Depth completion method for sparse depth map, computer device and storage medium

Abstract

The present invention discloses a depth completion method for a sparse depth map, a computer device and a storage medium. In the depth completion method, the process of training a neural network includes obtaining a color image and a true value of a depth map, performing equidistant sampling on the true value of the depth map to obtain a sparse depth map, extracting a multi-scale feature map from the color image and the sparse depth map, regressing the multi-scale feature map to obtain an initial depth map, calculating pixel correlation, performing multiple iterative filtering on the initial depth map to obtain a dense depth map, and performing multiple rounds of iterative processing on the dense depth map to train an image consistency optimization module. The present invention trains a neural network to improve the prediction effect of the image consistency optimization module on the pixel depth at the boundary of the dense depth map, so that the image consistency optimization module has the ability to predict the depth of its pixel points according to the dense depth map, thereby enabling the neural network to have the ability of depth completion. The present invention is widely used in the field of image processing technology.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a depth completion method for a sparse depth map, a computer device and a storage medium. Background Art

[0002] Depth perception is a basic task of three-dimensional vision. Through depth perception, the distance between the point on the object represented by each pixel in the image and a reference plane can be obtained, thereby providing important control parameters for applications in autonomous driving control, robotics, augmented reality, etc. Depth perception technology has made rapid progress in recent years, but how to obtain high-precision, high-resolution depth maps at low cost is still a challenging task. Low-cost, low-power depth sensors can usually only obtain low-resolution, very sparse depth maps, missing the depth information of a large number of pixels, while dense depth maps are required in actual applications.

[0003] Terminology explanation:

[0004] Depth map: refers to an image that uses the distance (depth) from the image collector to each point in the scene as the pixel value;

[0005] Sparse depth map: a depth map in which some pixel values ​​have clear depth values. Compared with dense depth maps, the number of pixels with clear depth values ​​in sparse depth maps accounts for a lower proportion of all pixels.

[0006] Dense depth map: a depth map in which some or all pixel values ​​have clear depth values. Compared with a sparse depth map, the number of pixels with clear depth values ​​in a dense depth map accounts for a higher proportion of all pixels. Summary of the invention

[0007] In view of at least one technical problem that the sparse depth map obtained in the current depth perception technology cannot meet the demand for dense depth maps in practical applications, the object of the present invention is to provide a depth completion method, a computer device and a storage medium for a sparse depth map.

[0008] In one aspect, an embodiment of the present invention includes a neural network training method, wherein the depth completion method of the sparse depth map includes:

[0009] Training a neural network; the neural network includes a first encoding network, a first decoding network, a pixel correlation calculation module, a pixel correlation optimization module and an image consistency optimization module;

[0010] Get the image to be processed;

[0011] Using the trained neural network, performing depth completion on the image to be processed;

[0012] The training of the neural network comprises:

[0013] Get color images;

[0014] Obtaining a true value of a depth map corresponding to the color image;

[0015] Performing equidistant sampling on the true value of the depth map to obtain a sparse depth map;

[0016] The first encoding network extracts a multi-scale feature map from the color image and the sparse depth map; the multi-scale feature map includes feature information of multiple different scales;

[0017] The first decoding network regresses the multi-scale feature map to obtain an initial depth map;

[0018] The pixel correlation calculation module calculates the pixel correlation according to the multi-scale feature map; the pixel correlation represents the degree of correlation between each pixel and adjacent pixels in the color image;

[0019] The pixel correlation optimization module performs multiple iterative filtering on the initial depth map to obtain a dense depth map;

[0020] The image consistency optimization module performs multiple rounds of iterative processing; in one round of iterative processing, the residual map of the iterative processing in this round is determined according to the target depth map and the color image, the processing result of the iterative processing in this round is determined according to the target depth map and the residual map, the loss function value is determined according to the processing result of the iterative processing in this round and the target depth map, and the parameters of the image consistency optimization module, the first encoding network, the first decoding network and the pixel correlation calculation module are adjusted according to the loss function value or the training process is ended; wherein, when the iterative processing in this round is the first round of iterative processing, the target depth map is the dense depth map, and when the iterative processing in this round is not the first round of iterative processing, the target depth map is the processing result of the previous round of iterative processing.

[0021] Furthermore, the performing equidistant sampling on the true value of the depth map to obtain a sparse depth map includes:

[0022] Establishing a grid; the distance between any two adjacent grid points in the grid is equal;

[0023] Taking the position corresponding to each of the grid points in the true value of the depth map as a reference point;

[0024] The reference point in the true value of the depth map and / or a point whose distance from the reference point does not exceed a distance threshold is used as a sampling point to sample the true value of the depth map to obtain the sparse depth map.

[0025] Furthermore, the extracting a multi-scale feature map from the color image and the sparse depth map includes:

[0026] Encoding the color image to obtain first feature encoding information;

[0027] Encode the sparse depth map to obtain second feature encoding information; the first feature encoding information and the second feature encoding information are in the same feature space;

[0028] fusing the first feature coding information and the second feature coding information to obtain third feature coding information;

[0029] The third feature encoding information is encoded to obtain the multi-scale feature map.

[0030] Further, calculating pixel correlation according to the multi-scale feature map includes:

[0031] Expanding feature information of each scale in the multi-scale feature map to the same size as the color image;

[0032] Acquire the channel dimension of the color image corresponding to the feature information of each scale;

[0033] The expanded feature information of each scale is spliced ​​together according to the corresponding channel dimension to obtain a spliced ​​feature map;

[0034] Perform multi-layer convolution processing on the spliced ​​feature map to obtain a multi-dimensional tensor; use the multi-dimensional tensor as the pixel correlation.

[0035] Further, determining the processing result of the iterative processing process in this round according to the target depth map and the residual map includes:

[0036] The sum of the target depth map and the residual map of this round of iterative processing is taken as the processing result of this round of iterative processing.

[0037] Further, determining the loss function value according to the processing result of the iterative processing process in this round includes:

[0038] For the processing result of the iterative processing process in round t, according to the formula Determine the loss function value; where Loss is the loss function value, D1 is the dense depth map, D gt is the true value of the depth map, is the result of the iterative process in the nth round, λ1 and is the weight coefficient, is the loss function.

[0039] Furthermore, the loss function is any one of the following functions or a combination of several functions: L1 function, L2 function, SmoothL1 function or Berhu function.

[0040] Furthermore, the using the trained neural network to perform depth completion on the image to be processed includes:

[0041] Inputting the image to be processed into a trained neural network for processing;

[0042] Obtaining a processing result of the neural network.

[0043] On the other hand, an embodiment of the present invention also includes a computer device, including a memory and a processor, wherein the memory is used to store at least one program, and the processor is used to load the at least one program to execute the depth completion method in the embodiment.

[0044] On the other hand, an embodiment of the present invention further includes a storage medium storing a program executable by a processor, wherein the program executable by the processor is used to execute the depth completion method in the embodiment when executed by the processor.

[0045] The beneficial effects of the present invention are: the depth completion method in the embodiment first trains the image consistency optimization module in the neural network, which can improve the prediction effect of the image consistency optimization module on the pixel depth at the boundary of the dense depth map, so that the image consistency optimization module has the ability to predict the depth of its pixel points based on the dense depth map. In each round of iterative processing during the training process, the processing result obtained has more accurate depth information than the processing result of the previous round of iterative processing. Therefore, the result obtained by processing the image consistency optimization module can obtain more accurate depth information than the dense depth map, which is equivalent to obtaining a dense depth map that is finer than the dense depth map input to the image consistency optimization module, thereby enabling the neural network to have the ability of depth completion; the trained neural network is used in the image processing method to perform depth completion on the image to be processed, thereby obtaining a dense depth map relative to the image to be processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of training a neural network in an embodiment;

[0047] Figure 2 A schematic diagram of training a neural network in an embodiment;

[0048] Figure 3 A schematic diagram of establishing a grid in an embodiment;

[0049] Figure 4This is a schematic diagram of the method of performing equidistant sampling of the true value of the depth map using the reference point itself as the sampling point in the embodiment;

[0050] Figure 5 3 is a schematic diagram of the principle of performing equidistant sampling on the true value of the depth map using non-reference points as sampling points in an embodiment. DETAILED DESCRIPTION

[0051] In this embodiment, the depth completion method of the sparse depth map includes the following steps:

[0052] S1. Train the neural network;

[0053] S2. Obtain the image to be processed;

[0054] S3. Use the trained neural network to perform depth completion on the image to be processed.

[0055] In step S1, the neural network to be trained includes a sampling module, a first encoding network, a first decoding network, a pixel correlation calculation module, a pixel correlation optimization module and an image consistency optimization module. By training the neural network, the neural network can have the performance of depth completion. Therefore, before describing the depth completion method, the neural network training method, that is, step S1, is described first.

[0056] Reference Figure 1 The neural network training method, that is, the process of training the neural network in step S1, includes the following steps:

[0057] P1. Get color image;

[0058] P2. Get the true value of the depth map corresponding to the color image;

[0059] P3. The sampling module performs equidistant sampling on the true value of the depth map to obtain a sparse depth map;

[0060] P4. The first encoding network extracts a multi-scale feature map from the color image and the sparse depth map; the multi-scale feature map includes feature information of multiple different scales;

[0061] P5. The first decoding network regresses the multi-scale feature map to obtain the initial depth map;

[0062] P6. The pixel correlation calculation module calculates the pixel correlation according to the multi-scale feature map; the pixel correlation represents the degree of correlation between each pixel and the adjacent pixels in the color image;

[0063] P7. The pixel correlation optimization module performs multiple iterative filtering on the initial depth map to obtain a dense depth map;

[0064] P8. The image consistency optimization module performs multiple rounds of iterative processing; in one round of iterative processing, the residual map of this round of iterative processing is determined according to the target depth map and the color image, the processing result of this round of iterative processing is determined according to the target depth map and the residual map, the loss function value is determined according to the processing result of this round of iterative processing and the target depth map, and the parameters of the image consistency optimization module, the parameters of the first encoding network, the parameters of the first decoding network and the parameters of the pixel correlation calculation module are adjusted according to the loss function value or the training process is ended; wherein, when this round of iterative processing is the first round of iterative processing, the target depth map is a dense depth map, and when this round of iterative processing is not the first round of iterative processing, the target depth map is the processing result of the previous round of iterative processing.

[0065] The principle of steps P1-P8 is as follows Figure 2 shown.

[0066] Before executing step P1, a pre-constructed RGB-D dataset can be used, in which each color image corresponds to a true depth map value, and the true depth map value can describe the depth value of some or all pixels in the corresponding color image, and the true depth map value is a dense depth value. In steps P1 and P2, the color image and the corresponding true depth map value are read from the RGB-D dataset.

[0067] In step P3, the sampling module performs equidistant sampling on the true value of the depth map to obtain a sparse depth map. When executing step P3, the sampling module specifically performs the following steps:

[0068] P301. Establish a grid; the distance between any two adjacent grid points in the grid is equal;

[0069] P302. Take the position corresponding to each grid point in the true value of the depth map as the reference point;

[0070] P303. Take the reference point in the true value of the depth map, or the point whose distance to the reference point does not exceed the distance threshold as the sampling point, sample the true value of the depth map to obtain a sparse depth map.

[0071] The grid established in step P301 has the characteristic of "the distance between any two adjacent grid points is equal", where "two adjacent grid points" may refer to two grid points connected by a line segment with no other grid points in the middle of the line segment. Figure 3 The equilateral triangle mesh shown is a mesh that meets the above characteristics. It is composed of multiple equilateral triangles. The vertices of each equilateral triangle can be regarded as grid points in the equilateral triangle mesh. The distance between two adjacent grid points is equal. Therefore, in step P302, the grid lines in the grid are removed and only the grid points are retained. Figure 4The result shown in FIG. 1 is that the reference points in the true value of the depth map are determined. These reference points correspond to the intersection points in the equilateral triangle mesh, so the distance between any two adjacent reference points is equal. Figure 4 The reference points shown are used as sampling points to sample the true value of the depth map, which can achieve equidistant sampling, wherein the result of sampling each sampling point is to obtain the depth value of the true value of the depth map at the location of the sampling point, and the depth values ​​collected from multiple sampling points form a sparse depth map in this embodiment. Figure 4 The reference points shown themselves are used as sampling points, e.g. Figure 5 , where point A1, point B and point C are all reference points, and they form a strict equilateral triangle △A1BC with a side length of L; point A2 is not a reference point, and the distance between point A2 and point A1 is ΔL. If a length threshold R expressed in percentage is set max , when R max Small enough (for example, R max Less than 10%) and ΔL / L<ΔR max , the difference caused by ΔL can be ignored, that is, △A2BC can be approximately regarded as an equilateral triangle. At this time, if the reference points B and C and the non-reference point A2 are used as sampling points, approximate equidistant sampling can also be achieved.

[0072] For a grid composed of equilateral triangles, the number of sampling points is related to the side length of the equilateral triangle. When the size of the true depth map remains unchanged and the entire true depth map is sampled, the smaller the side length of the equilateral triangle, the greater the number of sampling points. Therefore, the number of sampling points can be determined first, and the side length of each equilateral triangle can be determined based on the number of sampling points, and then the equilateral triangle grid can be established.

[0073] When executing step P3, the position of the reference point can also be determined by other means. For example, a grid composed of multiple equal-sized circles can be established, and the adjacent circles are tangent to each other. The center of each circle is used as the reference point, and equidistant sampling or approximately equidistant sampling can also be achieved.

[0074] Sampling a sparse depth map by equidistant sampling or approximately equidistant sampling, and using the sparse depth map in the subsequent training of the neural network is conducive to improving the generalization of the training of the neural network, especially compared with the method of randomly determining the sampling points, the optimization effect of generalization is more obvious. When the distance between adjacent sampling points is not strictly equal, but there is a certain deviation, it belongs to approximately equidistant sampling, and the effect of improving the generalization of training close to equidistant sampling can also be obtained.

[0075] The first encoding network and the first decoding network in steps P4 and P5 may be independent networks, or may be parts of the same encoding and decoding network, that is, the first encoding and decoding network.

[0076] In step P4, the first coding network processes the color image obtained in step P1 and the sparse depth map obtained in step P3 to extract a multi-scale feature map. When the first coding network executes step P4, specifically, the following steps may be performed:

[0077] P401. Encode the color image to obtain the first feature encoding information;

[0078] P402. Encode the sparse depth map to obtain second feature encoding information;

[0079] P403. Fusing the first feature coding information and the second feature coding information to obtain the third feature coding information;

[0080] P404. Encode the third feature coding information to obtain a multi-scale feature map.

[0081] In this embodiment, the first coding network includes multiple convolutional layers. When executing step P401, the first coding network encodes the color image through a part of the convolutional layers to obtain the first feature coding information. When executing step P402, the first coding network encodes the sparse depth map through another part of the convolutional layers to obtain the first feature coding information, and the first feature coding information and the second feature coding information are encoded into the same feature space. In step P403, the first coding network fuses the first feature coding information and the second feature coding information, and the result is the third feature coding information. In step P404, the first coding network then encodes the third feature coding information to obtain a multi-scale feature map.

[0082] The multi-scale feature map obtained by the first encoding network executing steps P401-P404 includes multiple feature information extracted from the color image and the sparse depth map, and these feature information correspond to different scales.

[0083] In step P5, the first decoding network regresses the multi-scale feature map obtained in step P4 to obtain an initial depth map. Specifically, the first decoding network is composed of multiple layers of deconvolution layers, and the first decoding network gradually restores the multi-scale feature map to a size like a color image, and regresses to obtain the initial depth map D0 in this process.

[0084] In step P6, the pixel correlation calculation module calculates the pixel correlation according to the multi-scale feature map.

[0085] The basis for executing step P6 is the local consistency of the distribution of depth values. In the depth map obtained by shooting and processing a real scene, the depth value distribution in most areas has local consistency, that is, for any pixel point, its corresponding depth value and the depth value of the pixel point in its adjacent area are more likely to be closer than the depth value of the pixel point in its non-adjacent area. In applying this principle, it can be directly considered that for any pixel point, its corresponding depth value and the depth value of the pixel point in its adjacent area are always closer than the depth value of the pixel point in its non-adjacent area. Therefore, for a pixel point in the depth map, the long-range information (such as the depth information of the pixel point in its non-adjacent area) is invalid, and the local depth information is similar or consistent.

[0086] Since the neighborhood pixel information of different receptive field sizes has been written into the multi-scale feature map during the extraction process of the multi-scale feature map, based on the principle of local depth consistency, the pixel correlation calculation module in this embodiment may specifically perform the following steps when executing step P6:

[0087] P601. Expand the feature information of each scale in the multi-scale feature map to the same size as the color image;

[0088] P602. Obtain the channel dimension of the color image corresponding to the feature information of each scale;

[0089] P603. Splice the expanded feature information of each scale together according to the corresponding channel dimension to obtain a spliced ​​feature map;

[0090] P604. Perform multi-layer convolution processing on the spliced ​​feature map to obtain a multi-dimensional tensor; use the multi-dimensional tensor as the pixel correlation.

[0091] In step P601, the feature information of each scale in the multi-scale feature map is expanded to the same size as the color image. Specifically, the original small-scale feature information can be copied multiple times and combined into an image with the same resolution size as the color image, or the small-scale feature information can be expanded to the same resolution size as the color image through interpolation methods such as nearest neighbor interpolation. In the process of executing step P601, the diffusion of the multi-scale feature map is naturally limited to the local area.

[0092] In step P602, the channel dimension of the color image corresponding to the feature information of each scale is obtained. In step P603, the expanded multi-scale feature maps are spliced ​​together according to the corresponding channel dimensions to obtain a spliced ​​feature map.

[0093] In step P604, multi-layer convolution processing is performed on the spliced ​​feature map obtained in step P603 to obtain a multi-dimensional tensor. The multi-dimensional tensor obtained in step P604 is the pixel correlation to be obtained by executing step P6.

[0094] A specific example of steps P601-P604 is: suppose the size of the original image, i.e., the color image, is H×W. If the multi-scale feature map has N layers, i.e., the multi-scale feature map contains feature information of N different scales, the sizes corresponding to these N feature information are respectively Among them C n Represents the number of channels of the nth layer of feature information in the multi-scale feature map. A certain interpolation method is used to expand the feature information of each layer to the size of the original image. The expanded n-layer feature information is C n ×H×W,n=1,2,…,N. Concatenate the expanded feature information according to the channel dimension to obtain After that, the pixel correlation is output through multiple convolution layers. Taking a 3×3 neighborhood as an example, it is necessary to measure the relationship between the central pixel and the 8 pixels in the neighborhood. Therefore, executing steps P601-P604 will obtain a 8×H×W multi-dimensional tensor, which can represent the correlation between all pixels in the entire color image and the pixels in the neighborhood, that is, it can be used as the pixel correlation to be obtained in step P6.

[0095] In step P7, the pixel correlation optimization module uses the pixel correlation calculated by the pixel correlation calculation module in step P6 to guide the optimization process of the initial depth map. Specifically, the pixel correlation optimization module normalizes the pixel correlation of 8×H×W and constructs H×W position-related convolution kernels, corresponding to H×W pixels respectively, and uses this set of convolution kernels to iteratively filter the initial depth map multiple times to output a dense depth map D1.

[0096] In step P8, the image consistency optimization module performs multiple rounds of iterative processing on the color image obtained in step P1 and the dense depth map obtained in step P7.

[0097] When executing step P8, the images to be processed by the image consistency optimization module, except for the color images, are collectively referred to as target depth maps in each round. Specifically, in the first round of iterative processing, the target depth map refers to the dense depth map obtained in step P7. In each round of iterative processing after the first round of iterative processing, the target depth map refers to the processing result obtained by executing the previous round of iterative processing. Therefore, each round of iterative processing performed by the image consistency optimization module can be described as:

[0098] In the first round of iterative processing, the image consistency optimization module determines the residual map of this round (first round) of iterative processing according to the dense depth map obtained in step P7 and the color image obtained in step P1; determines the processing result of this round (first round) of iterative processing according to the dense depth map and the residual map of this round (first round), determines the loss function value according to the processing result of this round of iterative processing, adjusts the parameters of the image consistency optimization module, the first encoding network, the first decoding network and the pixel correlation calculation module according to the loss function value (executes the second round of iterative processing after adjusting the parameters of the image consistency optimization module), or ends all training processes;

[0099] During the t (t>1)th round of iterative processing, the image consistency optimization module determines the residual map of this round (tth round) of iterative processing according to the processing result obtained in the t-1th round of iterative processing and the color image obtained in step P1; determines the processing result of this round (tth round) of iterative processing according to the processing result obtained in the t-1th round of iterative processing and the residual map of this round (tth round), determines the loss function value according to the processing result of this round of iterative processing, adjusts the parameters of the image consistency optimization module, the first encoding network, the first decoding network and the pixel correlation calculation module according to the loss function value (execute the t+1th round of iterative processing after adjusting the parameters of the image consistency optimization module), or ends the entire training process.

[0100] Since the principles of each round of iterative processing are the same, the difference between the first round of iterative processing and the other rounds of iterative processing mainly lies in that what is processed is the dense depth map obtained in step P7 or the processing result obtained in the previous round of iterative processing. Therefore, the entire iterative processing can be explained by explaining the tth (t>1) round of iterative processing.

[0101] During the tth round of iterative processing, the image consistency optimization module performs the following steps:

[0102] P801. The processing results obtained according to the t-1th round of iterative processing The residual image ΔD of this round (tth round) of iterative processing is determined by comparing it with the color image obtained in step P1. t ;

[0103] Specifically, the image consistency optimization module includes a second codec network, and the second codec network is a cyclic codec network. When executing step P801, the processing result obtained in the t-1th round of iterative processing is The color image obtained in step P1 is input into the second codec network together, and the residual image ΔD is output. t , residual graph ΔD tIndicates the processing result obtained in the t-1th round of iterative processing If you need to adjust it to be consistent with the color image, you need to adjust the depth value.

[0104] P802. The processing results obtained according to the t-1th round of iterative processing and the residual graph of this round (round t) ΔD t , determine the processing result of this round (round t) of iterative processing Right now

[0105] P803. Based on the processing results of this round (round t) of iterative processing Determine the loss function value. Specifically, according to the formula Determine the loss function value, where Loss is the loss function value, D1 is the dense depth map obtained in step P7, and D gt is the true value of the depth map obtained in step P2, is the result of the nth (n<t)th round of iterative processing, λ1 and is the weight coefficient, is the loss function. Specifically, It can be L1 function, L2 function, SmoothL1 function, Berhu function or a combination thereof, for example, It can be an L1 function, It can be an L2 function, It can also be a combination of L1 function and Berhu function.

[0106] P804. Set a loss value threshold, compare the loss function value determined in step P803 with the loss value threshold, if the loss function value is less than the loss value threshold, end the iterative processing process, and do not execute the t+1th round and subsequent rounds of iterative processing, that is, end the entire training process of the neural network; if the loss function value is not less than the loss value threshold, execute the t+1th round of iterative processing.

[0107] By training the image consistency optimization module through steps P801-P804, the prediction effect of the image consistency optimization module on the pixel depth at the boundary of the dense depth map can be improved, so that the image consistency optimization module has the ability to predict the depth of its pixel points based on the dense depth map. In each round of iterative processing during the training process, the processing result obtained has more accurate depth information than the processing result of the previous round of iterative processing. Therefore, the result obtained by the processing of the image consistency optimization module can obtain more accurate depth information than the dense depth map, which is equivalent to obtaining a dense depth map that is more refined than the dense depth map input to the image consistency optimization module. Specifically, the training method for the neural network in this embodiment also includes the following advantages:

[0108] (1) The consistency of the distribution of the sparse depth map in the image is maintained during sampling, so the diffusion range of the depth value learned by the neural network is relatively stable. Compared with the sparse depth map obtained by random sampling, the present invention can effectively improve the generalization of the neural network. The present invention uses a sparse depth map with variable density during training to improve the adaptability of the neural network to sparse inputs of different densities during the training stage. Compared with the sparse depth map with fixed density commonly used in existing training, the network trained with the sparse depth map obtained by the present invention has better robustness;

[0109] (2) Pixel correlation calculation method based on local consistency. The present invention explicitly introduces the local consistency constraint of depth distribution in real scenes when calculating pixel correlation, effectively improving the measurement accuracy of pixel correlation. Experimental results show that the optimization effect of the pixel correlation calculated based on the present invention on the initial depth map is better than other methods.

[0110] (3) Depth map optimization method based on image consistency. The depth map predicted by the prior art often has the problem of blurred depth edge prediction. The present invention adopts an optimization method based on image consistency to maximize the use of image information to correct the depth map, which can effectively improve the prediction effect of the scene fine structure.

[0111] In this embodiment, the neural network trained through steps P1-P8 is applied to the processing of the sparse depth map, and steps S2 and S3 in the depth completion method of the sparse depth map can be executed.

[0112] S1. Train the neural network;

[0113] S2. Obtain the image to be processed;

[0114] S3. Use the trained neural network to perform depth completion on the image to be processed.

[0115] The image to be processed obtained in step S2 is a sparse depth map relative to the image obtained after depth completion by the neural network.

[0116] In step S3, the image to be processed, which belongs to the sparse depth map, is input into the neural network. The neural network can predict the depth information of the sparse depth map. The output result of the neural network is a dense depth map relative to the sparse depth map, thereby achieving the effect of depth completion.

[0117] When executing step S3, the neural network used can be obtained by removing the sampling module from the neural network trained in steps P1-P8, that is, the neural network used in steps S1-S3 can only consist of the first encoding network, the first decoding network, the pixel correlation calculation module, the pixel correlation optimization module and the image consistency optimization module. In step S3, after the image to be processed is input into the neural network, the neural network can perform the following steps based on the principles of steps P1-P8:

[0118] S301. Obtain the image to be processed;

[0119] S302. Obtain the true value of the depth map corresponding to the image to be processed;

[0120] S303. The sampling module performs equidistant sampling on the true value of the depth map to obtain a sparse depth map;

[0121] S304. The first encoding network extracts a multi-scale feature map from the image to be processed and the sparse depth map; the multi-scale feature map includes feature information of multiple different scales;

[0122] S305. The first decoding network regresses the multi-scale feature map to obtain an initial depth map;

[0123] S306. The pixel correlation calculation module calculates the pixel correlation according to the multi-scale feature map; the pixel correlation represents the degree of correlation between each pixel and adjacent pixels in the image to be processed;

[0124] S307. The pixel correlation optimization module performs multiple iterative filtering on the initial depth map to obtain a dense depth map;

[0125] S308. The second encoding and decoding network in the image consistency optimization module encodes and predicts the dense depth map in step S307, and outputs a depth map that is more accurate than the dense depth map in step S307, thereby achieving depth completion of the image to be processed.

[0126] A computer program for executing the depth completion method for the sparse depth map in the present embodiment can be written and written into a computer device or a storage medium. When the computer program is read out and executed, the depth completion method for the sparse depth map in the present embodiment is executed, thereby achieving the same technical effect as the depth completion method for the sparse depth map in the embodiment.

[0127] It should be noted that, unless otherwise specified, when a feature is referred to as "fixed" or "connected" to another feature, it may be directly fixed or connected to another feature, or it may be indirectly fixed or connected to another feature. In addition, the descriptions of up, down, left, right, etc. used in this disclosure are only relative to the relative positional relationship of the components of the present disclosure in the accompanying drawings. The singular forms of "a", "said" and "the" used in this disclosure are also intended to include the plural forms, unless the context clearly indicates other meanings. In addition, unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as those generally understood by technicians in this technical field. The terms used in the specification of this embodiment are only for describing specific embodiments, not for limiting the present invention. The term "and / or" used in this embodiment includes any combination of one or more related listed items.

[0128] It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided by the present embodiment is only intended to better illustrate embodiments of the present invention, and unless otherwise required, the scope of the present invention will not be limited.

[0129] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose.

[0130] In addition, the operations of the process described in this embodiment may be performed in any suitable order, unless otherwise indicated in this embodiment or otherwise clearly contradicted by the context. The process described in this embodiment (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed jointly on one or more processors, by hardware or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0131] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or a portion thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described in this embodiment includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0132] The computer program can be applied to input data to perform the functions described in the present embodiment, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents a physical and tangible object, including a specific visual depiction of the physical and tangible object produced on the display.

[0133] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical scheme and / or implementation method may have various modifications and changes.

Claims

1. A depth completion method for a sparse depth map, characterized in that: The depth completion method of the sparse depth map comprises: Training a neural network; the neural network includes a first encoding network, a first decoding network, a pixel correlation calculation module, a pixel correlation optimization module and an image consistency optimization module; Get the image to be processed; Using the trained neural network, performing depth completion on the image to be processed; The training of the neural network comprises: Get color images; Obtaining a true value of a depth map corresponding to the color image; Performing equidistant sampling on the true value of the depth map to obtain a sparse depth map; The first encoding network extracts a multi-scale feature map from the color image and the sparse depth map; the multi-scale feature map includes feature information of multiple different scales; The first decoding network regresses the multi-scale feature map to obtain an initial depth map; The pixel correlation calculation module calculates the pixel correlation according to the multi-scale feature map; the pixel correlation represents the degree of correlation between each pixel and adjacent pixels in the color image; The pixel correlation optimization module performs multiple iterative filtering on the initial depth map to obtain a dense depth map; The image consistency optimization module performs multiple rounds of iterative processing; in one round of iterative processing, the residual map of the iterative processing in this round is determined according to the target depth map and the color image, the processing result of the iterative processing in this round is determined according to the target depth map and the residual map, the loss function value is determined according to the processing result of the iterative processing in this round and the target depth map, and the parameters of the image consistency optimization module, the first encoding network, the first decoding network and the pixel correlation calculation module are adjusted according to the loss function value or the training process is ended; wherein, when the iterative processing in this round is the first round of iterative processing, the target depth map is the dense depth map, and when the iterative processing in this round is not the first round of iterative processing, the target depth map is the processing result of the previous round of iterative processing.

2. The depth completion method according to claim 1, characterized in that: The step of performing equidistant sampling on the true value of the depth map to obtain a sparse depth map includes: Establishing a grid; the distance between any two adjacent grid points in the grid is equal; Taking the position corresponding to each of the grid points in the true value of the depth map as a reference point; The reference point in the true value of the depth map and / or a point whose distance from the reference point does not exceed a distance threshold is used as a sampling point to sample the true value of the depth map to obtain the sparse depth map.

3. The depth completion method according to claim 1, characterized in that: The step of extracting a multi-scale feature map from the color image and the sparse depth map comprises: Encoding the color image to obtain first feature encoding information; Encode the sparse depth map to obtain second feature encoding information; the first feature encoding information and the second feature encoding information are in the same feature space; fusing the first feature coding information and the second feature coding information to obtain third feature coding information; The third feature encoding information is encoded to obtain the multi-scale feature map.

4. The depth completion method according to claim 1, characterized in that: The calculating pixel correlation according to the multi-scale feature map comprises: Expanding feature information of each scale in the multi-scale feature map to the same size as the color image; Acquire the channel dimension of the color image corresponding to the feature information of each scale; The expanded feature information of each scale is spliced ​​together according to the corresponding channel dimension to obtain a spliced ​​feature map; Perform multi-layer convolution processing on the spliced ​​feature map to obtain a multi-dimensional tensor; use the multi-dimensional tensor as the pixel correlation.

5. The depth completion method according to claim 1, characterized in that: Determining the processing result of the iterative processing process in this round according to the target depth map and the residual map includes: The sum of the target depth map and the residual map of this round of iterative processing is taken as the processing result of this round of iterative processing.

6. The depth completion method according to claim 1, characterized in that: Determining the loss function value according to the processing result of the iterative processing process in this round includes: For the processing result of the iterative processing process in round t, according to the formula Determine the loss function value; where Loss is the loss function value, D1 is the dense depth map, and D gt is the true value of the depth map, is the result of the iterative process in the nth round, λ1 and is the weight coefficient, is the loss function.

7. The depth completion method according to claim 6, characterized in that: The loss function is any one of the following functions or a combination of several functions: L1 function, L2 function, SmoothL1 function or Berhu function.

8. The depth completion method according to any one of claims 1 to 7, characterized in that: The using the trained neural network to perform depth completion on the image to be processed includes: Inputting the image to be processed into a trained neural network for processing; Obtaining a processing result of the neural network.

9. A computer device, characterized in that: The device comprises a memory and a processor, wherein the memory is used to store at least one program, and the processor is used to load the at least one program to execute the depth completion method according to any one of claims 1 to 8.

10. A storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to execute the depth completion method according to any one of claims 1 to 8 when executed by the processor.

Images