Depth completion method, device, electronic device and storage medium

Abstract

The present invention provides a depth completion method, device, electronic device and storage medium, which are applied to the field of data processing technology. The method comprises: obtaining an RGB image, a sparse depth map and a camera intrinsic parameter matrix; performing normal estimation processing on the sparse depth map to obtain a sparse normal map; performing fusion coding processing on the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map through a depth completion network to obtain a depth code, and decoding the depth code to obtain a dense depth map; wherein the image density of the dense depth map is higher than the image density of the sparse depth map.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a depth completion method, device, electronic device and storage medium. Background Art

[0002] Depth sensors are widely used in technologies such as autonomous driving, 3D reconstruction, augmented reality, and pose estimation. Common depth sensors include RGB-D cameras and LiDAR. However, the depth map directly obtained by the depth sensor is usually too sparse and contains a lot of noise, so it cannot be directly applied to subsequent tasks.

[0003] In the prior art, the depth completion network is generally trained with sparse depth maps as input and dense depth truth values ​​as supervision. However, since dense depth truth values ​​are difficult to obtain and single-modal input can only provide less semantic information, the network accuracy of existing depth completion networks is low. Summary of the invention

[0004] The present invention provides a depth completion method, device, electronic device and storage medium, which are used to solve the problem of low network accuracy of existing depth completion networks.

[0005] The present invention provides a depth completion method, comprising: acquiring an RGB image, a sparse depth map and a camera intrinsic parameter matrix; performing normal estimation processing on the sparse depth map to obtain a sparse normal map; performing fusion encoding processing on the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map through a depth completion network to obtain a depth code, and decoding the depth code to obtain a dense depth map; wherein the image density of the dense depth map is higher than the image density of the sparse depth map.

[0006] According to a depth completion method provided by the present invention, the sparse normal map includes normal vectors of all valid pixels in the sparse depth map; the normal estimation processing of the sparse depth map to obtain the sparse normal map includes: back-projecting the valid pixels in the sparse depth map through the camera intrinsic parameter matrix to obtain a sparse point cloud; determining a set of neighboring points for each valid point in the sparse point cloud; determining a target fitting plane for the neighboring point set, wherein the distance between the neighboring points in the neighboring point set and the target fitting plane is the smallest; and determining the normal vector of the target fitting plane as the normal vector of the corresponding valid pixel.

[0007] According to a depth completion method provided by the present invention, the sparse depth map is subjected to normal estimation processing to obtain a sparse normal map, including: determining a ground area image of the sparse depth map according to a first proportional coefficient; and determining the direction of a normal vector in the ground area image whose angle with the vertical direction is less than a first angle as vertically upward.

[0008] According to a depth completion method provided by the present invention, the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map are subjected to fusion coding processing to obtain a depth code, and the depth code is decoded to obtain a dense depth map. The method further includes: performing a maximum pooling and a minimum pooling operation on the sparse depth map using kernels of different sizes to obtain a dense sparse depth map; and performing a maximum pooling and a minimum pooling operation on the sparse normal map based on the second normal component to obtain a dense sparse normal map.

[0009] According to a depth completion method provided by the present invention, the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map are subjected to fusion coding processing to obtain a depth code, comprising: obtaining a depth code output by a previous layer; determining a three-dimensional position code according to the depth code of the sparse depth map and the camera intrinsic parameter matrix, and determining a three-dimensional normal code of the three-dimensional position code; splicing and fusing the normal code of the sparse normal map and the three-dimensional normal code to obtain a new normal code; fusing the new normal code, the image code of the RGB image, the three-dimensional position code and the depth code output by the previous layer to obtain the depth code.

[0010] According to a depth completion method provided by the present invention, after the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map are subjected to fusion encoding processing to obtain the depth code, the method further includes: decoding the depth code to obtain a dense normal map.

[0011] According to a depth completion method provided by the present invention, the loss function of the depth completion network is: in, represents the loss of photometric consistency, represents the normal-depth consistency loss, represents the sparse depth consistency loss, represents the local smoothing loss; ω ph ,ω nd ,ω sd and ω sm is the corresponding weight balance coefficient.

[0012] The present invention also provides a depth completion device, comprising: an acquisition module and a processing module; the acquisition module is used to acquire an RGB image, a sparse depth map and a camera intrinsic parameter matrix; the processing module is used to perform normal estimation processing on the sparse depth map to obtain a sparse normal map; the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map are fused and encoded by a depth completion network to obtain a depth code, and the depth code is decoded to obtain a dense depth map; wherein the image density of the dense depth map is higher than the image density of the sparse depth map.

[0013] According to a depth completion device provided by the present invention, the sparse normal map includes normal vectors of all valid pixels in the sparse depth map; the processing module is used to: perform back-projection processing on the valid pixels in the sparse depth map through the camera intrinsic parameter matrix to obtain a sparse point cloud; determine a set of adjacent points for each valid point in the sparse point cloud; determine a target fitting plane for the adjacent point set, wherein the distance between the adjacent points in the adjacent point set and the target fitting plane is the smallest; and determine the normal vector of the target fitting plane as the normal vector of the corresponding valid pixel.

[0014] According to a depth completion device provided by the present invention, the processing module is used to: determine a ground area image of the sparse depth map according to a first proportional coefficient; and determine the direction of a normal vector in the ground area image whose angle with the vertical direction is less than a first angle as vertically upward.

[0015] According to a depth completion device provided by the present invention, the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map are subjected to fusion coding processing to obtain a depth code, and the depth code is decoded to obtain a dense depth map. The processing module is used to: use kernels of different sizes to perform quasi-maximum pooling and quasi-minimum pooling operations on the sparse depth map to obtain a dense sparse depth map; perform quasi-maximum pooling and quasi-minimum pooling operations on the sparse normal map based on the second normal component to obtain a dense sparse normal map.

[0016] According to a depth completion device provided by the present invention, the processing module is used to: obtain the depth code output by the previous layer; determine the three-dimensional position code according to the depth code of the sparse depth map and the camera intrinsic parameter matrix, and determine the three-dimensional normal code of the three-dimensional position code; perform splicing and fusion processing on the normal code of the sparse normal map and the three-dimensional normal code to obtain a new normal code; and perform fusion processing on the new normal code, the image code of the RGB image, the three-dimensional position code and the depth code output by the previous layer to obtain the depth code.

[0017] According to a depth completion device provided by the present invention, after the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map are subjected to fusion encoding processing to obtain the depth code, the processing module is further used to: decode the depth code to obtain a dense normal map.

[0018] According to a depth completion device provided by the present invention, the loss function of the depth completion network is: in, represents the loss of photometric consistency, represents the normal-depth consistency loss, represents the sparse depth consistency loss, represents the local smoothing loss; ω ph ,ω nd ,ω sd and ω sm is the corresponding weight balance coefficient.

[0019] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described depth completion methods are implemented.

[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the depth completion methods described above are implemented.

[0021] The depth completion method, device, electronic device and storage medium provided by the present invention can obtain an RGB image, a sparse depth map and a camera intrinsic parameter matrix; perform normal estimation processing on the sparse depth map to obtain a sparse normal map; perform fusion encoding processing on the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map through a depth completion network to obtain a depth code, and perform decoding processing on the depth code to obtain a dense depth map; wherein the image density of the dense depth map is higher than the image density of the sparse depth map. Through this scheme, the sparse depth map can be subjected to normal estimation processing to obtain a sparse normal map, and fusion encoding and decoding based on the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map can be performed to obtain a dense depth map with a higher image density. Since the sparse normal map is added in the process of image depth completion, it can not only provide more spatial geometric information, optimize the performance of the sparse depth map on the object boundary, and suppress noise, but also multimodal input can provide more semantic information for the depth completion network, thereby improving the network accuracy of the depth completion network. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 This is one of the flow charts of the depth completion method provided by the present invention;

[0024] Figure 2 It is a schematic diagram of the prediction process of the deep completion network provided by the present invention;

[0025] Figure 3 is a schematic diagram of determining a normal vector of a pixel point provided by the present invention;

[0026] Figure 4 It is a schematic diagram of the process of performing a densification operation on a sparse depth map provided by the present invention;

[0027] Figure 5 It is a schematic diagram of the process of multimodal fusion provided by the present invention;

[0028] Figure 6 is a schematic diagram of a process for determining photometric consistency loss provided by the present invention;

[0029] Figure 7 is a structural schematic diagram of a depth completion device provided by the present invention;

[0030] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] It should be noted that, in the embodiments of the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0033] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present invention is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0034] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, the words "first", "second", etc. are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. are not limiting the quantity and execution order.

[0035] The embodiments of the present invention describe some exemplary embodiments for the purpose of explanation. It should be understood that the present invention can be implemented in other ways that are not specifically shown in the drawings.

[0036] The above implementation is described in detail below with reference to specific embodiments and drawings.

[0037] like Figure 1 As shown, an embodiment of the present invention provides a depth completion method, which can be applied to a depth completion device. The depth completion method may include S101-S103:

[0038] S101. A depth completion device obtains an RGB image, a sparse depth map, and a camera intrinsic parameter matrix.

[0039] It should be noted that the camera intrinsic parameter matrix refers to the shooting parameters of the camera that shoots the RGB image. The sparse depth map is the image that needs to be completed in depth.

[0040] S102: The depth completion device performs normal estimation processing on the sparse depth map to obtain a sparse normal map.

[0041] Optionally, the sparse normal map includes normal vectors of all valid pixels in the sparse depth map; the depth completion device performs normal estimation processing including: back-projecting the valid pixels in the sparse depth map through the camera intrinsic parameter matrix to obtain a sparse point cloud; determining a set of neighboring points for each valid point in the sparse point cloud; determining a target fitting plane for the neighboring point set, wherein the distance between the neighboring points in the neighboring point set and the target fitting plane is the smallest; and determining the normal vector of the target fitting plane as the normal vector of the corresponding valid pixel.

[0042] like Figure 2 As shown, the depth completion device can generate a sparse normal map from a sparse depth map by using a method based on local surface fitting. The main idea is to back-project the sparse depth map into three-dimensional space, calculate the local fitting plane in the least squares sense for the k-neighborhood of each point in the generated point cloud, and use the normal of the local plane as the normal of the point.

[0043] Specifically, the depth completion device can first perform back-projection processing on the valid pixels in the sparse depth map through the camera intrinsic parameter matrix to obtain a sparse point cloud; then, by establishing a KD tree, the neighboring point set of each valid point in the sparse point cloud is determined, and the neighboring point set can be expressed as P:{p 1 ,p 2 ,…,p i ,…,p k}; Then, search for the target fitting plane of k adjacent points, that is, find a plane z = ax + by + c, so that the distance between these k points and this plane is the shortest. Then, according to the idea of ​​least squares method, find a set of plane parameters a, b, c so that S = ∑ (ax i +by i +cz) 2 minimum, that is, the derivatives of S with respect to a, b, and c are all zero, as shown in formula (1):

[0044]

[0045] By deducing formula (1), we can get formula (2):

[0046]

[0047] Arranging formula (2) yields formula (3):

[0048]

[0049] Changing formula (3) into a matrix equation gives formula (4):

[0050]

[0051] By solving the above formula (4), the plane parameters a, b, c of the target fitting plane can be obtained.

[0052] The equation in formula (4) can be simply expressed as A·X=b. Through matrix operation, we can get X=A -1 b. At this point, the plane equation z = ax + by + c can be obtained, and the normal direction of this plane can be expressed as (a, b, -1). After normalization, the normal vector of the target fitting plane can be obtained, that is, the normal vector of the corresponding valid pixel.

[0053] For example, Figure 3 As shown, taking pixel r as an example, the small dots around pixel r represent the k adjacent points searched. By fitting the plane using the least squares method, the target fitting plane D can be obtained. The normal vector n of the target fitting plane D can be used as the normal vector of pixel r.

[0054] Optionally, the sparse normal map generated by the above method can better reflect the geometric features in the scene. However, the normal vectors generated by some pixels have directional problems. After observation, such problems mostly occur in the ground area. Since the lower half of the image in the outdoor scene is mostly the ground area, the depth completion device can first determine the ground area image of the sparse depth map according to the first proportional coefficient; and then determine the direction of the normal vector in the ground area image whose angle with the vertical direction is less than the first angle as vertically upward. In this way, the direction problem of the normal map can be better solved.

[0055] Optionally, the first proportionality coefficient may be one third, and the first angle may be 5°.

[0056] S103, the depth completion device performs fusion encoding processing on the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map through a depth completion network to obtain a depth code, and decodes the depth code to obtain a dense depth map.

[0057] The image density of the dense depth map is higher than the image density of the sparse depth map.

[0058] Alternatively, for data that is too sparse, simple convolution operations usually cannot achieve the desired effect, because all pixels in the image are convolved regardless of whether they are valid or not, which introduces a large degree of randomness during training and testing, resulting in unstable performance. Figure 2As shown in the figure, before the sparse depth map and sparse normal map are input into the depth completion network, the sparse depth map needs to be densified by the depth densification module, and the sparse normal map needs to be densified by the normal densification module to avoid the sparse convolution problem. Since the geometric meanings represented by the sparse normal map and the sparse depth map are quite different, different densification strategies need to be designed:

[0059] like Figure 4 As shown, for a sparse depth map, the depth completion device can use a depth densification module to perform maximum pooling and minimum pooling operations on the sparse depth map using kernels of different sizes, thereby obtaining a densified sparse depth map.

[0060] Specifically, the depth densification module uses a series of minimum pooling and maximum pooling layers with different kernel sizes to perform multi-scale densification on the input sparse depth map. The size of the pooling kernel can be selected according to the sparsity of the point cloud collected by the radar.

[0061] Before the sparse depth map is input into the minimum pooling layer, it is necessary to preprocess the sparse depth map to avoid the negative impact of invalid pixels on pooling. That is, all pixels less than or equal to zero in the sparse depth map d are set to infinity:

[0062]

[0063] After that, the processed sparse depth map d′ is input into the minimum pooling layer with a pooling kernel size of k×k:

[0064] p = minpool(d′,k);

[0065] Finally, reset the infinite points caused by large void regions to zero:

[0066]

[0067] It should be noted that the purpose of selecting different pooling kernel sizes and using both maximum and minimum pooling is to preserve the detailed features of the scene as much as possible while densifying the sparse depth map, while balancing the processing of distant and near scenes. Among them, selecting a larger pooling kernel can effectively densify the depth map, while using a smaller pooling kernel can better preserve the details of the scene.

[0068] After the pooling operation, the depth densification module can weigh the outputs of multiple pooling layers through three 1×1 convolutions, and then fuse them with the original sparse depth map through a 3×3 convolution layer, finally generating a relatively dense depth representation.

[0069] It should be noted that the convolution operation is performed to better fuse the data obtained by selecting different pooling kernels and pooling methods, and connect it with the original sparse depth map through residual links to obtain the final dense depth representation.

[0070] For a sparse normal map, the depth completion device may perform a quasi-max pooling and a quasi-min pooling operation on the sparse normal map based on the second normal component, thereby obtaining a densified sparse normal map.

[0071] Specifically, for a single-channel depth map, maximum pooling and minimum pooling have clear physical meanings and operation methods. Through the pooling operation, the scene details and features can be retained while completing the densification. Compared with the depth map, the normal n×n x ,n y ,n z ) T and two degrees of freedom (||n|| 2 =1) has more complex geometric meanings. Processing it only through simple pooling operations will lead to serious data loss problems. Therefore, it is necessary to design a normal densification module based on its characteristics.

[0072] At present, the most common outdoor depth completion scenario is usually applied to autonomous driving technology. This type of depth map and the corresponding RGB image are collected by installing one or more street view cameras and on-board lidar on the work vehicle. The main axis of the camera is mostly facing forward and parallel to the ground. That is to say, in the camera coordinate system, the x-axis is parallel to the ground, the y-axis is perpendicular to the ground, and the z-axis points forward and is parallel to the ground.

[0073] Based on the above observations, the y-axis component of the normal in the camera coordinate system can reflect the geometric characteristics of the scene to a certain extent. Specifically, if the absolute value of the second component of the normal n |n y | is larger, the point is more likely to be on the ground, roof, or other horizontal plane; if the absolute value of the second component of the normal n |n y | is smaller, then the point is more likely to be on a vertical plane such as a wall or a car body. Therefore, the present invention uses the absolute value of the second component of the surface normal n |n y | is used as a reference item and the normal densification is performed to preserve the geometric information on both the plane and the elevation.

[0074] For example, a quasi-max pooling operation is performed on the normal map using a kernel of size n×n based on the second component of the normal. For a pixel p in the normal map with a missing normal, if there are no valid pixels in the n×n neighborhood of p, then after the operation, the normal of pixel p is still missing. However, if there are one or more pixels in the neighborhood of p that are valid points and can provide normal information, it is recorded as {n(p1 ),n(p 2 ),…,n(p m )}, where m is the number of valid values ​​in the neighborhood, then it can be stipulated that the normal n(p) of pixel point p can be replaced by a point in the neighborhood, that is, in It can be determined according to formula (5):

[0075] i * = arg max{|n y (p i )||i=1,2,…,m} (5);

[0076] Similarly, the second component based on the normal n |n y | is a minimum pooling operation, that is, finding the absolute value of the second normal component except the invalid point in the neighborhood |n y |The point with the minimum value replaces the normal of the pixel whose normal is missing with the normal of this point.

[0077] In the normal densification module, the present invention also uses kernels of different sizes to perform quasi-max pooling and quasi-min pooling operations, which have roughly the same meaning as the depth densification module. The use of larger kernels is to better complete the densification task, and the use of smaller kernels is to retain more detailed features. Quasi-max pooling and quasi-min pooling are designed based on the physical meaning of the normal. In the quasi-max pooling operation, by retaining the absolute value of the second component of the normal |n y |A larger position can better preserve the planar features of the image; similarly, the absolute value of the second normal component |n is retained by the minimum pooling method y |The smaller the location, the more favorable the treatment of the geometric features of the facade.

[0078] Similar to the depth densification module, the multiple feature maps output by the normal densification module after the pooling operation can be fused and weighed through 3 layers of 1×1 convolutions. The output of multiple layers of 1×1 convolutions is fused with the original sparse depth map through a 3×3 convolution layer to produce a relatively dense normal representation.

[0079] Optionally, the depth completion device performs fusion coding processing on the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map to obtain a depth code, including: obtaining the depth code output by the previous layer; determining the three-dimensional position code according to the depth code of the sparse depth map and the camera intrinsic parameter matrix, and determining the three-dimensional normal code of the three-dimensional position code; splicing and fusing the normal code of the sparse normal map and the three-dimensional normal code to obtain a new normal code; fusing the new normal code, the image code of the RGB image, the three-dimensional position code and the depth code output by the previous layer to obtain the depth code.

[0080] Specifically, Figure 5 As shown in the figure, an n-layer multimodal fusion module is set up in the deep completion network, and the first layer is encoded in the normal direction. The depth code φ(p), RGB image code ψ(p), and camera intrinsic parameter matrix K are input, and the remaining n-1 layers also combine the depth code σ(p) output by the previous layer and take the four feature codes as input. The generated four codes are passed to the next layer as the input of the next multimodal fusion module.

[0081] The multimodal fusion module first uses the camera intrinsic parameter matrix K to upgrade the depth code to three-dimensional space and generate a three-dimensional position code x 3D (p):

[0082]

[0083] Among them, x 3D (p), x(p) and They correspond to the three-dimensional coordinates, two-dimensional coordinates and two-dimensional homogeneous coordinates of pixel p respectively.

[0084] In order to better combine and utilize the three-dimensional information and the normal information, the present invention further uses the three-dimensional position code x 3D (p), generates a 3D normal encoding

[0085] Specifically, first obtain the three-dimensional position code x 3D The 3×3 neighborhood of each pixel in (p) is then fitted to a local plane using the least squares fitting method, and the normal of the fitted plane is used as the normal of the point. 3D Normal Coding It can be obtained by the following formula (6):

[0086]

[0087] in, The calculation formula of A(p) is shown in the following formula (7):

[0088]

[0089] Among them, {p i |i=1,2,…9} is the 3×3 neighborhood of pixel point p (including pixel point p).

[0090] The 3D normal code generated by the above method is different from the input normal code After splicing and fusion, it can be used as the updated normal encoding Perform follow-up operations.

[0091] The present invention uses a 1×1 convolution to encode the three-dimensional position x 3D (p), updated normal encoding And the RGB image code ψ(p) is fused. In addition to the first layer fusion module, the depth code σp) generated by the previous layer needs to be introduced at the same time during fusion.

[0092] The present invention extracts image features from the input RGB image and the densified depth representation and normal representation through independent convolution operations and inputs them into the first-layer multimodal fusion module. Afterwards, the present invention uses three independent 3×3 convolutions to extract the depth code φ(p) and RGB code φ(p) from the previous layer. New depth, normal, and RGB codes are generated from the updated normal code ψ″(p), and the generated depth code σ(p) is combined as the input of the next fusion module.

[0093] The present invention adopts the idea of ​​jump connection in ResNet, and jumps the depth code σ(p) output by the previous layer of the multimodal fusion module to the multimodal fusion module of this layer to generate the depth code of this layer. Since the method proposed in the present invention uses three modes of depth, RGB image, and normal as input, the number of parameters and the amount of information in the network are large. If not processed, more effective information will be lost. Therefore, the present invention can protect the integrity of information as much as possible through jump connection to avoid information loss and loss.

[0094] The multimodal fusion module proposed in the present invention makes full use of three-dimensional information, and replaces conventional RGB coding with depth coding σ(p) together with the depth coding normal coding jump to the decoder network, which is a more flexible and effective multimodal fusion method. Taking the intrinsic parameter matrix K as an additional input effectively improves the generalization of the model, making the model applicable to a variety of sensor structures. More importantly, due to the combination of three-dimensional information and surface normal information, the model is more accurate in processing object boundaries, small objects and areas with drastic depth changes, achieving higher performance.

[0095] Alternatively, if Figure 2As shown, after the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map are fused and encoded to obtain the depth code, the depth completion device can also decode the depth code to obtain a dense normal map.

[0096] That is to say, the above-mentioned depth coding is multi-channel data, and a dense depth map and a dense normal map can be obtained simultaneously by decoding the depth coding.

[0097] Optionally, the loss function of the above depth completion network is: in, represents the loss of photometric consistency, represents the normal-depth consistency loss, represents the sparse-depth consistency loss, represents the local smoothing loss; ω ph ,ω nd ,ω sd and ω sm is the corresponding weight balance coefficient.

[0098] The losses included in the loss function of the deep completion network are explained below.

[0099] (1) Loss of photometric consistency

[0100] Specifically, the depth completion device can predict the pose transformation relationship between adjacent frame images taken by the camera through the pose estimation network, and then project the images on the adjacent frame perspectives to the current frame perspective through the pose transformation relationship predicted by the pose estimation network to generate a reconstructed image. Finally, the loss function is constructed through the photometric consistency loss between the reconstructed image and the original input image.

[0101] The present invention designs an independent posture estimation network to obtain the relative posture between adjacent frame images. Since the intrinsic parameter matrix K has been introduced into the network as an additional input, the posture estimation network used in the present invention only needs to predict the relative posture g of adjacent frame images. τt , without the need to additionally predict the internal parameter matrix K.

[0102] The above pose estimation network uses a simple encoder-decoder network such as Figure 6 As shown, the current image I t and adjacent frame images I τ Input to the pose estimation network to predict the relative pose g between the adjacent frame τ and the current frame t τt , here the present invention takes the previous and next frame images, that is, τ∈ΓΓ is defined as {t-1, t+1}. By predicting the relative pose g τt , the internal parameter matrix K and the image I on the adjacent frame τAnd the predicted depth map on the adjacent frame The photometric consistency loss can be constructed.

[0103] Specifically, let the adjacent frames τ∈Γ, then The final photometric consistency loss constructed by the loss and the structural consistency (Structure Similarity Index Measure, SSIM) loss is:

[0104]

[0105] Among them, ω co and ω st are the weights of image reconstruction loss and SSIM loss respectively, Represents the entire image area.

[0106] The present invention uses the photometric consistency loss based on monocular sequence as the supervision of the network. Since the time interval between adjacent frames is small enough, the images on adjacent frames can be approximated as multiple images under different perspectives of the same scene. Therefore, by projecting the images on the perspectives of adjacent frames to the perspective of the current frame, a reconstructed image is generated, and self-supervised depth completion can be achieved through the photometric consistency loss between the current image and the reconstructed image.

[0107] (2) Normal-depth consistency loss

[0108] The present invention introduces normal constraints into the self-supervised depth completion method. Since the input sparse normal map is roughly estimated from the sparse depth map, if the normal is not constrained, it may cause a large gap between the normal information and the true value. If the information provided by the normal map to the depth completion network is inaccurate, it may cause the depth completion network to have an increased number of parameters and a decreased accuracy. In order to enhance the geometric consistency between the predicted normal and depth, the present invention proposes a normal-depth consistency loss term suitable for the self-supervised depth completion task based on the spatial geometric relationship between the surface normal and the depth, so as to make full use of the three-dimensional information provided by the normal and optimize the predicted dense depth map and dense normal map results.

[0109] Specifically, the pixel point p and its adjacent point q in the image coordinate system are projected into the camera coordinate system through the intrinsic parameter matrix K to generate two points P and Q. The line x between the two points is PQ It should be perpendicular to the normal of point P. Based on this, the normal depth consistency relationship between pixel p and its neighboring point q can be expressed as:

[0110]

[0111] in, represents the predicted normal vector of pixel p. Next, by dividing both sides of the formula by You can get:

[0112] D inv (p)c pq -D inv (q)c pp =0;

[0113] in,

[0114] Final Normal-Depth Consistency Loss It can be expressed as:

[0115]

[0116] in, is the neighborhood of pixel p, Regularization used to reduce image edges.

[0117] The introduction of the normal-depth consistency loss enhances the consistency between the predicted dense depth map and the dense normal map, more effectively constrains the normal information, and avoids the problem of poor results of the normal branch due to the lack of effective supervision.

[0118] (3) Sparse-depth consistency loss

[0119] Although the self-supervised depth completion method does not have the complete depth truth as a strong supervision, the input sparse depth itself can also serve as a supervisory signal. The introduction of sparse depth consistency loss helps to improve the accuracy of depth recovery and accelerate the convergence speed of the model in the training phase.

[0120] Since the present invention adopts an unsupervised learning method, the depth value that can be used as supervision has only sparse depth input and less effective value, so a simple Loss Penalization Prediction Depth The difference between the input sparse depth d(x) and the input sparse depth d(x) can meet the requirements. The sparse depth consistency loss used in the present invention is defined as:

[0121]

[0122] Where Ω represents the domain of sparse depth.

[0123] (4) Local smoothing loss

[0124] By smoothing the differences between spatially adjacent depth predictions through local smoothness loss, the model can predict dense depth maps through the texture information of RGB images, thereby avoiding local optimal solutions.

[0125] The local smoothing loss defined in the present invention is:

[0126]

[0127] in,

[0128] The local smoothness loss is penalized by To smooth the depth prediction, The negative exponent of is weighted to make the edge structure of the output dense depth map similar to that of the RGB image, so that the output depth map can achieve a smoother and more accurate boundary segmentation effect.

[0129] In an embodiment of the present invention, since the sparse depth map can be processed for normal estimation to obtain a sparse normal map, and fusion encoding and decoding are performed based on the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map to obtain a dense depth map with higher image density. Since the sparse normal map is added in the process of image depth completion, it can not only provide more spatial geometric information, optimize the performance of the sparse depth map on the object boundary, and suppress noise, but also the multimodal input can provide more semantic information for the depth completion network, thereby improving the network accuracy of the depth completion network.

[0130] The above mainly introduces the solution provided by the embodiment of the present invention from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed in this article, the embodiment of the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0131] The depth completion method provided in the embodiment of the present invention may be executed by a depth completion device or a control module for depth completion in the depth completion device. In the embodiment of the present invention, the depth completion device executing the depth completion method is taken as an example to illustrate the depth completion device provided in the embodiment of the present invention.

[0132] It should be noted that the embodiment of the present invention can divide the functional modules of the depth completion device according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. Optionally, the division of modules in the embodiment of the present invention is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0133] like Figure 7 As shown, an embodiment of the present invention provides a depth completion device 700. The depth completion device 700 includes: an acquisition module 701 and a processing module 702. The acquisition module 701 can be used to acquire an RGB image, a sparse depth map, and a camera intrinsic parameter matrix; the processing module 702 is used to perform normal estimation processing on the sparse depth map to obtain a sparse normal map; the RGB image, the sparse depth map, the camera intrinsic parameter matrix, and the sparse normal map are fused and encoded through a depth completion network to obtain a depth code, and the depth code is decoded to obtain a dense depth map; wherein the image density of the dense depth map is higher than the image density of the sparse depth map.

[0134] Optionally, the sparse normal map includes normal vectors of all valid pixels in the sparse depth map; the processing module 702 is used to: back-project the valid pixels in the sparse depth map through the camera intrinsic parameter matrix to obtain a sparse point cloud; determine a set of neighboring points for each valid point in the sparse point cloud; determine a target fitting plane for the neighboring point set, wherein the distance between the neighboring points in the neighboring point set and the target fitting plane is the smallest; and determine the normal vector of the target fitting plane as the normal vector of the corresponding valid pixel.

[0135] Optionally, the processing module 702 is used to: determine a ground area image of the sparse depth map according to a first proportionality coefficient; and determine the direction of a normal vector in the ground area image whose angle with the vertical direction is less than a first angle as vertically upward.

[0136] Optionally, before the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map are fused and encoded to obtain a depth code, and the depth code is decoded to obtain a dense depth map, the processing module 702 is used to: use kernels of different sizes to perform maximum pooling and minimum pooling operations on the sparse depth map to obtain a dense sparse depth map; perform maximum pooling and minimum pooling operations on the sparse normal map based on the second component of the normal to obtain a dense sparse normal map.

[0137] Optionally, the processing module 702 is used to: obtain the depth code output by the previous layer; determine the three-dimensional position code according to the depth code of the sparse depth map and the camera intrinsic parameter matrix, and determine the three-dimensional normal code of the three-dimensional position code; perform splicing and fusion processing on the normal code of the sparse normal map and the three-dimensional normal code to obtain a new normal code; perform fusion processing on the new normal code, the image code of the RGB image, the three-dimensional position code and the depth code output by the previous layer to obtain the depth code.

[0138] Optionally, after the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map are fused and encoded to obtain the depth code, the processing module 702 is further used to: decode the depth code to obtain a dense normal map.

[0139] Optionally, the loss function of the depth completion network is: in, represents the loss of photometric consistency, represents the normal-depth consistency loss, represents the sparse depth consistency loss, represents the local smoothing loss; ω ph ,ω nd ,ω sd and ω sm is the corresponding weight balance coefficient.

[0140] In an embodiment of the present invention, the sparse depth map can be processed for normal estimation to obtain a sparse normal map, and fusion encoding and decoding are performed based on the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map to obtain a dense depth map with higher image density. Since the sparse normal map is added in the process of image depth completion, it can not only provide more spatial geometric information, optimize the performance of the sparse depth map on the object boundary, and suppress noise, but also the multimodal input can provide more semantic information for the depth completion network, thereby improving the network accuracy of the depth completion network.

[0141] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the depth completion method, which includes: obtaining an RGB image, a sparse depth map and a camera intrinsic parameter matrix; performing normal phase estimation processing on the sparse depth map to obtain a sparse normal map; performing fusion encoding processing on the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map through a depth completion network to obtain a depth code, and decoding the depth code to obtain a dense depth map; wherein the image density of the dense depth map is higher than the image density of the sparse depth map.

[0142] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0143] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the depth completion method provided by the above methods, and the method includes: obtaining an RGB image, a sparse depth map, and a camera intrinsic parameter matrix; performing normal estimation processing on the sparse depth map to obtain a sparse normal map; performing fusion encoding processing on the RGB image, the sparse depth map, the camera intrinsic parameter matrix, and the sparse normal map through a depth completion network to obtain a depth code, and decoding the depth code to obtain a dense depth map; wherein the image density of the dense depth map is higher than the image density of the sparse depth map.

[0144] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned depth completion methods, the methods comprising: acquiring an RGB image, a sparse depth map, and a camera intrinsic parameter matrix; performing normal estimation processing on the sparse depth map to obtain a sparse normal map; performing fusion encoding processing on the RGB image, the sparse depth map, the camera intrinsic parameter matrix, and the sparse normal map through a depth completion network to obtain a depth code, and decoding the depth code to obtain a dense depth map; wherein the image density of the dense depth map is higher than the image density of the sparse depth map.

[0145] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A depth completion method, characterized in that: include: Get RGB image, sparse depth map and camera intrinsic parameter matrix; Performing normal estimation processing on the sparse depth map to obtain a sparse normal map; Performing fusion encoding processing on the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map through a depth completion network to obtain a depth code, and decoding the depth code to obtain a dense depth map; Wherein, the image density of the dense depth map is higher than the image density of the sparse depth map; The sparse normal map includes normal vectors of all valid pixels in the sparse depth map; The method of performing normal estimation processing on the sparse depth map to obtain a sparse normal map includes: performing back-projection processing on the effective pixels in the sparse depth map through the camera intrinsic parameter matrix to obtain a sparse point cloud; determining a neighboring point set for each effective point in the sparse point cloud; determining a target fitting plane of the neighboring point set, wherein the distance between the neighboring points in the neighboring point set and the target fitting plane is the smallest; and determining the normal vector of the target fitting plane as the normal vector of the corresponding effective pixel. The fusion coding process of the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map to obtain the depth code includes: obtaining the depth code output by the previous layer; determining the three-dimensional position code according to the depth code of the sparse depth map and the camera intrinsic parameter matrix, and determining the three-dimensional normal code of the three-dimensional position code; splicing and fusing the normal code of the sparse normal map and the three-dimensional normal code to obtain a new normal code; fusing the new normal code, the image code of the RGB image, the three-dimensional position code and the depth code output by the previous layer to obtain the depth code.

2. The depth completion method according to claim 1, characterized in that: The performing normal estimation processing on the sparse depth map to obtain a sparse normal map includes: Determine a ground area image of the sparse depth map according to a first scale factor; The direction of the normal vector in the ground area image whose angle with the vertical direction is smaller than the first angle is determined as vertically upward.

3. The depth completion method according to claim 1, characterized in that: Before the fusion encoding process is performed on the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map to obtain the depth code, and the depth code is decoded to obtain the dense depth map, the method further includes: Using kernels of different sizes to perform a maximum pooling and a minimum pooling operation on the sparse depth map to obtain a densified sparse depth map; Based on the second normal component, a quasi-max pooling and a quasi-min pooling operation are performed on the sparse normal map to obtain a densified sparse normal map.

4. The depth completion method according to claim 1, characterized in that: After the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map are subjected to fusion coding to obtain depth coding, the method further includes: The depth code is decoded to obtain a dense normal map.

5. The depth completion method according to claim 4, characterized in that: The loss function of the depth completion network is: in, represents the loss of photometric consistency, represents the normal-depth consistency loss, represents the sparse depth consistency loss, represents the local smoothing loss; ω ph ,ω nd ,ω sd and ω sm is the corresponding weight balance coefficient.

6. A depth completion device, characterized in that: include: Acquisition module and processing module; The acquisition module is used to acquire the RGB image, the sparse depth map and the camera intrinsic parameter matrix; The processing module is used to perform normal estimation processing on the sparse depth map to obtain a sparse normal map; Performing fusion encoding processing on the RGB image, the sparse depth map, the camera intrinsic parameter matrix and the sparse normal map through a depth completion network to obtain a depth code, and decoding the depth code to obtain a dense depth map; wherein the image density of the dense depth map is higher than the image density of the sparse depth map; The sparse normal map includes normal vectors of all valid pixels in the sparse depth map; the processing module is used to perform back-projection processing on the valid pixels in the sparse depth map through the camera intrinsic parameter matrix to obtain a sparse point cloud; determine a neighboring point set for each valid point in the sparse point cloud; determine a target fitting plane of the neighboring point set, where the distance between the neighboring points in the neighboring point set and the target fitting plane is the smallest; determine the normal vector of the target fitting plane as the normal vector of the corresponding valid pixel; The acquisition module is used to acquire the depth code output by the previous layer; the processing module is used to determine the three-dimensional position code according to the depth code of the sparse depth map and the camera intrinsic parameter matrix, and determine the three-dimensional normal code of the three-dimensional position code; the normal code of the sparse normal map and the three-dimensional normal code are spliced ​​and fused to obtain a new normal code; the new normal code, the image code of the RGB image, the three-dimensional position code and the depth code output by the previous layer are fused to obtain the depth code.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the depth completion method according to any one of claims 1 to 5 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the depth completion method according to any one of claims 1 to 5 are implemented.

Images