An unsupervised depth completion method and system based on visual-inertial system and confidence guidance

Abstract

An unsupervised depth completion method and system based on a visual-inertial system and confidence guidance relates to the field of robotic computer vision technology. This method addresses the problem of existing unsupervised depth completion methods having complex network parameters and limited widespread application. The method includes: using a visual-inertial system to acquire images, sparse depth maps, and camera poses; establishing an unsupervised dense depth completion network, which processes the sparse depth map to obtain confidence and unsupervised dense depth maps; using a supervised depth completion network to process confidence and unsupervised dense depth maps to obtain supervised dense depth maps; using a motion residual network to process supervised dense depth maps, images, and camera motion poses to obtain a translation residual matrix; using a loss function to train the supervised depth completion network to obtain a supervised depth completion network training model; and using a loss function to train the motion residual network to obtain a motion residual network training model. The method is applied to the field of image processing technology.

Description

Technical Field

[0001] The present invention relates to the field of robot computer vision technology, and in particular to an unsupervised depth completion method based on a visual inertial system and confidence guidance. Background Art

[0002] Spatial perception is an essential learning process for robots, helping them perceive and understand various aspects of their environment. Visual-inertial systems, through simultaneous localization and mapping, provide robots with precise self-localization information and a sparse map of their environment. However, the sparsity of this environmental map depth is insufficient for robots to perform complex tasks such as path planning and obstacle avoidance. Therefore, it is necessary to convert this sparse depth map into a dense one.

[0003] While some simultaneous localization and mapping methods can obtain dense depth information in an environment, these methods are limited in various ways. In the past few years, depth completion methods have become popular in robotics. They use convolutional neural networks to complete dense depth maps from sparse depth and color images.

[0004] There are two main types of depth completion methods: one is a supervised completion method that relies on true depth values ​​for training, and the other is an unsupervised completion method based on the motion structure principle (SFM).

[0005] Supervised depth completion methods primarily aim to minimize the difference between the true pixel depth of each image and the depth predicted from the RGB image and sparse depth during training. These methods primarily focus on network structure modeling, loss function optimization, and parameter lightweighting. Jaritz et al. proposed a post-processing fusion of RGB images and sparse depth using a U-shaped network, jointly learning depth completion and semantic segmentation. A prior art method also proposed a CNN-based belief propagation method for guiding sparse depth regression. By introducing normalized convolutional layers and generating point-by-point continuous confidence maps to guide depth completion, they achieved good accuracy. Building on the CNN-based belief propagation method for guiding sparse depth regression, Teixeira et al. added a new confidence guidance network while maintaining lightweight network parameters and developed it in an embedded system. The prior art also proposed the SemAttNet network, which uses attention-based semantic perception to guide depth completion. This network primarily consists of color-guided, semantic-guided, and depth-guided branches, and uses SAMMFAB to fuse features across these three branches. All of these methods require pixel-by-pixel true value labeling of training samples, which is a very difficult and complex task that requires a lot of manpower, and it is often very difficult to obtain the true depth.

[0006] Unsupervised depth completion methods, based on the principle of structure from motion, predict depth information at each pixel by minimizing the photometric error between the input image and the reconstructed image. Existing methods typically use PnP and random sample consistency to align the image's photometric terms with a second-order smoothness prior. However, this approach is limited to depth completion for static scenes. Yang et al. leveraged a recently introduced method to learn a conditional prior, rather than relying on a local smoothness assumption. Learning the conditional prior requires the introduction of a prior network, making the entire network less than completely unsupervised. Wong et al. employed a predictive cross-modal criterion similar to "self-supervision" to measure photometric consistency across time, previous and next pose consistency, and ensemble compatibility of sparse point clouds. Building on this, Wong et al. imposed a strong inductive bias on the Euclidean distance reconstruction within their architecture and incorporated a calibrated back-projection layer, reducing model size while maintaining the state-of-the-art. Similarly, these methods fail to account for interference caused by dynamic objects and occlusions in the environment. As can be seen, unsupervised depth completion methods avoid the complex task of labeling training samples but often suffer from many challenges of SFM, such as dynamic objects and occlusions. To overcome these challenges, most scholars added additional supervisory information such as optical flow and semantics, which made the network parameters complicated and not widely applicable. Summary of the Invention

[0007] The present invention solves the problem that the network parameters of the unsupervised depth completion method in the prior art are complex and cannot be widely used.

[0008] The specific scheme of the present invention is:

[0009] The present invention provides an unsupervised depth completion method based on a visual inertial system and confidence guidance, the method comprising:

[0010] Use a visual-inertial SLAM system to acquire RGB images, sparse depth maps, and camera poses;

[0011] Building an unsupervised dense depth completion network;

[0012] Processing the sparse depth map according to the unguided dense depth completion network to obtain a confidence score and an unguided dense depth map;

[0013] Processing the confidence and the unguided dense depth map according to the guided depth completion network to obtain a guided dense depth map;

[0014] Processing the guidance dense depth map, the RGB image, and the camera motion pose using a motion residual network to obtain a translation residual matrix;

[0015] Training the guided depth completion network according to the Losses loss function to obtain a guided depth completion network training model;

[0016] The motion residual network is trained according to the Losses loss function to obtain a motion residual network training model.

[0017] Furthermore, a preferred embodiment is provided, wherein the step of processing the sparse depth map according to the unguided dense depth completion network to obtain a confidence score and an unguided dense depth map comprises:

[0018] Using a normalized convolutional layer to separate the depth and confidence of the sparse depth map;

[0019] Confidence and unguided dense depth maps are obtained through confidence adaptation processing.

[0020] Furthermore, a preferred embodiment is provided, wherein the confidence and the non-guided dense depth map are processed according to the guided depth completion network to obtain the guided dense depth map, including:

[0021] The confidence and the non-guided dense depth are processed by 3*3 2D convolution + activation function, 3*3 2D convolution + Leaky activation function and 2D convolution to obtain a guided dense depth map.

[0022] Furthermore, a preferred embodiment is provided, wherein the motion residual network is specifically:

[0023] There are 5 convolution stacks with a stride of 2. The last convolution stack is an average pool with 1024 channels and a spatial resolution of 1*1.

[0024] The motion residual network also includes two 1*1 convolutional layers of three channels.

[0025] Furthermore, a preferred method is provided, wherein the guiding depth completion network is trained according to the Losses loss function to obtain the guiding depth completion network training model, specifically:

[0026] Acquire a dynamic region and a static region according to the translation residual matrix, wherein the dynamic region is a dynamic object region at the current moment, and the static region is a static structure region at the current moment;

[0027] Calculate the photometric consistency loss for pixels in the static area:

[0028]

[0029] in, is the photometric consistency loss of the pixels in the static area, N is the total number of pixels in the static area, I(P t ) is the pixel coordinate P t The pixel value of To reconstruct the pixel coordinate P t+1 The pixel value, P t is the coordinate of a pixel point in a static area at time t, is the coordinate of a pixel point in a static area at time t+1;

[0030] Calculate the photometric consistency loss for pixels in the dynamic area:

[0031]

[0032] in, is the photometric consistency loss of pixels in the dynamic area, is the pixel coordinate The pixel value, I(P t obj ) is the pixel coordinate The pixel value of is the coordinate of a target pixel point in a dynamic area at time t+1, P t obj is the coordinate of a target pixel point in a dynamic area at time t;

[0033] Calculate the parallax smoothing loss for all pixels:

[0034]

[0035] Among them, L ds is the parallax smoothing loss for all pixels, To perform partial derivative of u, d u,v is the disparity of pixel coordinates (u, v), I u,v is the pixel at pixel coordinate (u, v), e is a constant;

[0036] According to the photometric consistency loss of the static area and the dynamic area and the parallax smoothing loss of all pixels, the The total loss function is:

[0037]

[0038]

[0039] Among them, L E is the loss function without confidence guidance, p is the number of training times;

[0040] The guided depth completion network is trained according to the total loss function to obtain a guided depth completion network training model.

[0041] Furthermore, a preferred method is provided, wherein the motion residual network is trained according to the Losses loss function to obtain the motion residual network training model, specifically:

[0042]

[0043] in, is the loss function of the motion residual network, L1 is the mean absolute error, t obj (δu i ,δv i ) is the translation residual, δu i is the pixel deviation in the u direction, δv i is the pixel deviation in the v direction.

[0044] Based on the same inventive concept, the present invention also provides an unsupervised depth completion system based on a visual inertial system and confidence guidance, the system comprising:

[0045] An image information acquisition unit, used to acquire RGB images, sparse depth maps, and camera poses using a visual-inertial SLAM system;

[0046] An unsupervised dense depth completion network building unit, used to build an unsupervised dense depth completion network;

[0047] a confidence and unguided dense depth map acquisition unit, configured to process the sparse depth map according to the unguided dense depth completion network to acquire a confidence and unguided dense depth map;

[0048] a guided dense depth map acquisition unit, configured to process the confidence and the non-guided dense depth map according to the guided depth completion network to acquire a guided dense depth map;

[0049] A translation residual matrix acquisition unit, configured to process the guided dense depth map, the RGB image, and the camera motion pose using a motion residual network to acquire a translation residual matrix;

[0050] A guided depth completion network training model acquisition unit is used to train the guided depth completion network according to the Losses loss function to obtain a guided depth completion network training model;

[0051] The motion residual network training model acquisition unit is used to train the motion residual network according to the Losses loss function to obtain the motion residual network training model.

[0052] Furthermore, a preferred embodiment is provided, wherein the confidence and non-guided dense depth map acquisition unit comprises:

[0053] Using a normalized convolutional layer to separate the depth and confidence of the sparse depth map;

[0054] Confidence and unguided dense depth maps are obtained through confidence adaptation processing.

[0055] Based on the same inventive concept, the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes an unsupervised depth completion method based on a visual inertial system and confidence guidance as described in any one of the above items.

[0056] Based on the same inventive concept, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, an unsupervised depth completion method based on a visual-inertial system and confidence guidance as described in any one of the above items is executed.

[0057] The present invention is beneficial in that:

[0058] 1. This paper describes an unsupervised depth completion method based on a visual-inertial system and confidence guidance, leveraging monocular sequences and IMU data for supervision. This method achieves joint unsupervised learning of depth completion and dynamic object motion. By designing a tightly coupled network for depth completion and motion residuals, it enables real-time, dense depth prediction for robots in dynamic and complex environments, utilizing minimal computational resources.

[0059] 2. The unsupervised depth completion method based on a visual-inertial system and confidence guidance described in this invention can obtain depth completion results using only the color image, pose information, and sparse depth map provided by VIO, without requiring any other auxiliary information. The network parameters used are simple and have a wide range of applications.

[0060] 3. The unsupervised depth completion method based on the visual inertial system and confidence guidance described in the present invention divides the image into static areas, dynamic areas, and occluded areas based on the analysis of image pixel depth and dynamic object motion reconstruction process. The present invention takes into account the problems of dynamic object motion and occlusion, eliminates interference in the unsupervised training process, and designs a loss function guided by confidence for the static area, dynamic area, and occluded area. The present invention also intuitively demonstrates the impact of confidence on the loss function, which can improve data errors until convergence, making the completed dense depth map more accurate in terms of edge sharpness and details. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flow chart of an unsupervised depth completion method based on a visual-inertial system and confidence guidance according to embodiment 1;

[0062] Figure 2 Schematic diagram of the processing flow of the guided depth completion network and the motion residual network according to the first embodiment, wherein I t +IMUt is the image and IMU information of the current frame, I t+1 +IMU t+1 is the image and IMU information of frame t+1, I t+2 +IMU t+2 It is the image and IMU information of frame t+2;

[0063] Figure 3 Schematic diagram of the guided depth completion network described in implementation mode 3;

[0064] Figure 4 Schematic diagram of the bilinear interpolation process described in Implementation Method 4;

[0065] Figure 5 Schematic diagram of the motion residual network described in the fourth embodiment;

[0066] Figure 6 A schematic diagram of mutual occlusion between a dynamic object and a static structure according to the fifth embodiment;

[0067] Figure 7 This is a comparison chart of dense depth results of different algorithms in the KITTI dataset described in embodiment 11;

[0068] Figure 8 This is a comparison chart of dense depth results of different algorithms in the VOID dataset described in the eleventh embodiment;

[0069] Figure 9 Schematic diagram of the mean and standard deviation of the maximum output confidence and MAE error of images in the KITTI dataset described in Implementation 11, where Conf.Loss is the loss function with a confidence term and Loss is the loss function without a confidence term;

[0070] Figure 10 This is a comparison chart of dense depth results of different algorithms in the real scene described in the eleventh embodiment. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in combination with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0072] Implementation method 1, see Figure 1 and Figure 2 The present invention provides an unsupervised depth completion method based on a visual inertial system and confidence guidance, the method comprising:

[0073] Use a visual-inertial SLAM system to acquire RGB images, sparse depth maps, and camera poses;

[0074] Building an unsupervised dense depth completion network;

[0075] Processing the sparse depth map according to the unguided dense depth completion network to obtain a confidence score and an unguided dense depth map;

[0076] Processing the confidence and the unguided dense depth map according to the guided depth completion network to obtain a guided dense depth map;

[0077] Using a motion residual network to process the dense depth map, RGB image and camera motion pose of the guidance to obtain a translation residual matrix;

[0078] Training the guided depth completion network according to the Losses loss function to obtain a guided depth completion network training model;

[0079] The motion residual network is trained according to the Losses loss function to obtain a motion residual network training model.

[0080] In practical applications, a completed dense depth map can be obtained based on the guided depth completion network training model and the motion residual network training model. The completed dense depth map has higher accuracy in edge sharpness and details than the completion method in the existing technology.

[0081] This implementation uses a visual-inertial SLAM system to compute the sparse 3D point cloud in the scene and the camera's motion pose. Specifically, at each time frame, the visual-inertial SLAM system extracts ORB features from the image and tracks them through feature matching using descriptors. The visual-inertial SLAM system then fuses IMU measurements from the RGB image to perform factor graph optimization to estimate the camera's motion pose, which is the combination of the camera motion and feature point positions.

[0082] Each time an image is processed, the visual-inertial SLAM system projects all visible 3D features onto the image and uses the depths of these projected 3D features to create a metric-scale sparse depth image. It is particularly important to emphasize that the IMU plays a crucial role in metric depth completion. Without an IMU, the SLAM system cannot produce a metric-scale sparse depth image.

[0083] like Figure 2As shown, the sparse depth image is used as input to the unsupervised depth completion network, which outputs an unsupervised dense depth map and confidence. The supervised dense depth completion network then processes the unsupervised dense depth map and confidence to obtain a supervised dense depth map. Furthermore, the supervised dense depth map, RGB image, and camera motion pose are used as input to the motion residual network, which predicts the motion of each pixel relative to the background.

[0084] The unsupervised dense depth completion network described in this embodiment consists of a 5*5 normalized convolution layer, maximum pooling downsampling and upsampling.

[0085] This implementation aims to perform depth completion and dynamic object motion learning using only VIO, so extensive loss function normalization is required during training. This implementation explicitly considers occlusions to eliminate occlusion interference during unsupervised training of depth completion, and designs corresponding loss functions for different cases of occluded regions.

[0086] This embodiment avoids the defect of existing supervised depth completion methods that require pixel-by-pixel true value labeling of training samples. This embodiment skips the supervised depth completion method and trains the network through unlabeled samples to achieve the accuracy of supervised depth completion.

[0087] Implementation 2: This implementation further limits the unsupervised depth completion method based on a visual inertial system and confidence guidance described in Implementation 1. The unsupervised dense depth completion network processes the sparse depth map to obtain a confidence and unsupervised dense depth map, including:

[0088] Using a normalized convolutional layer to separate the depth and confidence of the sparse depth map;

[0089] Confidence and unguided dense depth maps are obtained through confidence adaptation processing.

[0090] In this embodiment, the 2D convolution layer of the sparse depth map is normalized, and after the maximum pooling downsampling process is performed, the corresponding convolution layer is upsampled to obtain the confidence and non-guided dense depth map.

[0091] Implementation method three, see Figure 3 This embodiment further defines the unsupervised depth completion method based on a visual inertial system and confidence guidance described in Embodiment 1, wherein the confidence and the unsupervised dense depth map are processed according to the guided depth completion network to obtain the guided dense depth map, including:

[0092] The confidence and the non-guided dense depth are processed by 3*3 2D convolution + activation function, 3*3 2D convolution + Leaky activation function and 2D convolution to obtain a guided dense depth map.

[0093] Specifically, this embodiment is described in conjunction with the second embodiment. Figure 3 As shown, the guided dense depth completion network in this embodiment is a hierarchical multi-scale architecture that acts as a universal estimator at different scales and approximates the dense output well at extremely low computational cost.

[0094] In the first stage, a normalized convolutional layer is used to separate depth and confidence, allowing confidence to be adaptively processed and output confidence and an unguided dense depth map. Since the dense depth learned from the sparse depth image estimated by visual inertial SLAM shows weaknesses in local areas with weak textures. Therefore, in the second stage, this problem is alleviated by fusing RGB images and a new form of auxiliary data. This auxiliary data is the confidence output from the first stage that preserves the reliability of the depth value of each pixel. Specifically:

[0095] The first stage is an unguided dense depth completion network that processes the sparse depth map. A normalized convolutional layer receives the sparse depth map. Then, two normalized convolutional layers are applied, followed by downsampling. The downsampling process is performed by applying a max pooling operator to the output confidences of the last normalized layer while maintaining the indices of the pooled values. These indices are then used to select the value from the feature map with the highest confidence. This enables the most reliable data to be propagated to subsequent scales. To maintain the absolute level of confidence after downsampling, the downsampled confidences are divided by the scaled Jacobian.

[0096] The above pipeline is repeated as needed based on the sparsity of the data. To fuse different scales, the output from the last normalized convolutional layer and the corresponding confidence scores are upsampled using nearest neighbor interpolation and concatenated with the corresponding scale via skip connections. After each cascade, a new normalized convolutional layer is used to fuse the data based on the confidence scores of the two scales. Finally, a 1x1 normalized convolutional layer is used to fuse the different channels into a single channel corresponding to the dense output. In addition to the dense output, a confidence output is also provided, which contains information about the distribution of output confidence scores. The output confidence scores can be used in security applications or in a second stage within the network.

[0097] The second stage is the guided dense depth completion network processing stage, the goal of which is to fuse the sparse depth, RGB image, and output confidence to generate a dense depth map. This implementation uses an encoder-decoder architecture with skip connections. Specifically, the unguided dense depth and output confidence from the first stage are first received, merged with the RGB image, and input into the encoder-decoder architecture with skip connections (i.e., the guided dense depth completion network processing stage), which is processed to obtain the guided dense depth map.

[0098] Implementation method 4, see Figure 4 and Figure 5 This embodiment further defines the unsupervised depth completion method based on a visual inertial system and confidence guidance described in the first embodiment, wherein the motion residual network is specifically:

[0099] There are 5 convolution stacks with a stride of 2. The last convolution stack is an average pool with 1024 channels and a spatial resolution of 1*1.

[0100] The motion residual network also includes two 1*1 convolutional layers of three channels.

[0101] Specifically, such as Figure 5 As shown, the motion residual network described in this embodiment is composed of a convolution stack with a stride of 2, and the last convolution stack is an average pool, which has 1024 channels and a spatial resolution of 1*1; the image input end of the motion residual network specifically has five channels, namely: RGB three channels, a guided dense depth channel and a motion pose channel; the two 1*1 convolution layers with 3 channels in the motion residual network predict the global rotation angle and the global translation vector respectively. The global rotation angle and translation vector represent the movement of the entire scene relative to the camera caused by the movement of the camera and the movement of dynamic objects. The subsequent convolution layers gradually refine the translation by a factor of 2 in the height and width dimensions each time, from a single vector to the translation residual matrix t obj (δu, δv). The output of the motion residual network is the translation residual matrix t obj (δu,δv) represents the translation size of the dynamic object in the pixel coordinate system.

[0102] Combined with the embodiment described in embodiment 1, the confidence and the non-guided dense depth are input to the guided depth completion network, and the current frame dense depth output by the guided depth completion network is Use VIO to obtain the camera motion poses of two adjacent frames, and the pixel coordinates of each pixel point in the current frame in the next frame image for:

[0103]

[0104] Among them, K is the camera internal parameter, To predict the depth value of each pixel in the next frame image, To predict the u-axis pixel coordinates of each pixel in the next frame image, To predict the v-axis pixel coordinates of each pixel in the next frame image.

[0105] The above calculation only considers the pixel changes caused by camera motion, and does not consider the pixel changes between two frames caused by the motion of the dynamic objects in the scene. As shown in the following formula, the translation residual matrix output by the motion residual network can be used to obtain the exact pixel coordinates of the dynamic object in the next frame of the image: and the corresponding pixel values

[0106]

[0107] At the same time Figure 4 As shown in the figure, during the projection reconstruction process between two frames, the pixel value I(P) of each pixel of the reconstructed image can be obtained by the differentiable bilinear interpolation method:

[0108]

[0109] Among them, P bl is the lower left pixel coordinate closest to the sub-pixel point P, P br is the lower right pixel coordinate closest to the sub-pixel point P, P tl is the upper left pixel coordinate closest to the sub-pixel point P, P tr is the upper right pixel coordinate closest to the sub-pixel point P.

[0110] Implementation method five, see Figure 6 This embodiment further defines the unsupervised depth completion method based on a visual inertial system and confidence guidance described in the first embodiment, wherein the guided depth completion network is trained according to the Losses loss function to obtain the guided depth completion network training model, specifically:

[0111] Acquire a dynamic region and a static region according to the translation residual matrix, wherein the dynamic region is a dynamic object region at a current moment, and the static region is a static structure region at a previous moment;

[0112] Calculate the photometric consistency loss for pixels in the static area:

[0113]

[0114] in, is the photometric consistency loss of the pixels in the static area, N is the total number of pixels in the static area, I(Pt ) is the pixel coordinate P t The pixel value of To reconstruct the pixel coordinate P t+1 The pixel value, P t is the coordinate of a pixel point in a static area at time t, is the coordinate of a pixel point in a static area at time t+1;

[0115] Calculate the photometric consistency loss for pixels in the dynamic area:

[0116]

[0117] in, is the photometric consistency loss of pixels in the dynamic area, is the pixel coordinate The pixel value, I(P t obj ) is the pixel coordinate The pixel value of is the coordinate of a target pixel point in a dynamic area at time t+1, P t obj is the coordinate of a target pixel point in a dynamic area at time t;

[0118] Calculate the parallax smoothing loss for all pixels:

[0119]

[0120] Among them, L ds is the parallax smoothing loss for all pixels, To perform partial derivative of u, d u,v is the disparity of pixel coordinates (u, v), I u,v is the pixel at pixel coordinate (u, v), e is a constant;

[0121] According to the photometric consistency loss of the static area and the dynamic area and the parallax smoothing loss of all pixels, the The total loss function is:

[0122]

[0123]

[0124] Among them, L E is the total loss without confidence guidance, p is the number of training times;

[0125] The guided depth completion network is trained according to the total loss function to obtain a guided depth completion network training model.

[0126] In particular, note that as the data error term begins to converge, the confidence term decays by dividing by the number of training epochs to prevent it from dominating the loss.

[0127] Specifically, this embodiment is described in conjunction with Embodiment 4. This embodiment aims to perform depth completion and dynamic object motion learning using only VIO, so a large amount of loss function normalization is required during training. In particular, the method proposed in this embodiment explicitly considers occlusion issues to eliminate occlusion interference in unsupervised training of depth completion, and designs corresponding loss functions for different situations of occluded areas.

[0128] The pixel coordinates P of the dynamic object at the current moment can be obtained through the RGB image and the translation residual output by the motion residual network. t obj and the corresponding pixel value I(P t obj ). The dynamic object area at the current moment is defined as the dynamic area, and the static structure area at the current moment is defined as the static area. When the current frame image is projected to the next frame, occlusion will occur due to the movement of the camera and the dynamic object. This area is defined as the occlusion area. The occlusion situation of the occlusion area is mainly divided into two types: the static structure depth is less than the dynamic object depth and the static structure depth is greater than the dynamic object depth. Figure 6 The mutual occlusion diagram of dynamic objects and static structures shown in the figure shows that when a dynamic object and a static structure or multiple dynamic objects are at the same pixel, the pixel belongs to the one with the smallest depth among them.

[0129] Formula (2) of the fourth embodiment obtains the pixel coordinates of the dynamic object in the next frame image and the corresponding pixel values ​​of the RGB image Note that the focus at this time is to determine whether the pixel belongs to the dynamic object or the static scene.

[0130] According to formula (4), the pixel coordinates Pixel coordinates that can be projected back to the current frame And get the corresponding pixel value through RGB image

[0131]

[0132] Furthermore, by judging and The pixel difference size is used to identify the category of occlusion. Less than If the static structure depth is small, that is, the static structure is not occluded, but the dynamic object is occluded, then the pixels in the occluded area are normalized to the pixels in the static area. Conversely, if the static structure depth is large, the pixels in the occluded area are normalized to the pixels in the dynamic area. Similarly, when multiple dynamic objects move to the same pixel, the pixel value of the pixel belongs to which dynamic object or static structure by determining the pixel difference between these dynamic objects and static structures.

[0133] Dense depth output of the depth completion network The regularization is done through the photometric consistency loss and the parallax smoothness loss that complement each other with the maximization of confidence. The photometric consistency loss is a combination of the L1 penalty of the average per-pixel reprojection error and SSIM, which is a perceptual measure of invariance to local illumination changes. The parallax smoothness loss is a loss of the disparity map d u,v Perform standard edge-aware smoothing regularization to ensure that the parallax smoothing loss weight is small when the image gradient is large.

[0134] Specifically, the photometric consistency loss is calculated for the pixels in the static area as shown in formula (5); the photometric consistency loss is calculated for the pixels in the dynamic area as shown in formula (6); and the parallax smoothing loss is calculated for all pixels as shown in formula (7).

[0135]

[0136]

[0137]

[0138] In the guided depth completion network with normalized convolutional layers, it is necessary to maximize the output confidence C while minimizing the data error. u,v Therefore, a loss function that achieves both goals is needed, and the total loss function is for:

[0139]

[0140]

[0141] Where p is the number of training times. When the data error term begins to converge, the confidence term is decayed by dividing by the number of training times p to prevent it from dominating the loss.

[0142] Implementation 6. This implementation further limits the unsupervised depth completion method based on a visual inertial system and confidence guidance described in Implementation 1. The motion residual network is trained according to the Losses loss function to obtain a motion residual network training model, including:

[0143]

[0144] in, is the loss function of the motion residual network, L1 is the mean absolute error, t obj (δu i ,δv i ) is the translation residual, δu i is the pixel deviation in the u direction, δv i is the pixel deviation in the v direction.

[0145] Specifically, the translation residual t output by the residual translation network obj (δu,δv) through The loss function is normalized. The loss function minimizes the translation difference of each dynamic object in the dynamic region, so that all pixels belonging to a dynamic object maintain consistent motion. It is defined as:

[0146]

[0147] The unsupervised depth completion method described in this embodiment is used for depth completion in highly dynamic scenes. It jointly solves the translation residual matrix and the dense depth map. In order to solve the adverse effects caused by the motion and occlusion of dynamic objects during training, this embodiment divides the image into static, dynamic and occluded areas, and designs loss functions guided by the visual inertial system and confidence respectively. This function can simultaneously minimize data errors and maximize output confidence. Finally, by combining depth and RGB information with confidence to fuse structural information, the completed dense depth map is more accurate in terms of edge sharpness and details.

[0148] Embodiment 7: This embodiment describes an unsupervised depth completion system based on a visual-inertial system and confidence guidance, the system comprising:

[0149] An image information acquisition unit, used to acquire RGB images, sparse depth maps, and camera poses using a visual-inertial SLAM system;

[0150] An unsupervised dense depth completion network building unit, used to build an unsupervised dense depth completion network;

[0151] a confidence and unguided dense depth map acquisition unit, configured to process the sparse depth map according to the unguided dense depth completion network to acquire a confidence and unguided dense depth map;

[0152] a guided dense depth map acquisition unit, configured to process the confidence and the non-guided dense depth map according to the guided depth completion network to acquire a guided dense depth map;

[0153] A translation residual matrix acquisition unit, configured to process the guided dense depth map, the RGB image, and the camera motion pose using a motion residual network to acquire a translation residual matrix;

[0154] A guided depth completion network training model acquisition unit is used to train the guided depth completion network according to the Losses loss function to obtain a guided depth completion network training model;

[0155] The motion residual network training model acquisition unit is used to train the motion residual network according to the Losses loss function to obtain the motion residual network training model.

[0156] Embodiment 8: This embodiment further limits the unsupervised depth completion system based on a visual inertial system and confidence guidance described in embodiment 6. The confidence and unguided dense depth map acquisition unit includes:

[0157] Using a normalized convolutional layer to separate the depth and confidence of the sparse depth map;

[0158] Confidence and unguided dense depth maps are obtained through confidence adaptation processing.

[0159] Implementation method 9. A computer device described in this implementation method includes a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes an unsupervised depth completion method based on a visual inertial system and confidence guidance as described in any one of implementation methods 1 to 5.

[0160] Implementation method ten, a computer-readable storage medium as described in this implementation method, on which a computer program is stored, and when the computer program is executed by a processor, an unsupervised depth completion method based on a visual inertial system and confidence guidance as described in any one of implementation methods one to five is executed.

[0161] Implementation method 11, see Figure 7 and Figure 8 This embodiment provides a specific example of the unsupervised depth completion method based on a visual inertial system and confidence guidance described in embodiment 1, and is also used to explain embodiments 2 to 5. Specifically:

[0162] This embodiment is compared with other advanced depth completion networks on the KITTI depth completion dataset and the VOID dataset. In addition, the weight model trained on the VOID dataset was tested in real scenes to verify the generalization ability of embodiment one. This embodiment adopts the standard evaluation indicators of the KITTI depth completion dataset: the mean absolute error (MAE) and the root mean square error (RMSE) calculated based on the depth values. MAE is an unbiased error estimate that estimates the average error of the entire image, while RMSE penalizes outliers. In addition, this embodiment also uses iMAE and iRMSE, which are calculated based on disparity instead of depth. The definitions of these four evaluation indicators are shown in Table 1.

[0163] Table 1 Error Metrics

[0164]

[0165] The KITTI depth completion dataset provides approximately 80,000 raw image frames of outdoor scenes and associated sparse depth maps. The sparse depth map is composed of point cloud depths with a density of approximately 5% of the output of a Velodyne lidar sensor. In order to better verify the performance of the method provided in this embodiment, this embodiment will use a sparse depth map with a density of approximately 0.5% generated by a visual-inertial SLAM system. This embodiment is verified using 1,000 officially selected test samples, and compares the unsupervised depth completion method based on a visual-inertial system and confidence guidance described in this embodiment with other advanced supervised / unsupervised depth completion methods.

[0166] VOID contains synchronized 640x480 RGB images (laboratory, corridor, classroom, and garden) and sparse depth maps with a density of approximately 0.5%. It is worth emphasizing that the sparse depth points provided by the dataset are a set of features tracked by SLAM systems, which exactly matches the sparse depth maps required by our depth completion network. The entire dataset is divided into 48 training samples and 8 test samples (containing 800 frames of images and sparse depth maps).

[0167] The OpenLORIS dataset is entirely derived from everyday life scenes and is the world's first large-scale dataset that includes indoor dynamic scenes. The dataset contains 848*480 RGB images, aligned depth images, and IMU information collected by InterD435i. It is worth noting that the dataset does not perform any modification on the collected images and can be used as a test set for real scenes. In order to verify the generalization ability of the method described in this embodiment in real scenes, the model trained with the VOID dataset will be tested in different scenes (coffee shop, home, supermarket).

[0168] In practice, the method proposed in this embodiment is implemented using PyTorch and trained on a machine equipped with an NVIDIA GTX3090Ti GPU with 24G memory. For all experiments in this embodiment, the Adam optimizer is used with a learning rate of 10 -4 Start with 10 and reduce it by a factor of 10 every three periods, down to 10 -6 The model is trained for 24 hours and the best epoch is reported. Standard PyTorch weights are used for ResNet and the batch size is 8. Given the diversity of depth in the dataset, the sparse depth needs to be normalized before inputting it into the network. Specifically, a visual inertial SLAM system is used to generate sparse depth maps with a density of about 0.5% of the KITTI dataset, OpenLORIS dataset, and VOID.

[0169] The visual-inertial system and confidence-guided real-time unsupervised depth completion method described in this embodiment are submitted to the KITTI Depth benchmark and compared with advanced unsupervised and supervised methods. Among them, the MDPC method of Liu et al. proposes an adaptive knowledge distillation method that integrates some existing excellent models while avoiding some errors in the model. The most accurate method among the supervised depth completion methods is the NLSPN method of Park et al., which proposes a robust and efficient end-to-end non-local spatial propagation network for depth completion. By introducing a learnable affinity normalization method, it has better robustness to the mixed depth problem of depth boundaries.

[0170] The quantitative results of the KITTI depth dataset test set using the above evaluation metrics and methods are shown in Table 2. The supervised depth completion method performs better overall than the unsupervised depth completion method. The method described in this embodiment is better than the most advanced unsupervised depth completion method and some supervised depth completion methods, and is slightly worse than the most advanced supervised depth completion method. In addition, the method described in this embodiment can improve MAE by 1.93%, RMSE by 1.76%, iMAE by 3.26%, and iRMSE by 3.79% compared with the most advanced unsupervised MDPC method in Table 2. It is worth emphasizing that the sparse depth of the input motion residual network is generated by VI-SLAM and accounts for only 0.5% of the image pixel density, while other methods use sparse depth maps with a density of 5% in the dataset. This embodiment achieves better depth completion effect with less diluted depth.

[0171] Table 2 Quantitative results of KITTI test dataset

[0172]

[0173]

[0174] Among them, the second line of the first column of Table 2 is Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera proposed by F.Ma, GVCavalheiro and S.Karaman, the third line of the first column is Dense Depth Posterior (DDP) From SingleImage and Sparse Range proposed by Y.Yang, A.Wong and S.Soatto, the fourth line of the first column is Unsupervised Depth Completion FromVisual Inertial Odometry proposed by A.Wong, X.Fei, S.Tsuei and S.Soatto, and the fifth line of the first column is Unsupervised Depth Completion with CalibratedBackprojection proposed by A.Wongand S.Soatto. Layers (unsupervised depth completion with calibrated back-projection layers, KBNET for short), the sixth row in the first column is Monitored Distillation for PositiveCongruent Depth Completion proposed by T.Liu, P.Agrawal, A.Chen, B.Hong (monitored distillation for regular consistent depth completion), the seventh row in the first column is Confidence Propagation through CNNsfor Guided Sparse Depth Regression proposed by A.Eldesokey, M.Felsberg and FSKhan (confidence propagation for sparse depth regression based on neural networks), the eighth row in the first column is Aerial Single-View Depth Completion With Image-Guided Uncertainty Estimation proposed by L.Teixeira and MROswald (aerial single-view depth completion based on image-guided uncertainty estimation), the ninth row in the first column is J.Park, K.Joo, Z.Hu, C.Liu and ISKweon proposed the Non-local Spatial Propagation Network for Depth Completion (Non-local Spatial Propagation Network for Depth Completion).

[0175] like Figure 7 As shown, it shows a positive qualitative comparison with advanced unsupervised methods. The first line is an RGB image, the second line is a sparse depth map generated by a visual-inertial SLAM system (the depth is represented by different colors), the third line is the unsupervised depth completion method of the calibrated back-projection layer proposed by Wong et al., and the fourth line is the method proposed in this embodiment, in which challenging areas are highlighted. The unsupervised depth completion method described in this embodiment performs better in these areas with mixed depths of small structures and in areas with obvious luminosity changes in the distance. The unsupervised depth completion method described in this embodiment performs better in these areas with mixed depths of small structures and in areas with obvious luminosity changes in the distance. It is particularly noteworthy that, for the purpose of visualization, this embodiment magnifies the sparse depth image and only displays the areas marked in green.

[0176] The unsupervised depth completion method described in this embodiment has been well tested for outdoor depth completion on the KITTI depth completion dataset. Because indoor scenes often have complex layouts and areas with weak textures, testing the unsupervised depth completion method described in this embodiment on the VOID dataset based on indoor scenes is both important and meaningful.

[0177] Using the evaluation indicators in Table 1, the unsupervised depth completion method described in this embodiment is compared with five excellent unsupervised depth completion methods and the most advanced supervised depth completion method. The comparison results are shown in Table 3. The unsupervised depth completion method described in this embodiment achieved the best results among the unsupervised methods. Specifically, it is better than the MDPC method by 17.05% MAE, 4.88% RMSE, 25.65% iMAE and 16.92% iRMSE. At the same time, the unsupervised depth completion method described in this embodiment is only 5.52% more RMSE than the most advanced supervised depth completion method. At the same time, the unsupervised depth completion method provided by this embodiment is much better than the supervised method in terms of completion time.

[0178] Table 3 Test results of VOID dataset

[0179]

[0180] To further illustrate the advantages of this embodiment, this embodiment selects challenging scenes from the VOID dataset and provides some qualitative examples. The fourth row in the first column shows the unsupervised depth completion from visual inertial odometry proposed by A. Wong, X. Fei, S. Tsuei and S. Soatto, and the fifth row in the first column shows the unsupervised depth completion with calibrated backprojection layers proposed by A. Wong and S. Soatto.

[0181] The unsupervised depth completion method provided by this embodiment and the existing unsupervised depth completion method of the calibration back projection layer can both reconstruct the sparse to dense dense mapping very well. However, in terms of edge sharpness and detail level, the method proposed in this embodiment is smoother and more consistent in prediction and reconstruction than the existing unsupervised depth completion method of the calibration back projection layer, especially in reconstruction along the edge. The reason is that this embodiment proposes a method and fusion strategy for propagating confidence between CNN layers, which makes it possible to generate continuous confidence values ​​of depth points for the output of the deep network and combine depth and RGB information to reconstruct edge structure information to the greatest extent. Figure 8 As shown, it can be clearly seen that the method of this embodiment is smoother and more consistent in terms of edge sharpness and detail level. Similarly, for image visualization, Figure 8 The sparse depth image is zoomed in and only the green marked area is shown.

[0182] This embodiment also designs a loss function for confidence, and trains the guided depth completion network twice. The first training uses a loss function with a confidence term, and the loss function is The second time, a loss function without a confidence term was used The training results are as follows Figure 9 As shown, Figure 9 The right axis represents the mean and standard deviation of the maximum output confidence of images in the KITTI dataset, and the left axis represents the MAE error. Figure 9 As can be seen, the loss function of this embodiment is trained to obtain a monotonically increasing confidence map, which improves the depth completion error until convergence. When the network is trained without confidence guidance, it will have a lower output confidence level and converge to a higher MAE.

[0183] The method described in this embodiment is also compared with some state-of-the-art methods in terms of parameter count and runtime on the KITTI dataset, where the parameter count and runtime are taken from relevant papers and the KITTI benchmark server.

[0184] As shown in Table 4, the real-time unsupervised depth completion method described in this embodiment ranks third in terms of the number of parameters. Compared to the first two networks developed for embedded systems with limited computing resources, this embodiment uses a relatively small number of parameters and can also be applied to embedded systems in robots. In addition, the method described in this embodiment completes a sparse depth map with a density of approximately 0.5% into a dense depth map in just 0.03 seconds. This makes the method provided in this embodiment equally advantageous among all methods and is a method with a high potential for maintaining real-time performance.

[0185] Table 4 Comparison of parameters and running time

[0186]

[0187] To further validate the generalization capabilities of the method provided in this embodiment in real-world scenarios, we compared the performance of this method and the unsupervised depth completion method using a calibrated back-projection layer trained on the VOID dataset with the OpenLORIS dataset. In the experiment, we selected 1,000 images from three complex real-world scenes: a cafe, a home, and a supermarket. We used the provided IMU data and RGB images to generate sparse depth images with a density of 0.5%, ensuring that both methods shared the same input source.

[0188] The qualitative results of the two methods are as follows Figure 10 As shown, it can be clearly seen in the marked areas that the method provided by this embodiment is more accurate in depth estimation, and the edge sharpness and details (buildings outside the window, child in the arms) after depth completion are better presented, proving that the method provided by this embodiment has stronger generalization ability.

[0189] To quantitatively determine the generalization error of the method provided by this embodiment in real-world scenarios, the method provided by this embodiment uses the depth image provided by the depth camera as the ground truth. The completion accuracy of this method was evaluated and compared with an unsupervised depth completion method using a calibrated back-projection layer. The comparison results, shown in Table 5, are consistent with the qualitative results of the method provided by this embodiment. The completion accuracy of the method provided by this embodiment is superior to that of the unsupervised depth completion method using a calibrated back-projection layer.

[0190] Table 5 Quantitative error comparison of depth completion results in real experimental scenes

[0191]

[0192] In Table 5, KBNet is an unsupervised depth completion method for calibrating the back-projection layer.

[0193] This embodiment proposes a new unsupervised depth completion method for depth completion in highly dynamic scenes. The unsupervised depth completion method jointly solves the translation residual matrix and the dense depth map. In order to solve the adverse effects of dynamic object motion and occlusion during training, the method proposed in this embodiment divides the RGB image into static, dynamic and occluded areas, and designs loss functions guided by the visual inertial system and confidence respectively. This function can simultaneously minimize data error and maximize output confidence. Finally, by combining the depth and RGB information with confidence to fuse structural information, the completed dense depth map is more accurate in terms of edge sharpness and details. The method described in this embodiment also comprehensively evaluates the generalization ability of the KITTI depth completion dataset, the VOID dataset, and real scenes. Compared with the most advanced unsupervised depth completion method, the method described in this embodiment achieves excellent performance with fewer network parameters.

[0194] It should be noted that the methods and detailed examples provided in the above embodiments can be combined with the devices and equipment provided in the embodiments, and references can be made to each other, and no further details will be given.

[0195] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0196] In the embodiments provided herein, it should be understood that the disclosed apparatus / terminal equipment and methods may be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For example, the division of the modules or units described above is merely a logical functional division, and actual implementation may employ other division methods, such as combining or integrating multiple units or components into another system, or omitting or not implementing certain features.

[0197] The above further describes the technical solution provided by the present invention in detail in conjunction with the accompanying drawings in order to highlight the advantages and benefits, and is not intended to limit the present invention. Any modification, combination of implementation methods, improvement and equivalent replacement of the present invention based on the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An unsupervised depth completion method based on a visual-inertial system and confidence guidance, characterized in that: The method comprises: Use a visual-inertial SLAM system to acquire RGB images, sparse depth maps, and camera poses; Building an unsupervised dense depth completion network; Processing the sparse depth map according to the unguided dense depth completion network to obtain confidence and unguided dense depth; Processing the confidence and the non-guided dense depth according to the guided depth completion network to obtain a guided dense depth map; Processing the guidance dense depth map, the RGB image, and the camera motion pose using a motion residual network to obtain a translation residual matrix; Training the guided depth completion network according to the Losses loss function to obtain a guided depth completion network training model; Training the motion residual network according to the Losses loss function to obtain a motion residual network training model; The guided depth completion network for estimating dense depth based on the confidence and the unguided dense depth includes: The confidence and non-guided dense depth are processed by 3*3 2D convolution + activation function, 3*3 2D convolution + Leaky activation function and 2D convolution to obtain a dense depth guided depth completion network; The motion residual network is specifically: There are 5 convolution stacks with a stride of 2, and the last convolution stack is an average pool with 1024 channels and a spatial resolution of 1*1; The motion residual network also includes two 1*1 convolutional layers of three channels.

2. The unsupervised depth completion method based on a visual inertial system and confidence guidance according to claim 1, characterized in that: The step of processing the sparse depth map according to the unguided dense depth completion network to obtain confidence and unguided dense depth includes: Using a normalized convolutional layer to separate the depth and confidence of the sparse depth map; Confidence and unguided dense depth are obtained through confidence adaptation processing.

3. The unsupervised depth completion method based on a visual inertial system and confidence guidance according to claim 1, characterized in that: The guiding depth completion network is trained according to the Losses loss function to obtain the guiding depth completion network training model, specifically: Acquire a dynamic region and a static region according to the translation residual matrix, wherein the dynamic region is a dynamic object region at a current moment, and the static region is a static structure region at a previous moment; Calculate the photometric consistency loss for pixels in the static area: ， in, is the photometric consistency loss of pixels in the static area, is the total number of pixels in the static area, is the pixel coordinate The pixel value of To reconstruct pixel coordinates The pixel value of for t The coordinates of a pixel point in a static area at a certain moment, for t The coordinates of a pixel point in a static area at time +1; Calculate the photometric consistency loss for pixels in the dynamic area: ， in, is the photometric consistency loss of pixels in the dynamic area, is the pixel coordinate The pixel value of is the pixel coordinate The pixel value of for t The coordinates of the target pixel point in a dynamic area at time +1, for t The coordinates of the target pixel point in a certain dynamic area at the moment; Calculate the parallax smoothing loss for all pixels: ， in, is the parallax smoothing loss for all pixels, For u Conduct partial guidance, is the pixel coordinate ( u,v ) parallax, is the pixel coordinate ( u,v ) pixels, is a constant; According to the photometric consistency loss of the static area and the dynamic area and the parallax smoothing loss of all pixels, the The loss function is: ， ， in, is the loss function without confidence guidance, p is the number of training sessions; The guided depth completion network is trained according to the loss function to obtain a guided depth completion network training model.

4. The unsupervised depth completion method based on a visual inertial system and confidence guidance according to claim 1, characterized in that: The step of training the motion residual network according to the Losses loss function to obtain a motion residual network training model includes: ， in, is the loss function of the motion residual network, is the mean absolute error, is the translation residual, for u Pixel deviation in direction, for v Pixel deviation in the direction.

5. An unsupervised depth completion system based on a visual-inertial system and confidence guidance, characterized in that: The system is implemented based on the unsupervised depth completion method based on a visual inertial system and confidence guidance according to claim 1, and the system includes: An image information acquisition unit, used to acquire RGB images, sparse depth maps, and camera poses using a visual-inertial SLAM system; An unsupervised dense depth completion network building unit, used to build an unsupervised dense depth completion network; A confidence and unguided dense depth acquisition unit, configured to process the sparse depth map according to the unguided dense depth completion network to acquire confidence and unguided dense depth; a guided dense depth map acquisition unit, configured to process the confidence and the non-guided dense depth according to the guided depth completion network to acquire a guided dense depth map; A translation residual matrix acquisition unit, configured to process the guidance dense depth map, the RGB image, and the camera motion pose using a motion residual network to acquire a translation residual matrix; A guided depth completion network training model acquisition unit is used to train the guided depth completion network according to the Losses loss function to obtain a guided depth completion network training model; The motion residual network training model acquisition unit is used to train the motion residual network according to the Losses loss function to obtain the motion residual network training model.

6. The unsupervised depth completion system based on a visual inertial system and confidence guidance according to claim 5, characterized in that: The confidence and non-guided dense depth acquisition unit includes: Using a normalized convolutional layer to separate the depth and confidence of the sparse depth map; Confidence and unguided dense depth are obtained through confidence adaptation processing.

7. A computer device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the unsupervised depth completion method based on a visual inertial system and confidence guidance according to any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the unsupervised depth completion method based on a visual-inertial system and confidence guidance as described in any one of claims 1 to 4.

Images