Depth completion method, system and terminal based on three-dimensional feature extraction and fusion
By employing a self-supervised depth completion method based on 3D feature extraction and fusion, a complete depth map is generated using LiDAR and camera data. This solves the problem of missing information in depth detection devices and improves the accuracy and robustness of computer vision tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN UNIV
- Filing Date
- 2024-03-19
- Publication Date
- 2026-07-10
Smart Images
Figure CN118115556B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision, and in particular to a self-supervised depth completion method, system, terminal, and computer-readable storage medium based on three-dimensional feature extraction and fusion. Background Technology
[0002] Computer vision, a research area within artificial intelligence, analyzes images in real or virtual environments. Through various specific analysis methods, it aims to extract information from images that is helpful for the current task. Similar to how the human eye perceives the external environment, computers capture images using cameras and other imaging devices, processing these images as input signals to obtain the desired results. However, color images captured by cameras only possess two-dimensional image information, providing only color and texture information, lacking information about the scene and the location of objects. To further improve the computer's ability to perceive environmental information, many current processing methods incorporate depth information into various computer vision tasks.
[0003] The acquisition of depth images containing depth information mainly relies on various depth sensors. The mainstream depth sensors include binocular stereo vision cameras, structured light depth sensors, and lidar sensors (LiDAR). However, existing depth detection devices all have the problem of missing information in some areas of the produced depth images, which will have a negative impact on the performance of subsequent computer vision tasks.
[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention
[0005] The main objective of this invention is to provide a self-supervised depth completion method, system, terminal, and computer-readable storage medium based on three-dimensional feature extraction and fusion. This invention aims to solve the problem that some regions of the depth map produced by depth detection equipment in the prior art have missing information, which will have a negative impact on the performance of subsequent computer vision tasks.
[0006] To achieve the above objectives, this invention provides a self-supervised depth completion method based on 3D feature extraction and fusion, which includes the following steps:
[0007] The discrete depth map acquired by the lidar and the color map acquired by the camera are obtained. The discrete depth map is preprocessed to obtain the target discrete depth map, and the color map is downsampled to obtain the target color feature map.
[0008] After performing downsampling and feature extraction on the target discrete depth map and the color map a preset number of times, a first color feature map and a first target discrete depth feature map are obtained. After performing channel parallel operation and upsampling operation on the first color feature map and the first target discrete depth feature map, a fused image feature map is obtained.
[0009] The fused image feature map and the target discrete depth map are input into the cross-attention feature fusion module, and the fused feature map is output.
[0010] The target color feature map and the fused feature map are connected in parallel through channels and then upsampled to obtain a complete depth map.
[0011] Optionally, the preprocessing of the discrete depth map to obtain the target discrete depth map specifically includes:
[0012] The discrete depth map and the color map are downsampled to obtain multi-scale feature output;
[0013] The multi-scale feature output is scaled up using the nearest neighbor interpolation method to obtain expanded multi-scale features.
[0014] The extended multi-scale features are input into a multilayer perceptron for feature forward propagation to obtain a confidence matrix. The confidence matrix is then processed according to a preset accuracy threshold to obtain a depth mask.
[0015] The discrete depth map is obtained by removing outliers based on the depth mask.
[0016] Optionally, the step of downsampling and feature extraction on the target discrete depth map and the color map a preset number of times to obtain a first color feature map and a first target discrete depth feature map specifically includes:
[0017] After performing three downsampling and feature extractions on the target discrete depth map and the color map, a second color feature map and a second target discrete depth feature map are obtained.
[0018] After downsampling the second color feature map and the second target discrete depth feature map twice, feature extraction is performed to obtain the first color feature map and the first target discrete depth feature map.
[0019] Optionally, the step of performing three downsampling operations and feature extractions on the target discrete depth map and the color map to obtain a second color feature map and a second target discrete depth feature map specifically includes:
[0020] After downsampling the target discrete depth map and the color map respectively, a first downsampled target discrete depth map and a first downsampled color map are obtained;
[0021] The first downsampled target discrete depth map and the first downsampled color map are input into the graph propagation-based feature extraction module for feature extraction to obtain the third target discrete depth feature map and the third color feature map.
[0022] After performing downsampling and feature extraction twice on the third target discrete depth feature map and the third color feature map, the second color feature map and the second target discrete depth feature map are obtained.
[0023] Optionally, the step of inputting the first downsampled target discrete depth map and the first downsampled color map into a graph propagation-based feature extraction module for feature extraction to obtain a third target discrete depth feature map and a third color feature map specifically includes:
[0024] The first downsampled target discrete depth map, the first downsampled color map, and the target discrete depth map are input into the graph propagation-based feature extraction module. The feature extraction module uses the neighborhood K-point distance operation method to obtain the first depth feature pixel distance coordinate matrix and the first color image feature pixel distance coordinate matrix based on the first downsampled target discrete depth map, the first downsampled color map, and the target discrete depth map.
[0025] The first depth feature pixel distance coordinate matrix and the first color image feature pixel distance coordinate matrix are respectively input into the multilayer perceptron in the feature extraction module to obtain the first updated depth feature pixel distance coordinate matrix and the first updated color image feature pixel distance coordinate matrix.
[0026] The first updated depth feature pixel distance coordinate matrix and the first downsampled target discrete depth map are multiplied together to obtain the third target discrete depth feature map. The first updated color map feature pixel distance coordinate matrix and the first downsampled color map are multiplied together to obtain the third color feature map.
[0027] Optionally, the step of inputting the fused image feature map and the target discrete depth map into the cross-attention feature fusion module and outputting the fused feature map specifically includes:
[0028] The fused image feature map and the target discrete depth map are input into the three-dimensional information extraction module in the cross-attention feature fusion module to obtain the first three-dimensional feature information;
[0029] Point cloud generation operations are performed on the three-dimensional feature information and the fused image feature map to obtain a two-dimensional feature image and a second three-dimensional feature image;
[0030] The two-dimensional feature image and the second three-dimensional feature image are input into the three-dimensional to two-dimensional feature cross-attention module in the cross-attention feature fusion module to obtain pre-fused features;
[0031] The pre-fused features are normalized to obtain the fused features.
[0032] Optionally, the step of inputting the fused image feature map and the target discrete depth map into the three-dimensional information extraction module in the cross-attention feature fusion module to obtain the first three-dimensional feature information specifically includes:
[0033] The fused image feature map and the target discrete depth map are input into the three-dimensional information extraction module in the cross-attention feature fusion module to extract the pixel values of the discrete depth effective point coordinates in the fused image feature map;
[0034] Based on the pixel values of the effective coordinates of the discrete depth points and the target discrete depth map, the neighborhood K-point distance calculation method is used to obtain the feature pixel distance and the point cloud geometric distance.
[0035] The feature pixel distance and the point cloud geometric distance are input into the multilayer perceptron in the 3D information extraction module of the cross-attention feature fusion module to obtain the attention matrix;
[0036] Based on the attention matrix and the target discrete depth map, a channel connection operation is performed to obtain the first three-dimensional feature information.
[0037] Furthermore, to achieve the above objectives, the present invention also provides a self-supervised depth completion system based on three-dimensional feature extraction and fusion, wherein the self-supervised depth completion system based on three-dimensional feature extraction and fusion includes:
[0038] The preprocessing module is used to acquire discrete depth maps collected by lidar and color maps collected by camera, and to preprocess the discrete depth maps to obtain target discrete depth maps, and to downsample the color maps to obtain target color feature maps;
[0039] The fused image feature map generation module is used to perform downsampling and feature extraction on the target discrete depth map and the color map a preset number of times to obtain a first color feature map and a first target discrete depth feature map. After performing channel parallel operation and upsampling operation on the first color feature map and the first target discrete depth feature map, a fused image feature map is obtained.
[0040] The fusion feature output module is used to input the fused image feature map and the target discrete depth map into the cross-attention feature fusion module and output the fused feature map;
[0041] The result output module is used to perform channel parallel connection on the target color feature map and the fused feature map, and to perform upsampling to obtain a completed depth map.
[0042] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a self-supervised depth completion program based on three-dimensional feature extraction and fusion stored in the memory and executable on the processor, wherein when the self-supervised depth completion program based on three-dimensional feature extraction and fusion is executed by the processor, it implements the steps of the self-supervised depth completion method based on three-dimensional feature extraction and fusion as described above.
[0043] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a self-supervised depth completion program based on three-dimensional feature extraction and fusion, and when the self-supervised depth completion program based on three-dimensional feature extraction and fusion is executed by a processor, it implements the steps of the self-supervised depth completion method based on three-dimensional feature extraction and fusion as described above.
[0044] In this invention, a discrete depth map acquired by a lidar and a color image acquired by a camera are obtained. The discrete depth map is preprocessed to obtain a target discrete depth map, and the color image is downsampled to obtain a target color feature map. After downsampling and feature extraction of the target discrete depth map and the color image a preset number of times, a first color feature map and a first target discrete depth feature map are obtained. The first color feature map and the first target discrete depth feature map are then subjected to channel parallel concatenation and upsampling operations to obtain a fused image feature map. The fused image feature map and the target discrete depth map are input into a cross-attention feature fusion module to output a fused feature map. Finally, the target color feature map and the fused feature map are subjected to channel parallel concatenation and upsampling to obtain a completed depth map. This invention extracts three-dimensional spatial information from a discrete depth map and applies attention to a color image based on this information, encouraging the network to learn the correlation information between features of neighboring locations in three-dimensional space. At the same time, based on the cross-attention feature fusion module, it learns the three-dimensional spatial feature information in the discrete depth map and fuses it with two-dimensional image features, thereby finally obtaining a complete and augmented depth map. With the augmented depth map, subsequent computer vision tasks can be accurately processed. Attached Figure Description
[0045] Figure 1 This is a flowchart of a preferred embodiment of the self-supervised depth completion method based on three-dimensional feature extraction and fusion of the present invention;
[0046] Figure 2 This is the overall network framework diagram of the self-supervised depth completion method based on three-dimensional feature extraction and fusion of the present invention;
[0047] Figure 3 This is a diagram illustrating the depth mask generation process of the self-supervised depth completion method based on 3D feature extraction and fusion, as described in this invention.
[0048] Figure 4 This is a structural diagram of the graph propagation-based feature extraction module of the self-supervised depth completion method based on 3D feature extraction and fusion of the present invention;
[0049] Figure 5 This is a structural diagram of the cross-attention feature fusion module of the self-supervised depth completion method based on 3D feature extraction and fusion of the present invention;
[0050] Figure 6 This is a structural diagram of the three-dimensional information extraction module of the self-supervised depth completion method based on three-dimensional feature extraction and fusion of the present invention;
[0051] Figure 7 This is a structural diagram of the cross-attention feature fusion module of the self-supervised depth completion method based on 3D feature extraction and fusion of the present invention;
[0052] Figure 8 This is a structural diagram of a preferred embodiment of the self-supervised depth completion system based on three-dimensional feature extraction and fusion of the present invention;
[0053] Figure 9 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation
[0054] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0055] Computer vision analyzes images in real or virtual environments, extracting information relevant to the current task through various specific analytical methods. Currently, with the continuous development of computer vision, its role in production work such as face recognition, object detection, autonomous driving, and medical image analysis is becoming increasingly important. Similar to how the human eye perceives the external environment, computers capture images through cameras and other imaging devices, processing them as input signals to obtain the desired results. However, color images captured by cameras only possess two-dimensional image information, providing only color and texture information, lacking information about the scene and the location of objects. To further improve the ability of computers to perceive environmental information, existing methods incorporate depth information into many computer vision tasks.
[0056] Depth information represents the distance between an image and a camera, representing the three-dimensional information received by the human eye when perceiving the environment. For many computer vision tasks such as autonomous driving and 3D scene reconstruction, the accurate perception of the location and distance of the current scene and objects is crucial, and the introduction of depth information can significantly improve the machine's ability to perceive position. Currently, the acquisition of depth images mainly relies on various depth sensors, with mainstream depth sensors including binocular stereo vision cameras, structured light depth sensors, time-of-flight cameras, and LiDAR sensors. However, these common sensors each have their own shortcomings. Binocular stereo vision cameras, in addition to requiring a long baseline and complex calibration and calculation processes, suffer from significant information loss in the disparity map of the output in areas lacking features. Structured light depth sensors are not suitable for outdoor environments because they are sensitive to lighting and will exhibit detection bias under strong light, resulting in the loss of some depth features. The two depth detectors mentioned above are mainly used for indoor depth detection. In outdoor scenes, LiDAR has high-precision depth information capture capabilities, but the number of pixels it scans is sparse, making the output depth image very sparse. Therefore, all existing depth detection equipment has the same shortcoming: some areas of the produced depth map have missing information.
[0057] This invention addresses one or more of the above-mentioned problems by proposing a self-supervised depth completion method based on 3D feature extraction and fusion, which can obtain a complete, completed depth map. The completed depth map can then be used to accurately process subsequent computer vision tasks.
[0058] The self-supervised depth completion method based on 3D feature extraction and fusion described in the preferred embodiment of the present invention, such as... Figure 1 As shown, the self-supervised depth completion method based on 3D feature extraction and fusion includes the following steps:
[0059] Step S10: Obtain the discrete depth map acquired by the lidar and the color map acquired by the camera, preprocess the discrete depth map to obtain the target discrete depth map, and downsample the color map to obtain the target color feature map.
[0060] Specifically, in this application, the discrete depth map is acquired by a LiDAR, while the color map is acquired by a camera. After acquiring the discrete depth map and the color map, the discrete depth map is preprocessed to obtain the target discrete depth map, and the color map is downsampled to obtain the target color feature map.
[0061] It should be noted that the self-supervised deep completion method based on 3D feature extraction and fusion in this application is implemented based on a deep completion network model based on 3D feature extraction and attention fusion. Furthermore, the models used in the self-supervised deep completion method based on 3D feature extraction and fusion in this application are all included in the deep completion network model based on 3D feature extraction and attention fusion. The specific structure of the deep completion network model based on 3D feature extraction and attention fusion is as follows: Figure 2 As shown, by Figure 2 It can be seen that the deep completion network model based on 3D feature extraction and fusion attention includes two main modules: a feature extraction module based on graph propagation and a cross-attention feature fusion module. Through the deep completion network model based on 3D feature extraction and fusion attention, discrete depth maps acquired by LiDAR and color maps acquired by cameras can be processed to obtain corresponding completed depth maps.
[0062] Furthermore, the preprocessing of the discrete depth map to obtain the target discrete depth map specifically includes:
[0063] The discrete depth map and the color map are downsampled to obtain multi-scale feature output;
[0064] The multi-scale feature output is scaled up using the nearest neighbor interpolation method to obtain expanded multi-scale features.
[0065] The extended multi-scale features are input into a multilayer perceptron for feature forward propagation to obtain a confidence matrix. The confidence matrix is then processed according to a preset accuracy threshold to obtain a depth mask.
[0066] The discrete depth map is obtained by removing outliers based on the depth mask.
[0067] Specifically, the information in a discrete depth map has the characteristics of sparse distribution but accurate depth values. However, not every discrete depth pixel has the correct depth information. The errors in the discrete depth acquired by the lidar mainly stem from the difference between the viewpoint of the lidar sensor and the viewpoint of the camera. This causes occlusion of some foreground and background images in the observed scene during the back projection process, resulting in incorrect depth values in these boundary areas. These incorrect depth values are generally referred to as outliers.
[0068] Therefore, in order to obtain accurate discrete depth information, this application uses preprocessing to obtain a target discrete depth map, as shown below. Figure 2 The outlier removal section. Outlier removal requires the application of a depth mask, and the depth mask generation process is as follows: Figure 3As shown, this process employs a deep learning strategy, inputting both the discrete depth map and the color map into a downsampled convolutional neural network to obtain multi-scale feature outputs. Then, the nearest neighbor difference method is used to expand the feature maps of different scales to match the scale of the input discrete depth map. Subsequently, the set of expanded multi-scale features is input into a multilayer perceptron for feature forward propagation to obtain a confidence matrix. By selecting pixels in the confidence matrix that are below a preset accuracy threshold, a depth mask is formed to remove outliers. The outlier removal operation of the discrete depth map is completed through the depth mask.
[0069] Step S20: After performing downsampling and feature extraction on the target discrete depth map and the color map a preset number of times, a first color feature map and a first target discrete depth feature map are obtained. After performing channel parallel operation and upsampling operation on the first color feature map and the first target discrete depth feature map, a fused image feature map is obtained.
[0070] Specifically, in this application, the target discrete depth map and color map are downsampled and feature extracted a preset number of times, which is 3 times. Then, the corresponding feature maps generated from the two maps are downsampled twice, and then feature extraction is performed once through a convolution to obtain the first color feature map and the first target discrete depth feature map. The obtained first color feature map and the first target discrete depth feature map are then connected in parallel through channels, and the obtained features are upsampled 4 times to obtain the fused image feature map.
[0071] Further, the step of downsampling and feature extraction on the target discrete depth map and the color map a preset number of times to obtain a first color feature map and a first target discrete depth feature map specifically includes:
[0072] After performing three downsampling and feature extractions on the target discrete depth map and the color map, a second color feature map and a second target discrete depth feature map are obtained.
[0073] After downsampling the second color feature map and the second target discrete depth feature map twice, feature extraction is performed to obtain the first color feature map and the first target discrete depth feature map.
[0074] Specifically, when the first color feature map and the first target discrete depth feature map are obtained, the target discrete depth map and the color map are downsampled and feature extracted three times to obtain the second color feature map and the second target discrete depth feature map. After downsampling the second color feature map and the second target discrete depth feature map twice, feature extraction is performed through convolution to obtain the first color feature map and the first target discrete depth feature map.
[0075] The step of performing three downsampling and feature extraction on the target discrete depth map and the color map to obtain the second color feature map and the second target discrete depth feature map specifically includes:
[0076] After downsampling the target discrete depth map and the color map respectively, a first downsampled target discrete depth map and a first downsampled color map are obtained;
[0077] The first downsampled target discrete depth map and the first downsampled color map are input into the graph propagation-based feature extraction module for feature extraction to obtain the third target discrete depth feature map and the third color feature map.
[0078] After performing downsampling and feature extraction twice on the third target discrete depth feature map and the third color feature map, the second color feature map and the second target discrete depth feature map are obtained.
[0079] Specifically, the results of downsampling the color image are a first downsampled color image and a target color feature image, where the first downsampled color image and the target color feature image are the same feature images, but they undergo different operations.
[0080] The target discrete depth map and color map are downsampled and feature extracted three times. Specifically, the target discrete depth map and color map are downsampled respectively and then input into the graph propagation-based feature extraction module for feature extraction. The results are then downsampled twice more and processed by the graph propagation-based feature extraction module to obtain the second color feature map and the second target discrete depth feature map.
[0081] The step of inputting the first downsampled target discrete depth map and the first downsampled color map into the graph propagation-based feature extraction module for feature extraction to obtain the third target discrete depth feature map and the third color feature map specifically includes:
[0082] The first downsampled target discrete depth map, the first downsampled color map, and the target discrete depth map are input into the graph propagation-based feature extraction module. The feature extraction module uses the neighborhood K-point distance operation method to obtain the first depth feature pixel distance coordinate matrix and the first color image feature pixel distance coordinate matrix based on the first downsampled target discrete depth map, the first downsampled color map, and the target discrete depth map.
[0083] The first depth feature pixel distance coordinate matrix and the first color image feature pixel distance coordinate matrix are respectively input into the multilayer perceptron in the feature extraction module to obtain the first updated depth feature pixel distance coordinate matrix and the first updated color image feature pixel distance coordinate matrix.
[0084] The first updated depth feature pixel distance coordinate matrix and the first downsampled target discrete depth map are multiplied together to obtain the third target discrete depth feature map. The first updated color map feature pixel distance coordinate matrix and the first downsampled color map are multiplied together to obtain the third color feature map.
[0085] Specifically, the structure of the graph propagation-based feature extraction module is as follows: Figure 4 As shown, the feature extraction process of this module consists of two parts: K-Nearest Neighbors (KNN) distance calculation and an attention-based feature extraction network. During the three downsampling and feature extraction processes on the discrete depth map and color map of the target, a graph propagation-based feature extraction module is used for each feature extraction step.
[0086] The neighborhood K-point matrix operation is used to perform 3D reconstruction of the coordinates of each point in a discrete depth image to extract the geometric relationships between the points in 3D space. First, for any discrete depth image as input, a 3D coordinate matrix in the depth camera coordinate system is constructed based on the depth value of each discrete depth point and its 2D coordinates in the image. The matrix structure is as follows: Here, h and w represent the height and width of the discrete depth map. The next step involves using the camera intrinsic matrix K to map the x and y coordinates from the 3D coordinate matrix to the world coordinate system, obtaining the 3D coordinate matrix G in world coordinates. It is important to note that during multi-scale map propagation, the camera intrinsic matrix K needs to be scaled during the mapping process for discrete depth coordinates at different scaling scales. Finally, the Euclidean distance matrix between each pair of coordinate points is calculated. Where N is the number of discrete depth points, T represents the transpose, and the Euclidean distance matrix is... The calculation process is as follows:
[0087] ;
[0088] The distance matrix allows us to obtain the distance between each coordinate point in matrix G and its neighboring coordinate points, thus identifying the K points with the closest Euclidean distance to each coordinate point, resulting in the KNN coordinate matrix. The KNN coordinate matrix stores the 3D coordinates of the pixels of multiple neighboring points of each depth pixel. By using the distance between each pixel, the central depth point in the corresponding discrete depth map is linked to its neighboring depth points in a spatially irregular distribution, which helps the network learn the 3D geometric structure information in the depth map.
[0089] It is important to note that when deploying multi-scale feature extraction modules in a network, the number of discrete points in different depth maps input during the same batch of training is inconsistent, and the scale of discrete depth maps required by feature extraction modules at different network layer depths is also different. Therefore, before inputting the discrete depth map into the feature extraction module of this method, it is necessary to perform a minimum pooling downsampling operation on the discrete depth map and randomly sample a uniform number of effective depth values to obtain a discrete depth map containing the same number of effective depth point clouds as the input of this module.
[0090] The second component of the graph propagation-based feature extraction module is an attention-based feature extraction network. Its function is to combine the 3D geometric information obtained above to extract features from the depth map and color map based on their geometric positional relationships. This attention-based feature extraction network is essentially the multilayer perceptron within the feature extraction module. The obtained KNN coordinate matrix is then used... The pixel values of depth features and color image features at these coordinates can be directly obtained. By directly calculating the difference between the pixel values of the center pixel and the K neighboring points, the depth feature pixel distance matrix and the color image feature pixel distance matrix are obtained, which are the first depth feature pixel distance coordinate matrix and the first color image feature pixel distance coordinate matrix in practical applications. After feature updates by a multi-layer perceptron (MLP), these two distance matrices yield corresponding matrices, namely the first updated depth feature pixel distance coordinate matrix and the first updated color image feature pixel distance coordinate matrix in practical applications. The first updated depth feature pixel distance coordinate matrix and the first updated color image feature pixel distance coordinate matrix serve as attention matrices combining three-dimensional spatial positional relationships. They are multiplied by their respective features to obtain discrete spatial attention-weighted depth and color image feature information. That is, in practical applications, the first updated depth feature pixel distance coordinate matrix and the first downsampled target discrete depth map are multiplied to obtain the third target discrete depth feature map, and the first updated color image feature pixel distance coordinate matrix and the first downsampled color map are multiplied to obtain the third color feature map.
[0091] Step S30: Input the fused image feature map and the target discrete depth map into the cross-attention feature fusion module, and output the fused feature map.
[0092] It should be noted that, in this application, in order to realize the self-supervised depth completion method based on three-dimensional feature extraction and fusion, this application uses a cross-attention feature fusion module to process the fused image feature map and the target discrete depth map, and outputs the fused features.
[0093] The cross-attention feature fusion module combines depth feature information from the network's convolutional layers with discrete depth point cloud information. During the multi-scale feature output stage of the decoder, it propagates the 3D geometric information of each point in the corresponding discrete depth map to its local region within the image features. In other words, it uses the 3D geometric position information from the corresponding discrete depth features to reconstruct the 3D depth information from the features obtained from the depth map and color image. Furthermore, the cross-attention feature fusion module alleviates the problem of incomplete extraction of discrete depth information features due to the limited field of view of 2D convolutional kernels, while also improving the network's robustness to discrete depth inputs with varying degrees of sparsity.
[0094] Furthermore, the network structure of the cross-attention feature fusion module is as follows: Figure 5 As shown, this feature fusion network module is mainly composed of two sub-modules: a 3D information extraction module and a 3D-to-2D feature cross-attention module.
[0095] The specific structure of the 3D information extraction module is as follows: Figure 6 As shown, the design of the 3D information extraction module borrows from the graph propagation-based feature extraction module. For the input image features, the 3D information extraction module extracts the pixel values at the effective depth point coordinates. For discrete depth information with the same feature scale as the current feature scale, it converts it into a 3D coordinate matrix, i.e., a point cloud matrix, and calculates the K nearest neighbors of each pixel. Based on the distance between them, the KNN matrix representing pixel intensity distance in the image feature domain is concatenated with the KNN matrix representing geometric distance in the discrete depth space. This is then processed by a multilayer perceptron to obtain an attention matrix for each point in the discrete depth point cloud coordinates, and attention is applied to the image features through this attention matrix. Afterwards, the attention-weighted image features are added to the discrete depth information (which has undergone channel dimensionality upscaling via linear embedding) and concatenated with the original discrete depth information to finally obtain the 3D feature information as output. Specifically, before performing the cross-attention operation between the 3D features and image features at discrete depth, the 3D information extraction module fuses the image features at the effective depth coordinates with the point cloud coordinates in 2D space, preparing for the next step of the 3D-to-2D feature cross-attention module.
[0096] As the core component of this feature fusion module, the 3D-to-2D feature cross-attention module adopts a partial network structure based on the residual attention mechanism as its module framework. Its main structure is as follows: Figure 7As shown, the cross-attention module for 3D-to-2D features uses both image feature information from the network and 3D feature information output from the 3D information extraction module as input for feature fusion between the two modalities. First, to ensure scale consistency between the features of the two modalities, a method for generating depth point cloud matrices is used. A convolution function is employed to generate depth information for mapping from the image features, and this is combined with the camera intrinsic parameter matrix to obtain the 3D spatial coordinates of the 2D image features. Then, the point cloud coordinate matrix of the image features is concatenated with the image features along the channel dimension as the output. It is important to note that the coordinate elements in the depth coordinate matrix are replaced with constants to avoid the generated depth affecting the subsequent KNN operation based on geometric distance between the image features and the 3D features.
[0097] Further, the step of inputting the fused image feature map and the target discrete depth map into the cross-attention feature fusion module and outputting the fused feature map specifically includes:
[0098] The fused image feature map and the target discrete depth map are input into the three-dimensional information extraction module in the cross-attention feature fusion module to obtain the first three-dimensional feature information;
[0099] Point cloud generation operations are performed on the three-dimensional feature information and the fused image feature map to obtain a two-dimensional feature image and a second three-dimensional feature image;
[0100] The two-dimensional feature image and the second three-dimensional feature image are input into the three-dimensional to two-dimensional feature cross-attention module in the cross-attention feature fusion module to obtain pre-fused features;
[0101] The pre-fused features are normalized to obtain the fused features.
[0102] Specifically, based on the cross-attention feature fusion module proposed in this application, the fused image feature map and the target discrete depth map are processed. That is, the fused image feature map and the target discrete depth map are first input into the three-dimensional information extraction module in the cross-attention feature fusion module. The three-dimensional information extraction module proposed in this application processes the data to obtain the first three-dimensional feature information, and then performs point cloud generation to obtain the two-dimensional feature image and the second three-dimensional feature image. Then, the two-dimensional feature image and the second three-dimensional feature image are input into the three-dimensional to two-dimensional feature cross-attention module proposed in this application to obtain the pre-fused features. Finally, the fused features are obtained through the normalization operation of Norm+MLP.
[0103] The step of inputting the fused image feature map and the target discrete depth map into the three-dimensional information extraction module of the cross-attention feature fusion module to obtain the first three-dimensional feature information specifically includes:
[0104] The fused image feature map and the target discrete depth map are input into the three-dimensional information extraction module in the cross-attention feature fusion module to extract the pixel values of the discrete depth effective point coordinates in the fused image feature map;
[0105] Based on the pixel values of the effective point coordinates of the discrete depth and the target discrete depth map, the neighborhood K-point distance calculation method is used to obtain the feature pixel distance and the point cloud geometric distance.
[0106] The feature pixel distance and the point cloud geometric distance are input into the multilayer perceptron in the 3D information extraction module of the cross-attention feature fusion module to obtain the attention matrix;
[0107] Based on the attention matrix and the target discrete depth map, a channel connection operation is performed to obtain the first three-dimensional feature information.
[0108] Specifically, the process of obtaining the first three-dimensional feature information through the three-dimensional information extraction module involves first extracting the pixel values of the discrete depth effective point coordinates in the feature map of the fused image; then, based on the pixel values of the discrete depth effective point coordinates and the target discrete depth map, using the neighborhood K-point distance calculation method, obtaining the feature pixel distance and the point cloud geometric distance; then, inputting the feature pixel distance and the point cloud geometric distance into the multilayer perceptron in the three-dimensional information extraction module of the cross-attention feature fusion module to obtain the attention matrix; then, performing linear embedding on the target discrete depth map, adding the discrete depth information obtained by channel upscaling through linear embedding to the attention matrix, and then performing channel concatenation with the target discrete depth map to obtain the first three-dimensional feature information.
[0109] Step S40: Connect the target color feature map and the fused feature map in parallel channels and upsample them to obtain a complete depth map.
[0110] Specifically, after obtaining the target color feature map and the fused feature map, the target color feature map and the fused feature map are connected in parallel through channels, and then upsampled again to obtain a complete depth map with the same scale as the discrete depth map.
[0111] Furthermore, the deep completion network model based on 3D feature extraction and attention fusion in this application is trained using a self-supervised learning method. The deep completion network model based on 3D feature extraction and attention fusion is composed of all the networks (models) in the self-supervised deep completion method based on 3D feature extraction and attention fusion.
[0112] The deep completion network model based on 3D feature extraction and attention fusion is trained end-to-end as a whole, and a loss function composed of depth consistency loss, optical structure loss and depth smoothing loss is introduced to optimize the network.
[0113] Since self-supervised learning training methods do not rely on true depth values for depth error calculation, the depth loss function compares the discrete depth map input with the network output's completed depth map, calculating their difference to obtain the depth consistency loss. The optical structure loss function evaluates the accuracy of information in the output depth map by comparing the structural differences between the target viewpoint image and the source viewpoint image after mapping. The depth smoothing loss function complements the above two loss functions. Because both depth consistency loss and optical structure loss only calculate the error of individual pixels, lacking continuity constraints between pixels, this may lead to high discontinuities on the same object surface in the output depth map. The total loss function is obtained by adding the three loss functions after weight adjustment; the calculation formulas for the three loss functions and the total loss function are shown below:
[0114] Deep consistency loss Represented as:
[0115] ;
[0116] Optical structure loss Represented as:
[0117] ;
[0118] Deep smoothing loss Represented as:
[0119] ;
[0120] Total loss function Represented as:
[0121] ;
[0122] Where x is the number of pixels in the discrete depth map acquired by the LiDAR during training. This is a discrete depth map. Output a depth map for the network; N is the number of valid values in the discrete depth map; , and The preset weight parameters were set to 0.6, 1.0, and 0.04 respectively. M The number of pixels in the image. Represents the source view image. express The reconstructed image, Represents the gradient operator. This represents the depth map output by the network.
[0123] As an auxiliary module that integrates with the backbone network to preprocess discrete depth information, the outlier removal network is also trained independently using a self-supervised training method. By generating pseudo-labels, the network takes the minimum non-zero value of the 9×9 neighborhood surrounding each valid point in the discrete depth map and replaces that pixel's depth value. This operation ensures that erroneous background pixels in areas where foreground and background depths are mixed are replaced by foreground pixels with smaller depth values. The pseudo-label depth obtained in this way, along with the depth confidence score output by the network, is used to optimize the network parameters using a negative log-likelihood loss function.
[0124] The loss function for outlier removal is shown below:
[0125] ;
[0126] in, Let d represent the depth confidence score of the network output, where d is the discrete depth. This is a deep pseudo-tag.
[0127] This invention extracts three-dimensional spatial information from a discrete depth map and applies attention to a color image based on this information, encouraging the network to learn the correlation information between features of neighboring locations in three-dimensional space. At the same time, based on the cross-attention feature fusion module, it learns the three-dimensional spatial feature information in the discrete depth map and fuses it with two-dimensional image features, thereby finally obtaining a complete and augmented depth map. With the augmented depth map, subsequent computer vision tasks can be accurately processed.
[0128] Furthermore, such as Figure 8 As shown, based on the above-mentioned self-supervised depth completion method based on 3D feature extraction and fusion, the present invention also provides a self-supervised depth completion system based on 3D feature extraction and fusion, wherein the self-supervised depth completion system based on 3D feature extraction and fusion includes:
[0129] The preprocessing module 81 is used to acquire the discrete depth map collected by the lidar and the color map collected by the camera, and to preprocess the discrete depth map to obtain the target discrete depth map, and to downsample the color map to obtain the target color feature map.
[0130] The fused image feature map generation module 82 is used to perform downsampling and feature extraction on the target discrete depth map and the color map a preset number of times to obtain a first color feature map and a first target discrete depth feature map. After performing channel parallel operation and upsampling operation on the first color feature map and the first target discrete depth feature map, a fused image feature map is obtained.
[0131] The fusion feature output module 83 is used to input the fused image feature map and the target discrete depth map into the cross-attention feature fusion module and output the fused feature map;
[0132] The result output module 84 is used to perform channel parallel connection on the target color feature map and the fused feature map, and to perform upsampling to obtain a completed depth map.
[0133] Furthermore, such as Figure 9 As shown, based on the above-mentioned self-supervised depth completion method and system based on three-dimensional feature extraction and fusion, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 9 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.
[0134] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a self-supervised depth completion program 40 based on 3D feature extraction and fusion, which can be executed by the processor 10 to implement the self-supervised depth completion method based on 3D feature extraction and fusion in this application.
[0135] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the self-supervised depth completion method based on 3D feature extraction and fusion.
[0136] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.
[0137] In one embodiment, when the processor 10 executes the self-supervised depth completion program 40 based on 3D feature extraction and fusion in the memory 20, the following steps are performed:
[0138] The discrete depth map acquired by the lidar and the color map acquired by the camera are obtained. The discrete depth map is preprocessed to obtain the target discrete depth map, and the color map is downsampled to obtain the target color feature map.
[0139] After performing downsampling and feature extraction on the target discrete depth map and the color map a preset number of times, a first color feature map and a first target discrete depth feature map are obtained. After performing channel parallel operation and upsampling operation on the first color feature map and the first target discrete depth feature map, a fused image feature map is obtained.
[0140] The fused image feature map and the target discrete depth map are input into the cross-attention feature fusion module, and the fused feature map is output.
[0141] The target color feature map and the fused feature map are connected in parallel through channels and then upsampled to obtain a complete depth map.
[0142] Specifically, the preprocessing of the discrete depth map to obtain the target discrete depth map includes:
[0143] The discrete depth map and the color map are downsampled to obtain multi-scale feature output;
[0144] The multi-scale feature output is scaled up using the nearest neighbor interpolation method to obtain expanded multi-scale features.
[0145] The extended multi-scale features are input into a multilayer perceptron for feature forward propagation to obtain a confidence matrix. The confidence matrix is then processed according to a preset accuracy threshold to obtain a depth mask.
[0146] The discrete depth map is obtained by removing outliers based on the depth mask.
[0147] Specifically, the step of downsampling and feature extraction on the target discrete depth map and the color map a predetermined number of times to obtain a first color feature map and a first target discrete depth feature map includes:
[0148] After performing three downsampling and feature extractions on the target discrete depth map and the color map, a second color feature map and a second target discrete depth feature map are obtained.
[0149] After downsampling the second color feature map and the second target discrete depth feature map twice, feature extraction is performed to obtain the first color feature map and the first target discrete depth feature map.
[0150] Specifically, the step of performing three downsampling operations and feature extractions on the target discrete depth map and the color map to obtain the second color feature map and the second target discrete depth feature map includes:
[0151] After downsampling the target discrete depth map and the color map respectively, a first downsampled target discrete depth map and a first downsampled color map are obtained;
[0152] The first downsampled target discrete depth map and the first downsampled color map are input into the graph propagation-based feature extraction module for feature extraction to obtain the third target discrete depth feature map and the third color feature map.
[0153] After performing downsampling and feature extraction twice on the third target discrete depth feature map and the third color feature map, the second color feature map and the second target discrete depth feature map are obtained.
[0154] Specifically, the step of inputting the first downsampled target discrete depth map and the first downsampled color map into a graph propagation-based feature extraction module for feature extraction to obtain a third target discrete depth feature map and a third color feature map includes:
[0155] The first downsampled target discrete depth map, the first downsampled color map, and the target discrete depth map are input into the graph propagation-based feature extraction module. The feature extraction module uses the neighborhood K-point distance operation method to obtain the first depth feature pixel distance coordinate matrix and the first color image feature pixel distance coordinate matrix based on the first downsampled target discrete depth map, the first downsampled color map, and the target discrete depth map.
[0156] The first depth feature pixel distance coordinate matrix and the first color image feature pixel distance coordinate matrix are respectively input into the multilayer perceptron in the feature extraction module to obtain the first updated depth feature pixel distance coordinate matrix and the first updated color image feature pixel distance coordinate matrix.
[0157] The first updated depth feature pixel distance coordinate matrix and the first downsampled target discrete depth map are multiplied together to obtain the third target discrete depth feature map. The first updated color map feature pixel distance coordinate matrix and the first downsampled color map are multiplied together to obtain the third color feature map.
[0158] Specifically, the step of inputting the fused image feature map and the target discrete depth map into the cross-attention feature fusion module and outputting the fused feature map includes:
[0159] The fused image feature map and the target discrete depth map are input into the three-dimensional information extraction module in the cross-attention feature fusion module to obtain the first three-dimensional feature information;
[0160] Point cloud generation operations are performed on the three-dimensional feature information and the fused image feature map to obtain a two-dimensional feature image and a second three-dimensional feature image;
[0161] The two-dimensional feature image and the second three-dimensional feature image are input into the three-dimensional to two-dimensional feature cross-attention module in the cross-attention feature fusion module to obtain pre-fused features;
[0162] The pre-fused features are normalized to obtain the fused features.
[0163] Specifically, inputting the fused image feature map and the target discrete depth map into the 3D information extraction module of the cross-attention feature fusion module to obtain the first 3D feature information includes:
[0164] The fused image feature map and the target discrete depth map are input into the three-dimensional information extraction module in the cross-attention feature fusion module to extract the pixel values of the discrete depth effective point coordinates in the fused image feature map;
[0165] Based on the pixel values of the effective point coordinates of the discrete depth and the target discrete depth map, the neighborhood K-point distance calculation method is used to obtain the feature pixel distance and the point cloud geometric distance.
[0166] The feature pixel distance and the point cloud geometric distance are input into the multilayer perceptron in the 3D information extraction module of the cross-attention feature fusion module to obtain the attention matrix;
[0167] Based on the attention matrix and the target discrete depth map, a channel connection operation is performed to obtain the first three-dimensional feature information.
[0168] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.
[0169] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.
[0170] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A self-supervised depth completion method based on 3D feature extraction and fusion, characterized in that, The self-supervised depth completion method based on 3D feature extraction and fusion includes: The discrete depth map acquired by the lidar and the color map acquired by the camera are obtained. The discrete depth map is preprocessed to obtain the target discrete depth map, and the color map is downsampled to obtain the target color feature map. After performing downsampling and feature extraction on the target discrete depth map and the color map a preset number of times, a first color feature map and a first target discrete depth feature map are obtained. After performing channel parallel operation and upsampling operation on the first color feature map and the first target discrete depth feature map, a fused image feature map is obtained. The fused image feature map and the target discrete depth map are input into the cross-attention feature fusion module, and the fused feature map is output. The step of inputting the fused image feature map and the target discrete depth map into the cross-attention feature fusion module and outputting the fused feature map specifically includes: The fused image feature map and the target discrete depth map are input into the three-dimensional information extraction module in the cross-attention feature fusion module to obtain the first three-dimensional feature information; Point cloud generation operations are performed on the three-dimensional feature information and the fused image feature map to obtain a two-dimensional feature image and a second three-dimensional feature image; The two-dimensional feature image and the second three-dimensional feature image are input into the three-dimensional to two-dimensional feature cross-attention module in the cross-attention feature fusion module to obtain pre-fused features; The pre-fused features are normalized to obtain the fused features; The step of inputting the fused image feature map and the target discrete depth map into the three-dimensional information extraction module of the cross-attention feature fusion module to obtain the first three-dimensional feature information specifically includes: The fused image feature map and the target discrete depth map are input into the three-dimensional information extraction module in the cross-attention feature fusion module to extract the pixel values of the discrete depth effective point coordinates in the fused image feature map; Based on the pixel values of the effective point coordinates of the discrete depth and the target discrete depth map, the neighborhood K-point distance calculation method is used to obtain the feature pixel distance and the point cloud geometric distance. The feature pixel distance and the point cloud geometric distance are input into the multilayer perceptron in the 3D information extraction module of the cross-attention feature fusion module to obtain the attention matrix; Based on the attention matrix and the target discrete depth map, a channel connection operation is performed to obtain the first three-dimensional feature information; The target color feature map and the fused feature map are connected in parallel through channels and then upsampled to obtain a complete depth map.
2. The self-supervised depth completion method based on 3D feature extraction and fusion according to claim 1, characterized in that, The preprocessing of the discrete depth map to obtain the target discrete depth map specifically includes: The discrete depth map and the color map are downsampled to obtain multi-scale feature output; The multi-scale feature output is scaled up using the nearest neighbor interpolation method to obtain expanded multi-scale features. The extended multi-scale features are input into a multilayer perceptron for feature forward propagation to obtain a confidence matrix. The confidence matrix is then processed according to a preset accuracy threshold to obtain a depth mask. The discrete depth map is obtained by removing outliers based on the depth mask.
3. The self-supervised depth completion method based on 3D feature extraction and fusion according to claim 1, characterized in that, After performing downsampling and feature extraction on the target discrete depth map and the color map a preset number of times, a first color feature map and a first target discrete depth feature map are obtained, specifically including: After performing three downsampling and feature extractions on the target discrete depth map and the color map, a second color feature map and a second target discrete depth feature map are obtained. After downsampling the second color feature map and the second target discrete depth feature map twice, feature extraction is performed to obtain the first color feature map and the first target discrete depth feature map.
4. The self-supervised depth completion method based on 3D feature extraction and fusion according to claim 3, characterized in that, The step of performing three downsampling and feature extraction on the target discrete depth map and the color map to obtain the second color feature map and the second target discrete depth feature map specifically includes: After downsampling the target discrete depth map and the color map respectively, a first downsampled target discrete depth map and a first downsampled color map are obtained; The first downsampled target discrete depth map and the first downsampled color map are input into the graph propagation-based feature extraction module for feature extraction to obtain the third target discrete depth feature map and the third color feature map. After performing downsampling and feature extraction twice on the third target discrete depth feature map and the third color feature map, the second color feature map and the second target discrete depth feature map are obtained.
5. The self-supervised depth completion method based on 3D feature extraction and fusion according to claim 4, characterized in that, The step of inputting the first downsampled target discrete depth map and the first downsampled color map into the graph propagation-based feature extraction module for feature extraction to obtain the third target discrete depth feature map and the third color feature map specifically includes: The first downsampled target discrete depth map, the first downsampled color map, and the target discrete depth map are input into the graph propagation-based feature extraction module. The feature extraction module uses the neighborhood K-point distance operation method to obtain the first depth feature pixel distance coordinate matrix and the first color image feature pixel distance coordinate matrix based on the first downsampled target discrete depth map, the first downsampled color map, and the target discrete depth map. The first depth feature pixel distance coordinate matrix and the first color image feature pixel distance coordinate matrix are respectively input into the multilayer perceptron in the feature extraction module to obtain the first updated depth feature pixel distance coordinate matrix and the first updated color image feature pixel distance coordinate matrix. The first updated depth feature pixel distance coordinate matrix and the first downsampled target discrete depth map are multiplied together to obtain the third target discrete depth feature map. The first updated color map feature pixel distance coordinate matrix and the first downsampled color map are multiplied together to obtain the third color feature map.
6. A self-supervised depth completion system based on 3D feature extraction and fusion, characterized in that, The self-supervised depth completion system based on 3D feature extraction and fusion is used to implement the self-supervised depth completion method based on 3D feature extraction and fusion as described in any one of claims 1-5. The self-supervised depth completion system based on 3D feature extraction and fusion includes: The preprocessing module is used to acquire discrete depth maps collected by lidar and color maps collected by camera, and to preprocess the discrete depth maps to obtain target discrete depth maps, and to downsample the color maps to obtain target color feature maps; The fused image feature map generation module is used to perform downsampling and feature extraction on the target discrete depth map and the color map a preset number of times to obtain a first color feature map and a first target discrete depth feature map. After performing channel parallel operation and upsampling operation on the first color feature map and the first target discrete depth feature map, a fused image feature map is obtained. The fusion feature output module is used to input the fused image feature map and the target discrete depth map into the cross-attention feature fusion module and output the fused feature map; The result output module is used to perform channel parallel connection on the target color feature map and the fused feature map, and to perform upsampling to obtain a completed depth map.
7. A terminal, characterized in that, The terminal includes: a memory, a processor, and a self-supervised depth completion program based on three-dimensional feature extraction and fusion stored in the memory and executable on the processor. When the self-supervised depth completion program based on three-dimensional feature extraction and fusion is executed by the processor, it implements the steps of the self-supervised depth completion method based on three-dimensional feature extraction and fusion as described in any one of claims 1-5.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a self-supervised depth completion program based on 3D feature extraction and fusion, which, when executed by a processor, implements the steps of the self-supervised depth completion method based on 3D feature extraction and fusion as described in any one of claims 1-5.