Camera-millimeter wave radar fusion depth completion method based on structural information guidance

Abstract

The invention discloses a camera-millimeter wave radar fusion depth completion method based on structural information guidance, and the method comprises the steps: firstly extracting image features and depth features from an RGB image and a millimeter wave depth map, and then guiding the depth features to carry out the enhancement through the image features, performing feature extraction and enhancement on the enhanced depth feature and the image feature under a larger scale, repeating the feature extraction and enhancement steps until the feature scale reaches a preset scale, and fusing the image feature and the depth feature under the last scale; and performing scale-by-scale up-sampling on the fusion features to obtain a dense depth map with the same size as the millimeter wave depth map. According to the method, operation is carried out on multi-scale features, so that the feature extraction process can be focused on channels and spatial positions strongly related to structure information, and finally, a dense depth map with good structure details and an accurate absolute depth value is recovered from a sparse millimeter wave radar depth map.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a camera-millimeter-wave radar fusion depth completion method based on structural information guidance. Background Art

[0002] With the development of autonomous driving technology, the perception of depth has become increasingly important for understanding the three-dimensional environment. Common depth sensors on the market, such as lidar and radar, can obtain depth directly from the environment. Among them, lidar can generate higher accuracy and denser depth information than millimeter-wave radar, but it is more expensive, larger in size, and more sensitive to environmental factors such as rain, fog, and snow. In contrast, millimeter-wave radar is easy to deploy and has strong robustness to adverse light and weather conditions, and has received increasing attention in recent years. However, due to the limitations of the millimeter-wave radar imaging mechanism, the depth data obtained by millimeter-wave radar is very sparse and lacks semantic information.

[0003] With the widespread application of deep learning technology, in order to make up for the inherent sparsity and semantic scarcity of millimeter-wave radar depth data, many methods have introduced deep neural networks to complete the sparse depth information and obtain dense depth. Early methods tried to use the deep neural network with UNet structure to directly complete the depth of sparse depth maps. However, due to the sparsity of depth and the lack of any semantic information, such methods have very limited performance in depth accuracy and structural details. Considering the limitations of using single-modal depth data, many studies in recent years have completed depth by fusing visible light images and millimeter-wave radar depth. This type of method often uses a deep neural network including two feature encoders and a feature decoder, one of which is used to extract image features and the other is used to extract millimeter-wave radar depth features. The features of the two modalities are then fused, and a dense depth map is restored through a decoder. This method complements the scene prior in the camera image with the depth clues in the millimeter-wave radar depth map, so that the absolute depth can be more accurate and with better structural details. Many experiments have verified the rationality and effectiveness of this technical route, making it the mainstream solution for millimeter-wave radar depth completion.

[0004] However, the fusion schemes adopted by existing camera-millimeter-wave radar fusion depth completion methods mostly simply splice the features of the image modality and the depth modality without sufficient interaction of the features of the two modalities, resulting in unsatisfactory accuracy and structural details of the generated dense depth map. Summary of the invention

[0005] In view of the limitations of feature fusion in existing camera-millimeter-wave radar fusion depth completion methods, the present invention starts from the perspective of how to build a better feature interaction and fusion solution, and designs a camera-millimeter-wave radar fusion depth completion method based on structural information guidance according to task characteristics. By extracting structural information from image features to guide the enhancement of millimeter-wave radar depth features, accurate depth completion with richer structural details can be achieved.

[0006] The present invention is implemented by the following technical solutions:

[0007] A camera-millimeter-wave radar fusion depth completion method based on structural information guidance is disclosed. The method comprises the following steps: firstly, image features and depth features are extracted from an RGB image and a millimeter-wave depth map respectively; then, the image features are used to guide the depth features for enhancement; then, the enhanced depth features and image features are extracted and enhanced at a larger scale; the feature extraction and enhancement steps are repeated until the feature scale reaches a preset scale; the image features and depth features at the last scale are fused; and the fused features are upsampled scale by scale to obtain a dense depth map with the same size as the millimeter-wave depth map.

[0008] In the above technical solution, further, the camera-millimeter wave radar fusion depth completion method based on structural information guidance has the following specific steps:

[0009] S1: Given a pair of RGB images and sparse mmWave depth maps The initial image features are obtained through a set of downsampling and dimension upscaling operations. and the initial depth feature H and W represent the height and width of the image respectively.

[0010] S2: respectively from f I and f d Extract image features and deep features Where i represents the scale of the feature, C g Represents the dimension of image features (i.e., guided features), C d Represents the dimension of deep features;

[0011] S3: Exploitation The structural information in Enhance and obtain enhanced deep features

[0012] S31: and Concatenate on the feature dimension to get fusion features

[0013] S32: Extraction using channel attention mechanism The structural information of the channel dimension is obtained to obtain a channel attention map

[0014] S33: Channel attention map With deep features Multiply pixel by pixel to obtain the deep features enhanced by channel level features

[0015] S34: Extraction using spatial attention mechanism The structural information of the spatial dimension in the middle is used to obtain a spatial attention map

[0016] S35: Spatial Attention Map And the deep features enhanced by channel-level features Multiply pixel by pixel to obtain the depth features that are further enhanced by spatial features.

[0017] S36: Image features and Concatenate on the feature dimension, and then use a layer of 3×3 convolution to match the feature dimension to Consistent, and then get the enhanced depth features

[0018] S4: and Set to new f I and f d , repeat the operations of S2-S3 at the i+1 scale until i reaches the preset scale;

[0019] S5: The image features obtained at the last scale and deep features Fusion is performed to obtain the fusion features at the last scale Combine the image features at the last scale and the enhanced depth features at the last scale to obtain the fusion features at the last scale. Perform upsampling and convolution operations to obtain the fusion features at the last-1 scale use replace Perform upsampling and convolution operations at the last-1th scale; repeat upsampling and convolution operations at the next scale until the image with the same size as d is restored. r Dense depth maps of the same size

[0020] Furthermore, the downsampling and dimension upscaling operations in step S1 are specifically as follows:

[0021] For the RGB image I, it first passes through a convolution Conv2d with a convolution kernel of 7×7 and a stride of 2. I The operation is used to increase the dimension and perform preliminary downsampling, so that the dimension is increased from 3 to 32 and the size is reduced to I. Then an adaptive maximum pooling operation AMP is used to further reduce the size to I Finally, the initial image feature f is obtained I =AMP(Conv2d I (I)). For the sparse mmWave depth map d r , using the same operation as I (first through Conv2d d The operation performs dimension upscaling and preliminary downsampling, and then further reduces the size to d through an adaptive maximum pooling operation AMP. r of ), we can get the initial depth feature f d =AMP(Conv2d d (d r )). Among them, Conv2d d The convolution kernel and step size settings are the same as Conv2d I Exactly the same, the difference is that the dimensionality increase process is from 3 to 16.

[0022] Furthermore, the specific method for extracting the image features and depth features in step S2 is as follows:

[0023] 1) For input features The first convolution Conv2d1 is used to extract features of x, and activated by the activation function σ to obtain the first layer feature a1=σ(Conv2d1(x)). The parameters of Conv2d1 can be adjusted as needed.

[0024] 2) The first layer feature a1 is sent to the second convolution layer Conv2d2, and the feature extraction is performed in the same way as in 1) to obtain the second layer feature a2 = σ(Conv2d2(x)). The parameters of Conv2d2 can also be adjusted as needed.

[0025] 3) The input feature x is added to the second layer feature a2 as a skip connection, and then the output feature y = σ(a2+x) of the ResNet module is obtained through the activation function σ.

[0026] The above input feature x is an image feature or depth feature at a certain scale. For example, when x is f I or dBased on the above steps 1)-3) we can get f I and f d Extract image features and deep features

[0027] Furthermore, the step S32 is specifically as follows: using an average pooling operation to calculate the fusion feature in the spatial dimension The average value of the channel number remains unchanged during the average pooling operation, and the shape becomes (C g +C d )×1×1; Use a multi-layer perceptron to extract features from the result of the average pooling operation and reduce the dimension to obtain the first intermediate feature

[0028] At the same time, the fusion features After a maximum pooling operation, the shape becomes (C g +C d )×1×1, and then use a multi-layer perceptron to extract features from the maximum pooling operation result and reduce the dimension to obtain the second intermediate feature

[0029] Will and Add, then pass a Sigmoid activation function, and finally get the channel attention map

[0030] Furthermore, the specific method of step S34 is as follows:

[0031] 1) Deep features enhanced by channel-level features The average operation is performed on the channel dimension while keeping the feature size unchanged, and the shape becomes 1×H×W.

[0032] 2) is in parallel with 1), Find the maximum value of the channel dimension while keeping the feature size unchanged, and the shape becomes 1×H×W.

[0033] 3) Concatenate the results of steps 1) and 2) in the feature dimension, and the shape becomes 2×H×W.

[0034] 4) Reduce the dimension of the result in 3) through a 3×3 convolution, and then activate it through a Sigmoid function to finally obtain the spatial attention map

[0035] Furthermore, step S5 is implemented based on a multi-scale jump connection decoder. The multi-scale jump connection decoder is mainly composed of a combination of 5 upsampling + convolution, which is used to gradually restore the fusion features at the last scale into a dense depth map; in this process, the multi-scale fusion features obtained by the S1-S4 feature extraction steps will be introduced into the decoder in the form of jump connections to maintain the receptive field at multiple scales. The specific method is:

[0036] 1) For the fusion features at the last scale Image features at the last scale And the enhanced deep features at the last scale First, The dimension is raised to Consistent, then added, to get the skip connection feature Among them, project represents the dimension enhancement operation; then the fusion feature After upsampling and convolution operation deconv, the deconv result and Concatenate on the feature dimension to get the features after upsampling the features at the last scale

[0037] 2) Use the last-1 scale replace Combined with the image features of the corresponding scale (last-1 scale) And the depth feature of its corresponding scale (last-1 scale) Repeat the operation in step 1) to obtain

[0038] 3) Repeat the operation of step 2) at the next scale (the number of cycles is equal to the preset scale in S4) until the same scale as d is restored. r Dense depth map d of the same size out .

[0039] Furthermore, the method is implemented based on multi-scale structural information to guide the deep neural network, and the training process of the multi-scale structural information to guide the deep neural network is as follows:

[0040] 1) Compared with millimeter wave radar, the point cloud of LiDAR is denser and the measurement value is more accurate due to the different imaging mechanism. Therefore, the LiDAR point cloud is projected onto the image plane to form the LiDAR depth map, which is used as the true value of the depth map d gt .

[0041] 2) Due to d gt is relatively sparse and lacks semantic information during supervision. Therefore, the depth map truth d of the multi-frame lidar point cloud isgt Fusion is performed and then projected to obtain a relatively dense lidar depth map d acc Furthermore, acc Perform linear interpolation to obtain a fully dense lidar depth map d int , and remove the depth values ​​of unreasonable areas (such as the sky).

[0042] 3) Use L2 loss to measure d out With d acc and d int The difference between acc is sparse, so only in d acc The loss is calculated at non-zero positions. The loss calculation process can be expressed as follows:

[0043]

[0044] Among them, Ω acc Indicates d acc The reasonable area in Ω int Indicates d int The reasonable area in d acc (u,v) represents the pixel at (u,v) in the relatively dense lidar depth map, d out (u,v) represents the pixel at (u,v) in the dense depth map output by the deep neural network, d int (u,v) is the pixel at (u,v) in the fully dense lidar depth map.

[0045] 4) Use the true value and loss function set in 1)-3) to train the multi-scale structural information to guide the deep neural network until the set number of iterations; during the training process, stochastic gradient descent is used to update the learnable parameters of the neural network.

[0046] The beneficial effects of the present invention are:

[0047] In view of the limitations of feature fusion in existing camera-millimeter-wave radar fusion depth completion methods, the present invention guides the sparse and semantically lacking millimeter-wave radar depth for depth completion by capturing the structural information in the camera RGB image. Given a pair of RGB images and a sparse millimeter-wave depth map, the method first extracts image features and depth features respectively, and then uses the structural information in the image features to guide the depth features to be enhanced to enhance the structural expression of the depth features, and then repeats the above feature extraction and enhancement steps for the enhanced depth features and image features, and then fuses the image features and depth features at the last scale, and finally upsamples the fused features to restore the dense depth map. The method of the present invention realizes the migration of image feature structural information to sparse depth features, and by operating on multi-scale features, the feature extraction process can focus on channels and spatial positions that are strongly related to structural information, and finally realizes the recovery of a dense depth map with good structural details and accurate absolute depth values ​​from a sparse millimeter-wave radar depth map. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of the structure of a multi-scale structural information guided deep neural network according to an embodiment of the present invention.

[0049] Figure 2 It is a schematic diagram of the structure of the structure information guiding module SGB according to an embodiment of the present invention.

[0050] Figure 3 A schematic diagram of the training process of a deep neural network guided by multi-scale structural information according to an embodiment of the present invention.

[0051] Figure 4 The following are example results of an embodiment of the present invention: (a) input RGB image; (b) input sparse millimeter-wave radar depth map; (c) dense depth map after completion; (d) single-frame lidar depth map; (e) denser lidar depth map after multi-frame fusion; (f) fully dense lidar depth map after linear interpolation. DETAILED DESCRIPTION

[0052] The technical solution of the present invention is further described below in conjunction with specific embodiments and drawings.

[0053] Example

[0054] The present invention provides a camera-millimeter-wave radar fusion depth completion method based on structural information guidance. A multi-scale structural information guided deep neural network is designed. The structural semantic information in the image features is migrated to the depth features through the structural information guidance module at multiple scales, thereby enhancing the depth features, and finally restoring a dense depth map with good structural details and accurate absolute depth.

[0055] This method is based on the multi-scale structural information guided deep neural network. Specifically, given a pair of RGB images and a sparse millimeter-wave depth map, firstly, the image features and depth features are extracted through two ResNet modules respectively, and then the image features and depth features are input into the structural information guided module to enhance the structural expression of the depth features. Then, the enhanced depth features and image features are sent to the ResNet module and the structural information guided module at the next scale to repeat the above steps, and then the image features and depth features at the last scale are fused. Finally, a UNet decoder based on jump connection is used to upsample the fused features scale by scale to obtain a dense depth map of the same size as the millimeter-wave depth map.

[0056] Use Figure 1 The multi-scale structural information shown guides the deep neural network to complete the depth. The specific method is:

[0057] Image branch: The shape of the input image I is 3×H×W, which becomes After the first ResNet module, the shape becomes After the second ResNet module, the shape becomes After the third ResNet module, the shape becomes Next, we will continue to downsample without increasing the dimension. After the fourth ResNet module, the shape becomes After the 5th ResNet module, the shape becomes This feature is the image feature of the last scale and will be used to fuse with the depth feature of the last scale.

[0058] Depth branch: input depth map d r The feature of is 1×H×W, which becomes after dimension upscaling and downsampling in S1. After the first ResNet module, the shape becomes After the first structural information guidance module SGB and the second ResNet module, the shape becomes After the second structural information guidance module and the third ResNet module, the shape becomes Next, we will continue to downsample without increasing the dimension. After the third structural information guidance module and the fourth ResNet module, the shape becomes After the 4th structural information guidance module and the 5th ResNet module, the shape becomes This feature is the depth feature of the last scale and is no longer structure-guided. It will be used to fuse with the image feature of the last scale.

[0059] The image features and depth features obtained at the last scale are fused to obtain the fused features at the last scale; the image features at the last scale extracted by the ResNet module and the enhanced depth features at the last scale are combined, and the fused features at the last scale are upsampled and convolved to obtain the fused features at the last-1 scale; the fused features at the last-1 scale are used to replace the fused features at the last-1 scale, and upsampled and convolved at the last-1 scale; the upsampling and convolution operations are repeated at the next scale until the fused features that are consistent with d are restored. r Dense depth maps of the same size.

[0060] A camera-millimeter-wave radar fusion depth completion method based on structural information guidance in this embodiment has the following specific steps:

[0061] 1) Given a pair of RGB images ( Figure 4 (a)) and sparse mmWave depth map ( Figure 4 (b)) The initial image features are obtained through a set of downsampling and dimension upscaling operations. and the initial depth feature For the RGB image I, it first passes through a convolution Conv2d with a convolution kernel of 7×7 and a stride of 2. I The operation is used to increase the dimension and perform preliminary downsampling, so that the dimension is increased from 3 to 32 and the size is reduced to I. Then, an adaptive maximum pooling operation AMP is used to further reduce the feature size to I. Finally, the initial image features input to the ResNet module are obtained For the input depth map d r , using the same operation as I, we can get the deep features input to the ResNet module Among them, Conv2d d The convolution kernel and step size settings are the same as Conv2d I Exactly the same, but the number of output channels is changed from 32 to 16.

[0062] 2) Through the ResNet module, respectively from f I and f d Extract image features and deep features Where i represents the scale of the feature, C g Represents the dimension of image features (i.e., guided features), C d Represents the dimension of deep features, Figure 1 The RB in the figure is the position of the ResNet module in the entire deep neural network.

[0063] 3) and Input structure information guide module SGB, through The structural information in Enhance and obtain enhanced deep features The position of the structure information guidance module SGB in the entire multi-scale structure information guidance deep neural network is as follows: Figure 1 As shown in SGB, its specific structure is as follows Figure 2 First, and Concatenate on the feature dimension to get fusion features Then use the channel attention mechanism to extract The structural information of the channel dimension is obtained to obtain a channel attention map The channel attention map With deep features Multiply pixel by pixel to obtain the deep features enhanced by channel level features Then use the spatial attention mechanism to extract The structural information of the spatial dimension in the middle is used to obtain a spatial attention map Finally, the spatial attention map And the deep features enhanced by channel-level features Multiply pixel by pixel to obtain the depth features that are further enhanced by spatial features.

[0064] Among them, the channel attention mechanism is used to extract The structural information of the channel dimension is obtained to obtain a channel attention map The specific method is:

[0065] Use the average pooling operation to calculate the fusion features in the spatial dimension The average value of the channel number remains unchanged during the average pooling operation, and the shape becomes (C g +C d )×1×1; Use a multi-layer perceptron to extract features from the result of the average pooling operation and reduce the dimension to obtain the first intermediate feature

[0066] At the same time, the fusion features After a maximum pooling operation, the shape becomes (C g +C d )×1×1, and then use a multi-layer perceptron to extract features from the maximum pooling operation result and reduce the dimension to obtain the second intermediate feature

[0067] Will and Add, then pass a Sigmoid activation function, and finally get the channel attention map

[0068] 4) and Set to new f I and f d , repeat the operations in 2) and 3), input it into the ResNet module at the i+1 scale to extract features, and then input the result into the SGB at the i+1 scale to enhance the deep features until i reaches the preset scale.

[0069] 5) The image features obtained at the last scale and deep features Fusion is performed to obtain the fusion features at the last scale Combine the image features at the last scale and the enhanced depth features at the last scale to obtain the fusion features at the last scale. Perform upsampling and convolution operations to obtain the fusion features at the last-1 scale use replace Perform upsampling and convolution operations at the last-1th scale; repeat upsampling and convolution operations at the next scale until the image with the same size as d is restored. r Dense depth maps of the same size Specifically, this step includes the following steps: 51) For the fusion feature at the last scale Image features at the last scale And the enhanced deep features at the last scale First, The dimension is raised to Consistent, then added, to get the skip connection feature Among them, project represents the dimension enhancement operation; then the fusion feature After upsampling and convolution operation deconv, the deconv result and Concatenate on the feature dimension to get the features after upsampling the features at the last scale 52) Use the last-1 scale replace Combined with the image features of the corresponding scale And the depth features of its corresponding scale Repeat the operation in step 51) to obtain 53) Repeat the operation of step 52) at the next scale until the same value as d is restored. rDense depth map d of the same size out ,like Figure 4 (c) as shown.

[0070] The multi-scale structural information guides the training method of the deep neural network as follows: a relatively dense lidar depth map is obtained through the lidar point cloud and dense lidar depth map It is used to supervise the training of the entire deep neural network. Compared with millimeter wave radar, the point cloud of lidar is denser and the measurement value is more accurate due to the different imaging mechanism. Therefore, the lidar point cloud is projected onto the image plane to form a lidar depth map, which is used as the true value of the depth map d gt ,like Figure 4 (d). gt It is relatively sparse and lacks semantic information during supervision, so the depth map truth d of the multi-frame lidar point cloud is gt Fusion is performed and then projected to obtain a relatively dense lidar depth map d acc ,like Figure 4 (e) is shown. Furthermore, acc Perform linear interpolation to obtain a fully dense lidar depth map d int , and delete the depth values ​​of some unreasonable areas (such as the sky), such as Figure 4 (f) is shown. Using L2 loss to measure d out With d acc and d int The difference between acc is sparse, so only in d acc Calculate loss at non-zero positions, depth loss The calculation process can be expressed as follows:

[0071]

[0072] Among them, Ω acc Indicates d acc The reasonable area in Ω int Indicates d int The reasonable area in d acc (u,v) represents the pixel at (u,v) in the relatively dense lidar depth map, d out (u,v) represents the pixel at (u,v) in the dense depth map output by the deep neural network, d int (u,v) is the pixel at (u,v) in the fully dense LiDAR depth map. Finally, the multi-scale structural information is trained to guide the deep neural network according to the set true value and loss function until the set number of iterations; during the training process, stochastic gradient descent is used to update the learnable parameters of the neural network.

[0073] The present invention realizes the migration of image feature structure information to sparse depth features, and by operating on multi-scale features, the feature extraction process can focus on channels and spatial positions that are strongly correlated with structural information, ultimately restoring a dense depth map with good structural details and accurate absolute depth values ​​from a sparse millimeter-wave radar depth map. Figure 4 (a) is the input RGB image I, Figure 4 (b) is the input sparse millimeter wave depth map d r , Figure 4 (c) is the dense depth map completed by the present invention. Figure 4 (c) The target objects in the dense depth map have clear depth boundaries and structural contours, and the depths between objects do not stick together, which demonstrates the effectiveness of the structural information migration in the present invention.

Claims

1. A camera-millimeter-wave radar fusion depth completion method based on structural information guidance, characterized in that: First, image features and depth features are extracted from the RGB image and the millimeter-wave depth map respectively, and then the image features are used to guide the depth features for enhancement. Next, the enhanced depth features and image features are extracted and enhanced at a larger scale. The feature extraction and enhancement steps are repeated until the feature scale reaches a preset scale. The image features and depth features at the last scale are fused, and the fused features are upsampled scale by scale to obtain a dense depth map with the same size as the millimeter-wave depth map.

2. The camera-millimeter-wave radar fusion depth completion method based on structural information guidance according to claim 1 is characterized in that: The specific steps are as follows: S1: Given a pair of RGB images and sparse mmWave depth maps The initial image features are obtained through a set of downsampling and dimension upscaling operations. and the initial depth feature H and W represent the height and width of the image, respectively; S2: respectively from f I and f d Extract image features and deep features Where i represents the scale of the feature, C g Represents the dimension of image features, C d Represents the dimension of deep features; S3: Exploitation The structural information in Enhance and obtain enhanced deep features S31: and Splice on the feature dimension to get fusion features S32: Extraction using channel attention mechanism The structural information of the channel dimension is obtained to obtain a channel attention map S33: Channel attention map With deep features Multiply pixel by pixel to obtain the deep features enhanced by channel level features S34: Extraction using spatial attention mechanism The structural information of the spatial dimension in the middle is used to obtain a spatial attention map S35: Spatial Attention Map And the deep features enhanced by channel-level features Multiply pixel by pixel to obtain the depth features that are further enhanced by spatial features. S36: Image features and Concatenate on the feature dimension, and then use a layer of 3×3 convolution to match the feature dimension to Consistent, and then get the enhanced depth features S4: and Set to new f I and f d , repeat the operations of S2-S3 at the i+1 scale until i reaches the preset scale; S5: The image features obtained at the last scale and deep features Fusion is performed to obtain the fusion features at the last scale Combine the image features at the last scale and the enhanced depth features at the last scale to obtain the fusion features at the last scale. Perform upsampling and convolution operations to obtain the fusion features at the last-1 scale use replace Perform upsampling and convolution operations at the last-1th scale; repeat upsampling and convolution operations at the next scale until the image with the same size as d is restored. r Dense depth maps of the same size 3. The camera-millimeter-wave radar fusion depth completion method based on structural information guidance according to claim 1 is characterized in that: The downsampling and dimension upscaling operations in step S1 are specifically as follows: 1) For the RGB image I, first pass a convolution Conv2d with a convolution kernel of 7×7 and a stride of 2 I The operation is used to increase the dimension and perform preliminary downsampling, so that the dimension is increased from 3 to 32 and the size is reduced to I. Then an adaptive maximum pooling operation AMP is used to further reduce the size to I Finally, the initial image feature f is obtained I =AMP(Conv2d I (I)); 2) For the sparse mmWave depth map d r , using similar operations as in 1) to obtain the initial depth feature f d =AMP(Conv2d d (d r )); Among them, Conv2d d The convolution kernel and step size settings are the same as Conv2d I Exactly the same; the difference from 1) is that the dimension increase process is to increase the dimension from 3 to 16.

4. The camera-millimeter-wave radar fusion depth completion method based on structural information guidance according to claim 1 is characterized in that: The extraction of image features and depth features in step S2 is implemented by a ResNet module, and the specific operation of the ResNet module is as follows: 1) For input features Use the first convolution Conv2d1 to extract features from x, and activate it through the activation function σ to obtain the first layer feature a1 = σ (Conv2d1 (x)); where the parameters of Conv2d1 are adjusted as needed; 2) Send the first layer feature a1 to the second convolutional layer Conv2d2, and extract the features in the same way as in 1) to obtain the second layer feature a2 = σ(Conv2d2(x)), where the parameters of Conv2d2 are adjusted as needed; 3) The input feature x is added to the second layer feature a2 using a skip connection, and then the output feature y = σ(a2 + x) of the ResNet module is obtained through the activation function σ.

5. The camera-millimeter-wave radar fusion depth completion method based on structural information guidance according to claim 1, characterized in that: The step S32 is specifically as follows: Use the average pooling operation to calculate the fusion features in the spatial dimension The average value of the channel number remains unchanged during the average pooling operation, and the shape becomes (C g +C d )×1×1; extract features from the result of the average pooling operation and perform dimensionality reduction to obtain the first intermediate feature At the same time, the fusion features After a maximum pooling operation, the shape becomes (C g +C d )×1×1, and then perform feature extraction and dimensionality reduction on the result of the maximum pooling operation to obtain the second intermediate feature Will and Add, then pass a Sigmoid activation function, and finally get the channel attention map 6. The camera-millimeter-wave radar fusion depth completion method based on structural information guidance according to claim 1 is characterized in that: The specific method of step S34 is as follows: 1) Deep features enhanced by channel-level features Perform an average operation on the channel dimension while keeping the feature size unchanged, and the shape becomes 1×H×W; 2) Yes Find the maximum value of the channel dimension while keeping the feature size unchanged, and the shape becomes 1×H×W; 3) Concatenate the results of steps 1) and 2) in the feature dimension, and the shape becomes 2×H×W; 4) Reduce the dimension of the result in 3) through a 3×3 convolution, then activate it through a Sigmoid function, and finally obtain the spatial attention map 7. The camera-millimeter-wave radar fusion depth completion method based on structural information guidance according to claim 1 is characterized in that: The step S5 is implemented based on a multi-scale skip connection decoder, and the specific steps are as follows: 1) For the fusion features at the last scale Image features at the last scale And the enhanced deep features at the last scale First, The dimension is raised to Consistent, then added, to get the skip connection feature Among them, project represents the dimension enhancement operation; then the fusion feature After upsampling and convolution operation deconv, the deconv result and Concatenate on the feature dimension to get the features after upsampling the features at the last scale 2) Use the last-1 scale replace Combined with the image features of the corresponding scale And the depth features of its corresponding scale Repeat the operation in step 1) to obtain 3) Repeat step 2) at the next scale until the same value as d is restored. r Dense depth map d of the same size out .

8. The camera-millimeter-wave radar fusion depth completion method based on structural information guidance according to claim 1, characterized in that: The method is implemented based on multi-scale structural information to guide the deep neural network. The training process of the multi-scale structural information to guide the deep neural network is as follows: 1) Project the LiDAR point cloud onto the image plane to form a LiDAR depth map, which is used as the true value of the depth map d gt ; 2) The depth map truth value d of the multi-frame lidar point cloud gt Fusion is performed and then projected to obtain a relatively dense lidar depth map d acc ; for d acc Perform linear interpolation to obtain a fully dense lidar depth map d int , and delete the depth values ​​of unreasonable areas; 3) Use L2 loss to measure d out With d acc and d int The difference between acc is sparse, so only in d acc Calculate loss at non-zero positions, depth loss The calculation process is expressed as follows: Among them, Ω acc Indicates d acc The reasonable area in Ω int Indicates d int The reasonable area in d acc (u,v) represents the pixel at (u,v) in the relatively dense lidar depth map, d out (u,v) represents the pixel at (u,v) in the dense depth map output by the deep neural network, d int (u,v) is the pixel at (u,v) in the fully dense lidar depth map; 4) Use the true value and loss function set in 1)-3) to train the multi-scale structural information to guide the deep neural network until the set number of iterations; during the training process, stochastic gradient descent is used to update the learnable parameters of the neural network.

Images