A camera-millimeter-wave radar fusion depth completion method based on structural information guidance
Patent Information
- Application Number
- CN202411758807.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-03
- Publication Date
- 2025-09-16
- Estimated Expiration
- 2044-12-03
Smart Images
Figure CN119649048B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image processing technology, and in particular relates to a camera-millimeter-wave radar fusion depth completion method based on structural information guidance. Background Art
[0002] With the development of autonomous driving technology, depth perception is becoming increasingly important for understanding the three-dimensional environment. Common depth sensors on the market, such as lidar and radar, can directly obtain depth from the environment. Among them, lidar can generate more accurate and denser depth information than millimeter-wave radar, but it is more expensive, larger in size, and more sensitive to environmental factors such as rain, fog, and snow. In contrast, millimeter-wave radar is easy to deploy and has strong robustness to adverse lighting and weather conditions, and has received increasing attention in recent years. However, due to the limitations of millimeter-wave radar's imaging mechanism, the depth data obtained by millimeter-wave radar is very sparse and lacks semantic information.
[0003] With the widespread application of deep learning technology, many methods have introduced deep neural networks to perform depth completion on sparse depth information to obtain dense depth data, in order to compensate for the inherent sparsity and lack of semantic information in millimeter-wave radar depth data. Early methods attempted to directly complete sparse depth maps using deep neural networks with a UNet structure. However, due to the sparsity of depth and the lack of any semantic information, these methods were very limited in terms of depth accuracy and structural detail. Considering the limitations of using single-modal depth data, many recent studies have combined visible light images and millimeter-wave radar depth for depth completion. These methods often use a deep neural network consisting of two feature encoders and a feature decoder: one encoder extracts image features and the other extracts millimeter-wave radar depth features. The features of the two modalities are then fused, and a decoder is used to recover a dense depth map. This approach complements the scene priors in the camera image with the depth cues in the millimeter-wave radar depth map, resulting in more accurate absolute depth with better structural detail. Numerous experiments have validated the rationality and effectiveness of this technical approach, making it the current mainstream solution for millimeter-wave radar depth completion.
[0004] However, most of the fusion schemes adopted by existing camera-millimeter-wave radar fusion depth completion methods simply splice the features of the image modality and the depth modality without sufficient interaction between the features of the two modalities, resulting in unsatisfactory accuracy and structural details of the generated dense depth map. Summary of the Invention
[0005] In view of the limitations of feature fusion in existing camera-millimeter-wave radar fusion depth completion methods, this paper starts from the perspective of how to build a better feature interaction and fusion solution. According to the task characteristics, a camera-millimeter-wave radar fusion depth completion method based on structural information guidance is designed. By extracting structural information from image features, the millimeter-wave radar depth features are enhanced, thereby achieving accurate depth completion with richer structural details.
[0006] The present invention is achieved by adopting the following technical solutions:
[0007] A camera-millimeter-wave radar fusion depth completion method based on structural information guidance is proposed. The method comprises the following steps: first, image features and depth features are extracted from an RGB image and a millimeter-wave depth map, respectively; then, the image features are used to guide the depth features for enhancement; then, the enhanced depth features and image features are extracted and enhanced at a larger scale; the feature extraction and enhancement steps are repeated until the feature scale reaches a preset scale; the image features and depth features at the last scale are fused; and the fused features are upsampled scale by scale to obtain a dense depth map of the same size as the millimeter-wave depth map.
[0008] In the above technical solution, further, the camera-millimeter-wave radar fusion depth completion method based on structural information guidance has the following specific steps:
[0009] S1: Given a pair of RGB images and sparse mmWave depth maps The initial image features are obtained through a set of downsampling and dimension upscaling operations respectively. and initial depth features H and W represent the height and width of the image respectively.
[0010] S2: respectively from f I and f d Extract image features and deep features Where i represents the scale of the feature, C g Represents the dimension of image features (i.e., guided features), C d The dimension representing the deep features;
[0011] S3: Exploitation Structural information in the Enhance and obtain enhanced depth features
[0012] S31: and Splicing on the feature dimension to obtain fusion features
[0013] S32: Extraction using channel attention mechanism The structural information of the channel dimension is obtained to obtain a channel attention map
[0014] S33: Channel attention map and deep features Perform pixel-by-pixel multiplication to obtain the depth features enhanced by channel-level features
[0015] S34: Extraction using spatial attention mechanism The structural information of the spatial dimension in the middle is used to obtain a spatial attention map
[0016] S35: Spatial Attention Map and deep features enhanced by channel-level features Perform pixel-by-pixel multiplication to obtain depth features that are further enhanced by spatial features.
[0017] S36: Image features and The feature dimension is concatenated and then matched to the feature dimension by a layer of 3×3 convolution. Consistent, and then get the enhanced depth features
[0018] S4: and Set to new f I and f d , repeat the operations S2-S3 at scale i+1 until i reaches the preset scale;
[0019] S5: The image features obtained at the last scale and deep features Perform fusion to obtain the fusion features at the last scale Combine the image features at the last scale and the enhanced depth features at the last scale to obtain the fusion features at the last scale. Perform upsampling and convolution operations to obtain the fusion features at the last-1 scale use replace Perform upsampling and convolution operations at the last-1 scale; repeat upsampling and convolution operations at the next scale until the image with the same resolution as d is restored. r Dense depth maps of the same size
[0020] Furthermore, the downsampling and dimension upscaling operations in step S1 are specifically as follows:
[0021] For the RGB image I, it first passes through a convolution Conv2d with a convolution kernel of 7×7 and a stride of 2. I The operation is to increase the dimension and perform preliminary downsampling, so that the dimension is increased from 3 to 32 and the size is reduced to 1. Then an adaptive maximum pooling operation AMP is used to further reduce the size to I Finally, the initial image feature f is obtained I =AMP(Conv2d I (I)). For the sparse mmWave depth map d r , using the same operation as I (first through Conv2d d The operation performs dimensionality increase and preliminary downsampling, and then further reduces the size to d through an adaptive maximum pooling operation AMP r of ), we can get the initial depth feature f d =AMP(Conv2d d (d r )). Among them, Conv2d d The convolution kernel and step size settings are the same as Conv2d I Exactly the same, the difference is that the dimensionality increase process is from 3 to 16 dimensions.
[0022] Furthermore, the specific method for extracting image features and depth features in step S2 is as follows:
[0023] 1) For input features The first convolution layer Conv2d1 is used to extract features from x and activated by the activation function σ to obtain the first-layer features a1 = σ(Conv2d1(x)). The parameters of Conv2d1 can be adjusted as needed.
[0024] 2) The first-layer features a1 are fed into the second convolutional layer Conv2d2, and feature extraction is performed using the same method as in 1) to obtain the second-layer features a2 = σ(Conv2d2(x)). The parameters of Conv2d2 can also be adjusted as needed.
[0025] 3) The input feature x is added to the second layer feature a2 as a skip connection, and then the output feature y = σ(a2 + x) of the ResNet module is obtained through the activation function σ.
[0026] The above input feature x is an image feature or depth feature at a certain scale. For example, when x is f I or f dBased on the above steps 1)-3) you can get f I and f d Extract image features and deep features
[0027] Furthermore, the step S32 is specifically as follows: using the average pooling operation to calculate the fusion feature in the spatial dimension The average value of the channel number remains unchanged during the average pooling operation, and the shape becomes (C g +C d )×1×1; Use a multi-layer perceptron to extract features from the result of the average pooling operation and perform dimensionality reduction to obtain the first intermediate feature
[0028] At the same time, the fusion features After a maximum pooling operation, the shape becomes (C g +C d )×1×1, and then use the multi-layer perceptron to extract features from the maximum pooling operation result and perform dimensionality reduction to obtain the second intermediate feature
[0029] Will and Add, then pass a Sigmoid activation function, and finally get the channel attention map
[0030] Furthermore, the specific method of step S34 is as follows:
[0031] 1) Deep features enhanced by channel-level features The average operation is performed on the channel dimension while keeping the feature size unchanged, and the shape becomes 1×H×W.
[0032] 2) is in parallel with 1), Find the maximum value of the channel dimension while keeping the feature size unchanged, and the shape becomes 1×H×W.
[0033] 3) Concatenate the results of steps 1) and 2) in the feature dimension, and the shape becomes 2×H×W.
[0034] 4) Reduce the dimension of the result in 3) through a 3×3 convolution, then activate it through a Sigmoid function, and finally obtain the spatial attention map
[0035] Furthermore, step S5 is implemented based on a multi-scale skip connection decoder. The multi-scale skip connection decoder is mainly composed of a combination of 5 upsampling + convolution, which is used to gradually restore the fused features at the last scale into a dense depth map; in this process, the multi-scale fused features obtained by the S1-S4 feature extraction steps will be introduced into the decoder in the form of skip connections to maintain the receptive field at multiple scales. The specific method is:
[0036] 1) For the fusion features at the last scale Image features at the last scale And the enhanced depth features at the last scale First, The dimension is raised to Consistent, then added to get the skip connection feature Among them, project represents the dimension improvement operation; then the fusion feature After upsampling and convolution operation deconv, the deconv result and Splicing on the feature dimension to obtain the features after upsampling the features at the last scale
[0037] 2) Use the last-1 scale replace Combined with the image features of its corresponding scale (last-1 scale) And the depth feature of its corresponding scale (last-1 scale) Repeat the operation in step 1) to obtain
[0038] 3) Repeat the operation of step 2) at the next scale (the number of cycles is equal to the preset scale in S4) until the same scale as d is restored. r Dense depth map d of the same size out .
[0039] Furthermore, the method is implemented based on multi-scale structural information to guide the deep neural network. The training process of the multi-scale structural information to guide the deep neural network is as follows:
[0040] 1) Compared with millimeter wave radar, the point cloud of lidar is denser and the measurement value is more accurate due to the different imaging mechanism. Therefore, the lidar point cloud is projected onto the image plane to form the lidar depth map, which is used as the true value of the depth map d gt .
[0041] 2) Due to d gt is relatively sparse and lacks semantic information during supervision. Therefore, the depth map truth value d of the multi-frame lidar point cloud isgt Fusion is performed and then projected to obtain a relatively dense lidar depth map d acc . Further, for d acc Perform linear interpolation to obtain a fully dense lidar depth map d int , and delete the depth values of unreasonable areas (such as the sky).
[0042] 3) Use L2 loss to measure d out with d acc and d int The difference between acc is sparse, so only in d acc The loss is calculated at non-zero positions. The loss calculation process can be expressed as follows:
[0043]
[0044] Among them, Ω acc Indicates d acc The reasonable area in Ω int Indicates d int The reasonable area in d acc (u,v) represents the pixel at (u,v) in the relatively dense lidar depth map, d out (u,v) represents the pixel at (u,v) in the dense depth map output by the deep neural network, d int (u,v) is the pixel at (u,v) in the fully dense lidar depth map.
[0045] 4) Use the true value and loss function set in 1)-3) to train the multi-scale structural information guided deep neural network until the set number of iterations; during the training process, use stochastic gradient descent to update the learnable parameters of the neural network.
[0046] The beneficial effects of the present invention are:
[0047] In response to the limitations of feature fusion in existing camera-millimeter-wave radar fusion depth completion methods, the present invention captures the structural information in the camera RGB image to guide the sparse and semantic-lacking millimeter-wave radar depth for depth completion. Given a pair of RGB images and a sparse millimeter-wave depth map, the method first extracts image features and depth features respectively, then uses the structural information in the image features to guide the depth features for enhancement to enhance the structural expression of the depth features, then repeats the above feature extraction and enhancement steps for the enhanced depth features and image features, and then fuses the image features and depth features at the last scale, and finally upsamples the fused features to restore a dense depth map. The method of the present invention realizes the migration of image feature structural information to sparse depth features, and by operating on multi-scale features, the feature extraction process can focus on channels and spatial positions that are strongly correlated with structural information, ultimately achieving the recovery of a dense depth map with good structural details and accurate absolute depth values from a sparse millimeter-wave radar depth map. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 This is a schematic diagram of the structure of a deep neural network guided by multi-scale structural information in an embodiment of the present invention.
[0049] Figure 2 Schematic diagram of the structure of the structure information guide module SGB according to an embodiment of the present invention.
[0050] Figure 3 Schematic diagram of the training process of a deep neural network guided by multi-scale structural information according to an embodiment of the present invention.
[0051] Figure 4 Example results of an embodiment of the present invention: (a) input RGB image; (b) input sparse millimeter-wave radar depth map; (c) completed dense depth map; (d) single-frame lidar depth map; (e) denser lidar depth map after multi-frame fusion; (f) fully dense lidar depth map after linear interpolation. DETAILED DESCRIPTION
[0052] The technical solution of the present invention is further described below with reference to specific embodiments and accompanying drawings.
[0053] Example
[0054] The present invention provides a camera-millimeter-wave radar fusion depth completion method based on structural information guidance. A multi-scale structural information-guided deep neural network is designed. The structural semantic information in image features is transferred to depth features through the structural information guidance module at multiple scales, thereby enhancing the depth features and finally restoring a dense depth map with good structural details and absolutely accurate depth.
[0055] This method is based on a multi-scale structural information-guided deep neural network. Specifically, given a pair of RGB images and a sparse millimeter-wave depth map, the image features and depth features are first extracted through two ResNet modules respectively. The image features and depth features are then input into the structural information-guided module to enhance the structural expression of the depth features. The enhanced depth features and image features are then sent to the ResNet module and structural information-guided module at the next scale to repeat the above steps. The image features and depth features at the last scale are then fused. Finally, a UNet decoder based on jump connections is used to upsample the fused features scale by scale to obtain a dense depth map of the same size as the millimeter-wave depth map.
[0056] Use Figure 1 The multi-scale structural information shown guides the deep neural network to perform depth completion. The specific method is as follows:
[0057] Image branch: The shape of the input image I is 3×H×W, which becomes After the first ResNet module, the shape becomes After the second ResNet module, the shape becomes After the third ResNet module, the shape becomes Next, we will continue to downsample without increasing the dimension. After the fourth ResNet module, the shape becomes After the 5th ResNet module, the shape becomes This feature is the image feature of the last scale and will be used to fuse with the depth feature of the last scale.
[0058] Depth branch: input depth map d r The feature of is 1×H×W, which becomes After the first ResNet module, the shape becomes After the first structural information guidance module SGB and the second ResNet module, the shape becomes After the second structural information guidance module and the third ResNet module, the shape becomes Next, we will continue to downsample without increasing the dimension. After the third structural information guidance module and the fourth ResNet module, the shape becomes After the 4th structural information guidance module and the 5th ResNet module, the shape becomes This feature is the depth feature of the last scale and is no longer structurally guided. It will be used to fuse with the image feature of the last scale.
[0059] The image features and depth features obtained at the last scale are fused to obtain the fused features at the last scale; the image features at the last scale extracted by the ResNet module and the enhanced depth features at the last scale are combined, and the fused features at the last scale are upsampled and convolved to obtain the fused features at the last-1 scale; the fused features at the last-1 scale are replaced by the fused features at the last-1 scale, and upsampling and convolution operations are performed at the last-1 scale; the upsampling and convolution operations are repeated at the next scale until the fused features are restored to the same level as d r Dense depth maps of the same size.
[0060] This embodiment provides a camera-millimeter-wave radar fusion depth completion method based on structural information guidance, and the specific steps are as follows:
[0061] 1) Given a pair of RGB images ( Figure 4 (a)) and sparse mmWave depth map ( Figure 4 (b)) The initial image features are obtained through a set of downsampling and dimension upscaling operations. and initial depth features For the RGB image I, it first passes through a convolution Conv2d with a convolution kernel of 7×7 and a stride of 2. I The operation is to increase the dimension and perform preliminary downsampling, so that the dimension is increased from 3 to 32 and the size is reduced to 1. Then, an adaptive maximum pooling operation AMP is used to further reduce the feature size to I. Finally, the initial image features input to the ResNet module are obtained For the input depth map d r , using the same operation as I, we can get the deep features input to the ResNet module Among them, Conv2d d The convolution kernel and step size settings are the same as Conv2d I Exactly the same, but the number of output channels is changed from 32 to 16.
[0062] 2) Through the ResNet module, respectively from f I and f d Extract image features and deep features Where i represents the scale of the feature, C g Represents the dimension of image features (i.e., guided features), C d Represents the dimension of deep features, Figure 1 The RB in the figure is the position of the ResNet module in the entire deep neural network.
[0063] 3) and Input structure information guide module SGB, through Structural information in the Enhance and obtain enhanced depth features The position of the structure information guidance module SGB in the entire multi-scale structure information guidance deep neural network is as follows: Figure 1 As shown in SGB, its specific structure is as follows Figure 2 First, and Splicing on the feature dimension to obtain fusion features Then use the channel attention mechanism to extract The structural information of the channel dimension is obtained to obtain a channel attention map The channel attention map and deep features Perform pixel-by-pixel multiplication to obtain the depth features enhanced by channel-level features Then use the spatial attention mechanism to extract The structural information of the spatial dimension in the middle is used to obtain a spatial attention map Finally, the spatial attention map and deep features enhanced by channel-level features Perform pixel-by-pixel multiplication to obtain depth features that are further enhanced by spatial features.
[0064] Among them, the channel attention mechanism is used to extract The structural information of the channel dimension is obtained to obtain a channel attention map The specific method is:
[0065] Use the average pooling operation to calculate the fusion features in the spatial dimension The average value of the channel number remains unchanged during the average pooling operation, and the shape becomes (C g +C d )×1×1; Use a multi-layer perceptron to extract features from the result of the average pooling operation and perform dimensionality reduction to obtain the first intermediate feature
[0066] At the same time, the fusion features After a maximum pooling operation, the shape becomes (C g +C d )×1×1, and then use the multi-layer perceptron to extract features from the maximum pooling operation result and perform dimensionality reduction to obtain the second intermediate feature
[0067] Will and Add, then pass a Sigmoid activation function, and finally get the channel attention map
[0068] 4) and Set to new f I and f d , repeat the operations in 2) and 3), input it into the ResNet module at scale i+1 to extract features, and then input the results into the SGB at scale i+1 to enhance the deep features until i reaches the preset scale.
[0069] 5) The image features obtained at the last scale and deep features Perform fusion to obtain the fusion features at the last scale Combine the image features at the last scale and the enhanced depth features at the last scale to obtain the fusion features at the last scale. Perform upsampling and convolution operations to obtain the fusion features at the last-1 scale use replace Perform upsampling and convolution operations at the last-1 scale; repeat upsampling and convolution operations at the next scale until the image with the same resolution as d is restored. r Dense depth maps of the same size Specifically, this step includes the following steps: 51) For the fusion features at the last scale Image features at the last scale And the enhanced depth features at the last scale First, The dimension is raised to Consistent, then added to get the skip connection feature Among them, project represents the dimension improvement operation; then the fusion feature After upsampling and convolution operation deconv, the deconv result and Splicing on the feature dimension to obtain the features after upsampling the features at the last scale 52) Use the last-1 scale replace Combined with the image features of the corresponding scale And the depth features of the corresponding scale Repeat the operation in step 51) to obtain 53) Repeat step 52) at the next scale until the same value as d is restored. rDense depth map d of the same size out ,like Figure 4 (c) shown.
[0070] The multi-scale structural information guided deep neural network training method is: obtain a relatively dense lidar depth map through the lidar point cloud and dense lidar depth map Used to supervise the training of the entire deep neural network. Compared with millimeter wave radar, the point cloud of lidar is denser and the measurement value is more accurate due to the different imaging mechanism. Therefore, the lidar point cloud is projected onto the image plane to form a lidar depth map, which is used as the true value of the depth map d gt ,like Figure 4 (d). gt It is relatively sparse and lacks semantic information during supervision, so the depth map truth value d of the multi-frame lidar point cloud is gt Fusion is performed and then projected to obtain a relatively dense lidar depth map d acc ,like Figure 4 (e) shows. Further, for d acc Perform linear interpolation to obtain a fully dense lidar depth map d int , and delete the depth values of some unreasonable areas (such as the sky), such as Figure 4 (f) shows that L2 loss is used to measure d out with d acc and d int The difference between acc is sparse, so only in d acc Calculate loss at non-zero positions, depth loss The calculation process can be expressed as follows:
[0071]
[0072] Among them, Ω acc Indicates d acc The reasonable area in Ω int Indicates d int The reasonable area in d acc (u,v) represents the pixel at (u,v) in the relatively dense lidar depth map, d out (u,v) represents the pixel at (u,v) in the dense depth map output by the deep neural network, d int (u, v) is the pixel at (u, v) in the fully dense LiDAR depth map. Finally, a multi-scale structural information-guided deep neural network is trained based on the set true value and loss function until the set number of iterations is reached. Stochastic gradient descent is used during training to update the learnable parameters of the neural network.
[0073] The present invention realizes the migration of image feature structural information to sparse depth features, and by operating on multi-scale features, the feature extraction process can focus on channels and spatial positions that are strongly correlated with structural information, ultimately restoring a dense depth map with good structural details and accurate absolute depth values from a sparse millimeter-wave radar depth map. Figure 4 (a) is the input RGB image I, Figure 4 (b) is the input sparse millimeter wave depth map d r , Figure 4 (c) is the dense depth map completed by the present invention. Figure 4 (c) The target objects in the dense depth map have clear depth boundaries and structural contours, and there is no depth adhesion between objects, which demonstrates the effectiveness of the structural information migration in the present invention.
Claims
1. A camera-millimeter-wave radar fusion depth completion method based on structural information guidance, characterized in that: First, image features and depth features are extracted from the RGB image and millimeter-wave depth map, respectively. The image features are then used to guide the depth features for enhancement. The enhanced depth features and image features are then extracted and enhanced at a larger scale. The feature extraction and enhancement steps are repeated until the feature scale reaches a preset scale. The image features and depth features at the last scale are fused, and the fused features are upsampled scale by scale to obtain a dense depth map of the same size as the millimeter-wave depth map. The specific steps are as follows: S1: Given a pair of RGB images and sparse mmWave depth maps , respectively, through a set of downsampling and dimension upscaling operations to obtain the initial image features and initial depth features , Represents the height and width of the image respectively; S2: respectively from and Extract image features and deep features ,in The scale of the feature, Represents the dimension of image features, The dimension representing the deep features; S3: Exploitation Structural information in the Enhance and obtain enhanced depth features ; Specifically include: S31: and Splicing is performed on the feature dimension to obtain fusion features ; S32: Extraction using channel attention mechanism The structural information of the channel dimension is obtained to obtain a channel attention map ; S33: Channel attention map and deep features Perform pixel-by-pixel multiplication to obtain the depth features enhanced by channel-level features ; S34: Extraction using spatial attention mechanism The structural information of the spatial dimension in the middle is used to obtain a spatial attention map ; S35: Spatial Attention Map and deep features enhanced by channel-level features Perform pixel-by-pixel multiplication to obtain depth features that are further enhanced by spatial features. ; S36: Image features and Concatenate on the feature dimension and then pass through a layer Convolution matches the feature dimension to Consistent, and then get the enhanced depth features ; S4: and Set as New and ,exist Repeat S2-S3 at +1 scale until Reaching the preset scale; S5: The image features obtained at the last scale and deep features Perform fusion to obtain the fusion features at the last scale , also includes: 1) For the fusion features at the last scale , image features at the last scale And the enhanced depth features at the last scale ; First, The dimension is raised to Consistent, then added to get the skip connection feature ,in Represents the dimension enhancement operation; then the fusion feature After upsampling and convolution operations , then Results and Splicing on the feature dimension to obtain the features after upsampling the features at the last scale ; 2) Use At the scale replace , combined with the image features of the corresponding scale And the depth features of the corresponding scale Repeat the operation in step 1) to obtain ; 3) Repeat step 2) at the next scale until the same Dense depth maps of the same size .
2. The camera-millimeter-wave radar fusion depth completion method based on structural information guidance according to claim 1 is characterized in that: The downsampling and dimension upscaling operations in step S1 are as follows: 1) For the RGB image , first pass a convolution kernel as And the step length is Convolution The operation is to increase the dimension and perform preliminary downsampling, so that the dimension is increased from 3 to , the size is reduced to of ; Then an adaptive maximum pooling operation AMP is used to further reduce the size to of ; Finally get the initial image features ; 2) For sparse mmWave depth maps , using similar operations as in 1) to obtain the initial depth features ;in, The convolution kernel and step size settings are the same as Exactly the same; the difference from 1) is that the dimension-raising process is to raise the dimension from 3 to .
3. The camera-millimeter-wave radar fusion depth completion method based on structural information guidance according to claim 1 is characterized in that: The extraction of image features and depth features in step S2 is implemented by the ResNet module. The specific operation of the ResNet module is as follows: 1) For input features , using the first convolution layer right Perform feature extraction and activate the function Activate and get the first layer features ;in, The parameters can be adjusted as needed; 2) The first layer features Send to the second convolutional layer , perform feature extraction in the same way as in 1) to obtain the second layer of features ,in, The parameters can be adjusted as needed; 3) Input features Using skip connection and second layer features Add, and then pass the activation function Get the output features of the ResNet module .
4. The camera-millimeter-wave radar fusion depth completion method based on structural information guidance according to claim 1 is characterized in that: The step S32 is specifically as follows: Use the average pooling operation to calculate the fusion features in the spatial dimension The average value of the channel remains unchanged during the average pooling operation, and the shape becomes ; Perform feature extraction and dimensionality reduction on the result of the average pooling operation to obtain the first intermediate feature ; At the same time, the fusion features After a maximum pooling operation, the shape becomes , then perform feature extraction and dimensionality reduction on the result of the maximum pooling operation to obtain the second intermediate feature ; Will and Add, then pass a Sigmoid activation function, and finally get the channel attention map .
5. The camera-millimeter-wave radar fusion depth completion method based on structural information guidance according to claim 1 is characterized in that: The specific method of step S34 is as follows: 1) Deep features enhanced by channel-level features Perform the average operation on the channel dimension while keeping the feature size unchanged, and the shape becomes ; 2) Yes Find the maximum value of the channel dimension while keeping the feature size unchanged, and the shape becomes ; 3) Concatenate the results of steps 1) and 2) in the feature dimension, and the shape becomes ; 4) Through a Convolution reduces the dimension of the result in 3), and then activates it through a Sigmoid function to obtain the spatial attention map. .
6. The camera-millimeter-wave radar fusion depth completion method based on structural information guidance according to claim 1 is characterized in that: This method is based on multi-scale structural information to guide the deep neural network. The training process of the multi-scale structural information to guide the deep neural network is as follows: 1) Project the LiDAR point cloud onto the image plane to form a LiDAR depth map, which is used as the ground truth of the depth map. ; 2) The depth map of the multi-frame lidar point cloud is Fusion and then projection to obtain a relatively dense lidar depth map ;right Perform linear interpolation to obtain a fully dense lidar depth map , and delete the depth values of unreasonable areas; 3) Measured by L2 loss and and The difference between is sparse, so only Calculate loss at non-zero positions, depth loss The calculation process is expressed as follows: ， in, express The reasonable area in express The reasonable area in Represents a relatively dense lidar depth map The pixel point at Represents the dense depth map output by a deep neural network The pixel point at For the fully dense lidar depth map Pixels at 4) Use the true values and loss functions set in 1)-3) to train the multi-scale structural information-guided deep neural network until the set number of iterations; during the training process, stochastic gradient descent is used to update the learnable parameters of the neural network.