Three-dimensional target detection method integrating depth clues in night traffic scene

Abstract

The invention discloses a depth clue-integrated three-dimensional target detection method in a night traffic scene, and the method comprises the steps: firstly obtaining all-round camera and radar point cloud data in the night traffic scene, inputting the data into a backbone network, and extracting features; an image feature map and a point cloud feature map are obtained, the image feature map and the point cloud feature map are input into a depth clue integration module to be subjected to depth integration and visual angle conversion, image BEV features and point cloud BEV features are generated, the generated BEV features are fused through a fusion module to generate fused BEV features, and finally the fused BEV features are input into a universal 3D detection head to be subjected to a three-dimensional target detection task. And completing three-dimensional target detection in a night traffic scene. According to the method, the advantages of radar data are fully utilized, the loss of semantic information of the camera in the night traffic scene can be overcome, the defect of the camera in depth estimation can be overcome, and the advantages in the night traffic scene are that the robustness of the detection effect is higher, and the generalization performance of the detection system in different weather conditions is better.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving environment perception, and specifically relates to a three-dimensional target detection method in nighttime traffic scenes integrating depth cues. Background Art

[0002] Autonomous driving has garnered significant attention in recent years. Object detection is one of the most critical tasks in vehicle perception systems. High-quality detection results serve as a reliable basis for subsequent tasks such as trajectory prediction and path planning. Accurate object detection enables vehicles to intelligently perceive their surroundings and even predict impending accidents and take appropriate control decisions, effectively avoiding traffic accidents caused by human error and improving road safety.

[0003] As the most common visual sensor, monocular cameras are the preferred choice for object detection due to their rich semantic information and low cost. Current pure vision methods are mostly based on the BEV (Bird's Eye View) concept. They first construct a voxel space in front of each camera and assign coordinates to each voxel. By predicting the depth probability of each pixel in the surround 2D image within the voxel space, the pixel's features are assigned to the corresponding voxel to generate a pseudo-point cloud, thereby "lifting" the 2D image into 3D space. Pooling is then performed on the features in voxels at different heights to "flatten" the 3D space into a 2D BEV feature map. BEV-based methods merge the receptive fields of multiple cameras into the BEV space. Under the unified BEV perspective, object detection is no longer isolated and fragmented. Continuous object movements can also be predicted from a global perspective, analyzing the target's intentions, such as lane changes and overtaking. This not only improves the perception system's prediction accuracy but also increases perception efficiency, freeing up more time for the regulation and control module. However, the natural defects of monocular cameras make obtaining three-dimensional information such as depth from them an ill-posed problem, especially in nighttime environments with lack of and uneven lighting, where the camera's semantic information is severely lost, making it even more difficult to obtain three-dimensional information.

[0004] Existing multimodal fusion target detection methods typically combine the camera modality with precise three-dimensional information provided by sensors such as lidar and millimeter-wave radar. The echo from the millimeter-wave radar does not attenuate at night, providing nearly identical three-dimensional information to that found during daytime, when illumination conditions are good. Therefore, it is more suitable for target detection tasks in nighttime scenes with poor and uneven illumination. However, existing multimodal fusion methods based on cameras and millimeter-wave radars typically process the data from the two modalities separately, convert it to a BEV perspective, and then fuse the features of the different modalities from the BEV perspective. This fusion method processes the data from different modalities independently, performing feature-level fusion only in the BEV space, and does not fully utilize the three-dimensional information provided by the millimeter-wave radar. Independently processing image data means that the challenge of "lifting" image features to three-dimensional space remains. As a result, many methods that perform well during the daytime experience significant degradation in scenes with poor and uneven illumination at night. In addition, this fusion method ignores the differences between the two data in the feature space. Since the point cloud data provided by the millimeter-wave radar is very sparse compared to the image, a lot of information about small-sized targets that are very important for traffic scenes, such as pedestrians and bicycles, will be lost during the fusion process, which is not conducive to the safe driving of autonomous vehicles. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a three-dimensional target detection method in nighttime traffic scenes that integrates depth cues, providing a more reliable three-dimensional target detection solution for the development of autonomous driving and intelligent transportation systems in nighttime traffic scenes.

[0006] The technical solution adopted by the present invention is: a three-dimensional target detection method in nighttime traffic scenes integrating depth cues, the specific steps are as follows:

[0007] S1. Obtain surround-view camera and radar point cloud data in nighttime traffic scenes, input them into the backbone network to extract features, and obtain image feature maps and point cloud feature maps;

[0008] The backbone network includes: an image backbone network and a radar backbone network.

[0009] The surround view camera data is input into the image backbone network to extract features and obtain the image feature map C I The image backbone network first extracts the multi-scale features of the image from the bottom up through the convolutional network, and then fuses the multi-scale information of the image from the top down through the feature pyramid.

[0010] The radar point cloud data is input into the radar backbone network to extract features, and the point cloud feature map C is obtained by first voxel encoding and then feature extraction. P , as follows:

[0011] (1) Voxel encoding;

[0012] First, the input point cloud is divided into a uniform voxel grid in three-dimensional space, without any height division. The divided voxels are cylindrical. The voxelized point cloud data is then encoded point by point through two multi-layer perceptrons. The points within the voxel grid are then aggregated using a max pooling operation, aggregating the multiple point features within the voxel into a single voxel feature.

[0013] (2) Feature extraction;

[0014] The voxel-encoded point cloud data is used to extract multi-scale features from the bottom up using a sparse convolutional network, and then the multi-scale feature information is fused from the top down through a feature pyramid.

[0015] S2. Input the image feature map and point cloud feature map obtained in step S1 into the depth cue integration module for depth integration and perspective conversion, and generate feature maps of the image and point cloud under the BEV perspective, i.e., image BEV features and point cloud BEV features;

[0016] The depth clue integration module includes: a depth estimation network, a point cloud distribution network, and a view conversion module.

[0017] S3, fusing the image BEV features and point cloud BEV features generated in step S2 through a fusion module to generate a fused BEV feature;

[0018] S4. Based on step S3, the fused BEV features are input into a universal 3D detection head to perform a 3D target detection task, thereby completing 3D target detection in nighttime traffic scenarios.

[0019] Furthermore, the step S2 is specifically as follows:

[0020] S21, inputting the image feature map obtained in step S1 into the depth estimation network to obtain the depth distribution of the feature points in the three-dimensional space;

[0021] Image feature map C I The input depth estimation network first passes through a dimensionality reduction convolution layer, then a depth convolution layer, and finally a sigmoid operation to normalize the features into a depth distribution D. I , D I The dimensions are [N1,D1,H1,W1].

[0022] The dimensionality reduction convolution layer includes convolution, normalization, and the ReLU activation function. The depthwise convolution layer includes three basic residual blocks and a convolution operation. Each residual block undergoes a sub-downsampling convolution, followed by an output dimension adaptation convolution to extract depth distribution features. N1 represents the batch size, D1 represents D1 depth probabilities, i.e., the probability that a feature point belongs to each depth. H1 and W1 represent the height and width of the feature map, respectively.

[0023] S22, inputting the radar cross-sectional area of the radar point cloud data and the point cloud feature map obtained in step S1 into a point cloud distribution network to obtain a point cloud distribution;

[0024] First, based on the distance between the point cloud and the vehicle and the radar cross-sectional area rcs, the point cloud is diffused into a circular area to obtain a diffuse point cloud feature map. The center of the diffusion circle is the original point cloud, and the diffusion radius r is calculated as follows:

[0025]

[0026] Among them, (u, v) represents the normalized horizontal position of the valid point cloud in three-dimensional space.

[0027] The standard deviation is set according to the diffusion radius r A two-dimensional Gaussian kernel is used to draw a two-dimensional Gaussian distribution with each valid point cloud as the center, and finally a Gaussian heat map G is generated. The heat map G is spliced with the diffuse point cloud feature map and input into a convolution layer. The number of channels is adjusted to keep the same as the point cloud feature C. P The number of channels is consistent with that of the point cloud feature map C P The radar cross section is encoded into the point cloud feature C through a convolutional layer. P In the convolution operation, the fusion features are obtained, and then the fusion features are subjected to a two-step convolution operation. The normalization operation and the relu activation function are used in the first layer, and the sigmoid function is used for normalization in the second layer. That is, the number of channels is adjusted to 1 and the feature values are normalized by the sigmoid function to obtain a point cloud distribution map D with a dimension of 1 and a size of D×W. P , D P The dimensions are [N2,D2,H2,W2].

[0028] Where N2 represents the batch size, D2 and W2 represent the width and length of the point cloud distribution map, respectively. N2, D2, and W2 are equal to N1, D1, and W1, respectively. If the millimeter-wave radar does not provide height information, H2 is 1.

[0029] S23, input the depth distribution and point cloud distribution obtained in steps S21 and S22 into the view conversion module, and combine them with the image feature map obtained in step S1 to generate image BEV features;

[0030] The view conversion module includes conversion and refinement operations.

[0031] The conversion operation is the depth distribution D I and point cloud distribution D P Respectively with the image feature map C I Perform outer product to elevate the image features to three-dimensional space, where the point cloud distribution D P If the height dimension is 1, the image features are folded by summing along the height axis; image feature map C I and depth distribution D I The outer product of is summed along the height axis. The results are then stacked to generate the preliminary image BEV feature B I , the specific operation expression is as follows:

[0032]

[0033] Among them, Conv() represents the convolution operation, (.;.) represents the cascade operation along the channel, Represents the outer product operation.

[0034] The refinement operation receives the preliminary image BEV features B I ,include:

[0035] 1) Feature compression layer: compresses high-dimensional features to intermediate dimensions through dimensionality reduction convolutional layers;

[0036] 2) Nonlinear enhancement layer: After two 3×3 convolutions, batch normalization and ReLU activation are applied after each convolution;

[0037] 3) Output adaptation layer: Project the features to the number of channels matching the BEV grid, where the output feature map resolution remains consistent with the input.

[0038] S24, diffuse the point cloud feature map generated in step S22 After splicing with the Gaussian heat map G, the channel dimension is adjusted to 1, and then combined with the point cloud feature map C P Perform point-by-point multiplication and encode the radar cross-sectional area as the representation of the spatially significant information enhanced point cloud feature to form the final point cloud BEV feature B P .

[0039] Furthermore, the step S3 is specifically as follows:

[0040] The fusion module first performs BEV feature B on the input image. I and point cloud BEV feature B PThe layers are normalized separately, and the dimensions of the two modalities are flattened through the linear projection layer. The features of the two modalities are then spliced in the channel dimension and projected into a unified dimension through the linear layer. Finally, cross-modal feature fusion is performed through the cascade of n fusion layers to generate the fused BEV feature B. F .

[0041] Among them, the number of fusion layers n is set according to actual conditions, and each fusion layer includes: an inter-modal feature alignment module based on a deformable cross-attention mechanism, a fully connected feedforward network layer, and the alignment module and the fully connected feedforward network layer are followed by a normalization layer.

[0042] The alignment module aligns the image and point cloud features; the fully connected feedforward network layer, after the deformable attention completes the preliminary modality alignment, the fully connected feedforward network layer further mixes the features of different modalities through the fully connected layer; the normalization layer, through normalization of the mean and variance.

[0043] Furthermore, the step S4 is specifically as follows:

[0044] First, the fused multi-scale features B are further extracted through the shallow feature extraction network and feature pyramid. F ,The further extracted features are then fed into the universal 3D detection head for ,3D object detection task, completing 3D object detection in night ,traffic scenes.

[0045] The loss functions include regression loss and classification loss. The regression loss uses the L1 loss function for bounding box regression, and the classification loss uses the Gaussian perceptron loss function for object classification.

[0046] The beneficial effects of the present invention are as follows: the method of the present invention first obtains the surround-view camera and radar point cloud data in the night traffic scene, inputs them into the backbone network to extract features, obtains image feature maps and point cloud feature maps, inputs them into the depth clue integration module for depth integration and perspective conversion, generates image BEV features and point cloud BEV features, and then fuses the generated BEV features through the fusion module to generate fused BEV features, and finally inputs the fused BEV features into the universal 3D detection head to perform three-dimensional target detection tasks, thereby completing three-dimensional target detection in night traffic scenes. The method of the present invention can effectively utilize the precise three-dimensional information provided by radar point cloud data in nighttime traffic scenes by introducing a depth clue integration module, and improve the reliability of the depth estimation network by utilizing the size prior information provided by the radar reflection cross-sectional area, thereby improving the accuracy of the operation of promoting the two-dimensional image to three-dimensional space, and further improving the accuracy and robustness of three-dimensional target detection; the method of the present invention fully utilizes the advantages of radar data, which can not only overcome the loss of camera semantic information in nighttime traffic scenes, but also make up for the shortcomings of the camera in depth estimation. The advantages in nighttime traffic scenes are reflected in the stronger robustness of the detection effect and the better generalization performance of the detection system under different weather conditions, providing a more reliable three-dimensional target detection solution for the development of autonomous driving and intelligent transportation systems in nighttime traffic scenes.

[0047] Compared with existing multimodal target detection methods, the innovation of the present invention lies in providing a more accurate BEV feature representation of the camera and millimeter-wave radar modalities for the subsequent general fusion detection module through a series of operations in the depth cue integration module, especially in nighttime traffic scenarios. The method of the present invention uses the radar cross-sectional area information provided by the millimeter-wave radar modality as a priori information on the target size to diffuse the sparse radar point cloud features. On the one hand, the diffused point cloud features are encoded into the point cloud distribution through the point cloud distribution network, and then combined with the image depth distribution generated by the depth estimation network to guide the view conversion of image features from the perspective view to the BEV view, solving the pain point that it is difficult to estimate depth from image features under nighttime conditions due to severe damage to the camera semantic information; on the other hand, it enhances the representation of point cloud features as spatially significant information, preventing the features of small targets in nighttime scenes from being smoothed out by background features, thereby making subsequent feature fusion more accurate and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flowchart of a three-dimensional target detection method in nighttime traffic scenes that integrates depth cues according to the present invention.

[0049] Figure 2 2 is a network structure diagram of a depth clue integration module in an embodiment of the present invention.

[0050] Figure 3The figure is a schematic diagram showing the visualization of detection results in a common nighttime traffic scene according to an embodiment of the present invention.

[0051] Figure 4 The figure is a visualization diagram of the small target detection results in a nighttime traffic scene according to an embodiment of the present invention.

[0052] Figure 5 This is a schematic diagram showing the visualization of detection results in a complex nighttime traffic scene according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The method of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0054] like Figure 1 As shown in FIG, a flow chart of a method for three-dimensional target detection in nighttime traffic scenes integrating depth cues of the present invention is shown, and the specific steps are as follows:

[0055] S1. Obtain surround-view camera and radar point cloud data in nighttime traffic scenes, input them into the backbone network to extract features, and obtain image feature maps and point cloud feature maps;

[0056] The backbone network includes: an image backbone network and a radar backbone network. In this embodiment, the surround view images provided by the nuScenes dataset and the corresponding radar point cloud information are used as training and testing data.

[0057] The surround view camera data is input into the image backbone network to extract features and obtain the image feature map C I . The image backbone network first extracts the multi-scale features of the image from the bottom up through the convolutional network, and then fuses the multi-scale information of the image from the top down through the feature pyramid, so as to retain the high-resolution spatial details and low-resolution semantic information of the image to support subsequent tasks. The image backbone network in this embodiment uses a pre-trained ResNet-50 pair to extract the multi-scale features of the image, outputs the multi-scale image features of the first four stages, and uses SECONDFPN as the feature pyramid to fuse the multi-scale information of the image.

[0058] The radar point cloud data is input into the radar backbone network to extract features, and the point cloud feature map C is obtained by first voxel encoding and then feature extraction. P , as follows:

[0059] (1) Voxel encoding;

[0060] This embodiment uses a point cloud voxel encoding operation based on the PointPillars architecture. First, the input point cloud is divided into a uniform voxel grid in three-dimensional space along the x, y, and z directions of the ego-vehicle coordinate system. The ego-vehicle is the origin, the x-axis is the direction facing the ego-vehicle, the y-axis is the left direction facing the ego-vehicle, and the z-axis is upward. The x- and y-axis ranges at ground level are both (-51.2m, 51.2m), with intervals of 0.8m. The z-axis range, representing altitude, is (-5, 3). Since millimeter-wave radar cannot provide altitude information, the z-axis is not divided. The divided voxels are cylindrical voxels. The entire scene is divided into a 128×128×1 uniform voxel grid. The original point cloud is assigned to the corresponding voxel, and points outside the above range are ignored. Each grid retains a maximum of 8 valid radar point clouds. If the original point cloud in a voxel exceeds 8, it is downsampled to 8. The x, y, and z coordinates, radar cross-section (rcs), compensated velocity (v), and offset relative to the voxel center of all points within each downsampled voxel are used as the eight original features of the voxel midpoint, i.e., the initial feature dimension of the valid points is 8. The voxelized point cloud data is then encoded point by point using two multilayer perceptrons with input and output channels of [8, 32] and [32, 64], respectively. Max pooling is then performed on the points within the voxel grid, aggregating multiple point features within a voxel into a single voxel feature. This voxel encoding operation transforms the unordered, sparse original point cloud into a structured representation for efficient feature extraction.

[0061] (2) Feature extraction;

[0062] This embodiment uses a radar point cloud processing network based on the SECOND architecture, including a sparse convolutional network and a feature pyramid. First, the voxelized encoded point cloud data is input into the SECOND network for sparse convolution, and features are extracted from the voxelized point cloud data to generate a multi-scale feature map. The image is similar to the ResNet50 in the backbone network, but is optimized for sparse point clouds. The number of input channels is 64, the number of output channels in each stage is 64, 128, and 256 respectively, the number of layers in each stage is 3, 5, and 5 respectively, and the downsampling step size of the first layer in each stage is 1, 2, and 2 respectively. The multi-scale point cloud features obtained by sparse convolution are input into the SECONDFPN feature pyramid network to fuse multi-scale features and enhance semantic and spatial information at different resolutions.

[0063] S2. Input the image feature map and point cloud feature map obtained in step S1 into the depth cue integration module for depth integration and perspective conversion, and generate feature maps of the image and point cloud under the BEV perspective, i.e., image BEV features and point cloud BEV features;

[0064] Among them, such as Figure 2As shown, the depth clue integration module includes: a depth estimation network, a point cloud distribution network, and a view conversion module.

[0065] S3, fusing the image BEV features and point cloud BEV features generated in step S2 through a fusion module to generate a fused BEV feature;

[0066] S4. Based on step S3, the fused BEV features are input into a universal 3D detection head to perform a 3D target detection task, thereby completing 3D target detection in nighttime traffic scenarios.

[0067] like Figure 2 As shown, in this embodiment, the step S2 is specifically as follows:

[0068] S21, inputting the image feature map obtained in step S1 into the depth estimation network to obtain the depth distribution of the feature points in the three-dimensional space;

[0069] Image feature map C I The input depth estimation network first passes through a dimensionality reduction convolution layer, then a depth convolution layer, and finally a sigmoid operation to normalize the features into a depth distribution D. I , D I The dimensions are [N1,D1,H1,W1].

[0070] In this embodiment, the dimensionality reduction convolution layer includes: convolution [3×3, 1, 256] (convolution kernel size is 3×3, stride is 1, output dimension is 256), normalization operation and ReLU activation function; the depth convolution layer includes: three basic residual blocks, a [3×3] convolution operation, each residual block undergoes a [3×3, 2, 256] convolution for 2x downsampling, and then undergoes a [1×1, 1, 56] convolution to extract depth distribution features. N1 represents the batch size, D1 represents D1 depth probabilities, that is, the probability that a feature point belongs to each depth, and H1 and W1 represent the height and width of the feature map, respectively.

[0071] S22, inputting the radar cross-sectional area of the radar point cloud data and the point cloud feature map obtained in step S1 into a point cloud distribution network to obtain a point cloud distribution;

[0072] First, based on the distance between the point cloud and the vehicle and the radar cross-sectional area rcs, the point cloud is diffused into a circular area to obtain a diffuse point cloud feature map. The center of the diffusion circle is the original point cloud, and the diffusion radius r is calculated as follows:

[0073]

[0074] Among them, (u, v) represents the normalized horizontal position of the valid point cloud in three-dimensional space.

[0075] In order to prevent this hard diffusion from introducing too much noise, the standard deviation is set according to the diffusion radius r. A two-dimensional Gaussian kernel is used to draw a two-dimensional Gaussian distribution with each valid point cloud as the center, and finally a Gaussian heat map G is generated. The heat map G is spliced with the diffuse point cloud feature map and input into a [1×1,1,64] convolution layer. The number of channels is adjusted to keep the same as the point cloud feature C. P The number of channels is consistent with that of the point cloud feature map C P The radar cross section is encoded into the point cloud feature C through a [3×3,1,64] convolution layer. P In the fusion feature, the fusion feature is obtained, and then a two-step convolution operation is performed on the fusion feature. The convolution layer sizes are [3×3,1,32] and [3×3,1,1] respectively. The normalization operation and relu activation function are used in the first layer, and the sigmoid function is used for normalization in the second layer. That is, the number of channels is adjusted to 1 and the feature value is normalized by the sigmoid function to obtain a point cloud distribution map D with a dimension of 1 and a size of D×W. P , D P The dimensions are [N2,D2,1,W2].

[0076] Where N2 represents the batch size, D2 and W2 represent the width and length of the point cloud distribution map, respectively. N2, D2, and W2 are equal to N1, D1, and W1, respectively. If the millimeter-wave radar does not provide height information, H2 is 1.

[0077] S23, input the depth distribution and point cloud distribution obtained in steps S21 and S22 into the view conversion module, and combine them with the image feature map obtained in step S1 to generate image BEV features;

[0078] The view conversion module includes conversion and refinement operations.

[0079] The conversion operation is the depth distribution D I and point cloud distribution D P Respectively with the image feature map C I Perform outer product to elevate the image features to three-dimensional space, where due to the point cloud distribution D P The height dimension is 1, so the image features are collapsed to fit the point cloud distribution D by summing along the height axis. P The size of the image feature map C I and depth distribution D I The outer product of is summed along the height axis to achieve the purpose of flattening the image features from the three-dimensional space to the BEV space. The results are stacked to generate the preliminary image BEV feature B I , the specific operation expression is as follows:

[0080]

[0081] Among them, Conv() represents the convolution operation, (.;.) represents the cascade operation along the channel, Represents the outer product operation.

[0082] The refinement operation receives the preliminary image BEV features B I ,include:

[0083] 1) Feature compression layer: The convolution layer compresses high-dimensional features to intermediate dimensions, reducing computational complexity while retaining key information.

[0084] 2) Nonlinear enhancement layer: After two 3×3 convolutions, batch normalization and ReLU activation are applied after each convolution;

[0085] 3) Output adaptation layer: Project the features to the number of channels matching the BEV grid, where the output feature map resolution remains consistent with the input.

[0086] S24, diffuse the point cloud feature map generated in step S22 After splicing with the Gaussian heat map G, the channel dimension is adjusted to 1, and then combined with the point cloud feature map C P Perform point-by-point multiplication and encode the radar cross-sectional area as the representation of the spatially significant information enhanced point cloud feature to form the final point cloud BEV feature B P .

[0087] In this embodiment, step S3 is specifically as follows:

[0088] The fusion module first performs BEV feature B on the input image. I and point cloud BEV feature B P The layers are normalized separately, and the dimensions of the two modalities are flattened through a linear projection layer. The features of the two modalities are then concatenated in the channel dimension and projected into a unified dimension (128 in this embodiment) through a linear layer. The cross-modal features are then fused through a cascade of n=6 fusion layers to generate a fused BEV feature B. F .

[0089] Wherein, N represents the batch size, which is 24 in this embodiment; H and W represent the feature map size, which is 128 in this embodiment; and C represents the number of feature channels, which is 80 in this embodiment. The fusion layer in this embodiment uses a multimodal variability cross-attention module improved based on the Transformer network architecture. Each fusion layer includes: an inter-modal feature alignment module based on a deformable cross-attention mechanism, and a fully connected feedforward network layer. The alignment module and the fully connected feedforward network layer are both followed by a normalization layer.

[0090] First, the image and point cloud features are aligned through the alignment module, and then through the fully connected feedforward network layer. After the deformable attention completes the preliminary modality alignment, the fully connected feedforward network layer further mixes the features of different modalities through the fully connected layer. In this embodiment, the fully connected feedforward network structure is fully connected layer-relu activation layer-fully connected layer; and the normalization layer reduces the distribution offset of image and point cloud features by normalizing the mean and variance, thereby accelerating convergence.

[0091] In this embodiment, step S4 is specifically as follows:

[0092] The detection head described in this embodiment adopts the CenterPoint architecture adapted for night scenes and includes two stages: further feature extraction and detection. First, the fused multi-scale features B are further extracted through the shallow feature extraction network and feature pyramid. F ,The further extracted features are then fed into the universal 3D detection head for ,3D object detection task, completing 3D object detection in night ,traffic scenes.

[0093] In this embodiment, the feature extraction phase first uses ResNet-18 as the backbone network, outputting feature maps at three stages corresponding to different resolutions and channel counts. A SECONDFPN is then used to extract the image's feature pyramid and further fuse multi-scale BEV features. This involves four upsampling steps with strides of 1, 2, 4, and 8, with each upsampling outputting 64 channels. The detection phase predicts 10 values: the offset of the bounding box center (two values, corresponding to the horizontal and vertical offsets), the object's height (one value), the object's size (three values, corresponding to the length, width, and height), the object's rotation angle (two values, corresponding to the horizontal and vertical rotation angles), and the object's speed (two values, corresponding to the horizontal and vertical speeds). Because the scales of the objects to be detected vary, including trucks and pedestrians, five detection heads are set based on the target scale: cars, trucks, pedestrians, motorcycles and bicycles, and construction obstacles. The bounding box is encoded and decoded for each of the five heads, and the final 3D bounding box is regressed and a category score is given.

[0094] The loss functions include regression loss and classification loss. The regression loss uses the L1 loss function for bounding box regression, and the classification loss uses the Gaussian perceptron loss function for object classification.

[0095] This example further conducted experimental verification on the nuScenes dataset, comparing the existing pure vision method BEVDepth and the method CRN that integrates vision and millimeter-wave radar. The detection results of the three methods for various types of targets in daytime and nighttime scenes are recorded, as shown in Table 1.

[0096] Table 1

[0097]

[0098] Among them, it can be seen from Table 1 that the detection performance (mAP) of the method of the present invention in the daytime scene reaches 54.35, and the detection performance in the nighttime scene reaches 49.18. The nighttime scene is only attenuated by 5.17 (9.5%) compared with the daytime scene. The detection results of the BEVDepth and CRN methods in the nighttime scene are attenuated by 16.85 (40.7%) and 7.86 (14.4%) respectively compared with the daytime scene. It can be seen that the detection performance of the method of the present invention in the nighttime scene is significantly lower than that of the existing method in terms of attenuation of the daytime scene. In addition, for the traffic scene For the more common car target, the detection result of the night scene reached 74.0, an improvement of 1.7 compared with the daytime scene; for large vehicles such as trucks, the detection result of the night scene reached 50.8, an improvement of 5.9 compared with the daytime scene; in addition, in this embodiment, the method of the present invention significantly improves the detection performance of small targets under nighttime traffic conditions, among which the detection results of pedestrians, bicycles and obstacles reached 44.9, 41.5 and 52.0 respectively, all higher than the other two methods, and the attenuation degree of such small targets compared with the daytime scenes was significantly lower than that of the other two methods.

[0099] The visualization results of this embodiment in the night scene of the nuScenes dataset show that the method of the present invention performs well in common traffic scenes, such as Figure 3 In addition, the method of the present invention is more accurate in detecting small targets, such as Figure 4 shown by Figure 5 It can be seen that the method of the present invention still performs well in complex traffic scenarios.

[0100] In summary, the method of the present invention fully combines the rich semantic information of the camera with the precise three-dimensional information of the radar. By introducing the size prior information provided by the radar cross-sectional area, the point cloud features are diffused. On the one hand, the depth estimation network and the point cloud distribution network are combined to guide the view transformation of image features. On the other hand, the representation of point cloud features is enhanced by generating a radar cross-sectional area attention map, preventing small targets from losing features in the higher-dimensional feature space. Compared with other multimodal fusion methods, the method of the present invention effectively overcomes the limitations of each modality and fuses multimodal information to generate a BEV scene representation with rich semantic features and accurate spatial information. Its advantages in traffic scenarios are reflected in the stronger robustness of the detection effect and the better generalization performance of the detection system under different weather conditions.

[0101] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A 3D object detection method for nighttime traffic scenes that integrates depth cues. The specific steps are as follows: S1. Obtain surround-view camera and radar point cloud data in nighttime traffic scenes, input them into the backbone network to extract features, and obtain image feature maps and point cloud feature maps; The backbone network includes: Image backbone network, radar backbone network; The surround view camera data is input into the image backbone network to extract features and obtain the image feature map C I The image backbone network first extracts multi-scale features of the image from bottom to top through the convolutional network, and then fuses the multi-scale information of the image from top to bottom through the feature pyramid. The radar point cloud data is input into the radar backbone network to extract features, and the point cloud feature map C is obtained by first voxel encoding and then feature extraction. P , as follows: (1) Voxel encoding; First, the input point cloud is divided into a uniform voxel grid in three-dimensional space, and the height is not divided. The divided voxels are cylindrical voxels. Then, the voxelized point cloud data is encoded point by point through two multi-layer perceptrons. The points in the voxel grid are aggregated through the maximum pooling operation, and the multiple point features in the voxel are aggregated into a single voxel feature. (2) Feature extraction; The voxel-encoded point cloud data is used to extract multi-scale features from the bottom up using a sparse convolutional network, and then the multi-scale feature information is fused from the top down through a feature pyramid. S2. Input the image feature map and point cloud feature map obtained in step S1 into the depth cue integration module for depth integration and perspective conversion, and generate feature maps of the image and point cloud under the BEV perspective, i.e., image BEV features and point cloud BEV features; The depth clue integration module includes: a depth estimation network, a point cloud distribution network, and a view conversion module; S3, fusing the image BEV features and point cloud BEV features generated in step S2 through a fusion module to generate a fused BEV feature; S4. Based on step S3, the fused BEV features are input into a universal 3D detection head to perform a 3D target detection task, thereby completing 3D target detection in nighttime traffic scenarios.

2. The method for three-dimensional object detection in nighttime traffic scenes integrating depth cues according to claim 1, characterized in that: The step S2 is specifically as follows: S21, inputting the image feature map obtained in step S1 into the depth estimation network to obtain the depth distribution of the feature points in the three-dimensional space; Image feature map C I The input depth estimation network first passes through a dimensionality reduction convolution layer, then a depth convolution layer, and finally a sigmoid operation to normalize the features into a depth distribution D. I , D I The dimensions are [N1, D1, H1, W1]; The dimensionality reduction convolution layer includes: convolution, normalization operation and relu activation function; the depth convolution layer includes: three basic residual blocks and a convolution operation. Each residual block undergoes a sub-downsampling convolution and then undergoes an output dimension adaptation convolution to extract depth distribution features. N1 represents the batch size, D1 represents D1 depth probabilities, that is, the probability that the feature point belongs to each depth, H1 and W1 represent the height and width of the feature map respectively. S22, inputting the radar cross-sectional area of the radar point cloud data and the point cloud feature map obtained in step S1 into a point cloud distribution network to obtain a point cloud distribution; First, based on the distance between the point cloud and the vehicle and the radar cross-sectional area rcs, the point cloud is diffused into a circular area to obtain a diffuse point cloud feature map. The center of the diffusion circle is the original point cloud, and the diffusion radius r is calculated as follows: Among them, (u, v) represents the normalized horizontal position of the valid point cloud in three-dimensional space; The standard deviation is set according to the diffusion radius r A two-dimensional Gaussian kernel is used to draw a two-dimensional Gaussian distribution with each valid point cloud as the center, and finally a Gaussian heat map G is generated. The heat map G is spliced with the diffuse point cloud feature map and input into a convolution layer. The number of channels is adjusted to keep the same as the point cloud feature C. P The number of channels is consistent with that of the point cloud feature map C P The radar cross section is encoded into the point cloud feature C through a convolutional layer. P In the convolution operation, the fusion features are obtained, and then the fusion features are subjected to a two-step convolution operation. The normalization operation and the relu activation function are used in the first layer, and the sigmoid function is used for normalization in the second layer. That is, the number of channels is adjusted to 1 and the feature values are normalized by the sigmoid function to obtain a point cloud distribution map D with a dimension of 1 and a size of D×W. P , D P The dimensions are [N2, D2, H2, W2]; Where N2 represents the batch size, D2 and W2 represent the width and length of the point cloud distribution map respectively; N2, D2 and W2 are equal to N1, D1 and W1 respectively, and the millimeter wave radar does not provide height information, then H2 is 1; S23, input the depth distribution and point cloud distribution obtained in steps S21 and S22 into the view conversion module, and combine them with the image feature map obtained in step S1 to generate image BEV features; The view conversion module includes conversion and refinement operations; The conversion operation is the depth distribution D I and point cloud distribution D P Respectively with the image feature map C I Perform outer product to elevate the image features to three-dimensional space, where the point cloud distribution D P If the height dimension is 1, the image features are folded by summing along the height axis; image feature map C I and depth distribution D I The outer product of is summed along the height axis; the results are then stacked to generate a preliminary image BEV feature B I , the specific operation expression is as follows: Among them, Conv() represents the convolution operation, (.;.) represents the cascade operation along the channel, represents the outer product operation; The refinement operation receives the preliminary image BEV features B I ,include: 1) Feature compression layer: compresses high-dimensional features to intermediate dimensions through dimensionality reduction convolutional layers; 2) Nonlinear enhancement layer: After two 3×3 convolutions, batch normalization and ReLU activation are applied after each convolution; 3) Output adaptation layer: projects the features to the number of channels that matches the BEV grid, where the output feature map resolution is consistent with the input; S24, diffuse the point cloud feature map generated in step S22 After splicing with the Gaussian heat map G, the channel dimension is adjusted to 1, and then combined with the point cloud feature map C P Perform point-by-point multiplication and encode the radar cross-sectional area as the representation of the spatially significant information enhanced point cloud feature to form the final point cloud BEV feature B P .

3. The method for three-dimensional object detection in nighttime traffic scenes integrating depth cues according to claim 1, characterized in that: The step S3 is specifically as follows: The fusion module first performs BEV feature B on the input image. I and point cloud BEV feature B P The layers are normalized separately, and the dimensions of the two modalities are flattened through the linear projection layer. The features of the two modalities are then spliced in the channel dimension and projected into a unified dimension through the linear layer. Finally, cross-modal feature fusion is performed through the cascade of n fusion layers to generate the fused BEV feature B. F ; The number of fusion layers n is set according to actual conditions. Each fusion layer includes: an inter-modal feature alignment module based on a deformable cross-attention mechanism, a fully connected feedforward network layer, and a normalization layer after each of the alignment module and the fully connected feedforward network layer. The alignment module aligns the image and point cloud features; the fully connected feedforward network layer, after the deformable attention completes the preliminary modality alignment, the fully connected feedforward network layer further mixes the features of different modalities through the fully connected layer; the normalization layer, through normalization of the mean and variance.

4. The method for three-dimensional object detection in nighttime traffic scenes integrating depth cues according to claim 1, characterized in that: The step S4 is specifically as follows: First, the fused multi-scale features B are further extracted through the shallow feature extraction network and feature pyramid. F ,Then the further extracted features are fed into the universal 3D detection head to perform the 3D object detection task, completing the 3D object detection in night traffic scenes; Among them, the loss functions include: regression loss and classification loss; the regression loss uses the L1 loss function for bounding box regression, and the classification loss uses the Gaussian perception loss function for target classification.

Images