Motion compensation based target detection system
By employing motion compensation and radar depth guidance, the spatiotemporal misalignment and sparsity issues between the camera and radar in adverse weather and complex scenarios were resolved. This enabled the alignment and fusion of camera and radar features, thereby improving the accuracy and robustness of target detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HENAN UNIV OF SCI & TECH
- Filing Date
- 2026-03-20
- Publication Date
- 2026-07-03
AI Technical Summary
Existing technologies suffer from insufficient target detection accuracy due to spatiotemporal misalignment and sparsity issues between camera and radar modes in adverse weather conditions and complex scenarios, making it difficult to achieve effective information fusion and complementarity.
By employing motion compensation mechanisms and radar depth guidance, the system achieves alignment and complementary information exchange between camera and radar features through density processing of millimeter-wave radar point clouds and depth guidance and channel-level fusion combined with image features.
It significantly improves target detection accuracy under adverse weather conditions, achieves consistent alignment of cross-modal features and interactive fusion of fine-grained complementary information, thereby enhancing detection accuracy and robustness.
Smart Images
Figure CN122336736A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of target detection technology, and more specifically, relates to a target detection system based on motion compensation. Background Technology
[0002] Multimodal target perception has become an important direction for improving the accuracy and robustness of environmental perception, and has demonstrated wide value in applications such as autonomous driving and intelligent transportation. As two typical heterogeneous modalities, cameras can provide fine-grained texture and rich semantic information, but are prone to performance degradation under drastic changes in lighting, inclement weather, or long-distance scenarios. In contrast, millimeter-wave radar, although limited in spatial resolution, can provide stable radial distance and velocity measurements and has all-weather anti-interference capabilities. Therefore, the effective fusion of the two is widely regarded as the key to improving the robustness and accuracy of perception systems.
[0003] Some studies have improved the reliability of target detection by associating radar-detected targets with bounding boxes in images and designing weighted or sampling mechanisms to fuse the confidence information of both. However, this often depends on the quality of the initial bounding boxes or targets generated by a single modality; if there are missed or false detections, subsequent fusion is difficult to compensate for. Other studies have mapped radar point clouds to images or BEV spaces using coordinate projection and utilized convolution or cross-attention mechanisms to promote the effective fusion of intermodal information. However, cameras and radars have inherent physical heterogeneity, and relying solely on geometric projection is insufficient to eliminate the spatiotemporal misalignment between modalities. In dynamic environments or under occlusion interference, this deviation can easily cause feature matching ambiguity, further hindering the effective interaction and aggregation of complementary information between modalities. Previous studies have explored various algorithms to improve the fusion of complementary information from cameras and radar, such as dynamically allocating modal contributions through attention mechanisms or weighted fusion strategies to leverage their respective advantages in different scenarios. However, cameras and radars exhibit significant modal differences and signal-to-noise ratio fluctuations in different scenarios. The semantic granularity differences between radar point clouds and image features are not fully considered, and the reliability variations of different modalities in the channel dimension are ignored, which limits the model's generalization ability in real open scenarios. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a target detection system based on motion compensation. The system uses motion compensation mechanism and radar depth guidance to alleviate the inherent sparsity and spatiotemporal misalignment of radar point clouds, thereby significantly improving the target detection accuracy in adverse weather and complex scenarios.
[0005] To achieve the above-mentioned objectives, the motion-compensated target detection system of the present invention includes the following steps:
[0006] The system comprises a camera, an image feature extraction module, a millimeter-wave radar, a motion compensation module, a radar depth guidance module, an image BEV feature extraction module, a radar BEV feature extraction module, a BEV feature fusion module, and a target detection module, among which:
[0007] The camera is used to periodically acquire panoramic image data of the target area. And send it to the image feature extraction module, Indicates time;
[0008] The image feature extraction module is used to extract features from panoramic image data. Extracting image features And send it to the radar depth guidance module and the image BEV feature extraction module;
[0009] Millimeter-wave radar is used to periodically collect point cloud data in the environment. And send it to the motion compensation module;
[0010] The motion compensation module is used for point cloud data Motion compensation is performed on the dense point cloud data after compensation. The data is then sent to the radar depth guidance module and the radar BEV feature extraction module; the specific method for motion compensation is as follows:
[0011] S1: Set the number of backtracking frames ;
[0012] S2: Transfer point cloud data Projected onto the bird's-eye view grid space, the space occupancy map and velocity map are calculated separately in each grid cell;
[0013] S3: Traverse each grid cell. If its speed is lower than a preset threshold, it is determined to be a stationary cell; otherwise, the grid cell is determined to be a dynamic cell.
[0014] S4: For each dynamic unit obtained in step S3, map each point within it to the current time to obtain the compensated point cloud data. ;
[0015] S5: Determine if , This indicates the preset number of frames. If so, proceed to step S6; otherwise, proceed to step S7.
[0016] S6: Set the number of backtracking frames Return to step S3;
[0017] S7: Will Frame-compensated point cloud data and current frame point cloud data Spatial merging is performed to obtain dense point cloud data. ;
[0018] The radar depth guidance module is used to utilize dense point cloud data. Image features Perform depth guidance to obtain a depth map And send it to the image BEV feature extraction module;
[0019] Image BEV feature extraction module for depth map-based From image features BEV features were extracted from the image. And send it to the BEV feature fusion module;
[0020] The radar BEV feature extraction module is used to extract features from dense point cloud data. Radar BEV features were extracted. And send it to the BEV feature fusion module;
[0021] The BEV feature fusion module is used to process BEV features in images. and radar BEV characteristics The fusion process is performed, and the resulting fused features are sent to the target detection module.
[0022] The target detection module is used to obtain target detection results based on fused features.
[0023] This invention relates to a motion-compensated target detection system. A camera acquires panoramic image data of the target area, and an image feature extraction module extracts image features. A millimeter-wave radar acquires point cloud data. A motion compensation module compensates the point cloud data to obtain dense point cloud data. A radar depth guidance module uses the dense point cloud data to guide the image features to obtain a depth map. An image BEV feature extraction module extracts image BEV features from the image features based on the depth map. A radar BEV feature extraction module extracts radar BEV features from the dense point cloud data. A BEV feature fusion module fuses the image BEV features and the radar BEV features. A target detection module obtains the target detection result based on the fused features.
[0024] The present invention has the following benefits:
[0025] 1) Under adverse weather conditions, this invention utilizes motion compensation to obtain dense point cloud data and uses it as a depth prior to assist the camera branch, thereby achieving consistent alignment of cross-modal features and improving the accuracy of target detection.
[0026] 2) When fusing image BEV features and radar BEV features, this invention can adopt an adaptive fusion strategy based on channel-level determination. By using channel-level determination to adaptively learn the importance of features, fine-grained complementary information can be fused to improve the quality of fused features and further improve the target detection accuracy.
[0027] 3) Experiments have verified the effectiveness of this invention in various adverse weather conditions. The experimental results show that under normal weather conditions, the target detection accuracy can reach 62.9%, and the comprehensive detection score can reach 65.8%. Under adverse conditions such as rain, night, fog, and snow, the target detection accuracy can reach 62.4%, 56.7%, 58.6%, and 54.7%, respectively. Attached Figure Description
[0028] Figure 1 This is a structural diagram of a specific implementation of the target detection system based on motion compensation of the present invention;
[0029] Figure 2 This is a structural diagram of the image feature extraction module in this embodiment;
[0030] Figure 3 This is a flowchart of the motion compensation method in this invention;
[0031] Figure 4 This is a schematic diagram of single-frame motion compensation in this embodiment;
[0032] Figure 5 This is a structural diagram of the BEV feature fusion module in this embodiment;
[0033] Figure 6 This is a flowchart of gating fusion in this embodiment;
[0034] Figure 7 This is a comparison chart of the average accuracy of each configuration in the ablation experiment in this embodiment;
[0035] Figure 8 This is a comparison chart of the delay performance of each configuration in the ablation experiment in this embodiment. Detailed Implementation
[0036] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.
[0037] Example
[0038] Figure 1 This is a structural diagram of a specific implementation of the motion-compensated target detection system of the present invention. Figure 1As shown, the target detection system based on motion compensation and channel-level fusion of the present invention includes a camera 1, an image feature extraction module 2, a millimeter-wave radar 3, a motion compensation module 4, a radar depth guidance module 5, an image BEV feature extraction module 6, a radar BEV feature extraction module 7, a BEV feature fusion module 8, and a target detection module 9. Each module will be described in detail below.
[0039] Camera 1 is used to periodically acquire panoramic image data of the target area. And send it to image feature extraction module 2, Indicates the time.
[0040] Image feature extraction module 2 is used to extract features from panoramic image data. Extracting image features It is then sent to the radar depth guidance module 5 and the image BEV feature extraction module 6.
[0041] Figure 2 This is a structural diagram of the image feature extraction module in this embodiment. For example... Figure 2 As shown, in this embodiment, the image feature extraction module 2 includes an image preprocessing module 21, a feature extraction backbone network 22, and a feature pyramid network 23, wherein:
[0042] The image preprocessing module 21 is used to preprocess the original panoramic image data using preset operations (such as distortion removal and cropping), and then sends the preprocessed panoramic image data to the feature extraction backbone network 22.
[0043] The feature extraction backbone network 22 is used to extract initial image features from the preprocessed panoramic image data and send them to the feature pyramid network 23. In this embodiment, the feature extraction backbone network 22 adopts the ResNet-50 network.
[0044] Feature pyramid network 23 is used to further extract features from the initial image features to obtain image features. .
[0045] After processing by the image feature extraction module in this embodiment, the original panoramic image data is converted into image features containing rich texture and semantic information, and the extracted image features... While preserving high-level semantics, the FPN structure also ensures the ability to detect small targets.
[0046] Millimeter-wave radar 3 is used to periodically acquire point cloud data in the environment. And send it to motion compensation module 4.
[0047] Motion compensation module 4 is used for point cloud data Motion compensation is performed on the dense point cloud data after compensation. It is then sent to the radar depth guidance module 5 and the radar BEV feature extraction module 7.
[0048] Point cloud data can provide crucial information such as the radial distance, azimuth, relative velocity, and radar cross section of a target. In practical autonomous driving applications, due to hardware limitations, single-frame radar point clouds typically exhibit significant sparsity and are accompanied by clutter and multipath noise, making accurate pixel-level matching with high-resolution dense images difficult. Traditional point cloud accumulation methods often directly superimpose multiple frames of data, leading to severe feature ghosting when vehicles or targets are moving at high speeds. Therefore, to overcome the physical limitations of sparsity and accurately capture the motion trends of dynamic objects, this invention designs a motion compensation method to obtain compensated dense point cloud data. . Figure 3 This is a flowchart of the motion compensation method in this invention. For example... Figure 3 As shown, the specific method of motion compensation in this invention is as follows:
[0049] S301: Set the number of backtracking frames .
[0050] S302: Computation of space occupancy map and velocity map:
[0051] Point cloud data Projected onto the bird's-eye view (BEV) grid space, the space occupancy map and velocity map are calculated separately in each grid cell.
[0052] In this embodiment, the velocity used when constructing the velocity map is the median velocity of all radar points in the neighborhood of the grid cell. That is, median filtering is used instead of mean filtering, which can effectively filter out instantaneous velocity noise caused by multipath reflection and improve the robustness of velocity estimation.
[0053] S303: Filtering dynamic units:
[0054] Each grid cell is traversed. If its speed is lower than a preset threshold, it is determined to be a stationary cell (such as a building or curb) and no additional motion compensation is performed. Otherwise, the grid cell is determined to be a dynamic cell.
[0055] S304: Dynamic Element Mapping
[0056] For each dynamic unit obtained in step S303, map each point within it to the current time to obtain compensated point cloud data. .
[0057] The mapping function used in this embodiment is shown in the following equation:
[0058] ,
[0059] in, Representing point cloud data Points in the dynamic unit, Point Mapped to the current time virtual points, Point The speed of the dynamic unit in which it is located This indicates the time interval between the number of point clouds in two frames.
[0060] By using the above mapping method, not only is the density of the point cloud increased, but the spatial position shift caused by the time difference is also corrected.
[0061] S305: Determine if , This indicates the preset number of frames. If so, proceed to step S306; otherwise, proceed to step S307.
[0062] S306: Set back the frame number Return to step S302.
[0063] S307: Spatial merging yields dense point cloud data:
[0064] Will Frame-compensated point cloud data and current frame point cloud data Spatial merging is performed to obtain dense point cloud data. .
[0065] Figure 4 This is a schematic diagram of single-frame motion compensation in this embodiment. For example... Figure 4 As shown, first, the radar point cloud from the previous frame is... Projected onto the bird's-eye view (BEV) grid space, the space occupancy map and velocity map are calculated within each grid cell. Then, dynamic cells are selected based on velocity, and the points in the dynamic cells are mapped to their current time. Then compared with the radar point cloud of the current frame. Spatial merging is performed to form dense point cloud data with spatiotemporal alignment and significantly improved density. By utilizing velocity and occupancy information from previous radar point cloud data, virtual points are predicted and generated for the current frame, thereby increasing point cloud density and reducing the impact of motion inaccuracies, effectively mitigating the inherent sparsity problem of radar point clouds.
[0066] Radar depth guidance module 5 is used to utilize point cloud data Image features Perform depth guidance to obtain a depth map And send it to the image BEV feature extraction module 6.
[0067] In monocular vision perception, due to the lack of absolute scale information, cameras often struggle to accurately determine the absolute distance of objects. Dense point cloud data with motion compensation provides sparse yet precise absolute depth geometric constraints. Therefore, this invention uses dense point cloud data to calibrate the camera's depth estimation. In this embodiment, the specific method of depth guidance by the radar depth guidance module 5 is as follows:
[0068] First, the dense point cloud data Projected onto a two-dimensional image plane to form a sparse depth map To fill the gaps between sparse points, max pooling is used to refine the sparse depth map. A densification process is performed to obtain a dense depth prior that matches the image feature size. Then, the dense depth prior is used. and image features Perform channel splicing, and then apply the splicing features. The data is input into a depth prediction head to obtain a depth map. .
[0069] In this embodiment, the depth prediction head includes convolutional layers and softmax layers, wherein the convolutional layers are used to process the concatenated features. Convolutional operations are performed to adaptively fuse visual texture and radar geometric information. The softmax layer is used to process the convolutional features using the softmax function to obtain the depth map. .
[0070] Image BEV feature extraction module 6 is used for depth map-based features. From image features BEV features were extracted from the image. And send it to BEV feature fusion module 8.
[0071] In this embodiment, the image BEV feature extraction module 6 first uses the LSS (Lift-Splat-Shoot) mechanism to extract 2D image features. The image is elevated to 3D space and then projected onto the BEV plane to generate BEV features. .
[0072] Radar BEV feature extraction module 7 is used to extract features from point cloud data. Radar BEV features were extracted. And send it to BEV feature fusion module 8.
[0073] BEV feature fusion module 8 is used to process BEV features of the image. and radar BEV characteristics The fusion process is performed, and the resulting fused features are sent to the target detection module 9.
[0074] To improve the quality of fused features, this embodiment designs a BEV feature fusion module based on reliability gating. Figure 5 This is a structural diagram of the BEV feature fusion module in this embodiment. For example... Figure 5 As shown, in this embodiment, the BEV feature fusion module 8 includes an attribute alignment module 81, a reliability gating calculation module 82, and a gating fusion module 83, wherein:
[0075] Attribute alignment module 81 is used to align the BEV features of the image respectively. and radar BEV characteristics Mapping to a structured space with fixed attribute channels, we obtain the attribute features and reliability scores of the corresponding channels in each BEV feature. We then denote the image BEV feature attribute features as... Image BEV feature reliability score Record the radar BEV characteristic attributes. Radar BEV characteristic reliability score ,in Indicates the feature size, The number of attribute channels is indicated, and then the reliability score of each BEV feature is sent to the reliability gating calculation module 82, and the attribute features of each BEV feature are sent to the gating fusion module 83.
[0076] The reliability gating calculation module 82 is used to calculate the reliability score based on the BEV features of the image. and radar BEV characteristic reliability score Calculate the reliability gating for each attribute channel And send it to the gating fusion module 83 for reliability gating. The calculation formula is as follows:
[0077] ,
[0078] in, , These represent the image BEV feature reliability scores, respectively. and radar BEV characteristic reliability score Medium attribute channel The reliability score matrix, , This indicates the goal of finding the norm.
[0079] The gated fusion module 83 is used for reliability-based gated control. BEV features of the image Attributes and characteristics and radar BEV characteristics Attributes and characteristics By performing fusion, fusion characteristics are obtained. . Figure 6 This is a flowchart of gating fusion in this embodiment. For example... Figure 6 As shown, the specific steps of gating fusion in this embodiment include:
[0080] S601: Attribute Channel Number .
[0081] S602: Determine if it is an attribute channel Reliability gating , This indicates a preset threshold. If so, proceed to step S603; otherwise, proceed to step S604.
[0082] S603: Retains high reliability attributes as a fusion feature:
[0083] Based on the reliability score of BEV features in the image and radar BEV characteristic reliability score Medium attribute channel Reliability score matrix , High-reliability attribute features are selected and retained to obtain attribute channels. Fusion characteristics The expression is as follows:
[0084] ,
[0085] in, , , Representing attribute channels Fusion features, image BEV feature attributes Radar BEV characteristic attributes median coordinate The attributes and characteristics, , Representing attribute channels Reliability score matrix , median coordinate The reliability score.
[0086] Then proceed to step S608.
[0087] S604: Calculate the baseline characteristics:
[0088] Attribute channels are generated by weighted averaging based on reliability scores. The baseline features This helps to further reduce random noise and improve measurement stability. The specific calculation formula is as follows:
[0089] ,
[0090] in, This represents the preset minimum value.
[0091] S605: Determine if , This indicates calculating the similarity. This indicates a preset similarity threshold. If so, proceed to step S606; otherwise, proceed to step S607.
[0092] In terms of similarity, when attribute channels For scalar channels (such as height and size), similarity can be achieved using the absolute difference of attribute features. For vector channels (such as velocity), cosine similarity can be used for similarity.
[0093] S606: Use the baseline feature as the fusion feature:
[0094] When attribute similarity If the value is higher than a preset threshold, it indicates that the camera and millimeter-wave radar have consistent observations of the same physical property, therefore the baseline feature is set. As an attribute channel Fusion characteristics ,Right now Then proceed to step S608.
[0095] S607: Complementary residual processing yields fused features:
[0096] When attribute similarity Below a preset threshold, although the observations from the camera and millimeter-wave radar are inconsistent, since their reliability is comparable, this inconsistency is not noise but rather contains complementary information. For example, millimeter-wave radar is extremely sensitive to radial velocity, while the camera is better at capturing lateral tangential motion; their velocity vector observations may be inconsistent, but combined, they can describe the complete motion. Therefore, this embodiment introduces a joint reliability term. As a scaling factor, different residual compensation strategies are employed for different types of channels.
[0097] When attribute channel For scalar attributes, the difference residuals are directly calculated and superimposed on the baseline features. The above yields fusion features. :
[0098] ,
[0099] in, This represents the difference residual.
[0100] When attribute channel As vector attributes, since millimeter-wave radar excels at radial velocity measurement while cameras excel at lateral motion capture, an orthogonal residual strategy is employed, resulting in its fused features. The calculation method is as follows:
[0101] First, calculate the attribute difference. :
[0102] ,
[0103] Then the attribute difference Projected onto reference feature Extract orthogonal residuals from orthogonal subspaces. :
[0104] ,
[0105] The superscript T indicates transpose.
[0106] Then proceed to step S608.
[0107] S608: Determine if If yes, proceed to step S609; otherwise, the fusion process ends.
[0108] S609: Order Return to step S602.
[0109] As described above, the BEV feature fusion module in this embodiment can achieve dynamic comparison and selective fusion of features along the attribute channel dimension, while modeling the reliability, consistency, and complementarity between attribute channels. Specifically, if the reliability difference between two modalities in a certain attribute channel exceeds a preset threshold, only the feature of the attribute modality with higher reliability is retained; if both are reliable in that channel, their similarity is further evaluated: if the similarity is higher than the threshold, a reliability-based weighted average is performed; if the consistency is lower than the threshold, complementary information is inferred, and the feature differences are used to enhance complementarity, thereby obtaining fused features.
[0110] The target detection module 9 is used to obtain target detection results based on the fusion features.
[0111] In this embodiment, the target detection module adopts the CenterPoint detection architecture, which uses convolutional layers and fully connected layers to regress parameters such as the target's center point heatmap, 3D size, orientation angle, and velocity, and finally outputs the target detection results.
[0112] Since the target detection system of this invention contains a deep neural network, it is necessary to train the target detection system with training samples in practical applications. In order to assign clear physical meaning to intermediate features during training and guide the network to learn the correct physical properties, this embodiment adopts a multi-task joint loss function. To optimize this, the loss function consists of three parts: the main detection loss. Multi-channel reliability monitoring loss and attribute feature regression loss Among them, the main detection loss The settings can be customized according to actual needs. This embodiment follows the standard settings of CenterPoint, including heatmap loss for key point localization and L1 loss for bounding box regression. Multi-channel reliability supervision loss. The reliability of adaptive learning modes in a supervised system is calculated using the following formula:
[0113] ,
[0114] in, It is a Gaussian heatmap generated based on the truth center. This indicates the calculation of Gaussian Focal Loss.
[0115] Attribute feature regression loss To force attribute features to learn real physical attributes (such as velocity and size), a sparse L1 regression loss is used, calculated as follows:
[0116] ,
[0117] in, Indicates the actual quantity of the target. Indicates the first One goal, , , Representing image attribute features respectively and radar attribute characteristics The Middle The attribute characteristics of each target Indicates the first The true attribute characteristics of each target.
[0118] Attribute feature regression loss This ensures that the attribute alignment module outputs not only abstract feature vectors, but also attribute values with clear physical meanings, thereby making subsequent channel-level fusion logic (such as difference calculation and orthogonal projection) physically interpretable.
[0119] Total loss function The calculation formula is as follows:
[0120] ,
[0121] in, , These represent the preset weighting coefficients.
[0122] By adopting the above total loss function, the target detection system of this invention is not only constrained at the output end, but also incorporates explicit physical guidance in the feature extraction and fusion process, which effectively improves the interpretability and generalization ability of the overall framework.
[0123] To better illustrate the technical effects of the present invention, specific examples are used to experimentally verify the invention. To verify the present invention (denoted as C...) 2 F) Performance advantages in complex scenarios: In this embodiment, the nuscenes-C dataset is used to compare and test the present invention with seven baseline models in four typical severe weather environments: rain, night, fog, and snow. The baseline models include:
[0124] CRN, see the document "Kim Y, Shin J, Kim S, et al. Crn: Camera radar net for accurate, robust, efficient 3d perception[C] / / Proceedings of the IEEE / CVFInternational Conference on Computer Vision. 2023: 17615-17626.";
[0125] RCTrans, see the document "Li Y, Yang Y, Lei Z. Rctrans: Radar-cameratransformer via radar densifier and sequential decoder for 3d object detection[C] / / Proceedings of the AAAI Conference on Artificial Intelligence.2025, 39(5): 5048-5056.";
[0126] CRT-Fusion, see the document "Kim J, Seong M, Choi J W. Crt-fusion: Camera, radar, temporal fusion using motion information for 3d object detection[J]. Advances in Neural Information Processing Systems, 2024, 37: 108625-108648.";
[0127] RaCFormer, see the document "Chu
[0128] CRAB, see the document "Lee IJ, Hwang S, Kim Y, et al. Crab: Camera-radarfusion for reducing depth ambiguity in backward projection based viewtransformation[C] / / 2025 IEEE International Conference on Robotics andAutomation (ICRA). IEEE, 2025: 15719-15725.";
[0129]
[0130] Table 1
[0131] Table 1 is a performance comparison chart of the present invention and the baseline model in this embodiment. As shown in Table 1, C 2 F demonstrates the best overall performance. Specifically, C 2F achieves 65.8% NDS and 62.9% mAP, both outperforming all baseline methods. Compared to pure vision methods lacking geometric measurements (such as Gaussian LSS), this invention significantly reduces mATE by introducing radar depth guidance; compared to pure radar methods lacking texture information (such as RadarDistill), this invention greatly improves classification accuracy by fusing camera features. Furthermore, thanks to fine-grained channel-level fusion, this invention also achieves the lowest level of mAVE, demonstrating the model's advantage in capturing motion information of dynamic objects.
[0132]
[0133] Table 2
[0134]
[0135] Table 3
[0136]
[0137] Table 4
[0138]
[0139] Table 5
[0140] Tables 2-5 show the performance comparison results of the present invention with the baseline model in four scenarios: rainy, low light, foggy, and snowy weather. The results show that the present invention exhibits more comprehensive and robust adaptability in the vast majority of scenarios. Detailed analysis follows:
[0141] (1) In rainy scenes, considering the visual depth estimation bias caused by road surface reflection and raindrop occlusion, C 2 F injects precise physical ranging information through a radar depth guidance module, effectively correcting the geometric projection error of the camera branch. Experimental results show that this invention achieves 62.4% mAP and 65.5% NDS in rainy weather, significantly outperforming methods that do not utilize radar geometric priors.
[0142] (2) In nighttime environments, the present invention still leads with 56.7% mAP, which is a huge improvement over the pure visual baseline and outperforms other fusion methods. To address the problem of visual texture loss caused by nighttime or insufficient lighting, the CWAF module of the present invention adaptively reduces the weight of camera features through a reliability gating mechanism, relying more on radar features for detection.
[0143] (3) In foggy conditions, to address the visual feature blurring caused by fog scattering, the MC-VPA module of this invention enhances the radar's penetration capability and point cloud density through multi-frame radar point cloud accumulation. Experimental results show that this invention maintains extremely high stability in foggy scenarios, achieving 58.6% mAP and 60.2% NDS, significantly outperforming the baseline model severely affected by fog interference.
[0144] (4) In snowy conditions, the MC-VPA module effectively filters out random environmental noise caused by snowflake obstruction and multipath reflection, generating enhanced radar characterization by utilizing temporal motion consistency. Experimental results show that the present invention achieves 54.7% mAP and 57.8% NDS in snowy conditions, and reduces mAVE to 0.207, demonstrating its robustness under dynamic interference.
[0145] In summary, the core reason why this invention maintains high accuracy in scenarios such as occlusion, lighting changes, and background interference lies in the following: MC-VPA effectively overcomes the inherent sparsity of radar and suppresses random environmental noise by utilizing the temporal motion consistency of multi-frame point clouds, thus constructing an enhanced radar representation; Radar Depth Guidance injects radar physical ranging information as a strong geometric prior into the visual branch, correcting depth deviations under poor visual conditions and ensuring BEV projection accuracy; CWAF achieves accurate capture of complementary information and adaptive suppression of invalid features through channel-level decision-making to evaluate and dynamically reconstruct cross-modal feature weights in real time. In actual autonomous driving and safety perception tasks, this model can cope with non-ideal visual conditions through multi-modal fusion mechanisms and meet the real-time requirements of edge devices with its lightweight design, demonstrating broad engineering application potential.
[0146] Next, to analyze the impact of each model of the present invention on detection accuracy and latency, this embodiment conducted an ablation experiment and designed the following three sets of comparative configurations:
[0147] (i)C 2 F w / o MC-VPA: This model removes the motion-compensated virtual point cloud enhancement component from the radar branch and only receives a single frame of raw radar point cloud as input;
[0148] (ii) C 2 F w / o Radar Depth Guidance: This model removes the radar depth prior injection mechanism, and the camera branch relies solely on monocular image features to regress the depth distribution when 2D features are lifted into the BEV space.
[0149] (iii) C 2F w / o CWAF: This model adopts a naive feature stitching strategy, which directly stitches the aligned camera and radar BEV features in the channel dimension for detection;
[0150] (iv)C 2 F: The complete design of this invention.
[0151] Figure 7 This is a comparison chart of the average accuracy of each configuration in the ablation experiment of this embodiment. According to... Figure 7 It can be observed intuitively that:
[0152] (1) Under different confidence threshold conditions, C 2 The full F model significantly outperformed other variant models in terms of mAP. Particularly when the confidence threshold was increased to a more stringent 0.7, the performance of other variant models declined significantly, while the model of this invention exhibited the slowest decay trend, maintaining an mAP as high as 58.2%, demonstrating its robustness under high confidence requirements.
[0153] (2) C 2 The F w / o MC-VPA, by removing the motion compensation module and relying solely on a single frame of sparse radar point cloud, suffers a significant decrease in mAP. The lack of temporal motion compensation makes spatiotemporal geometric alignment of dynamic objects difficult, and the sparse point cloud cannot provide sufficient geometric prior, resulting in compromised detection accuracy.
[0154] (3) C 2 The F w / o Radar Depth Guidance lags behind the full model by more than 4% across all threshold settings. This is because removing radar depth guidance degenerates the camera branch into an ill-posed monocular depth estimate, leading to projection drift in the BEV space. Even with high classification confidence, predicted bounding boxes are often classified as false positives due to inadequate IoU.
[0155] (4) C 2 F w / o CWAF employs a simple feature stitching strategy. Its performance is close to that of this invention at low confidence thresholds, but the gap widens at high confidence thresholds. This indicates that simple stitching cannot resolve semantic conflicts between radar and camera (e.g., radar detects a target but image confidence is low due to occlusion). CWAF effectively preserves complementary information through a channel-level decision mechanism, preventing correct detections from being filtered out.
[0156] Figure 8 This is a comparison chart of the delay performance of each configuration in the ablation experiment of this embodiment. According to... Figure 8 It can be observed intuitively that:
[0157] (1) At a low IoU threshold (0.3), C 2The latency of the full F model is approximately 82ms, while that of the w / o MC-VPA variant model is only 62ms. This indicates that during the feature extraction and preprocessing stages, the MC-VPA module needs to perform a large number of point cloud indexing and matrix transformation operations to achieve spatiotemporal alignment of multi-frame point clouds. Despite this increased computational overhead, it is necessary to address radar sparsity.
[0158] (2) When the IoU threshold rises to 0.5, the variant with the adaptive fusion module removed (w / o CWAF) has the closest latency to the full model. This is attributed to the efficient design of the CWAF module, which mainly relies on basic linear operations such as vector dot products and does not introduce a computationally intensive global attention mechanism. Therefore, while significantly improving the fusion quality, it adds almost no additional computational burden.
[0159] (3) As the IoU threshold is further increased to 0.7, the variant without radar depth guidance has a lower latency than the full model because it eliminates the projection process from the 3D point cloud to the 2D plane. Although the full model takes slightly longer at this stage, it gains a significant accuracy advantage at high thresholds, proving that the present invention achieves a good balance between computational efficiency and detection performance;
[0160] (4) The complete model achieves a balance between detection accuracy and computational efficiency at different IoU thresholds through the synergistic effect of its components. Although the latency increases with the increase of the IoU threshold, this increase is a necessary price to pay for improving detection performance, reflecting the depth and comprehensiveness of the model in handling complex tasks, especially in application scenarios that require high detection accuracy.
[0161] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.
Claims
1. A motion-compensation-based object detection system, characterized by It includes a camera, an image feature extraction module, a millimeter-wave radar, a motion compensation module, a radar depth guidance module, an image BEV feature extraction module, a radar BEV feature extraction module, a BEV feature fusion module, and a target detection module, among which: The camera is used to periodically acquire panoramic image data of the target area. And send it to the image feature extraction module, Indicates time; The image feature extraction module is configured to extract image features from the surround view image data and send to the radar depth guidance module and the image BEV feature extraction module and send to the radar depth guidance module and the image BEV feature extraction module Millimeter wave radar is used to periodically collect point cloud data in the environment and sent to the motion compensation module; The motion compensation module is used for motion compensation on the point cloud data , and the compensated dense point cloud data is sent to the radar depth guidance module and the radar BEV feature extraction module; the specific method of motion compensation is as follows: S1: let numBacktraceFrames ; S2: Project the point cloud data into the bird's eye view grid space, and calculate the spatial occupancy map and velocity map in each grid cell respectively; S3: Traverse each grid cell. If its speed is lower than a preset threshold, it is determined to be a stationary cell; otherwise, the grid cell is determined to be a dynamic cell. S4: For each dynamic unit obtained by screening in step S3, each point is mapped to the current time to obtain compensated point cloud data ; S5: Determine if , This indicates the preset number of frames. If so, proceed to step S6; otherwise, proceed to step S7. S6: Let the number of backtracking frames , return to step S3; S7: Will Frame-compensated point cloud data and current frame point cloud data Spatial merging is performed to obtain dense point cloud data. ; The radar depth guidance module is used to utilize dense point cloud data. Image features Perform depth guidance to obtain a depth map And send it to the image BEV feature extraction module; The image BEV feature extraction module is used for extracting the image BEV feature based on the depth map The image feature is extracted from the image The image BEV feature is obtained by extraction and sent to the BEV feature fusion module; The radar BEV feature extraction module is configured to extract radar BEV features from the dense point cloud data The radar BEV features are extracted and sent to the BEV feature fusion module; The BEV feature fusion module is configured to fuse the image BEV features and the radar BEV features and send the obtained fused features to the target detection module. The target detection module is used to obtain target detection results based on fused features.
2. The object detection system of claim 1, wherein, The image feature extraction module includes an image preprocessing module, a feature extraction backbone network, and a feature pyramid network, wherein: The image preprocessing module is used to preprocess the original panoramic image data using preset operations, and then sends the preprocessed panoramic image data to the feature extraction backbone network. The feature extraction backbone network is used to extract initial image features from the preprocessed panoramic image data and send them to the feature pyramid network; Feature pyramid networks are used to further extract features from the initial image features, resulting in image features. .
3. The target detection system according to claim 1, characterized in that, The mapping function in step S4 of the motion compensation module is as follows: ， wherein, representing point cloud data points in a dynamic cell, representing a point mapped to a current time instant, a virtual point, representing a point velocity of the dynamic cell where the point represents a time interval between the number of point clouds of two frames.
4. The object detection system of claim 1, wherein, The specific method of depth guidance in the radar depth guidance module is as follows: First, the dense point cloud data Projected onto a two-dimensional image plane to form a sparse depth map Max pooling is used to process sparse depth maps. A densification process is performed to obtain a dense depth prior that matches the image feature size. Then, the dense depth prior is... and image features Perform channel splicing, and then apply the splicing features. The data is input into a depth prediction head to obtain a depth map. .
5. The target detection system according to claim 4, characterized in that, The depth prediction head includes convolutional layers and softmax layers, where the convolutional layers are used to concatenate features. The convolution operation is performed, and the softmax layer is used to process the convolutional features using the softmax function to obtain the depth map. .
6. The target detection system according to claim 1, characterized in that, The image BEV feature extraction module uses the LSS mechanism to extract 2D image features. The image is elevated to 3D space and then projected onto the BEV plane to generate BEV features. .
7. The target detection system according to claim 1, characterized in that, The feature fusion module includes an attribute alignment module, a reliability gating calculation module, and a gating fusion module, wherein: The attribute alignment module is used to separate the BEV features of the image. and radar BEV characteristics Mapping to a structured space with fixed attribute channels, we obtain the attribute features and reliability scores of the corresponding channels in each BEV feature. We then denote the image BEV feature attribute features as... Image BEV feature reliability score Record the radar BEV characteristic attributes. Radar BEV characteristic reliability score ,in Indicates the feature size, The number of attribute channels is indicated, and then the reliability score of each BEV feature is sent to the reliability gating calculation module, and the attribute features of each BEV feature are sent to the gating fusion module. The reliability gating calculation module is used to calculate the reliability score based on the BEV features of the image. and radar BEV characteristic reliability score Calculate the reliability gating for each attribute channel And send it to the gating fusion module for reliability gating. The calculation formula is as follows: ， in, , These represent the image BEV feature reliability scores, respectively. and radar BEV characteristic reliability score Medium attribute channel The reliability score matrix, , This indicates the process of finding the norm; The gating fusion module is used for reliability gating. Image attribute features and radar attribute characteristics By performing fusion, fusion characteristics are obtained. The specific steps of gating fusion include: 1) Set the attribute channel number ; 2) Determine if it is an attribute channel Reliability gating , This indicates a preset threshold; if so, proceed to step 3; otherwise, proceed to step 4. 3) Based on the reliability score of BEV features in the image and radar BEV characteristic reliability score Medium attribute channel Reliability score matrix , High-reliability attribute values are selected and retained to obtain attribute channels. Fusion characteristics The expression is as follows: ， in, , , Representing attribute channels Fusion features, image BEV feature attributes Radar BEV characteristic attributes median coordinate The attribute value, , Representing attribute channels Reliability score matrix , median coordinate Reliability score; Then proceed to step 8); 4) Generate attribute channels by weighted averaging based on reliability scores. The baseline features : ， in, This indicates the preset minimum value; 5) Determine if , This indicates calculating the similarity. This indicates a preset similarity threshold; if so, proceed to step 6; otherwise, proceed to step 7. 6) Reference features As an attribute channel Fusion characteristics ,Right now Then proceed to step 8). 7) Calculate the joint reliability term ; When attribute channel For scalar attributes, the fused features are obtained using the following formula. : ， in, Indicates the difference residual; When attribute channel For vector attributes, first calculate the attribute difference. : ， Then the attribute difference Projected onto reference feature Extract orthogonal residuals from orthogonal subspaces. : ； The superscript T indicates transpose; Then proceed to step 8); 8) Determine whether If yes, proceed to step S9; otherwise, the fusion ends. 9) Order (Return to step 2).
8. The target detection system according to claim 1, characterized in that, The formula for calculating the loss function during training of the object detection system is as follows: ， in, , These represent the preset weighting coefficients. Indicates the main detection loss. The multi-channel reliability monitoring loss is represented by the following formula: ， in, It is a Gaussian heatmap generated based on the truth center. This indicates the calculation of Gaussian focusing loss; The regression loss for attribute features is calculated using the following formula: ， in, Indicates the actual quantity of the target. Indicates the first One goal, , , Representing image attribute features respectively and radar attribute characteristics The Middle The attribute characteristics of each target Indicates the first The true attribute characteristics of each target.