A method and system for fusing millimeter wave radar data and camera data
By adopting a top-down fusion model, using residual networks and max pooling encoders for multi-scale feature extraction, and combining a feature pyramid fusion module, the problems of high model complexity and poor fusion effect in existing technologies are solved, and high-precision target recognition is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-01
- Publication Date
- 2026-03-03
AI Technical Summary
In existing technologies, the data fusion methods for millimeter-wave radar and cameras suffer from problems such as high model complexity, difficulty in fully integrating low-level details and high-level semantic knowledge, and susceptibility to weather conditions.
A top-down fusion approach is adopted, which guides low-level texture information through high-level semantic knowledge, uses residual networks and max pooling encoders for multi-scale feature extraction, and combines a feature pyramid fusion module for data fusion.
It reduces model complexity, improves the perceptual accuracy of target recognition, and enhances the data fusion effect in different environments.
Smart Images
Figure CN114200441B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data fusion technology, and in particular to a method and system for fusing millimeter-wave radar data and camera data. Background Technology
[0002] Autonomous vehicles rely on the collaborative efforts of artificial intelligence, computer vision, radar, monitoring devices, and global positioning systems to enable computers to operate motor vehicles automatically and safely without any human intervention.
[0003] Key technologies for autonomous vehicles fall into four categories: environmental perception, precise positioning, path planning, and drive-by-wire execution. Among these, environmental perception serves as the data foundation for other key technologies. Environmental perception technologies primarily include target state perception and target recognition. Since autonomous vehicles need to make different decisions based on the types of surrounding objects, target recognition is a core technology in the field of autonomous driving. Due to the complexity of the environment, a single sensor or multiple homogeneous sensors cannot fully perceive the situation around the vehicle; multiple heterogeneous sensors need to be combined for joint perception. Currently, most technologies employ fusion of LiDAR and cameras, or fusion of LiDAR and millimeter-wave radar. However, LiDAR is costly and easily affected by weather conditions.
[0004] Regarding millimeter-wave radar and camera fusion, existing technologies use a simple merging of millimeter-wave radar channels and image channels for data fusion. This may result in the loss of multi-scale features from different images. Furthermore, achieving high accuracy inevitably involves too many redundant fusion structures, significantly increasing model complexity. These methods all belong to bottom-up fusion structures, which, due to a lack of information from high-level semantics, struggle to fully fuse low-level details. As fusion progresses, feature maps typically become increasingly smaller, making it difficult to recover and utilize high-level semantic knowledge from high-resolution images. Moreover, the increasing number of auxiliary structures added to the fusion model to improve the results greatly increases the difficulty of model training. Summary of the Invention
[0005] In view of this, the present invention provides a method and system for fusing millimeter-wave radar data and camera data. It adopts a top-down fusion mode, in which the data from millimeter-wave radar and camera are fused by guiding low-level texture information through high-level semantic knowledge, thereby improving the perception accuracy of target recognition.
[0006] To achieve the above objectives, the present invention provides the following solution:
[0007] A method for fusing millimeter-wave radar data and camera data, comprising:
[0008] Acquire millimeter-wave radar point cloud data and camera image data;
[0009] The millimeter-wave radar point cloud data is converted to obtain radar image data of the same size as the camera image data.
[0010] Multi-scale feature extraction is performed on the camera image data to obtain camera image feature data; multi-scale feature extraction is performed on the radar image data to obtain radar image feature data;
[0011] The camera image feature data and the radar image feature data are fused from top to bottom to obtain fused data;
[0012] The fused data is decoded to obtain fused image data.
[0013] Preferably, the step of converting the millimeter-wave radar point cloud data to obtain radar image data of the same size as the image data includes:
[0014] The millimeter-wave radar point cloud data in the three-dimensional coordinate system is transformed and mapped onto the camera image data in the two-dimensional coordinate system to obtain two-dimensional radar data.
[0015] The radar image data is generated based on the two-dimensional radar data.
[0016] Preferably, the step of performing multi-scale feature extraction on the camera image data to obtain camera image feature data; and performing multi-scale feature extraction on the radar image data to obtain radar image feature data, includes:
[0017] Multi-scale feature extraction is performed on the camera image data based on the residual network encoder to obtain the camera image feature data;
[0018] Multi-scale feature extraction is performed on the radar image data based on a max-pooling encoder to obtain the radar image feature data.
[0019] Preferably, the step of fusing the camera image feature data and the radar image feature data from top to bottom to obtain fused data specifically involves:
[0020] The camera image feature data and the radar image feature data are fused using a top-down feature pyramid fusion module to obtain the fused data.
[0021] The present invention also provides a system for fusing millimeter-wave radar data and camera data, comprising:
[0022] The data acquisition module acquires millimeter-wave radar point cloud data and camera image data;
[0023] The data conversion module converts the millimeter-wave radar point cloud data to obtain radar image data of the same size as the camera image data;
[0024] The feature extraction module performs multi-scale feature extraction on the camera image data to obtain camera image feature data; and performs multi-scale feature extraction on the radar image data to obtain radar image feature data.
[0025] The data fusion module fuses the camera image feature data and the radar image feature data from top to bottom to obtain fused data;
[0026] The decoding module decodes the fused data to obtain fused image data.
[0027] Preferably, the data conversion module includes:
[0028] The coordinate transformation unit performs coordinate transformation on the millimeter-wave radar point cloud data in the three-dimensional coordinate system and maps it onto the camera image data in the two-dimensional coordinate system to obtain two-dimensional radar data.
[0029] The data generation unit generates the radar image data based on the two-dimensional radar data.
[0030] Preferably, the feature extraction module includes:
[0031] The camera data feature extraction unit performs multi-scale feature extraction on the camera image data based on the residual network encoder to obtain the camera image feature data;
[0032] The radar data feature extraction unit performs multi-scale feature extraction on the radar image data based on a maximum pooling encoder to obtain the radar image feature data.
[0033] Preferably, the data fusion module specifically comprises:
[0034] The camera image feature data and the radar image feature data are fused using a top-down feature pyramid fusion module to obtain the fused data.
[0035] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0036] This invention relates to a method and system for fusing millimeter-wave radar data and camera data. First, the millimeter-wave radar point cloud data is converted to obtain radar image data of the same size as the camera image data. Then, multi-scale feature extraction is performed on both the camera image data and the radar image data to obtain radar image feature data. Next, the camera image feature data and the radar image feature data are fused from top to bottom to obtain fused data. Finally, the fused data is decoded to obtain fused image data. This invention employs a top-down fusion mode, fusing millimeter-wave radar and camera data at multiple scales through feature extraction. This not only reduces model complexity but also improves fusion accuracy, further enhancing the perception accuracy of target recognition. Attached Figure Description
[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0038] Figure 1 This is a flowchart of the method for fusing millimeter-wave radar data and camera data according to the present invention;
[0039] Figure 2 This is a structural diagram of the millimeter-wave radar data and camera data fusion system of the present invention;
[0040] Figure 3 This is a structural diagram of the residual network encoder, max pooling encoder, and feature pyramid fusion module of the present invention.
[0041] Symbol explanation: 1-Data acquisition module, 2-Data conversion module, 3-Feature extraction module, 4-Data fusion module, 5-Decoding module. Detailed Implementation
[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0043] The purpose of this invention is to provide a method and system for fusing millimeter-wave radar data and camera data. It adopts a top-down fusion mode, in which the data from millimeter-wave radar and camera are fused by guiding low-level texture information through high-level semantic knowledge, thereby improving the perception accuracy of target recognition.
[0044] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0045] Figure 1 This is a flowchart of the millimeter-wave radar data and camera data fusion method of the present invention. As shown in the figure, the present invention provides a method for fusion of millimeter-wave radar data and camera data, including:
[0046] Step S1: Acquire millimeter-wave radar point cloud data and camera image data. The millimeter-wave radar point cloud data is obtained based on millimeter-wave radar, and the camera image data is obtained by capturing images with a camera.
[0047] Step S2: Convert the millimeter-wave radar point cloud data to obtain radar image data of the same size as the camera image data.
[0048] Specifically, step S2 is as follows:
[0049] The millimeter-wave radar point cloud data in the three-dimensional coordinate system is transformed and mapped onto the camera image data in the two-dimensional coordinate system to obtain two-dimensional radar data; the radar image data is generated based on the two-dimensional radar data.
[0050] A point in a three-dimensional coordinate system can be found to a corresponding pixel in the camera image data. The millimeter-wave radar point cloud data is transformed into the pixel coordinate system of the camera image data. The transformed millimeter-wave radar point cloud data is then used to generate the radar image data, which has the same size as the camera image data.
[0051] Step S3: Perform multi-scale feature extraction on the camera image data to obtain camera image feature data; perform multi-scale feature extraction on the radar image data to obtain radar image feature data.
[0052] Further, step S3 includes:
[0053] Step S31: Perform multi-scale feature extraction on the camera image data based on the residual network encoder to obtain the camera image feature data.
[0054] Step S32: Perform multi-scale feature extraction on the radar image data based on the max pooling encoder to obtain the radar image feature data.
[0055] like Figure 3As shown, in this embodiment, the residual network encoder includes a four-layer structure, which are defined as a first residual network encoder, a second residual network encoder, a third residual network encoder, and a fourth residual network encoder, respectively; the max pooling encoder includes a four-layer structure, which are defined as a first max pooling encoder, a second max pooling encoder, a third max pooling encoder, and a fourth max pooling encoder, respectively.
[0056] The first residual network encoder extracts features from the camera image data at a 1 / 4 scale to obtain a first camera feature image; the second residual network encoder extracts features from the first camera feature image at a 1 / 8 scale to obtain a second camera feature image; the third residual network encoder extracts features from the second camera feature image at a 1 / 16 scale to obtain a third camera feature image; and the fourth residual network encoder extracts features from the third camera feature image at a 1 / 32 scale to obtain a fourth camera feature image. The camera image feature data includes the first camera feature image, the second camera feature image, the third camera feature image, and the fourth camera feature image.
[0057] The first max-pooling encoder extracts features from the radar image data at a 1 / 4 scale to obtain a first radar feature image; the second max-pooling encoder extracts features from the first radar feature image at a 1 / 8 scale to obtain a second radar feature image; the third max-pooling encoder extracts features from the second radar feature image at a 1 / 16 scale to obtain a third radar feature image; and the fourth max-pooling encoder extracts features from the third radar feature image at a 1 / 32 scale to obtain a fourth radar feature image. The radar image feature data includes the first radar feature image, the second radar feature image, the third radar feature image, and the fourth radar feature image.
[0058] Step S4: The camera image feature data and the radar image feature data are fused from top to bottom to obtain fused data.
[0059] Preferably, such as Figure 3 As shown, the camera image feature data and the radar image feature data are fused based on the top-down feature pyramid fusion module to obtain the fused data.
[0060] The feature pyramid fusion module includes a four-layer structure, which are defined from top to bottom as the first fusion layer, the second fusion layer, the third fusion layer, and the fourth fusion layer.
[0061] The first fusion layer fuses the feature image of the fourth camera and the feature image of the fourth radar to obtain a first fused feature map.
[0062] The second fusion layer upsamples the first fusion feature map to obtain a first upsampled feature map, and then fuses the first upsampled feature map, the third radar feature image, and the third camera feature image to obtain a second fusion feature map.
[0063] The third fusion layer upsamples the second fusion feature map to obtain a second upsampled feature map, and then fuses the second upsampled feature map, the second radar feature image, and the second camera feature image to obtain a third fusion feature map.
[0064] The fourth fusion layer upsamples the third fusion feature map to obtain a third upsampled feature map, and then fuses the third upsampled feature map, the first radar feature image, and the first camera feature image to obtain a fourth fusion feature map.
[0065] The fourth fusion feature map is the fusion data.
[0066] Step S5: Decode the fused data to obtain fused image data.
[0067] The main function of the decoder is to restore the fused sparse features into a dense information result. Empirically, deep features typically contain rich semantic information but lack low-level spatial details. To better recover image texture details, traditional symmetric decoding methods always fuse low-level information from the encoder multiple times through jump connections during the decoding process. However, in this invention, based on a top-down fusion method, the final output of the feature pyramid fusion module already contains rich high- and low-level information. Therefore, accurate image results can be obtained simply by organizing and fusing the multi-level information and then reconstructing it to the original resolution. This invention reconstructs multi-scale features in the top-down feature pyramid fusion module into an image with the original resolution through multiple upsampling operations. Compared with general symmetric decoding methods, the top-down pyramid structure decoding scheme has lower complexity. Since only one block is used per layer during decoding, the model complexity is greatly reduced, achieving good accuracy while maintaining a lightweight model structure.
[0068] Figure 2 This is a structural diagram of the millimeter-wave radar data and camera data fusion system of the present invention. As shown in the figure, the present invention provides a fusion system for millimeter-wave radar data and camera data, including: a data acquisition module 1, a data conversion module 2, a feature extraction module 3, a data fusion module 4, and a decoding module 5.
[0069] The data acquisition module 1 acquires millimeter-wave radar point cloud data and camera image data.
[0070] The data conversion module 2 converts the millimeter-wave radar point cloud data to obtain radar image data of the same size as the camera image data.
[0071] The feature extraction module 3 performs multi-scale feature extraction on the camera image data to obtain camera image feature data; and performs multi-scale feature extraction on the radar image data to obtain radar image feature data.
[0072] The data fusion module 4 fuses the camera image feature data and the radar image feature data from top to bottom to obtain fused data.
[0073] The decoding module 5 decodes the fused data to obtain fused image data.
[0074] As an optional implementation, the data conversion module 2 of the present invention includes: a coordinate conversion unit and a data generation unit.
[0075] The coordinate transformation unit performs coordinate transformation on the millimeter-wave radar point cloud data in the three-dimensional coordinate system and maps it onto the camera image data in the two-dimensional coordinate system to obtain two-dimensional radar data.
[0076] The data generation unit generates the radar image data based on the two-dimensional radar data.
[0077] As an optional implementation, the feature extraction module 3 of the present invention includes: a camera data feature extraction unit and a radar data feature extraction unit.
[0078] The camera data feature extraction unit performs multi-scale feature extraction on the camera image data based on the residual network encoder to obtain the camera image feature data.
[0079] The radar data feature extraction unit performs multi-scale feature extraction on the radar image data based on a maximum pooling encoder to obtain the radar image feature data.
[0080] As an optional implementation, the data fusion module 4 of the present invention specifically comprises:
[0081] The camera image feature data and the radar image feature data are fused using a top-down feature pyramid fusion module to obtain the fused data.
[0082] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.
[0083] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A method for fusing millimeter-wave radar data and camera data, characterized in that, include: Acquire millimeter-wave radar point cloud data and camera image data; The millimeter-wave radar point cloud data is converted to obtain radar image data of the same size as the camera image data. Multi-scale feature extraction is performed on the camera image data to obtain camera image feature data; multi-scale feature extraction is performed on the radar image data to obtain radar image feature data; The camera image feature data and the radar image feature data are fused from top to bottom to obtain fused data; The fused data is decoded to obtain fused image data; The process of extracting camera image feature data from the camera image data using multi-scale features specifically involves: Multi-scale feature extraction is performed on the camera image data based on the residual network encoder to obtain the camera image feature data; the residual network encoder includes a first residual network encoder, a second residual network encoder, a third residual network encoder and a fourth residual network encoder; The first residual network encoder extracts features at a 1 / 4 scale from the camera image data to obtain a first camera feature image; the second residual network encoder extracts features at a 1 / 8 scale from the first camera feature image to obtain a second camera feature image; the third residual network encoder extracts features at a 1 / 16 scale from the second camera feature image to obtain a third camera feature image; and the fourth residual network encoder extracts features at a 1 / 32 scale from the third camera feature image to obtain a fourth camera feature image. The process of extracting radar image feature data from the radar image data using multi-scale features specifically involves: Multi-scale feature extraction is performed on the radar image data based on the max pooling encoder to obtain the radar image feature data; the max pooling encoder includes a four-layer structure, which are defined as the first max pooling encoder, the second max pooling encoder, the third max pooling encoder and the fourth max pooling encoder respectively. The first max-pooling encoder extracts features at a 1 / 4 scale from the radar image data to obtain a first radar feature image; the second max-pooling encoder extracts features at a 1 / 8 scale from the first radar feature image to obtain a second radar feature image; the third max-pooling encoder extracts features at a 1 / 16 scale from the second radar feature image to obtain a third radar feature image; and the fourth max-pooling encoder extracts features at a 1 / 32 scale from the third radar feature image to obtain a fourth radar feature image. The process of fusing the camera image feature data and the radar image feature data from top to bottom to obtain fused data is as follows: Based on the top-down feature pyramid fusion module, the camera image feature data and the radar image feature data are fused to obtain the fused data; the feature pyramid fusion module includes a four-layer structure, which are defined from top to bottom as the first fusion layer, the second fusion layer, the third fusion layer and the fourth fusion layer. The first fusion layer fuses the fourth camera feature image and the fourth radar feature image to obtain a first fused feature map; the second fusion layer upsamples the first fused feature map to obtain a first upsampled feature map, and fuses the first upsampled feature map, the third radar feature image, and the third camera feature image to obtain a second fused feature map; the third fusion layer upsamples the second fused feature map to obtain a second upsampled feature map, and fuses the second upsampled feature map, the second radar feature image, and the second camera feature image to obtain a third fused feature map; the fourth fusion layer upsamples the third fused feature map to obtain a third upsampled feature map, and fuses the third upsampled feature map, the first radar feature image, and the first camera feature image to obtain a fourth fused feature map; the fourth fused feature map is the fused data.
2. The fusion method according to claim 1, characterized in that, The process of converting the millimeter-wave radar point cloud data to obtain radar image data of the same size as the image data includes: The millimeter-wave radar point cloud data in the three-dimensional coordinate system is transformed and mapped onto the camera image data in the two-dimensional coordinate system to obtain two-dimensional radar data. The radar image data is generated based on the two-dimensional radar data.
3. A system for fusing millimeter-wave radar data and camera data, characterized in that, include: The data acquisition module acquires millimeter-wave radar point cloud data and camera image data; The data conversion module converts the millimeter-wave radar point cloud data to obtain radar image data of the same size as the camera image data; The feature extraction module performs multi-scale feature extraction on the camera image data to obtain camera image feature data; and performs multi-scale feature extraction on the radar image data to obtain radar image feature data. The feature extraction module includes: The camera data feature extraction unit performs multi-scale feature extraction on the camera image data based on the residual network encoder to obtain the camera image feature data; the residual network encoder includes a first residual network encoder, a second residual network encoder, a third residual network encoder, and a fourth residual network encoder; The first residual network encoder extracts features at a 1 / 4 scale from the camera image data to obtain a first camera feature image; the second residual network encoder extracts features at a 1 / 8 scale from the first camera feature image to obtain a second camera feature image; the third residual network encoder extracts features at a 1 / 16 scale from the second camera feature image to obtain a third camera feature image; and the fourth residual network encoder extracts features at a 1 / 32 scale from the third camera feature image to obtain a fourth camera feature image. The radar data feature extraction unit performs multi-scale feature extraction on the radar image data based on the max pooling encoder to obtain the radar image feature data; the max pooling encoder includes a four-layer structure, which are defined as the first max pooling encoder, the second max pooling encoder, the third max pooling encoder and the fourth max pooling encoder respectively. The first max-pooling encoder extracts features at a 1 / 4 scale from the radar image data to obtain a first radar feature image; the second max-pooling encoder extracts features at a 1 / 8 scale from the first radar feature image to obtain a second radar feature image; the third max-pooling encoder extracts features at a 1 / 16 scale from the second radar feature image to obtain a third radar feature image; and the fourth max-pooling encoder extracts features at a 1 / 32 scale from the third radar feature image to obtain a fourth radar feature image. The data fusion module fuses the camera image feature data and the radar image feature data from top to bottom to obtain fused data; The data fusion module is specifically as follows: Based on the top-down feature pyramid fusion module, the camera image feature data and the radar image feature data are fused to obtain the fused data; the feature pyramid fusion module includes a four-layer structure, which are defined from top to bottom as the first fusion layer, the second fusion layer, the third fusion layer and the fourth fusion layer. The first fusion layer fuses the fourth camera feature image and the fourth radar feature image to obtain a first fused feature map; the second fusion layer upsamples the first fused feature map to obtain a first upsampled feature map, and fuses the first upsampled feature map, the third radar feature image, and the third camera feature image to obtain a second fused feature map; the third fusion layer upsamples the second fused feature map to obtain a second upsampled feature map, and fuses the second upsampled feature map, the second radar feature image, and the second camera feature image to obtain a third fused feature map; the fourth fusion layer upsamples the third fused feature map to obtain a third upsampled feature map, and fuses the third upsampled feature map, the first radar feature image, and the first camera feature image to obtain a fourth fused feature map; the fourth fused feature map is the fused data; The decoding module decodes the fused data to obtain fused image data.
4. The fusion system according to claim 3, characterized in that, The data conversion module includes: The coordinate transformation unit performs coordinate transformation on the millimeter-wave radar point cloud data in the three-dimensional coordinate system and maps it onto the camera image data in the two-dimensional coordinate system to obtain two-dimensional radar data. The data generation unit generates the radar image data based on the two-dimensional radar data.
Citation Information
Patent Citations
Self-adaptive reinforced fusion real-time instance segmentation method based on camera and laser radar
CN111967373A