A Transformer-based cross-modal fusion target detection method

Abstract

The present invention relates to a Transformer-based cross-modal fusion target detection method, which belongs to the field of image processing technology. The present invention uses a radar sensor to collect radar data while a camera collects image data. Millimeter-wave radar information and image information are aligned, including time alignment and spatial coordinate conversion, and radar features are mapped to the image plane. Radar information is preprocessed according to the physical properties of the radar. A feature extraction network is constructed to extract radar features and image features respectively. A radar and visual feature interactive fusion module is designed, fully considering cross-modal information interaction and supplementary learning of contextual environment information. Compared with the prior art, the present invention combines surrounding environment information with cross-modal information for joint learning. It solves the problem that traditional methods can only obtain the same position information and lack perception of surrounding environment information, while reducing the false detection rate and missed detection rate in bad weather and complex scenes.

Description

Technical Field

[0001] The present invention relates to a Transformer-based cross-modal fusion target detection method, and belongs to the technical field of image processing. Background Art

[0002] Assisted driving tasks require real-time and accurate detection. Single sensors struggle to adapt to target detection and real-time decision-making in complex scenarios, increasing driving risks. Utilizing multi-sensor information fusion can effectively reduce missed and false detection rates in complex weather and scenarios, ensuring the safety of assisted driving. With deep learning becoming the mainstream approach for target detection, multi-sensor fusion is increasingly trending toward the use of deep learning frameworks to fuse multimodal features. However, current fusion methods have limitations, focusing on modeling local information while neglecting the attention and perception of the surrounding environment. Furthermore, the lack of analysis of the characteristics of different sensors leads to variability in collected data, making it impossible to rationally utilize and complement the information from different sensors. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a Transformer-based cross-modal fusion target detection method, which combines the surrounding environment information with cross-modal information for joint learning, thereby solving the problem that traditional methods can only obtain the same location information and lack perception of the surrounding environment information, while reducing the false detection rate and missed detection rate in severe weather and complex scenarios.

[0004] The technical solution of the present invention is: a cross-modal fusion target detection method based on Transformer, characterized by:

[0005] Step 1: Extract radar data information and visual image information from the millimeter-wave radar sensor and camera respectively;

[0006] Step 2: Present the radar information in the form of point cloud data and align the two modal data in time and space to ensure consistency in data format.

[0007] Step 3: Use coordinate transformation to map the 3D radar point cloud information to the 2D plane of the image. The mapped radar information is a matrix with 18 channels, and its size is consistent with the image matrix.

[0008] Step 4: Expand and enhance a single radar point to make it more suitable for the actual size of the target;

[0009] Step 5: Extract and fuse the enhanced radar information with the image data;

[0010] Step 6: Finally, classify and identify them.

[0011] The specific steps of expanding and enhancing a single radar point in Step 4 are as follows:

[0012] Step 4.1: Determine the spatial coordinates of the radar point mapping on the image;

[0013] Step 4.2: Model the target based on its distance from the sensor and the radar cross-section (RCS) sent back by the radar to obtain the target's height displayed on the image. This will supplement the lack of target height information when the radar collects information, and the height will be recorded as

[0014] Step 4.3: Expand the original radar point into a line of length 1 on the image based on the acquired height. The line segment is used to supplement the missing target height information;

[0015] Step 4.4: Based on the radar cross-sectional area information, the enhanced line segment is expanded horizontally with a width of S to supplement the missing width information of the target. At this time, after the complete enhancement, a rectangular area is formed. This area is the radar focus area, and all information in this area is consistent with the single radar point information before enhancement.

[0016] The specific steps of Step 5 are:

[0017] Step 5.1: Use VGG16 as the feature extraction network, input radar information and image information respectively, and perform feature fusion after each block;

[0018] Step 5.2: Construct RC-Attention to calculate the similarity of different modalities;

[0019] Step 5.3: Construct a feature fusion module Multi-Transformer based on RC-Attention to achieve cross-modal fusion of features and interaction of contextual information.

[0020] The dataset used in this paper is the nuScenes dataset. It consists of 1,000 scenes, each 20 seconds long, encompassing a wide variety of scenarios. Each scene contains 40 keyframes. The nuScenes dataset uses six cameras, one lidar, five millimeter-wave radars, a GPS, and one inertial measurement sensor for data collection. The experimental validation was performed using data from the front-facing radar and camera.

[0021] The feature extraction network uses VGG16 and performs cross-modal feature fusion between each block.

[0022] The fusion method designs a multimodal information interaction module, Multi-Transformer, to achieve joint complementary learning of cross-modal information interaction and global context information. It also constructs a novel attention mechanism to achieve deep calculation of similarity weights between different modalities.

[0023] The beneficial effects of this invention are as follows: Compared with existing technologies, it primarily addresses the issue of insufficient fusion caused by differences in data collected by different sensors. It effectively reduces false detection rates in extremely dim conditions or in inclement weather, while also enabling effective recognition of occluded targets. It fully perceives cross-modal environmental information and global context, enhancing target detection and recognition capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of the steps of the present invention;

[0025] Figure 2 It is the overall network diagram of the cross-modal interactive fusion method of the present invention;

[0026] Figure 3 This is a structural diagram of the fusion module Multi-Transformer designed in the present invention;

[0027] Figure 4 It is the effect diagram of the final detection of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0029] Example 1: Figure 1 As shown in Figure 1, a cross-modal fusion target detection method based on Transformer is shown in Figure 1. The specific steps are as follows:

[0030] Step 1: Read the front millimeter-wave radar and front camera keyframe information from the nuScenes dataset. The millimeter-wave radar returns physical information such as target distance, speed, and radar cross-section (RCS), while the camera captures the target's RGB information.

[0031] The nuScenes dataset used consists of 1,000 scenes, each 20 seconds long, covering a wide variety of scenarios. Each scene contains 40 keyframes. The nuScenes dataset uses six cameras, one lidar, five millimeter-wave radars, GPS, and one inertial measurement sensor for data collection. The experimental validation was performed using data from the front-facing radar and camera.

[0032] Step 2: Spatially align the millimeter-wave radar information with the image information, and map the radar features onto the image plane.

[0033] Specifically, the coordinates in the millimeter-wave coordinate system are first converted to the world coordinate system; second, the coordinates in the world coordinate system are converted to the image coordinate system. After this conversion, the radar information is stored as two-dimensional points. The dimensions of the radar signature map and image signature map are 900 × 1600 × 18 and 900 × 1600 × 3, respectively. The radar signature and image signature maps are spliced ​​along the channel to map the radar signature to the two-dimensional image.

[0034] Step 3: Preprocess the radar information based on the physical characteristics of the radar.

[0035] Step 3.1: First, determine the initial position of the radar point mapped to the image plane, denoted as O i .

[0036]

[0037] Among them, x i is the vertical offset of the i-th radar point, h is the initial height of the radar sensor installation, and in this method h = 0.5. τ and are hyperparameters, namely the position factor and the vertical offset factor.

[0038] Step 3.2: In order to supplement the lack of target height information when the radar sensor detects the target, the present invention expands a single radar point into a longitudinally extended line segment. The length of the line segment is based on the radar cross section (RCS) of different targets. i and distance dist i Adaptively adjust to fit the actual height of the target in the image.

[0039]

[0040]

[0041]

[0042] Among them, α and β are hyperparameters, which are the distance factor and scattering cross-section factor respectively.

[0043] Step 3.3: Based on the initial position O of the radar point i , to enhance the height. Divided into two parts l1 and l2.

[0044]

[0045]

[0046]

[0047] Step 3.4: Based on radar cross section RCS i Perform horizontal enhancement on the target with an enhancement width of S i , to fill in the missing width information.

[0048]

[0049] Step 4: Build a feature extraction network to extract radar features and image features respectively;

[0050] The present invention first performs feature extraction, taking as input a radar feature map after preprocessing and enhancement, and visual features. The feature map dimensions are 900 × 1600 × 2 and 900 × 1600 × 3, respectively. Principal component analysis is used to select two distinct channels from the radar's 18 channels: range and radar cross-section. The image has three feature channels, including RGB information.

[0051] Step 5: Design a radar and visual feature interaction fusion module, fully considering cross-modal information interaction and supplementary learning of contextual environment information.

[0052] like Figure 2 As shown in Figure 2, after the third, fourth, and fifth blocks of the feature extraction network, the Multi-Transformer module is used to fuse multimodal features. The overall structure of the fusion module is shown in Figure 2. Figure 3 shown.

[0053] The module's inputs are radar features and image features. Due to spatial alignment and fusion operations, the input radar features and image features have the same dimensions and size.

[0054] The similarity calculation between modalities and within modalities is implemented in Encoder and Decoder, such as Figure 4 shown.

[0055] RC-Attention calculates the similarity of information at any position in the two modal feature maps, assigns similarity weights between different modalities, and achieves cross-modal feature matching, thereby achieving cross-modal information interaction. The output is radar features fused with image information, and image features fused with radar information, such as Figure 3 shown.

[0056]

[0057]

[0058]

[0059]

[0060]

[0061] V β =Linear(X β +PE(X β ))

[0062] Among them, Linear(·) is a linear transformation, Separate(·) further splits the heads of different features, and PE(·) is a position encoding.

[0063] The above describes the specific embodiments of the present invention in detail with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A Transformer-based cross-modal fusion target detection method, characterized by: Step 1: Extract radar data information and visual image information from the millimeter-wave radar sensor and camera respectively; Step 2: Present the radar information in the form of point cloud data and align the two modal data in time and space; Step 3: Use coordinate transformation to map the 3D radar point cloud information to the 2D plane of the image. The mapped radar information is a matrix with 18 channels, and its size is consistent with the image matrix. Step 4: Expand and enhance a single radar point; Step 5: Extract and fuse the enhanced radar information with the image data; Step 6: Finally, classify and identify them; The specific steps of Step 3 are: Step 3.1: First, determine the initial position of the radar point mapped to the image plane, denoted as O i ; Among them, x i is the vertical offset of the i-th radar point, h is the initial height of the radar sensor installation, τ and are hyperparameters, namely the position factor and the vertical offset factor; Step 3.2: To supplement the missing target height information when the radar sensor detects the target, the single radar point is expanded into a longitudinally extended line segment. The length of the line segment is based on the radar cross section (RCS) of different targets. i and distance dist i Perform adaptive adjustments to fit the actual height of the target in the image: Among them, α and β are hyperparameters, which are distance factor and scattering cross-sectional area factor respectively; Step 3.3: Based on the initial position O of the radar point i , to enhance the height, and enhance the line segment Divided into two parts l1 and l2: Step 3.4: Based on radar cross section RCS i Perform horizontal enhancement on the target with an enhancement width of S i , to supplement the missing width information; The specific steps of Step 4 are: Step 4.1: Determine the spatial coordinates of the radar point mapping on the image; Step 4.2: Model the target based on its distance from the sensor and the radar cross-sectional area transmitted by the radar. Obtain the target's height as displayed on the image. Record the height as Step 4.3: Expand the original radar point into a line of length 1 on the image based on the acquired height. line segment; Step 4.4: Expand the enhanced line segment horizontally based on the radar cross-sectional area information. The expansion width is S. The fully enhanced area is now a rectangular area. This area is the radar focus area, and all information within this area is consistent with the single radar point information before enhancement. The specific steps of Step 5 are: Step 5.1: Use VGG16 as the feature extraction network, input radar information and image information respectively, and perform feature fusion after each block; Step 5.2: Construct RC-Attention to calculate the similarity of different modalities; Step 5.3: Build a feature fusion module Multi-Transformer based on RC-Attention to achieve cross-modal feature fusion and interaction of contextual information; The feature extraction network uses VGG16 and performs cross-modal feature fusion between each block; After the third, fourth, and fifth blocks of the feature extraction network, the Multi-Transformer module is used to fuse multimodal features; The module inputs radar features and image features, respectively. The encoder and decoder perform inter-modal and intra-modal similarity calculations. RC-Attention calculates the similarity of information at any location in the feature maps of two modalities, assigns similarity weights between different modalities, and achieves cross-modal feature matching, thereby enabling cross-modal information interaction. The output is radar features fused with image information, and image features fused with radar information.

Images