A dual-modal target detection method and system based on cross-modal attention mechanism fusion

Abstract

The application provides a kind of based on the fusion of cross-modal attention mechanism dual-mode target detection method and system comprising: obtaining image to be detected and millimeter wave radar data;The millimeter wave radar data is pretreated;The image to be detected and the pretreated millimeter wave radar data are input into the trained dual-mode target detection model based on the fusion of cross-modal attention mechanism, and the detection result is obtained;Wherein, the dual-mode target detection model based on the fusion of cross-modal attention mechanism extracts radar features and image features by using radar feature extraction network and CSPDarkNet53 image feature extraction network based on point cloud Transformer and sparse coding convolution respectively, fuses radar features and image features through cross-modal attention composite feature fusion module at the input and output end of PANet at different scale levels, finally inputs the fusion features into the detection network of YOLOv5-X for detection and obtains the detection result after NMS processing;The effectiveness of the application is verified under nuScenes dataset.

Description

Technical Field

[0001] This invention relates to a dual-modal target detection method and system based on cross-modal attention mechanism fusion, and relates to the fields of deep learning and target detection technology. Background Technology

[0002] In recent years, thanks to the rapid development of deep learning, research on object detection technology has become quite mature, and the performance of object detection algorithms based on visual images has been greatly improved. In good environments, such as with sufficient light and high visibility, visual image detection algorithms can achieve high detection accuracy. However, in harsh environments, such as rain, fog, blizzards, and other low-visibility weather conditions, as well as weak light intensity at night, the detection accuracy of these algorithms drops significantly, resulting in numerous false positives and false negatives, and even complete algorithm failure. In road traffic or outdoor operations, when human vision struggles to identify certain targets in adverse weather or lighting conditions, environmental perception equipment is essential to ensure traffic or production safety. However, visual image-based object detection algorithms are not suitable for detection tasks in such harsh environments.

[0003] To address the aforementioned issues, introducing other types of sensors and achieving target detection under broader conditions through information fusion is an inevitable trend. Compared to camera sensors and LiDAR, millimeter-wave radar's detection performance is less affected by extreme weather and has stronger anti-interference capabilities, but it cannot provide detailed information such as target shape contained in visual images. The two sensors have highly complementary detection capabilities, making the use of millimeter-wave radar combined with image fusion a more advantageous solution to improve the algorithm's detection performance in harsh environments.

[0004] Currently, 2D target detection algorithms based on the fusion of millimeter-wave radar and visual images are still in the research stage. Existing fusion methods for millimeter-wave radar and images are generally divided into data-level fusion, feature-level fusion, and decision-level fusion. Data-level fusion first generates Regions of Interest (ROIs) based on radar points, then extracts corresponding regions of the image based on the ROIs, and then performs the detection task. However, the detection performance of this method is limited by the number of effective radar points. Decision-level fusion fuses the detection results of different sensors to generate the final output, but decision-level fusion methods are very difficult to model in terms of the joint density function of the sensors. Feature-level fusion, on the other hand, can learn radar and visual features simultaneously, fully explore feature information, and achieve better detection performance, making it a highly anticipated fusion method.

[0005] However, feature fusion of millimeter-wave radar data and visual images still faces several challenges. First, due to the heterogeneity of millimeter-wave radar data and visual images, directly using a general convolutional model cannot effectively capture the effective features of the radar data, nor can it fully exploit the semantic relationships contained within the data. Second, existing feature fusion methods often employ concatenation, addition, or spatial attention to fuse the depth features of radar and images; however, these single fusion methods struggle to simultaneously consider the differences and information correlations between different modalities. Therefore, how to design effective feature fusion methods based on the characteristics of different modalities requires further in-depth research. Summary of the Invention

[0006] This invention provides a dual-modal target detection method and system based on cross-modal attention mechanism fusion to achieve high-precision two-dimensional target detection in adverse weather and low-light environments. Building upon the YOLOv5-X detection network, this invention proposes a radar feature extraction network based on point cloud Transformer and sparse coding convolution to output multi-scale radar features. A composite feature fusion module based on cross-modal attention then fuses radar and visual features at multiple scales, ultimately achieving robust target detection for visibility conditions. Specifically, in the radar feature extraction network based on point cloud Transformer and sparse coding convolution (GLRFENet), the permutation invariance of the Transformer is utilized to learn the long-range dependencies of millimeter-wave radar point cloud data, and 3D sparse convolution is used to aggregate local point cloud features to obtain multi-scale radar features suitable for fusion with image features. Furthermore, a composite feature fusion module based on cross-modal attention was designed. First, radar features were used to guide the feature fusion network to learn image features and obtain pseudo-radar features. Then, the information correlation between image features and pseudo-radar features was learned through a cross-attention mechanism to improve cross-modal feature fusion, thereby enhancing the robustness and generalization ability of the model.

[0007] Specifically, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a dual-modal target detection method based on cross-modal attention mechanism fusion, comprising the following steps:

[0009] Acquire the image and millimeter-wave radar data to be detected;

[0010] Preprocess millimeter-wave radar data;

[0011] The image to be detected and the preprocessed millimeter-wave radar data are input into a trained bimodal target detection model based on cross-modal attention mechanism fusion to obtain the detection result output by the bimodal target detection model based on cross-modal attention mechanism fusion.

[0012] The dual-modal target detection model based on cross-modal attention mechanism fusion extracts millimeter-wave radar features and image features using a radar feature extraction network based on point cloud Transformer and sparse coding convolution and a CSPDarkNet53 image feature extraction network, respectively. The radar features and image features are fused at different scale stages at the input and output of PANet through a cross-modal attention composite feature fusion module. Finally, the fused features are input into the YOLOv5-X detection network for detection and processed by NMS to obtain the detection result.

[0013] The aforementioned dual-modal target detection method based on cross-modal attention mechanism fusion further includes the method for extracting millimeter-wave radar features based on a radar feature extraction network using point cloud Transformer and sparse coding convolution, comprising:

[0014] Two-dimensional millimeter-wave radar point clouds that have been preprocessed and projected onto the image plane Introducing depth information to reconstruct 2D point cloud data into 3D point cloud data in Let H represent the set of real numbers, H and W represent the height and width of the initial point cloud map, respectively, and D represent the depth of the radar points. The points are uniformly downsampled to N points, and if the number of points is less than N, they are padded with 0.

[0015] The initial features of each point in the point cloud are set to include 6 dimensions: α j ,β j ,γ j ,rcs j ,v j 1 ,v j 2 , where α j ,β j ,γ j Let rcs represent the spatial location of the j-th radar point in dimensions W, H, and D, respectively. j ,v j 1 ,v j 2 Let represent the radar cross-section, lateral velocity, and radial velocity of the j-th radar point, respectively, thus obtaining the initial point cloud features. Simultaneously, the spatial coordinates of each point are recorded, and then the long-range dependencies of the radar point cloud data are captured through the point cloud Transformer sub-network to enhance the global features of the data and obtain the output point cloud features.

[0016] F is implemented through a sparse coding layer outThe features of each point are encoded according to the recorded spatial coordinates, with zeros used to fill in the positions of non-existent points, in order to construct a four-dimensional radar feature sparse tensor. Where C represents the number of characteristic channels of the radar point;

[0017] We further learn the local features of the point cloud using a 3D sparse convolutional sub-network, and combine it with downsampling to obtain global-local multi-scale radar features;

[0018] The aforementioned dual-modal target detection method based on cross-modal attention mechanism fusion further includes the specific step of uniformly downsampling the millimeter-wave radar point cloud as follows:

[0019] Determine whether the number of points in the point cloud is greater than or less than N. If it is less than N, no sampling operation is performed, and all points are retained. If it is greater than N, the 3D point cloud is divided into voxel grids.

[0020] The point cloud is divided using voxel grids, and the points in the point cloud are assigned to the corresponding voxel grids according to their positions. Each voxel grid contains several points.

[0021] For each voxel grid, samples are taken, and the points in each voxel grid are taken as a point set. First, the centroid of the voxel grid is calculated, and then the Kd-Tree nearest neighbor search algorithm is used to select the point closest to the centroid of the voxel grid as the first round of sampling points, until the number of sampling points is N.

[0022] If the number of sampling points is less than N, a specified number of points are randomly sampled from the remaining radar points as a supplement;

[0023] The bimodal target detection method based on cross-modal attention mechanism fusion further includes a 3D sparse convolutional sub-network comprising five feature extraction stages (Stage 1 to Stage 5), with downsampling step sizes of S1, S2, S3, S4, and S5 for each stage. The step size for each stage is determined based on the required radar feature scale. The five feature extraction stages (Stage 1 to Stage 5) have identical and fixed network structures, each consisting of one conventional sparse convolutional layer and two identical submanifold sparse convolutional layers. The sparse convolutional layers consist of 3×3 regular sparse convolutions, a BatchNorm1d normalization function, and a ReLU activation function. The submanifold sparse convolutional layers also consist of 3×3 submanifold sparse convolutions, a BatchNorm1d normalization function, and a ReLU activation function. The stride of the regular sparse convolutions in each stage is consistent with the downsampling stride of each stage, while the stride of the submanifold sparse convolutions is always 1. The kernel size of 3×3 is a fixed value. After processing by the 3D sparse convolutional subnetwork, Stages 3 through 5 output radar features at different scales. and Then, by merging the depth dimension into the channel dimension, the final multi-scale radar features are obtained. and Used for fusing with multi-scale image features extracted by the CSPDarkNet53 image feature extraction network, where Li represents the i-th stage, C i D i H i and W i Let C represent the number of channels, depth, height, and width of the radar feature map in the i-th stage, respectively. i D i This represents the number of radar feature map channels after merging the depth dimension into the channel dimension in the i-th stage, where i = 3, 4, 5;

[0024] The aforementioned dual-modal target detection method based on cross-modal attention mechanism fusion further includes the following process: fusing radar features and image features through a cross-modal attention composite feature fusion module.

[0025] Radar characteristics of the same stage and image features To splice, among which and The radar and image features of stage i are represented, and the feature scales of corresponding stages are the same. Then, a 1×1 convolution is used to perform channel compression and inter-channel information exchange on the concatenated features, with a compression ratio of e, to obtain intermediate features. Reusing spatial attention Spatial feature optimization is performed to obtain pseudo-radar features. Here, e is set to 2 in order to make the number of channels of the obtained pseudo radar features consistent with the number of channels of the image features, so as to adapt to subsequent fusion.

[0026] Calculating image features using multi-head cross-attention mechanism and pseudo-radar characteristics The correlation between them is used to fuse pseudo-radar features and image features, wherein the number of heads is a fixed value of 8;

[0027] The bimodal target detection method based on cross-modal attention mechanism fusion further includes the following step: Optimizing spatial features using spatial attention, which includes:

[0028] Use global max pooling and global average pooling to extract intermediate features. The channel dimensions are aggregated into a scalar, and the following are obtained respectively. and Then and The result is obtained by splicing along the channel dimension.

[0029] Captured using 3×3 convolutions and 7×7 convolutions with stride of 1 respectively Different spatial information is processed and the number of channels is reduced to 1, where the stride, kernel size and number of channels are fixed values, and then the features are concatenated.

[0030] Spatial attention weights are obtained by performing global average pooling on the channel dimension and then passing it through the Sigmoid function, and the intermediate features are then... The output feature is obtained by multiplying it with the spatial attention weights.

[0031] The bimodal target detection method based on cross-modal attention mechanism fusion further includes the following step: Calculating the correlation between image features and pseudo-radar features using a cross-attention mechanism, and fusing features from different modalities based on this correlation, including:

[0032] false radar characteristics and image features Flattening the data in space yields the pseudo-radar feature sequences. and image feature sequences Where H i W i This represents the length of the feature sequence after spatial expansion of the pseudo-radar features and image features in the i-th stage;

[0033] Then Obtained through linear transformation As a query vector (Query) Obtained through linear transformations and As the key vector (Key) and value vector (Value), for each radar feature point and its corresponding image region, Q is used... 2 and K 2 Calculate the correlation and weights between pseudo-radar features and image features, determine the importance of each pseudo-radar feature point to the image features, and then use the calculated weights to apply to V. 2 Weighted summation yields attention feature sequences

[0034] X Attn With the pseudo radar feature sequence The parts are spliced ​​together and folded to restore their original shapes, resulting in a fused feature.

[0035] Secondly, the present invention also provides a dual-modal target detection system based on cross-modal attention mechanism fusion, the system comprising:

[0036] The data acquisition module is used to acquire the image to be detected and the millimeter-wave radar data;

[0037] The data preprocessing module is used to preprocess millimeter-wave radar data;

[0038] The target detection module is used to input the image to be detected and the preprocessed millimeter-wave radar data into a trained bimodal target detection model based on cross-modal attention mechanism fusion, and obtain the detection result output by the bimodal target detection model based on cross-modal attention mechanism fusion. Specifically, the bimodal target detection model based on cross-modal attention mechanism fusion extracts millimeter-wave radar features and image features using a radar feature extraction network based on point cloud Transformer and sparse coding convolution and a CSPDarkNet53 image feature extraction network, respectively. The radar features and image features are then fused at different stages at the PANet input and output ends through a cross-modal attention composite feature fusion module. Finally, the fused features are input into the YOLOv5-X detection network for detection and processed by NMS to obtain the detection result.

[0039] The inventiveness of this invention is mainly reflected in:

[0040] (1) This invention proposes a radar feature extraction network based on point cloud Transformer and sparse coding convolution. This network can learn the global features of millimeter-wave radar point cloud by utilizing the permutation invariance of Transformer, and further aggregate the local features of point cloud using 3D sparse convolution to obtain multi-scale radar features suitable for fusion with image features.

[0041] (2) This invention proposes a cross-modal attention composite feature fusion module. First, the radar feature guides the feature fusion network to learn the image features to obtain pseudo radar features. Then, the image features and pseudo radar features are used to learn the information correlation between the modalities through a cross-attention mechanism, which improves the effectiveness of cross-modal feature fusion and thus enhances the robustness and generalization ability of the model.

[0042] (3) This invention combines a radar feature extraction network based on point cloud Transformer and sparse coding convolution with a cross-modal attention composite feature fusion module to construct a dual-modal target detection network based on cross-modal attention mechanism fusion, thereby improving the target detection accuracy in complex environments;

[0043] In summary, the method of the present invention is applicable to target detection in adverse weather conditions such as rain and night, and in low light environments, and has broad application prospects. Attached Figure Description

[0044] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. In the drawings:

[0045] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0046] Figure 2 This is a diagram of the radar feature extraction network structure based on point cloud Transformer and sparse coding convolution in an embodiment of the present invention.

[0047] Figure 3 This is a structural diagram of the offset attention module according to an embodiment of the present invention;

[0048] Figure 4 This is a cross-modal attention composite feature fusion module according to an embodiment of the present invention, wherein SA is a spatial attention module;

[0049] Figure 5 This is a structural diagram of the spatial attention module according to an embodiment of the present invention;

[0050] Figure 6 This is a diagram of the bimodal target detection network structure based on cross-modal attention mechanism fusion according to an embodiment of the present invention;

[0051] Figure 7 This is a comparison of the detection results of the bimodal target detection method based on cross-modal attention mechanism fusion and the pure visual image target detection method YOLOv5-X according to an embodiment of the present invention; Detailed Implementation

[0052] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art.

[0053] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0054] For ease of description, spatial relative terms may be used in the text to describe the relationship of one element or feature relative to another element or feature as shown in the figure. These relative terms include, for example, "inside," "outside," "middle," "outer," "below," "above," etc. Such spatial relative terms are intended to include different orientations of the device in use or operation, other than those depicted in the figure.

[0055] Example 1

[0056] To address the poor performance of purely visual image object detection networks in adverse weather and low-light environments, this embodiment proposes a bimodal object detection method based on cross-modal attention mechanism fusion, including the following:

[0057] S1. Acquire the image and millimeter-wave radar data to be detected;

[0058] S2. Preprocess the millimeter-wave radar data;

[0059] Specifically, preprocessing includes the following three steps:

[0060] First, radar data quality is improved by combining radar signals from multiple cycles and filtering out noise points using a three-dimensional bounding box.

[0061] Then, the point was expanded to a 3-meter vertical line to compensate for the lack of height in the radar data;

[0062] Finally, spatial alignment between radar and image is achieved by mapping radar data onto the image plane;

[0063] S3. Input the image to be detected and the preprocessed millimeter-wave radar data into a trained bimodal target detection model based on cross-modal attention mechanism fusion to obtain the detection result output by the bimodal target detection model based on cross-modal attention mechanism fusion;

[0064] The dual-modal target detection model based on cross-modal attention mechanism fusion extracts millimeter-wave radar features and image features using a radar feature extraction network based on point cloud Transformer and sparse coding convolution and a CSPDarkNet53 image feature extraction network, respectively. The radar features and image features are fused at different stages at the PANet input and output ends through a cross-modal attention composite feature fusion module. Finally, the fused features are input into the YOLOv5-X detection network for detection and processed by NMS to obtain the detection result.

[0065] S31. Specifically, the radar feature extraction network based on point cloud Transformer and sparse coding convolution has the following structure: Figure 2 As shown, the specific process of radar feature extraction includes:

[0066] The network maps the pre-processed 2D radar point cloud onto the image plane. As input, the depth information of the radar points (i.e., the radial distance between the target and the sensor) is first projected onto the direction perpendicular to the image as extended third-dimensional spatial coordinate information to construct three-dimensional radar point cloud data based on the image coordinate system. in Let H represent the set of real numbers, and W represent the height and width of the initial point cloud map, respectively. H = 384, W = 640, and D represents the depth of the radar point, D = 64. Then, the points are uniformly downsampled to 2048 points. 2048 is an empirical value. If the number of points is less than 2048, it is padded with 0.

[0067] The specific steps for uniform downsampling the point cloud are as follows: First, determine whether the number of points in the point cloud is greater than or less than N. If it is less than N, no sampling operation is performed, and all points are retained. If it is greater than N, the 3D point cloud is divided into voxel grids, and the points in the point cloud are assigned to the corresponding voxel grids according to their positions. Each voxel grid contains several points. Then, each voxel grid is sampled, and the points in each voxel grid are treated as a point set. First, the centroid of the voxel grid is calculated, and then the Kd-Tree nearest neighbor search algorithm is used to select the point closest to the centroid of the voxel grid as the first round of sampling points, until the number of sampling points is N. If the number of sampling points is less than N, a specified number of points are randomly sampled from the remaining radar points as supplementary sampling points.

[0068] Furthermore, a point cloud Transformer subnetwork is used to capture the long-range dependencies of radar point cloud data, enhancing the global features of the data. The point cloud Transformer subnetwork structure includes an input embedding layer, an attention layer, and a linear output layer. First, the features of the radar points are initialized; the initial features of each radar point can be represented as:

[0069]

[0070] Among them, Z j Let α represent the initial characteristics of the j-th radar point. j ,β j ,γ j Let rcs represent the spatial coordinates of W, H, and D corresponding to the j-th radar point, respectively. j ,v j 1 ,v j 2 Let represent the radar cross-section, lateral velocity, and radial velocity of the j-th radar point, respectively, and N represent the number of radar points; the input embedding layer contains two linear layers used to embed the initial radar features. N=2048, mapped to a new high-dimensional feature space, to obtain embedded features.

[0071] The embedded features are then fed into an attention layer consisting of four stacked offset attention modules, where 4 is a fixed value, to learn the long-range dependencies between each point; the offset attention module can be represented as:

[0072]

[0073] Among them, F emb Q represents the embedded features. 1 K 1 and V 1 W represents the Query, Key, and Value matrices, respectively. q 1 W k 1 and W v 1 Both are learnable parameter matrices, where A(·) represents the computation of attention features, l 1norm Let L1 be the norm, T be the matrix transpose, LBR be the linear layer, and F be the linear norm. Attention This represents the output features of the attention module; the offset attention module normalizes the attention map using the L1 norm, such as... Figure 3 As shown, attention features are then used in conjunction with F. emb The offset is used to replace the original attention features, which enhances the attention weights and reduces noise interference. Then, the output features of the multi-level attention modules are concatenated to enrich the feature representation and reduce information loss. Finally, a linear layer further fuses the features to output point cloud features.

[0074] A sparse coding layer is used to encode features based on the spatial distribution of the 3D point cloud P using global point cloud features, in order to construct a 4D radar feature tensor. Wherein, C = 160, D = 64, H = 384, W = 640;

[0075] G is then input into a multi-scale 3D sparse convolutional sub-network for efficient local aggregation, learning the local features of the radar point cloud and saving a lot of unnecessary computing resources. The 3D sparse convolutional sub-network is similar to the image feature extraction network, including five feature extraction stages: Stage 1 to Stage 5. The main parameters are summarized in Table 1.

[0076] Table 1. Structure and parameters of 3D sparse convolutional layers

[0077]

[0078] Based on the table above, the downsampling step sizes for each stage of radar feature extraction are S1, S2, S3, S4, and S5, respectively. The step size for each stage is determined by the desired radar feature scale, as shown in Table 1. The network structures for the five feature extraction stages (Stages 1-5) are identical and fixed, each consisting of one conventional sparse convolutional layer and two identical submanifold sparse convolutional layers. The conventional sparse convolutional layer includes a 3×3 conventional sparse convolution, a BatchNorm1d normalization function, and a ReLU activation function. The submanifold sparse convolutional layers include a 3×3 submanifold sparse convolution, a BatchNorm1d normalization function, and a ReLU activation function. The step size of the conventional sparse convolution in each stage is consistent with the downsampling step size, while the step size of the submanifold sparse convolution is always 1. The kernel size of 3×3 is a fixed value. After processing by the 3D sparse convolutional subnetwork, Stages 3-5 output radar features at different scales. and Then, by merging the depth dimension into the channel dimension, the final multi-scale radar features are obtained. and Used for fusing with multi-scale image features extracted by the CSPDarkNet53 image feature extraction network, where Li represents the i-th stage, C i D i H i and W i Let C represent the number of channels, depth, height, and width of the radar feature map in the i-th stage, respectively. i D iThis represents the number of radar feature map channels after merging the depth dimension into the channel dimension in the i-th stage, i = 3, 4, 5, C3, C4, C5 = 40, 80, 160, D3, D4, D5 = 8, 8, 8, H3, H4, H5 = 48, 24, 12, W3, W4, W5 = 80, 40, 20, C3D3, C4D4, C5D5 = 320, 640, 1280;

[0079] S32. Specifically, the cross-modal attention composite feature fusion module is embedded in the input and output of PANet, fusing radar features and image features from different stages, with the structure as follows: Figure 4 As shown, the process of fusing radar features and image features through the cross-modal attention composite feature fusion module includes:

[0080] Radar characteristics of the same stage and image features To splice, where F L 1 i and F L 2 i The radar and image features of the i-th stage are represented, and the feature scales of corresponding stages are the same. Then, a 1×1 convolution is used to perform channel compression and inter-channel information exchange on the concatenated features, with a compression ratio of e=2, to obtain intermediate features. Reusing spatial attention Spatial feature optimization is performed to obtain pseudo-radar features. Here, e is set to 2 to ensure that the number of channels in the obtained pseudo-radar features is consistent with the number of channels in the image features, thus adapting to subsequent fusion.

[0081] The spatial attention, such as Figure 5 As shown, global max pooling and global average pooling are first used to pool the intermediate features. The channel dimensions are aggregated into a scalar, and the following are obtained respectively. and Then and The result is obtained by splicing along the channel dimension.

[0082] Then, capture the 3×3 convolution and 7×7 convolution with stride of 1 respectively. Different spatial information is used, and the number of channels is reduced to 1, where the stride, kernel size and number of channels are fixed values, and then the features are concatenated again.

[0083] Finally, spatial attention weights are obtained by performing global average pooling on the channel dimension and passing it through the Sigmoid function, and then using the intermediate features. The output feature is obtained by multiplying it with the spatial attention weights. Spatial attention can be represented as:

[0084]

[0085] Where SA(g) represents computational spatial attention, Indicates intermediate features, * indicates kernel size, F * This represents the features obtained by convolving concatenated pooling features. *×* This indicates a convolutional layer with a kernel size of *×*. GMP and GAP represent global max pooling and global average pooling, respectively. Concat represents the concatenation operation.

[0086] Furthermore, image features are calculated using a cross-attention mechanism. and pseudo-radar characteristics The correlation between them is used to fuse pseudo-radar features and image features, wherein the number of heads is a fixed value of 8;

[0087] Specifically, pseudo-radar characteristics and image features Flattening the image according to the spatial dimensions yields the corresponding feature sequences. and Where H i W i This indicates the sequence length after spatial expansion: H3W3 = 3840, H4W4 = 960, H5W5 = 240.

[0088] Then Obtained through linear transformation As a query vector (Query) Obtained through linear transformations and As the key vector (Key) and value vector (Value), for each radar feature point and its corresponding image region, Q is used... 2 and K 2 Calculate the correlation and weights between pseudo-radar features and image features, determine the importance of each pseudo-radar feature point to the image features, and then use the calculated weights to apply to V. 2 Weighting is performed to obtain the attention feature sequence.

[0089] Finally, X Attn With the pseudo radar feature sequence The parts are spliced ​​together and folded to restore their original shapes, resulting in a fused feature. Therefore, the proposed cross-modal attention composite feature fusion module can be represented by the following formula:

[0090]

[0091] Where Flat represents performing a flattening operation on the feature, Q 2 K 2 and V 2 W represents the Query, Key, and Value matrices, respectively. q 2 W k 2 and W v 2 Both are learnable parameter matrices, dim i This represents the dimension of Query / Value at different stages. The value is 320, 640 and 1280 depending on the stage. nh represents the number of heads in multi-head attention. nh=8 is a fixed value. PE represents the learnable relative position encoding. Reshape represents the operation of folding to restore the original feature shape.

[0092] S33. Further, the fusion features output from each stage of PANet are input into the corresponding stage of the YOLOv5-X detection network for processing to obtain the detection boxes and class parameters of each target in the image. Then, redundant detection boxes are filtered out by NMS non-maximum suppression post-processing to obtain the final detection result.

[0093] The experimental process and results of the bimodal target detection method based on cross-modal attention mechanism fusion of the present invention are described in detail below through specific embodiments;

[0094] 1. Experimental platform configuration

[0095] This embodiment is implemented in the PyTorch framework, and the running platform is based on an Intel i7-12700 CPU, an NVIDIA GeForce RTX 4090 GPU, and an Ubuntu 20.04.3LTS operating system for training and testing.

[0096] 2. Train a bimodal object detection network model based on cross-modal attention mechanism fusion.

[0097] 2.1 Dataset and Evaluation Metrics

[0098] In the experiment, the nuScenes dataset was used as the training dataset. nuScenes contains millimeter-wave radar and image data of 1000 different scenes captured in real scenes. The data of 850 scenes with annotations were used as experimental data. The dataset was preprocessed and divided into training set, validation set and test set in a ratio of 6:2:2. This ratio is an empirical value. A total of 20480 pairs of radar and image data were used as the training set and 6830 pairs of radar and image data were used as the test set. In order to adapt to network training, the resolution of input image and radar data was adjusted to 384×640.

[0099] In this embodiment, mAP (IoU = 0.5) is used as the evaluation index for the bimodal target detection method based on cross-modal attention mechanism fusion. Generally speaking, the higher the value of this index, the stronger the detection performance of the algorithm.

[0100] 2.2 Training Parameter Settings

[0101] The network in this embodiment is implemented in the PyTorch framework and trained using the SGD optimizer on a GeForce RTX 4090 GPU. The batch size is set to 4, the epoch is set to 100, and the initial learning rate is set to 0.01. The batch size is limited by the device's memory size, and the epoch and initial learning rate are empirical values ​​that are adaptively adjusted based on the training. The network is trained based on a YOLOv5-X pre-trained model.

[0102] 2.3 Results Analysis

[0103] The bimodal object detection model based on cross-modal attention mechanism fusion of the present invention was tested on the nuScenes dataset, and its performance was compared with that of a pure visual image object detection model using the mAP evaluation metric.

[0104] The performance comparison results are shown in Table 2. Under the mAP evaluation index, the best results are indicated in bold.

[0105] In the comparison of algorithm performance based on the mAP evaluation metric, the dual-modal target detection method based on cross-modal attention mechanism fusion of the present invention is superior to the pure visual image target detection method.

[0106] Table 2 Comparison of Experimental Results

[0107]

[0108] Figure 7The detection results of the pure visual image object detection algorithm YOLOv5-X and the dual-modal object detection algorithm of this invention are visualized respectively. The results show that the detection accuracy and recall of this invention are higher than those of the pure visual image algorithm. Figure 7 As shown, YOLOv5-X cannot completely and accurately identify people and cars in low light, while the present invention can accurately identify targets in low light.

[0109] Therefore, experimental results show that the dual-modal target detection method of the present invention achieves robust target detection in complex environments.

[0110] Example 2

[0111] The above embodiment 1 provides a bimodal target detection method based on cross-modal attention mechanism fusion. Correspondingly, this embodiment provides a bimodal target detection system based on cross-modal attention mechanism fusion. The bimodal target detection system based on cross-modal attention mechanism fusion provided in this embodiment can implement the bimodal target detection method based on cross-modal attention mechanism fusion in embodiment 1. The system can be implemented by software, hardware, or a combination of both. For example, the system may include integrated or separate functional modules or functional units to execute the corresponding steps in the methods of embodiment 1. Since the bimodal target detection system based on cross-modal attention mechanism fusion in this embodiment is basically similar to the method embodiment, the description process of this embodiment is relatively simple. For relevant parts, please refer to the description of embodiment 1. The bimodal target detection system based on cross-modal attention mechanism fusion in this embodiment is merely illustrative.

[0112] This embodiment provides a dual-modal target detection system based on cross-modal attention mechanism fusion, the system comprising:

[0113] The data acquisition module is used to acquire the image to be detected and the millimeter-wave radar data;

[0114] The data preprocessing module is used to preprocess millimeter-wave radar data;

[0115] The target detection module is used to input the image to be detected and the preprocessed millimeter-wave radar data into a trained bimodal target detection model based on cross-modal attention mechanism fusion, and obtain the detection result output by the bimodal target detection model based on cross-modal attention mechanism fusion. Specifically, the bimodal target detection model based on cross-modal attention mechanism fusion extracts millimeter-wave radar features and image features using a radar feature extraction network based on point cloud Transformer and sparse coding convolution and a CSPDarkNet53 image feature extraction network, respectively. At different stages at the PANet input and output, the radar features and image features are fused through a cross-modal attention composite feature fusion module. Finally, the fused features are input into the YOLOv5-X detection network for detection and processed by NMS to obtain the detection result.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dual-modal target detection method based on cross-modal attention mechanism fusion, characterized in that, include: Acquire the image and millimeter-wave radar data to be detected; Preprocess millimeter-wave radar data; The image to be detected and the preprocessed millimeter-wave radar data are input into a trained bimodal target detection model based on cross-modal attention mechanism fusion to obtain the detection result output by the bimodal target detection model based on cross-modal attention mechanism fusion. The dual-modal target detection model based on cross-modal attention mechanism fusion extracts millimeter-wave radar features and image features using a radar feature extraction network based on point cloud Transformer and sparse coding convolution and a CSPDarkNet53 image feature extraction network, respectively. The radar features and image features are fused at different stages at the PANet input and output ends through a cross-modal attention composite feature fusion module. Finally, the fused features are input into the YOLOv5-X detection network for detection and processed by NMS to obtain the detection result. The process of fusing radar features and image features through the cross-modal attention composite feature fusion module includes: Radar characteristics of the same stage and image features To splice, among which and The radar and image features of stage i are represented, and the feature scales of corresponding stages are the same. Then, a 1×1 convolution is used to perform channel compression and inter-channel information exchange on the concatenated features, with a compression ratio of e, to obtain intermediate features. Then use spatial attention to Spatial feature optimization is performed to obtain pseudo-radar features. Where e is 2, in order to make the number of channels of the obtained pseudo radar features consistent with the number of channels of the image features, so as to adapt to subsequent fusion; Calculating image features using multi-head cross-attention mechanism and pseudo-radar characteristics The correlation between them is used to fuse pseudo-radar features and image features, where the number of heads is a fixed value of 8.

2. The dual-modal target detection method based on cross-modal attention mechanism fusion according to claim 1, characterized in that, The method for extracting millimeter-wave radar features based on point cloud Transformer and sparse coded convolution includes: For the preprocessed two-dimensional millimeter-wave radar point cloud P0∈ℝ projected onto the image plane H×W By introducing depth information, two-dimensional point cloud data is reconstructed into three-dimensional point cloud data P∈ℝ D×H×W , where ℝ represents the set of real numbers, H and W represent the height and width of the initial point cloud map respectively, D represents the depth of the radar points, and is uniformly downsampled to N points, with 0s padded if the number of points is less than N; The initial features of each point in the point cloud are set to include features with 6 dimensions: ,in These represent the spatial positions of the j-th radar point in dimensions W, H, and D, respectively. Let F represent the radar cross-section, lateral velocity, and radial velocity of the j-th radar point, respectively, thus yielding the initial point cloud features F. init ∈ℝ N×6 Simultaneously, the spatial coordinates of each point are recorded, and then the long-range dependencies of the radar point cloud data are captured through a point cloud Transformer sub-network to enhance the global features of the data, resulting in the output point cloud feature F. out ∈ℝ N×160 ; F is implemented through a sparse coding layer out The features of each point are encoded according to the recorded spatial coordinates, with zeros used to fill in the positions of non-existent points, to construct a four-dimensional radar feature sparse tensor G∈ℝ. C×D×H×W , where C represents the number of characteristic channels of the radar point; We further learn the local features of point clouds using 3D sparse convolutional subnetworks, and combine them with downsampling to obtain global-local multi-scale radar features.

3. The dual-modal target detection method based on cross-modal attention mechanism fusion according to claim 2, characterized in that, The specific steps for uniformly downsampling millimeter-wave radar point clouds are as follows: Determine whether the number of points in the point cloud is greater than or less than N. If it is less than N, no sampling operation is performed, and all points are retained. If it is greater than N, the 3D point cloud is divided into voxel grids. The point cloud is divided using voxel grids, and the points in the point cloud are assigned to the corresponding voxel grids according to their positions. Each voxel grid contains several points. For each voxel grid, samples are taken, and the points in each voxel grid are taken as a point set. First, the centroid of the voxel grid is calculated, and then the Kd-Tree nearest neighbor search algorithm is used to select the point closest to the centroid of the voxel grid as the first round of sampling points, until the number of sampling points is N. If the number of sampling points is less than N, a specified number of points are randomly sampled from the remaining radar points as supplementary sampling points.

4. The dual-modal target detection method based on cross-modal attention mechanism fusion according to claim 2, characterized in that, The 3D sparse convolutional subnetwork comprises five feature extraction stages, Stages 1 to 5, with downsampling strides of S1, S2, S3, S4, and S5 for each stage. The stride of each stage is determined based on the desired radar feature scale. The five feature extraction stages (Stages 1 to 5) have identical and fixed network structures, each consisting of one conventional sparse convolutional layer and two identical submanifold sparse convolutional layers. The conventional sparse convolutional layer includes a 3×3 conventional sparse convolutional layer... The BatchNorm1d normalization function and ReLU activation function are used. The sub-manifold sparse convolutional layer includes 3×3 sub-manifold sparse convolution, BatchNorm1d normalization function, and ReLU activation function. The stride of the regular sparse convolution in each stage is consistent with the downsampling stride of each stage, while the stride of the sub-manifold sparse convolution is 1. The kernel size of 3×3 is a fixed value. After processing by the 3D sparse convolutional sub-network, Stages 3 through 5 output radar features of different scales. , and Then, by merging the depth dimension into the channel dimension, the final multi-scale radar features are obtained. , and This is used to fuse with multi-scale image features extracted by the CSPDarkNet53 image feature extraction network, where Li represents the i-th stage and C i D i H i and W i Let C represent the number of channels, depth, height, and width of the radar feature map in the i-th stage, respectively. i D i This represents the number of radar feature map channels after merging the depth dimension into the channel dimension in the i-th stage, where i = 3, 4, 5.

5. The dual-modal target detection method based on cross-modal attention mechanism fusion according to claim 1, characterized in that, Spatial feature optimization using spatial attention includes: Use global max pooling and global average pooling to extract intermediate features. The channel dimensions are aggregated into a scalar, and the following are obtained respectively. and Then and The data is obtained by splicing according to the channel dimension. ; Captured using 3×3 convolutions and 7×7 convolutions with stride of 1 respectively Different spatial information is processed and the number of channels is reduced to 1, where the stride, kernel size and number of channels are fixed values, and then the features are concatenated. Spatial attention weights are obtained by performing global average pooling on the channel dimension and then passing it through the Sigmoid function, and the intermediate features are then... The output feature is obtained by multiplying it with the spatial attention weights. .

6. The dual-modal target detection method based on cross-modal attention mechanism fusion according to claim 1, characterized in that, The method of calculating the correlation between image features and pseudo-radar features using a multi-head cross-attention mechanism, and fusing features from different modalities based on the correlation, includes: false radar characteristics and image features Flattening the data in space yields the pseudo-radar feature sequences. and image feature sequences H i W i This represents the length of the feature sequence after spatial expansion of the pseudo-radar features and image features in the i-th stage; Then Obtained through linear transformation As a query vector (Query) Obtained through linear transformations and As key vectors and value vectors, for each radar feature point and its corresponding image region, using... and The correlation and weights between pseudo-radar features and image features are calculated to determine the importance of each pseudo-radar feature point to the image features. Then, the calculated weights are used to... Weighting is performed to obtain the attention feature sequence. ; Will With the pseudo radar feature sequence The parts are spliced ​​together and folded to restore their original shapes, resulting in a fused feature. .

7. A system applying the method as described in any one of claims 1-6, characterized in that, The system includes: The data acquisition module is used to acquire the image to be detected and the millimeter-wave radar data; The data preprocessing module is used to preprocess millimeter-wave radar data; The target detection module is used to input the image to be detected and the preprocessed millimeter-wave radar data into a trained bimodal target detection model based on cross-modal attention mechanism fusion, and obtain the detection result output by the bimodal target detection model based on cross-modal attention mechanism fusion. Specifically, the bimodal target detection model based on cross-modal attention mechanism fusion extracts millimeter-wave radar features and image features using a radar feature extraction network based on point cloud Transformer and sparse coding convolution and a CSPDarkNet53 image feature extraction network, respectively. The radar features and image features are then fused at different stages at the PANet input and output ends through a cross-modal attention composite feature fusion module. Finally, the fused features are input into the YOLOv5-X detection network for detection and processed by NMS to obtain the detection result.

Images