A multi-sensor target detection method based on visual self-attention

Abstract

The present invention discloses a multi-sensor target detection method based on visual self-attention, which belongs to the field of target detection technology. It is used to solve the technical problems of high cost and poor real-time performance of current target detection solutions based on lidar and images. The present invention uses a visual self-attention model to fuse the image features of millimeter-wave radar and images, thereby replacing the currently commonly used feature fusion method based on convolution and element-level addition to achieve more efficient fusion. Since the targets in driving scenarios usually have large changes in target scale, it is crucial to use multi-scale features for target detection. Therefore, a decoder structure that fuses multi-scale features is proposed. That is, the network RCFormer proposed by the present invention is superior to the existing optimal solution based on millimeter-wave radar and images in target detection accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to a multi-sensor target detection method based on visual self-attention. Background Art

[0002] Object detection is one of the most important and challenging branches of computer vision, widely used in autonomous driving, smart healthcare, surveillance, and other fields. Over the past few decades, convolutional neural networks have made significant progress in object detection.

[0003] Object detection aims to determine the category and location of all objects in an image. It is a core challenge in the field of autonomous driving. Here, these approaches are categorized into anchor-based and anchor-free detectors. Anchor-based detectors can be divided into two categories: two-stage and single-stage methods. Two-stage algorithms generally have higher accuracy than single-stage algorithms, but are slower. Two-stage algorithms primarily consist of two stages: 1) generating region proposals from the image and 2) generating final detection bounding boxes from the region proposals. Representative works include RCNN, SPPNet, Faster RCNN, and FPN. One-stage algorithms, on the other hand, do not require the region proposal generation stage, resulting in faster detection speeds. Representative works include the YOLO series, SSD, and RetinaNet. Anchor-free detectors dispense with anchors and instead detect objects by identifying keypoints, thereby reducing the number of network hyperparameters. CornerNet is a pioneering work in this category, transforming the detection of an object's bounding box into the detection of a pair of keypoints (the top-left and bottom-right corners of the bounding box). In addition, CenterNet, FSAF, FCOS, and other projects have also explored this approach. CenterNet's overall network structure is lightweight and simple, making it more advantageous for application scenarios with limited computing power and real-time performance.

[0004] The existing object detection algorithms mentioned above are all based solely on vision. The autonomous driving field places extremely high demands on object detection accuracy, as the consequences of false detection are often unacceptable. Multi-sensor fusion perception is an inevitable trend in autonomous driving, aiming to integrate information from multiple sensors to improve overall system accuracy. Obviously, the use of multiple sensors significantly increases the amount of information that needs to be processed, which can be redundant or even contradictory. Therefore, effectively fusing multimodal information is a challenge to address. Furthermore, current academic research has largely focused on fusing LiDAR and images, while millimeter-wave radar, already deployed in vehicles and widely used in adaptive cruise control and automatic emergency braking systems, has been neglected. During the implementation of the present invention's technical solution, the inventors discovered that while millimeter-wave radar point clouds are highly sparse compared to LiDAR point clouds, they offer advantages such as long detection range and reduced weather impact, making them attractive and promising research areas. Fusing highly sparse point clouds with images has also become another research challenge.

[0005] Current methods for fusing millimeter-wave radar and images are based on convolutional neural networks. These networks have inherent inductive biases, making them easy to train but with a low upper limit. Visual self-attention models, on the other hand, introduce less inductive bias, making them naturally adaptable to processing information from multiple modalities. They are currently being applied to text, speech, images, point clouds, and other fields. However, the use of visual self-attention models to fuse multimodal information is still in its early stages of exploration. Summary of the Invention

[0006] The present invention provides a multi-sensor target detection method based on visual self-attention. Unlike using a visual self-attention model to process information of a single modality, the present invention uses a visual self-attention model to fuse millimeter-wave radar and images to achieve more efficient fusion.

[0007] The technical solution adopted in the present invention is as follows:

[0008] A multi-sensor target detection method based on visual self-attention, the method comprising:

[0009] Step 1: Set up and train a multi-sensor object detection network based on visual self-attention;

[0010] The multi-sensor target detection network based on visual self-attention includes: an encoder 1 and an encoder 2 based on a residual network, the back ends of the encoder 1 and the encoder 2 are respectively connected to a first reshaping network, the outputs of the two first reshaping networks are fed into a fusion module based on a visual self-attention model, and the back end of the fusion module based on the visual self-attention model is sequentially connected to a second reshaping network, a decoder, and a detection head;

[0011] in,

[0012] The input of encoder 1 is the image captured by the camera acquisition device, which is used to extract image features. The input of encoder 2 is the millimeter radar wave input, which is used to extract radar features. The output feature maps of the two encoders have the same size, H×W×C, where H×W represents the image size of the feature map and C represents the number of channels.

[0013] The first reshaping network is used to reshape the output feature map of the encoder into a flattened two-dimensional patch mark, and the second reshaping network is the inverse operation of the first reshaping network;

[0014] The fusion module based on the visual self-attention model adopts full-label fusion or cross-label fusion for fusion processing. If full-label fusion is adopted, the labels output by the two first reshaping networks are spliced ​​according to the channel dimension and then fused through the self-attention module; if cross-label fusion is adopted, the labels output by the two first reshaping networks are directly fused through the cross-attention module;

[0015] The decoder includes multiple upsampling modules, which are used to upsample the multi-scale features output by different stages to the same resolution, concatenate the upsampled features at each scale, and then map the concatenated features into decoder features through a convolutional layer (preferably a convolution layer with a 1x1 kernel). The decoded features are fed into the detection head to predict the heatmap, offset, and scale of the bounding box.

[0016] Concatenate the upsampled features from different decoding stages, and then

[0017] Step 2: Input the image data to be detected and the radar point cloud image collected by the millimeter radar into the multi-sensor target detection network based on visual self-attention trained in step 1, and obtain the target detection result based on its output.

[0018] Furthermore, the first reshaping network is used to reshape the output feature map of the encoder into a flattened two-dimensional patch label specifically as follows;

[0019] The output feature map of the encoder is evenly divided into multiple blocks to obtain multiple image blocks, and the size of each image block is expressed as P×P;

[0020] Each image block is flattened into a one-dimensional vector to obtain a reshaped flat two-dimensional patch mark with a dimension of N×(P 2 ×C), where represents the number of image blocks, that is, the number of patches generated, and R represents the downsampling rate.

[0021] Furthermore, the fusion module based on the visual self-attention model adopts the full-label fusion processing as follows:

[0022] Define x to represent the mark after splicing according to the channel dimension;

[0023] The multi-head self-attention module based on dynamic position encoding obtains the multi-head self-attention representation MSA(Q, K, V):

[0024]

[0025] Among them, Softmax() represents the output of the Softmax function, and the query Q = xW q , key K = xW k , value V = xW V , denotes the weights of query Q, key K and value V respectively, M and h denote the embedding dimension and number of heads respectively, K T represents the transpose of the key K, d represents the dimension of the query Q, and Q, K and V have the same dimension, B represents the output of the dynamic position encoding module, which takes the coordinate distance between the markers as input and outputs the relative position encoding;

[0026] According to z=y+MLP(y), the fused multimodal feature z is obtained, that is, the output feature of the fusion module based on the visual self-attention model; where y=x+MSA(LN(x)), MSA() represents the output of the multi-head self-attention module, MLP() represents the output of the multi-layer perceptron, and LN() represents the output of the linear layer.

[0027] Furthermore, the fusion module based on the visual self-attention model adopts the cross-label fusion process as follows:

[0028] Define X img 、X radar They represent the image mark and radar mark after the first reshaping network, respectively, and the image mark X img Generate query Q, mark X with radar radar Generate key K and value V: Q = X img W q , K=X radar W k , V=X radar W V , Denote the weights of query Q, key K, and value V respectively, M and h denote the embedding dimension and number of heads respectively;

[0029] according to Get the multi-head self-attention representation MSA(Q, K, V), where K T represents the transpose of the key K, d represents the dimension of the query Q, and B represents the output of the dynamic position encoding module, which takes the coordinate distance between the markers as input and outputs the relative position encoding;

[0030] According to z′=y′+MLP(y′), the fused multimodal feature z′ is obtained, where y′=X img +MSA(LN(Q, K, V)), where MSA() represents the output of the multi-head self-attention module, MLP() represents the output of the multi-layer perceptron, and LN() represents the output of the layernormalization layer, i.e., the output of the normalization layer.

[0031] Furthermore, the network structure of the dynamic position encoding module includes: a fully connected layer, a normalization layer, an activation function layer and a fully connected layer in sequence.

[0032] Preferably, each detection head consists of a 3x3 convolutional layer, an activation layer, and a 1x1 convolutional layer.

[0033] Compared with the existing millimeter wave radar and image fusion target detection method or pure visual target detection method, the present invention has the following beneficial effects:

[0034] 1) The accuracy of target detection is better than that of existing fusion target detection methods and pure visual target detection methods.

[0035] 2) The addition of millimeter radar information reduces the possible false detection and missed detection that may occur in pure visual methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 is a schematic diagram of the structure of the RCFormer network used in an embodiment of the present invention;

[0038] Figure 2 2 is a schematic diagram of the structure of the fusion module based on the visual self-attention model used in an embodiment of the present invention, wherein (2-a) represents full-label fusion and (2-b) represents cross-label fusion;

[0039] Figure 3 is a schematic structural diagram of a dynamic position encoding module used in an embodiment of the present invention;

[0040] Figure 4 is a schematic structural diagram of a decoder module used in an embodiment of the present invention;

[0041] Figure 5This is a visualization comparison effect diagram of the method adopted in the embodiment of the present invention and CenterNet, where (5-1) represents a daytime scene and (5-2) represents a nighttime scene. (a) in (5-1) and (5-2) represents the detection result based on the pure visual target detection network CenterNet, and (b) in (5-1) and (5-2) represents the detection result of the RCFormer network in the embodiment of the present invention. DETAILED DESCRIPTION

[0042] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0043] Considering that multi-sensor fusion perception is an inevitable trend in autonomous driving, while current solutions based on lidar and images suffer from high costs and poor real-time performance, the present invention proposes a target detection solution based on millimeter-wave radar and images. Furthermore, considering that the visual self-attention model introduces less inductive bias, making it naturally capable of processing information from multiple modalities, it has currently been applied in fields such as text, speech, images, and point clouds. Therefore, the present invention proposes two methods for fusing millimeter-wave radar and images using the visual self-attention model, thereby replacing the currently commonly used feature fusion methods based on convolution and element-wise addition to achieve more efficient fusion. In addition, since targets in driving scenarios typically vary greatly in scale, it is crucial to utilize multi-scale features for target detection. Therefore, the present invention proposes a decoder structure that fuses multi-scale features. In combination with this decoder structure, the present invention establishes a target detection network named RCFormer, which outperforms the existing optimal solutions based on millimeter-wave radar and images in terms of detection accuracy. That is, the target detection network provided by the embodiment of the present invention is a multi-sensor target detection network (RCFormer) based on visual self-attention, which includes two methods of fusing millimeter-wave radar and images using a visual self-attention model and a decoder structure that fuses multi-scale features.

[0044] In this invention, the overall architecture of the multi-sensor target detection network based on visual self-attention (RCFormer) is as follows Figure 1As shown, it includes: an encoder 1 and an encoder 2 based on a residual network, the back ends of the encoder 1 and the encoder 2 are respectively connected to a first reshaping network, the outputs of the two first reshaping networks are fed into a fusion module based on a visual self-attention model, and the back end of the fusion module based on the visual self-attention model is sequentially connected to a second reshaping network, a decoder and a detection head (for detection and classification of target objects, and output size categories and positions), wherein the input of the encoder 1 is an image captured by a camera acquisition device, which is used to extract image features, and the input of the encoder 2 is a millimeter radar wave input (radar point cloud image captured by a millimeter radar, also called a radar pixel map), which is used to extract radar features, and the sizes of the output feature maps of the two encoders are the same. The same; the first reshaping network is used to reshape the output feature map of the encoder into a flattened two-dimensional patch label sequence, and the second reshaping network is the inverse operation of the first reshaping network, that is, it is used to reshape the flattened two-dimensional label output by the fusion module based on the visual self-attention model into the form of a feature map and send it to the decoder; the fusion module based on the visual self-attention model can adopt two fusion modes: full label fusion and cross label fusion. If full label fusion is adopted, the labels output by the two first reshaping networks are first spliced ​​according to the channel dimension and then fused through the self-attention module; if cross label fusion is adopted, the labels output by the two first reshaping networks are directly fused through the cross attention module.

[0045] Specifically, the present invention projects radar points onto the image and fills the depth at the corresponding pixel position to obtain the single-channel image. Represents that its size (H×W) is the same as that of the image. Considering that images usually contain rich information and are dense, while radar pixel maps are highly sparse, with values ​​only at some pixels, their representations are very different. Therefore, using the same backbone to extract features will lead to poor performance. The present invention adopts a dual encoder structure to extract features of images and radars respectively, which is achieved through the residual network ResNet. That is, two encoding networks are used to extract features for image input and millimeter-wave radar input respectively. As a possible implementation method, in an embodiment of the present invention, ResNet-34 is used for images and ResNet-18 is used for millimeter-wave radar. Among them, 34-18 represents ResNet with different network layers. The dual encoder structure outputs feature maps of the same size, which are image features (denoted as x img , ) and millimeter-wave radar characteristics (denoted as x radar , ), and then reshape the image features and radar features into a flattened two-dimensional patch label sequence Where C is the number of input channels, (P,P) is the resolution of each 2D patch, is the number of patches generated, and R is the downsampling rate. That is, the two labeled sequences (image labels and radar labels) obtained after reshaping come from image features and millimeter-wave radar features, respectively, but have the same shape (X PATCH ). The image label and radar label are then sent to the fusion module based on the visual self-attention model to fuse features (full label fusion is input after splicing, and cross label fusion is directly input). That is, for the fusion module based on the visual self-attention model, the present invention proposes two fusion methods, namely full label fusion and cross label fusion, both of which are input with dynamic position encoding and label dimension. The label output by the fusion module based on the visual self-attention model is reshaped into the form of a feature map and then sent to the decoder. That is, in the present invention, the second reshaping network is the inverse process of the first reshaping network, thereby realizing the 2-dimensional (N×(P 2 ×C)) is reshaped back to a 3D (H×W×C) feature map. Considering the importance of multi-scale features in object detection, current methods predict multi-scale features multiple times separately, but this increases time and resources. Unlike existing methods, the present invention upsamples and merges features of different scales, and uses the merged features to replace the original single-scale features for prediction. Specifically, the upsampled multi-scale features are connected in series in the decoder, and 1x1 convolution is used to aggregate the features to obtain the decoded features x DEC , then x DEC The prediction head predicts the heatmap, offset, and scale of the bounding box.

[0046] As a possible implementation method, the specific implementation steps of the embodiment of the present invention based on the set RCFormer network are as follows:

[0047] 1. Using full-label fusion (such as Figure 2 (2-a) in the figure) is used to fuse millimeter-wave radar features and image features to obtain fused multimodal features.

[0048] 1.1. First, the image feature map x img and radar signature x radar They are reshaped into labels through the first reshaping network respectively, and then all labels of the two branches are spliced ​​in the channel dimension, which can be understood as expanding the image features with radar features.

[0049] x=[reshape(x img ); reshape(x radar )] (1)

[0050] Among them, x represents the concatenated label, and reshape represents the reshaping operation of the first reshaping network.

[0051] 1.2. Use self-attention module to fuse features.

[0052] y=x+MSA(LN(x)) (2)

[0053] z=y+MLP(y) (3)

[0054] Among them, MSA represents the multi-head self-attention module, MLP represents the multi-layer perceptron, and z represents the fused multimodal features. That is, MSA() represents the output of the multi-head self-attention module, MLP() represents the output of the multi-layer perceptron, and LN() represents the output of the linear layer. In the calculation of the multi-head self-attention module, relative position encoding is used, and dynamic position encoding is introduced to make the calculation of relative position encoding more flexible. It is based on the coordinate distance (Δx ij , Δy ij ) is input and output position code B ij , where the subscripts i and j are used to represent the two embeddings. The structure of dynamic position encoding is as follows Figure 3 As shown, it includes in sequence: a fully connected layer, a normalization layer, an activation function layer and a fully connected layer.

[0055] Multi-head self-attention using dynamic position encoding is represented as:

[0056]

[0057] Where, query Q = xW q , key K = xW k ,value are learnable parameters, i.e., the weights of query Q, key K, and value V, C and h are the embedding dimension and number of heads, K T represents the transposition of K, d represents, and B represents the output of the dynamic position encoding module.

[0058] Of course, in the embodiment of the present invention, cross-label fusion is used to fuse millimeter-wave radar features and image features to obtain fused multimodal features, such as Figure 2 As shown in (2-b), the specific processing process is:

[0059] (1) Generate query Q using image features, generate key K and value V using radar features, Q = reshape(x img )W q ,K=reshape(x radar )W k ,V=reshape(x radar )W V .

[0060] (2) Use the cross-attention module to fuse features.

[0061] First, based on formula (4), we get the multi-head self-attention representation MSA(Q, K, V) using dynamic position encoding, which is the output of the second multiplication operation in the forward propagation direction in Figure (2-b);

[0062] Then, the obtained MSA (Q, K, V) is passed through the linear layer and combined with the reshaped image feature X img Add the label y', that is, y'=X img +MSA(LN(Q,K,V)), where X img =reshape(x img );

[0063] Finally, the label y′ is passed through a multi-layer perceptron to obtain the fused multimodal feature z′, that is, z′=y′+MLP(y′).

[0064] 2. Use the decoder that fuses multi-scale features to generate decoder features with better representation capabilities, which are then input into the detection head to predict the scale of the heatmap, offset, and bounding box.

[0065] 2.1. For the outputs of different decoder stages, the decoder outputs of different stages are upsampled to the same resolution by stacking different numbers of upsampling modules, such as Figure 4 As shown in Figure 2, each upsampling module includes: a convolutional layer, a group normalization layer, an activation function layer, and a bilinear upsampling layer.

[0066] x DECUP_i =Up(ReLU(GN(Conv(X DEC_i )))) (5)

[0067] Among them, x DECUP_i represents the features after upsampling at the i-th stage of the decoder, x DEC_i Denotes the decoder stage i feature. Conv denotes the convolutional layer, GN denotes the group normalization layer, ReLU denotes the activation function, and Up denotes the bilinear upsampling layer.

[0068] 2.2. Concatenate the upsampled features from different decoding stages, and then use a 1x1 convolutional layer to map the concatenated features into the final decoder features, as shown in the following example: Figure 4 shown.

[0069] 2.3. Input the decoder features into different detection heads, namely, the heatmap detection head, the offset detection head, and the detection box scale detection head. Preferably, in the embodiment of the present invention, each detection head consists of a 3x3 convolution layer, an activation layer, and a 1x1 convolution layer.

[0070] In the embodiment of the present invention, the reshaping operation of the first reshaping network is specifically as follows:

[0071] The original input image data H×W×C is evenly divided into multiple blocks to obtain multiple blocks. Assuming that the length and width of each block are (P, P), the number of blocks can be expressed as: N = H×W / (P×P);

[0072] Each image block is then flattened into a one-dimensional vector, each vector size is P×P×C, so that the reshaped flattened two-dimensional patch tag sequence is obtained based on the flattened one-dimensional vectors of all image blocks, that is, the total input transformation is: N×(P 2 ×C).

[0073] As a possible implementation, the nuScenes dataset is used for both training and testing in this embodiment of the present invention. The nuScenes dataset contains not only camera and lidar data, but also millimeter-wave radar data, making it one of the few large datasets to include millimeter-wave radar data. Each scene in this dataset lasts 20 seconds and contains 40 keyframes, each with a resolution of 1600×900. Furthermore, nuScenes includes driving scenarios in various conditions, such as rainy days and at night, which increases the difficulty of object detection in this dataset. Because the dataset only provides 3D labels, this embodiment converts them to 2D labels and removes scenes without objects or with low-quality labels. This embodiment uses PyTorch to deploy the network and trains on four NVIDIA GeForce GTX TITAN Xs. Images are resized from 1600×900 to 1344×896 for training. The batch size is set to 12, and the Adam optimizer is used with a learning rate of 0.000125.

[0074] As shown in Table 1, the embodiment of the present invention achieves 41.4% AP (where AP represents the average precision, the mean of the precision values ​​on the PR curve), which is 5.2% higher than the baseline method CenterNet and 3.7% higher than BIRANet. In addition, Figure 5 A visual comparison between the embodiment of the present invention and CenterNet is shown. It can be seen that the embodiment of the present invention achieves better target detection performance. This is because the radar provides additional information to the model, and the radar and image can complement each other in terms of features, thus avoiding the missed detection phenomenon that may be caused by some purely image-based methods.

[0075] Table 1 Target detection results

[0076]

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

[0078] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the inventive concept of the present invention, which all fall within the scope of protection of the present invention.

Claims

1. A multi-sensor target detection method based on visual self-attention, characterized in that: The method comprises: Step 1: Set up and train a multi-sensor object detection network based on visual self-attention; The multi-sensor target detection network based on visual self-attention includes: an encoder 1 and an encoder 2 based on a residual network, the back ends of the encoder 1 and the encoder 2 are respectively connected to a first reshaping network, the outputs of the two first reshaping networks are fed into a fusion module based on a visual self-attention model, and the back end of the fusion module based on the visual self-attention model is sequentially connected to a second reshaping network, a decoder, and a detection head; in, The input of encoder 1 is the image captured by the camera acquisition device, which is used to extract image features. The input of encoder 2 is the millimeter-wave radar input, which is used to extract radar features. The output feature maps of the two encoders have the same size, H×W×C, where H×W represents the image size of the feature map and C represents the number of channels. The first reshaping network is used to reshape the output feature map of the encoder into a flattened two-dimensional patch mark, and the second reshaping network is the inverse operation of the first reshaping network; The fusion module based on the visual self-attention model adopts full-label fusion or cross-label fusion for fusion processing. If full-label fusion is adopted, the labels output by the two first reshaping networks are spliced ​​according to the channel dimension and then fused through the self-attention module; if cross-label fusion is adopted, the labels output by the two first reshaping networks are directly fused through the cross-attention module; The decoder includes multiple upsampling modules. Based on the upsampling process, the decoder generates features of multiple scales. Each scale feature is sampled to the same resolution by an upsampling module and then concatenated. The concatenated features are then mapped to decoder features through a convolutional layer. The decoder features are fed into the detection head to predict the scale of the heatmap, offset, and bounding box. Step 2: Input the image data to be detected and the radar point cloud image collected by the millimeter radar into the multi-sensor target detection network based on visual self-attention trained in step 1, and obtain the target detection result based on its output.

2. The method according to claim 1, wherein The first reshaping network is used to reshape the output feature map of the encoder into a flattened two-dimensional patch labeled as follows; The output feature map of the encoder is evenly divided into blocks to obtain multiple image blocks; Each image block is flattened into a one-dimensional vector, and the reshaped flattened two-dimensional patch label is obtained based on the one-dimensional vectors of all image blocks.

3. The method according to claim 1, wherein The fusion module based on the visual self-attention model uses the full-label fusion process as follows: Define x to represent the mark after splicing according to the channel dimension; The multi-head self-attention module based on dynamic position encoding obtains the multi-head self-attention representation MSA(Q,K,V): Where, query Q = xW q , key K = xW k , value V = xW V , denotes the weights of query Q, key K and value V respectively, M and h denote the embedding dimension and number of heads respectively, K T represents the transpose of the key K, d represents the dimension of the query Q, and Q, K and V have the same dimension, B represents the output of the dynamic position encoding module, which takes the coordinate distance between the markers as input and outputs the relative position encoding; The fused multimodal feature z is obtained according to z=y+MLP(y), where y=x+MSA(LN(x)), MSA() represents the output of the multi-head self-attention module, MLP() represents the output of the multi-layer perceptron, and LN() represents the output of the normalization layer.

4. The method according to claim 1, wherein The fusion module based on the visual self-attention model uses the cross-label fusion process as follows: Define X img 、X radar They represent the image mark and radar mark after the first reshaping network, respectively, and the image mark X img Generate query Q, mark X with radar radar Generate key K and value V: Q = X img W q , K=X radar W k , V=X radar W V , Denote the weights of query Q, key K and value V respectively, M and h denote the embedding dimension and number of heads respectively, according to Get the multi-head self-attention representation MSA(Q,K,V), where K T represents the transpose of the key K, and B represents the output of the dynamic position encoding module, which takes the coordinate distance between the marks as input and outputs the relative position code; According to z ′ =y+MLP(y ′ ) to obtain the fused multimodal feature z ′ , where y ′ =X img +MSA(LN(Q,K,V)), where MSA() represents the output of the multi-head self-attention module, MLP() represents the output of the multi-layer perceptron, and LN() represents the output of the normalization layer.

5. The method according to claim 3 or 4, wherein: The network structure of the dynamic position encoding module includes: a fully connected layer, a normalization layer, an activation function layer and a fully connected layer in sequence.

6. The method according to claim 1, wherein The detection head sequentially comprises a convolution layer with a convolution kernel of 3x3, an activation layer, and a convolution layer with a convolution kernel of 1x1.

Images