Multi-sensor fusion target detection method based on Transformer

Abstract

The present invention discloses a Transformer-based multi-sensor fusion target detection method, which includes first acquiring raw data from three sensors: a camera, a lidar, and a millimeter-wave radar, inputting the acquired data as three different modalities into a Transformer-based multi-sensor fusion network, then repeatedly strengthening the camera target modality with the features of the lidar source modality, outputting a potentially adapted new modality, and then repeatedly strengthening the features of the new modality with the features of the millimeter-wave radar source modality, outputting the final fusion result, and obtaining a supplemented and corrected road target detection frame. The present invention can avoid the harm caused by the misdetection or false detection of a single sensor during vehicle driving, thereby ensuring that more accurate and rich road target information is provided for the decision-making and planning part.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a target detection method that integrates a camera, a laser radar, and a millimeter-wave radar. Background Art

[0002] In recent years, with the rapid development of intelligent driving technology, the automotive industry has placed increasing emphasis on the development of advanced driver assistance systems (ADAS) to enhance driving safety. Accurate and efficient environmental perception is a primary task in achieving ADAS. Currently, intelligent vehicle environmental perception technology primarily relies on on-board sensors such as radar and vision. Vision sensors are primarily used for target recognition, offering advantages such as a wide detection range and relatively complete target information. However, they require high computational effort and lack depth information. Millimeter-wave radar sensors are primarily used for target position and velocity detection. They offer advantages such as direct access to obstacle distance and velocity information and the ability to penetrate rain, fog, and dense smoke. However, they exhibit high noise levels and lack the ability to capture target outlines. LiDAR is primarily used for obstacle detection and depth information acquisition. Its advantages include direct access to three-dimensional environmental information and strong anti-interference capabilities. However, its disadvantages include susceptibility to weather conditions such as rain, snow, and fog, and its high cost. Due to its operating principles and other factors, a single sensor typically only captures partial features of the detected target, failing to fully describe the road environment and thus failing to meet the environmental perception requirements of intelligent driving.

[0003] At present, the more popular rule-based post-fusion algorithm requires pairwise joint calibration of multiple sensors during the fusion stage because different sensors have different data processing methods. One of the difficulties is spatiotemporal alignment, unifying the data of different sensors into the same coordinate system. The second difficulty is the formulation of the fusion strategy. In the event of sensor failure, the fusion process is difficult to continue. Summary of the Invention

[0004] In response to the problems existing in the existing methods, the present invention provides a Transformer-based multi-sensor fusion target detection method, in order to realize multi-sensor target fusion for sensing direction in the field of intelligent driving, obtain more accurate and rich road target information, and thus avoid the disadvantages caused by false detection or misdetection of a single sensor during vehicle driving.

[0005] In order to achieve the above-mentioned object, the present invention adopts the following technical solutions:

[0006] The present invention provides a multi-sensor fusion target detection method based on a Transformer network, which comprises the following steps:

[0007] A. The original image data is obtained by the camera and input into the pre-trained target detection neural network YOLOv4 for processing, and the image recognition results of the road target are output. Among them, the image feature sequence composed of the image recognition results of any road target is recorded as N C , including: category information of road targets class cam and 2D detection box data bbox cam_2D And confidence score cam , the 2D detection box data bbox cam_2D Includes: the center coordinates of the 2D detection frame (x C_2D ,y C_2D ), the length w of the 2D detection box C_2D and width h C_2D ;

[0008] B. The point cloud data is collected by the lidar and input into the pre-trained point cloud target detection network Pointpillars for processing, and the point cloud recognition results of the road target are output. Among them, the point cloud feature sequence composed of the point cloud recognition results of any road target is recorded as N L , including: category information of road targets class lidar , 2D detection box data bbox lidar_2D , 3D detection box data bbox lidar_3D And confidence score lidar ; The 2D detection box data bbox lidar_2D Including the center coordinates of the 2D detection box (x L_2D ,y L_2D ), the length w of the 2D detection box L_2D and width h L_2D , the 3D detection box data bbox lidar_3D Includes: the center coordinates of the 3D detection frame (x L_3D ,y L_3D , z L_3D ), the length w of the 3D detection box L_3D 、Width h L_3D and Gaol L_3D ;

[0009] C. Use millimeter wave radar to collect original 2D point cloud data and perform clustering processing to obtain the clustering results of road targets. The 2D point cloud feature sequence composed of the clustering results of any road target is recorded as N R , including: category information of road targets class radar , 2D detection box data bbox radar_2D , road target speed information velocity radar And confidence score radar; The 2D detection box data bbox radar_2D Includes: the center coordinates of the 2D detection frame (x R_2D ,y R_2D ), the length w of the 2D detection box R_2D and width h R_2D ;

[0010] D. Image feature sequence N C , point cloud feature sequence N L , 2D point cloud feature sequence N R Fill the feature dimension to a uniform size, thereby obtaining the first mode C, the second mode L, and the third mode R after filling;

[0011] E. Use a Transformer-based multimodal perception fusion network to process the first modality C, the second modality L, and the third modality R:

[0012] E1. Use the Transformer encoder to associate and map the first modality C and the second modality L, so that the low-order features of the second modality L can potentially adapt to the feature data of the first modality C, and output the fused modality C. * ;

[0013] E2, using Transformer encoder to encode the fusion modality C * Associate and map it with the third modality R, so that the low-order features of the third modality R can be potentially adapted to the fusion modality C. * The feature data of the fusion model is output ** ;

[0014] E3, the final fusion mode C ** The data with confidence less than σ is filtered, and then the non-maximum suppression algorithm is used to remove duplicate data from the filtered data, and finally the predicted detection box of the multimodal perception fusion network is output.

[0015] The characteristic of the multi-sensor fusion target detection method based on Transformer described in the present invention is that the fusion modality C in step E1 is * It is obtained by the following steps:

[0016] A. Processing the first modality C and the second modality L through a first residual block layer, including:

[0017] A1. Normalize the first modality C and the second modality L through the LayerNorm layer to obtain the standardized first modality sequence N C1 And the second modal sequence N L1 ;

[0018] A2. Using μ different linear transformation matrices W q For the first modal sequence N C1 Perform multi-scale linear transformation to obtain the first modal sequence Q after transformation C ;

[0019] Using μ different linear transformation matrices W k and W v For the second modal sequence N L1 Perform multi-scale linear transformation to obtain the transformed second modal sequence K L and V L ;

[0020] A3, sequence Q C , K L 、V L After processing by the cross-modal attention layer, the sequence Y representing the two modal data information is obtained. C ;

[0021] A4. C With the first modal sequence N C1 Perform splicing operation and output the spliced ​​sequence Z C ;

[0022] B. The sequence Z C After the second residual block layer processing, including:

[0023] B1. Sequence Z through LayerNorm layer C Perform standardization and output the standardized sequence Z C1 ;

[0024] B2. For sequence Z C1 Perform nonlinear processing, including: C1 First, through a fully connected layer, the sequence representing the degree of association between the two modes is mapped to a high-dimensional space, and then the mapped result Z C2 After processing by the ReLU activation function, the result Z C3 Then, through a fully connected layer, the sequence representing the degree of association between the two modes is mapped to a low-dimensional space, thereby outputting a sequence Z representing the degree of association. C4 ;

[0025] B3, sequence Z C4 With Z C1 After the splicing operation, the splicing feature map Z is output C5 That is, fusion mode C * .

[0026] The present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the multi-sensor fusion target detection method, and the processor is configured to execute the program stored in the memory.

[0027] The present invention provides a computer-readable storage medium having a computer program stored thereon. The computer-readable storage medium is characterized in that when the computer program is run by a processor, the steps of the multi-sensor fusion target detection method are executed.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The present invention proposes an improved method to address the problem of large deviation in road target detection by a single sensor. The Transformer-based multi-sensor fusion target detection method proposes a multimodal perception fusion network structure based on the traditional Transformer network, which solves the decision-making problem caused by the failure of a single sensor, improves the defect of low accuracy of traditional target detection, and realizes the feature correction between multi-sensor and multi-modal targets.

[0030] 2. The difficulty of traditional projection-based target fusion methods lies in the temporal and spatial matching between different data. To address these difficulties, this paper proposes a Transformer-based multi-sensor fusion target detection method. It can use soft links to realize the association between different modal data, eliminating the joint calibration between multiple sensors in traditional fusion methods, completing the fusion of target features through potential adaptation, and outputting corrected, more accurate and rich road target information. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is an overall flow chart of a Transformer-based multi-sensor fusion target detection method of the present invention;

[0032] Figure 2 A flowchart of the present invention for associating the output data of a visual object detection network with the output data of a point cloud object detection network;

[0033] Figure 3 This is a flow chart of the present invention for associating the fusion data of vision and lidar with the detection output data of millimeter-wave radar. DETAILED DESCRIPTION

[0034] In this embodiment, a multi-sensor fusion target detection method based on Transformer is used. Figure 1 As shown, the following steps are included:

[0035] A. The camera acquires raw image data and inputs it into the pre-trained object detection neural network YOLOv4 for processing, and outputs the image recognition results of road objects, mainly including:

[0036] A1. Use the first 10,000 images from the public dataset RADIATE. Use 8,750 images as the training set and the remaining images as the validation set, with a training set to validation set ratio of 7:1. Only vehicles and pedestrians are labeled.

[0037] A2. Using the object detection neural network YOLOv4, input the images in the training set into the network, output several object detection boxes, and perform backpropagation by matching the predicted boxes with the true values ​​in the dataset. A loss function is established and the network is trained using gradient descent until the set number of iterations, 18,000, is reached. This results in a trained object detection neural network.

[0038] Among them, the image feature sequence composed of the image recognition results of any road target is recorded as N C , including: category information of road targets class cam and 2D detection box data bbox cam_2D And confidence score cam , 2D detection box data bbox cam_2D Includes: the center coordinates of the 2D detection frame (x C_2D ,y C_2D ), the length w of the 2D detection box C_2D and width h C_2D ;

[0039] B. The LiDAR collects point cloud data and inputs it into the pre-trained point cloud target detection network Pointpillars for processing, and outputs the point cloud recognition results of road targets, which mainly include:

[0040] B1. Using the lidar point cloud image from the public dataset RADIATE, we first filter out the lidar ground point cloud. That is, we judge the point cloud in the column. If the column is the ground, we can leave it blank and replace it with a tensor of zero value. We then judge the distance difference between the highest and lowest points in the column, that is:

[0041] H ij -h ij <s (1)

[0042] In formula (1), i and j are the x-axis and y-axis coordinates of the column, H is the highest point of the column, h is the lowest point of the column, and s is the height difference threshold. If the distance difference does not exceed the threshold, the point cloud in this column can be judged as the ground.

[0043] B2. The LiDAR point cloud with the ground point cloud filtered out is passed to the Pillar Feature Net (PFN) point cloud feature network for pseudo image processing. A pseudo image of size (C, H, W) is generated, where C represents the number of channels, H represents the height of the pseudo image, and W represents the width of the pseudo image. The (C, H, W) pseudo image is used as input to a two-dimensional convolutional neural network to extract features from the LiDAR point cloud.

[0044] B3. Use the SSD detection module to identify the extracted features, perform feature matching between the prior box and the annotated ground truth, perform backpropagation, establish a loss function, and train the network using the gradient descent method until the set number of iterations reaches 300, thereby obtaining a trained point cloud detection neural network;

[0045] B4. Input the point cloud information collected by the laser radar into the trained point cloud target detection neural network in real time to obtain the data of the road target. The point cloud feature sequence composed of the point cloud recognition results of any road target is recorded as N L , including: category information of road targets class lidar , 2D detection box data bbox lidar_2D , 3D detection box data bbox lidar_3D And confidence score lidar ; 2D detection box data bbox lidar_2D Including the center coordinates of the 2D detection box (x L_2D ,y L_2D ), the length w of the 2D detection box L_2D and width h L_2D , 3D detection box data bbox lidar_3D Includes: the center coordinates of the 3D detection frame (x L_3D ,y L_3D , z L_3D ), the length w of the 3D detection box L_3D 、Width h L_3D and Gaol L_3D ;

[0046] C. Using the RANSAC algorithm, the original ground point cloud and noise point cloud acquired by the millimeter-wave radar are filtered out, and the point cloud data of all road targets are segmented. The original 2D point cloud data is collected by the millimeter-wave radar and clustered. All road target point clouds are clustered into multiple point cloud clusters. Each point cloud cluster contains all the point clouds of a road target. The clustering results of the road targets are obtained. The 2D point cloud feature sequence composed of the clustering results of any road target is recorded as N R , including: category information of road targets class radar , 2D detection box data bbox radar_2D, road target speed information velocity radar And confidence score radar ; 2D detection box data bbox radar_2D Includes: the center coordinates of the 2D detection frame (x R_2D ,y R_2D ), the length w of the 2D detection box R_2D and width h R_2D ;

[0047] D. Image feature sequence N C , point cloud feature sequence N L , 2D point cloud feature sequence N R Fill the feature dimensions with 0 to a uniform size. In this embodiment, the uniform size is set to 20, thereby obtaining the first mode C, the second mode L, and the third mode R after filling.

[0048] E. Use a Transformer-based multimodal perception fusion network to process the first modality C, the second modality L, and the third modality R:

[0049] E1, such as Figure 2 As shown, the first modality C and the second modality L are input through the first residual block layer, and the input is first normalized through the LayerNorm layer, that is:

[0050] N C1 =LayerNorm(N C ) (2)

[0051] N L1 =LayerNorm(N L ) (3)

[0052] In formula (2), N C1 is the result of mode C after standardization;

[0053] In formula (3), N L1 is the result after the mode L is standardized,

[0054] Secondly, using μ different linear transformation matrices W q For the first modal sequence N C1 Perform multi-scale linear transformation to obtain the first modal sequence Q after transformation C , using μ different linear transformation matrices W k and W v For the second modal sequence N L1 Perform multi-scale linear transformation to obtain the transformed second modal sequence K L and V L, the present invention sets μ=8, that is:

[0055] Q C =N C1 W q (4)

[0056] K L =N L1 W k (5)

[0057] V L =N L1 W v (6)

[0058] In formula (4), Q C N C1 The sequence obtained after linear transformation;

[0059] In formula (5) and formula (6), K L and V L N L1 The sequence obtained after linear transformation is mapped into a feature sequence with special expression information through linear transformation;

[0060] The sequence Q C , K L 、V L After processing by the cross-modal attention layer, the sequence Y representing the two modal data information is obtained. C ,Right now:

[0061]

[0062] In formula (7), Y C To associate the sequences representing the two modal data information, the sequence Y C With the first modal sequence N C1 Perform splicing operation and output the spliced ​​sequence Z C Right now:

[0063] Z C =addition(N C1 , Y C ) (8)

[0064] In formula (8), Z C is the result sequence after the splicing operation;

[0065] E2, the output of step E1 is processed by the second residual layer, and the sequence Z is processed by the LayerNorm layer. C Perform standardization and output the standardized sequence Z C1 ;Right now:

[0066] Z C1 =LayerNorm(Z C ) (9)

[0067] In formula (9), Z C1 For sequence Z C After the normalization process, the sequence Z C1 Perform nonlinear processing, including: C1 First, through a fully connected layer, the sequence representing the degree of association between the two modes is mapped to a high-dimensional space, and then the mapped result Z C2 After processing by the ReLU activation function, the result Z C3 Then, through a fully connected layer, the sequence representing the degree of association between the two modes is mapped to a low-dimensional space, thereby outputting a sequence Z representing the degree of association. C4 ;Right now:

[0068] Z C2 =fully connected(Z C1 )=w1·Z C1 +b1 (10)

[0069] Z C3 =ReLU(Z C2 )=max(0,Z C2 ) (11)

[0070] Z C4 =fully connected(Z C3 )=w2·Z C3 +b2 (12)

[0071] In formula (10), Z C2 is composed of the sequence Z C1 The result obtained after processing by the fully connected layer, where w1 and b1 are the fitting linear parameters, respectively, indicating that the two modal correlation degree sequences are mapped into a high-dimensional space;

[0072] In formula (11), Z C3 is composed of the sequence Z C2 The result obtained after processing by the ReLU activation function;

[0073] In formula (12), Z C4 is composed of the sequence Z C3 The result obtained after processing by the fully connected layer, where w2 and b2 are the fitting linear parameters, respectively, indicating that the correlation degree sequence of the two modes is mapped from the high-dimensional space to the low-dimensional space;

[0074] Finally, the sequence Z C4 With Z C1After the splicing operation, the splicing feature map Z is output C5 ,Right now:

[0075] Z C5 =addition(Z C1 , Z C4 ) (13)

[0076] In formula (13), Z C5 is the result sequence after the splicing operation, that is, the fusion mode C * .

[0077] E3, such as Figure 3 As shown, the fusion mode C * The third modality R input is processed by the first residual block layer, and the input is first normalized by the LayerNorm layer, that is:

[0078] N C*1 =LayerNorm(Z C5 ) (14)

[0079] N R1 =LayerNorm(N R ) (15)

[0080] In formula (14), N C*1 For mode C * The results after standardization;

[0081] In formula (15), N R1 is the result of the normalization of the modal R;

[0082] Secondly, using μ different linear transformation matrices W q For the fusion modality sequence N C*1 Perform multi-scale linear transformation to obtain the transformed fusion modal sequence Using μ different linear transformation matrices W k and W v For the third modal sequence N R1 Perform multi-scale linear transformation to obtain the transformed third modal sequence K R and V R In this embodiment, μ is set to 8, that is:

[0083] Q C* =N C*1 W q (16)

[0084] K R =N R1 W k (17)

[0085] V R =N R1 W v (18)

[0086] In formula (16), N C*1 The sequence obtained after linear transformation;

[0087] In formula (17) and formula (18), K R and V R N R1 The sequence obtained after linear transformation is mapped into a feature sequence with special expression information through linear transformation;

[0088] will sequence K R 、V R After processing by the cross-modal attention layer, a sequence representing the two modal data information is obtained. Right now:

[0089]

[0090] In formula (19), To associate the sequences representing the two modal data information, the sequence With the third modal sequence N C*1 Perform splicing operation and output the spliced ​​sequence Right now:

[0091]

[0092] In formula (20), is the result sequence after the splicing operation;

[0093] E4, the output of step E3 is processed by the second residual layer, and the sequence is normalized by the LayerNorm layer. Perform standardization and output the standardized sequence Z C*1 ,Right now:

[0094]

[0095] In formula (21), Z C*1 For sequence After the normalization process, the sequence Z C*1 Perform nonlinear processing, including: C*1 First, through a fully connected layer, the sequence representing the degree of association between the two modes is mapped to a high-dimensional space, and then the mapped result Z C*2 After processing by the ReLU activation function, the result ZC*3 Then, through a fully connected layer, the sequence representing the degree of association between the two modes is mapped to a low-dimensional space, thereby outputting a sequence Z representing the degree of association. C*4 ;Right now:

[0096] Z C*2 =fully connected(Z C*1 )=w3·Z C*1 +b3 (22)

[0097] Z C*3 =ReLU(Z C*2 )=max(0,Z C*2 ) (twenty three)

[0098] Z C*4 =fully connected(Z C*3 )=w4·Z C*3 +b4 (24)

[0099] In formula (22), Z C*2 is composed of the sequence Z C*1 The result obtained after processing by the fully connected layer, where w3 and b3 are the fitting linear parameters, respectively, indicating that the two modal correlation degree sequences are mapped into a high-dimensional space;

[0100] In formula (23), Z C*3 is composed of the sequence Z C*2 The result obtained after processing by the ReLU activation function;

[0101] In formula (24), Z C*4 is composed of the sequence Z C*3 The result obtained after processing by the fully connected layer, where w4 and b4 are the fitting linear parameters, respectively, indicating that the two modal correlation degree sequences are mapped from high-dimensional space to low-dimensional space;

[0102] Finally, the sequence Z C*4 With Z C*1 After the splicing operation, the splicing feature map Z is output C*5 ,Right now::

[0103] Z C*5 =addition(Z C*1 , Z C*4 ) (25)

[0104] In formula (25), Z C*5 is the result sequence after the splicing operation, that is, the fusion mode C ** , where the image feature sequence composed of the image recognition results of any road target is recorded as Includes: Class information of the fused road target class fusion , 2D detection frame data bbox after fusion fusion , velocity information of the detected target after fusion fusion , confidence score of the detected target after fusion fusion , 2D detection box data bbox fusion Includes: the center coordinates of the 2D detection frame (x fusion ,y fusion ), the length w of the 2D detection box fusion and width h fusion ;

[0105] E5, the final fusion mode C ** The data with confidence less than σ is filtered, and then the non-maximum suppression algorithm is used to remove duplicate data from the filtered data, and finally the predicted detection box of the multimodal perception fusion network is output.

[0106] In this embodiment, an electronic device includes a memory and a processor. The memory is used to store a program that supports the processor to execute the above-mentioned multi-sensor fusion target detection method, and the processor is configured to execute the program stored in the memory.

[0107] In this embodiment, a computer-readable storage medium stores a computer program, which executes the steps of the multi-sensor fusion target detection method when executed by a processor.

Claims

1. A multi-sensor fusion target detection method based on Transformer network, characterized in that: The steps include: A. The original image data is obtained by the camera and input into the pre-trained target detection neural network YOLOv4 for processing, and the image recognition results of the road target are output. Among them, the image feature sequence composed of the image recognition results of any road target is recorded as N C , including: category information of road targets class cam and 2D detection box data bbox cam_2D And confidence score cam , the 2D detection box data bbox cam_2D Includes: the center coordinates of the 2D detection frame (x C_2D ,y C_2D ), the length w of the 2D detection box C_2D and width h C_2D ; B. The point cloud data is collected by the lidar and input into the pre-trained point cloud target detection network Pointpillars for processing, and the point cloud recognition results of the road target are output. Among them, the point cloud feature sequence composed of the point cloud recognition results of any road target is recorded as N L , including: category information of road targets class lidar , 2D detection box data bbox lidar_2D , 3D detection box data bbox lidar_3D And confidence score lidar ; The 2D detection box data bbox lidar_2D Including the center coordinates of the 2D detection box Length of the 2D detection box He Kuan The 3D detection box data bbox lidar_3D Includes: center coordinates of the 3D detection frame Length of the 3D detection frame Width and high C. Use millimeter wave radar to collect original 2D point cloud data and perform clustering processing to obtain the clustering results of road targets. The 2D point cloud feature sequence composed of the clustering results of any road target is recorded as N R , including: category information of road targets class radar , 2D detection box data bbox radar_2D , road target speed information velocity radar And confidence score radar ; The 2D detection box data bbox radar_2D Includes: the center coordinates of the 2D detection frame (x R_2D ,y R_2D ), the length w of the 2D detection box R_2D and width h R_2D ; D. Image feature sequence N C , point cloud feature sequence N L , 2D point cloud feature sequence N R Fill the feature dimension to a uniform size, thereby obtaining the first mode C, the second mode L, and the third mode R after filling; E. Use a Transformer-based multimodal perception fusion network to process the first modality C, the second modality L, and the third modality R: E1. Use the Transformer encoder to associate and map the first modality C and the second modality L, so that the low-order features of the second modality L can potentially adapt to the feature data of the first modality C, and output the fused modality C. * ; E2, using Transformer encoder to encode the fusion modality C * Associate and map it with the third modality R, so that the low-order features of the third modality R can be potentially adapted to the fusion modality C. * The feature data of the fusion model is output ** ; E3, the final fusion mode C ** The data with confidence less than σ is filtered, and then the non-maximum suppression algorithm is used to remove duplicate data from the filtered data, and finally the predicted detection box of the multimodal perception fusion network is output.

2. The multi-sensor fusion target detection method based on Transformer according to claim 1, characterized in that: In step E1, the fusion modality C * It is obtained by the following steps: A. Processing the first modality C and the second modality L through a first residual block layer, including: A1. Normalize the first modality C and the second modality L through the LayerNorm layer to obtain the standardized first modality sequence N C1 And the second modal sequence N L1 ; A2. Using μ different linear transformation matrices W q For the first modal sequence N C1 Perform multi-scale linear transformation to obtain the first modal sequence Q after transformation C ; Using μ different linear transformation matrices W k and W v For the second modal sequence N L1 Perform multi-scale linear transformation to obtain the transformed second modal sequence K L and V L ; A3, sequence Q C , K L 、V L After processing by the cross-modal attention layer, the sequence Y representing the two modal data information is obtained. C ; A4. C With the first modal sequence N C1 Perform splicing operation and output the spliced ​​sequence Z C ; B. The sequence Z C After the second residual block layer processing, including: B1. Sequence Z through LayerNorm layer C Perform standardization and output the standardized sequence Z C1 ; B2. For sequence Z C1 Perform nonlinear processing, including: C1 First, through a fully connected layer, the sequence representing the degree of association between the two modes is mapped to a high-dimensional space, and then the mapped result Z C2 After processing by the ReLU activation function, the result Z C3 Then, through a fully connected layer, the sequence representing the degree of association between the two modes is mapped to a low-dimensional space, thereby outputting a sequence Z representing the degree of association. C4 ; B3, sequence Z C4 With Z C1 After the splicing operation, the splicing feature map Z is output C5 That is, fusion mode C * .

3. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the multi-sensor fusion target detection method according to claim 1 or 2, and the processor is configured to execute the program stored in the memory.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-sensor fusion target detection method according to claim 1 or 2 are executed.

Images