Three-dimensional object detection method based on transformer multi-modal feature fusion

Abstract

The application discloses a three-dimensional target detection method based on a multi-modal feature fusion of a Transform, and comprises the following steps: 1. collecting point cloud data by using a laser radar and sampling the point cloud data, and collecting image data by using a camera; 2. inputting the data collected by the laser radar and the camera into a multi-modal feature fusion RPN network based on a Transform to extract a region proposal frame; and 3. inputting the region proposal frame information into a refinement network to obtain a final prediction frame. The application can avoid the problems of false detection and missed detection in the target detection process based on the multi-modal feature fusion, so that the accuracy of the evaluation of the perceived environment can be ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of environment perception based on multi-sensor fusion, in particular to a three-dimensional target detection method based on multi-modal feature fusion of Transformer. BACKGROUND

[0002] Current 3D target detection techniques can be roughly divided into laser radar only method and multi-modal method. Laser radar only method uses point cloud captured by laser radar to extract depth and geometric structure information to understand 3D scene. However, the point cloud collected by laser radar only lacks semantic information, so it may confuse foreground and background with similar structures and cause false detection, interfering with normal driving. At the same time, the sparsity of point cloud affects the accuracy of laser radar only method in identifying remote objects and small objects.

[0003] In this case, multi-modal method is increasingly concerned, and researchers are committed to perceiving the surrounding environment by fusing information collected by laser radar and camera. Existing multi-modal fusion methods can be divided into pre-fusion and post-fusion. The post-fusion method is robust, but is limited by the detection accuracy of single modal detector, and the accuracy needs to be improved. The pre-fusion method can be divided into data level fusion and feature level fusion, and its accuracy is usually better than that of post-fusion method, but it is difficult to solve the alignment problem of heterogeneous data. Specifically, MV3D and AVOD algorithms predict by fusing regional features. Huang Tengteng et al. estimate the importance of multi-modal features by learning, and generate weights between multi-modal to fuse multi-modal features. However, the above methods are difficult to accurately establish the corresponding relationship between different sensor data during execution, resulting in that the semantic information contained in the point cloud features and image features participating in fusion is not consistent, and the feature map after fusion cannot accurately describe the actual situation, which seriously affects the accuracy of detection. SUMMARY

[0004] The present application provides a three-dimensional target detection method based on multi-modal feature fusion of Transformer to solve the problems in the prior art, so as to realize multi-source heterogeneous feature fusion in the three-dimensional detection process, thereby ensuring the accuracy of the perceived environment evaluation.

[0005] To achieve the above application purposes, the present application adopts the following technical solutions:

[0006] The three-dimensional target detection method based on multi-modal feature fusion of Transformer has the following characteristics:

[0007] A, using a camera to collect images P of a target scene i Meanwhile, the point cloud data of the target scene collected by the laser radar is sampled to obtain a laser radar feature sequence Fp ; wherein the image P i has a dimension of (H r , W r , 3), wherein H r represents a height of the image P i , W r represents a width of the image P i , and 3 represents a number of RGB channels of the image P i ;

[0008] the laser radar feature sequence F p has a dimension of (N r , 4), wherein N r represents a number of point cloud data, and 4 represents a number of information of each point cloud data; the information includes a center coordinate (x o , y o , z o ) and a reflection intensity r o ;

[0009] B, establishing a multi-modal feature fusion RPN network based on a Transformer, comprising: K SetAbstraction layers, K+1 convolution modules, K+1 Transformer multi-modal feature fusion modules, K Feature Propagation layers, a two-dimensional transpose convolution layer, a classification layer and a regression layer; wherein the convolution module comprises, in sequence: a first convolution layer, a batch normalization layer, a ReLU activation function layer and a second convolution layer;

[0010] B1, defining a current iteration number as k, and initializing k=0, and letting K represent a total number of iterations;

[0011] taking the image P i as image features F i k of the kth iteration, and taking the laser radar feature sequence F p as laser radar point cloud features

[0012] B2, the k+1th SetAbstraction layer encodes the laser radar point cloud features F p k of the kth iteration to obtain high-level laser radar point cloud features of the k+1th iteration, and the dimension of is wherein N r / (4(k+1)) represents a number of point cloud data of , and represents The number of feature channels contained in each point cloud data;

[0013] B3, the (k+1)th convolutional module processes the image features F in the kth iteration. i k Encode the high-level image features F for the (k+1)th iteration. i k+1 And F i k+1 The dimension is Among them, H r / 2 k+1 ×W r / 2 k+1 F represents i k+1 The number of feature element points included. F represents i k+1 The number of feature channels contained in each feature element point;

[0014] B4. The (k+1)th Transformer multimodal feature fusion module performs high-level LiDAR point cloud features on the (k+1)th iteration. and the high-level image features F in the (k+1)th iteration i k+1 Feature fusion is performed to obtain the enhanced point cloud features for the (k+1)th iteration.

[0015] B5, will Assign to F i k+1 Assigned to F i k After assigning k+1 to k, return to step B2 and execute sequentially until k = K, thereby obtaining K high-level lidar point cloud features. and high-level image features {F i k |k=1,2,…,K};

[0016] B6, K Feature Propagation layers sequentially... In Perform K decoding iterations to obtain the final point cloud features. and The dimension is (N) r / 4,C p ), where C p express The number of feature channels for each point cloud data;

[0017] B7. The two-dimensional transposed convolutional layer pairs {Fi k |k=1,2,…,K} are decoded to obtain the transposed image features wherein, denotes high-level image features F i k the transposed image features;

[0018] B8, the concatenated and input into the K+1th convolution module to obtain the final image features F i K+1 , and the dimension of F i K+1 is (H r / 2,W r / 2,C i ), wherein C i represents the number of feature channels of each point data in F i K+1 ;

[0019] B9, the K+1th Transformer multi-modal feature fusion module performs feature fusion on the final point cloud features and the final image features F i K+1 to obtain point cloud features F ep with fused semantic information, and the dimension of F ep is (N r / 4,C), wherein C represents the number of feature channels of each point cloud data in F ep ;

[0020] B10, F ep is input into the classification layer and the regression layer respectively for processing to obtain region proposal information, including: region proposal object class type RPN , region proposal object 2D bounding box size bbox RPN , region proposal object 3D bounding box size dimensions RPN , region proposal object 3D bounding box location location RPN , region proposal object 3D bounding box space direction rotation RPN , and region proposal network detection confidence score RPN ;

[0021] C, the region proposal information is input into the refinement network for processing to obtain final prediction frame information, including: final prediction object class type result , final prediction object 2D prediction frame size bbox resultThe final predicted object's 3D prediction box dimensions result The final predicted 3D bounding box location of the object. result The final predicted object's 3D prediction box spatial orientation rotation result and the confidence score of the network's final prediction result .

[0022] The 3D target detection method based on Transformer multimodal feature fusion described in this invention is characterized in that the Transformer multimodal feature fusion module in B4 includes: two fully connected layers and two Transformer cross-attention modules, and performs feature fusion according to the following steps:

[0023] B4.1, the high-level image features F of the (k+1)th iteration i k+1 Compressed to a dimension of High-level compressed image features in the (k+1)th iteration Then, the high-level lidar point cloud features of the (k+1)th iteration are... With the high-level compressed image features The data are input into the first fully connected layer for processing to obtain the LiDAR point cloud features to be fused in the (k+1)th iteration. The high-level image features F to be fused with the (k+1)th iteration i ′ k+1 ,in, The dimension is (N) r / (4(k+1)),C k+1 ), F i ′ k+1 The dimension is ((H) r ×W r ) / 4 k+1 C k+1 ), C k+1 The number of channels representing the features to be fused;

[0024] B4.2, The first Transformer cross-attention module With F i ′ k+1 Processing is performed to obtain the adaptive point cloud features for the (k+1)th iteration. in, The dimension is ((H) r ×W r ) / 4 k+1 C k+1 );

[0025] B4.3, Fi k+1 with obtaining the cascaded connection image feature of the k+1th iteration obtaining the enhanced image feature of the k+1th iteration by inputting the second full connection layer wherein, the dimension of is ((H r ×W r ) / 4 k+1 , 2C k+1 ), the dimension of is ((H r ×W r ) / 4 k+1 , C k+1 );

[0026] B4.4, the second Transformer cross attention module pair and obtaining the adaptive image feature of the k+1th iteration wherein, the dimension of is (N r / (4(k+1)), C k+1 );

[0027] B4.5, after the concatenation of and obtaining the enhanced point cloud feature of the k+1th iteration wherein, the dimension of is (N r / (4(k+1)), 2(C k+1 )).

[0028] The electronic device comprises a memory and a processor, and the characteristic of the electronic device is that the memory is used for storing a program supporting the processor to execute the three-dimensional target detection method, and the processor is configured to execute the program stored in the memory.

[0029] The computer readable storage medium comprises a computer program stored thereon, and the characteristic of the computer readable storage medium is that the computer program is executed by the processor to execute the steps of the three-dimensional target detection method.

[0030] Compared with the prior art, the beneficial effects of the present application are that:

[0031] ​1.The present application aims at the problem of heterogeneous data alignment difficulty in three-dimensional target detection method based on multi-sensor feature level fusion, and proposes a multi-modal feature fusion three-dimensional target detection method based on Transformer, which uses the characteristics of traditional Transformer network that can dynamically estimate the correlation between two modalities, so that the network automatically aligns multi-source heterogeneous data, thereby improving the accuracy of target detection based on multi-sensor fusion, and is beneficial to the improvement of intelligent vehicle environmental perception ability.

[0032] 2.The present application proposes a bidirectional enhancement fusion strategy for multi-source heterogeneous features based on Transformer.Compared with previous multi-modal fusion methods based on Transformer, the method of the present application first extracts attention-based adaptive point cloud features using a Transformer network, then cascades image features and adaptive point cloud features to enhance image features, then generates attention-based adaptive image features based on a Transformer network using enhanced image features, and finally cascades point cloud features and adaptive image features to obtain point cloud features enhanced by semantic information, solving the problem that the multi-modal feature fusion strategy based on Transformer is too single and cannot fully utilize the superiority of the Transformer network, thereby optimizing the adaptability of feature level fusion and improving the accuracy of target detection based on multi-sensor fusion, which is beneficial to the improvement of intelligent vehicle environmental perception ability. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The overall flowchart of the multi-modal feature fusion three-dimensional target detection method of the present application is shown in the figure.

[0034] Figure 2 The flowchart of the bidirectional enhancement fusion strategy for multi-source heterogeneous features based on Transformer of the present application is shown in the figure.

[0035] Figure 3 The machine vision detection effect diagram of the present application is shown in the figure. DETAILED DESCRIPTION

[0036] In this embodiment, a multi-modal feature fusion three-dimensional target detection method based on Transformer includes the following steps as shown in the figure. Figure 1

[0037] A, using a camera to collect the image P of the target scene i , and simultaneously using a laser radar to collect the point cloud data of the target scene and then sampling, the sampling process includes: recording the number of point clouds collected by the laser radar as N o , and stipulating that the number of point clouds after sampling N r , if N o >N r ​Then, the point cloud collected by the lidar is randomly downsampled until the number of point clouds is N. r If N o <N r Then, the point cloud collected by the lidar is randomly copied until the number of point clouds reaches N. r Output lidar feature sequence F p , where, where, image P i The dimension is (H) r W r ,3), where H r Represents image P i Height, W r Represents image P i The width, 3 represents the image P i The number of RGB channels;

[0038] LiDAR feature sequence F p The dimension is (N) r ,4), where N r This indicates the number of point cloud data points, where 4 represents the amount of information in each point cloud data point; the information includes the center coordinates (x, y, 0). o ,y o ,z o and reflection intensity r o In this embodiment, N is set. o =16384;

[0039] B. The Transformer-based multimodal feature fusion RPN network consists of: K PointNet++ Set Abstraction (SA) layers, K+1 convolutional modules, K+1 Transformer multimodal feature fusion modules, K PointNet++ Feature Propagation (FP) layers, a two-dimensional transposed convolutional layer, a classification layer, and a regression layer; wherein the convolutional modules sequentially include: a first convolutional layer, a batch normalization layer, a ReLU activation function layer, and a second convolutional layer;

[0040] B1. Define the current iteration number as k, and initialize k = 0. Let K represent the total number of iterations.

[0041] Image P i Image features F in the kth iteration i k LiDAR feature sequence F p F, representing the point cloud features of the lidar in the k-th iteration. p k ;

[0042] B2, the (k+1)th Set Abstraction layer's LiDAR point cloud features for the k-th iteration F p k Encoding is performed to obtain the high-level lidar point cloud features of the (k+1)th iteration. This includes: using the FPS algorithm to select N with the largest mutual distance. o In this embodiment, N points are set. o '=N r / (4(k+1)); then using the ball query algorithm, with the above N o 'A point is the centroid, and in a radius of r o Sampling K within the sphere o In this embodiment, r is set to [number of points]. o =0.5(k+1), K o =32; Finally, this N o Each sphere is input into a PointNet layer to obtain high-level LiDAR point cloud features. in, The dimension is Where, N r / (4(k+1)) represents The number of point cloud data points express The number of feature channels contained in each point cloud data;

[0043] B3, the (k+1)th convolutional module's image features F during the k-th iteration. i k Encode the high-level image features F for the (k+1)th iteration. i k+1 , of which F i k+1 The dimension is H r / 2 k+1 ×W r / 2 k+1 F represents i k+1 The number of feature element points included. F represents i k+1 The number of feature channels contained in each feature element point;

[0044] B4. The (k+1)th Transformer multimodal feature fusion module performs high-level LiDAR point cloud features on the (k+1)th iteration. and the high-level image features F in the (k+1)th iteration i k+1 Feature fusion is performed to obtain the enhanced point cloud features for the (k+1)th iteration. The flowchart of the Transformer multimodal feature fusion module is as follows: Figure 2 As shown:

[0045] B4.1, the high-level image features F in the (k+1)th iteration i k+1 Compressed to a dimension of High-level compressed image features in the (k+1)th iteration Then, the high-level lidar point cloud features of the (k+1)th iteration are... High-level compressed image features The data are input into the first fully connected layer for processing to obtain the LiDAR point cloud features to be fused in the (k+1)th iteration. The high-level image features F to be fused with the (k+1)th iteration i ′ k+1 ,in, The dimension is (N) r / (4(k+1)),C k+1 ), F i ′ k+1 The dimension is ((H) r ×W r ) / 4 k+1 C k+1 ), C k+1 This represents the number of channels for the features to be fused; in this invention, the following settings are used:

[0046] B4.2, The first Transformer cross-attention module With F i ′ k+1 Processing is performed to obtain the adaptive point cloud features for the (k+1)th iteration. Obtain adaptive point cloud features The Transformer cross-attention module is represented by equation (1), which includes: first, the high-level image features F to be fused are... i k+1 'Mapping to Image Query Matrix' Simultaneously, the high-level lidar point cloud features to be fused will be... Mapping to point cloud key matrix and point cloud value matrix Right now:

[0047]

[0048]

[0049]

[0050] In formula (1)-(3), W i Q 、 are learnable linear transformation matrices;

[0051] Then the image Query matrix is multiplied by the point cloud Key matrix , and is subjected to Softmax processing to obtain a point cloud attention weight matrix , that is:

[0052]

[0053] Then the point cloud attention weight matrix is multiplied by the point cloud Value matrix to obtain adaptive point cloud features , that is:

[0054]

[0055] In formula (5), the dimension of is ((H r ×W r ) / 4 k+1 , C k+1 );

[0056] B4.3, F i ′ k+1 is concatenated with to obtain a concatenated image feature of the k+1th iteration , which is input into a second fully connected layer for processing to obtain an enhanced image feature of the k+1th iteration , that is:

[0057]

[0058] In formula (6), FC represents a fully connected layer, represents Concatenation concatenation;

[0059] wherein, the concatenation refers to splicing in the 2nd dimension of the feature tensor, the concatenation refers to splicing in the 2nd dimension of the feature tensor, the size of the concatenated image feature is ((H r ×W r ) / 4 k+1 , 2C k+1 ), and the enhanced image feature output by the fully connected layer is a feature tensor with a size of ((H r ×W r ) / 4 k+1 , C k+1 );

[0060] B4.4, processing the second high-level lidar point cloud feature and using a second Transformer cross-attention module, the adaptive image feature of the k+1th iteration is where the Transformer cross-attention module is characterized by formula (7), including: first, the to-be-fused high-level lidar point cloud feature is mapped to a point cloud query matrix The enhanced image feature is mapped to an image key matrix and an image value matrix V i k+1 that is:

[0061]

[0062]

[0063] V i k+1 = F i k+1 '·W i V (9)

[0064] In formula (7) to formula (9), W i K , W i V are all learnable linear transformation matrices;

[0065] Then, the point cloud query matrix is multiplied by the image key matrix , and is subjected to Softmax processing to obtain an image attention weight matrix that is:

[0066]

[0067] Finally, the image attention weight matrix is multiplied by the image value matrix V i k+1 to obtain the adaptive image feature that is:

[0068]

[0069] In formula (11), the adaptive image feature is a feature tensor with a size of (N r / (4(k+1)), C k+1 )

[0070] B4.5, concatenate the high-level lidar point cloud features to be fused with adaptive image features to obtain enhanced point cloud features i.e.

[0071]

[0072] assign the enhanced point cloud features to the lidar point cloud features wherein the concatenation refers to splicing in the 2nd dimension of the feature tensor, and the enhanced point cloud feature of the k+1th iteration is a feature tensor with a size of (N r / (4(k+1)), 2(C k+1 ))

[0073] B5, assign to assign F i k+1 to F i k After k+1 is assigned to k, return to step B2 for sequential execution until k=K, thereby obtaining K high-level lidar point cloud features and high-level image features {F i k |k=1,2,…,K}P i 4 ;

[0074] B6, K Feature Propagation layers sequentially decode in K times, including up-sampling the features by using the IDW interpolation method, which can be expressed as:

[0075]

[0076] In formula (13), ω i (x) calculates the point weight, which is inversely proportional to the distance, and the closer the distance, the greater the influence. f i (j) represents the value of the known point, f (j) represents the value of the unknown point, s represents taking s points in the known point set for interpolation calculation, and e represents the influence degree of the distance on the weight;

[0077] After the above operation, the final point cloud feature is obtained, wherein the dimension of is (N r / 4, Cp ), C p express The number of feature channels for each point cloud data;

[0078] B7, Two-dimensional transposed convolutional layer pairs {F i k Decode the transposed image features by processing |k=1,2,…,K}. in, Representing high-level image features F i k Transposed image features;

[0079] B8, will After cascading, the input is given to the (K+1)th convolutional module to obtain the final image features F. i K+1 , of which F i K+1 The dimension is (H) r / 2,W r / 2,C i ), C i F represents i K+1 The number of feature channels for each data point in the dataset;

[0080] B9. The (K+1)th Transformer multimodal feature fusion module performs final point cloud feature processing. and final image features F i K+1 Perform feature fusion to obtain point cloud features F with fused semantic information. ep , of which F ep The dimension is (N) r / 4,C), C represents F ep The number of feature channels for each point cloud data;

[0081] B10, F ep The data is processed separately in the classification and regression layers to obtain region suggestion box information, including: the type of the suggested object. RPN The suggested 2D bounding box size (bbox) of the region object. RPN Region suggestion: 3D bounding box dimensions of objects RPN 3D bounding box location of the suggested object in the region RPN Rotation of the 3D bounding box space of the region suggested object RPN and the confidence score of the regional suggestion network detection RPN The suggested 2D bounding box size (bbox) of the region object. RPN Includes: 2D bounding box center point coordinates (xRPN_2D , y RPN_2D , long w RPN_2D , and width h RPN_2D ; the 3D bounding box dimensions of the region proposal object RPN include: long w RPN_3D , width h RPN_3D , and height l RPN_3D ; the 3D bounding box location of the region proposal object RPN includes: (x RPN_3D , y RPN_3D , z RPN_3D );

[0082] C, input the region proposal frame information in step B10 into the refinement network of the PointRCNN algorithm, the refinement network includes: three SA layers for extracting global feature, and two sub-networks, both of which have two cascaded 1x1 convolution layers for classification and regression, respectively, so as to obtain the final prediction frame information, wherein the final prediction frame information includes: final prediction object type result , final prediction object 2D prediction frame size bbox result , final prediction object 3D prediction frame dimensions dimensions result , final prediction object 3D prediction frame location location result , final prediction object 3D prediction frame space direction rotation result , and network final prediction confidence score result ; the final prediction object 2D prediction frame size bbox result includes: 2D prediction frame center point coordinates (x result_2D , y result_2D ), long w result_2D , and width h result_2D ; the final prediction object 3D prediction frame dimensions dimensions result include: long w result_3D , width h result_3D , and height l result_3D ; the final prediction object 3D prediction frame location location result includes: (x result_3D , y result_3D , z result_3D ), and the detection effect diagram is as shown in Figure 3 , wherein the white frame in the diagram represents the final prediction frame detected by the network.

[0083] In this embodiment, an electronic device is designed, comprising a memory and a processor, the memory is used to store a program supporting the processor to execute the multi-modal feature fusion three-dimensional target detection method, and the processor is configured to execute the program stored in the memory.

[0084] In this embodiment, a computer readable storage medium is designed, and a computer program is stored on the computer readable storage medium. When the computer program is run by a processor, the steps of the multi-modal feature fusion three-dimensional target detection method are executed.

Claims

1. A three-dimensional target detection method based on Transformer multi-modal feature fusion, characterized in that, Comprising the following steps: A. Use a camera to capture images of the target scene. Simultaneously, point cloud data of the target scene is collected using lidar and then sampled to obtain lidar feature sequences. The image The dimension is ,in, Representing an image height, Representing an image width, Representing an image The number of RGB channels; The laser radar feature sequence The dimension of Wherein, The number of point cloud data, The information number of each point cloud data; the information includes the central coordinates And the reflection intensity ; B, a multi-modal feature fusion RPN network based on Transformer is established, comprising: K Set Abstraction layers, K+1 convolution modules, K+1 Transformer multi-modal feature fusion modules, K Feature Propagation layers, a two-dimensional transpose convolution layer, a classification layer and a regression layer; wherein the convolution module comprises in turn: a first convolution layer, a batch normalization layer, a ReLU activation function layer and a second convolution layer; B1, define the current iteration number as and initialize = 0, let K denote the total number of iterations; said image image features as the kth iteration said lidar feature sequence lidar point cloud features as the kth iteration ; B2, the k+1th Set Abstraction layer is applied to the laser radar point cloud features of the kth iteration encoded to obtain high-level laser radar point cloud features of the k+1th iteration , and the dimension of , wherein represents the number of point cloud data represents the number of feature channels contained in each point cloud data B3. The (k+1)th convolutional module processes the image features of the kth iteration. Encode the high-level image features for the (k+1)th iteration. ,and The dimension is ,in, express The number of feature element points included. express The number of feature channels contained in each feature element point; B4. a k+1th Transformer multimodal feature fusion module for performing feature fusion on the high-level lidar point cloud features of the k+1th iteration and the k+1th iteration high-level image features to obtain enhanced point cloud features of the k+1th iteration ; B5、will be assigned to will be assigned to will be assigned to will be assigned to will be assigned to will be assigned to After that, return to step B2 for sequential execution until =K, thereby obtaining K high-level lidar point cloud features and high-level image features ; B6, K Feature Propagation layers sequentially... In Perform K decoding iterations to obtain the final point cloud features. ,and The dimension is ,in, express The number of feature channels for each point cloud data; B7. the two-dimensional transpose convolutional layer pair to obtain a transpose image feature ; wherein represents a high-level image feature transpose image feature B8、will After cascading connection, input the K+1th convolution module to obtain the final image feature , and The dimension of Wherein, Indicates the feature channel number of each point data in ; B9, the K+1th Transformer multimodal feature fusion module performs feature fusion on the final point cloud feature and the final image feature to obtain a point cloud feature with fused semantic information , and the dimension of the point cloud feature is , wherein represents the number of feature channels of each point cloud data in . B10、will Respectively input the classification layer and the regression layer for processing to obtain the region proposal frame information, including: region proposal object category , 2D bounding box size of region proposal object , 3D bounding box size of region proposal object , 3D bounding box position of region proposal object , 3D bounding box space direction of region proposal object , and confidence of region proposal network detection ; C. inputting the region proposal frame information into a refinement network of the PointRCNN algorithm for processing to obtain final prediction frame information, including: final prediction object category , 2D prediction frame size of the final prediction object , 3D prediction frame size of the final prediction object , 3D prediction frame position of the final prediction object , 3D prediction frame space direction of the final prediction object , and confidence of the final prediction of the network .

2. The Transformer-based multi-modal feature fusion three-dimensional object detection method according to claim 1, characterized in that, The Transformer multi-modal feature fusion module in B4 comprises two fully connected layers and two Transformer cross-attention modules, and performs feature fusion according to the following steps: B4.1, the high-level image features of the (k+1)th iteration Compressed to a dimension of High-level compressed image features in the (k+1)th iteration Then, the high-level lidar point cloud features of the (k+1)th iteration are... With the high-level compressed image features The data are input into the first fully connected layer for processing to obtain the LiDAR point cloud features to be fused in the (k+1)th iteration. High-level image features to be fused with the (k+1)th iteration ,in, The dimension is , The dimension is , The number of channels representing the features to be fused; B4.

2. the first Transformer cross-attention module pair is processed to obtain adaptive point cloud features of the k+1th iteration is processed to obtain adaptive point cloud features of the k+1th iteration wherein, the dimension of is ; B4.3, the concatenated to obtain the k+1 iteration of the concatenated image features concatenated to obtain the k+1 iteration of the concatenated image features processed in the second fully connected layer, to obtain the k+1 iteration of the enhanced image features wherein the dimension of the dimension of the , dimension of the ; B4.

4. the second Transformer cross-attention module pair and is processed to obtain the adaptive image feature of the k+1th iteration wherein, has a dimension of ; B4.5、will with After cascading connection, the enhanced point cloud feature of the k+1th iteration is obtained ; wherein, The dimension of .

3. An electronic device comprising a memory and a processor, characterized in that The memory is used to store a program supporting the processor to execute the three-dimensional target detection method of claim 1 or 2, and the processor is configured to execute the program stored in the memory.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to perform the steps of the three-dimensional target detection method of claim 1 or 2.

Images