A road vehicle perception method based on multi-sensor fusion

Abstract

The application discloses a road vehicle perception method based on multi-sensor fusion and belongs to the technical field of unmanned driving. In the road vehicle perception model adopting a double-flow network, an attention mechanism and a Transformer structure are introduced to aggregate different modal features, so that the two modal features are enhanced mutually, the network learns global dependency, and global context information is integrated in the feature extraction stage. By using the self-attention mechanism of the Transformer, the model can naturally perform intra-modal and inter-modal fusion simultaneously and robustly capture potential interactions between the image domain and the laser radar domain, so that the vehicle detection performance is significantly improved, and the limitation of the previous fusion method is improved. By using the re-fusion of RoI, the detection result is refined based on the re-fusion result, and accurate perception of vehicles on the road is realized.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, specifically relating to a road vehicle perception method based on multi-sensor fusion. Background Technology

[0002] In the implementation of autonomous driving technology, multi-sensor fusion technology is often used to fully perceive the global contextual information of a 3D scene. Multi-sensor fusion involves fusing information from multiple different types of sensors to obtain more valuable features. Furthermore, by fusing complementary information from different modalities, the perceptibility, reliability, and robustness of detection algorithms can be further improved. Due to different fusion strategies, multi-sensor deep fusion architectures can be broadly classified into two categories: early fusion and late fusion.

[0003] Early fusion refers to the process where the input representation incorporates information from multiple modalities before feature extraction using deep neural networks. This means the input representation integrates data from different sensors, often at the pixel level. The most common method for this fusion is to perform simple concatenation operations on the input signals, leveraging the correlations and interactions between low-level features of each modality.

[0004] Late fusion, on the other hand, fuses features from different sensor branches, which is typically feature-level fusion. Late fusion uses separate subnetworks to generate feature representations for each modality and usually integrates the encoding results of all individual modality models using fully connected layers. Because late fusion methods can extract different networks for each branch, they can better model each modality of data, achieving greater flexibility.

[0005] Existing early fusion schemes perform fusion operations at the data input layer. However, it is generally believed that the information contained in the data streams of different sensors can only be correlated at a high-level dimension. In particular, the raw data of RGB images and point cloud data relied upon by autonomous driving technology have poor correlation, making it difficult to directly fuse the two types of data using early fusion schemes. In later fusion schemes, new fusion approaches are constantly emerging. The most commonly used approach now is geometry-based fusion, which uses a carefully designed feature projection mechanism to fuse features from two modalities. Under this fusion mechanism, information is usually gathered from the local neighborhood around each feature in the projected two-dimensional or three-dimensional space. Although these methods are better than simply adding or stitching features, we have observed that LiDAR point cloud data is much sparser than RGB images, resulting in very limited fusion capabilities. The limitations in their architectural design hinder their performance in real-world, complex, and dynamic scenes.

[0006] Autonomous vehicles aim to obtain complementary information from different sensors. The core issue in autonomous road vehicle detection is how to effectively fuse features from both modalities. However, real-world 3D scenes often contain numerous complex objects, and because different modalities typically describe the same object differently, accurately fusing information from corresponding objects to form a more effective representation is extremely difficult. Furthermore, the sheer volume of information in the entire 3D scene means that not every feature extracted by the network is valid. Summary of the Invention

[0007] This invention provides a road vehicle perception method based on multi-sensor fusion, which can be used to improve the perception accuracy of autonomous vehicles.

[0008] The technical solution adopted in this invention is as follows:

[0009] A road vehicle perception method based on multi-sensor fusion, the method comprising the following steps:

[0010] Step 1: Obtain target detection data for training the road vehicle perception model. The target detection data includes image data collected by the camera device and lidar point cloud data collected by the laser scanner; and convert the lidar point cloud data into a lidar bird's-eye view.

[0011] Step 2: Construct and train a road vehicle perception model;

[0012] The road vehicle perception model includes a point-level feature fusion module and a region of interest (RoI)-level feature fusion module.

[0013] The point-level feature fusion module includes an image branch backbone network and a LiDAR branch backbone network. The image branch backbone network is used to extract image features from image data at multiple scales, obtaining several first intermediate feature maps at different scales, as well as the final output feature map of the image branch backbone network. The LiDAR branch backbone network is used to extract image features from LiDAR bird's-eye view at multiple scales, obtaining several second intermediate feature maps at different scales, as well as the final output feature map of the LiDAR branch backbone network. The first and second intermediate feature maps have the same number of scales, and the feature map dimensions of the first and second intermediate feature maps at the same level are consistent.

[0014] The first and second intermediate feature maps of the first level are processed by the attention-based fusion module to obtain the first level fused intermediate feature map. The first level fused intermediate feature map is added to the first intermediate feature map of the first level and then continues to participate in the forward calculation on the image branch backbone network. The first level fused intermediate feature map is added to the second intermediate feature map of the first level and then continues to participate in the forward calculation on the lidar branch backbone network.

[0015] Furthermore, starting from the second-level intermediate feature map, the intermediate feature maps of the same level of the image branch backbone network and the lidar branch backbone network are fused through the Transformer-based fusion module to obtain the current-level fused intermediate feature map. Then, it is added to the first and second intermediate feature maps of the current level respectively, and then continues to participate in the forward calculation on the two backbone networks respectively.

[0016] The RoI-level feature fusion module performs road vehicle detection on the final output feature map of the image branch backbone network through convolutional layers, obtaining several 3D candidate boxes and their road vehicle recognition results. After score thresholding and non-maximum suppression, several 3D detection boxes are obtained. The 3D detection boxes are projected onto the LiDAR bird's-eye view space and the two-dimensional image space of the image data, respectively. Then, RoI features of the image data and LiDAR bird's-eye view are obtained through RoI feature extraction. The RoI features of the two modalities are concatenated and input into the thinning module based on at least two fully connected layers to predict the thinning correction of each 3D detection box. The thinning correction result and the 3D detection box are combined to obtain the final road vehicle detection result.

[0017] When training a road vehicle perception model based on target detection data and corresponding label data, the total loss function of the model is the sum of classification loss and regression loss. When the preset training convergence condition is met, a road vehicle perception model based on multi-sensor fusion is obtained, which can be used for road vehicle perception during the driving process of autonomous vehicles.

[0018] The technical solution provided by the invention brings at least the following beneficial effects:

[0019] This invention introduces an attention mechanism and a Transformer structure to aggregate features from different modalities in a road vehicle perception model employing a two-stream network. This allows the two modal features to mutually enhance each other and enables the network to learn global dependencies, integrating global contextual information during the feature extraction stage. Utilizing the self-attention mechanism of the Transformer, the model can naturally perform intra-modal and inter-modal fusion simultaneously and robustly capture potential interactions between the image domain and the LiDAR domain, thereby significantly improving vehicle detection performance and overcoming the limitations of previous fusion methods. Furthermore, this invention utilizes RoI (Region of Interest) re-fusion, and refines the detection results based on the re-fusion results to achieve accurate vehicle perception on the road. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the road vehicle perception model used in a road vehicle perception method based on multi-sensor fusion provided in an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of the Transformer-based fusion module used in the implementation of this invention;

[0023] Figure 3 This is a schematic diagram of the attention-based fusion module used in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0025] Real-world driving tasks are often complex and dynamic. To achieve advanced operations such as obstacle avoidance and traffic rule compliance, global contextual information of the 3D scene is crucial. Although joint processing of camera data and LiDAR data is considered a possible solution, the high sparsity and irregularity of point clouds themselves lead to misalignments between the camera view and the commonly used bird's-eye view (BEV) of LiDAR in terms of modal cooperation, occlusion, and long-distance scale changes. This often results in the fused representation still not fully representing the entire 3D scene, leading to perception blind spots.

[0026] This invention provides a road vehicle perception method based on multi-sensor fusion, proposing a two-level detector with a densely fused dual-stream backbone network and performing feature fusion in two dimensions: point-level feature fusion and RoI (Region of Interest) level feature fusion. In the first stage, the detector employs point-level feature fusion technology in the feature extraction network, utilizing an attention-based fusion module and a Transformer-based fusion module to aggregate information from both modalities, ultimately outputting a small number of high-quality 3D detection boxes. In the second stage, RoI-level feature fusion is applied for more accurate 3D detection box regression, ultimately yielding accurate vehicle path detection results.

[0027] like Figure 1As shown, in this embodiment of the invention, the road vehicle perception model based on multi-sensor fusion includes a two-level fusion mechanism: (1) a point-level feature fusion module and (2) a RoI-level feature fusion module. The point-level feature fusion module includes an image branch backbone network and a lidar branch backbone network. For image features of different scales extracted from the intermediate layers of the two backbone networks, point-level feature fusion is achieved using an attention-based fusion module and a Transformer-based fusion module. For the three-dimensional detection boxes obtained based on the image features finally extracted from the two backbone networks, RoI-level feature fusion is used to perform more accurate three-dimensional detection box regression, resulting in the RoI-level feature fusion result. After processing by the refinement module, the final prediction result is obtained.

[0028] There are various applications for point cloud data. This embodiment of the invention adopts a common method of converting point cloud data into a bird's-eye view. This method is simple to implement and can fully utilize the advantage of LiDAR's accurate depth information. Therefore, this embodiment converts the point cloud data acquired by LiDAR into a 2-bin histogram, which is then input into the corresponding image data. Figure 1 In the model shown, after point-level feature fusion and RoI-level feature fusion, the output is a road vehicle detection box and category information. The two-stage feature fusion is specifically implemented as follows:

[0029] (1) Point-level feature fusion

[0030] Since images and point clouds are two vastly different types of data, it is difficult to process information from both modalities simultaneously using a single network. Therefore, this embodiment of the invention employs a dual-stream network to extract features from both modalities separately. To ensure efficiency in real-time detection on the road, ResNet-34 can be used as the backbone network for the image branch, while for the LiDAR branch, due to its sparsity, ResNet-18 can be used as the backbone network for feature extraction to balance feature extraction capability and computational cost. The fusion operation is performed on feature maps of the two different resolutions.

[0031] The Transformer-based fusion module structure is as follows: Figure 2 As shown, the core idea is to leverage the self-attention mechanism of the Transformer to aggregate complementary global contextual information from image and LiDAR modalities. The aim is to use the Transformer to learn the relationships between the features of the two modalities and achieve long-range dependencies and higher-order spatial interactions through self-attention operations, ultimately extracting a global understanding of the visual scene. The specific fusion operation involves: providing the intermediate feature map F from the two modalities... RGB ∈R C×H×W and F LiDAR ∈R C×H×W Where C represents the number of channels, H×W represents the resolution (i.e., spatial size) of the feature map, and F RGBF LiDAR Expanding the permutation matrix in sequence, each sequence I is formed by discrete labels. RGB ∈R HW ×C and I LiDAR ∈R HW×C Then, the sequences of the two modes are concatenated to form an input sequence I∈R. 2HW×C The 2HW labels in the sequence are each represented by a feature vector of dimension C. Each feature vector is supplemented by a learnable positional encoding to incorporate positional induction bias and distinguish the spatial information of different labels. The input sequence I is linearly projected onto three weight matrices to calculate a set of Q, K, and V values, as shown in the following formula:

[0032] Q = IM q K = IM k V = IM v

[0033] in, The weight matrices for Q, K, and V are respectively, and D q =D k =D v =2HW. The self-attention layer uses the scaled dot product between Q and K to compute attention weights, and then aggregates the values ​​for each query to infer the refined output Z:

[0034]

[0035] To encapsulate multiple complex relationships in different locations within different representation subspaces, a multi-head attention mechanism is employed:

[0036] Z′=MultiHead(Q,K,V)=Concat(Z1,...,Z n M 0

[0037] Where n represents the number of heads in the multi-head attention mechanism. M 0 ∈R nC×2HW This means Concat(Z1,...,Z) n The projection matrix of the input sequence I is then calculated using a nonlinear transformation. The output sequence O, with the same scale as the input sequence I, is then calculated.

[0038] O = MLP(Z″) + Z″

[0039] Here, MLP stands for Multilayer Perceptron, and Z″ = Z′ + I. A Transformer-based fusion module has 6 repeated Transformer blocks, performing self-attention operations multiple times. Finally, following the inverse operation of the permutation matrix in the first step, the input...

[0040] Transform the output sequence O into feature map F' RGB and F' LiDAR This information is then added as supplementary information to the corresponding layers of the original modality coding branches (i.e., the image branch backbone network and the lidar branch backbone network) to complete the point-level feature fusion operation.

[0041] In this embodiment of the invention, the feature extractors for RGB images and LiDAR encode different aspects of the scene at different extraction layers. Therefore, these features can be fused at multiple scales. Considering the high computational cost of processing feature maps at high resolution, global average pooling is used to downsample the high-resolution feature maps of the encoder block to reduce expensive computations. Before passing the feature maps to the Transformer-based fusion module, they are downsampled to a specified resolution, preferably a low and fixed resolution of H = W = 8, and used as input to the fusion module. The output feature maps are upsampled using bilinear interpolation until the original resolution is reached, and then element-wise added to the original feature maps to obtain the fused image feature map and LiDAR feature map, respectively.

[0042] Because the first layer of the network encodes a huge number of low-level features, the above strategy would result in significant feature loss. To balance model performance, this invention employs the following approach in the first layer of the network: Figure 3 The structure shown is used for fusion operations. To fully fuse features, the features of the branch networks are first concatenated, and convolutional layers are used to initially fuse the two modalities, while reducing the number of channels in the fused features to prevent excessive computational cost. The initially fused feature map can be represented as F. fuse ∈R C×H×W :

[0043] F fuse =f([F RGB ;F LiDAR ])

[0044] Among them, F RGB and F LiDAR This represents the intermediate feature maps obtained from the RGB and LiDAR branches, where [·; · represents the concatenation operation, and f· represents the convolution operation. That is, the features of the same size C×H×W from the branch networks are first concatenated, and a convolutional layer is used to initially fuse the two modalities.

[0045] Because the two modalities differ significantly, simple concatenation and convolution operations only perform a lightweight fusion, resulting in a fused feature map F. fuse Most of the features remain modally independent, which fails to effectively aggregate their respective advantageous visual features and thus cannot adequately encode the entire 3D scene. Therefore, embodiments of the present invention require the integration of the insufficiently fused feature map F. fusePerform deep fusion. For feature map F fuse An attention mechanism is applied to it, weighting each feature in the feature map, and the output is an attention map F. map ∈R C×H×W Then combine it with F fuse Deep integration through accumulation:

[0046]

[0047] in, F represents element-wise multiplication. out This represents the feature map after fusing information from the two branches. Finally, the fused feature map is added back to the two branches to complete the fusion of the two modalities.

[0048] For example, corresponding to a feature map scale of 512×8×8, after dense feature fusion under multi-resolution feature maps, each of the two modalities obtains an 8×8×512-dimensional feature map. Finally, mean pooling is used to flatten them into a 512-dimensional feature vector. The element-wise summation of the two vectors yields a 512-dimensional encoding vector, which encodes the global context information of the entire 3D scene, forming a compact representation of the environment.

[0049] The above mechanism constitutes feature fusion at a single scale of the feature map. In conjunction with the Transformer-based fusion module, the embodiments of the present invention apply the fusion operation multiple times to feature maps of different resolutions, so that the model can fuse both the rich semantics of the high-level features of the two modalities and the rich detailed texture features contained in the low-level features of the two modalities.

[0050] (2) RoI-level feature fusion

[0051] The motivation for RoI-level feature fusion is to further refine the localization accuracy of detection in three-dimensional space. To achieve this goal, embodiments of the present invention perform the following fusion operation.

[0052] Considering the diversity of object scales in real 3D scenes, after the point-level feature fusion operation, 1×1 convolutions and bilinear upsampling Feature Pyramid Networks (FPNs) are applied to both streams to combine multi-scale features, fusing information from different levels to form a global representation of the 3D scene. The final feature maps of the two streams have a downsampling factor of 4 compared to the input.

[0053] However, while LiDAR sensors provide accurate depth measurements, the observations are often sparse (especially at long distances) and lack the richness of image information; for example, it is difficult to distinguish between pedestrians and cyclists using a LiDAR detector. Conversely, images are rich in information, and LiDAR provides additional 3D scene distance information, resulting in a more compact environmental representation in the feature map obtained from the RGB branch compared to the LiDAR branch. Therefore, this invention adds a 1×1 convolution to the final RGB branch feature map for dense road vehicle detection, yielding a series of 3D candidate boxes and classifying them. After score thresholding and non-maximum suppression, a smaller but high-quality series of 3D detection boxes are retained. These are then projected onto the LiDAR bird's-eye view space and the 2D image space, and their RoL features are cropped from the final feature map of each stream using precise RoL feature extraction. Specifically, this embodiment uses ROIAlign to extract features from both feature maps, extracting bilinear interpolated features into an n×n (n is a preset value) regular grid. Finally, the RoL features from the two modalities are concatenated and fed into a refinement module with two 256-dimensional fully connected layers to predict the refinement correction for each 3D detection box. The final road vehicle detection result is obtained by combining the refinement correction result with the 3D detection boxes mapped onto the feature map.

[0054] As one possible implementation, an embodiment of the present invention provides a road vehicle perception method based on multi-sensor fusion, comprising the following steps:

[0055] (1) Obtain training data

[0056] This embodiment uses the KITTI object detection dataset for model training, which has 7481 training frames and 7518 test frames. Each frame consists of a pair of data points: an RGB camera image captured by a front-facing camera mounted on the vehicle's roof, and LiDAR point cloud data captured by a LiDAR scanner (Velodyne HDL-64E) mounted on the vehicle's roof. KITTI annotates objects appearing in the camera view with ground-aligned 3D bounding boxes.

[0057] Due to the limited training data for KITTI, this embodiment employs multiple data augmentation techniques to mitigate overfitting. For each frame during training, this embodiment applies random scaling (0.9–1.1 for all three coordinate axes), translation (-5–5 meters for x and y axes, -1–1 meter for z axis), and rotation (-5–5 degrees along the z axis) to the point cloud data, and randomly scales (0.9–1.1) and translates (-50–50 pixels) the camera image. The transformation matrix from LiDAR to the camera is modified accordingly to ensure the correspondence between the two.

[0058] (2) Model Training

[0059] The point cloud data obtained from the lidar in step (1) is converted into a 2-bin histogram using existing technology, and then input with the corresponding image data. Figure 1 In the model shown (road vehicle perception model), after point-level feature fusion and RoI-level feature fusion, the output road vehicle detection box and category information are obtained.

[0060] After the model obtains the prediction results, it uses a loss function to calculate the similarity between the predicted results and the true values ​​provided in the dataset. The difference between the true and predicted values ​​is then used to optimize the model using the backpropagation algorithm. Following common practices in object detection, we define the loss function as the sum of the classification loss and the regression loss, as shown in the following formula:

[0061] L = L cls +λL reg

[0062] Where λ is the balance parameter, and L is the classification loss. cls and regression loss L reg The definitions are as follows:

[0063]

[0064] Where, p c For the predicted classification score, l c The labels are binary (car and background), and N is the total number of samples.

[0065]

[0066] Where (x, y, z) are the coordinates of the 3D bounding box, (w, h, d) represent the dimensions of the bounding box, i.e., the dimensions in the x, y, and z directions, and t represents the direction. N pos Where is the number of positive samples, and D is the smoothed L1 norm, defined as follows:

[0067]

[0068] Where, p k and l k These are the offsets of the predicted and actual values, respectively.

[0069] For k∈(x,y,z):

[0070] p k =(ka) k ) / a k

[0071] Among them, a k Let the coordinates be the anchor points, for k∈(w,h,d):

[0072] pk =log(k / a) k )

[0073] Here a k For anchor point dimensions. Direction offset is simply defined as the difference between the predicted direction and the marked direction:

[0074] p k =ka k

[0075] All training data is fed into the network, completing one forward computation and backpropagation cycle, which is then repeated. With the help of the Adam optimizer, the loss function gradually decreases with the increase in training iterations, and its graph curve plotted with the number of iterations gradually converges, indicating that model optimization is complete. In other words, the model training terminates when the preset maximum number of iterations is reached or the loss value converges.

[0076] (3) Model Application

[0077] The trained model is deployed on autonomous vehicles equipped with cameras and LiDAR. As the autonomous vehicle travels on the road, it collects real-time road condition information. The point cloud data obtained from the LiDAR is converted into a 2-bin histogram, which, along with the corresponding RGB images captured by the cameras, is input into the model, enabling it to perceive vehicles on the road. Based on this perception, the autonomous vehicle's control system performs subsequent logical judgments and controls the vehicle to perform actions such as turning or moving forward.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0079] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A road vehicle perception method based on multi-sensor fusion, characterized in that, Includes the following steps: Step 1: Obtain target detection data for training the road vehicle perception model. The target detection data includes image data collected by the camera device and lidar point cloud data collected by the laser scanner. Then, convert the lidar point cloud data into a lidar bird's-eye view. Step 2: Construct and train a road vehicle perception model; The road vehicle perception model includes a point-level feature fusion module and a region of interest (RoI)-level feature fusion module. The point-level feature fusion module includes an image branch backbone network and a LiDAR branch backbone network. The image branch backbone network is used to extract image features from image data at multiple scales, obtaining several first intermediate feature maps at different scales, as well as the final output feature map of the image branch backbone network. The LiDAR branch backbone network is used to extract image features from LiDAR bird's-eye view at multiple scales, obtaining several second intermediate feature maps at different scales, as well as the final output feature map of the LiDAR branch backbone network. The first and second intermediate feature maps have the same number of scales, and the feature map dimensions of the first and second intermediate feature maps at the same level are consistent. The first and second intermediate feature maps of the first level are processed by the attention fusion module to obtain the first level fused intermediate feature map. The first level fused intermediate feature map is added to the first intermediate feature map of the first level and then continues to participate in the forward calculation on the image branch backbone network. The first level fused intermediate feature map is added to the second intermediate feature map of the first level and then continues to participate in the forward calculation on the lidar branch backbone network. Furthermore, starting from the second-level intermediate feature map, the intermediate feature maps of the same level of the image branch backbone network and the lidar branch backbone network are fused through the fusion module of the Transformer to obtain the current-level fused intermediate feature map. Then, it is added to the first and second intermediate feature maps of the current level respectively, and then continues to participate in the forward calculation on the two backbone networks respectively. The RoI-level feature fusion module performs road vehicle detection on the final output feature map of the image branch backbone network through convolutional layers, obtaining several 3D candidate boxes and their road vehicle recognition results. After score thresholding and non-maximum suppression, several 3D detection boxes are obtained. The 3D detection boxes are projected onto the LiDAR bird's-eye view space and the two-dimensional image space of the image data, respectively. Then, RoI features of the image data and LiDAR bird's-eye view are obtained through RoI feature extraction. The RoI features of the two modalities are concatenated and input into the thinning module based on at least two fully connected layers to predict the thinning correction of each 3D detection box. The thinning correction result and the 3D detection box are combined to obtain the final road vehicle detection result. When training a road vehicle perception model based on target detection data and corresponding label data, the total loss function of the model is the sum of classification loss and regression loss. When the preset training convergence condition is met, a road vehicle perception model based on multi-sensor fusion is obtained, which can be used for road vehicle perception during the driving process of autonomous vehicles.

2. The method as described in claim 1, characterized in that, The color space of the image data is in RGB format.

3. The method as described in claim 1, characterized in that, The attention fusion module is specifically: For the first intermediate feature map F1 of the current level i i Second intermediate feature map After concatenation along the channel dimension, the data is input into the convolutional layer to obtain the preliminary fused feature map of the current level i. An attention mechanism is used to initially fuse feature maps. The various features in the dataset are weighted, and the output is the attention map. Then and Element-wise multiplication is performed to obtain the fused intermediate feature map output by the attention fusion module.

4. The method as described in claim 1, characterized in that, The fusion module of the Transformer is specifically as follows: Define C×H×W as the first intermediate feature map F1 of the current level i. i Second intermediate feature map The feature map dimension is given by C, where C represents the number of channels in the intermediate feature map and H×W represents the resolution of the intermediate feature map. For the first intermediate feature map F1 of the current level i i Second intermediate feature map Each sequence is expanded according to the order of the permutation matrix, forming a sequence composed of discrete labels. and Where, sequence and The dimension is HW×C; splicing sequence and Obtain the fusion sequence I of the current level i. i Among them, fusion sequence I i The dimension is 2HWxC, and the fusion sequence I i The 2HW labels in the model are each represented by a feature vector of dimension C. Each feature vector is supplemented by a learnable positional encoding to incorporate positional induction bias and distinguish the spatial information of different labels. Fusion Sequence I i A set of query Q, key K, and value V are calculated using linear projection onto three weight matrices: Q<IM q ,K=IM k ,V=IM v Among them, M q M k M v The weight matrices are Q, K, and V, respectively. The self-attention layer uses the scaled dot product between Q and K to compute attention weights and aggregates the values ​​for each query to infer the refined output Z: Based on the projection mapping of several outputs Z, the output Z′ of the multi-head attention mechanism is obtained, and then a nonlinear transformation is used to calculate the input sequence I. i Output sequence of the same scale: O = MLP(Z″) + Z″, where MLP() represents a multilayer perceptron, and Z″ = Z′ + I i ; According to the formation sequence and The inverse operation of the permutation matrix transforms the output sequence O into the corresponding first intermediate feature map F1. i Second intermediate feature map Feature map F1′ i and Feature map F1′ i and Each is compared with the first intermediate feature map F1 i Second intermediate feature map After addition, it continues to participate in forward computation on the lidar branch backbone network.

5. The method according to any one of claims 1 to 4, characterized in that, The model's total loss function is specifically set as follows: L=L cls +λL reg Where λ represents the balance parameter, and the classification loss L cls and regression loss L reg The ones are respectively: Where, p c Indicates the predicted classification score, l c Binary labels representing the vehicle and the background, where N represents the total number of samples; Where (x, y, z) are the coordinates of the 3D bounding box, (w, h, d) represent the dimensions of the 3D bounding box in the x, y, z directions, t represents the direction, and N... pos p represents the number of positive samples, D represents the smoothed L1 norm, and p k and l k These are the offsets of the predicted and actual values, respectively.

6. The method as described in claim 5, characterized in that, p k Specifically: If k∈(x, y, z), then p k =(ka) k ) / a k , where a k The coordinates of the anchor point; If k∈(w, h, d), then p k =log(k / a′) k ), where a′ k This refers to the anchor point size.

Images