A cross-view fusion 3D object detection method based on cross-attention

Abstract

The present invention proposes a cross-view fusion three-dimensional target detection method based on cross-attention, which belongs to the field of three-dimensional target detection. The method comprises the following steps: extracting features from multi-view images using an image backbone network improved by an expert perception layer to obtain image features of multiple views; extracting depth information of lidar point clouds of multiple views to obtain dense point cloud depth maps; pixel-by-pixel splicing the dense point cloud depth maps of corresponding views with image features to obtain multi-view image features with depth information, generating an image pseudo point cloud according to the depth information, and flattening the image along the z-axis to obtain BEV features; voxelizing the lidar point cloud and using a multi-view backbone network to respectively obtain BEV features and RV features of the point cloud and splicing them; introducing a cross-attention mechanism to aggregate the fused BEV features and RV features of the point cloud, and finally obtaining fused BEV features with attention weighting, and using a regression detection head to obtain three-dimensional target detection results.

Description

Technical Field

[0001] The present invention relates to the fields of autonomous driving and deep learning, and in particular to a cross-view fusion three-dimensional target detection method based on cross-attention. Background Art

[0002] In recent years, autonomous driving has garnered increasing attention due to its potential to reduce driver burden and improve driving safety. Perception systems are an integral component of modern autonomous driving systems, accurately estimating the state of the surrounding environment and providing reliable observation data for prediction and planning. Three-dimensional object detection is a crucial component of perception systems, intelligently predicting the location, size, and category of key three-dimensional objects near the autonomous vehicle.

[0003] Cameras and lidar are the most commonly used sensors in autonomous driving solutions. Each has its own advantages and limitations, but they are two complementary sensor types for 3D object detection. Cameras provide color information from which rich semantic features can be extracted, while lidar sensors excel at 3D localization and provide rich 3D structural information. The complementary advantages and disadvantages of these two sensor types are highly evident, leading to the fusion of lidar and cameras, a major research direction both domestically and internationally. Currently, most classic object detection algorithms use either a bird's-eye view (BEV) or a point view (PV) for object detection. The BEV effectively addresses occlusion issues between objects, but the point cloud data is relatively sparse. The point view contains complete object information, but suffers from occlusion issues and a large number of useless background points. While the range view (RV) suffers from the problem of objects appearing larger near and smaller far, it allows for fast neighborhood queries, making it suitable as a complementary feature to other view types. Currently, the bird's-eye view is the most commonly used view in object detection algorithms. This is not limited to lidar point cloud-based object detection algorithms; many image-based object detection algorithms also utilize the bird's-eye view feature. However, the limitations of a single view are obvious. Therefore, using multi-view fusion methods for target detection can obtain more comprehensive target features.

[0004] The first algorithm to fuse camera and lidar was MV3D, which projects the lidar point cloud into the front view (FV) and bird's-eye view (BEV). The bird's-eye view then generates a rough 3D region proposal. The front view, bird's-eye view, and image are then fused based on the 3D region proposal to obtain the detection result. Although the accuracy has been improved, the computational complexity of MV3D is relatively high, making it difficult to perform real-time detection. F-PointNet has pioneered a new fusion method, which uses the camera's perspective projection to generate a 3D frustum of the two-dimensional detection frame generated by the 2D image plane. The point cloud within the frustum is then combined for segmentation and regression to obtain the final target detection frame. This method reduces computational complexity, but relies too much on 2D image information and is prone to missed detections due to occlusion and inaccurate depth information. In 2019, the publication of the milestone work Pointpainting took the fusion algorithm to a higher level. It proposed a new fusion strategy to perform semantic segmentation on two-dimensional image information, and then directly draw the semantic segmentation information onto the lidar point cloud. It then uses pure point cloud methods such as Voxelnet, PointPillar, PointRCNN, etc. for target detection. This method uses two-dimensional image information as an auxiliary to the radar point cloud for target detection. It has low computational complexity and does not lose depth information. It has neither restrictions on the 3D detection framework nor problems with feature or depth blur. Many subsequent fusion algorithms are optimized and improved on this basis. In 2017, Google proposed the concept of the attention mechanism (Transformer), which uses the attention mechanism to encode and calculate the features of the target, completely abandoning the RNN and CNN structures, and greatly reducing the computational complexity. In 2022, the release of DeepFusion introduced the Transformer into multi-sensor fusion target detection for the first time, performing fine-grained fusion in a focused manner. Later, TransFusion designed an image-guided query initialization module based on it to handle objects that are difficult to detect in point clouds, and used the attention mechanism to interactively fuse the features of image information and point cloud information, with high accuracy and robustness. Summary of the Invention

[0005] In order to address the shortcomings of existing point cloud image fusion target detection methods, the present invention proposes a cross-view fusion three-dimensional target detection method based on cross-attention. By introducing the RV features of the lidar point cloud to accelerate neighborhood queries, an expert perception module is proposed to improve the image backbone network, and a module for depth completion of image features using dense point cloud depth maps is proposed to solve the problem of large error in depth information when converting 2D images to BEV view features. At the same time, a cross-view fusion cross-attention network is introduced to fuse the BEV and RV features of the lidar point cloud, enhance multi-view feature interaction, and improve the accuracy and robustness of the detection network.

[0006] The present invention is achieved by at least one of the following technical solutions:

[0007] A cross-view fusion 3D object detection method based on cross-attention includes the following steps:

[0008] S1. Obtain multi-view images collected by cameras at different positions, and use the image backbone network optimized by the expert perception module to extract image features to obtain image feature information;

[0009] S2. Obtain lidar point cloud information, use the dense point cloud depth information completion module to generate a dense depth map containing feature depth information from the point cloud, and perform depth completion by pixel-by-pixel stitching of the feature depth information of the depth map with the feature information of the image to generate image features with accurate depth information;

[0010] S3, generating an image pseudo point cloud based on the depth information, and performing BEV view transformation on the pseudo point cloud to generate image BEV features with depth information;

[0011] S4. Use the multi-view point cloud backbone network to extract the lidar point cloud and obtain the BEV features and RV features of the lidar point cloud respectively;

[0012] S5, concatenating the BEV features of the point cloud with the BEV features of the image;

[0013] S6, use the cross-view cross attention network to fuse the total BEV features and the RV features of the point cloud;

[0014] S7. Use the regression detection head to process and transform the total fusion features to obtain the final detection result; the final 3D target detection result includes the category and position parameters of the target bounding box.

[0015] Furthermore, the image backbone network uses SwinTransformer as its basis, and the expert perception module optimizes the basic image backbone network. The expert perception module activates GPU parallel processing. Each GPU corresponds to an expert network to process the input image features, thereby speeding up the image feature processing speed.

[0016] Furthermore, the expert perception module includes a gating network, an expert selection network and a collective communication network based on the ALL to ALL algorithm, the gating network includes a fully connected layer, and the expert network is an FFN feedforward neural network; the image is input into the expert perception module after feature extraction and preliminary processing through the image backbone network, first, each input feature is scored through the gating network, and corresponding weight values ​​are generated according to the type and complexity of the input features to obtain different weight values, i.e., score values, and the weight values ​​are normalized using the softmax function, and then in the first ALL to ALL (collective communication), the GPU of each input is determined and communication between each GPU is established to achieve parallel processing, and the input features are divided into blocks. After the first ALL to ALL, the expert selection network selects an expert network according to the weight value to perform feature processing on each feature block. Each GPU has its own expert network for feature processing. After feature processing, a second ALL to ALL is performed, and the output of each expert network is summarized. The feature blocks processed according to feature correlation are spliced ​​to obtain the processed image features.

[0017] Furthermore, in step S2, the dense point cloud depth information completion module uses the dense point cloud depth information to complete the depth feature of the image features with estimated depth generated by the LSS algorithm, and projects the point cloud into the multi-view depth map. where N v Indicates the number of views, N H and N W Represents the length and height of the depth map, d ij Represents the depth value of the corresponding position in the depth map D. The multi-view depth map D is divided into blocks of size k×k along the X-axis and Y-axis, where k is the step size. Then, the maximum depth value of each block is used to fill the entire block through the expansion operation. Most of the scenes in the point cloud depth map are connected to obtain a dense point cloud depth map; the dense point cloud depth map D of one of the views is selected v , the point cloud depth map D v After Conv 3×3 , BatchNorm normalization layer and ReLU activation function processing for feature refinement, and then use ResNet18 network to obtain the point cloud point-by-point depth feature F of the view D , the point cloud point-by-point depth feature F DPixel-by-pixel concatenation with image features and pixel-by-pixel depth loss As a constraint to supervise the loss between the depth information of non-zero pixels in the spliced ​​image features and the depth feature information of the point cloud, where n is the number of non-zero pixels in the image features, H D The view category representing the depth information of the point cloud, is the estimated depth value of the i-th non-zero pixel, h i,v is the depth value of the point closest to the pixel, α v Represents the weight, γ is the defined loss hyperparameter, and then the BEV conversion of image features is performed.

[0018] Furthermore, in step S3, the multi-view image features with accurate depth information generate image pseudo point cloud cone features, which are projected into voxels to obtain image pseudo point cloud voxel features; the image pseudo point cloud voxel features are flattened along the Z axis to obtain the image BEV features.

[0019] Furthermore, the multi-view point cloud backbone network includes BEV convolutional layer and RV convolutional layer;

[0020] The BEV convolution is a three-dimensional sparse convolution with 128 input channels, 32 output channels, a convolution kernel size of (1, 1, 3), and a step size of (1, 1, 2).

[0021] The RV convolution includes three sparse convolution layers. The number of input channels of the first convolution layer is 128, and the number of output channels is 32. The number of input and output channels of the second and third convolution layers are both 32. The convolution kernel sizes of the three convolution layers are all (1, 1, 3), and the step sizes are all (1, 1, 2). The generated BEV feature size is (N, C*D, H, W), and the generated RV feature size is (N, C*W, H, D), where N is the number of input samples, C is the number of channels, W is the width, H is the height, and D is the depth. The BEV convolution layer and the RV convolution layer respectively extract features of the lidar point cloud to obtain the BEV features and RV features of the lidar point cloud, and obtain the position information of the two features respectively.

[0022] Furthermore, in step S5, the image BEV features and the lidar point cloud BEV features are spliced ​​together along the number of channels to obtain fused BEV features.

[0023] Furthermore, in step S6, the RPN network is used to refine the fused BEV features and RV features of the lidar point cloud, and the corresponding position information is added to the fused BEV features and RV features respectively. The cross-view cross-attention network is used for feature fusion. Each view is a feature sequence of length J, and the attention weight matrix between each item in the different view sequences is obtained. Apply it to the value matrix V j , and processed by a fully connected layer FC to obtain the fusion feature matrix after attention weighted aggregation The calculation formula of attention weighted fusion BEV features is as follows:

[0024]

[0025]

[0026]

[0027] in is the query item feature of BEV, is the key item feature of RV, V is the value item feature of RV, i and j are the numbers of each item, J is the total number of value items, d q Indicates the number of feature channels and obtains fusion features with attention weights

[0028] Furthermore, in step S7, the regression detection head is used to obtain various regression parameters of the target frame, and the final prediction result is obtained by calculating the fusion features and attention weights.

[0029] Furthermore, the loss function Loss is defined, Loss = λ1L cls +λ2L reg +λ3L var , where λ1, λ2, λ3 are loss weights, L cls , L reg , L var Represents classification loss, regression loss and variance constraint loss. The classification loss uses CrossEntropyLoss to calculate the error between the predicted sample result and the real sample result. The regression loss uses SmoothL1Loss to calculate the loss between the detection box generated by the regression detection head and the real box. The variance constraint loss is used to constrain the variance of the attention weight, focusing attention on meaningful targets in the scene. The variance constraint formula of all real bounding boxes is:

[0030]

[0031] Among them, N box is the number of bounding boxes in the scene, N q is the bounding box b q The number of enclosed cylinders, A q [i] represents the attention weight of the i-th ground-truth bounding box, and Var(·) is the variance of the given quantity.

[0032] The present invention has the following advantages and strengths over the existing technology:

[0033] (1) The present invention discloses a cross-view fusion three-dimensional target detection method based on cross-attention. The method integrates the features of BEV and RV views, allowing the two views to complement each other and avoid the feature loss caused by a single view.

[0034] (2) The present invention discloses a cross-view fusion three-dimensional target detection method based on cross-attention. The image backbone network SwinTransformer is optimized using an expert perception module so that it can be processed in parallel on multiple GPUs, effectively improving the inference speed of target detection and increasing the processable image resolution.

[0035] (3) The present invention discloses a cross-view fusion three-dimensional target detection method based on cross-attention, which uses point cloud depth information as supervision to perform BEV conversion of image features. Compared with the traditional LSS algorithm, it has precise depth information and the obtained image BEV features are more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Flowchart of a cross-view fusion three-dimensional object detection method based on cross-attention in an embodiment of the present invention;

[0037] Figure 2 Schematic diagram of the overall framework of a cross-view fusion 3D object detection method based on cross-attention in an embodiment of the present invention;

[0038] Figure 3 This is a structural diagram of the expert perception module in an embodiment of the present invention;

[0039] Figure 4 This is a structural diagram of an image BEV feature generation module guided by point cloud depth information in an embodiment of the present invention;

[0040] Figure 5 This is a conversion relationship diagram of voxel sampling points, frustum depth structure, and image BEV features in an embodiment of the present invention;

[0041] Figure 6 Schematic diagram of a multi-view point cloud backbone network in an embodiment of the present invention;

[0042] Figure 7 This is a structural diagram of the cross-view cross-attention fusion module in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Of course, the specific embodiments described below are only used to explain the present invention and are not used to limit the present invention.

[0044] Figure 1 Flowchart of a cross-view fusion 3D object detection method based on cross-attention in an embodiment of the present invention. Figure 2 Schematic diagram of the overall framework of a cross-view fusion 3D target detection method based on cross-attention in an embodiment of the present invention. Figure 1 and Figure 2 .

[0045] In an embodiment of the present invention, a cross-view fusion 3D object detection method based on cross-attention includes the following steps:

[0046] S1. Obtain multi-view images collected by cameras at different positions, and use the image backbone network optimized by the expert perception module to extract image features to obtain image feature information;

[0047] like Figure 3 As shown in the figure, the collected multi-view image size is H×W×3, which is input into the SwinTransformer image backbone network to extract image features. The last layer of each SwinTransformer block (sliding window transformer network) adds an expert perception module. The expert perception module includes a gating network, an expert selection network and a collective communication network based on the ALL to ALL algorithm. The image is extracted and preliminarily processed by the image backbone network and then input into the expert perception module. First, the input feature blocks are scored by the gating network. The gating network is usually composed of a fully connected layer. The corresponding weight value, that is, the score, is generated according to the type and complexity of the input feature. After obtaining different weight values, the softmax function is used to normalize the weight value. Then, in the first ALL to ALL (collective communication), the GPU of each input is determined and communication between the GPUs is established to achieve parallel processing, and the input features are divided into blocks. After the first ALL to ALL, the expert selection network selects the expert network for feature processing according to the feature weight value. Each GPU has its own expert network for feature processing. After feature processing, a second ALL to ALL, summarizes the output of each expert network, and combines the feature blocks processed according to feature correlation to obtain the processed image features. The expert network is mainly a FFN feedforward neural network.

[0048] S2. Obtain the LiDAR point cloud information and use the dense point cloud depth information completion module to generate a dense depth map containing feature depth information from the point cloud. The feature depth information of the depth map is then combined with the feature information of the image pixel by pixel for depth completion to generate image features with accurate depth information. The specific steps are as follows:

[0049] like Figure 4 As shown, the image feature F of view v c,v , point cloud projection to multi-view depth map where d ij Indicates the depth value of the corresponding position in the depth map D, N v Indicates the number of views, N H and N W Represents the length and height of the depth map. The multi-view depth map D is divided into blocks of size k×k along the X-axis and Y-axis, where k is the step size. Then, the maximum depth value of each block is used to fill the entire block through the expansion operation to connect most of the scenes in the point cloud depth map. After block division and expansion operations, the multi-view depth map is converted into a multi-view dense depth map with the same dimension. The point cloud depth map D of one of the views is selected. v , the point cloud depth map D v Conv 3×3 , BatchNorm normalization layer and ReLU activation function are processed to obtain the point cloud depth feature F D , and then use the ResNet18 network to obtain the deep feature F D The depth information H(u,v) of the view v is concatenated pixel by pixel with the depth information and the image features of the view v to obtain the image features with depth features of the view v. 3×3 Perform feature refinement, and then store the sequence obtained by splicing the features into an empty list. Perform this operation on each view, and finally obtain a multi-view image feature list with depth features, which is the final multi-view image feature with depth information.

[0050] Because the depth features obtained from the point cloud are relatively sparse, even in the converted dense depth map, the resolution of the image pixels is still much higher than that of the dense depth map of the point cloud. Therefore, after the image and point cloud depth features are spliced ​​together, there will be a lot of non-zero pixel depth information that is still the original estimated depth information. Therefore, a fine-grained module is proposed to constrain the depth network pixel by pixel, and pixel-by-pixel depth loss (PDL) is used as a constraint to supervise the loss between the depth information of the non-zero pixels in the spliced ​​image features and the point cloud depth feature information:

[0051]

[0052] Where n is the number of non-zero pixels in the image feature, H D The view category representing the depth information of the point cloud, h i,v is the estimated depth value of the i-th non-zero pixel, h i,v is the depth value of the point closest to the pixel, α v Denotes the weight, and γ is the defined loss hyperparameter. The depth information of each non-zero pixel in each image feature is constrained by the PDL loss function, making it closer to the true depth value.

[0053] S3, such as Figure 5 As shown in the figure, a pseudo point cloud of the image is generated based on the depth information, and the pseudo point cloud is transformed into a BEV view to generate BEV features of the image with depth information; the image features are associated with the depth estimation information using the frustum depth estimation network, and the image with depth information is projected into the three-dimensional space to obtain three-dimensional data in the form of a pseudo point cloud. The feature pixel point F(u,v) is weighted with the corresponding depth information H(u,v) to generate the image frustum feature G(u,v). c,v In the example, (u, v, c) is used to represent the pixel information, and in the depth information H, (u, v, h i ) represents the depth information of the corresponding pixel, h i is the index value of the depth unit, and the depth probability of F(u,v) and H(u,v) is weighted to fill the Z axis of the image pixel. The image feature pixel and depth information are weighted using the outer product formula to obtain the frustum feature G(u,v):

[0054]

[0055] Since the frustum depth structure cannot be directly used in the 3D target detection network, it is necessary to use camera calibration mixed sampling to convert it into voxel data, that is, Transform to V∈R X×Y×Z×C In voxel space, each pixel center generates a sampling point The sampling points in the depth space of the viewing cone are obtained by the correspondence between the camera's 3D coordinate system and the radar coordinate system. Due to d c It is obtained after voxel space quantization and needs to be further quantized into depth unit space to obtain the depth unit index d i , recorded as Then fill the corresponding information into the voxel space to obtain the voxel features of the image, and then superimpose the voxel features of the image on the Z axis, and use the convolution operation to reduce the channel to obtain the BEV feature F of the image bev =Conv(Stack(F voxel, dim=Z)), and then it can be spliced ​​with the BEV features of the point cloud to obtain the fused BEV features.

[0056] S4, such as Figure 6 As shown in the figure, the multi-view point cloud backbone network is used to extract the lidar point cloud, and the BEV features and RV features of the lidar point cloud are obtained respectively; the BEV features and RV features of the point cloud are generated by the multi-view backbone network. First, the lidar point cloud is voxelized using VoxelNet to generate voxel features with a range of [-51.2, -51.2, -5.0, 51.2, 51.2, 3.0] and a size of [0.08, 0.08, 0.08]. An intermediate convolution layer is first used to refine the point cloud voxel features. The intermediate convolution layer consists of a three-dimensional sparse convolution, which consists of multiple convolution blocks with gradually increasing numbers of input and output channels and a ReLU activation function. Then, the BEV convolution and RV convolution in the multi-view backbone network are used to refine the voxelized point cloud respectively. Feature extraction is performed to obtain the BEV features and RV features of the point cloud. The BEV convolution is a three-dimensional sparse convolution with 128 input channels, 32 output channels, a convolution kernel size of (1, 1, 3), and a step size of (1, 1, 2). The RV convolution consists of three sparse convolution layers. The first convolution layer has 128 input channels and 32 output channels. The second and third convolution layers have 32 input and output channels, respectively. The convolution kernel sizes of the three convolution layers are all (1, 1, 3), and the step sizes are all (1, 1, 2). The generated BEV feature size is (N, C*D, H, W), and the generated RV feature size is (N, C*W, H, D), where N is the number of input samples, C is the number of channels, W is the width, H is the height, and D is the depth.

[0057] S5, the BEV features of the image and the BEV features of the point cloud are spliced ​​together, and feature refinement is performed using two-dimensional convolution to obtain fused BEV features. After obtaining the fused BEV features and point cloud RV features, as shown in Figure 7 As shown in the figure, the RPN network (region proposal generation network) is used to extract and integrate multi-scale features to obtain high-resolution fused BEV features and point cloud RV features, and then the position encoding information of the BEV features and the position encoding tensor of the RV features are introduced into the corresponding features respectively.

[0058] Define the query item convolution layer q_conv, the key item convolution layer k_conv and the value item convolution layer v_conv respectively, the convolution kernel size is 3×3, and the high-resolution BEV feature is RV characteristics are Use the query convolution layer q_conv to obtain the query feature of BEV Use the key convolution layer k_conv to obtain the key features of RV Use the value item convolution layer v_conv to obtain the value item feature Value(V) of RV where d q , d k , d v The characteristic dimensions of Q, K, and V are:

[0059]

[0060]

[0061]

[0062] S6. To obtain the attention-weighted features, a cross-view cross-attention network is used to obtain cross-view fusion features. First, the number of feature channels d q Perform feature scaling to obtain the similarity matrix of the BEV feature of the i-th query item and the RV feature of the j-th value item Use the softmax function to normalize and get the cross-view attention weight

[0063]

[0064]

[0065] The cross attention weight matrix Acting on the value matrix V j , and processed by a fully connected layer FC to obtain the fusion feature matrix after attention weighted aggregation

[0066]

[0067] Where J is the number of value items, and the cross attention weight matrix The meaning of is the degree to which the fusion feature of each point is affected by the fusion features of other points, using the attention weight matrix The value matrix V is weightedly integrated to achieve the weighted integration of attention to the fusion features.

[0068] Attention-weighted fusion features With high resolution BEV features F BEV Perform point-by-point splicing to achieve the attention weighting of the fused BEV features, so that each point in the fused BEV features has its own attention score;

[0069] S7. After the processed features are input into the regression detection head, the detection head will pay different degrees of attention to each point according to the attention score, so that it can pay more attention to useful features to obtain more accurate detection results.

[0070] Since 3D scenes contain a large number of background points (about 95%), and only a small number of them are points of interest that contribute to the detection results, during the training of cross-view attention, a large number of background points will bring unexpected noise to the attention module. In addition, the occlusion problem in complex 3D scenes also brings inevitable distortion to the attention.

[0071] To enable the attention module to focus on specific targets rather than general points, the present invention imposes a constraint on the variance of the learned attention weights. By combining the attention variance constraint with traditional classification and regression supervision signals, the attention module focuses on meaningful targets in the scene, thereby producing high-quality fused features.

[0072] The constraint method proposed in this invention is formulated as an auxiliary loss during training.

[0073] Given a learned attention weight where N BEV and N RV are the number of cylinders in BEV and RV respectively, and the set of the size and center position of the real bounding box on the xy plane is b q is the bounding box numbered q, w q 、h q 、x q 、y q N is the height, width and center coordinates of the bounding box. box For each cylinder in the BEV, the method proposed in this invention calculates the real coordinates of its center according to the voxel size and obtains the set c j is the center coordinate of the voxel numbered j, x j ,y j is the coordinate value, N bev represents the number of cylinders in BEV and the attention weight of each ground-truth bounding box is:

[0074]

[0075] Then, the proposed method formulates the variance constraint of all ground-truth bounding boxes as:

[0076]

[0077] where N q is the bounding box b q The number of enclosed cylinders, A q[i] represents the attention weight of the i-th ground-truth bounding box, and Var(·) is the variance of the given quantity.

[0078] As an embodiment, a FFN-based regression detection head is used to extract and transform target features, attention-weighted BEV semantic features are used for target classification, and attention-weighted BEV geometric features are used for target regression to obtain various classification and regression parameters of the target frame. The various classification and regression parameters are input into the loss function. After iterative calculation, the final more accurate detection frame is obtained. There are ten main detection categories, namely car, truck, bus, trailer, construction, pedestrian, motorcycle, bicycle, traffic_cone, barrier, among which pedestrian, motorcycle, bicycle, traffic_cone, and barrier are small targets. The detection accuracy of small targets is an important indicator to measure the excellence of an algorithm.

[0079] The target detection network in the method of the present invention is composed of an image pipeline and a point cloud pipeline. Before the overall training, the image pipeline must be trained first to obtain the optimal image pipeline parameters before the overall network training. The loss function is used during the image pipeline training. As described above, in the three-dimensional target detection model based on multi-view image point cloud fusion, all the features are finally input into the detection head and then the loss function is used during training to optimize the network results so that it converges and obtains the optimal network parameters after training. The loss function defined here is the loss function of the entire target detection network. The entire target detection network is composed of the above-mentioned networks or modules, and its loss function is defined as the sum of the target loss and the variance constraint loss:

[0080]

[0081] in is the target loss, is the variance constraint loss, λ3 is the balance coefficient of the variance constraint loss, where the target loss for:

[0082]

[0083] Where λ1 and λ2 are classification losses respectively and regression loss The weight coefficient of the classification loss is used to calculate the true label The loss value between the predicted label y is calculated using the CrossEntropyLoss function, and the regression loss is used to calculate the true bounding box. The loss value between the predicted box b is SmoothL1Loss.

[0084] The present invention proposes a cross-view fusion three-dimensional target detection method based on cross-attention, which realizes the feature-level fusion of images and lidar point clouds based on BEV views and RV views. In order to solve the problem of limitations in single BEV view target detection, the RV view is added to enhance the neighborhood query. The image features are enhanced using the depth information of the point cloud, and the BEV view conversion of the image features is guided by the depth feature information. The SwinTransformer image backbone network is optimized using the expert perception module, which gives the network parallel processing capabilities, improves the processing speed of image features, and increases the amount of information processed. Increasing the input image resolution can further improve the detection performance of small targets. Finally, the cross-attention mechanism is used to complete the weighted fusion of multi-view features, and the fusion features with attention weights are obtained. The fusion features are point-by-point spliced ​​and feature integrated with the point cloud BEV high-dimensional features, ultimately achieving higher fusion accuracy and achieving better three-dimensional target detection effects.

[0085] The following experiments are used to verify the technical effects of the present invention.

[0086] This experiment used nuScenes, a large-scale public autonomous driving dataset. The nuScenes dataset was collected in Boston and Singapore by two vehicles equipped with six cameras and a 32-axis LiDAR. The dataset covers various scenes, including urban, residential, suburban, and industrial areas, and encompasses different weather conditions, including daytime, nighttime, sunny, and rainy. This experiment used the nuScenes training set for training, and the trained network was validated using the nuScenes validation set.

[0087] As an embodiment, the model training of the present invention uses the Adam optimizer, the learning rate is set to 0.0001, and four RTX3090 graphics cards are used for training. The batch_size of each card is set to 2, the total batch_size is 8, and a total of 20 epochs are trained.

[0088] In terms of model evaluation, the mAP and NDS indicators proposed by the nuScenes dataset are used for evaluation. mAP is the average detection precision, and NDS is the weighted average of detection accuracy and positioning accuracy. To verify the excellent detection performance of the present invention and the improvement in small target detection accuracy, it is compared with the current more representative target detection network. At the same time, in order to verify the improvement in small target detection accuracy, the detection accuracy of the algorithm of the present invention for five types of small target objects is separately compared with the baseline. The experimental comparison results are shown in Tables 2 and 3.

[0089] Table 2 Performance comparison between the algorithm of the present invention and the algorithm in the field of target detection

[0090] Algorithm Name mAP NDS VISTA 63.6％ 69.5％ PointAugmenting 66.4％ 71.0％ BEVFusion 68.6％ 71.4％ Algorithm of the present invention 69.5％ 71.9％

[0091] Table 3 Detection accuracy of the proposed algorithm compared with the baseline for small targets

[0092]

[0093] Table 4 Effect of the expert perception module in the algorithm of the present invention

[0094]

[0095] Table 2 shows the detection accuracy comparison between the algorithm proposed in this paper and the classic algorithms in this field. mAP and NDS are two measurement indicators. It can be seen that the mAP and NDS indicators of the algorithm proposed in this paper are both higher than those of other excellent algorithm models in this field, showing feasibility and effectiveness.

[0096] Table 3 shows the detection accuracy of small targets of the algorithm proposed in this invention compared with the baseline algorithm. It can be seen that the algorithm proposed in this invention has significantly improved the detection accuracy of the five small targets compared with the baseline, indicating that the algorithm proposed in this invention has achieved a better effect in small target detection.

[0097] Table 4 shows how the expert perception module improves the inference speed of the entire model and optimizes computing power requirements. When the image size is 704×256, the module can reduce the inference time by 23.3 seconds. When the image size is increased to 1056×384, the model without the module cannot run due to computing power limitations. However, the model using the expert perception module can complete the entire inference process in 135.3 seconds, demonstrating the effectiveness of the expert perception module.

[0098] The above embodiments are preferred implementations of the present invention, but the implementation of the present invention is not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A cross-view fusion 3D object detection method based on cross-attention, characterized by: The following steps are involved: S1. Obtain multi-view images collected by cameras at different positions, and use the image backbone network optimized by the expert perception module to extract image features to obtain image feature information; The expert perception module includes a gating network, an expert selection network and a collective communication network based on an ALL to ALL algorithm, the gating network includes a fully connected layer, and the expert network is an FFN feedforward neural network; the image is input into the expert perception module after feature extraction and preliminary processing through the image backbone network, firstly, each input feature is scored through the gating network, and corresponding weight values ​​are generated according to the type and complexity of the input features to obtain different weight values, i.e., score values, and the weight values ​​are normalized using the softmax function, and then in the first ALL to ALL, the GPU of each input is determined and communication between the GPUs is established to achieve parallel processing, and the input features are divided into blocks. After the first ALL to ALL, the expert selection network selects an expert network according to the weight value to perform feature processing on each feature block. Each GPU has its own expert network for feature processing. After the feature processing, a second ALL to ALL is performed to summarize the output of each expert network, and the feature blocks processed according to the feature correlation are spliced ​​to obtain the processed image features; S2. Obtain lidar point cloud information, use the dense point cloud depth information completion module to generate a dense depth map containing feature depth information from the point cloud, and perform depth completion by pixel-by-pixel stitching of the feature depth information of the depth map with the feature information of the image to generate image features with accurate depth information; S3, generating an image pseudo point cloud based on the depth information, and performing BEV view transformation on the pseudo point cloud to generate image BEV features with depth information; S4. Use the multi-view point cloud backbone network to extract the lidar point cloud and obtain the BEV features and RV features of the lidar point cloud respectively; S5, concatenating the BEV features of the point cloud with the BEV features of the image; S6, use the cross-view cross attention network to fuse the total BEV features and the RV features of the point cloud; S7. Use the regression detection head to process and transform the total fusion features to obtain the final detection result; the final 3D target detection result includes the category and position parameters of the target bounding box.

2. The cross-view fusion 3D object detection method based on cross-attention according to claim 1 is characterized in that: The image backbone network uses SwinTransformer as its basis, and the expert perception module optimizes the basic image backbone network. The expert perception module activates GPU parallel processing. Each GPU corresponds to an expert network to process the input image features, thereby speeding up the image feature processing speed.

3. The cross-view fusion 3D object detection method based on cross-attention according to claim 1 is characterized in that: In step S2, the dense point cloud depth information completion module uses the dense point cloud depth information to complete the depth feature of the image features with estimated depth generated by the LSS algorithm, and projects the point cloud into the multi-view depth map. where N v Indicates the number of views, N H and N W Represents the length and height of the depth map, d ij Represents the depth value of the corresponding position in the depth map D. The multi-view depth map D is divided into blocks of size k×k along the X-axis and Y-axis, where k is the step size. Then, the maximum depth value of each block is used to fill the entire block through the expansion operation. Most of the scenes in the point cloud depth map are connected to obtain a dense point cloud depth map; the dense point cloud depth map D of one of the views is selected v , the point cloud depth map D v After Conv 3×3 , BatchNorm normalization layer and ReLU activation function processing for feature refinement, and then use ResNet18 network to obtain the point cloud point-by-point depth feature F of the view D , the point cloud point-by-point depth feature F D Pixel-by-pixel concatenation with image features and pixel-by-pixel depth loss As a constraint to supervise the loss between the depth information of non-zero pixels in the spliced ​​image features and the depth feature information of the point cloud, where n is the number of non-zero pixels in the image features, H D The view category representing the depth information of the point cloud, is the estimated depth value of the i-th non-zero pixel, h i,v is the depth value of the point closest to the i-th non-zero pixel, α v Represents the weight, γ is the defined loss hyperparameter, and then the BEV conversion of image features is performed.

4. The cross-view fusion 3D object detection method based on cross-attention according to claim 1, characterized in that: In step S3, the multi-view image features with accurate depth information generate the image pseudo point cloud cone features, and the pseudo point cloud cone features are projected into voxels to obtain the image pseudo point cloud voxel features; the image pseudo point cloud voxel features are flattened along the Z axis to obtain the image BEV features.

5. The cross-view fusion 3D object detection method based on cross-attention according to claim 1, characterized in that: The multi-view point cloud backbone network includes BEV convolution layer and RV convolution layer; The BEV convolution is a three-dimensional sparse convolution with 128 input channels, 32 output channels, a convolution kernel size of (1, 1, 3), and a step size of (1, 1, 2). The RV convolution includes three sparse convolution layers. The number of input channels of the first convolution layer is 128, and the number of output channels is 32. The number of input and output channels of the second and third convolution layers are both 32. The convolution kernel sizes of the three convolution layers are all (1, 1, 3), and the step sizes are all (1, 1, 2). The generated BEV feature size is (N, C*D, H, W), and the generated RV feature size is (N, C*W, H, D), where N is the number of input samples, C is the number of channels, W is the width, H is the height, and D is the depth. The BEV convolution layer and the RV convolution layer respectively extract features of the lidar point cloud to obtain the BEV features and RV features of the lidar point cloud, and obtain the position information of the two features respectively.

6. The cross-view fusion 3D object detection method based on cross-attention according to claim 1, characterized in that: In step S5, the image BEV features and the lidar point cloud BEV features are spliced ​​together along the channel number to obtain a fused BEV feature.

7. The cross-view fusion 3D object detection method based on cross-attention according to claim 1, characterized in that: In step S6, the RPN network is used to refine the fused BEV features and RV features of the lidar point cloud, and the corresponding position information is added to the fused BEV features and RV features respectively. The cross-view cross-attention network is used for feature fusion. Each view is a feature sequence of length J, and the attention weight matrix between each item in the different view sequences is obtained. Apply it to the value matrix V j , and processed by a fully connected layer FC to obtain the fusion feature matrix after attention weighted aggregation The calculation formula of attention weighted fusion BEV features is as follows: Where Q is the query feature of BEV, is the key item feature of RV, V is the value item feature of RV, i and j are the numbers of each item, J is the total number of value items, d q Indicates the number of feature channels and obtains fusion features with attention weights 8. The cross-view fusion 3D object detection method based on cross-attention according to claim 1, characterized in that: In step S7, the regression detection head is used to obtain various regression parameters of the target box, and the final prediction result is obtained by calculating the fusion features and attention weights.

9. The cross-view fusion 3D object detection method based on cross-attention according to any one of claims 1 to 8, characterized in that: Define the loss function Loss, Loss = λ1L cls +λ2L reg +λ3L var , where λ1, λ2, λ3 are loss weights, L cls , L reg , L var Represents classification loss, regression loss and variance constraint loss. The classification loss uses CrossEntropyLoss to calculate the error between the predicted sample results and the real sample results. The regression loss uses SmoothL1Loss to calculate the loss between the detection box generated by the regression detection head and the real box. The variance constraint loss is used to constrain the variance of the attention weight, focusing attention on meaningful targets in the scene. The variance constraint formula of all real bounding boxes is: Among them, N box is the number of bounding boxes in the scene, N q is the number of cylinders enclosed by the bounding box q, A q [i] represents the attention weight of the i-th ground-truth bounding box, and Var(·) is the variance of the given quantity.

Images