Image and point cloud fusion multi-modal three-dimensional target detection method

Abstract

The invention discloses an image and point cloud fusion multi-modal three-dimensional target detection method, which comprises the following steps: dynamically establishing bidirectional mapping of point cloud and voxels in a Bev view angle based on a voxelization method of double hash tables, and then extracting geometric features of each Bev voxel; after the point cloud is converted from a LiDAR coordinate system to an image coordinate system, the reflection intensity of the point cloud serves as a new channel to be attached to an image, four-channel image tensor representation is obtained, and global multi-level image semantic features are obtained through an adaptive feature extraction module based on perspective voxels; obtaining a regional semantic feature corresponding to the perspective voxel geometric feature through a perspective voxel projection method, and achieving the correlation alignment of the regional semantic feature and the point cloud geometric feature; a channel weight adaptive module based on a cross attention mechanism is adopted to weight geometric features and semantic features respectively, the weighted features are spliced to generate a feature tensor, and then a region proposal RPN network is adopted to carry out classification and regression tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing. Specifically, it relates to a multimodal three-dimensional object detection method for image and point cloud fusion. Background Art

[0002] In order to achieve robust and accurate environmental perception, intelligent vehicles are usually equipped with multiple sensors. Among them, cameras and lidars are widely used in three-dimensional object detection due to the complementarity of data. Cameras provide rich color and texture information in the form of high-resolution two-dimensional pixels, while lidars provide spatial position and size shape information in the form of sparse three-dimensional point clouds. Given the differences between the data of these two types of sensors, how to effectively integrate image information to strengthen point cloud data and then improve the accuracy of the detection model has become a crucial research topic.

[0003] After effectively integrating image information to strengthen point cloud data and obtaining the high-dimensional features encoded by the image and the point cloud, the next step is to consider how to perform fusion to achieve the interaction between the two types of features, so as to obtain a more comprehensive, effective, and robust data representation, with the expectation of achieving better three-dimensional object detection performance.

[0004] The existing implementation methods of fusion mainly differ in whether learning is required during the fusion process, and can be divided into two categories: learning-agnostic methods and learning-aware methods. Learning-agnostic methods fuse feature vectors or feature maps of different modalities by performing arithmetic operations (such as addition, averaging, etc.) and / or concatenation in a certain dimension. Learning-agnostic fusion methods are indeed simple and computationally efficient in feature fusion, but they do have certain limitations in terms of scalability and robustness. These operations mainly combine features based on preset arithmetic rules and cannot automatically learn the optimal feature fusion method. When dealing with complex tasks or large-scale data, simply relying on these simple operations may not be able to fully extract and utilize the deep relationships between features, thus limiting the further improvement of model performance. On the other hand, during the feature fusion process, there is a problem of uneven information carried by different perception methods, as introduced in the first chapter, which directly stems from their respective inherent perception limitations.

[0005] In summary, the existing image and point cloud fusion methods may not be able to fully extract and utilize the deep relationships between features when dealing with complex tasks or large-scale data, and there are limitations in terms of scalability and robustness. Summary of the Invention

[0006] Aiming at the problem that the existing image and point cloud fusion methods using learning-agnostic methods cannot fully extract and utilize the deep relationships between features only relying on these simple operations, the present invention provides a multimodal three-dimensional object detection method for image and point cloud fusion.

[0007] To achieve the above technical objectives, the technical solution adopted by the present invention is as follows:

[0008] An image and point cloud fusion multi-modal three-dimensional object detection method, comprising the steps of:

[0009] S1. Dynamically establish a two-way mapping relationship between the point cloud and the voxel based on the double hash table voxelization method, and then extract the geometric features of each bird's-eye voxel through the point cloud processing network;

[0010] S2. Convert the point cloud from the LiDAR coordinate system to the image coordinate system through the mapping relationship. After determining the corresponding position of each LiDAR point in the image, take the reflection intensity of each LiDAR point as a new channel and attach it to the image to obtain a 4-channel image tensor representation, and then send it to the adaptive image semantic feature acquisition module based on perspective voxels to obtain global multi-level image features;

[0011] S3. Obtain the regional semantic features corresponding to the geometric features of each perspective voxel through the perspective voxel projection method, and realize the association and alignment of the regional semantic features and the point cloud geometric features;

[0012] S4. Adaptively learn the weights between the point cloud and the image channels during the fusion by the model, determine the importance of each channel, obtain the respective channel weights of the corresponding geometric features and semantic features through the channel weight self-adaptation module using the cross-attention mechanism, weight the geometric features and semantic features respectively, and finally splice the weighted features to obtain the final fusion feature representation that is double-optimized by spatial attention and channel weights;

[0013] S5. Process the fusion features through the VFE voxel feature encoding layer. The output features combine point-wise features, local aggregation features, extended aggregation features, and dimension conversion. Convert the fused 4D feature tensor into a 3D feature tensor through preliminary feature extraction and dimension conversion, splice and fuse the feature maps of the feature fusion to generate a feature tensor, and use the region proposal RPN network for classification and regression tasks.

[0014] Further, the detailed steps of step S1 include:

[0015] S101. Hash dynamic point cloud voxelization: Convert the irregular point cloud in the three-dimensional space into equally spaced rasterized voxels Voxel. If the fixed size of each voxel is v w 、v H 、v D , then the point cloud space with lengths of W, H, and D along the x, y, and z axes will be divided into a number of W′ = W / v w 、H' = H / v H 、D' = D / vD Voxel grid, while setting the starting coordinates of the x, y, and z axes to x 0 , y 0 , z 0 ; Let v w , v H and v D have an integer multiple relationship with the corresponding W, H, and D. The voxel index formula for any point P(x, y, z) in the point cloud space is calculated as follows:

[0016]

[0017] Create two empty hash tables V(P) and P(V). V(P) is used to store the voxel index corresponding to each point, and P(V) is used to store the indices of all points within each voxel to establish a two-way relationship between points and corresponding voxels;

[0018] S102. Point cloud feature statistics: Use V to represent a non-empty voxel, and the voxel contains a point set:

[0019] p i =[x i , y i , z i , r i ∈ R 4} i = 1, 2,..., t,

[0020] where r i is the reflection intensity of the point; x i , y i , z i are the corresponding coordinates of the point on the x, y, and z axes; t represents the number of points in the voxel;

[0021] Calculate the coordinate average value of all points within the voxel V to obtain the centroid (c x , c y , c z ). Further, normalize the coordinates of the point p i with the centroid to obtain a new coordinate representation (x i - c x , y i - c y , z i - c z );

[0022] Obtain the geometric center point coordinates of the voxel (v x , v y , v z ), and obtain the distance d from this point to the coordinate origin through encoding; After splicing Finally, obtain a more abundant 11-dimensional feature description;

[0023] S103, Voxel feature extraction: The hash table P(V) stores the indices of all points within each voxel. By using the voxel indices, the point sets within the voxels can be quickly located, and a max pooling operation is performed to obtain the overall feature representation of each voxel.

[0024] Furthermore, the detailed steps of step S2 include:

[0025] S201, Global multi-level image semantic feature extraction:

[0026] Utilize prior knowledge and spatial geometric constraints to process the interaction between images and LiDAR point clouds:

[0027] The mapping relationship converts the sparse 3D point cloud points from the LiDAR coordinate system to the image coordinate system;

[0028] After determining the corresponding positions of each LiDAR point in the image, the reflection intensity of the point is attached to the image as a new channel, obtaining a new 4-channel image tensor representation (RGB-I);

[0029] Adopt the method of neighborhood averaging to average the sparse LiDAR points within the square neighborhood centered on the point cloud as the intensity information of the point cloud. The calculation formula is:

[0030]

[0031] Where N area is the number of points contained in the adjacent area area, I i represents the intensity value corresponding to the i-th point, and I * is the intensity value after interpolation at this position;

[0032] S202, Multi-level image semantic feature extraction based on the ResNet-50 backbone network and the Feature Pyramid Network FPN:

[0033] Extract features using a 2D object detection network pre-trained on the COCO dataset;

[0034] Extract the features and input them into the ResNet-50 basic network to obtain the interaction between the deep features. After obtaining the combined features, perform a lateral convolution operation again. Use a 3×3 convolution to aggregate each of the generated 5-layer features respectively, eliminate the overlapping effect brought by the upsampling process, and generate the final feature map;

[0035] Input the channel image tensor representation and convert it into feature maps of different sizes for output, corresponding to abstract information of different depths respectively.

[0036] Furthermore, obtain the regional semantic features corresponding to the perspective voxel geometric features:

[0037] Voxelize the 3D point cloud space based on the angle of LiDAR laser emission;

[0038] When determining the size of the perspective voxel, the interval between the horizontal and vertical directions is subdivided according to the resolution of the LiDAR;

[0039] Obtain the positions of the four sides and their corner points of the cross-section where the projection points are located in the LiDAR coordinate system through geometric relationships;

[0040] Map the alignment relationship to the feature map to obtain the corresponding regional semantic features.

[0041] Furthermore, the detailed steps of step S4 include:

[0042] Feature fusion with adaptive channel weights:

[0043] After obtaining the regional semantic features and point cloud geometric features of the image, first add the corresponding position elements of the two feature vectors for combination, and use the Tanh activation function to process the combined features;

[0044] Introduce a fully connected layer to further process the features and achieve non-linear combination with each other;

[0045] In the final stage of feature processing, use the Sigmoid activation function to normalize the feature values to the interval of 0 to 1. The output of the Sigmoid function is interpreted as probability and used to convert it into the weight of the image features;

[0046] Finally, fuse the semantic and geometric features by weighting them with the w weight respectively and then concatenating them;

[0047] Feature selection based on cross-attention mechanism:

[0048] Obtain the point cloud geometric feature F G and the corresponding image regional semantic feature F S After that, respectively pass through three coefficient matrices w Q 、w K 、w V Map them to their respective feature spaces. The geometric features of the corresponding point cloud are converted into queries Q (Query), and the semantic features of the image are converted into keys K (Key) and values V (Value) to obtain high-dimensional vector representations;

[0049] Steps of the cross-attention mechanism:

[0050] Use the dot product method to perform Attention calculation between each query Q and key K to obtain a spatial attention A vector matrix;

[0051] Written in vector form Q·K T , where T represents the transpose of the matrix;

[0052] Divided by a scaling factor

[0053] The result is normalized by Softmax to obtain a spatial cross-attention weight matrix, and this weight is used to weight the value V to obtain the final image semantic feature representation after attention;

[0054] The aggregated image semantic features are processed by a fully connected layer, and then the corresponding channel weights for geometric features and semantic features are obtained by using a channel weight self-adaptive module, and F G and F S are weighted respectively, and finally the weighted features are concatenated together to obtain the final fused feature representation that is double-optimized by spatial attention and channel weights.

[0055] Furthermore, the detailed steps of step S5 include:

[0056] Voxel feature encoding: The fused features are processed by a VFE voxel feature encoding layer, and the output features combine point-wise features and local aggregation features;

[0057] The combination of sparse convolution and sub-manifold sparse convolution forms four processing stages to expand the aggregated features;

[0058] Dimension conversion: Compress the two dimensions of the feature map tensor in the z-axis direction (that is, corresponding to the D channel) to obtain a three-dimensional feature tensor representation of (128, 200, 176), and obtain the features corresponding to the Bev view;

[0059] Two layers of 2D transposed convolution layers are used for feature depth fusion;

[0060] The two feature maps of feature depth fusion are concatenated and fused to generate a feature tensor;

[0061] Object classification prediction classifies the generated feature tensor to obtain the confidence score of each anchor box belonging to each class for the branch output;

[0062] Bounding box regression uses regression parameters of the center point coordinates, length and width dimensions, and orientation angle to obtain the branch output;

[0063] Orientation classification, corresponding to the orientation confidence of each anchor box.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] By associating and aligning semantic features with point cloud geometric features, then weighting the geometric features and semantic features respectively through the respective channel weights of the corresponding geometric features and semantic features, and finally splicing the weighted features together, a fused feature representation that is double-optimized by spatial attention and channel weights is obtained; finally, through the processing of the backbone network, the fused 4D feature tensor is initially feature-extracted and dimension-converted into a 3D feature tensor, the feature maps of feature fusion are spliced and fused to generate a feature tensor, and multiple classification methods are used to obtain the confidence levels of different classifications. It realizes the full extraction and utilization of the deep relationships between features when processing complex tasks or large-scale data, achieves the deep fusion of image and point cloud data, and improves the scalability and robustness of the entire image and point cloud fusion model or method. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 FIG. is an overall flowchart of a multi-modal three-dimensional object detection method for image and point cloud fusion according to an embodiment of the present invention;

[0067] Figure 2 FIG. is a schematic diagram of the framework of the entire multi-modal three-dimensional object detection model for image and point cloud fusion according to an embodiment of the present invention;

[0068] Figure 3 FIG. is a schematic diagram of dynamic voxelization based on a double hash table according to an embodiment of the present invention;

[0069] Figure 4 FIG. is a flowchart of point cloud feature extraction based on double hashing according to an embodiment of the present invention;

[0070] Figure 5 FIG. is a schematic diagram of the structure of an image feature extractor according to an embodiment of the present invention;

[0071] Figure 6 FIG. is a schematic diagram of the structure composed of a backbone network according to an embodiment of the present invention;

[0072] Figure 7 FIG. is a schematic diagram of the structure composed of a head network according to an embodiment of the present invention;

[0073] Figure 8 FIG. is a structural block diagram of point projection and voxel projection based on an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0074] For the convenience of understanding by those skilled in the art, the present invention will be further described below in conjunction with the embodiments and the drawings. The content mentioned in the embodiments does not limit the present invention.

[0075] As Figure 1 and 2 shown, this embodiment provides a multi-modal three-dimensional object detection method for image and point cloud fusion, including the steps:

[0076] S1. Dynamically establish a bidirectional mapping relationship between the point cloud and voxels based on the double hash table method, and then extract the geometric features of each bird's-eye view voxel through the point cloud processing network;

[0077] S2. Convert the point cloud from the LiDAR coordinate system to the image coordinate system through the mapping relationship. After determining the corresponding positions of each LiDAR point in the image, attach the reflection intensity of each LiDAR point as a new channel to the image to obtain a 4-channel image tensor representation, and then send it into the adaptive image semantic feature acquisition module based on perspective voxels to obtain global multi-level image features;

[0078] S3. Obtain the regional semantic features corresponding to the geometric features of each perspective voxel through the perspective voxel projection method, and realize the correlation alignment between the regional semantic features and the point cloud geometric features;

[0079] S4. Adaptively learn and fuse the weights between the point cloud and image channels during model adaptation to determine the importance of each channel. Use the channel weight self-adaptation module of the cross-attention mechanism to obtain the respective channel weights of the corresponding geometric features and semantic features, weight the geometric features and semantic features respectively, and finally splice the weighted features to obtain the final fusion feature representation that is double-optimized by spatial attention and channel weights;

[0080] S5. Process the fusion features through the VFE voxel feature encoding layer. The output features combine point-wise features, local aggregation features, extended aggregation features, and dimension conversion. Convert the fused 4D feature tensor into a 3D feature tensor through preliminary feature extraction and dimension conversion, splice and fuse the feature maps of the feature fusion to generate a feature tensor, and use the region proposal RPN network for classification and regression tasks.

[0081] Detailed steps of the voxelization method based on the double hash table:

[0082] As Figure 3 shown, the design of the hash dynamic point cloud voxelization module: Point cloud voxelization is to convert the irregular point cloud in three-dimensional space into equally spaced rasterized voxels. If the fixed size of each voxel is v w 、v H 、v D , then the point cloud space with lengths of W, H, and D along the x, y, and z axes will be divided into a voxel grid with quantities of W′ = W / v w 、H' = H / v H 、D' = D / v D , which is also the detection range. At the same time, set the starting coordinates of the x, y, and z axes to be x 0 , y 0 、z0 . To simplify the calculation process, assume that v w , v H and v D are in an integer multiple relationship with the corresponding W, H, and D. The voxel index to which any point P(x, y, z) in the point cloud space belongs can be quickly calculated through the following formula.

[0083]

[0084] Create two empty hash tables V(P) and P(V). V(P) is used to store the voxel index corresponding to each point, and P(V) is used to store the indices of all points within each voxel to establish a two-way relationship between points and corresponding voxels.

[0085] In the specific implementation, a specific three-dimensional space will be taken as the detection area. The x-axis corresponds to [0m, 70.4m], x 0 = 0m; the y-axis corresponds to [-40m, 40m], y 0 = -40m; the z-axis corresponds to [-3m, 1m], z 0 = -3m. Correspondingly, W = 70.4m, H = 80m, D = 4m. At the same time, it is projected onto the image plane, and the points outside those boundaries will be deleted to ensure alignment. In this paper, the voxel size is selected as V W = 0.05m, V H = 0.05m, and V D = 0.1m, and the corresponding voxel space dimensions are W' = 1408, H' = 1600, D' = 40.

[0086] Assume that in the original point cloud data of the current frame, there are a total of 6 voxels after voxel partitioning in this area. For each point in the point cloud, according to the above point cloud grouping based on indices, it can be determined which numbered voxel it should belong to, and then V(P) and P(V) are updated: for example, if there are two points with indices 1 and 2 in voxel No. 1 and one point with index 0 in voxel No. 6, then P(1) = {1, 2}, P(6) = {0}. According to the fact that point 0 falls within voxel No. 6, point 1 falls within voxel No. 1, and point 2 falls within voxel No. 1, then V(0) = {6}, V(1) = {1}, V(2) = {1}. Finally, the embedding of each voxel can be calculated according to P(V) (for example, by averaging or max-pooling the features of all points within the voxel), and then these voxel embeddings are used for subsequent detection tasks. In this way, a complete two-way mapping between the point cloud and voxels is established. There is no need to sample points into voxels with a fixed number and fixed capacity. The number of voxels and the number of points contained in each voxel are both dynamic, which eliminates the loss of random points and voxels and ensures that deterministic voxel embeddings can be generated, thereby obtaining more stable detection results.

[0087] Point cloud feature statistics: Let V represent a non-empty voxel, and the voxel contains a point set p i =[x i ,y i ,z i ,r i ∈R 4 [[ID=13}] i = 1, 2,..., t, where r i is the reflection intensity of the point; x i ,y i ,z i are the corresponding coordinates of the point on the x, y, and z axes; t represents the number of points in the voxel, and this value may vary due to the properties of the point cloud and the differences in dynamic voxelization processing. To describe the characteristics of the voxel in more depth, further calculations are performed here. First, this paper calculates the average value of the coordinates of all points in the voxel V to obtain the centroid (c x ,c y ,c z ), and further normalizes the coordinates of the point p i with the centroid to obtain a new coordinate representation (x i -c x ,y i -c y ,z i -c z ). Then, the coordinates of the geometric center point of the voxel (v x ,v y ,v z ) are also obtained, and the distance d from this point to the coordinate origin is obtained through encoding. After splicing the above information a more abundant 11-dimensional feature description is obtained.

[0088] Voxel feature extraction: In the previous step, a rich feature description of the points in the voxel was obtained. Next, this paper will obtain the high-dimensional feature representation of each voxel through a specially designed voxel feature extraction module. Specifically, as Figure 4As shown, the feature extraction module incorporates a fully connected layer for learning complex transformations of features, a BN (Batch Normalization) layer for stabilizing the training process and accelerating convergence, and a ReLU activation function for introducing non-linear characteristics. Next, using the hash table P(V), which stores the indices of all points within each voxel, the point set within the voxel can be quickly located through the index of the voxel, and a max pooling operation is performed to obtain the overall feature representation of each voxel. To make full use of global structural information and local detail information, this paper also concatenates the overall feature of the voxel with the feature of each point to form a 128-dimensional feature vector. In this process, the hash table V(P) plays a key role. It stores the voxel index corresponding to each point, enabling the voxel to which each point belongs to be quickly found here and implementing the concatenation operation with the voxel feature. The concatenated feature is then processed by a neural network layer to transform the original 128-dimensional feature into a more advanced and interactive 64-dimensional feature. This step not only reduces the dimension of the data, reduces the storage space and computational resource requirements, but also improves the generalization expression ability of the feature, providing a more effective feature representation for subsequent fusion and detection tasks.

[0089] The detailed steps of step S2 include:

[0090] Global multi-level image semantic feature extraction:

[0091] Using prior knowledge and spatial geometric constraints to process the image and LiDAR point cloud to achieve interaction:

[0092] The mapping relationship converts the sparse 3D point cloud points from the LiDAR coordinate system to the image coordinate system;

[0093] After determining the corresponding position of each LiDAR point in the image, the reflected intensity of the point is attached to the image as a new channel I to obtain a new 4-channel image tensor representation (RGB-I);

[0094] The method of neighborhood averaging is used to average the sparse lidar points within the square neighborhood centered on the point cloud as the intensity information of the point cloud. The calculation formula is:

[0095]

[0096] Where N area is the number of points contained in the adjacent area area, I i represents the intensity value corresponding to the i-th point, and I * is the intensity value after interpolation at this position.

[0097] Such as Figure 5As shown in the figure, multi-level image semantic feature extraction based on the ResNet-50 backbone network and the Feature Pyramid Network (FPN):

[0098] Use a pre-trained 2D object detection network on the COCO dataset for feature extraction, and perform fine-tuning and transfer learning training on the KITTI dataset for multi-level input and output requirements. Faster R-CNN is selected for training.

[0099] The ResNet-50 basic network has a total of 5 downsampling output feature layers, namely conv1 to conv5. Select the output results of the 2nd, 3rd, 4th, and 5th layers and perform a 2x upsampling operation to make their shapes and sizes consistent with the previous layer's feature map. After performing a 1x1 convolution, the output feature channels of the previous layer also become 256. Then, add the upsampled result element-wise to it, thus realizing the interaction between features of different depths. After obtaining the combined features, perform a horizontal convolution operation next. Use a 3x3 convolution to aggregate each of the 5 generated features again, eliminate the overlapping effect brought by the upsampling process, and generate the final feature map. For a frame of input RGB-I image with a shape of (1280, 384, 4), representing the width, height, and number of channels of the image respectively, after the data passes through the MSF multi-level image semantic feature module, it is converted into feature maps of different sizes for output, as Figure 5 shown. There are a total of 5 layers: P5(40, 12, 256), P4(80, 24, 256), P3(160, 48, 256), P2(320, 96, 256), and P1(640, 192, 256). They respectively correspond to abstract information of different depths, providing a rich feature basis for subsequent object detection tasks.

[0100] As Figure 8 shown, the regional semantic features corresponding to the perspective voxel geometric features:

[0101] Regional semantic feature extraction based on perspective voxels:

[0102] Voxelize the 3D point cloud space based on the angle of LiDAR laser emission.

[0103] When determining the size of the perspective voxel, the intervals between the horizontal and vertical directions are subdivided according to the LiDAR resolution.

[0104] Obtain the positions of the four sides and their corner points of the cross-section where the projection point P(x, y, z) is located through geometric relationships in the LiDAR coordinate system. The specific calculation formula:

[0105] y max = x·tan(arctan(y / x) + α / 2)

[0106] zmax = x·tan(arctan(z / x) + β / 2)

[0107] y min = x·tan(arctan(y / x) - α / 2)

[0108] z min = x·tan(arctan(z / x) - β / 2)

[0109] α and β correspond to the horizontal resolution and vertical resolution of the LiDAR in the dataset, and the cross-section corner coordinates are (x, y max , z min ), (x, y max , z max ), (x, y min , z min ), (x, y min , z max );

[0110] Map the alignment relationship onto the feature map to obtain the corresponding regional semantic features.

[0111] Based on the size change relationship in the above mapping transformation and convolution process, it can be determined that the corresponding 4 corner coordinates of the region of interest in the feature map plane are (x i , y i ), i = 1, 2, 3, 4. Calculate the extreme values of the x and y coordinates for normalization to obtain the new corner coordinates of the rectified region of interest (max{x i}, min{y i}), (max{x i}, max{y i}), (min{x i}, min{y i}), (min{x i}, max{y i)。To convert the feature maps of regions of interest (ROIs) of any size into a fixed shape and size, while more precisely aligning the features in the region to avoid information loss, RoI-Align in Faster RCNN is introduced to extract the features of this region. Specifically, the region of interest is first evenly divided into a 3×3 grid, and then bilinear interpolation is performed within each grid to obtain a feature map of a fixed size. The output of the FPN layer consists of 5 feature maps of different sizes, and each feature map is also in the form of a unified 256-dimensional tensor. Next, the features of the regions of interest are extracted from these feature maps, and then they are concatenated at the dimension level and finally merged into a 1280-dimensional vector representation. After that, to simplify the processing flow of the subsequent fusion module to be learned, a fully connected layer is introduced to integrate and reduce the dimensions of these features, and finally a 64-dimensional semantic feature vector representation representing the corresponding region of the perspective voxel is obtained.

[0112] Finally, it can adaptively align with the distance and obtain the features of the region of interest, effectively solving the problem of insufficient utilization of image information and loss of important information in the method based on point projection, and the problem of inaccurate feature alignment in the method based on voxel projection.

[0113] Detailed steps of the fusion module based on learned knowledge:

[0114] Feature fusion method with adaptive channel weights:

[0115] After obtaining the regional semantic features and point cloud geometric features of the image, first add the corresponding position elements of these two feature vectors and combine them, and use the Tanh activation function to process the added features;

[0116] Introduce a fully connected layer to further process the features and achieve non-linear combination with each other;

[0117] In the final stage of feature processing, use the Sigmoid activation function to normalize the feature values to the range of 0 to 1. The output of the Sigmoid function can be interpreted as a probability, so it is very suitable for converting it into the weight of image features.

[0118] Finally, adaptive channel weights are achieved through the above method, and the final fusion method is expressed by formula (4).

[0119]

[0120] w = σ(ω · tanh(F G + F S )) (3)

[0121] F fused = ConCat(w · F S , FG ) (4)

[0122] Among them, σ represents the activation function used, i.e., Sigmoid as shown in Equation (2); ω represents the learnable weight in the fully connected layer mentioned above for aggregation and dimensionality reduction; Tanh is the activation function shown in Equation (1); F G 、F S respectively represent the geometric and semantic features obtained correspondingly in step S3, and their dimensions are both 64. The above process is learnable, w is the learned weight value, and the model can learn the optimal fully connected layer parameter matrix through training. Finally, the semantic and geometric features are weighted respectively by the w weight and then concatenated to achieve fusion, as shown in formula (4), to obtain a 128-dimensional feature representation.

[0123] Feature selection based on cross-attention mechanism:

[0124] After obtaining the point cloud geometric feature F G and the corresponding image region semantic feature F S , they are respectively mapped to their respective feature spaces through three coefficient matrices w Q 、w K 、w V . The geometric feature corresponding to the point cloud is converted into a query Q (Query), while the semantic feature of the image is converted into a key K (Key) and a value V (Value) to obtain a high-dimensional vector representation;

[0125] Steps of the cross-attention mechanism:

[0126] Use the dot product method to perform Attention calculation between each query Q and key K to obtain a spatial attention A vector matrix;

[0127] Written in vector form Q·K T , where T represents the transpose of the matrix;

[0128] Divide by a scaling factor

[0129] Normalize the result through Softmax to obtain a spatial cross-attention weight matrix, and use this weight to weight the value V to obtain the final image semantic feature representation after attention;

[0130] The aggregated image semantic feature is processed by a fully connected layer, and then at this time, the respective channel weights corresponding to the geometric feature and the semantic feature obtained by the channel weight self-adaptive module can be used to weight F G and F S respectively, and finally the weighted features are concatenated together to obtain the final fusion feature representation that is double-optimized by spatial attention and channel weights;

[0131] Cross-attention mechanism calculation formula:

[0132] As Figure 6 and Figure 7 shown, the backbone network includes voxel feature encoding, sparse convolution (Spa_Conv3D), sub-manifold sparse convolution (Sub_Conv3D), and dimension transformation;

[0133] Voxel feature encoding: The fused features are processed through the VFE voxel feature encoding layer, and the output features combine point-wise features and local aggregation features; its tensor shape is (128, 40, 1600, 1408), representing the feature dimension and the D, H, W size dimensions in sequence.

[0134] The combination of sparse convolution and sub-manifold sparse convolution forms four processing stages to expand the aggregated features;

[0135] Dimension transformation: Compress two dimensions of the feature map tensor in the z-axis direction (i.e., corresponding to the D channel) to obtain a three-dimensional feature tensor representation of (128, 200, 176), and obtain the features corresponding to the Bev view.

[0136] The head network includes two layers of 2D transposed convolution layers, a concatenation layer, object classification, bounding box regression, and orientation classification;

[0137] The two layers of 2D transposed convolution layers perform deep feature fusion;

[0138] Perform concatenation fusion on the two feature maps with deep feature fusion to generate a feature tensor;

[0139] Object classification prediction classifies the generated feature tensor to obtain the confidence score of each anchor box belonging to each category for each branch output;

[0140] Bounding box regression uses regression parameters to obtain the center point coordinates, length and width dimensions, and orientation angle to obtain the branch output;

[0141] Orientation classification, corresponding to the orientation confidence of each anchor box.

[0142] Compared with the prior art, the present invention has the following beneficial effects:

[0143] By associatively aligning semantic features with point cloud geometric features, then weighting the geometric features and semantic features respectively through their respective channel weights corresponding to the geometric features and semantic features, and finally splicing the weighted features together, a fused feature representation that is doubly optimized by spatial attention and channel weights is obtained; finally, through the processing of the backbone network, the fused 4D feature tensor is initially feature-extracted and dimensionally transformed into a 3D feature tensor, the feature maps of feature fusion are spliced and fused to generate a feature tensor, and multiple classification methods are used to obtain the confidence levels of different classifications. This realizes the full extraction and utilization of the deep relationships between features when dealing with complex tasks or large-scale data, achieves the deep fusion of image and point cloud data, and improves the scalability and robustness of the entire image and point cloud fusion model or method.

[0144] The above has introduced in detail a multi-modal three-dimensional object detection method for image and point cloud fusion provided by this application. The description of specific embodiments is only used to help understand the method and its core idea of this application. It should be noted that for those of ordinary skill in the art of this technology, without departing from the principle of this application, several improvements and modifications can still be made to this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A multimodal three-dimensional object detection method by fusing images and point clouds, characterized in that: Includes steps: S1. The voxelization method based on the double hash table dynamically establishes a bidirectional mapping relationship between point cloud and voxel, and then extracts the geometric features of each bird's-eye view voxel through the point cloud processing network; S2, transform the point cloud from the LiDAR coordinate system to the image coordinate system through the mapping relationship, determine the corresponding position of each LiDAR point in the image, attach the reflection intensity of each LiDAR point to the image as a new channel, obtain a 4-channel image tensor representation, and then send it to the perspective voxel-based adaptive image semantic feature acquisition module to obtain global multi-level image features; S3, obtaining the regional semantic features corresponding to each perspective voxel geometric feature through the perspective voxel projection method, and realizing the association alignment between the regional semantic features and the point cloud geometric features; S4. The importance of each channel is determined by learning the weights between the point cloud and the image channels through model adaptive learning. The channel weight adaptive module of the cross-attention mechanism obtains the corresponding geometric features and the channel weights of the semantic features. The geometric features and the semantic features are weighted respectively, and finally the weighted features are spliced ​​to obtain the fused feature representation that is finally optimized by both spatial attention and channel weights. S5. The fused features are processed through the VFE voxel feature encoding layer. The output features combine point-by-point features, local aggregation features, extended aggregation features and dimensionality conversion. The fused 4-dimensional feature tensor is subjected to preliminary feature extraction and dimensionality conversion to become a 3-dimensional feature tensor. The feature maps of the feature fusion are concatenated and fused to generate a feature tensor. The regional proposal RPN network is used for classification and regression tasks.

2. The method for multimodal 3D target detection by fusion of image and point cloud according to claim 1, characterized in that: The detailed steps of step S1 include: S101, Double hash dynamic point cloud voxelization: Convert irregular point clouds in three-dimensional space into equally spaced gridded voxels. If each voxel has a fixed size of v w 、v H 、v D , then the point cloud space with lengths of W, H, and D along the three directions of the x, y, and z axes will be divided into a number of W′=W / v w 、H'=H / v H , D′=D / v D Voxel grid, and set the starting coordinates of x, y, and z axes to x0, y0, and z0 respectively; let V W 、v H and v D The corresponding W, H and D are integer multiples. The formula for calculating the voxel index of any point P (x, y, z) in the point cloud space is as follows: Create two empty hash tables V(P) and P(V). V(P) is used to store the voxel index corresponding to each point, and P(V) is used to store the index of all points in each voxel to establish a bidirectional relationship between the point and the corresponding voxel. S102. Point cloud feature statistics: V is used to represent a non-empty voxel, which contains a point set: p i =[x i ,y i ,z i ,r i ]∈R 4 }i=1,2,...,t, where r i is the reflection intensity of the point; x i ,y i ,z i are the x, y, and z coordinates of the point; t represents the number of points in the voxel; The average coordinates of all points in the voxel V are calculated to obtain the center of mass of the voxel (c x ,c y ,c z ), further point p i The coordinates are normalized with the center of mass to obtain a new coordinate representation (x i -c x ,y i -c y ,z i -c z ); Obtain the geometric center coordinates of the voxel (v x ,v y ,v z ), and the distance d from the point to the origin of the coordinates is obtained by encoding; after splicing Finally, we get a richer 11-dimensional feature description; S103, voxel feature extraction: The hash table P(V) is used to store the indexes of all points in each voxel, and the point set in the voxel is quickly located through the voxel index, and a maximum pooling operation is performed to obtain the overall feature representation of each voxel.

3. The method for multimodal 3D target detection by fusion of image and point cloud according to claim 2, characterized in that: The detailed steps of step S2 include: S201, global multi-level image semantic feature extraction: Use prior knowledge and spatial geometric constraints to process images and LiDAR point clouds to achieve interaction: The mapping relationship transforms the sparse 3D point cloud points from the LiDAR coordinate system to the image coordinate system; After determining the corresponding position of each LiDAR point in the image, the reflection intensity of the point is attached to the image as a new channel to obtain a new 4-channel image tensor representation (RGB-I); The neighborhood averaging method is used to average the sparse lidar points in the square neighborhood centered on the point cloud as the intensity information of the point cloud. The calculation formula is: Where N area is the number of points contained in the adjacent area, I i Indicates the intensity value corresponding to the i-th point, I * is the interpolated intensity value at that position; S202, Multi-level image semantic feature extraction based on ResNet-50 backbone network and feature pyramid network FPN: Extract features from a pre-trained 2D object detection network on the COCO dataset; The extracted features are input into the ResNet-50 basic network to obtain the interaction between the deep features. After obtaining the combined features, the horizontal convolution operation is performed again. The generated 5 layers of features are aggregated again using 3×3 convolution to eliminate the overlapping effect caused by the upsampling process and generate the final feature map. The input channel image tensor representation is converted into feature map outputs of different sizes, corresponding to abstract information of different depths.

4. The method for multimodal 3D target detection by fusion of image and point cloud according to claim 3, characterized in that: Get the regional semantic features corresponding to the perspective voxel geometric features: Voxelize the 3D point cloud space based on the angle of the LiDAR laser emission; When determining the size of the perspective voxel, the horizontal and vertical intervals are subdivided according to the resolution of the LiDAR; The positions of the four sides and corner points of the cross section where the projection point is located in the LiDAR coordinate system are obtained through geometric relationships; The alignment relationship is projected onto the feature map to obtain the corresponding regional semantic features.

5. The method for multimodal 3D target detection by fusion of image and point cloud according to claim 4, characterized in that: The detailed steps of step S4 include: Feature fusion with adaptive channel weights: After obtaining the regional semantic features and point cloud geometric features of the image, the elements of the corresponding positions of the two feature vectors are first added together, and the added features are processed using the Tanh activation function; A fully connected layer is introduced to further process the features and realize nonlinear combination between them; In the final stage of feature processing, the Sigmoid activation function is used to normalize the feature values ​​to the range of 0 to 1. The output of the Sigmoid function is interpreted as a probability, which is used to convert it into the weight of the image feature; Finally, the semantic and geometric features are weighted by w weights and then spliced ​​to achieve fusion; Feature selection based on cross-attention mechanism: Get the point cloud geometric features F G and the corresponding image region semantic feature F S After that, three coefficient matrices w are used respectively. Q 、w K 、w V Map them to their respective feature spaces, the geometric features of the corresponding point cloud are converted into query Q (Query), and the semantic features of the image are converted into key K (Key) and value V (Value) to obtain a high-dimensional vector representation; Steps of the Cross-Attention Mechanism: Use the dot product method to calculate the attention of each query Q and key K to obtain a spatial attention vector matrix A; Written in vector form Q·K T , where T represents the transpose of the matrix; Divide by a scaling factor The result is normalized by Softmax to obtain a spatial cross-attention weight matrix, and the value V is weighted by this weight to obtain the final image semantic feature representation after attention; The aggregated image semantic features are processed through a fully connected layer, and then the channel weights corresponding to the geometric features and semantic features are obtained by the channel weight adaptation module. G and F S Each of them is weighted, and finally the weighted features are spliced ​​together to obtain the final fused feature representation that is doubly optimized by spatial attention and channel weights.

6. The method for multimodal 3D target detection by fusion of image and point cloud according to claim 5, characterized in that: The detailed steps of step S5 include: Voxel feature encoding: The fused features are processed by the VFE voxel feature encoding layer, and the output features combine point-by-point features and local aggregation features; Sparse convolution and submanifold sparse convolution are combined to form four processing stages to expand the aggregated features; Dimension conversion: compress the two dimensions of the feature map tensor in the z-axis direction (corresponding to the D channel) to obtain a three-dimensional feature tensor representation of (128, 200, 176) to obtain the features corresponding to the Bev perspective; Two 2D deconvolution layers are used for deep feature fusion; Concatenate and fuse the two feature maps of feature depth fusion to generate a feature tensor; The target classification prediction classifies the generated feature tensor and obtains the confidence score of each anchor box belonging to each category corresponding to the branch output; The bounding box regression uses the regression parameters of the center point coordinates, length and width, and orientation angle to obtain branch outputs; Direction classification, corresponding to the orientation confidence of each anchor box.

Images