Three-dimensional target detection method and system based on point cloud-image multi-cross hybrid and storage medium

Abstract

This invention provides a 3D target detection method, system, and storage medium based on point cloud-image multi-cross hybridization, comprising the following steps: Step 1: Extracting features from the image and the original point cloud respectively; combining the original point position encoding with image features through a coordinate thinning module; iteratively moving the sampling points towards the bounding box to enhance the coupling between the image and the point cloud; Step 2: In the voxel bird's-eye view feature extraction process, integrating fine-grained regional information within the voxel into global features to supplement the lack of local information in the Transformer, thereby obtaining a bird's-eye view feature map with a wider receptive field. Advantages of this invention: This invention integrates the advantages of both LiDAR and camera sensors to achieve 3D target detection technology in the field of autonomous driving. It can accurately identify and locate targets such as vehicles, pedestrians, and cyclists, and can be applied in real-world scenarios.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a three-dimensional target detection method, system, and storage medium based on point cloud-image multi-cross hybridization. Background Technology

[0002] 3D object detection is a crucial task in computer vision, aiming to detect the location, size, class, and other attributes of objects within a three-dimensional spatial structure. To overcome the limitations of single-mode cameras and LiDAR, multimodal 3D object detection technology has emerged. This technology primarily utilizes information from multiple different media to detect and locate objects in three-dimensional space. Multimodal 3D object detection is better at detecting objects in complex scenes, such as those with occlusion, reflections, and complex backgrounds. Despite progress in multimodal 3D object detection, significant differences exist between the object information detected by LiDAR and cameras due to the inherent characteristics of sensors. This leads to imbalanced training data, affecting model effectiveness. Furthermore, the different data dimensions and processing methods result in feature differences, impacting the coupling between multimodal data. Finally, current advanced multimodal Transformer methods primarily perform global fusion, integrating image features and point cloud features across the entire scene. This approach lacks fine-grained region-level information and fails to fully explore global contextual information, making it difficult to capture small, distant targets. Summary of the Invention

[0003] This invention provides a three-dimensional target detection method based on point cloud-image multi-cross hybridization, comprising the following steps:

[0004] Step 1: Extract features from the image and the original point cloud respectively. Combine the original point position encoding with the image features through the coordinate thinning module, and iteratively move the sampling points towards the bounding box to enhance the coupling between the image and the point cloud.

[0005] Step 2: During the voxel bird's-eye view feature extraction process, the fine-grained regional information within the voxel is integrated into the global feature to supplement the lack of local information in the Transformer, thereby obtaining a bird's-eye view feature map with a wider receptive field.

[0006] Step 3: Based on image features, bird's-eye view features, and point-level features, a hybrid representation is fused through a multi-intersecting Transformer. Finally, a region proposal network and a classification and regression learning network are used to achieve 3D object detection.

[0007] The present invention also provides a three-dimensional target detection system based on point cloud-image multi-cross hybridization, comprising: a memory, a processor, and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the three-dimensional target detection method of the present invention when called by the processor.

[0008] The present invention also provides a computer-readable storage medium storing a computer program configured to, when invoked by a processor, implement the steps of the three-dimensional target detection method described herein.

[0009] The beneficial effects of this invention are: This invention integrates the advantages of both lidar and camera sensors to realize 3D target detection technology in the field of autonomous driving, which can accurately identify and locate targets such as vehicles, pedestrians, and cyclists, and can be applied in real-world scenarios. Attached Figure Description

[0010] Figure 1 This is a framework diagram of a 3D target detection method based on point cloud-image multi-cross hybridization;

[0011] Figure 2 This is a diagram of the image self-attention module;

[0012] Figure 3 Point cloud coordinate refinement module diagram;

[0013] Figure 4 It is a diagram of a two-stream feature extraction network;

[0014] Figure 5 This is a diagram of a multi-cross-fusion Transformer module. Detailed Implementation

[0015] This invention discloses a three-dimensional target detection method (PIMFormer) based on point cloud-image multi-cross hybridization, which reduces the mismatch between local and global information during the dimensionality reduction process of multimodal information fusion in target detection, and improves the localization accuracy of 3D target detection.

[0016] This invention is applicable to multimodal 3D target detection for autonomous driving. The vehicle uses multiple sensors to collect target data and iteratively fuses data from different modalities to improve the accuracy of target detection.

[0017] This invention effectively fuses image features, bird's-eye view features, and point-level features using a multimodal Transformer to improve the localization accuracy of 3D target detection.

[0018] like Figures 1 to 5 As shown, this invention discloses a three-dimensional target detection method based on point cloud-image multi-cross hybridization, the specific steps of which are as follows:

[0019] Step 1: Extract features from the image and the original point cloud respectively. Design a coordinate thinning module to combine the original point position encoding with image features and iteratively move the sampling points towards the bounding box to enhance the coupling between the image and the point cloud and solve the matching problem caused by offset.

[0020] Step 1 includes:

[0021] Step 1: Design an image self-attention module based on the visual Transformer. This invention uses a modified ResNet as the 2D backbone network, where I∈R H×W×3 The image is taken as input, where H×W is the resolution of the original image, and the image has 3 RGB channels. A backbone network is then used to generate an image of size [size missing]. Feature map F I The image self-attention module of this invention consists of two sub-encoder layers: self-attention (SA) and feedforward neural network (FFN). Each sub-layer uses residual connections followed by normalization (LN) processing. First, to further process the 2D image using Transformer encoding, this invention incorporates image features... Adjusted to a series of 2D feature blocks F of size . IB ∈R N×D , where P 2 This is the resolution of each image feature block, therefore the number of feature blocks is... This is also the length of the input sequence for the Transformer. Specifically, it refers to the length of the input image features. Transform into query Q I Key K I Sum V I Q I =F IB W Q K I =F IB W K V I =F IB W V ,in and This is a linear projection matrix. The relevance values ​​are then normalized using the softmax function to obtain the attention weights. Finally, the self-attention weighted value is obtained by calculating the similarity between different labels using a dot product. Therefore, the self-attention weighted value is: Subsequently, the obtained self-attention weighted values ​​are input into the FFN to perform pointwise operations. This invention uses a multilayer perceptron (MLP) as the FFN, thus obtaining FFN(x) = MLP(x) = σ((xw1+b1)w2+b2), where... As the weights of the first layer, x is projected onto dimension K. These are the weights for the second layer. b1∈R K and It represents the bias, and σ(·) is a non-linear activation. In summary, the output of each encoder is expressed as... Finally, the transformed vector feature block F′ IB Reshape it into a two-dimensional feature map F I This is to facilitate subsequent cross-integration.

[0022] Step 2: Extracting features using point cloud coordinates. This invention introduces a two-stream network to acquire point cloud features and compensate for the lack of depth information of the target in the image. First, point cloud features are extracted from the original point cloud. Given an input point cloud P = {p1, p2, ..., p...} N To better cover the entire point cloud, farthest point sampling (FPS) is used to select a subset of points. Make It is relative to other sets of the same point. The point with the longest total distance. Then, a sphere query is used to find the center point. Construct a local region set S from the k nearest neighbors. i In this way, features associated with the target are obtained. Then, for P... s Points within the domain are clustered, and each resulting local feature is then subjected to convolution and pooling operations. The resulting multi-resolution features are used as local features for the center points. Finally, PointNet is used to encode the local region patterns into feature vectors. in Both μ and are MLP networks. Simultaneously, to establish the positional correlation between three-dimensional space and image features, this invention introduces a three-dimensional position encoder into the feature extraction network, aiming to extract two-dimensional image features... To obtain 3D features by associating with 3D location information Therefore, a 3D position encoder is described as Here, θ(·) represents the positional encoding (PE) part of the positional encoding function. Specifically, given a two-dimensional feature F I and three-dimensional coordinates p i First, p i The data is fed into a multilayer perceptron (MLP) network and converted into a 3D location embedding (PE). Then, the 2D features F are processed... I Perform a 1×1 convolutional layer transformation and add a 3D PE to form 3D position-aware features. Finally, the three-dimensional position features were respectively... and original point cloud features The merger was carried out in, For linking operations.

[0023] Step 2: In the voxel-based bird's-eye view feature extraction process, a dual-stream feature extraction network is employed. Fine-grained regional information within voxels is integrated into global features to supplement the lack of local information in the Transformer, resulting in a bird's-eye view feature map with a wider receptive field.

[0024] Step two includes:

[0025] Step A1: Local Transformer explores the neighborhood relationships of points within a voxel. First, the W×H×D point cloud space is divided into a uniformly distributed voxel mesh along the X, Y, and Z axes, where each voxel is V in size. W ×V H ×V D Due to the sparsity of point clouds, the number of points in each voxel may vary. To reduce learning bias, the same number of points N are randomly sampled for each voxel; zero-padding is used if a voxel has too few points. The points closest to the center within each voxel are used as a centroid set. Where N is the number of voxels. (The voxels are...) Points within the voxel are fed into a self-focused local structure for voxel-internal information aggregation. Given input... Output F L available: Where "⊙" represents the Hadamard product, and j represents a voxel. Sampling point p within j Index. The characteristic linear transformation function β generates the adaptive weight vector α(c i p j The aggregated feature vector β(p) j ρ is the softmax function. The attention weights α are a function used to calculate the weights c. i and p j The weights between them are used to decompose the function α as follows: The relational function δ outputs the fusion feature c. i and p j For a single feature vector, this invention uses the dot product method. ψ and ψ are characteristic linear projections. The mapping function γ is an MLP with two linear layers and a ReLU nonlinearity.

[0026] Step A2: The global Transformer focuses on the overall contextual relationships of voxels. Although local layers explore long-range dependencies, local transformation layers perform local processing of the point cloud. This invention also uses an additional global transformation layer to provide overall contextual information. It has a transformation structure similar to the local transformation layers, but focuses on all voxels F. L As input, rather than a local subset Features generated by the local and global transformer layers are cascaded to integrate local and global contexts. The output voxel F V available: Finally, submanifold sparse convolution is used to extract features, and spatial reconstruction is used to convert the voxel features F V Downsampling is converted into bird's-eye view (BEV) features F B And it is fed into the multi-cross fusion module. Meanwhile, based on F... B Generate a high-quality 3D preselection box. In short, 3D features along the Z-axis are superimposed, and the resulting box is of size [size missing]. F B The evaluation process generates pre-selection boxes, which are then optimized in the future.

[0027] Step 3: Based on image features, bird's-eye view features, and point-level features, a hybrid representation is fused into a multi-cross Transformer. Finally, a region proposal network and a classification and regression learning network are used to achieve 3D object detection.

[0028] Step three includes:

[0029] Step B1: Similar to the data processing method of the image self-attention module, the image feature block coordinates P corresponding to the point-level feature coordinates P are obtained by point-level feature projection. I and bird's-eye view feature block coordinates P B Projection yields: Among them, T LiDAR→cam and T LiDAR→bev C represents the transformation matrix from LiDAR coordinates to camera and bird's-eye view coordinates, respectively. rect and R rect These are the camera's calibration matrix and rotation correction, respectively.

[0030] After obtaining F p Corresponding F I and F B Then, similar to image self-attention, this invention will input point-level features F p F I and F B Transform into query Q * Key K * Sum V * : in, and This is a linear projection matrix. To enable the Transformer to adaptively learn weights between different modalities for complementary enhancement, this invention cross-fuses the question matrices of image, point-level, and bird's-eye view features to obtain a new fused query. Then, the fusion query is combined with the key K of another modality to obtain a multimodal multi-cross fusion weight. For example: With K B Combined. Then, using the normalization function, a new weighted score is obtained: New image, point-level, and bird's-eye view features are obtained through dot product calculation. Finally, features with multi-fusion cross-modal interactions are aggregated into new features. in Represents a connection.

[0031] Step B2: The present invention first uses a set of predicted bounding boxes B = {b} in a voxel bird's-eye view. k}, where each bounding box b = (u, v, d, w, l, h, θ) consists of a center position (u, v, d), a bounding box size (w, l, h), and a yaw rotation angle θ. This invention uses CenterPoint to predict the heatmap, target size, refinement position, and rotation angle for a specified class.

[0032] In summary, this invention integrates the advantages of both LiDAR and camera sensors to achieve 3D target detection technology in the field of autonomous driving. It can accurately identify and locate targets such as vehicles, pedestrians, and cyclists, and can be applied in real-world scenarios.

[0033] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A three-dimensional target detection method based on point cloud-image multi-cross hybridization, characterized in that, Includes the following steps: Step 1: Extract features from the image and the original point cloud respectively. Combine the original point position encoding with the image features through the coordinate thinning module, and iteratively move the sampling points towards the bounding box to enhance the coupling between the image and the point cloud. Step 2: During the voxel bird's-eye view feature extraction process, the fine-grained regional information within the voxel is integrated into the global feature to supplement the lack of local information in the Transformer, thereby obtaining a bird's-eye view feature map with a wider receptive field. Step 3: Based on image features, bird's-eye view features, and point-level features, a hybrid representation is fused through a multi-intersecting Transformer. Finally, a region proposal network and a classification and regression learning network are used to achieve 3D object detection. Step three includes the following steps: Step B1: Obtain the image feature block coordinates P corresponding to the point-level feature coordinates P by point-level feature projection. I and bird's-eye view feature block coordinates P B Projection yields: Among them, T LiDAR→cam and T LiDAR→bev C represents the transformation matrix from LiDAR coordinates to camera and bird's-eye view coordinates, respectively. rect and R rect These are the camera's calibration matrix and rotation correction; after obtaining F... p Corresponding F I and F B Then, input point-level features F p F I and F B Transform into query Q * Key K * Sum V * The question matrix of image, point-level, and bird's-eye view features is cross-fused to obtain a new fused query Q. ** Then, the fusion query is combined with the key K of another modality to obtain a multimodal multi-cross fusion weight; new image, point-level, and bird's-eye view features F′ are obtained through dot product calculation. * Finally, features with multi-fusion cross-modal interactions are aggregated into a new feature F: in Represents a connection; Step B2: Using a set of predicted bounding boxes B = {b} from the voxel bird's-eye view. k }, where each bounding box b = (u, v, d, w, l, h, θ) consists of the center position (u, v, d), the bounding box size (w, l, h), and the yaw rotation angle θ; CenterPoint is applied to predict the heatmap, target size, refinement position, and rotation angle of the specified class.

2. The three-dimensional target detection method according to claim 1, characterized in that, Step one includes the following steps: Step 1, design an image self-attention module based on the visual Transformer: use a modified ResNet as the 2D backbone network, and take the image of I as input; generate feature maps F through the backbone network. I The image self-attention module consists of two sub-encoder layers: a self-attention layer and a feedforward neural network layer. Each sub-layer uses residual connections and then performs normalization processing. The image features F are then processed. I Adjusted to a series of 2D feature blocks F of size . IB The number of feature blocks is N, which is also the length of the input sequence of the Transformer; Step 2, refine and extract features using point cloud coordinates: First, extract point cloud features from the original point cloud. Given an input point cloud P, use farthest point sampling to select a subset P of points. s , making It is relative to other sets of the same point. Find the point with the furthest total distance; then use a sphere query to find the center point. Construct a local region set s from k neighboring points. i Obtain features associated with the target; then analyze P. s Points within the domain are clustered, and each local feature is then subjected to convolution and pooling operations. The resulting multi-resolution features are used as local features of the center points. Finally, PointNet encodes the local region patterns into feature vectors. in Both μ and λ are MLP networks; a 3D position encoder is introduced into the feature extraction network, which extracts 2D image features F... I To obtain 3D features by associating with 3D location information The 3D position encoder is described as follows: in This is the position encoding part of the position encoding function.

3. The three-dimensional target detection method according to claim 2, characterized in that, In step 1, the input image features F Ip Transform into query Q I Key K I Sum V I Then, the relevant values ​​are normalized using the softmax function to obtain the attention weights; Finally, the self-attention weighted value is obtained by calculating the similarity between different labels through dot product; therefore, the self-attention weighted value is obtained. Subsequently, the obtained self-attention weights are input into the FFN for pointwise operation; a multilayer perceptron is used as the FFN, thus obtaining FFN(x) = MLP(x) = σ((xw1+b1)w2+b2), where w1 is used as the weight of the first layer to project x onto dimension K, w2 is the weight of the second layer, b1 and b2 are biases, and σ(·) is the nonlinear activation; the output of each encoder is represented as... Finally, the transformed vector feature block F′ IB Reshape it into a two-dimensional feature map F I This is to facilitate subsequent cross-integration.

4. The three-dimensional target detection method according to claim 2, characterized in that, In step 1, I∈R H×W×3 The image is taken as input, where H×W is the resolution of the original image, and the image has 3 RGB channels; a data structure of size [size missing] is generated through the backbone network. Feature map F I Image features Adjusted to a series of 2D feature blocks F of size . IB ∈R N×D , where P 2 This is the resolution of each image feature block, therefore the number of feature blocks is...

5. The three-dimensional target detection method according to claim 2, characterized in that, In step 2, given two-dimensional feature F I and three-dimensional coordinates p i First, p i The data is fed into a multilayer perceptron and converted into a 3D location embedding. Then, the 2D features F are processed. I Perform a 1×1 convolutional layer transformation and add a 3D PE to form 3D position-aware features. Finally, the three-dimensional position features were respectively... and original point cloud features The merger was carried out in, For linking operations.

6. The three-dimensional target detection method according to claim 1, characterized in that, Step two includes the following steps: Step A1: First, divide the W×H×D point cloud space into a uniformly distributed voxel grid along the X, Y, and Z axes, where each voxel is V in size. W ×V H ×V D For each voxel, randomly sample the same number of points N. If a voxel has data points less than a set value, zero-padding is used. The points closest to the center within each voxel are used as a centroid set C, where N is the number of voxels. Points within the voxel are fed into a self-focused local structure for voxel-internal information aggregation; given input Output F L available: Where ⊙ is the Hadamard product, and j is a voxel. Sampling point p within j Index; The characteristic linear transformation function β generates the adaptive weight vector α(c) i p j The aggregated feature vector β(p) j ); ρ is the softmax function; α is a function of attention weights to calculate the weights c. i and p j Weights between them; Step A2: Use a global transformation layer to provide overall contextual information for all voxels F L As input, features generated by the local and global transformer layers are cascaded to integrate local and global context; the output voxel F V available: Finally, submanifold sparse convolution is used to extract features, and spatial reconstruction is used to convert the voxel features F V Downsampling to bird's-eye view feature F B And feed it into the multi-cross fusion module; at the same time, based on F B Generate high-quality 3D preselection boxes.

7. The three-dimensional target detection method according to claim 6, characterized in that, In step A1, the function α is decomposed as follows: The relational function δ outputs the fusion feature c. i and p j A single feature vector, ψ is the characteristic linear projection, and the mapping function γ is an MLP with two linear layers and a ReLU nonlinearity.

8. A three-dimensional target detection system based on point cloud-image multi-cross hybridization, characterized in that, include: A memory, a processor, and a computer program stored on the memory, the computer program being configured to implement the steps of the three-dimensional target detection method of any one of claims 1-7 when invoked by the processor.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program configured to implement the steps of the three-dimensional target detection method according to any one of claims 1-7 when invoked by a processor.

Images