A pure vision target detection method based on image-pseudo point cloud feature fusion

Abstract

The application discloses a kind of pure vision target detection methods based on image-pseudo point cloud feature fusion, belong to the field of automatic driving technology.The application mainly includes the following five steps:1. based on the basis of pure vision three-dimensional target detection model training of image;2. three-dimensional target detection model training based on laser radar point cloud;3. design mask automatic encoder-decoder realizes pseudo point cloud feature generation;4. image feature and pseudo point cloud feature adaptive fusion;5. fine-tuning obtains pure vision target detection model based on pseudo feature fusion.Based on the pre-training scheme proposed in the application, only the corresponding laser radar point cloud data needs to be provided during model training, and only the camera image needs to be input during inference, to generate high-quality pseudo point cloud features, through the cross-modal feature fusion of image features-pseudo point cloud features, to improve the performance based on the original pure vision model.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, specifically a pure visual target detection method based on image-pseudo-point cloud feature fusion. Background Technology

[0002] Currently, autonomous driving vehicle sensors primarily consist of LiDAR and cameras. In the field of object detection algorithms, these mainly fall into three categories: LiDAR point cloud-based, camera image-based, and multi-modal fusion-based solutions. Among these, point cloud-based and multi-modal fusion-based solutions exhibit superior performance in 3D object detection tasks due to their ability to obtain spatial prior information. In contrast, purely vision-based 3D object detection models lag significantly behind the former in performance. A crucial reason for this is the lack of realistic depth information in images, which hinders 3D scene reconstruction. Therefore, purely vision-based 3D object detection remains an extremely challenging task.

[0003] While LiDAR can provide high-quality point cloud data, its high cost and susceptibility to environmental factors make cameras, as a general-purpose automotive sensor, still essential. Compared to LiDAR sensors, cameras offer unique advantages such as low cost, rich color information, dense sensing capabilities, and ease of deployment. They hold significant commercial value for the industrialization and promotion of autonomous driving, thus image-based 3D object detection research continues to receive considerable attention despite its relatively lower performance.

[0004] Given the inherent advantages of LiDAR point clouds over images in 3D object detection, LiDAR-based models can be used as teacher models to guide purely visual 3D object detection models. However, common approaches to cross-modal knowledge distillation merely mimic the teacher model in feature extraction, attempting to obtain higher-quality feature maps and thus improve model performance. This is problematic because images inherently lack depth information, and the vast differences between modalities often make it difficult to learn effective high-dimensional features. Therefore, effectively learning high-quality scene features from LiDAR point cloud models using only image input and then rationally utilizing these features to improve final model performance remains a critical technical challenge. Summary of the Invention

[0005] This invention proposes a pure visual target detection method based on image-pseudo-point cloud feature fusion. The aim is to use an effective pre-training scheme to provide high-quality feature maps using a LiDAR point cloud model only during the pre-training stage. This allows a 3D target detection model with pure visual input to learn pseudo-point cloud features containing certain point cloud-specific spatial information. Multimodal feature fusion technology is then used to fuse image modal features and pseudo-point cloud modal features, thereby significantly improving performance on the basis of the original pure visual model.

[0006] The technical solution adopted in this invention is as follows:

[0007] The subscripts "img" and "pc" are used to distinguish between the image and point cloud branches. The subscript "bev" represents the bird's-eye view format. The subscript "pseudo" represents pseudo point cloud features. The subscript "fus" represents cross-modal fusion features.

[0008] A purely visual object detection method based on image-pseudo-point cloud feature fusion includes the following steps:

[0009] Step (1). Obtain the surround view image data collected by the vehicle camera, use the image-based BEV feature generation network to extract features and project them to generate the three-dimensional bird's-eye view features of the image modality, use the image target detection output head to realize the image-based target detection task, and pre-train the first model composed of the image-based BEV feature generation network and the image target detection output head.

[0010] Step (2). Obtain the point cloud data collected by the lidar, and after data preprocessing, generate the three-dimensional bird's-eye view features of the point cloud modality based on the BEV feature generation network of the point cloud. Use the point cloud target detection output head to realize the target detection task based on the point cloud. Pre-train the second model composed of the point cloud-based BEV feature generation network and the point cloud target detection output head.

[0011] Step (3). The 3D bird's-eye view features of the image modality generated by the pre-trained model in step (1) are randomly masked according to the proportion and input into the encoder. The encoding result of the unmasked part is output. A learnable feature vector is designed as the encoding result of the masked part. Both are input into the decoder to obtain pseudo point cloud features. The 3D bird's-eye view features of the point cloud modality generated by the pre-trained model in step (2) are used as supervision signals to pre-train the encoder and decoder. The pseudo point cloud feature generation network is composed of the pre-trained encoder and decoder.

[0012] Step (4). Fine-tune the pure visual object detection model consisting of the pseudo point cloud feature generation network, the multimodal feature fusion network, the fused target output head and the image-based BEV feature generation network in the pre-trained first model. After training, a pure visual object detection model based on pseudo feature fusion is obtained.

[0013] Step (5). Using only the surround view image captured by the vehicle camera as input, generate the target detection result using a pure visual target detection model based on pseudo-feature fusion.

[0014] Furthermore, the image-based BEV feature generation network employs a two-dimensional neural network. Preferably, the two-dimensional neural network uses a convolutional neural network combined with projection based on camera intrinsic and extrinsic parameters (LSS: Lift, Splat, Shoot), and the two-dimensional neural network is a prior art technology.

[0015] Furthermore, in step (2), the second model also includes a voxel feature extraction network, which is used to initially encode irregular point cloud data into voxel features, thereby realizing data preprocessing of the point cloud data acquired by the lidar. Preferably, the voxel feature extraction network is implemented using a fully connected network (FC).

[0016] Furthermore, the point cloud-based BEV feature generation network employs a three-dimensional neural network. Preferably, the three-dimensional neural network can be implemented using a three-dimensional sparse convolutional network (Sparse Convolutional Network).

[0017] Furthermore, in step (3), the mask ratio is 50%-75%.

[0018] Furthermore, in step (3), the encoder and decoder adopt a Transformer structure.

[0019] Furthermore, in step (3), the encoder and decoder adopt an asymmetric lightweight design, with the number of encoder layers and the number of decoder layers being 3:1.

[0020] Furthermore, in step (3), the learnable feature vector has the same feature dimension as the three-dimensional bird's-eye view of the image modality.

[0021] Furthermore, step (4) includes:

[0022] Step (4.1). Combine the image-based BEV feature generation network and the pseudo-point cloud feature generation network in the pre-trained first model, and design a multimodal feature fusion network and a fusion target output head; the multimodal feature fusion network includes a splicing layer and a convolutional neural network layer;

[0023] Step (4.2). Using the surround view image captured by the vehicle camera as input, the image-based BEV feature generation network in the pre-trained first model extracts features and projects them to generate a three-dimensional bird's-eye view feature of the image modality.

[0024] Step (4.3). Using the 3D bird's-eye view features of the image modality obtained in step (4.2) as input, the pseudo point cloud features are generated by the pseudo point cloud feature generation network;

[0025] Step (4.4). Using the 3D bird's-eye view features of the image modality obtained in step (4.2) and the pseudo point cloud features obtained in step (4.3) as input, the two are stitched together by the stitching layer in the multimodal feature fusion network and then the adaptive fusion features are generated by the convolutional neural network layer.

[0026] Step (4.5). Using the adaptive fusion features obtained in step (4.3) as input, the target detection result is generated by the fusion target output head;

[0027] Step (4.6). Using the real target results of the surround view image collected by the vehicle camera in step (4.2) as labels, perform end-to-end training on the image-based BEV feature generation network, pseudo-point cloud feature generation network, multimodal feature fusion network and fusion target output head in the pre-trained first model, fine-tune the parameters, and obtain a pure visual target detection model based on pseudo-feature fusion.

[0028] Furthermore, the image target detection output head, point cloud target detection output head, and fusion target output head are three independent multilayer perceptrons.

[0029] Advantages of this invention:

[0030] This invention proposes a complete design scheme for a pure visual object detection model based on image-pseudo-point cloud feature fusion, including multiple stages such as basic pure visual object detection model pre-training, LiDAR point cloud object detection model pre-training, cross-modal pseudo-point cloud feature generation pre-training, and cross-modal feature fusion fine-tuning. In the pre-training stage, the trained point cloud model serves as a teacher model, providing relatively high-quality scene information features. Pseudo-point cloud features are learned from image features through the attention mechanism in the mask and encoder / decoder. In the fine-tuning stage, image features and pseudo-point cloud features are adaptively fused to obtain the object detection results. After a series of pre-training and fine-tuning steps, the final model achieves a significant performance improvement over the original pure visual model. Attached Figure Description

[0031] Figure 1 This is a flowchart of the pure visual target detection method based on image-pseudo-point cloud feature fusion proposed in this invention. Detailed Implementation

[0032] The present invention will be further described below with reference to the accompanying drawings. The drawings are merely illustrative diagrams of the present invention. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0033] like Figure 1 As shown, the specific implementation steps of the pure visual target detection method based on image-pseudo-point cloud feature fusion of the present invention are as follows:

[0034] Step (1). Input the surround view image data collected by the vehicle-mounted camera sensor. After feature extraction and feature projection, generate a bird's-eye view feature in three-dimensional space. Input the feature into the target detection output head to realize the target detection task, specifically:

[0035] Step (1.1). Input the panoramic image data I collected by the camera. img Through an image-based BEV feature generation network (two-dimensional neural network) Θ img Extracting image features F img :

[0036] F img =Θ img (I img )

[0037] Step (1.2). Project the image features generated in step (1.1) into the three-dimensional bird's-eye view space to form the three-dimensional bird's-eye view features F of the image modality. ibev :

[0038]

[0039] Step (1.3). Analyze the 3D bird's-eye view features of the image modality F. ibev Input 3D target detection output head Generate target detection results:

[0040]

[0041] Among them, the product generated in step (1.2) Representing the bird's-eye view features in three-dimensional space, X×Y corresponds to the horizontal coverage area, which can be set to 128×128, while the vertical dimension is compressed to 1. C represents the feature dimension, which can be set to 256; O det This indicates the target detection result. It represents the projection of two-dimensional planar features onto three-dimensional spatial bird's-eye view features.

[0042] The image-based BEV feature generation network Θ described above img and 3D target detection output head A basic pure visual object detection model can be constructed. A loss function for pre-training can be designed based on the object detection results to complete the pre-training of the basic pure visual object detection model, which is denoted as image modality pre-training.

[0043] Step (2). Input the point cloud data collected by the lidar, process the data and extract features to generate a bird's-eye view feature in three-dimensional space, input the feature into the target detection output head to achieve the target detection task, specifically:

[0044] Step (2.1). Input the point cloud data I collected by the lidar. pc The irregular point cloud data is initially encoded into voxel features through a voxel feature extraction network v.

[0045]

[0046] Step (2.2). The voxel feature F generated in step (2.1) voxel Input: Point cloud-based BEV feature generation network (3D neural network) Θ pc 3D bird's-eye view features forming point cloud modalities

[0047] F pbev =Θ pc (F voxel )

[0048] To facilitate the subsequent generation of pseudo-point cloud features, the spatial resolution and feature dimension are kept consistent with those of the image features.

[0049] Step (2.3). The bird's-eye view features F of the point cloud modality... pbev Input 3D target detection output head Generate target detection results:

[0050]

[0051] Among them, the generated in step (2.1) For the encoded voxel features, X×Y×Z corresponds to each direction in three-dimensional space, representing the coverage area of ​​the voxel features. This is generally determined based on the point cloud range and voxel size. For example, if the point cloud range is [-51.2, -51.2, -5, 51.2, 51.2, 3] and the voxel size is [0.1, 0.1, 0.2], then X×Y×Z = 1024×1024×40, and D is the dimension of the encoded voxel features, which can be set to 10. The F generated in step (2.2) pbevFor the 3D bird's-eye view features of the point cloud modality, to facilitate feature learning, the feature dimensions and spatial resolution are set to the same as in step (1.2); O det This indicates the target detection result.

[0052] The aforementioned voxel feature extraction network V and point cloud-based BEV feature generation network Θ pc and 3D target detection output head A complete point cloud object detection model can be constructed. A loss function for pre-training can be designed based on the object detection results to complete the pre-training of the point cloud object detection model, which is called point cloud modal pre-training.

[0053] Step (3). Generate pseudo-point cloud features using a pseudo-point cloud feature generation network. The pseudo-point cloud feature generation network adopts a mask auto-encoder / decoder, which includes a random mask generator, an encoder and a decoder based on the Transformer structure. The image features generated by the model in step (1) are randomly masked according to a certain proportion and input into the encoder. The feature mask positions output by the encoder are replaced with learnable feature vectors of the same feature dimension and then input into the decoder. The three-dimensional bird's-eye view features of the point cloud modality generated by the model in step (2) are used as supervision signals to train the pseudo-point cloud feature generation network to obtain pseudo-point cloud features F. pseudo Specifically:

[0054] Step (3.1). Calculate the 3D bird's-eye view features F of the image modality generated in step (1.2). ibev Randomly code the remaining visible portion according to a certain proportion (e.g., 75%), and then feed the remaining visible portion into the pseudo-point cloud feature encoder ε to obtain the encoding result T of the visible portion:

[0055] T = ε(Mask(F) ibev ))

[0056] Step (3.2). Transfer the learnable feature vectors As the encoding result of the mask part in step (3.1), where C represents the dimension of the feature vector, which is the same as the dimension of the image features, both are fed into the pseudo-point cloud feature decoder D to generate pseudo-point cloud features F. pseudo :

[0057]

[0058] Step (3.3). Use the 3D bird's-eye view features F of the point cloud modality generated in step (2.2). pbev As a supervisory signal, the pseudo-point cloud feature generation network is trained:

[0059]

[0060]

[0061] Where MSE represents the mean squared error loss function, and the pseudo-point cloud feature generation network is pre-trained based on this loss function; y i This represents the i-th element in the pseudo-point cloud feature. The i-th element in the 3D bird's-eye view features of the point cloud modalities, where n represents the number of features, and Mask(.) represents the random mask generator, encoder ε, and decoder used to perform the masking operation. Both are based on the Transformer architecture, consisting of k identical cascaded networks. Each layer includes attention-weighted summation, multilayer perceptrons, and layer normalization modules. Because the decoder incorporates a large number of learnable mask features, computational resource consumption increases significantly; therefore, an asymmetric design is adopted. e k d Set to 9 and 3 respectively, Q, K, and V used in the attention-weighted summation calculation are all linear mappings of the input features, and the specific calculation method is as follows:

[0062] Q = XW Q K = XW K V = XW V

[0063]

[0064] Q←Q+MLP(Q)

[0065] The feature F generated in step (3) pseudo The pseudo-point cloud features obtained by relying on a pure visual model in this invention can be used for subsequent cross-modal feature fusion to improve model performance.

[0066] Step (4). Reorganization and fine-tuning of the pure visual object detection model

[0067] In the image-based BEV feature generation network Θ img Based on the pseudo-point cloud feature generation network, a multimodal feature fusion network and a fused target detection output head are added to construct a reconstructed pure visual target detection model. The multimodal feature fusion network will combine the 3D bird's-eye view features F of the image modality generated by the model in step (1). ibev and the pseudo-point cloud features F generated in step (3) pseudo To perform adaptive fusion, the two features are first concatenated along their feature dimensions. Then, a convolutional neural network is used to reduce the dimensionality of the fused features, generating the adaptive fused feature F. fus :

[0068]

[0069] Where Φ represents a convolutional neural network, Represents the splicing operation, adaptive feature fusion. The concatenated feature channel C is identical to the original image features, so the spatial resolution remains unchanged. The adaptive fusion feature will replace the image features in the basic pure visual object detection model to obtain the object detection result. That is, the adaptive fusion feature is used as the input to the fusion object detection output head to output the object detection result.

[0070] The fine-tuning process of the recombined pure visual object detection model is as follows: using only the panoramic images captured by the camera as input, and passing them through a pre-trained image-based BEV feature generation network Θ. img After image feature projection, a bird's-eye view feature F is generated in the image modality. ibev The pre-trained pseudo-point cloud feature generation network is used to convert it into pseudo-point cloud features F. pseudo Then, cross-modal feature fusion is achieved through a multimodal feature fusion network, and the fused target detection output head is connected. Output the target detection results, use the real detection results as labels, and fine-tune the recombined pure visual target detection model to obtain a pure visual target detection model based on pseudo-feature fusion.

[0071] In practical applications, only the panoramic images captured by the camera are used as input, and target detection is achieved through a pure visual target detection model based on pseudo-feature fusion. Compared with the basic pure visual target detection model in step (1), the detection accuracy is significantly improved.

[0072] The above description is merely a specific embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technologies, but should also cover other technical solutions formed by arbitrary combinations of the above-described technologies or their equivalents without departing from the concept of this application. For example, technical solutions formed by substituting the above-described technologies with (but not limited to) technologies with similar functions disclosed in this application.

Claims

1. A purely visual object detection method based on image-pseudo-point cloud feature fusion, characterized by the following steps: Step (1). Obtain the surround view image data collected by the vehicle camera, use the image-based BEV feature generation network to extract features and project them to generate the three-dimensional bird's-eye view features of the image modality, use the image target detection output head to realize the image-based target detection task, and pre-train the first model composed of the image-based BEV feature generation network and the image target detection output head. Step (2). Acquire point cloud data collected by lidar, and after data preprocessing, generate three-dimensional bird's-eye view features of point cloud modality based on the BEV feature generation network of point cloud. Use point cloud target detection output head to realize target detection task based on point cloud. Pre-train the second model composed of point cloud-based BEV feature generation network and point cloud target detection output head. Step (3). The 3D bird's-eye view features of the image modality generated by the pre-trained model in step (1) are randomly masked proportionally by a random mask generator and then input into the encoder. The output is the encoding result of the unmasked part. A learnable feature vector is designed as the encoding result of the masked part. Both are input into the decoder to obtain pseudo point cloud features. The 3D bird's-eye view features of the point cloud modality generated by the pre-trained model in step (2) are used as supervision signals. A random mask generator, encoder and decoder are pre-trained, and a pseudo point cloud feature generation network is formed by the pre-trained random mask generator, encoder and decoder. Step (4). Fine-tune the pure visual object detection model consisting of the pseudo point cloud feature generation network, the multimodal feature fusion network, the fused target output head and the image-based BEV feature generation network in the pre-trained first model. After training, a pure visual object detection model based on pseudo feature fusion is obtained. Step (5). Using only the surround view image captured by the vehicle camera as input, generate the target detection result using a pure visual target detection model based on pseudo-feature fusion.

2. The pure visual target detection method based on image-pseudo-point cloud feature fusion according to claim 1 is characterized in that the image-based BEV feature generation network adopts a two-dimensional neural network.

3. The pure visual target detection method based on image-pseudo-point cloud feature fusion according to claim 1 is characterized in that, in step (2), the second model further includes a voxel feature extraction network, which is used to initially encode irregular point cloud data into voxel features to realize data preprocessing of point cloud data collected by lidar.

4. The pure visual target detection method based on image-pseudo-point cloud feature fusion according to claim 1 is characterized in that the point cloud-based BEV feature generation network adopts a three-dimensional neural network.

5. The pure visual target detection method based on image-pseudo-point cloud feature fusion according to claim 1 is characterized in that, in step (3), the mask ratio is 50%-75%.

6. The pure visual target detection method based on image-pseudo-point cloud feature fusion according to claim 1 is characterized in that, in step (3), the encoder and decoder adopt the Transformer structure.

7. The pure visual target detection method based on image-pseudo-point cloud feature fusion according to claim 6 is characterized in that, in step (3), the encoder and decoder adopt an asymmetric lightweight design, and the number of encoder layers and the number of decoder layers are 3:

1.

8. The pure visual target detection method based on image-pseudo-point cloud feature fusion according to claim 1 is characterized in that, in step (3), the learnable feature vector has the same dimension as the three-dimensional bird's-eye view feature of the image modality.

9. The pure visual target detection method based on image-pseudo-point cloud feature fusion according to claim 1, characterized in that step (4) includes: Step (4.1). Combine the image-based BEV feature generation network and the pseudo-point cloud feature generation network in the pre-trained first model, and design a multimodal feature fusion network and a fusion target output head; the multimodal feature fusion network includes a splicing layer and a convolutional neural network layer; Step (4.2). Using the surround view image captured by the vehicle camera as input, the image-based BEV feature generation network in the pre-trained first model extracts features and projects them to generate a 3D bird's-eye view feature of the image modality; Step (4.3). Using the 3D bird's-eye view features of the image modality obtained in step (4.2) as input, the pseudo-point cloud features are generated by the pseudo-point cloud feature generation network; Step (4.4). Using the 3D bird's-eye view features of the image modality obtained in step (4.2) and the pseudo point cloud features obtained in step (4.3) as input, the two are stitched together by the stitching layer in the multimodal feature fusion network and then the adaptive fusion features are generated by the convolutional neural network layer. Step (4.5). Using the adaptive fusion features obtained in step (4.3) as input, the target detection result is generated by the fusion target output head; Step (4.6). Using the real target results of the surround view images collected by the vehicle camera in step (4.2) as labels, perform end-to-end training on the image-based BEV feature generation network, pseudo-point cloud feature generation network, multimodal feature fusion network, and fusion target output head in the pre-trained first model, fine-tune the parameters, and obtain a pure visual target detection model based on pseudo-feature fusion.

10. The pure visual target detection method based on image-pseudo-point cloud feature fusion according to claim 1, characterized in that the image target detection output head, the point cloud target detection output head, and the fused target output head are three independent multilayer perceptrons.

Images