Single-stage three-dimensional target detection method of variational auto-encoder based on label guidance

Abstract

The invention relates to a single-stage three-dimensional target detection method of a variational auto-encoder based on label guidance. The method comprises the following steps: inputting a point cloud; voxelizing the point cloud, and dividing an irregular point cloud space into uniformly stacked voxels; 3D backbone networking processing is carried out on the voxelized point cloud, and multi-scale voxel features are extracted; tag information is introduced as a prior condition, and a variational auto-encoder LG-CVAE is constructed; performing multi-task loss function processing, calculating KL divergence between a feature map generated by LG-CVAE and an input feature map in combination with target perception adversarial loss in a multi-task loss function, and enabling generated potential features to be closer to real data distribution; and outputting the three-dimensional frame. According to the method, the advantages of CVAE, an attention mechanism and antagonism training are combined, more effective feature representation can be learned, the most effective features are selected from numerous feature maps generated by CNN, and the feature extraction capability of LG-CVAE is further improved.

Description

Technical Field

[0001] This patent application belongs to the field of three-dimensional target detection technology, and more specifically, it relates to a single-stage three-dimensional target detection method based on a label-guided variational autoencoder, which is especially used in autonomous driving, robotics and computer vision. Background Art

[0002] 3D object detection has become increasingly important in recent years and is widely used in fields such as autonomous driving, robotics, and computer vision. With the advancement of LiDAR sensor technology, point cloud data has become the main source of information for these tasks, providing rich geometric and spatial information, making it possible to directly predict the 3D bounding box of an object.

[0003] Due to inherent limitations, point clouds are disordered, sparse, and invariant, which poses challenges for object detection in incomplete point clouds. Recently, many single-stage 3D object detection methods based on point clouds have been developed. These methods borrow from the detection paradigm based on 2D images and adopt a multi-scale feature fusion architecture for detection. However, these methods do not take into account the differences between point clouds and images. In 2D images, the apparent size of an object is affected by its distance from the camera because of the presence of perspective information. In contrast, objects in point clouds maintain their actual size, represented by precise coordinates. Due to the extreme sparsity of point clouds and the effects of upsampling or pooling operations, high-level features usually lose many object details, making it difficult for the model to distinguish the boundaries of objects for the following two reasons:

[0004] First, the single-stage 3D object detection method directly detects from the entire point cloud space, which contains a lot of background information, but relatively few object points. The extremely sparse features are not robust enough in predicting objects. The main reason is that after passing through multiple layers of CNN, the boundaries of objects become blurred and the features of neighboring objects are difficult to distinguish.

[0005] Secondly, the nature of the BEV (bird’s eye view) feature map is to compress the height to the ground, which makes it difficult for the network to effectively focus on spatial features. This may lead to inaccurate object edge detection or the model ignoring small but critical features.

[0006] To address these challenges, a 3D object detection method with label-guided attention mechanism is needed. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a single-stage 3D object detection method based on a label-guided variational autoencoder, which uses dataset labels to learn the distribution of object point clouds and extract valuable information from single-scale feature maps. Unlike CNN-based methods that have difficulty maintaining fine-grained spatial details, this method utilizes the inherent structure of dataset labels. By focusing on specific areas indicated by these labels, the network is able to focus on key areas, thereby enhancing the extraction of significant features. In contrast, the label-guided attention mechanism can identify the data distribution within the object and ensure more discriminative feature learning.

[0008] In order to solve the above problems, the technical solution adopted by the present invention is:

[0009] A single-stage three-dimensional object detection method based on a label-guided variational autoencoder comprises the following steps:

[0010] S1, input point cloud;

[0011] S2, voxelize point cloud, divide the irregular point cloud space into evenly stacked voxels, and process the point cloud at the voxel level;

[0012] S3, perform 3D backbone network processing on the voxelized point cloud and extract multi-scale voxel (spatial) features through 3D convolution;

[0013] S4, introduce label information as a priori condition and construct a label-guided conditional variational autoencoder LG-CVAE to learn the feature distribution in the latent space, thereby focusing on effective features and discarding irrelevant information;

[0014] S5. Multi-task loss function processing: In the multi-task loss function, the objective-aware adversarial loss (OAL) is combined. By calculating the KL (Kullback-Leibler) divergence between the feature map generated by LG-CVAE and the input feature map, the potential features generated by the model are closer to the real data distribution.

[0015] S6. The generated potential features are displayed by outputting a three-dimensional box.

[0016] Due to the adoption of the above technical solution, the beneficial effects achieved by the present invention are:

[0017] 1. This paper proposes an attention-guided variational autoencoder (LG-CVAE) combined with adversarial training as a feature extraction technique to enhance point cloud-based single-stage 3D object detection. This novel method combines the advantages of CVAE, attention mechanism and adversarial training to learn more effective feature representations and select the most effective features from the numerous feature maps generated by CNN.

[0018] 2. This paper proposes an innovative loss function that combines CVAE loss with adversarial training to effectively guide the learning process and further enhance the feature extraction capability of LG-CVAE.

[0019] 3. Through a large number of experiments on the widely used KITTI dataset, the present invention proves that AG-VAE has advanced results compared with the existing state-of-the-art methods, demonstrating the effectiveness of the method of the present invention.

[0020] 4. In addition, the results of the present invention are compared with other state-of-the-art 3D object detection methods, highlighting the superiority of the present method.

[0021] 5. Finally, the invention has high value for promotion and application, and this method provides a promising direction for future research in the fields of autonomous driving, robotics, and computer vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a network structure diagram of the present invention.

[0023] Figure 2 Schematic diagram of a multi-layer perceptron of the present invention.

[0024] Figure 3 Enhanced coordinate transformation for the distributed sensing data of the present invention Figure 1 .

[0025] Figure 4 Enhanced coordinate transformation for the distributed sensing data of the present invention Figure 2 .

[0026] Figure 5 This is the distribution-aware data enhancement diagram of the present invention.

[0027] Figure 6 This is a loss curve diagram of the present invention during the model training process.

[0028] Figure 7 It is a visualization diagram of the detection results of the present invention. DETAILED DESCRIPTION

[0029] The present invention is further described in detail below with reference to the embodiments.

[0030] First, let me introduce the current technology, which mainly focuses on CNN-based feature extraction methods and Transformer-based feature extraction methods.

[0031] CNN-based feature extraction methods. A variety of CNN-based feature extraction techniques have been proposed for 3D object detection, such as PointNet, VoxelNet, SECOND, PointPillars, PillarNet, etc. PointNet is a widely used CNN-based feature extraction method that can directly build deep neural networks from point clouds and is suitable for applications such as autonomous driving, augmented reality, and robotics. VoxelNet is one of the pioneering works in this field. It voxelizes point clouds and uses 3D convolution for feature extraction, then compresses 3D features into 2D bird's-eye view (BEV) perspectives, and uses 2D CNN for further feature extraction to generate region proposals. SECOND improves the 3D convolution operation of VoxelNet by introducing 3D sparse convolutions, performing convolution operations only on non-empty voxels, thereby improving the efficiency of 3D convolutions. PointPillars directly compresses point clouds into 2D BEV perspectives through PillarFeature Encoding (PFE), uses 2DCNN for feature extraction, and does not use 3D convolutions, which greatly improves efficiency. PillarNet further enhances pillar encoding and uses CNN to extract multi-scale features to generate region proposals, thereby improving detection accuracy.

[0032] In addition, there are PV-RCNN, AFDetV2, VoxelFPN, PVB-SSD, etc. PV-RCNN uses the PointNet++ module to extract features from the point cloud, and then generates regional proposals for the target object through the region proposal network (RPN). PV-RCNN then further optimizes the proposal and extracts more features through the multi-layer perceptron (MLP) and RoIAlign. In experiments on KITTI and Waymo public datasets, PV-RCNN surpassed other advanced methods in accuracy and efficiency. AFDetV2 introduced self-calibrated convolutions in the feature extraction stage to extract more effective target features and improve detection performance. VoxelFPN obtains multi-scale features of the target object by fusing voxel features of different resolutions through 3D convolution, showing the importance of multi-scale feature extraction in 3D target detection. PVB-SSD combines 3D voxel features and BEV features to generate region proposals.

[0033] These methods essentially rely on CNN feature extraction and improve the performance of 3D object detection by fusing multi-scale or multi-channel features. Although these methods have achieved encouraging results, the robustness of these methods in scenes with limited data or complex object geometry still needs further research and improvement.

[0034] On the other hand, there are Transformer-based feature extraction methods. With the rapid development of Transformer architecture in the fields of natural language processing (NLP) and computer vision (CV), many researchers have begun to explore how to apply Transformer to point cloud-based 3D object detection tasks, such as TANet, VoTr, VoxSeT, RDIoU, CT3D, and Li3DeTr.

[0035] TANet integrates channel attention, point attention, and voxel attention to emphasize key target information, making the model more robust when dealing with noisy points. Since the efficiency of directly using Transformer to process point clouds is low, VoTr introduces sparse voxel modules and submanifold voxel modules to effectively extract features from non-empty voxels. VoxSeT introduces Set-Transformer in the voxel encoding stage to retain more spatial information in the point cloud. RDIoU combines Swin Transformer and CNN to extract local and global features of the target and achieve better detection results. CT3D pioneered the application of Transformer in the two-stage target optimization process, and adopted a self-attention mechanism on the point features in the RoI area to improve the detection accuracy. Li3DeTr uses sparse convolution and multi-scale deformable attention to encode the local and global features of LiDAR data respectively, and promotes target query interaction through multi-head self-attention, thereby improving model performance. In addition, there are some hybrid models that combine Transformer-based feature extraction and other methods. For example, the PointPainting model combines point cloud-based and image-based features, and image features are extracted by a Transformer-based model. The above model PointPainting achieved state-of-the-art results on multiple 3D object detection benchmarks such as KITTI and nuScenes.

[0036] Although CNN-based methods have achieved remarkable success in 3D object detection, they still have some limitations. For example, they have difficulty in effectively capturing complex spatial dependencies and may have limited ability to focus on the most discriminative features. On the other hand, while Transformer-based methods have shown promising results by addressing these limitations, they are potentially computationally expensive and require a lot of memory, especially when processing high-resolution input data. In addition, both CNNs and Transformers tend to generate a large number of feature maps, which makes it challenging to select the most useful features for object detection.

[0037] Therefore, attention must be paid to the work related to variational autoencoders.

[0038] Variational Autoencoder (VAE) is a generative model that learns to encode and decode data by optimizing the lower bound of the likelihood of observed data. It uses variational inference techniques to approximate the true posterior distribution, thereby obtaining a compact and continuous latent space representation. VAE has variants such as VQ-VAE and DC-VAE. For example, VQ-VAE uses discrete latent variables and learns the prior distribution of data through a separate autoregressive model. DC-VAE integrates instance-level discriminative loss and set-level adversarial loss, optimizing reconstruction and sampling at the same time.

[0039] In addition, some research works use VAE to generate images or videos. Yan et al. use the variational autoencoder framework to use decoupled latent variables for image reconstruction and completion. Videogpt uses VQ-VAE and GPT-like models combined with spatiotemporal position encoding for video generation.

[0040] VAE offers the potential to capture more effective features. By introducing VAE into the feature extraction process, one can learn a more efficient and compact representation of the input data while enabling the model to focus on the most discriminative features. This combination can improve performance in 3D object detection tasks because VAE is able to effectively filter and learn the most relevant feature maps generated by CNN and Transformer. In addition, the attention mechanism in VAE can provide additional guidance to help the model better understand and represent the underlying structure of the input data.

[0041] Therefore, a method combining label-guided conditional variational autoencoder (CVAE) with target-aware adversarial loss function is proposed to generate more effective feature maps. The overall steps are as follows:

[0042] Step 1: Input point cloud;

[0043] Step 2: voxelize the point cloud, divide the irregular point cloud space into evenly stacked voxels, and process the point cloud at the voxel level;

[0044] Step 3: Perform 3D backbone network processing on the voxelized point cloud and extract multi-scale voxel features through 3D convolution;

[0045] Step 4: Introduce label information as a priori conditions and construct a label-guided conditional variational autoencoder LG-CVAE to learn the feature distribution in the latent space, thereby focusing on effective features and discarding irrelevant information;

[0046] Step 5: Multi-task loss function processing. In the multi-task loss function, the target perception adversarial loss is combined. By calculating the KL divergence between the feature map generated by LG-CVAE and the input feature map, the potential features generated by LG-CVAE are made closer to the real data distribution.

[0047] Step 6: Display the generated potential features by outputting a three-dimensional box.

[0048] In general, the above method mainly consists of two parts: label-guided CVAE for reconstructing feature maps and objective-aware adversarial loss (OAL) for enhancing the training process. The network structure is as follows: Figure 1 shown.

[0049] exist Figure 1 In the above example, the point cloud is represented as P = {p i}, where p i ={x i ,y i ,z i ,r i} represents each point, including the three-dimensional coordinates x, y, z and the reflectivity feature r. The index of each point in the point cloud is represented by i. The predicted box B R and the real frame B G It can be expressed as {x, y, z, l, w, h, θ}, where x, y, z represent the center coordinates; l, w, h represent the length, width, and height relative to the center of the box, respectively, and θ represents the rotation angle of the three-dimensional box relative to the x-axis.

[0050] Since point clouds are disordered, irregular, and invariant, it is challenging to extract features using traditional convolution operations. Voxelization methods initially divide the irregular point cloud space into evenly stacked voxels and process the point cloud at the voxel level. Subsequently, 3D convolution is used to extract multi-scale spatial features.

[0051] The processing process of voxelized point cloud is:

[0052] The point cloud is first voxelized into R through a quantization process D×W×H ,in and Represents the number of voxels in the x, y, and z directions respectively. x 、v y and v z represents the size of each voxel in three directions. During training, the number of voxels is limited to a maximum of 16000. max 、x min ,y max ,y min 、z max 、z min Respectively represent the actual maximum and minimum coordinates of the point cloud scene in the three directions of x, y, and z axes, in meters, and are used to represent the actual scene size of the point cloud.

[0053] Although in the preprocessing stage according to the range {x min ,y min ,z min ,x max ,y max ,z max}The point cloud has been filtered, and the space still contains a large number of non-uniformly distributed points. In addition, after the point cloud is voxelized, the number of point clouds contained in each voxel grid is uneven. In order to reduce the computational complexity of voxel encoding, the number of points in the point cloud in each voxel is limited. If the number of points exceeds the threshold T, T points are randomly selected and the remaining unselected points are discarded. If the number of points is lower than T, it is filled with zeros. During the voxelization training process, the threshold T = 5, of course, it can also be other values, depending on the specific situation.

[0054] like Figure 2 As shown, the coordinates of a single point in the point cloud (v x,n ,v y,n ,v z,n ), the coordinates of a single point are encoded using a shared multi-layer perceptron (MLP), and then the encoding is mapped to a high-dimensional feature space, where

[0055] {a i,1 ,a i,2 ,...,a i,M} represents the transformed eigenvector.

[0056] Next, the voxelized point cloud is processed by 3D backbone network, such as Figure 2As shown in the figure, four convolution modules, including submanifold convolution layers and 3D sparse convolution layers, are used to learn multi-scale voxel features. Four downsampling modules are used to obtain multi-scale dense features, and each downsampling module consists of several 3D sparse convolution layers and submanifold convolution layers (this is not an innovative point and will not be repeated here). The submanifold convolution layer maintains the sparsity of the point cloud by converting the voxel feature channel, while the 3D sparse convolution layer only performs convolution calculations on non-empty voxels, which improves the convolution efficiency.

[0057] Each downsampling module first performs a sparse convolution layer to obtain features of different scales, and then performs channel conversion through several sub-manifold convolution layers to enhance the nonlinear expression of multi-scale dense features.

[0058] The output features of each downsampling module can be expressed as in Represents the multi-scale voxel features obtained by n-fold downsampling. Usually contains more detailed information, while The feature of each scale is obtained by downsampling the previous feature by 2 times, so that the 3D backbone network can obtain features at different levels and thus make a more comprehensive representation of the point cloud. The bird’s-eye view is compressed as the map view feature and fed into the 2D backbone network to generate region proposals.

[0059] The Label-guided CVAE (label-guided conditional variational autoencoder) proposed in this invention improves the feature learning ability of traditional CVAE by introducing prior label information.

[0060] The LG-CVAE of the present invention consists of an encoder and a decoder. The encoder is responsible for learning the compact potential representation of the input point cloud data, and the decoder is used for decoding the point cloud data; the encoder adopts a hierarchical structure and contains multiple hidden layers, each of which captures a different level of abstraction in the point cloud data. The attention mechanism is integrated into the encoder, enabling it to focus on the most relevant part of the input data and suppress irrelevant information, expressed as:

[0061]

[0062] Where μ represents the mean of the learned point cloud data distribution, and var represents the variance of the learned point cloud data distribution. Encoder is the encoder (marked in purple below). Given the input and condition c n , the encoder maps the input to the latent space. n The dimensions and Consistently, a set of 0 and 1 is used to represent the background and foreground. At the same time, this information is fused by addition to enhance the foreground information.

[0063] During the training phase, downsampling of the point cloud ensures that multi-scale information is captured and improves processing efficiency. However, as the point cloud is gradually downsampled, spatial information is lost. This loss makes it difficult to effectively learn from low-resolution feature maps. To compensate for the loss of spatial information, we use the true labels to enhance the feature information of the foreground objects, thereby alleviating the information loss problem.

[0064] The encoder consists of a series of CBR blocks (convolution, batch normalization, ReLU). The kernel size of the first convolution layer is 3×3, and the subsequent convolution layers are kept of the same size to optimize the balance between feature extraction accuracy and computational efficiency. The stride of all convolutional layers is unified to 2, and each convolution operation effectively halves the spatial dimension of the feature map. This systematic dimensionality reduction method helps to generate a compact representation of the input data, which is crucial for efficient feature encoding in latent space models. The downsampled features are flattened and passed into two fully connected layers to generate the latent space feature distribution of the data.

[0065] The decoder reconstructs the input point cloud data from the latent representation, using the multi-scale features learned by the encoder. The decoder also adopts a hierarchical structure with the same layers as the encoder. During the decoding process, the attention mechanism is also introduced, enabling the decoder to focus on the most relevant parts of the latent representation, thereby improving the reconstruction quality and generation ability.

[0066] To further improve the performance of LG-CVAE, the researchers introduced an adversarial loss function called target-aware adversarial loss. The target-aware adversarial loss function encourages the model to generate more realistic and diverse samples by pitting the generator (decoder) against the discriminator that tries to distinguish between real data and generated data. The adversarial loss function helps the model learn a more accurate input data distribution, thereby improving feature extraction and generation capabilities.

[0067] Next, we introduce the target-aware adversarial loss.

[0068] The target-aware adversarial loss plays a crucial role in assisting the training of the LG-CVAE network. The main purpose of this adversarial loss is to calculate the Kullback-Leibler (KL) divergence between the feature map generated by LG-CVAE and the input feature map. By minimizing this KL divergence, the adversarial loss ensures that the generated feature map is closer to the true distribution of the input data.

[0069] The adversarial loss can be expressed as:

[0070] L adv =KL(P||Q)

[0071] Among them, P represents the probability distribution of the input feature map, and Q represents the probability distribution of the feature map generated by LG-CVAE. KL divergence is used to measure the difference between the two distributions. Minimizing this divergence can make the generated feature map closer to the input feature map.

[0072] During the training process, the LG-CVAE network and the discriminator are jointly optimized. The task of the discriminator is to distinguish between the real input feature map and the generated feature map, while LG-CVAE is trained by generating feature maps that can deceive the discriminator. This adversarial loss process improves LG-CVAE's ability to learn the underlying distribution of input data and enhances its generation ability.

[0073] Introducing adversarial loss into the LG-CVAE network not only improves the quality of generated feature maps, but also helps the model better capture the complex structures and relationships in point cloud data, which in turn can improve feature extraction and achieve better performance in various 3D object detection and scene understanding tasks.

[0074] In this LG-CVAE model, three loss functions are combined to effectively train the model: classification loss, regression loss, and adversarial loss. Multi-task loss facilitates joint learning and optimization, thereby improving the overall performance of the model.

[0075] The multi-task loss function can be expressed as:

[0076] L total =λ 1 *L cls +λ 2 *L reg +λ 3 *L adv

[0077] Among them, L cls is the classification loss, which is used to measure the difference between the predicted object category and the true label; L reg is the regression loss, quantifying the difference between the predicted bounding box coordinates and the true coordinates; L adv λ is the adversarial loss introduced earlier, which is used to calculate the KL divergence between the input feature map and the feature map generated by LG-CVAE. 1 , 2 and λ 3 are the weight coefficients of each loss term to the total loss.

[0078] In order to improve the performance of detection algorithms, it is necessary to simulate various situations that may occur in real scenes through data augmentation. Existing augmentation methods usually perform basic geometric transformations such as scaling, translation or rotation, but these operations will destroy the geometric distribution of point clouds.

[0079] In order to solve the above problems, the present invention also proposes a distribution-aware data enhancement method before voxelizing the point cloud, which can better simulate the sparsity of the distant target point cloud instead of just focusing on the number of points.

[0080] Based on the inherent characteristics of the LiDAR system, as the distance increases, the spacing between points in the point cloud will also increase. Based on this characteristic, the present invention designs a distribution-aware data enhancement method to simulate the sparsity of distant targets in a way that is more in line with realistic scenarios.

[0081] Specifically, for the internal point P in The close distance truth value G near First, P in,i The coordinates are converted to the spherical coordinate system, such as Figure 3 As shown, the spherical coordinate system is represented by P s,i (θ, φ, r), where θ, φ and r represent polar angle, azimuth angle and radius respectively. For a clearer explanation, the internal points are projected onto the θ-φ plane, as Figure 4 As shown. It can be seen that the distribution of internal points along the θ axis is orderly. Then along the θ axis, s Sort and get the index Ind of the point i Next, start sampling from the beginning in interval order to obtain Ind sample , where sample<i. After this step, the sampling point P[index sample ].

[0082] The sampling point P[index sample ] is converted back to Cartesian coordinates, such as Figure 5 As shown, the sampled ground truth (GT) is added to the scene for data supplementation during the training phase. By adding these sparser GTs during the training process, the method of the present invention can better simulate the sparsity changes of distant target point clouds, thereby improving the performance of the object detection algorithm.

[0083] After the method of the present invention is designed, it needs to be experimentally verified. First, the experimental environment is set up. The hardware and software configurations used in this experiment are as follows: the CPU is Intel (R) Xeon (R) Gold5218R (10 cores), the GPU is RTX3090, equipped with 24G video memory; the operating system is Ubuntu 18.04; the Python version is 3.8; the deep learning framework is PyTorch 1.8.1; the CUDA version is 11.1, and the cuDNN version is 8.0.

[0084] Then the data set is processed.

[0085] The KITTI dataset contains 7481 training images / point clouds and 7518 test images / point clouds, covering multiple categories such as vehicles, pedestrians, bicycles, etc. According to the number of points in the object, the degree of occlusion and the degree of truncation, the difficulty level is divided into simple, medium and difficult. The training set contains 3712 point clouds and the validation set contains 3769 point clouds. The model is trained on the training set and evaluated on the validation set and the test set respectively. For fair comparison, the present invention adopts the average precision (AP) indicator officially recommended by KITTI, and evaluates it on 11 and 40 recall points respectively.

[0086] In terms of experimental parameters, for the KITTI dataset, the detection range of the point cloud is set to [0,70.4] meters on the x-axis, [-40.0,40.0] meters on the y-axis, and [-3.0,1.0] meters on the z-axis. The voxel size is set to (0.05,0.05,0.05) meters. For the nuScenes dataset, the detection range of the point cloud is set to [-51.2,51.2] meters on the x-axis, [-51.2,51.2] meters on the y-axis, and [-5.0,3.0] meters on the z-axis. The voxel size is set to (0.1,0.1,0.2) meters.

[0087] Then comes training and reasoning. The present invention uses the Adam_onecycle optimizer, with the learning rate set to 0.001, the split factor to 10, the momentum range to 0.95 to 0.85, and the weight decay to 0.01.

[0088] In order to avoid overfitting, this paper adopts four commonly used data enhancement strategies: 1) real value sampling; 2) global scaling, the scaling factor is randomly selected between [0.95, 1.05], and the global rotation angle around the z-axis is randomly selected. 3) Rotate the real value around the z axis Randomly rotate the image by an angle to simulate turning; 4) randomly flip along the x-axis.

[0089] During inference, the non-maximum suppression (NMS) threshold is set to 0.7 for the vehicle class and 0.5 for the bicycle. The prediction boxes with the top 160 classification scores are selected as the input for the second stage of refinement. In the post-processing stage, the prediction boxes with classification scores lower than 0.55 are removed, and the redundant prediction boxes are filtered out using the NMS threshold of 0.1. Inference is performed on a single RTX 3090 GPU with a batch size of 1.

[0090] The experimental results are shown in Table 1 below.

[0091] Table 1 Detection accuracy of 11 models for 3 types of targets on the KITTI dataset (%)

[0092]

[0093] The evaluation criteria and experimental settings can be seen from Table 2, which shows the comparison of the detection accuracy of the proposed model and the existing mainstream 3D target detection model on the KITTI dataset, covering three categories of targets: cars, pedestrians, and bicycles, each category including three difficulty levels: simple, medium, and difficult. This method follows the evaluation criteria of KITTI, keeps the calculation method of IoU threshold and mean average precision (mAP) consistent with other models in the table, sets the IoU threshold of the car category to 0.7, the IoU threshold of the pedestrian and bicycle categories to 0.5, and mAP is calculated based on 40 recall points.

[0094] In terms of result analysis, Table 2 shows the classification according to the single-stage and two-stage detection models, and lists the experimental results of the model of the present invention. The present invention uses a unified model to train the three types of targets, and does not train the categories independently. As a single-stage model, the present invention focuses on comparing with other single-stage models. From the results, it can be seen that the performance of the model of the present invention in the single-stage model is significantly improved, especially when detecting difficult-level targets. For car targets at simple, medium and difficult levels, the detection accuracy of the model of the present invention is 88.81%, 78.32% and 76.15% respectively; the pedestrian category is 54.71%, 46.29% and 44.72% respectively; the bicycle category is 78.24%, 65.28% and 58.12% respectively. In the difficult level detection, the model of the present invention performs well compared with other mainstream single-stage models, proving that the label-guided conditional self-encoder proposed in this case and the target perception adversarial loss effectively enhance the modeling ability of point cloud data by introducing label information, and improve the accuracy of feature representation.

[0095] Figure 6 The loss curve during model training is shown. Figure 6 It can be seen that the loss of this method is high at the beginning, but then it drops rapidly and gradually stabilizes with the increase of training iterations. The overall trend shows that the model training is stable and effective, and no abnormalities occur.

[0096] In terms of visual analysis, the researchers visualized the detection results of the model, such as Figure 7 As shown in the visualization results, it can be seen that the model can accurately detect targets in different point cloud scenarios, even sparse objects in the distance can be effectively detected, and the detection effect is also good for occluded or overlapping objects.

[0097] To further verify the effectiveness of the LG-CVAE and OAL modules proposed in this paper, an ablation experiment was designed. Taking the SECOND model as the baseline, the LG-CVAE and OAL modules were introduced into the baseline model respectively to evaluate the contribution of each module to the model performance. The experimental results are shown in Table 2.

[0098] The results show that both LG-CVAE and OAL modules significantly improve the detection performance. After adding the LG-CVAE module, the detection accuracy of the car category in the simple, medium and difficult difficulty levels increased by about 2.78, 4.13 and 8.16 percentage points respectively; the accuracy of the simple, medium and difficult samples of pedestrian detection increased by about 2.39, 1.96 and 3.46 percentage points respectively; the accuracy of bicycle detection in the three difficulty levels increased by 4.94, 8.38 and 9.42 percentage points respectively. After further introducing the OAL module, the detection accuracy of the car category in the three difficulty levels increased by 2.68, 0.36 and 1.52 percentage points respectively; the accuracy of the simple, medium and difficult samples of the pedestrian category increased by about 1.04, 1.63 and 2.53 percentage points respectively; the detection accuracy of the bicycle category increased by 1.21, 2.52 and 1.11 percentage points respectively.

[0099] Table 2 Ablation experiment detection accuracy %

[0100]

[0101] The LG-CVAE of the present invention significantly improves the performance of detecting distant sparse objects, especially at difficult difficulty levels, the detection effect of bicycles and pedestrians is better than that of the case where the OAL module is introduced alone. By applying the LG-CVAE and OAL modules to the SECOND model at the same time, the detection performance of the three categories at different difficulties is further improved, proving that the module proposed by the present invention can effectively enhance the overall performance of 3D object detection.

[0102] This proposal comprehensively explains the related work on 3D object detection, variational autoencoders, and attention mechanisms, and then introduces in detail the architecture, training process, and loss function including adversarial training of the LG-CVAE proposed in this method.

[0103] The proposed label-guided VAE attention mechanism uses a variational autoencoder (VAE) combined with adversarial training as a feature extraction technique to enhance point cloud-based single-stage 3D object detection. This method combines the advantages of VAE, attention mechanism, and adversarial training to learn more effective feature representations. Experimental results show that this method not only outperforms the existing state-of-the-art, but also significantly improves the detection performance when integrated into the 3D object detection architecture.

Claims

1. A single-stage 3D object detection method based on label-guided variational autoencoder, characterized in that The steps include: S1, input point cloud; S2, voxelize point cloud, divide the irregular point cloud space into evenly stacked voxels, and process the point cloud at the voxel level; S3, perform 3D backbone network processing on the voxelized point cloud and extract multi-scale voxel features through 3D convolution; S4, introduce label information as a priori condition and construct a label-guided conditional variational autoencoder LG-CVAE to learn the feature distribution in the latent space, thereby focusing on effective features and discarding irrelevant information; S5, multi-task loss function processing, combining the target perception adversarial loss in the multi-task loss function, and making the potential features generated by LG-CVAE approach the real data distribution by calculating the KL divergence between the feature map generated by LG-CVAE and the input feature map; S6. Output a three-dimensional box to display the generated potential features.

2. The single-stage three-dimensional object detection method based on label-guided variational autoencoder according to claim 1, characterized in that: In S1, the point cloud is represented as P = {p i }, where p i ={x i ,y i ,z i ,r i } represents each point, including three-dimensional coordinates x, y, z and reflectivity feature r; The index of each point in the point cloud is represented by i; the predicted box B R and the real frame B G It is expressed as {x, y, z, l, w, h, θ}, where x, y, z represent the center coordinates; l, w, h represent the length, width, and height relative to the center of the box, respectively; and θ represents the rotation angle of the three-dimensional box relative to the x-axis.

3. The single-stage three-dimensional object detection method based on label-guided variational autoencoder according to claim 2, characterized in that: In S2, the processing of voxelized point cloud is: The point cloud is first voxelized into R through a quantization process D×W×H ,in Represents the number of voxels in the x, y, and z directions respectively; v x 、v y and v z Indicates the size of each voxel in three directions; x max 、x min ,y max ,y min 、z max 、z min Respectively represent the actual maximum and minimum coordinates of the point cloud scene in the three directions of x, y, and z axes, in meters, and are used to represent the actual scene size of the point cloud.

4. The single-stage three-dimensional object detection method based on label-guided variational autoencoder according to claim 3, characterized in that: The number of points in each voxel is limited. If the number of points exceeds the threshold T, T points are randomly selected and the remaining unselected points are discarded; if the number of points is lower than the threshold T, it is filled with zeros; During voxelized training, the number of voxels is limited to a maximum of 16,000.

5. The single-stage three-dimensional object detection method based on label-guided variational autoencoder according to claim 4, characterized in that: In S3, the voxelized point cloud is processed by 3D backbone network, which specifically refers to: The coordinates of a single point in the point cloud (v x,n ,v y,n ,v z,n ), use the shared multi-layer perceptron MLP to encode, and then map the encoding to the high-dimensional feature space to form the transformed feature vector {a i,1 ,a i,2 ,...,a i,M }.

6. The single-stage three-dimensional object detection method based on label-guided variational autoencoder according to claim 5, characterized in that: The multi-layer perceptron MLP uses four downsampling modules to obtain multi-scale dense features. Each downsampling module includes multiple coordinated 3D sparse convolutional layers and sub-manifold convolutional layers. The sub-manifold convolutional layer maintains the sparsity of the point cloud by converting the voxel feature channel. The 3D sparse convolutional layer only performs convolution calculations on non-empty voxels, thereby improving the convolution efficiency. Each downsampling module first passes through a 3D sparse convolution layer to obtain features of different scales, and then performs channel conversion through multiple sub-manifold convolution layers to enhance the nonlinear expression of multi-scale dense features; The output features of each downsampling module can be expressed as in Represents the multi-scale voxel features obtained by n-fold downsampling; Contains detailed information, and The feature of each scale is obtained by downsampling the previous feature by 2 times, so that the 3D backbone network can obtain features at different levels and then comprehensively represent the point cloud. Finally, The bird’s-eye view is compressed as the map view feature and fed into the 2D backbone network to generate region proposals.

7. The single-stage three-dimensional object detection method based on label-guided variational autoencoder according to claim 6, characterized in that: In S4, LG-CVAE includes an encoder and a decoder. The encoder is responsible for learning the compact potential representation of the input point cloud data, and the decoder is used for decoding the point cloud data. The encoder adopts a hierarchical structure, which contains multiple hidden layers. Each hidden layer captures a different level of abstraction in the point cloud data. The attention mechanism is integrated into the encoder, enabling it to focus on the most relevant part of the input data and suppress irrelevant information. The expression is: μ,var=Encoder(f v n× ,c n ) Among them, μ represents the mean of the learned point cloud data distribution, var represents the variance of the learned point cloud data distribution; Encoder is the encoder; given the input f v n× and condition c n , the input is mapped to the latent space through the encoder; c n The dimension and f v n× Consistently, a set of 0 and 1 is used to represent the background and foreground.

8. The single-stage three-dimensional object detection method based on label-guided variational autoencoder according to claim 7, characterized in that: In S5, the target-aware adversarial loss is expressed as: L adv =KL(P||Q) Among them, P represents the probability distribution of the input feature map, Q represents the probability distribution of the feature map generated by LG-CVAE; KL divergence is used to measure the difference between the two distributions. Minimizing the KL divergence can make the generated feature map close to the input feature map.

9. The single-stage three-dimensional object detection method based on label-guided variational autoencoder according to claim 8, characterized in that: The multi-task loss function is expressed as: L total =λ1*L cls +λ2*L reg +λ3*L adv Among them, L cls is the classification loss, which is used to measure the difference between the predicted object category and the true label; L reg is the regression loss, quantifying the difference between the predicted bounding box coordinates and the true coordinates; L adv is the target-aware adversarial loss, which is used to calculate the KL divergence between the input feature map and the feature map generated by LG-CVAE; λ1, λ2, and λ3 are the weight coefficients of each loss term to the total loss.

10. A single-stage three-dimensional object detection method based on label-guided variational autoencoder according to any one of claims 1 to 9, characterized in that: Before voxelizing the point cloud, distribution-aware data enhancement is performed. The distribution-aware data enhancement is as follows: For the internal point P in The close distance truth value G near First, P in,i The coordinates of are converted into spherical coordinates, which are represented by P s,i (θ, φ, r), where θ, φ, and r represent the polar angle, azimuth, and radius, respectively; Secondly, project the internal points onto the θ-φ plane, so that we can see that the distribution of the internal points along the θ axis is orderly; Then along the θ axis, P s Sort and get the index Ind of the point i ; Next, start sampling from the beginning in interval order to get Ind sample , where sample<i, after this step, the sampling point P[index sample ]; The sampling point P[index sample ] is converted back to the Cartesian coordinate system, and the sampled true values ​​are added to the scene for data supplementation during the training phase.

Images