A method and system for 3D vehicle target detection that integrates voxel and image features

Abstract

This invention provides a method and system for 3D vehicle target detection that integrates voxel and image features. The method includes: acquiring an initial LiDAR point cloud and an initial scene image of a sample detection scene; performing data augmentation on the LiDAR point cloud and the initial scene image using a cross-modal ground truth bounding box data augmentation method to obtain an enhanced LiDAR point cloud and an enhanced scene image; performing color difference balancing on the enhanced scene image to obtain a balanced scene image; performing image transformation on the balanced scene image to obtain a transformed scene image; inputting the enhanced LiDAR point cloud and the transformed scene image into an initial target detection model to obtain a trained 3D vehicle target detection model; and inputting the target detection scene's test data into the 3D vehicle target detection model to obtain the detection result. This invention enables accurate and rapid identification of 3D vehicle targets.

Description

Technical Field

[0001] This invention relates to the field of vehicle target detection, and in particular to a three-dimensional vehicle target detection method and system that integrates voxel and image features. Background Technology

[0002] Voxelization divides the entire point cloud scene into several regular voxels, transforming disordered point cloud data into ordered voxel data. It is one of the important methods used in point cloud-based 3D vehicle target detection. Compared to massive, disordered point cloud data, voxelized point cloud data has a regular structure and smaller memory footprint. Furthermore, image data, being a regular data type, is easily fused with voxelized point cloud data. Existing voxel-based 3D target detection methods follow a similar process. Figure 1 As shown, the algorithm is divided into a "single-stage" target detection algorithm and a "two-stage" target detection algorithm.

[0003] A single-stage object detection algorithm based on voxelization first encodes the overall unordered point cloud scene into voxels. It then extracts voxel features using 3D sparse convolution and 3D submanifold sparse convolution, encoding the voxelized point cloud into a high-dimensional semantic space. A bird's-eye view projection method is used to project the output features onto a bird's-eye view, effectively avoiding target occlusion in the 3D scene and reducing the 3D features to 2D features, significantly improving the inference speed of subsequent algorithms. A 2D convolution is performed on the bird's-eye view feature map, and finally, the object detection head directly outputs the target bounding box position and classification confidence. Compared to two-stage object detection algorithms, the single-stage algorithm has higher inference speed and lower memory consumption, but its detection accuracy is slightly lower.

[0004] The two-stage object detection algorithm based on voxelization follows the same feature extraction process as the single-stage algorithm. After the object detection head outputs the bounding box position and classification confidence, the two-stage algorithm samples the output bounding box to obtain the region of interest (RoI). Further pooling is performed on the RoI, and finally, the generated new features are used to correct the previously generated bounding box position and category confidence, producing the final detection result. Compared with single-stage object detection, the two-stage algorithm has higher detection accuracy, but its execution speed is slightly slower.

[0005] Therefore, how to perform accurate and rapid target detection on point cloud-based 3D vehicles has become an urgent problem to be solved. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, the purpose of this invention is to provide a three-dimensional vehicle target detection method and system that integrates voxel and image features, enabling accurate and rapid target detection.

[0007] To achieve the above objectives, the present invention provides the following solution:

[0008] A method for detecting 3D vehicle targets by fusing voxel and image features, comprising:

[0009] Acquire the initial LiDAR point cloud and initial scene image of the sample detection scene;

[0010] The lidar point cloud and the initial scene image are augmented using a cross-modal ground truth bounding box data augmentation method to obtain an augmented lidar point cloud and an augmented scene image.

[0011] The enhanced scene image is then subjected to color difference balancing to obtain a balanced scene image;

[0012] The balanced scene image is transformed to obtain a transformed scene image; the transformed scene image is in the HSV color space.

[0013] The enhanced LiDAR point cloud and the transformed scene image are input into an initial target detection model to obtain a trained 3D vehicle target detection model. The initial target detection model includes a multimodal feature fusion module, a multimodal Voxel-RoI pooling module, and a prediction module connected in sequence. The multimodal feature fusion module is used to perform multimodal feature fusion based on the enhanced LiDAR point cloud and the transformed scene image to obtain fused features. The multimodal Voxel-RoI pooling module is used to perform RoI pooling operation based on the fused features to obtain the final RoI pooled output features. The prediction module is used to perform classification confidence and location regression prediction based on the output features.

[0014] The target detection scene data is input into the three-dimensional vehicle target detection model to obtain the detection results.

[0015] Preferably, the enhanced scene image is subjected to color difference balancing to obtain a balanced scene image, including:

[0016] Calculate the average pixel value of the enhanced scene image in the three RGB channels;

[0017] Calculate the balance coefficient for each color channel based on the pixel mean;

[0018] The color difference is balanced by using the balance coefficient to obtain the balanced scene image.

[0019] Preferably, the formula for calculating the average pixel value is: Where D represents the pixel mean, This represents the pixel mean value of the enhanced scene image in the red channel. This represents the pixel mean value of the enhanced scene image in the green channel. This represents the average pixel value of the enhanced scene image in the blue channel.

[0020] Preferably, the formula for calculating the balance coefficient is:

[0021]

[0022] Among them, K R K represents the balance coefficient of the red channel. G K represents the balance coefficient of the green channel. B R' represents the balance coefficient of the blue channel, G' represents the pixel value of the red channel after balance, B' represents the pixel value of the green channel after balance, R represents the pixel value of the enhanced scene image in the red channel, G represents the pixel value of the enhanced scene image in the green channel, and B represents the pixel value of the enhanced scene image in the blue channel.

[0023] Preferably, the balanced scene image is transformed to obtain a transformed scene image, including:

[0024] Normalize each color channel of the balanced scene image to obtain the normalized result;

[0025] Using formula Select the extreme values ​​from the normalization results; where C max C represents the maximum value in the normalization result. min This represents the minimum value in the normalization result;

[0026] A color space conversion formula is constructed based on the normalization results and the extreme values ​​of the normalization results;

[0027] The balanced scene image is converted into a transformed scene image using the color space conversion formula.

[0028] Preferably, the color space conversion formula is:

[0029]

[0030] V = C max

[0031] Where H represents the hue component, S represents the saturation component, and V represents the lightness component.

[0032] Preferably, the processing steps of the multimodal feature fusion module include:

[0033] The voxel features in the enhanced lidar point cloud are extracted using the Voxel-RCNN 3D feature extraction network;

[0034] Extract image features from the transformed scene image;

[0035] The voxel features and the image features are fused using the first feature fusion submodule to generate initial features;

[0036] The SE channel attention mechanism is used to enhance the attention of foreground features in the initial features and reduce the attention of background points;

[0037] The second feature fusion submodule is used to fuse image features with the generated initial features to obtain the fused features.

[0038] Preferably, the processing steps of the multimodal Voxel-RoI pooling module include:

[0039] Divide the candidate boxes into a preset number of sub-grids, and calculate the pseudo-point coordinates in the voxel coordinate system based on the sub-grids.

[0040] Based on the pseudo-point coordinates, the image features in the fused features are sampled and voxelized to generate image voxel features;

[0041] The voxel features of the image, the voxel features of the point cloud with a downsampling factor of 4, and the voxel features of the point cloud with a downsampling factor of 8 are obtained respectively, and the voxel features are aggregated to obtain three aggregated features.

[0042] The aggregated features are fused to obtain the final RoI pooled output features.

[0043] A 3D vehicle target detection system that integrates voxel and image features includes:

[0044] The data acquisition unit is used to acquire the initial lidar point cloud and initial scene image of the sample detection scene;

[0045] A cross-modal enhancement unit is used to perform data enhancement on the lidar point cloud and the initial scene image using a cross-modal ground truth box data enhancement method to obtain an enhanced lidar point cloud and an enhanced scene image.

[0046] A color difference balancing unit is used to perform color difference balancing on the enhanced scene image to obtain a balanced scene image.

[0047] An image conversion unit is used to convert the balanced scene image to obtain a converted scene image; the converted scene image is in the HSV color space.

[0048] The model building unit is used to input the enhanced LiDAR point cloud and the transformed scene image into an initial target detection model to obtain a trained 3D vehicle target detection model. The initial target detection model includes a multimodal feature fusion module, a multimodal Voxel-RoI pooling module, and a prediction module connected in sequence. The multimodal feature fusion module is used to perform multimodal feature fusion based on the enhanced LiDAR point cloud and the transformed scene image to obtain fused features. The multimodal Voxel-RoI pooling module is used to perform RoI pooling operation based on the fused features to obtain the final RoI pooled output features. The prediction module is used to perform classification confidence and location regression prediction based on the output features.

[0049] The detection unit is used to input the target detection scene data into the three-dimensional vehicle target detection model to obtain the detection result.

[0050] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0051] This invention provides a method and system for fusion of voxel and image features in 3D vehicle target detection. The method includes: acquiring an initial LiDAR point cloud and an initial scene image of a sample detection scene; performing data augmentation on the LiDAR point cloud and the initial scene image using a cross-modal ground truth bounding box data augmentation method to obtain an enhanced LiDAR point cloud and an enhanced scene image; performing color difference balancing on the enhanced scene image to obtain a balanced scene image; performing image transformation on the balanced scene image to obtain a transformed scene image; the transformed scene image is in the HSV color space; and inputting the enhanced LiDAR point cloud and the transformed scene image into an initial target detection model to obtain a trained 3D target. A vehicle target detection model is provided. The initial target detection model includes a multimodal feature fusion module, a multimodal Voxel-RoI pooling module, and a prediction module connected in sequence. The multimodal feature fusion module is used to fuse multimodal features based on the enhanced LiDAR point cloud and the transformed scene image to obtain fused features. The multimodal Voxel-RoI pooling module is used to perform RoI pooling operations based on the fused features to obtain the final RoI pooled output features. The prediction module is used to perform classification confidence and position regression prediction based on the output features. The test data of the target detection scene is input into the three-dimensional vehicle target detection model to obtain the detection results. This invention can achieve accurate and rapid identification of three-dimensional vehicle targets. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram of the process of a voxelization-based point cloud target detection method provided in the prior art.

[0054] Figure 2 A flowchart of the method provided in an embodiment of the present invention;

[0055] Figure 3 This is a flowchart of the real frame sampling method provided in an embodiment of the present invention;

[0056] Figure 4 This is a flowchart of multimodal feature fusion provided in an embodiment of the present invention;

[0057] Figure 5 This is a structural diagram of the feature fusion module provided in an embodiment of the present invention;

[0058] Figure 6 This is a basic principle diagram of RoI pooling provided in an embodiment of the present invention;

[0059] Figure 7 A diagram illustrating the multimodal Voxel-RoI pooling method provided in this embodiment of the invention;

[0060] Figure 8 Voxel sampling diagrams provided in embodiments of the present invention;

[0061] Figure 9 A flowchart of the target refinement regression provided in this embodiment of the invention;

[0062] Figure 10 This is a flowchart of a 3D vehicle target detection algorithm that integrates point cloud and image data according to an embodiment of the present invention.

[0063] Figure 11 This is a schematic diagram illustrating the change of the relevant loss value of the VoxelImage algorithm over time steps, provided in an embodiment of the present invention.

[0064] Figure 12 A schematic diagram illustrating the change of loss values ​​of each branch with time step during the training process of the multimodal Voxel-RoI pooling and bounding box refinement stages provided in this embodiment of the invention.

[0065] Figure 13The algorithm-related loss value for point cloud and image fusion provided in this embodiment of the invention varies with time step. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] The purpose of this invention is to provide a method and system for detecting three-dimensional vehicle targets by fusing voxel and image features, which can achieve accurate and rapid identification of three-dimensional vehicle targets.

[0068] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] Figure 2 The method flowchart provided in the embodiments of the present invention is as follows: Figure 2 As shown, this invention provides a three-dimensional vehicle target detection method that integrates voxel and image features, including:

[0070] Step 100: Obtain the initial LiDAR point cloud and initial scene image of the sample detection scene;

[0071] Step 200: Use cross-modal ground truth bounding box data augmentation method to perform data augmentation on the lidar point cloud and the initial scene image to obtain the augmented lidar point cloud and the augmented scene image;

[0072] Step 300: Perform color difference balancing on the enhanced scene image to obtain a balanced scene image;

[0073] Step 400: Perform image transformation on the balanced scene image to obtain a transformed scene image; the transformed scene image is in the HSV color space;

[0074] Step 500: Input the enhanced LiDAR point cloud and the transformed scene image into the initial target detection model to obtain a trained 3D vehicle target detection model. The initial target detection model includes a multimodal feature fusion module, a multimodal Voxel-RoI pooling module, and a prediction module connected in sequence. The multimodal feature fusion module is used to perform multimodal feature fusion based on the enhanced LiDAR point cloud and the transformed scene image to obtain fused features. The multimodal Voxel-RoI pooling module is used to perform RoI pooling operation based on the fused features to obtain the final RoI pooled output features. The prediction module is used to perform classification confidence and location regression prediction based on the output features.

[0075] Step 600: Input the target detection scene data into the 3D vehicle target detection model to obtain the detection results.

[0076] Specifically, step 100 in this embodiment is cross-modal data augmentation. Data augmentation refers to expanding data during the model training phase through random translation, rotation, scaling, etc., to generate various training data and improve the robustness of the model. In 3D object detection, point cloud data augmentation can avoid overfitting and enhance the model's generalization ability and robustness. 3D object detection algorithms based on single LiDAR point clouds widely employ ground truth (GT) sampling data augmentation methods to enhance the point cloud. This method copies the point clouds of ground truth bounding boxes from other scenes into the training scene, strengthening the diversity of the point cloud scene and improving the model's robustness and object detection accuracy. The ground truth sampling method flow is as follows: Figure 3 As shown, the red box represents the ground truth bounding box of the vehicle target in other scenarios, and the orange point cloud represents the noise point cloud added to the target scene after sampling the ground truth bounding box.

[0077] In multimodal 3D target detection, directly applying point cloud bounding box sampling can disrupt the alignment consistency between LiDAR point clouds and images. To address this issue, a cross-modal bounding box sampling method for point clouds and images is proposed. This method copies point cloud bounding boxes from other scenes into the training scene and simultaneously copies the corresponding image bounding boxes of the sampled point cloud bounding boxes into the corresponding training images in order from farthest to nearest, thus ensuring data consistency between LiDAR point clouds and images. However, since images are two-dimensional data with limited field of view, the above bounding box sampling method can lead to the occlusion of many distant objects, affecting the detection accuracy of distant training targets. Near targets are larger than distant targets; therefore, when near targets from other image scenes are copied into the training image scene, it is highly likely that some distant targets in the training image scene will be completely occluded. To solve this problem, this embodiment designs a cross-modal bounding box data augmentation method to balance near and distant targets during the bounding box sampling process, reducing the probability of distant objects being occluded. This method primarily removes sampled ground truth bounding boxes of distant targets in a completely occluded training scene, while retaining sampled ground truth bounding boxes with 3D IoU equal to 0 in the 3D point cloud space and 2D IoU less than 0.5 in the 2D image. The pseudocode for the cross-modal ground truth bounding box data augmentation method is shown in Table 1.

[0078] Table 1. Cross-modal ground truth frame data augmentation process

[0079]

[0080]

[0081]

[0082] Preferably, the enhanced scene image is subjected to color difference balancing to obtain a balanced scene image, including:

[0083] Calculate the average pixel value of the enhanced scene image in the three RGB channels;

[0084] Calculate the balance coefficient for each color channel based on the pixel mean;

[0085] The color difference is balanced by using the balance coefficient to obtain the balanced scene image.

[0086] Preferably, the formula for calculating the average pixel value is: Where D represents the pixel mean, This represents the pixel mean value of the enhanced scene image in the red channel. This represents the pixel mean value of the enhanced scene image in the green channel. This represents the average pixel value of the enhanced scene image in the blue channel.

[0087] Preferably, the formula for calculating the balance coefficient is:

[0088]

[0089] Among them, K R K represents the balance coefficient of the red channel. G K represents the balance coefficient of the green channel. B R' represents the balance coefficient of the blue channel, G' represents the pixel value of the red channel after balance, B' represents the pixel value of the green channel after balance, R represents the pixel value of the enhanced scene image in the red channel, G represents the pixel value of the enhanced scene image in the green channel, and B represents the pixel value of the enhanced scene image in the blue channel.

[0090] Specifically, the color difference balancing process in step 300 of this embodiment improves color consistency, minimizing the impact of different lighting conditions and scene backgrounds on the image. By correcting color deviations in the image, it ensures that the resulting balanced scene image is more color-consistent, helping the model accurately identify target objects. This process enables the model to maintain stability and accuracy in recognizing diverse real-world scenarios. In applications such as autonomous driving and intelligent transportation, changes in ambient lighting are common. By implementing color difference balancing during data preprocessing, the model's sensitivity to lighting changes is reduced, thereby improving its robustness in dynamic and complex environments. Furthermore, color difference balancing helps enhance the visibility of features in the image, making them more apparent in subsequent processing and feature extraction, thus improving the feature fusion effect in the subsequent neural network.

[0091] Preferably, the balanced scene image is transformed to obtain a transformed scene image, including:

[0092] Normalize each color channel of the balanced scene image to obtain the normalized result;

[0093] Using formula Select the extreme values ​​from the normalization results; where C max C represents the maximum value in the normalization result. min This represents the minimum value in the normalization result;

[0094] A color space conversion formula is constructed based on the normalization results and the extreme values ​​of the normalization results;

[0095] The balanced scene image is converted into a transformed scene image using the color space conversion formula.

[0096] Preferably, the color space conversion formula is:

[0097]

[0098] V = C max

[0099] Where H represents the hue component, S represents the saturation component, and V represents the lightness component.

[0100] Specifically, in step 400 of this embodiment, the image is converted to the HSV color space, which offers greater flexibility in feature extraction: the HSV (hue, saturation, brightness) color space is more consistent with human visual perception than the RGB color space, enabling more effective extraction and representation of color features in the image. Processing in the HSV space makes the color features of vehicle targets more prominent, thereby improving detection accuracy. Hue and saturation in the HSV color space can, to some extent, eliminate the influence of lighting changes, making target recognition more consistent under different lighting conditions. This lighting adaptability enhances the model's detection capability, making it better able to cope with different environmental conditions. By converting the image to the HSV color space, this embodiment can better combine the color features in LiDAR point cloud data and image data, providing a richer and more meaningful feature expression for the multimodal feature fusion module, which helps the fusion deep learning model understand and process multidimensional information.

[0101] Furthermore, step 300 in this embodiment significantly improves the color consistency of the image and the robustness of the model by implementing color difference balancing, while step 400 enhances the flexibility of feature extraction and illumination adaptability by converting the image to the HSV color space. These two innovations respectively improve data preprocessing and feature representation, thereby enhancing the overall performance of 3D vehicle target detection.

[0102] Preferably, the processing steps of the multimodal feature fusion module include:

[0103] The voxel features in the enhanced lidar point cloud are extracted using the Voxel-RCNN 3D feature extraction network;

[0104] Extract image features from the transformed scene image;

[0105] The voxel features and the image features are fused using the first feature fusion submodule to generate initial features;

[0106] The SE channel attention mechanism is used to enhance the attention of foreground features in the initial features and reduce the attention of background points;

[0107] The second feature fusion submodule is used to fuse image features with the generated initial features to obtain the fused features.

[0108] Specifically, LiDAR point clouds exhibit a "dense near, sparse far" characteristic. In 3D space, distant vehicle targets consist of only a small portion of the point cloud. Consequently, the voxels of such vehicle targets contain only a few foreground points, making it difficult to extract sufficient semantic features during voxel feature extraction, thus resulting in low detection accuracy. Image data, on the other hand, compresses a 3D scene into a 2D space, unaffected by depth and distance, allowing for better extraction of distant vehicle target features during feature extraction. Based on this analysis, this embodiment proposes a multimodal feature fusion method to fuse image features and voxel features, enriching the semantic features of sparse point cloud vehicle targets, thereby improving the detection accuracy to some extent. The entire multimodal feature fusion process is as follows: Figure 4 As shown.

[0109] Furthermore, the Voxel-RCNN 3D feature extraction network employs 3D sparse convolution to extract voxel features. Each 3D sparse convolutional layer contains one spatial sparse convolution (SparseConv3d) and two sub-manifold sparse convolutions (SubMConv3d). The spatial sparse convolution is used to increase the receptive field of the convolution kernel to extract voxel contextual information features, while the sub-manifold sparse convolutions are used to ensure that the position of the sparse voxel features remains unchanged. Considering that both point cloud voxel features and image features consume a large amount of memory, multimodal feature fusion is only performed in the first layer of the backbone network. Figure 4 As can be seen, image features and voxel features are first processed by feature fusion module 1 (using the Concat method to concatenate features) to generate fused features. An SE channel attention mechanism is used to enhance the focus on foreground features of the target in the fused features and reduce the focus on background points. Finally, feature fusion module 2 (using the Sum method to add features) fuses the image features with the generated fused features. The feature fusion module designed in this embodiment is as follows: Figure 5 As shown.

[0110] Optionally, during the training phase, the algorithm employs data augmentation methods such as random flipping, scaling, and rotation on the original point cloud data. This leads to a misalignment between the stored 3D index after voxelization and the corresponding image. To ensure that the point cloud data can be correctly mapped back to the original image, it is necessary to first transform the voxel 3D index to the original point cloud coordinate system. The point cloud data restoration method in the feature fusion module restores the point cloud data by storing parameters such as the flip axis, scaling factor, and rotation angle during the point cloud data augmentation process. The correction transformation method first transforms the 3D voxel data according to the voxelization parameters. Convert to raw point cloud dataset Then, the scaling factor, rotation angle, and other parameters stored in the data augmentation stage are used to perform an inverse transformation on the point cloud data to align it with the corresponding image features. Based on the calibration parameters and intrinsic / extrinsic parameter matrices provided by the training dataset, the point cloud is projected onto the image feature map. Finally, bilinear interpolation is used to sample the image features to obtain the sampled image features. The sampled image features are voxelized and fused with voxel features via Concat (or Sum). However, simple concatenation or addition operations cannot fully utilize the image features. To better fuse image features and voxel features and enhance feature extraction from sparse targets, submanifold sparse convolution is performed on the fused features to obtain deep fused features. The above correction transformation process is shown below.

[0111] F P =B(P(T) -1 (G(V))),F img_feature )

[0112] Among them, F P For the sampled image features, B(·) represents bilinear interpolation, P(·) represents the point cloud projection process, and T... -1 (·) represents the inverse data augmentation transformation process, G(·) represents the process of converting 3D voxel data into raw point cloud data, V represents 3D voxel data, and F ingfeature Represents image features.

[0113] Preferably, the processing steps of the multimodal Voxel-RoI pooling module include:

[0114] Divide the candidate boxes into a preset number of sub-grids, and calculate the pseudo-point coordinates in the voxel coordinate system based on the sub-grids.

[0115] Based on the pseudo-point coordinates, the image features in the fused features are sampled and voxelized to generate image voxel features;

[0116] The voxel features of the image, the voxel features of the point cloud with a downsampling factor of 4, and the voxel features of the point cloud with a downsampling factor of 8 are obtained respectively, and the voxel features are aggregated to obtain three aggregated features.

[0117] The aggregated features are fused to obtain the final RoI pooled output features.

[0118] Specifically, the multimodal Voxel-RoI pooling method in this embodiment is as follows:

[0119] RoI pooling, a common method in two-stage network training, aims to further refine the prediction of target location and size by combining the point cloud inside the candidate box with the point cloud or voxel features of the surrounding area. The basic idea of ​​RoI pooling is as follows: First, the target boxes obtained in the first stage of the algorithm are sampled to obtain candidate boxes. These candidate boxes are then divided into equidistant sub-grids. Using the center point of each sub-grid as the origin, the sampling strategy in PointNet++ is used to fuse the features of the point cloud (or voxels) within its radius. Figure 6 As shown. The red dots represent the center points of the equidistant sub-grids where the candidate boxes are divided. They only indicate the location and are a virtual point cloud rather than actual point cloud data. The blue dots represent the original point cloud, the yellow dots are the sampled points obtained by the PointNet++ sampling strategy, and the green arrows indicate that the features of the sampled points are aggregated onto the grid center points.

[0120] This embodiment, based on the RoI pooling strategy of Voxel-RCNN, further integrates image features and proposes a multimodal Voxel-RoI pooling method, such as... Figure 7 As shown, the main components include the generation of candidate box grid center points and pseudo points, the association of pseudo points with image features, and the fusion of image features and voxel features at the grid center points.

[0121] Traditional fusion methods only project the center points of the grid divided by the candidate bounding boxes onto the image features and sample them. Due to the resolution imbalance between the point cloud and the image, this method results in a low image feature sampling rate, affecting the overall detection performance. To better sample image features, this embodiment, while equally dividing the candidate bounding box grid center points (querypoints), enlarges the candidate bounding box size and uses a different scale to divide the new candidate bounding boxes, generating pseudopoints. The pseudopoints are mapped back to the image features for sampling. Based on the 3D coordinates of the pseudopoints and the sampled image features, feature voxelization is performed to obtain image voxel features. The image voxel features are aggregated onto the grid center points, and simultaneously, two voxel features with downsampling ratios of 4 and 8 from the 3D feature extraction network are aggregated to generate fused features. Finally, the target bounding boxes are refined.

[0122] like Figure 7 As shown, the RoI pooling process used in this embodiment is as follows:

[0123] Step 1: Generate candidate bounding boxes. First, divide the candidate bounding boxes (RoIs) into 6×6×6 sub-grids, and calculate the position coordinates G of the center point of each grid in the point cloud coordinate system. xyz The calculation formulas are shown in the first two formulas below; simultaneously, according to the third formula below, the candidate bounding box (RoI) is enlarged, the RoI is divided into sections with a scale of 16×8×22, pseudopoints are generated, and the position coordinates of the pseudopoints in the point cloud coordinate system are calculated. The calculation formulas are shown in the fourth and fifth formulas below:

[0124]

[0125] Among them, G temp and These represent the coordinates of the center point and the pseudo-point in the RoI coordinate system, respectively. `center_idx` refers to the index coordinates of the grid center point, and `sizee`... RoI and size grid These represent the current RoI size and the size of each grid cell, respectively. RoI_(l,w,h) These represent the length, width, and height of the current RoI, respectively. and These represent the size of the enlarged RoI and the division scale of the enlarged RoI, respectively. θ is the rotation angle between the RoI coordinate system and the point cloud coordinate system, and centerr is the rotation angle between the RoI coordinate system and the point cloud coordinate system. RoI These are the coordinates of the center point of the RoI.

[0126] Since voxel features are obtained by voxelizing and encoding the original point cloud and then processing it with a 3D sparse convolutional neural network, grid points and pseudo-points are difficult to align directly with voxel features. Therefore, it is necessary to calculate the corresponding voxel coordinates based on the coordinates of the grid points and pseudo-points, as shown in the following formula.

[0127]

[0128] Among them, G voxel and Here are the voxel coordinates corresponding to the grid points and pseudo-points, R represents the left boundary of the point cloud range, d represents the downsampling factor (2, 4, 8 respectively), and size... voxel and This refers to the voxel dimensions corresponding to the grid points and pseudo-points, which are (0.05, 0.05, 0.1) and (0.2, 0.2, 0.1), respectively.

[0129] Step 2: Sample image features and perform voxelization. After obtaining the pseudo-point coordinates in the voxel coordinate system, voxelization parameters are used to convert them to pseudo-point coordinates in the original point cloud coordinate system. Based on the correction parameters provided by the training dataset, the pseudo-points are mapped to the image coordinate system. Pseudo-points outside the image range are filtered out. Image features are sampled using bilinear interpolation. The sampled image features are used as voxel features, and the voxel coordinates of the pseudo-points within the image range are used as indices for voxelization to generate image voxel features.

[0130] Step 3: Voxel Feature Aggregation. Using the central voxel in the voxel features as the center, set multiple sampling radii, and obtain the K non-empty voxels within the neighborhood of each sampling radius, such as... Figure 8As shown (example sampling radius is 2), the PointNet++ network is used to further extract voxel features from the sampled non-empty voxels, and feature aggregation is performed through max pooling, aggregating the features to the center point of the grid. Red voxels represent the center voxel, and blue voxels represent non-empty voxels in the neighborhood of the center voxel with a radius of 2.

[0131] Step 4: Multimodal Feature Aggregation. Image voxel features, point cloud voxel features with a downsampling factor of 4, and point cloud voxel features with a downsampling factor of 8 are obtained separately. Multiple central voxel neighborhood radii are set for each of these three types of voxel features. Step 3, voxel feature aggregation, is repeated, outputting three aggregated features. The three aggregated features are then fused together as the output features for the final RoI pooling.

[0132] Furthermore, the target box refinement steps in this embodiment are as follows:

[0133] The target bounding box classification and localization results generated in the first stage may deviate somewhat from the true bounding boxes. Fine-tuning regression of the target bounding boxes is then used to further adjust the output results. The final output features obtained by the multimodal Voxel-RoI pooling method are first pooled to a fixed scale of N×41472 through a fully connected layer. Classification fully connected layers and regression fully connected layers are then used to perform classification confidence and location regression predictions, respectively. Figure 9 As shown.

[0134] The classification fully connected layer calculates the class confidence of each RoI, classifying each RoI as either a vehicle target object or a background object. The regression fully connected layer calculates the positional offset between each RoI and the true target, determining the predicted target position based on the offset and the RoI position. To prevent overfitting, the RoI pooling features (N×13824) aggregated from image voxel features are separately classified and regressed for target classification, forming an auxiliary branch to calculate the pooling loss. The above branch network structure is shown in Table 2.

[0135] Table 2 Target Classification and Regression Network Structure

[0136]

[0137] Furthermore, the loss function in this embodiment is designed as follows:

[0138] The loss function of the VoxelImage algorithm proposed in this embodiment is shown in the following formula:

[0139]

[0140] in The region network proposal loss represents the first stage of the algorithm and includes two parts: target classification loss and regression loss. This is the loss for the second stage of the algorithm, which involves multimodal Voxel-RoI pooling and target box refinement. It is the total loss of the algorithm.

[0141] (1) RPN loss

[0142] Considering the imbalance between positive and negative samples, this embodiment uses the Focal Loss function to calculate the target classification loss and the SmoothL1 function to calculate the regression loss. The RPN loss is shown in the following formula:

[0143]

[0144] Where, N fg This refers to the number of foreground needle points, p i and δ i The outputs of the target classification network and the location regression network are c, respectively. i and t i This indicates the target's true category and true location. This indicates that only the regression loss of the foreground needle points is calculated. The target classification loss is calculated using the Focal Loss function. The regression loss is calculated using the SmoothL1 function.

[0145] (2) RCNN loss

[0146] Multimodal networks are highly likely to be dominated by one modality during training. In the algorithm proposed in this embodiment, since the 3D information in the generated image voxel features is far less than that in the LiDAR point cloud voxel features, the point cloud voxel features often play a dominant role in the multimodal Voxel-RoI pooling process, which may lead to overfitting. To solve the above problem, this embodiment introduces an auxiliary branch to calculate the RCNN loss based on the main branch RCNN loss calculation, in order to avoid single-modality dominance and alleviate overfitting. The RCNN loss is shown in the following formula.

[0147]

[0148] in The total loss for multimodal Voxel-RoI pooling and bounding box refinement is... This represents the RCNN loss calculated in the main branch. This represents the RCNN loss calculated by the auxiliary branch.

[0149] The main branch uses an existing detector head loss, and the input is the image features and voxel features aggregated from the center points of the RoI grid, as shown in the following formula:

[0150]

[0151] Where N R The number of target candidate boxes participating in multimodal Voxel-RoI pooling. This represents the target classification loss calculated by FocalLoss. The position regression loss is calculated using SmoothL1. and c represents the target category prediction result and target location prediction result output by the multimodal Voxel-RoI pooling and target box refinement network, respectively. i and t i These represent the target's true category and the target's true location, respectively. This refers to calculating the regression loss only for the foreground target. λ1 is the classification loss weight, set to 1.0, and λ2 is the regression loss weight, set to 1.0.

[0152] The auxiliary branch is similar to the main branch, except that it takes the image features aggregated by the center points of the RoI grid as input and only predicts the location regression results, as shown in the following formula.

[0153]

[0154] The parameters in this formula have the same meaning as above, except that the classification loss weight λ1 is set to 0 and the regression loss weight λ2 is set to 0.5.

[0155] Furthermore, this embodiment uses the KITTI3DObject dataset to train a 3D vehicle target detection model that fuses voxel and image features. The vehicle target detection performance of this algorithm is qualitatively and quantitatively analyzed. Ablation experiments are conducted to evaluate the effectiveness of related modules. The performance differences between the feature fusion algorithm proposed in this embodiment and other methods are compared, verifying the performance differences of fusing image features at different voxel layers and the effectiveness of cross-modal data augmentation methods. To comprehensively analyze the performance differences between the 3D target detection algorithm that fuses point clouds and images and the traditional single-LiDAR point cloud 3D target detection algorithm, this embodiment, based on the fusion of voxel features and image features, introduces a pixel-level fusion module and a decision-level fusion module, respectively, and designs a system as follows: Figure 10 The 3D vehicle target detection method is shown. This embodiment also uses the KITTI3DObject dataset to train the model. By introducing ablation experiments of pixel-level fusion, decision-level fusion, and overall fusion separately, the effectiveness of point cloud and image fusion is further verified. By introducing pixel-level fusion, feature-level fusion, and decision-level fusion methods separately into the Voxel-RCNN benchmark algorithm, the performance differences between the three different fusion methods are analyzed.

[0156] For the 3D vehicle target detection algorithm that integrates voxel and image features, this embodiment uses the PyTorch and OpenPCDet frameworks to implement the basic algorithm flow and trains it on a single RTX 3090 GPU. The Adam model parameter update optimizer is used, with a maximum learning rate of 0.008, a weight decay rate of 0.01, a batch size of 2 samples per training iteration, a momentum of 0.9, and a total of 80 iterations. The target detection loss during the algorithm's training process changes with time steps as follows: Figure 11 As shown. By Figure 11 It can be seen that the total loss, RPN loss and RCNN loss of the model reach convergence at time step 140001, at which point the model has iterated to the 77th time.

[0157] Figure 12 This graph shows the change of loss values ​​for each branch during the multimodal Voxel-RoI pooling and bounding box refinement stages over time steps. The blue, red, gray, orange, and light blue curves represent the total loss, classification loss, localization loss, localization loss of the auxiliary branch, and classification loss of the auxiliary branch, respectively, for each stage. The classification loss of the auxiliary branch is set to 0 due to its weight being zero. At time step 140001, the losses of each branch converge, indicating that the model has iterated for 77 epochs.

[0158] For the 3D vehicle target detection algorithm that fuses point clouds and images, this embodiment also uses PyTorch and OpenPCDet as the baseline framework to implement the basic algorithm modules. A single RTX 3090 GPU is used to train the model, and the Adam optimizer is used to update the model parameters. The maximum learning rate is 0.01, the weight decay rate is 0.01, and the batch size is 2 samples, with a total of 80 iterations. The target detection loss of this algorithm changes as follows: Figure 13 As shown. By Figure 13 It can be seen that at time step 50001 (i.e., 80 iterations of the model), the total loss, RPN loss, and RCNN loss reach a convergent state.

[0159] The evaluation indicators used in this embodiment mainly include AP. 3D _R11、AP 3D _R40 is used as the average detection accuracy indicator. 3D IoU and BEV IoU are used to determine whether the predicted bounding box hits the real bounding box. The IoU threshold for the vehicle target is set to 0.7 for all three difficulty levels.

[0160] This embodiment evaluates the vehicle target detection performance of the proposed feature fusion algorithm from both quantitative and qualitative perspectives. Specifically, it includes: comparing the average target detection accuracy of the proposed algorithm with the benchmark algorithm Voxel-RCNN, the 3D vehicle target detection algorithm based on adaptive feature fusion (Algorithm 2), and the 3D vehicle target detection algorithm that fuses image texture features and prior information (Algorithm 3); comparing the average target detection accuracy of the proposed algorithm with other 3D target detection algorithms; comparing the running time of the multimodal Voxel-RoI pooling method with the benchmark RoI pooling method; and visualizing some scenes from the KITTI 3DObject dataset to qualitatively analyze the effectiveness of the proposed algorithm in vehicle target detection.

[0161] Specifically, the quantitative analysis of the vehicle target detection performance of the feature fusion algorithm proposed in this embodiment is as follows:

[0162] This experiment uses two different methods, R11 (11 interpolation points) and R40 (40 interpolation points), to calculate the average detection accuracy AP3D. 3D The performance of the feature fusion algorithm proposed in this embodiment for 3D vehicle target detection was evaluated, and the results are shown in Table 3. With an IoU threshold of 0.7 and using R11 to calculate the average detection accuracy, the feature fusion algorithm proposed in this embodiment achieved average detection accuracies (APs) of 90.02%, 86.45%, and 85.51% for easy, medium, and hard difficulty levels, respectively. 3D When calculating the average detection accuracy using the R40 method (R11), the average detection accuracy (AP) for three-dimensional targets was 93.09%, 88.19%, and 85.85% respectively at the three difficulty levels mentioned above. 3D _R40. The above experimental results effectively demonstrate that the algorithm proposed in this embodiment can accurately detect vehicle targets in 3D scenes. Compared with Algorithm 2 and the benchmark algorithm Voxel-RCNN, the method in this embodiment enhances the semantic feature richness of sparse vehicle target point clouds by fusing image features, thereby improving the average detection accuracy of the algorithm on difficult samples. For example, the AP on difficult samples... 3D The R11 scores were improved by 6.58% and 8.15%, respectively. Compared to pixel-level fusion and decision-level fusion 3D object detection algorithms, the algorithm in this embodiment has slightly lower detection accuracy on easy-difficulty samples, but achieves better detection performance on medium- and difficult-difficulty samples, for example, with higher AP on medium- and difficult-difficulty samples. 3D R11 increased by 6.06% and 7.08%, respectively.

[0163] Table 3. Quantitative evaluation results of the feature fusion algorithm proposed in this embodiment on the KITTI 3D Object validation set.

[0164]

[0165] Table 4 compares the vehicle target detection accuracy of the feature fusion algorithm proposed in this embodiment with other typical 3D target detection algorithms, using the R11 (11 recall locations) method for measurement. The detection accuracy results of other methods are from their respective literature.

[0166] Table 4 compares the accuracy of the method in this embodiment with other methods in vehicle target detection on the KITTI 3D Object validation set.

[0167]

[0168] As shown in Table 4, compared with single-LiDAR point cloud-based 3D target detection algorithms such as PointRCNN, SECOND, 3DSSD, and PV-RCNN, the algorithm in this embodiment achieves superior 3D target detection results at all three difficulty levels (easy, medium, and hard). This demonstrates that fusing image features significantly improves the accuracy of 3D target detection, and that the multi-modal sensor 3D target detection method effectively compensates for the shortcomings of a single sensor. Compared with pixel-level fusion-based 3D target detection algorithms such as F-ConvNet and PointPainting, this embodiment... The proposed method demonstrates significantly superior performance, proving that feature-level fusion methods outperform pixel-level fusion methods in detection accuracy. Compared with 3D target detection algorithms based on feature-level fusion, such as PI-RCNN, 3D-CVF, EPNet, PVFusion, PA3DNet, MLF-DET, and LoGoNet, the proposed method achieves competitive results, showing significant improvement on difficult samples. The average detection accuracy of AP3D is improved by 7.76%, 12.4%, 5.37%, 5.35%, 7.63%, 7.62%, and 1.52%, respectively.

[0169] Furthermore, the execution time comparison of the multimodal Voxel-RoI pooling proposed in this embodiment is as follows:

[0170] This embodiment compares the execution time required during the model inference stage between the proposed multimodal Voxel-RoI pooling method and the RoI pooling method of the benchmark algorithm Voxel-RCNN. Table 5 shows the comparison results of the average execution time of RoI pooling for the two different methods on the KITTI3DObject validation set. The comparative experimental results show that the proposed method inevitably increases the time overhead of the RoI pooling stage due to the need to generate pseudo-points and fuse image features during the RoI pooling stage. To reduce the impact of the high time overhead caused by introducing image features, this embodiment reduces the average number of RoI pooling operations from the initial value of 100 set by the benchmark method Voxel-RCNN to 40, a reduction of 2.5 times compared to Voxel-RCNN. While maintaining the detection accuracy of the algorithm, the average RoI pooling time only increases by 2.4ms, indicating that the proposed multimodal Voxel-RoI pooling method is a lightweight method that improves the detection performance of the algorithm with minimal time overhead.

[0171] Table 5 compares the execution time of the method in this embodiment with that of the Voxel-RCNN method in RoI pooling.

[0172]

[0173] Corresponding to the above method, this embodiment also provides a three-dimensional vehicle target detection system that fuses voxel and image features, including:

[0174] The data acquisition unit is used to acquire the initial lidar point cloud and initial scene image of the sample detection scene;

[0175] A cross-modal enhancement unit is used to perform data enhancement on the lidar point cloud and the initial scene image using a cross-modal ground truth box data enhancement method to obtain an enhanced lidar point cloud and an enhanced scene image.

[0176] A color difference balancing unit is used to perform color difference balancing on the enhanced scene image to obtain a balanced scene image.

[0177] An image conversion unit is used to convert the balanced scene image to obtain a converted scene image; the converted scene image is in the HSV color space.

[0178] The model building unit is used to input the enhanced LiDAR point cloud and the transformed scene image into an initial target detection model to obtain a trained 3D vehicle target detection model. The initial target detection model includes a multimodal feature fusion module, a multimodal Voxel-RoI pooling module, and a prediction module connected in sequence. The multimodal feature fusion module is used to perform multimodal feature fusion based on the enhanced LiDAR point cloud and the transformed scene image to obtain fused features. The multimodal Voxel-RoI pooling module is used to perform RoI pooling operation based on the fused features to obtain the final RoI pooled output features. The prediction module is used to perform classification confidence and location regression prediction based on the output features.

[0179] The detection unit is used to input the target detection scene data into the three-dimensional vehicle target detection model to obtain the detection result.

[0180] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0181] This embodiment uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention; at the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. In summary, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for detecting three-dimensional vehicle targets by fusing voxel and image features, characterized in that, include: Acquire the initial LiDAR point cloud and initial scene image of the sample detection scene; The LiDAR point cloud and the initial scene image are augmented using a cross-modal ground truth bounding box data augmentation method to obtain an augmented LiDAR point cloud and an augmented scene image. The cross-modal ground truth bounding box data augmentation method removes sampled ground truth bounding boxes of distant targets in a completely occluded training scene, while retaining sampled ground truth bounding boxes with a 3D IoU of 0 in the 3D point cloud space and a 2D IoU of less than 0.5 in the 2D image. The enhanced scene image is then subjected to color difference balancing to obtain a balanced scene image; The balanced scene image is transformed to obtain a transformed scene image; the transformed scene image is in the HSV color space. The enhanced LiDAR point cloud and the transformed scene image are input into an initial target detection model to obtain a trained 3D vehicle target detection model. The initial target detection model includes a multimodal feature fusion module, a multimodal Voxel-RoI pooling module, and a prediction module connected in sequence. The multimodal feature fusion module is used to perform multimodal feature fusion based on the enhanced LiDAR point cloud and the transformed scene image to obtain fused features. The multimodal Voxel-RoI pooling module is used to perform RoI pooling operation based on the fused features to obtain the final RoI pooled output features. The prediction module is used to perform classification confidence and location regression prediction based on the output features. The target detection scene data is input into the three-dimensional vehicle target detection model to obtain the detection results.

2. The three-dimensional vehicle target detection method based on the fusion of voxels and image features according to claim 1, characterized in that, Perform color difference balancing on the enhanced scene image to obtain a balanced scene image, including: Calculate the average pixel value of the enhanced scene image in the three RGB channels; Calculate the balance coefficient for each color channel based on the pixel mean; The color difference is balanced by using the balance coefficient to obtain the balanced scene image.

3. The three-dimensional vehicle target detection method based on the fusion of voxel and image features according to claim 2, characterized in that, The formula for calculating the average pixel value is: Where D represents the pixel mean, This represents the pixel mean value of the enhanced scene image in the red channel. This represents the pixel mean value of the enhanced scene image in the green channel. This represents the average pixel value of the enhanced scene image in the blue channel.

4. The three-dimensional vehicle target detection method based on the fusion of voxel and image features according to claim 3, characterized in that, The formula for calculating the balance coefficient is: Among them, K R K represents the balance coefficient of the red channel. G K represents the balance coefficient of the green channel. B R' represents the balance coefficient of the blue channel, G' represents the pixel value of the red channel after balance, B' represents the pixel value of the green channel after balance, R represents the pixel value of the enhanced scene image in the red channel, G represents the pixel value of the enhanced scene image in the green channel, and B represents the pixel value of the enhanced scene image in the blue channel.

5. The three-dimensional vehicle target detection method based on the fusion of voxels and image features according to claim 4, characterized in that, The balanced scene image is transformed to obtain a transformed scene image, including: Normalize each color channel of the balanced scene image to obtain the normalized result; Using formula Select the extreme values ​​from the normalization results; where C max C represents the maximum value in the normalization result. min This represents the minimum value in the normalization result; A color space conversion formula is constructed based on the normalization results and the extreme values ​​of the normalization results; The balanced scene image is converted into a transformed scene image using the color space conversion formula.

6. The three-dimensional vehicle target detection method based on the fusion of voxel and image features according to claim 5, characterized in that, The color space conversion formula is: V=C max Where H represents the hue component, S represents the saturation component, and V represents the lightness component.

7. The three-dimensional vehicle target detection method based on the fusion of voxel and image features according to claim 1, characterized in that, The processing steps of the multimodal feature fusion module include: The voxel features in the enhanced lidar point cloud are extracted using the Voxel-RCNN 3D feature extraction network; Extract image features from the transformed scene image; The voxel features and the image features are fused using the first feature fusion submodule to generate initial features; The SE channel attention mechanism is used to enhance the attention of foreground features in the initial features and reduce the attention of background points; The second feature fusion submodule is used to fuse image features with the generated initial features to obtain the fused features.

8. The three-dimensional vehicle target detection method based on the fusion of voxel and image features according to claim 1, characterized in that, The processing steps of the multimodal Voxel-RoI pooling module include: Divide the candidate boxes into a preset number of sub-grids, and calculate the pseudo-point coordinates in the voxel coordinate system based on the sub-grids. Based on the pseudo-point coordinates, the image features in the fused features are sampled and voxelized to generate image voxel features; The voxel features of the image, the voxel features of the point cloud with a downsampling factor of 4, and the voxel features of the point cloud with a downsampling factor of 8 are obtained respectively, and the voxel features are aggregated to obtain three aggregated features. The aggregated features are fused to obtain the final RoI pooled output features.

9. A three-dimensional vehicle target detection system that integrates voxel and image features, characterized in that, include: The data acquisition unit is used to acquire the initial lidar point cloud and initial scene image of the sample detection scene; A cross-modal augmentation unit is used to perform data augmentation on the LiDAR point cloud and the initial scene image using a cross-modal ground truth box data augmentation method to obtain an augmented LiDAR point cloud and an augmented scene image. The cross-modal ground truth box data augmentation method removes sampled ground truth boxes of distant targets in a completely occluded training scene, while retaining sampled ground truth boxes with 3D IoU equal to 0 in the 3D point cloud space and 2D IoU less than 0.5 in the 2D image. A color difference balancing unit is used to perform color difference balancing on the enhanced scene image to obtain a balanced scene image. An image conversion unit is used to convert the balanced scene image to obtain a converted scene image; the converted scene image is in the HSV color space. The model building unit is used to input the enhanced LiDAR point cloud and the transformed scene image into an initial target detection model to obtain a trained 3D vehicle target detection model. The initial target detection model includes a multimodal feature fusion module, a multimodal Voxel-RoI pooling module, and a prediction module connected in sequence. The multimodal feature fusion module is used to perform multimodal feature fusion based on the enhanced LiDAR point cloud and the transformed scene image to obtain fused features. The multimodal Voxel-RoI pooling module is used to perform RoI pooling operation based on the fused features to obtain the final RoI pooled output features. The prediction module is used to perform classification confidence and location regression prediction based on the output features; The detection unit is used to input the target detection scene data into the three-dimensional vehicle target detection model to obtain the detection result.

Images