Three-dimensional vehicle target detection method and system fusing voxel and image features

Abstract

The invention provides a voxel and image feature fused three-dimensional vehicle target detection method and system. The method comprises the steps of obtaining an initial laser radar point cloud and an initial scene image of a sample detection scene; performing data enhancement on the laser radar point cloud and the initial scene image by using a cross-modal real frame data enhancement method to obtain an enhanced laser radar point cloud and an enhanced scene image; performing chromatic aberration balance on the enhanced scene image to obtain a balanced scene image; performing image conversion on the balanced scene image to obtain a converted scene image; inputting the enhanced laser radar point cloud and the converted scene image into the initial target detection model to obtain a trained three-dimensional vehicle target detection model; and inputting to-be-detected data of the target detection scene into the three-dimensional vehicle target detection model to obtain a detection result. According to the invention, accurate and rapid identification of the three-dimensional vehicle target can be realized.

Description

Technical Field

[0001] The present invention relates to the field of vehicle target detection, and in particular to a three-dimensional vehicle target detection method and system integrating voxel and image features. Background Art

[0002] "Voxelization" divides the entire point cloud scene into several regular voxels, converting disordered point cloud data into ordered voxel data. It is one of the important means of point cloud-based 3D vehicle target detection. Compared with huge and disordered point cloud data, the voxelized point cloud data has a regular structure and small memory usage. At the same time, image data, as a regular data, is easy to merge with the voxelized point cloud data. The existing voxel-based 3D target detection method process is as follows: Figure 1 As shown, the algorithm is divided into a "single-stage" target detection algorithm and a "two-stage" target detection algorithm.

[0003] The voxelization-based single-stage target detection algorithm first performs voxel encoding on the overall disordered point cloud scene, uses three-dimensional sparse convolution and three-dimensional submanifold sparse convolution to extract voxel features, and encodes the voxelized point cloud into a high-dimensional semantic space. The output features are projected to a bird's-eye view using a bird's-eye view projection method, which effectively avoids the problem of target occlusion in the three-dimensional space scene. At the same time, the three-dimensional features are reduced to two-dimensional features, which greatly improves the reasoning speed of subsequent algorithms. Two-dimensional convolution is performed on the feature map after bird's-eye view, and finally the target detection head is used to directly output the target box position and classification confidence. Compared with the two-stage target detection algorithm, the single-stage target detection algorithm has a higher reasoning speed and lower memory consumption, but is slightly insufficient in detection accuracy.

[0004] The feature extraction process of the voxel-based two-stage target detection algorithm is consistent with that of the single-stage algorithm. After the target detection head outputs the target box position and classification confidence, the two-stage target detection algorithm samples the output target box to obtain the region of interest (RoI), performs further pooling operations on the RoI, and finally uses the generated new features to correct the previously generated target box position and category confidence to produce the final detection result. Compared with the single-stage target detection, the two-stage target detection algorithm has higher detection accuracy, but is slightly insufficient in execution speed.

[0005] Based on this, how to accurately and quickly detect three-dimensional vehicles based on point clouds has become an urgent problem to be solved. Summary of the invention

[0006] In order to overcome the deficiencies of the prior art, an object of the present invention is to provide a three-dimensional vehicle target detection method and system that integrates voxel and image features, which can perform accurate and rapid target detection.

[0007] To achieve the above object, the present invention provides the following solutions:

[0008] A three-dimensional vehicle target detection method integrating voxel and image features, comprising:

[0009] Obtaining an initial lidar point cloud and an initial scene image of a sample detection scene;

[0010] Performing data enhancement on the laser radar point cloud and the initial scene image using a cross-modal true box data enhancement method to obtain an enhanced laser radar point cloud and an enhanced scene image;

[0011] Performing color difference balance on the enhanced scene image to obtain a balanced scene image;

[0012] Performing image conversion on the balanced scene image to obtain a converted scene image; the converted scene image is in the HSV color space;

[0013] The enhanced laser radar point cloud and the converted scene image are input into the initial target detection model to obtain a trained three-dimensional vehicle target detection model; the initial target detection model includes a multimodal feature fusion module, a multimodal Voxel-RoI pooling module and a prediction module connected in sequence; the multimodal feature fusion module is used to perform multimodal feature fusion according to the enhanced laser radar point cloud and the converted scene image to obtain fusion features; the multimodal Voxel-RoI pooling module is used to perform RoI pooling operation according to the fusion features to obtain the final RoI pooling output features; the prediction module is used to perform classification confidence and position regression prediction according to the output features;

[0014] The data to be tested of the target detection scene is input into the three-dimensional vehicle target detection model to obtain the detection result.

[0015] Preferably, performing color difference balancing on the enhanced scene image to obtain a balanced scene image comprises:

[0016] Calculate the pixel mean of the enhanced scene image in the three RGB channels;

[0017] Calculating a balance coefficient for each color channel according to the pixel mean;

[0018] The color difference of the enhanced scene image is balanced by using the balance coefficient to obtain the balanced scene image.

[0019] Preferably, the pixel mean calculation formula is: Where D represents the pixel mean, represents the pixel mean value of the enhanced scene image in the red channel, represents the pixel mean value of the enhanced scene image in the green channel, Represents the pixel mean value of the enhanced scene image in the blue channel.

[0020] Preferably, the calculation formula of the balance coefficient is:

[0021]

[0022] Among them, K R Indicates the balance coefficient of the red channel, K G Indicates the balance coefficient of the green channel, K B represents the balance coefficient of the blue channel, R′ represents the pixel value of the red channel after balance, G′ represents the pixel value of the green channel after balance, B′ represents the pixel value of the blue channel after balance, R represents the pixel value of the enhanced scene image in the red channel, G represents the pixel value of the enhanced scene image in the green channel, and B represents the pixel value of the enhanced scene image in the blue channel.

[0023] Preferably, performing image conversion on the balanced scene image to obtain a converted scene image comprises:

[0024] Performing normalization processing on each color channel of the balanced scene image to obtain a normalized result;

[0025] Using formula Select the extreme value in the normalized result; where C max Indicates the maximum value in the normalized result, C min Indicates the minimum value in the normalized result;

[0026] Constructing a color space conversion formula according to the normalization result and the extreme value of the normalization result;

[0027] The balanced scene image is converted into a converted scene image using the color space conversion formula.

[0028] Preferably, the color space conversion formula is:

[0029]

[0030] V=C max

[0031] Among them, H represents the hue component, S represents the saturation component, and V represents the lightness component

[0032] Preferably, the processing steps of the multimodal feature fusion module include:

[0033] Extracting voxel features in the enhanced lidar point cloud using a three-dimensional feature extraction network of Voxel-RCNN;

[0034] Extracting image features from the conversion scene image;

[0035] Using a first feature fusion submodule to fuse the voxel feature and the image feature to generate an initial feature;

[0036] The SE channel attention mechanism is used to strengthen the attention of the target foreground point features in the initial features and reduce the attention of the background points;

[0037] The second feature fusion submodule is used to fuse the image feature with the generated initial feature to obtain the fused feature.

[0038] Preferably, the processing steps of the multimodal Voxel-RoI pooling module include:

[0039] Dividing a preset candidate box to divide the candidate box into a plurality of sub-grids, and calculating pseudo-point coordinates in a voxel coordinate system according to the sub-grids;

[0040] Sampling and voxelizing the image features in the fusion features according to the pseudo point coordinates to generate image voxel features;

[0041] Respectively obtain the image voxel features, the point cloud voxel features with a downsampling ratio of 4, and the point cloud voxel features with a downsampling ratio of 8, and perform voxel feature aggregation to obtain three aggregated features;

[0042] The aggregated features are fused to obtain the final RoI pooled output features.

[0043] A three-dimensional vehicle target detection system integrating voxel and image features, comprising:

[0044] A data acquisition unit, used to acquire an initial laser radar point cloud and an initial scene image of a sample detection scene;

[0045] A cross-modal enhancement unit, used to perform data enhancement on the laser radar point cloud and the initial scene image using a cross-modal real-box data enhancement method to obtain an enhanced laser radar point cloud and an enhanced scene image;

[0046] A color difference balancing unit, used for performing color difference balancing on the enhanced scene image to obtain a balanced scene image;

[0047] An image conversion unit, used for performing image conversion on the balanced scene image to obtain a converted scene image; the converted scene image is in the HSV color space;

[0048] A model building unit, used for inputting the enhanced laser radar point cloud and the converted scene image into an initial target detection model to obtain a trained three-dimensional vehicle target detection model; the initial target detection model includes a multimodal feature fusion module, a multimodal Voxel-RoI pooling module and a prediction module connected in sequence; the multimodal feature fusion module is used for performing multimodal feature fusion according to the enhanced laser radar point cloud and the converted scene image to obtain fusion features; the multimodal Voxel-RoI pooling module is used for performing RoI pooling operation according to the fusion features to obtain the final RoI pooling output features; the prediction module is used for performing classification confidence and position regression prediction according to the output features;

[0049] The detection unit is used to input the data to be tested of the target detection scene into the three-dimensional vehicle target detection model to obtain the detection result.

[0050] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0051] The present invention provides a three-dimensional vehicle target detection method and system that integrates voxel and image features. The method comprises: obtaining an initial laser radar point cloud and an initial scene image of a sample detection scene; performing data enhancement on the laser radar point cloud and the initial scene image using a cross-modal real frame data enhancement method to obtain an enhanced laser radar point cloud and an enhanced scene image; performing color difference balance on the enhanced scene image to obtain a balanced scene image; performing image conversion on the balanced scene image to obtain a converted scene image; the converted scene image is in the HSV color space; inputting the enhanced laser radar point cloud and the converted scene image into an initial target detection model to obtain a trained three-dimensional vehicle target detection method. Vehicle target detection model; the initial target detection model includes a multimodal feature fusion module, a multimodal Voxel-RoI pooling module and a prediction module connected in sequence; the multimodal feature fusion module is used to perform multimodal feature fusion according to the enhanced laser radar point cloud and the converted scene image to obtain fusion features; the multimodal Voxel-RoI pooling module is used to perform RoI pooling operation according to the fusion features to obtain the final RoI pooling output features; the prediction module is used to perform classification confidence and position regression prediction according to the output features; the data to be tested of the target detection scene is input into the three-dimensional vehicle target detection model to obtain the detection result. The present invention can realize accurate and rapid recognition of three-dimensional vehicle targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0053] Figure 1 A schematic diagram of the flow of a voxel-based point cloud target detection method provided in the prior art;

[0054] Figure 2 A flow chart of a method provided by an embodiment of the present invention;

[0055] Figure 3 A flow chart of a real frame sampling method provided by an embodiment of the present invention;

[0056] Figure 4 A multimodal feature fusion flow chart provided for an embodiment of the present invention;

[0057] Figure 5 A structural diagram of a feature fusion module provided in an embodiment of the present invention;

[0058] Figure 6 A basic principle diagram of RoI pooling provided by an embodiment of the present invention;

[0059] Figure 7 A diagram of a multimodal Voxel-RoI pooling method provided by an embodiment of the present invention;

[0060] Figure 8 A voxel sampling map provided by an embodiment of the present invention;

[0061] Fig. 9 A target refinement regression flow chart provided for an embodiment of the present invention;

[0062] Fig.10 A flow chart of a three-dimensional vehicle target detection algorithm for overall fusion of point cloud and image provided by an embodiment of the present invention;

[0063] Fig.11 A schematic diagram showing the change of the loss value of the VoxelImage algorithm with time steps provided by an embodiment of the present invention;

[0064] Fig.12 A schematic diagram of the change of the loss value of each branch over time during the training process of the multimodal Voxel-RoI pooling and target box refinement stage provided by an embodiment of the present invention;

[0065] Fig.13The algorithm-related loss value of the overall fusion of the point cloud and the image provided in the embodiment of the present invention changes with the time step. The algorithm-related loss value of the overall fusion of the point cloud and the image changes with the time step. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] The purpose of the present invention is to provide a three-dimensional vehicle target detection method and system that integrates voxel and image features, which can realize accurate and rapid recognition of three-dimensional vehicle targets.

[0068] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] Figure 2 A flow chart of a method provided by an embodiment of the present invention, such as Figure 2 As shown, the present invention provides a three-dimensional vehicle target detection method integrating voxel and image features, comprising:

[0070] Step 100: Obtaining an initial LiDAR point cloud and an initial scene image of a sample detection scene;

[0071] Step 200: Performing data enhancement on the lidar point cloud and the initial scene image using a cross-modal ground truth box data enhancement method to obtain an enhanced lidar point cloud and an enhanced scene image;

[0072] Step 300: performing color balance on the enhanced scene image to obtain a balanced scene image;

[0073] Step 400: performing image conversion on the balanced scene image to obtain a converted scene image; the converted scene image is in the HSV color space;

[0074] Step 500: inputting the enhanced laser radar point cloud and the converted scene image into the initial target detection model to obtain a trained three-dimensional vehicle target detection model; the initial target detection model includes a multimodal feature fusion module, a multimodal Voxel-RoI pooling module and a prediction module connected in sequence; the multimodal feature fusion module is used to perform multimodal feature fusion according to the enhanced laser radar point cloud and the converted scene image to obtain fusion features; the multimodal Voxel-RoI pooling module is used to perform RoI pooling operation according to the fusion features to obtain the final RoI pooling output features; the prediction module is used to perform classification confidence and position regression prediction according to the output features;

[0075] Step 600: input the data to be tested of the target detection scene into the three-dimensional vehicle target detection model to obtain the detection result.

[0076] Specifically, step 100 of this embodiment is cross-modal data enhancement. Data enhancement refers to the generation of a variety of training data to improve the robustness of the model by expanding the data through random translation, rotation, scaling, etc. during the model training phase. In three-dimensional target detection, point cloud data enhancement can avoid overfitting and enhance the generalization ability and robustness of the model. Three-dimensional target detection algorithms based on single lidar point clouds widely use the data enhancement method of ground truth (GT) sampling to enhance point clouds. This method enhances the diversity of point cloud scenes by copying the point clouds of the target ground truth boxes in other scenes to the training scenes, thereby improving the robustness of the model and the accuracy of target detection. The process of the ground truth sampling method is as follows: Figure 3 As shown, the red box represents the real box of the vehicle target in other scenes, and the orange point cloud represents the newly added noise point cloud in the target scene after sampling the real box.

[0077] In multimodal three-dimensional target detection, directly applying the point cloud real frame sampling method will destroy the alignment consistency between the laser radar point cloud and the image. To solve the above problem, a cross-modal real frame sampling method for point cloud and image is proposed. The method copies the point cloud real frame of other scenes to the training scene, and at the same time copies the image real frame corresponding to the sampled point cloud real frame to the corresponding training image in order from far to near, thereby ensuring the data consistency between the laser radar point cloud and the image. However, since the image is two-dimensional data and the field of view is limited, the above real frame sampling method will cause a large number of distant objects to be blocked, affecting the detection accuracy of distant training targets. The size of the near target is larger than the size of the distant target. Therefore, when the near target of other image scenes is copied to the training image scene, it is highly likely that a part of the distant target of the training image scene will be completely blocked. To solve this problem, this embodiment designs a cross-modal real frame data enhancement method to balance the near and far targets in the real frame sampling process and reduce the probability of distant objects being blocked. This method mainly removes the sampled real-box targets that completely occlude distant targets in the training scene, while retaining the real-box sampled targets with 3D IoU equal to 0 in the 3D point cloud space and 2D IoU less than 0.5 in the 2D image. The pseudo code of the cross-modal real-box data enhancement method is shown in Table 1.

[0078] Table 1 Cross-modal ground truth box data enhancement process

[0079]

[0080]

[0081]

[0082] Preferably, performing color difference balancing on the enhanced scene image to obtain a balanced scene image comprises:

[0083] Calculate the pixel mean of the enhanced scene image in the three RGB channels;

[0084] Calculating a balance coefficient for each color channel according to the pixel mean;

[0085] The color difference of the enhanced scene image is balanced by using the balance coefficient to obtain the balanced scene image.

[0086] Preferably, the pixel mean calculation formula is: Where D represents the pixel mean, represents the pixel mean value of the enhanced scene image in the red channel, represents the pixel mean value of the enhanced scene image in the green channel, Represents the pixel mean value of the enhanced scene image in the blue channel.

[0087] Preferably, the calculation formula of the balance coefficient is:

[0088]

[0089] Among them, K R Indicates the balance coefficient of the red channel, K G Indicates the balance coefficient of the green channel, K B represents the balance coefficient of the blue channel, R′ represents the pixel value of the red channel after balance, G′ represents the pixel value of the green channel after balance, B′ represents the pixel value of the blue channel after balance, R represents the pixel value of the enhanced scene image in the red channel, G represents the pixel value of the enhanced scene image in the green channel, and B represents the pixel value of the enhanced scene image in the blue channel.

[0090] Specifically, the color balance processing of step 300 of this embodiment can improve color consistency, and the color balance processing minimizes the impact of different lighting conditions and scene backgrounds on the image. By correcting the color deviation in the image, it is ensured that the obtained balanced scene image is more consistent in color, which helps the model to accurately identify the target object. This process enables the model to maintain recognition stability and accuracy when processing a variety of actual scenes. In applications such as autonomous driving and intelligent transportation, changes in ambient lighting are very common. By implementing color balance in the data preprocessing stage, the model's sensitivity to lighting changes is reduced, thereby improving its robustness in dynamic and complex environments. And color balance helps to enhance the visibility of features in the image, making it more obvious during subsequent processing and feature extraction, which helps to improve the effect of feature fusion in subsequent neural networks.

[0091] Preferably, performing image conversion on the balanced scene image to obtain a converted scene image comprises:

[0092] Performing normalization processing on each color channel of the balanced scene image to obtain a normalized result;

[0093] Using formula Select the extreme value in the normalized result; where C max Indicates the maximum value in the normalized result, C min Indicates the minimum value in the normalized result;

[0094] Constructing a color space conversion formula according to the normalization result and the extreme value of the normalization result;

[0095] The balanced scene image is converted into a converted scene image using the color space conversion formula.

[0096] Preferably, the color space conversion formula is:

[0097]

[0098] V=C max

[0099] Among them, H represents the hue component, S represents the saturation component, and V represents the lightness component.

[0100] Specifically, the image in step 400 of this embodiment is converted to the HSV color space, which has the flexibility of feature extraction: the HSV (hue, saturation, brightness) color space is more consistent with human visual perception than the RGB color space, and can more effectively extract and represent the color features in the image. Processing in the HSV space can make the color features of the vehicle target more prominent, thereby improving the accuracy of detection. The hue and saturation in the HSV color space can eliminate the impact of lighting changes to a certain extent, making target recognition under different lighting conditions more consistent. This lighting adaptability enhances the detection capability of the model, making it more capable of coping with different environmental conditions. By converting the image into the HSV color space, this embodiment can better combine the color features in the lidar point cloud data and the image data, provide a richer and more meaningful feature expression for the multimodal feature fusion module, and help integrate the deep learning model to understand and process multidimensional information.

[0101] Furthermore, step 300 of this embodiment significantly improves the color consistency of the image and the robustness of the model by implementing color difference balance, while step 400 enhances the flexibility and illumination adaptability of feature extraction by converting the image to the HSV color space. These two innovations improve data preprocessing and feature expression respectively, thereby improving the overall performance of three-dimensional vehicle target detection.

[0102] Preferably, the processing steps of the multimodal feature fusion module include:

[0103] Extracting voxel features in the enhanced lidar point cloud using a three-dimensional feature extraction network of Voxel-RCNN;

[0104] Extracting image features from the conversion scene image;

[0105] Using a first feature fusion submodule to fuse the voxel feature and the image feature to generate an initial feature;

[0106] The SE channel attention mechanism is used to strengthen the attention of the target foreground point features in the initial features and reduce the attention of the background points;

[0107] The second feature fusion submodule is used to fuse the image feature with the generated initial feature to obtain the fused feature.

[0108] Specifically, the lidar point cloud has the characteristic of “dense near and sparse far”. There are only a few point clouds for vehicle targets far away in the three-dimensional space. Correspondingly, the voxels of such vehicle targets only contain a small number of foreground points, and many semantic features cannot be obtained during the voxel feature extraction process. Therefore, the detection accuracy of such vehicle targets is low. Image data compresses the three-dimensional scene into a two-dimensional space and is not affected by depth and distance. It can better extract the features of distant vehicle targets during the feature extraction process. Based on the above analysis, this embodiment proposes a multimodal feature fusion method for fusing image features with voxel features, enriching the semantic features of sparse point cloud vehicle targets, thereby improving the detection accuracy of such vehicle targets to a certain extent. The entire multimodal feature fusion process is as follows: Figure 4 shown.

[0109] Furthermore, the 3D feature extraction network of Voxel-RCNN uses 3D sparse convolution to extract voxel features. Each 3D sparse convolution layer contains a spatial sparse convolution (SparseConv3d) and two sub-manifold sparse convolutions (SubMConv3d). The spatial sparse convolution is used to increase the receptive field of the convolution kernel to extract voxel context information features, and the sub-manifold sparse convolution is used to ensure that the position of the sparse voxel features remains unchanged. Considering that both point cloud voxel features and image features occupy a large amount of memory, multimodal feature fusion is only performed in the first layer of the backbone network. Figure 4 As can be seen from the figure, the image features and voxel features are first generated by the feature fusion module 1 (using the Concat method to splice features), and the SE channel attention mechanism is used to strengthen the attention of the target foreground point features in the fusion features and reduce the attention of the background points. Finally, the feature fusion module 2 (using the Sum method to add features) is used to fuse the image features and the generated fusion features. The feature fusion module designed in this embodiment is as follows Figure 5 shown.

[0110] Optionally, during the training phase, the algorithm uses data enhancement methods such as random flipping, scaling, and rotation on the original point cloud data, which causes the three-dimensional index stored after voxelization to be misaligned with the corresponding image. To ensure that the point cloud data can be correctly mapped back to the original image, the voxel three-dimensional index must first be converted to the original point cloud coordinate system. The point cloud data restoration method in the feature fusion module is to restore the point cloud data by storing parameters such as the flip axis, scaling factor, and rotation angle during the point cloud data enhancement process. The correction transformation method first converts the three-dimensional voxel data into a pixel according to the voxelization parameters. Convert to raw point cloud data set Then, the point cloud data is inversely transformed using the scaling factor, rotation angle and other parameters stored in the data enhancement stage to align it with the corresponding image features; the point cloud is projected onto the image feature map based on the calibration parameters and internal and external parameter matrices provided by the training data set; finally, the image features are sampled by bilinear interpolation to obtain the sampled image features. The sampled image features are voxelized and fused with the voxel features through Concat splicing (or Sum addition). However, since simple splicing or addition operations cannot fully play the role of image features, in order to better fuse image features and voxel features and increase the feature extraction of sparse targets, the fused features are subjected to sub-manifold sparse convolution to obtain deep fused features. The above correction transformation process is as follows:

[0111] F P =B(P(T -1 (G(V))),F img_feature )

[0112] Among them, F P is the image feature after sampling, B(·) represents bilinear interpolation, P(·) represents the point cloud projection process, T -1 (·) represents the data enhancement inverse transformation process, G(·) represents the process of converting 3D voxel data into original point cloud data, V represents 3D voxel data, and F ingfeature Represents image features.

[0113] Preferably, the processing steps of the multimodal Voxel-RoI pooling module include:

[0114] Dividing a preset candidate box to divide the candidate box into a plurality of sub-grids, and calculating pseudo-point coordinates in a voxel coordinate system according to the sub-grids;

[0115] Sampling and voxelizing the image features in the fusion features according to the pseudo point coordinates to generate image voxel features;

[0116] Respectively obtain the image voxel features, the point cloud voxel features with a downsampling ratio of 4, and the point cloud voxel features with a downsampling ratio of 8, and perform voxel feature aggregation to obtain three aggregated features;

[0117] The aggregated features are fused to obtain the final RoI pooled output features.

[0118] Specifically, the multimodal Voxel-RoI pooling method of this embodiment is as follows:

[0119] RoI pooling is one of the common methods for two-stage training networks. It aims to use the point cloud inside the candidate box and the point cloud or voxel features near the candidate box to further accurately predict the position and size of the target. The basic idea of ​​RoI pooling is: first, sample the target box obtained in the first stage of the algorithm to obtain the candidate box, divide the candidate box into equidistant sub-grids, take the center point of each sub-grid as the origin, and use the sampling strategy in PointNet++ to perform feature fusion on the point cloud (or voxel) within its radius, such as Figure 6 As shown in the figure. The red dots represent the center points of the equidistant sub-grids divided by the candidate boxes. They only indicate the position and are a virtual point cloud rather than actual point cloud data. The blue dots represent the original point cloud, the yellow dots are the sampling points obtained by the PointNet++ sampling strategy, and the green arrows indicate the aggregation of the features of the sampling points to the center point of the grid.

[0120] This embodiment further integrates image features based on the RoI pooling strategy of Voxel-RCNN and proposes a multimodal Voxel-RoI pooling method. Figure 7 As shown, it mainly includes the generation of candidate box grid center points and pseudo points, the association of pseudo points with image features, and the fusion of image features and voxel features at the grid center points.

[0121] The traditional fusion method only projects the center points of the grid divided by the candidate box onto the image features and samples them. Due to the imbalance in the resolution of the point cloud and the image, this method results in a low sampling rate of image features, affecting the overall detection performance. In order to better sample image features, this embodiment divides the center points of the candidate box grid (querypoints) equidistantly, enlarges the size of the candidate box, divides the new candidate box at another different scale, and generates pseudopoints. The pseudopoints are mapped back to the image features to sample the image features, and the features are voxelized based on the three-dimensional coordinates of the pseudopoints and the sampled image features to obtain the image voxel features. The image voxel features are aggregated to the grid center points, and the two voxel features with downsampling ratios of 4 and 8 in the three-dimensional feature extraction network are aggregated to generate fused features, and finally the target box is refined.

[0122] like Figure 7 As shown, the RoI pooling process used in this embodiment is as follows:

[0123] Step 1: Divide the candidate box. First, divide the candidate box RoI into 6×6×6 sub-grids, and calculate the position coordinates G of the center point of the divided grid in the point cloud coordinate system. xyz , the calculation formula is shown in the first two formulas below; at the same time, according to the third formula below, the candidate box RoI is enlarged, the RoI is divided into 16×8×22 scales, pseudo points are generated and the position coordinates of the pseudo points in the point cloud coordinate system are calculated. The calculation formulas are shown in the fourth and fifth formulas below:

[0124]

[0125] Among them, G temp and They represent the coordinates of the center point and the pseudo point in the RoI coordinate system, center_idx refers to the index coordinates of the center point of the grid, and size RoI and size grid The size of the current RoI and the size of each grid are respectively RoI_(l,w,h) Represent the length, width, and height of the current RoI respectively. and They represent the size of the enlarged RoI and the division scale of the enlarged RoI, θ is the rotation angle between the RoI coordinate system and the point cloud coordinate system, centerr RoI are the coordinates of the center point of the RoI.

[0126] Since voxel features are the features of the original point cloud after voxel encoding and three-dimensional sparse convolutional neural network, it is difficult to align grid points and pseudo points directly with voxel features. Therefore, it is necessary to calculate the corresponding voxel coordinates based on the grid point and pseudo point coordinates, as shown in the following formula:

[0127]

[0128] Among them, G voxel and are the voxel coordinates corresponding to the grid points and pseudo points, R represents the left boundary of the point cloud range, d represents the downsampling ratio, which are 2, 4, and 8 respectively, and size voxel and Refers to the voxel sizes corresponding to the grid points and pseudo points, which are (0.05, 0.05, 0.1) and (0.2, 0.2, 0.1), respectively.

[0129] Step 2: Sample image features and voxelize them. After obtaining the pseudo-point coordinates in the voxel coordinate system, use the voxelization parameters to convert them into the pseudo-point coordinates in the original point cloud coordinate system. Map the pseudo-points to the image coordinate system according to the correction parameters provided by the training data set, filter the pseudo-points outside the image range, and sample the image features by bilinear interpolation. Use the sampled image features as voxel features, and the pseudo-point voxel coordinates within the image range as indexes to perform voxelization and generate image voxel features.

[0130] Step 3: Voxel feature aggregation. Taking the central voxel in the voxel feature as the sphere center, set multiple sampling radii and obtain K non-empty voxels in the neighborhood of the sampling radius respectively, such as Figure 8As shown in the figure (the sampling radius is 2), the PointNet++ network is used to further extract voxel features from the sampled non-empty voxels and aggregate the features through maximum pooling, and aggregate the features to the center point of the grid. The red voxels represent the central voxels, and the blue voxels represent the non-empty voxels in the neighborhood of the central voxel with a radius of 2.

[0131] Step 4: Multimodal feature aggregation. Obtain image voxel features, point cloud voxel features with a downsampling ratio of 4, and point cloud voxel features with a downsampling ratio of 8, respectively. Set multiple central voxel neighborhood radii for these three voxel features, repeat the third step of voxel feature aggregation, and output three aggregate features. The three aggregate features obtained are fused as the output features of the final RoI pooling.

[0132] Furthermore, the target frame refinement steps of this embodiment are as follows:

[0133] The target box classification and positioning results generated in the first stage may have a certain deviation from the real box. The target box fine regression is used to further adjust the output results. The final output features obtained by the multimodal Voxel-RoI pooling method are first pooled to a fixed scale of N×41472 through a fully connected layer, and the classification fully connected layer and regression fully connected layer are used to perform classification confidence and position regression prediction, respectively. Fig. 9 shown.

[0134] The classification fully connected layer calculates the category confidence of each RoI and divides each RoI into a vehicle target object or a background object; the regression fully connected layer calculates the position offset between each RoI and the real target, and determines the target prediction position based on the offset and the RoI position. To prevent overfitting, the RoI pooling features (N×13824) aggregated by the image voxel features need to be separately classified and regressed to form an auxiliary branch to calculate the pooling loss. The above branch network structure is shown in Table 2.

[0135] Table 2 Target classification and regression network structure

[0136]

[0137] Furthermore, the loss function of this embodiment is designed as follows:

[0138] The loss function of the VoxelImage algorithm proposed in this embodiment is shown in Formula 5.9:

[0139]

[0140] in Represents the region network proposal loss in the first stage of the algorithm, which includes target classification loss and regression loss. It is the multimodal Voxel-RoI pooling and target box refinement loss in the second stage of the algorithm. is the total loss of the algorithm.

[0141] (1) RPN loss

[0142] Considering the imbalance of positive and negative samples, this embodiment uses the Focal Loss function to calculate the target classification loss, and uses the SmoothL1 function to calculate the regression loss. The RPN loss is shown in Formula 5.10.

[0143]

[0144] Among them, N fg refers to the number of foreground pinpoints, p i and δ i are the output results of the target classification network and the position regression network, respectively. i and t i Indicates the true target category and the true target location, Indicates that only the regression loss of the foreground pinpoints is calculated. Represents the target classification loss, calculated by the Focal Loss function, is the regression loss, calculated by the SmoothL1 function.

[0145] (2) RCNN loss

[0146] During the training process, the multimodal network is highly likely to be dominated by one of the modes. In the algorithm proposed in this embodiment, since the three-dimensional information in the generated image voxel features is far less than the lidar point cloud voxel features, the point cloud voxel features often play a dominant role in the multimodal Voxel-RoI pooling process, which may lead to overfitting. To solve the above problem, this embodiment introduces an auxiliary branch to calculate the RCNN loss based on the main branch RCNN loss calculation to avoid single-mode dominance and reduce overfitting. The RCNN loss is shown in the following formula:

[0147]

[0148] in is the total loss of multimodal Voxel-RoI pooling and target box refinement, represents the RCNN loss calculated by the main branch, represents the RCNN loss calculated by the auxiliary branch.

[0149] The main branch uses the same detection head loss as in

[70] , and the input is the image features and voxel features aggregated by the center points of the RoI grid, as shown in Equation 5.12:

[0150]

[0151] Where N R is the number of target candidate boxes participating in multimodal Voxel-RoI pooling, represents the target classification loss calculated by FocalLoss, is the position regression loss calculated by SmoothL1, and They represent the target category prediction results and target prediction positions output by the multimodal Voxel-RoI pooling and target box refinement network, respectively. i and t i They represent the real category and real position of the target respectively. It means that only the regression loss of the foreground target is calculated. λ1 is the classification loss weight, which is set to 1.0, and λ2 is the regression loss weight, which is set to 1.0.

[0152] The auxiliary branch is similar to the main branch, except that the auxiliary branch takes the image features aggregated by the center points of the RoI grid as input and only predicts the position regression results, as shown in the following formula:

[0153]

[0154] The meanings of the parameters in this formula are the same as above, except that the classification loss weight λ1 is set to 0 and the regression loss weight λ2 is set to 0.5.

[0155] Furthermore, this embodiment uses the KITTI3DObject dataset to train a three-dimensional vehicle target detection model that fuses voxel and image features, qualitatively and quantitatively analyzes the vehicle target detection performance of the algorithm, conducts ablation experiments to evaluate the effectiveness of related modules, compares the performance differences between the feature fusion algorithm proposed in this embodiment and other methods, verifies the performance differences of image features fused at different voxel layers and the effectiveness of cross-modal data enhancement methods. In order to comprehensively analyze the performance differences between the three-dimensional target detection algorithm that fuses point clouds and images and the traditional single-lidar point cloud three-dimensional target detection algorithm, this embodiment introduces a pixel-level fusion module and a decision-level fusion module on the basis of the fusion of voxel features and image features, and designs the following: Fig.10 The three-dimensional vehicle target detection method shown in the figure. This embodiment also uses the KITTI3DObject dataset to train the model, and further verifies the effectiveness of point cloud and image fusion by introducing pixel-level fusion, decision-level fusion, and overall fusion ablation experiments separately; by separately introducing pixel-level fusion method, feature-level fusion method, and decision-level fusion method in the Voxel-RCNN benchmark algorithm, the performance differences between the three different fusion methods are analyzed.

[0156] For the three-dimensional vehicle target detection algorithm that integrates voxel and image features, this embodiment uses Pytorch and OpenPCDet basic framework to implement the basic process of the algorithm and trains on an RTX3090 GPU. The Adam model parameter update optimizer is used, with a maximum learning rate of 0.008, a weight decay rate of 0.01, a batch size of 2 samples per training, a momentum of 0.9, and a total of 80 iterations. The target detection loss during the algorithm training process changes with time steps as shown in Fig.11 As shown. Fig.11 It can be seen that the total loss, RPN loss and RCNN loss of the model reached convergence at time step 140001, and the model iterated to the 77th time.

[0157] Fig.12 This is a graph showing the changes in the loss values ​​of each branch over time during the training process of the multimodal Voxel-RoI pooling and target box refinement stage. The blue, red, gray, orange, and light blue curves represent the total loss, classification loss, positioning loss, positioning loss of the auxiliary branch, and classification loss of the auxiliary branch. The classification loss of the auxiliary branch has a value of 0 because its weight is set to 0. At time step 140001, the losses of each branch in this stage converged, and the model iterated to 77 rounds.

[0158] For the 3D vehicle target detection algorithm that integrates point cloud and image, this embodiment also uses Pytorch and OpenPCDet as the benchmark framework to implement the basic module of the algorithm, uses 1 RTX 3090 GPU to train the model, uses Adam optimizer to update the model parameters, the maximum learning rate is 0.01, the weight decay rate is 0.01, the batch size is 2, and the total number of iterations is 80. The target detection loss of this algorithm changes as follows: Fig.13 As shown. Fig.13 It can be seen that at time step 50001 (i.e., 80 model iterations), the total loss, RPN loss, and RCNN loss reached convergence.

[0159] The evaluation indicators used in this embodiment experiment are mainly AP 3D _R11, AP 3D _R40 is used as the average detection accuracy indicator. 3D IoU and BEV IoU are used to determine whether the predicted box hits the real box. The IoU thresholds of the three difficulty levels of vehicle targets are set to 0.7.

[0160] This embodiment evaluates the vehicle target detection performance of the feature fusion algorithm proposed in this embodiment from both quantitative and qualitative perspectives, specifically including the following: comparing the average target detection accuracy of the algorithm proposed in this embodiment with the benchmark algorithm Voxel-RCNN, the cylindrical three-dimensional vehicle target detection algorithm based on adaptive feature fusion (Algorithm 2), and the three-dimensional vehicle target detection algorithm that integrates image texture features and prior information (Algorithm 3), and comparing the average target detection accuracy of the algorithm proposed in this embodiment with other three-dimensional target detection algorithms; comparing the running time of the multimodal Voxel-RoI pooling method and the benchmark RoI pooling method; visualizing some scenes of the KITTI 3DObject dataset, and qualitatively analyzing the effectiveness of vehicle target detection of the algorithm in this embodiment.

[0161] Specifically, the vehicle target detection performance of the feature fusion algorithm proposed in this embodiment is quantitatively analyzed as follows:

[0162] This experiment uses two different methods, R11 (11 interpolation points) and R40 (40 interpolation points), to calculate the average detection accuracy AP3D. 3D The 3D vehicle target detection performance of the feature fusion algorithm proposed in this embodiment is evaluated, and the results are shown in Table 3. When the IoU threshold is set to 0.7 and the average detection accuracy is calculated using the R11 method, the feature fusion algorithm proposed in this embodiment achieves 90.02%, 86.45%, and 85.51% of the 3D target average detection accuracy AP at the three difficulty levels of simple, medium, and difficult, respectively. 3D _R11; When the R40 method is used to calculate the average detection accuracy, the average detection accuracy AP of 3D objects is 93.09%, 88.19%, and 85.85% respectively at the above three difficulty levels. 3D The above experimental results effectively show that the algorithm proposed in this embodiment can accurately detect vehicle targets in three-dimensional scenes. Compared with Algorithm 2 and the benchmark algorithm Voxel-RCNN, the method in this embodiment enhances the richness of semantic features of sparse vehicle target point clouds by fusing image features, and enhances the average detection accuracy of the algorithm for difficult samples, such as AP on difficult samples. 3D _R11 increased by 6.58% and 8.15% respectively. Compared with the 3D object detection algorithms of pixel-level fusion and decision-level fusion, the detection accuracy of the algorithm in this embodiment is slightly inferior on simple difficulty samples, but it achieves better detection performance on medium and difficult difficulty samples. For example, AP 3D _R11 increased by 6.06% and 7.08% respectively.

[0163] Table 3 Quantitative evaluation results of the feature fusion algorithm proposed in this embodiment on the KITTI 3D Object validation set

[0164]

[0165] (1) Comparison of the feature fusion algorithm proposed in this embodiment with other methods

[0166] Table 4 compares the vehicle target detection accuracy of the feature fusion algorithm proposed in this embodiment and other typical three-dimensional target detection algorithms, and uses the R11 (11 recall positions) method for measurement. The detection accuracy results of other methods are from their respective literature.

[0167] Table 4 Comparison of vehicle target detection accuracy of the method in this embodiment and other methods in the KITTI 3D Object verification set

[0168]

[0169] As can be seen from Table 5.4, the algorithm of this embodiment has achieved better 3D target detection results at three difficulty levels: simple, medium and difficult, compared with PointRCNN, SECOND, 3DSSD, PV-RCNN and other 3D target detection algorithms based on single lidar point cloud. This proves that the fused image features have a good effect on improving the accuracy of 3D target detection, and the 3D target detection method of multimodal sensors can effectively make up for the shortcomings of a single sensor; compared with F-ConvNet, PointPainting and other 3D target detection algorithms based on pixel-level fusion, The method of this embodiment has significantly leading performance, proving that the feature-level fusion method is superior to the pixel-level fusion method in terms of detection accuracy. Compared with PI-RCNN, 3D-CVF, EPNet, PVFusion, PA3DNet, MLF-DET, LoGoNet and other three-dimensional target detection algorithms based on feature-level fusion, the method of this embodiment has achieved competitive results, with significant improvements on difficult samples, and the average detection accuracy AP3D is improved by 7.76%, 12.4%, 5.37%, 5.35%, 7.63%, 7.62% and 1.52%, respectively.

[0170] Furthermore, the execution time comparison of the multimodal Voxel-RoI pooling proposed in this embodiment is as follows:

[0171] This embodiment experiment compares the execution time required for the multimodal Voxel-RoI pooling method proposed in this embodiment and the RoI pooling method of the benchmark algorithm Voxel-RCNN in the model reasoning stage. Table 5 is a comparison of the average execution time of RoI pooling of two different methods on the KITTI3DObject validation set. From the comparative experimental results, it can be seen that the method proposed in this embodiment inevitably increases the time overhead of the RoI pooling stage because it needs to generate pseudo points and fuse image features at the RoI pooling stage, so the execution time overhead is high. In order to reduce the impact of high time overhead caused by the introduction of image features, the method of this embodiment reduces the average RoI pooling number from the initial value of 100 set by the benchmark method Voxel-RCNN to 40, which is 2.5 times less than the Voxel-RCNN method as a whole. On the basis of ensuring the detection accuracy of the algorithm, the average RoI pooling time is only increased by 2.4ms, indicating that the multimodal Voxel-RoI pooling method proposed in this embodiment is a lightweight method, which improves the detection performance of the algorithm with a small amount of time overhead.

[0172] Table 5 Comparison of RoI pooling execution time between the method in this embodiment and the Voxel-RCNN method

[0173]

[0174] Corresponding to the above method, this embodiment further provides a three-dimensional vehicle target detection system integrating voxel and image features, including:

[0175] A data acquisition unit, used to acquire an initial laser radar point cloud and an initial scene image of a sample detection scene;

[0176] A cross-modal enhancement unit, used to perform data enhancement on the laser radar point cloud and the initial scene image using a cross-modal real-box data enhancement method to obtain an enhanced laser radar point cloud and an enhanced scene image;

[0177] A color difference balancing unit, used for performing color difference balancing on the enhanced scene image to obtain a balanced scene image;

[0178] An image conversion unit, used for performing image conversion on the balanced scene image to obtain a converted scene image; the converted scene image is in the HSV color space;

[0179] A model building unit, used for inputting the enhanced laser radar point cloud and the converted scene image into an initial target detection model to obtain a trained three-dimensional vehicle target detection model; the initial target detection model includes a multimodal feature fusion module, a multimodal Voxel-RoI pooling module and a prediction module connected in sequence; the multimodal feature fusion module is used for performing multimodal feature fusion according to the enhanced laser radar point cloud and the converted scene image to obtain fusion features; the multimodal Voxel-RoI pooling module is used for performing RoI pooling operation according to the fusion features to obtain the final RoI pooling output features; the prediction module is used for performing classification confidence and position regression prediction according to the output features;

[0180] The detection unit is used to input the data to be tested of the target detection scene into the three-dimensional vehicle target detection model to obtain the detection result.

[0181] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0182] In this embodiment, specific examples are used to illustrate the principle and implementation of the present invention. The above description of the embodiment is only used to help understand the method and core idea of ​​the present invention; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A three-dimensional vehicle target detection method integrating voxel and image features, characterized in that: include: Obtaining an initial lidar point cloud and an initial scene image of a sample detection scene; Performing data enhancement on the laser radar point cloud and the initial scene image using a cross-modal true box data enhancement method to obtain an enhanced laser radar point cloud and an enhanced scene image; Performing color difference balance on the enhanced scene image to obtain a balanced scene image; Performing image conversion on the balanced scene image to obtain a converted scene image; the converted scene image is in the HSV color space; The enhanced laser radar point cloud and the converted scene image are input into the initial target detection model to obtain a trained three-dimensional vehicle target detection model; the initial target detection model includes a multimodal feature fusion module, a multimodal Voxel-RoI pooling module and a prediction module connected in sequence; the multimodal feature fusion module is used to perform multimodal feature fusion according to the enhanced laser radar point cloud and the converted scene image to obtain fusion features; the multimodal Voxel-RoI pooling module is used to perform RoI pooling operation according to the fusion features to obtain the final RoI pooling output features; the prediction module is used to perform classification confidence and position regression prediction according to the output features; The data to be tested of the target detection scene is input into the three-dimensional vehicle target detection model to obtain the detection result.

2. The three-dimensional vehicle target detection method integrating voxel and image features according to claim 1, characterized in that: The enhanced scene image is subjected to color difference balancing to obtain a balanced scene image, comprising: Calculate the pixel mean of the enhanced scene image in the three RGB channels; Calculating a balance coefficient for each color channel according to the pixel mean; The color difference of the enhanced scene image is balanced by using the balance coefficient to obtain the balanced scene image.

3. The three-dimensional vehicle target detection method integrating voxel and image features according to claim 2, characterized in that: The pixel mean calculation formula is: Where D represents the pixel mean, represents the pixel mean value of the enhanced scene image in the red channel, represents the pixel mean value of the enhanced scene image in the green channel, Represents the pixel mean value of the enhanced scene image in the blue channel.

4. The three-dimensional vehicle target detection method integrating voxel and image features according to claim 3, characterized in that: The calculation formula of the balance coefficient is: Among them, K R Indicates the balance coefficient of the red channel, K G Indicates the balance coefficient of the green channel, K B represents the balance coefficient of the blue channel, R′ represents the pixel value of the red channel after balance, G′ represents the pixel value of the green channel after balance, B′ represents the pixel value of the blue channel after balance, R represents the pixel value of the enhanced scene image in the red channel, G represents the pixel value of the enhanced scene image in the green channel, and B represents the pixel value of the enhanced scene image in the blue channel.

5. The three-dimensional vehicle target detection method integrating voxel and image features according to claim 4, characterized in that: Performing image conversion on the balanced scene image to obtain a converted scene image includes: Performing normalization processing on each color channel of the balanced scene image to obtain a normalized result; Using formula Select the extreme value in the normalized result; where C max Indicates the maximum value in the normalized result, C min Indicates the minimum value in the normalized result; Constructing a color space conversion formula according to the normalization result and the extreme value of the normalization result; The balanced scene image is converted into a converted scene image using the color space conversion formula.

6. The three-dimensional vehicle target detection method integrating voxel and image features according to claim 5, characterized in that: The color space conversion formula is: V=C max Among them, H represents the hue component, S represents the saturation component, and V represents the lightness component.

7. The three-dimensional vehicle target detection method integrating voxel and image features according to claim 1, characterized in that: The processing steps of the multimodal feature fusion module include: Extracting voxel features in the enhanced lidar point cloud using a three-dimensional feature extraction network of Voxel-RCNN; Extracting image features from the conversion scene image; Using a first feature fusion submodule to fuse the voxel feature and the image feature to generate an initial feature; The SE channel attention mechanism is used to strengthen the attention of the target foreground point features in the initial features and reduce the attention of the background points; The second feature fusion submodule is used to fuse the image feature with the generated initial feature to obtain the fused feature.

8. The three-dimensional vehicle target detection method integrating voxel and image features according to claim 1, characterized in that: The processing steps of the multimodal Voxel-RoI pooling module include: Dividing a preset candidate box to divide the candidate box into a plurality of sub-grids, and calculating pseudo-point coordinates in a voxel coordinate system according to the sub-grids; Sampling and voxelizing the image features in the fusion features according to the pseudo point coordinates to generate image voxel features; Respectively obtain the image voxel features, the point cloud voxel features with a downsampling ratio of 4, and the point cloud voxel features with a downsampling ratio of 8, and perform voxel feature aggregation to obtain three aggregated features; The aggregated features are fused to obtain the final RoI pooled output features.

9. A three-dimensional vehicle target detection system integrating voxel and image features, characterized in that: include: A data acquisition unit, used to acquire an initial laser radar point cloud and an initial scene image of a sample detection scene; A cross-modal enhancement unit, used to perform data enhancement on the laser radar point cloud and the initial scene image using a cross-modal real-box data enhancement method to obtain an enhanced laser radar point cloud and an enhanced scene image; A color difference balancing unit, used for performing color difference balancing on the enhanced scene image to obtain a balanced scene image; An image conversion unit, used for performing image conversion on the balanced scene image to obtain a converted scene image; the converted scene image is in the HSV color space; A model building unit, used for inputting the enhanced laser radar point cloud and the converted scene image into an initial target detection model to obtain a trained three-dimensional vehicle target detection model; the initial target detection model includes a multimodal feature fusion module, a multimodal Voxel-RoI pooling module and a prediction module connected in sequence; the multimodal feature fusion module is used for performing multimodal feature fusion according to the enhanced laser radar point cloud and the converted scene image to obtain fusion features; the multimodal Voxel-RoI pooling module is used for performing RoI pooling operation according to the fusion features to obtain the final RoI pooling output features; The prediction module is used to perform classification confidence and position regression prediction according to the output features; The detection unit is used to input the data to be tested of the target detection scene into the three-dimensional vehicle target detection model to obtain the detection result.

Images