An image point cloud three-dimensional target detection method based on fine feature extraction

Abstract

The application discloses a kind of based on fine feature extraction image point cloud three-dimensional target detection method, including the following steps: obtaining look-around two-dimensional image feature map and point cloud voxel feature map, according to point cloud image space projection relationship, using deformable cross attention mechanism, fusion point cloud voxel center point and the feature near corresponding image point, application backbone network extracts voxel three-dimensional feature, by constructing three-dimensional feature space distribution weight chart, adaptive fusion height dimension feature, generate multi-scale BEV feature and fusion, again according to high-resolution BEV feature map generation three-dimensional target detection frame, obtain the specific location and class of target.The application constructs three-dimensional feature space distribution weight chart, adaptive fusion height dimension feature, reduces the information loss existing when three-dimensional feature dimension reduction, and introduces an additional multi-scale BEV feature fusion branch, improves the perception ability of network model to geometric information, realizes more optimal detection effect.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional target detection based on computer vision, and in particular to a three-dimensional target detection method based on image point cloud and refined feature extraction. Background Art

[0002] In recent years, autonomous driving has garnered increasing attention due to its potential to reduce driver burden and improve driving safety. Modern autonomous vehicle systems can be divided into three modules: perception, planning, and control. The perception module is responsible for environmental perception and localization, and its output provides a reliable data source for downstream planning and control modules. The perception module encompasses many common visual tasks, such as object detection and segmentation, lane detection, and semantic and instance segmentation. Among these visual tasks, three-dimensional object detection is one of the most indispensable tasks for perception systems. It aims to detect the position, size, and category of key objects in the driving environment, ultimately outputting them as three-dimensional object detection boxes.

[0003] Commonly used sensors for 3D object detection tasks include lidar and cameras. LiDAR can obtain a 360-degree field of view and accurate depth information, ultimately presenting it in the form of a point cloud, where each point contains spatial coordinates (x, y, z) and reflection intensity r; cameras present the surrounding environment as RGB images containing semantic information. Initially, single-modality methods based solely on lidar or cameras developed rapidly. However, this model of using only point clouds or images has inherent flaws, as point clouds cannot provide information about mutual occlusion of targets or texture information, and point clouds are even sparser at long distances. In contrast, image-based methods provide sufficient semantic information but lack geometric information. Therefore, to address the inherent limitations of a single modality, a large number of researchers have developed 3D object detection methods that fuse image point clouds with lidar and cameras.

[0004] The three-dimensional target detection method based on image point cloud fusion can be roughly divided into three categories: front fusion, middle fusion and post fusion.(1) Front fusion, that is, fusing image and point cloud at data level. For example, MV3D fuses RGB image and point cloud bird's eye view and point cloud front view, and finally generates 3D bounding box through feature extraction and feature fusion; PointPainting uses image semantic segmentation to obtain pixel-level semantic information to enhance point cloud features.(2) Middle fusion, that is, fusing image and point cloud at feature level. For example, SFD uses a projection matrix to realize depth completion of the image, obtains a pseudo point cloud, extracts corresponding image features, fuses the image features with the point cloud features, and then further detects; BEVFusion extracts BEV features of the point cloud and the image respectively, splices the two features, and then detects.(3) Post fusion, that is, fusing at the detection result. For example, CLOCS obtains 2D image detection box and 3D point cloud detection box respectively, and updates the 3D detection box by calculating the coincidence degree of the two. In the existing methods, when fusing features, simple splicing, addition and other operations are used, and when obtaining BEV features, the height dimension features are mostly flattened, and then a 2D convolution network is used to obtain the detection result. However, this method is simple, but it loses part of the geometric information, which is not conducive to the detection of small targets.

[0005] In the existing point cloud image fusion model, PointAugmenting proposes to use image features instead of pixel-level semantic information, project the point cloud to the image plane through the camera and laser radar external parameter matrix, and realize the fusion of corresponding point cloud voxel features and image features. However, its defects are that the alignment relationship generated by this projection method has the defects of multi-value mapping and failure to fully utilize image features, and the fusion accuracy is low. At the same time, when the baseline model Centerpoint reduces the three-dimensional voxel features to two-dimensional BEV features, the flattening operation is used, which will lose a large amount of three-dimensional geometric information, which is not conducive to the regression of the subsequent detection box. SUMMARY

[0006] In order to solve the coarse granularity problem existing in the feature extraction of the existing image point cloud fusion three-dimensional target detection, the present application proposes an image point cloud three-dimensional target detection method based on fine feature extraction, which aggregates image features in the point cloud field through a deformable cross-attention mechanism, constructs a three-dimensional feature space distribution weight map, adaptively aggregates height dimension features, and introduces an additional multi-scale BEV feature fusion branch, thereby achieving better detection effect.

[0007] The present application is realized at least by one of the following technical solutions.

[0008] An image point cloud three-dimensional target detection method based on fine feature extraction, comprising the following steps:

[0009] Obtaining two-dimensional images captured by multiple surround-view cameras and performing feature extraction on the two-dimensional images to obtain multiple two-dimensional image feature maps;

[0010] Obtain a point cloud image and perform feature encoding on the point cloud image using a voxel encoding network to obtain point cloud voxel features;

[0011] According to the spatial projection relationship of the point cloud image, a deformable cross-attention mechanism is used to extract the center point of the point cloud voxel and project it to the two-dimensional image features near the corresponding point after the image is projected;

[0012] Adding the point cloud voxel features to the corresponding two-dimensional image features to obtain fused multimodal features;

[0013] The multimodal features are fed into the point cloud 3D object detection backbone network. While downsampling, a feature weight map is introduced to aggregate the height dimension features to obtain BEV features of multiple different scales.

[0014] Using convolution and addition operations to fuse the multiple BEV features of different scales to a specific scale to obtain a fused BEV feature;

[0015] The fused BEV features are fed into the RPN network to generate a high-resolution feature map;

[0016] Generate a three-dimensional target detection frame based on the high-resolution feature map to obtain the specific location and category of the target.

[0017] Furthermore, the feature extraction of the two-dimensional image to obtain a plurality of two-dimensional image feature maps specifically includes:

[0018] The image features of the two-dimensional image are extracted using the ResNet50 backbone network of the deeplabv3 detector pre-trained on the COCO dataset.

[0019] Furthermore, the voxel encoding network is used to obtain the point cloud voxel features, including:

[0020] The space where the point cloud is located is divided into voxels of the same size, and the point cloud data is assigned to the corresponding voxels according to its coordinates. According to the sparsity of the point cloud, the voxels are divided into empty voxels that do not contain point clouds and voxels that contain point clouds. Sparse three-dimensional convolution is used to extract voxel features from voxels that contain point clouds.

[0021] Furthermore, the point cloud is divided into multiple voxels, and feature encoding and feature extraction are performed on the voxels to obtain the features of each voxel point cloud, including:

[0022] The entire three-dimensional space occupied by the point cloud is divided into multiple voxels, and spatial classification is performed according to the three-dimensional coordinates of the point cloud, and the point cloud is classified into the corresponding voxels.

[0023] Filter the point cloud data and divide the entire space into multiple voxels in a 3D perspective. The resolution of the division is D×W×H, where D, W, and H represent the depth, width, and height of the divided space, respectively. Classify the point cloud voxels according to the 3D coordinates and classify the point cloud into the corresponding voxels. Only non-empty voxels with a point count greater than zero are retained.

[0024] For voxels with more than T points in the voxels, T point clouds are randomly screened out, and voxels with less than T points are filled with zeros to achieve structural processing of irregular point clouds; for voxels with more than T points, T points are randomly screened out, and voxels with less than T points are filled with zeros;

[0025] Aggregate all points within the voxel, use the point cloud coordinate mean to represent the initial voxel feature, and use the sparse 3D convolution block to extract the voxel feature F P (N, C), where N represents the number of valid point cloud points within the divided voxel, and C represents the number of feature channels.

[0026] Furthermore, based on the spatial projection relationship of the point cloud image, the two-dimensional image features near the corresponding point after the center point of the point cloud voxel is projected onto the image are extracted and added to the voxel features to obtain multimodal features, specifically including:

[0027] According to the calibrated lidar extrinsic parameters and camera extrinsic parameters, the coordinate transformation matrix of the point cloud-image is obtained;

[0028] According to the coordinate transformation matrix, the point cloud coordinates of the voxel center point are projected to the corresponding image to obtain the coordinate position of the corresponding image, and the image features of the corresponding coordinate position are obtained using bilinear interpolation;

[0029] Generate an attention matrix and an offset according to the image features, obtain image features at the image offset position using bilinear interpolation, and generate an image feature matrix;

[0030] Apply the attention matrix to the image feature matrix and pass it through a fully connected layer to obtain weighted aggregated image features;

[0031] The point cloud voxel features are added to the corresponding image features to obtain fused multimodal features.

[0032] Furthermore, the multimodal features are fed into the point cloud 3D object detection backbone network. While downsampling, a feature weight map is introduced to aggregate the height dimensional features to obtain BEV features of multiple different scales, specifically including:

[0033] According to the multi-modal feature, a multi-layer perception is used to generate feature weight maps of different high dimensions, after Softmax normalization, the feature weight maps are applied to high-dimensional features to obtain BEV features after dimension reduction by adaptive weighting, and the dimension reduction method is applied to 3D voxel features of multiple different scales to obtain BEV features of multiple different scales.

[0034] Further, the multiple different scale BEV features are fused to a specific scale using convolution and addition operations to obtain fused BEV features, specifically including:

[0035] The original maximum scale BEV feature is down-sampled by 2 using 2D convolution, and then added to the BEV feature of the same scale, and the 2D convolution down-sampling operation is repeatedly applied and added to the BEV feature of the same size, and finally the fused BEV feature is obtained by splicing with the 8 times down-sampled original feature map.

[0036] Further, the fused BEV feature is sent to the RPN network to generate a high-resolution feature map, specifically including:

[0037] The fused BEV feature is extracted and integrated using the RPN network, and the RPN network used includes two convolution feature extraction modules and two up-sampling modules, two convolution feature extraction modules, which gradually extract the input feature, and two up-sampling modules, which are decoders that restore the feature to a higher resolution.

[0038] Further, a three-dimensional target detection frame is generated according to the high-resolution feature map to obtain the specific position and category of the target, specifically including:

[0039] Six CenterHead detection heads are used to obtain detection results of six large categories from the high-resolution feature map;

[0040] The category and position parameters of the target detection frame are integrated according to the detection results of the six large categories as the three-dimensional target detection results.

[0041] Further, the detection result of each large category includes a heat map representing the position of the target center point and the category, a horizontal offset of the target center point, a height of the target center point, a size of the target, and an orientation.

[0042] Compared with the prior art, the beneficial effects of the present application are:

[0043] The present application realizes multi-scale BEV feature extraction of fine granularity. In view of the feature loss problem existing in the dimension reduction of point cloud voxel height features, a three-dimensional feature space distribution weight map is designed to adaptively fuse high-dimensional features, and multi-scale BEV feature maps are aggregated, finally realizing higher fusion accuracy and achieving better three-dimensional target detection effect. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0045] Figure 1 A flowchart of the image point cloud three-dimensional target detection method based on fine feature extraction of the present application,

[0046] Figure 2 A schematic diagram of the overall framework of the image point cloud three-dimensional target detection method based on fine feature extraction of the present application,

[0047] Figure 3 A process schematic diagram of the deformable cross-attention mechanism provided by the present application,

[0048] Figure 4 A process schematic diagram of the adaptive height dimension feature fusion provided by the present application,

[0049] Figure 5 A process schematic diagram of the multi-scale BEV feature fusion provided by the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0051] The purpose of the present application is to propose an image point cloud three-dimensional target detection method based on adaptive feature fusion, in order to solve the problems of coarse granularity in image point cloud feature fusion and loss of height dimension features in feature dimension reduction in existing methods, and to achieve better detection effect.

[0052] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0053] Figure 1 A flowchart of the image point cloud three-dimensional target detection method based on fine feature extraction of the present application, Figure 2 A schematic diagram of the overall framework of the image point cloud three-dimensional target detection method based on fine feature extraction of the present application. As Figure 1 and Figure 2As shown, the present invention provides a method for three-dimensional target detection based on image point cloud based on refined feature extraction, comprising the following steps:

[0054] Step 1: Obtain two-dimensional images captured by multiple surround-view cameras and perform feature extraction on the two-dimensional images to obtain multiple two-dimensional image feature maps.

[0055] LiDAR and cameras, as commonly used sensors in autonomous driving perception systems, play a key role in three-dimensional target detection algorithms. These algorithms use sensor data from LiDAR or cameras to perceive the surrounding environment, accurately identify and locate various objects on the road, and are a key link in achieving safe driving of autonomous vehicles. For LiDAR sensors, they usually use multiple laser beams to scan the surrounding environment and generate three-dimensional point cloud data. These point cloud data consist of three-dimensional coordinates (x, y, z) and reflection intensity r. LiDAR provides high-precision spatial information. Camera sensors collect visual information from the surroundings of the vehicle. They obtain two-dimensional images and detect and identify target objects by analyzing and processing features such as object boundaries, color and texture in the image. In autonomous driving systems, the data of these two types of sensors are usually fused to comprehensively utilize the advantages of both and improve the accuracy and robustness of target detection. Step 1 of the present invention is mainly used to extract features from the two-dimensional image (hereinafter referred to as the image) collected by the surround-view camera to obtain two-dimensional image features.

[0056] Specifically, step 1 extracts features from the two-dimensional image to obtain multiple two-dimensional image features, specifically including:

[0057] Step 1) using the DeepLabV3 detector to obtain a feature map of the two-dimensional image;

[0058] Specifically, the two-dimensional image data is passed through the DeepLabV3 detector to obtain the first layer features F of all images. c ={F1,F2,…,Fn}; where n is the number of images and Fn represents the feature map of the nth image;

[0059] Step 2) Obtain a point cloud image and use a voxel encoding network to perform feature encoding on the point cloud image to obtain point cloud voxel features.

[0060] In step 2), the point cloud is divided into multiple voxels, and feature encoding and feature extraction are performed on the voxels to obtain the features of each voxel point cloud, specifically including:

[0061] Step 2.1) Divide the entire three-dimensional space occupied by the point cloud into multiple voxels, and perform spatial classification based on the three-dimensional coordinates of the point cloud, and classify the point cloud into corresponding voxels.

[0062] Filter point cloud data within a certain spatial range and divide the entire space into multiple voxels from a 3D perspective. The resolution of the partition is D×W×H, where D, W, and H represent the depth, width, and height of the partitioned space, respectively. The point cloud is voxel-classified according to the 3D coordinates, and the point cloud is classified into the corresponding voxel. Only non-empty voxels with a point count greater than zero are retained.

[0063] Step 2.2) For voxels with more than T points, randomly select T point clouds, and fill the voxels with less than T points with zeros to achieve structural processing of irregular point clouds; for voxels with more than T points, randomly select T points, and fill the voxels with less than T points with zeros. The above operations achieve structural processing of irregular point clouds.

[0064] Step 2.3) Aggregate all points in the voxel, use the mean of the point cloud coordinates to represent the initial voxel features, and use three-dimensional sparse convolution to extract the voxel features F P (N, C), where N represents the number of valid point cloud points within the divided voxel, and C represents the number of feature channels.

[0065] Step 3: Based on the spatial projection relationship of the point cloud image, a deformable cross-attention mechanism is used to extract the two-dimensional image features near the corresponding point after the center point of the point cloud voxel is projected onto the image.

[0066] like Figure 3 As shown in the figure, based on the calibrated lidar and camera extrinsic parameters, a point cloud-to-image coordinate transformation matrix is ​​derived. This matrix is ​​then used to project the point cloud onto the image, obtaining the pixel coordinates of each point in the image. For each pixel generated after projection, bilinear interpolation is used to obtain its corresponding image features. A linear layer is then used to generate offsets and weight matrices for the image features. Bilinear interpolation is then used to obtain the image features at the offsets. The weight matrix is ​​then applied to the output image features.

[0067] Step 3 uses a deformable cross attention mechanism based on the spatial projection relationship of the point cloud image to extract the two-dimensional image features near the corresponding point after the center point of the point cloud voxel is projected onto the image, specifically including:

[0068] Step 3.1) Obtain the coordinate transformation matrix IT of the point cloud-image based on the pre-calibrated lidar external parameters and camera internal and external parameters. l→c The coordinate transformation matrix of the point cloud-image includes the camera internal parameter matrix I and the external parameter matrix T from the laser radar to the camera l→c .

[0069] The camera intrinsic parameter matrix I can be expressed as:

[0070]

[0071] where fx , f y , c x , c y are intrinsic parameters of the camera, where, dxand dyrepresent the physical size of a single pixel in x and y direction, respectively, (c x , c y ) is the center pixel coordinate of the image plane.

[0072] The extrinsic matrix T l→c of the laser-radar-to-camera can be represented as:

[0073]

[0074] where the extrinsic matrix T l→c of the laser-radar-to-camera includes a rotation matrix and a translation vector

[0075] Step 3.2), project the point cloud voxel center points into a two-dimensional image according to the coordinate transformation matrix, to obtain the two-dimensional pixel coordinates of each point cloud in the two-dimensional image.

[0076] According to the spatial projection transformation matrix (i.e. the coordinate transformation matrix IT l→c ), project the point cloud into the image to obtain the pixel coordinates of each voxel center point, the formula is as follows:

[0077]

[0078] where (P x , P y , P z ) is the three-dimensional position coordinate of the point cloud under the laser-radar coordinate system, (P u , P v ) is the two-dimensional pixel coordinate of the point cloud projected onto the image plane, is the depth value of the point cloud under the camera coordinate system, I is the camera intrinsic matrix, T l→c is the extrinsic matrix of the laser-radar-to-camera.

[0079] Step 3.3), use the bilinear interpolation method to obtain the image features at the corresponding pixel coordinates.

[0080] Given the feature values f1, f2, f3, f4 at four points (x1, y1), (x2, y1), (x1, y2), (x2, y2) near the projection point (P u , P v ), by bilinear interpolation, the feature value at the projection point (P u , P v ) is:

[0081]

[0082] Step 3.4), a linear layer is applied to the image features at the projection point to generate a query matrix, and an offset Δk and an attention weight matrix A are obtained after the query matrix and a fully connected layer, respectively, the image features at the projection point plus the offset are obtained using bilinear interpolation, the weight matrix A is applied, and the image features F retrieved near the projection point are output c .

[0083]

[0084] where Q is a query matrix, W and W' are learnable weights, A is an attention weight matrix, R is a projection point, Δmqk is an offset, k is the number of sampling points, M is the number of heads, and F is an image feature map. i m m mqk i

[0085] Step 4, the point cloud voxel features and the corresponding image features are added to obtain the fused multi-modal features:

[0086] F' = F p +F c (6)

[0087] where F is a point cloud voxel feature, and F is the image feature. p c

[0088] Step 5, the multi-modal features are sent into a traditional point cloud three-dimensional target detection backbone network, and feature weight maps are introduced to aggregate high-dimensional features while down-sampling to obtain multiple BEV features of different scales, which specifically include:

[0089] Step 5.1, multiple sparse three-dimensional convolution blocks are used to down-sample the multi-modal features to obtain three-dimensional feature maps of different scales with different numbers of channels

[0090] Step 5.2, as shown in Figure 4 , the three-dimensional feature maps of different scales are applied to generate weight maps of different height dimensions using a multi-layer perceptron, and after normalization by a Softmax function, the high-dimensional features are weighted and summed to obtain multiple BEV feature maps of different scales

[0091] As an embodiment, a sparse three-dimensional convolution block is composed of a three-dimensional sparse convolution, a regularization function, and an activation function, H = 1600, W = 1200, D = 41, and C = 16.

[0092] ​​​​​​​Step 6, the BEV features of different scales are fused into a specific scale using convolution and addition operation to obtain the fused BEV feature.

[0093] As shown in Figure 5 , the BEV feature x with a size of bev1 is down-sampled to a BEV feature x′ with a size of bev1 using 2D convolution, then the BEV features x′ bev1 and x bev2 with the same size are added, and 2D convolution is applied for down-sampling to obtain a BEV feature x′ with a size of bev2 , which is added with a BEV feature x bev3 with the same size to obtain a BEV feature x′ bev3 with a size of , which is then spliced with a BEV feature x bev4 obtained by down-sampling the original feature map by 8 times to obtain a fused BEV feature x bev with a size of

[0094] Step 7, the fused BEV feature is sent into an RPN network to generate a high-resolution feature map.

[0095] The fused BEV feature is extracted and integrated using an RPN network, the input of the RPN network is the BEV feature map obtained in step 6 with a size of , and the output is a high-resolution feature map with a size of (180, 180, 256).

[0096] As a preferred embodiment, the RPN network adopted contains 2 convolution feature extraction modules and 2 up-sampling modules, the 2 convolution feature extraction modules are in series to gradually extract the input feature, and the 2 up-sampling modules are similar to Decoder to restore the feature to a higher resolution.

[0097] Step 8, a three-dimensional target detection box is generated according to the high-resolution feature map to obtain the specific position and category of the target.

[0098] After the high-resolution feature map is extracted using the RPN structure, a CenterHead detection head is used to predict a heat map representing the position and category of the target center point, the size of the target, the offset of the target center point, the size and orientation of the target, and finally regress the detection parameters.

[0099] The three-dimensional target detection box is generated according to the high-resolution feature map, specifically including:

[0100] Step 8.1: Divide the 10 types of targets annotated in the Nuscenes dataset into six categories (represented as Head_0 to Head_5) based on their sizes, as shown in Table 1 below. Then, use six detection heads to detect each of the six categories, so that each detection head can focus on detecting targets of a specific size.

[0101] Table 1 Classification of target detection heads

[0102]

[0103] In step 8.2, use six CenterHeads to obtain detection results for six major categories from the high-resolution feature map. Each major category detection result includes a heatmap of the target center point location and type (180, 180, C), the target size (180, 180, 3), the target center point offset (180, 180, 2), the target size (180, 180, 3), and the orientation (180, 180, 2). Each major category detection result is added to the detection result list to form the final detection result.

[0104] Step 8.3: Integrate the category and position parameters of the target bounding box as the three-dimensional target detection result based on the detection results of the six major categories.

[0105] Finally, the detection results of the above six categories are integrated to obtain the category and position parameters (x, y, z, w, l, h, θ) of the target bounding box, where (x, y, z) are the three-dimensional coordinates of the target center point, (w, l, h) are the length, width and height of the target, and θ is the yaw angle (heading angle) of the target.

[0106] The present invention proposes a method for 3D object detection based on image point clouds and adaptive feature fusion. The method can be considered as a 3D object detection model based on image point clouds and adaptive feature fusion. The loss function is defined as the sum of the classification loss and the regression loss:

[0107] L all = λ1L cls +λ2L reg (7)

[0108] Among them, L cls is the classification loss, L reg is the regression loss, λ1 and λ2 represent the balance coefficients of the two types of losses. cls Acting on the heat map of the predicted output, Focal Loss is used to address the imbalance problem of positive and negative samples.

[0109] The following experiments are used to verify the technical effect of the method of the present invention.

[0110] This experiment used the public nuScense dataset. The nuScense dataset was collected using two electric vehicles with identical sensor configurations. These vehicles were equipped with six cameras and a 32-beam LiDAR, providing 360° field of view. The data was collected in Boston and Singapore. The data covers urban, residential, suburban, and industrial areas, and encompasses different weather conditions, including daytime, nighttime, sunny, rainy, and cloudy. The proposed method addresses multi-sensor spatiotemporal registration by searching for the six camera data points with the nearest timestamps for each point cloud frame to form a data set. For spatial registration, the calibrated registration parameters of the dataset were used. To account for the fact that point clouds and images are not strictly aligned in time, ego-vehicle motion compensation was applied to the different sensor timestamps. This resulted in a projection transformation matrix from the point cloud to the six image frames in each data set. Furthermore, to account for the sparsity of the point cloud, ego-vehicle motion compensation was used to integrate 10 frames. Based on these operations, a total of 28,130 frames of training data and 6,019 frames of testing data were generated.

[0111] The model training of the present invention uses the adam optimizer, the initial learning rate is set to 0.0001, two RTX3090s are used for training, the batch size of each card is set to 2, and a total of 20 epochs are trained, and the model converges well.

[0112] In terms of model evaluation, the mAP and NDS indicators proposed by the nuScense dataset are used for evaluation. To fully verify the excellent performance of the method of the present invention, it is compared with the baseline algorithm and other representative detection models (3DCVF, PointPainting, Centerpoint, MVP). The experimental comparison results are shown in Table 2.

[0113] Table 2 Performance evaluation of the model of the present invention and existing models in the field

[0114] Method / model name mAP (%) NDS (%) PointPainting 46.4 58.1 3DCVF 52.7 62.3 Centerpoint 57.9 65.6 MVP 62.8 66.2 Invention method (model) 64.1 69.3

[0115] Table 2 compares the experimental results of the proposed model with other representative models in the field, using mAP and NDS as the two measurement metrics. As can be seen from the data in Table 2, the proposed model achieves higher mAP and NDS than other representative models in the field, demonstrating its feasibility and effectiveness.

[0116] The method model provided by the application introduces a deformable cross-attention mechanism in the image and point cloud voxel feature fusion process, adaptively aggregates image features near the center of the point cloud voxel, simultaneously, constructs a three-dimensional feature space distribution weight map, adaptively fuses high-dimensional features, reduces the information loss existing in the three-dimensional feature dimension reduction, introduces an additional multi-scale BEV feature fusion branch, improves the perception ability of the network model to geometric information, and realizes better detection effect.

[0117] The preferred embodiments of the application disclosed above are only used to help explain the application. The preferred embodiments do not describe all the details and limit the application to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of the specification. The specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the application, so that those skilled in the art can well understand and use the application. The application is limited only by the claims and their full scope and equivalents.

Claims

1. A method for three-dimensional object detection from image point clouds based on refined feature extraction, characterized in that: The following steps are involved: Obtaining two-dimensional images captured by multiple surround-view cameras and performing feature extraction on the two-dimensional images to obtain multiple two-dimensional image feature maps; Obtain a point cloud image and perform feature encoding on the point cloud image using a voxel encoding network to obtain point cloud voxel features; According to the spatial projection relationship of the point cloud image, a deformable cross-attention mechanism is used to extract the center point of the point cloud voxel and project it to the two-dimensional image features near the corresponding point after the image is projected; The point cloud voxel features are added to the corresponding two-dimensional image features to obtain fused multimodal features, specifically including: According to the calibrated lidar extrinsic parameters and camera extrinsic parameters, the coordinate transformation matrix of the point cloud-image is obtained; According to the coordinate transformation matrix, the point cloud coordinates of the voxel center point are projected to the corresponding image to obtain the coordinate position of the corresponding image, and the image features of the corresponding coordinate position are obtained using bilinear interpolation; Generate an attention matrix and an offset according to the image features, obtain image features at the image offset position using bilinear interpolation, and generate an image feature matrix; Apply the attention matrix to the image feature matrix and pass it through a fully connected layer to obtain weighted aggregated image features; Adding the point cloud voxel features to the corresponding image features to obtain fused multimodal features; The multimodal features are fed into the point cloud 3D object detection backbone network. While downsampling, a feature weight map is introduced to aggregate the height dimension features to obtain BEV features of multiple different scales, including: According to the multimodal features, a multi-layer perceptron is used to generate feature weight maps of different height dimensions. After normalization, the feature weight map is applied to the height dimensional features, and adaptive weighting is performed to obtain the reduced-dimensionality BEV features. The dimensionality reduction method is applied to 3D voxel features of multiple different scales to obtain BEV features of multiple different scales. The convolution and addition operations are used to fuse the BEV features of the multiple different scales to a specific scale to obtain the fused BEV features, which specifically include: Use 2D convolution to downsample the original maximum-scale BEV feature by 2 times, then add it to the BEV feature of the same scale, repeat the 2D convolution downsampling operation, add it to the BEV feature of the same size, and finally concatenate it with the 8-fold downsampled original feature map to obtain the fused BEV feature; The fused BEV features are fed into the RPN network to generate high-resolution feature maps, including: The RPN network is used to extract and integrate the fused BEV features. The RPN network used contains two convolutional feature extraction modules and two upsampling modules. Two convolutional feature extraction modules gradually extract input features. The two upsampling modules are decoders that restore features to a higher resolution. Generate a three-dimensional target detection frame based on the high-resolution feature map to obtain the specific location and category of the target.

2. The method for three-dimensional object detection based on image point cloud and refined feature extraction according to claim 1, characterized in that: The feature extraction of the two-dimensional image to obtain a plurality of two-dimensional image feature maps specifically includes: The image features of the two-dimensional image are extracted using the ResNet50 backbone network of the deeplabv3 detector pre-trained on the COCO dataset.

3. The method for three-dimensional object detection based on image point cloud and refined feature extraction according to claim 1, characterized in that: The voxel encoding network is used to obtain the point cloud voxel features, including: The space where the point cloud is located is divided into voxels of the same size, and the point cloud data is assigned to the corresponding voxels according to its coordinates. According to the sparsity of the point cloud, the voxels are divided into empty voxels that do not contain point clouds and voxels that contain point clouds. Sparse three-dimensional convolution is used to extract voxel features from voxels that contain point clouds.

4. The method for three-dimensional object detection based on image point cloud and refined feature extraction according to claim 1, characterized in that: The point cloud is divided into multiple voxels, and the voxels are encoded and extracted to obtain the features of each voxel point cloud, including: The entire three-dimensional space occupied by the point cloud is divided into multiple voxels, and spatial classification is performed according to the three-dimensional coordinates of the point cloud, and the point cloud is classified into the corresponding voxels. Filter the point cloud data and divide the entire space into multiple voxels in a three-dimensional perspective. The resolution of the division is , where D, W, and H represent the depth, width, and height of the divided space, respectively; the point cloud is classified into voxels according to the three-dimensional coordinates, and the point cloud is classified into the corresponding voxels, and only the non-empty voxels with a point cloud point count greater than zero are retained; For voxels with more than T points in the voxels, T point clouds are randomly screened out, and voxels with less than T points are filled with zeros to achieve structural processing of irregular point clouds; for voxels with more than T points, T points are randomly screened out, and voxels with less than T points are filled with zeros; Aggregate all points within the voxel, use the point cloud coordinate mean to represent the initial voxel features, and use sparse 3D convolution blocks to extract voxel features , where N represents the number of valid point cloud points in the divided voxel, and C represents the number of feature channels.

5. The method for three-dimensional object detection based on image point cloud and refined feature extraction according to any one of claims 1 to 4, characterized in that: Generate a 3D target detection frame based on the high-resolution feature map to obtain the specific location and category of the target, including: Use six CenterHead detection heads to obtain detection results of six major categories from high-resolution feature maps; According to the detection results of the six major categories, the category and position parameters of the target detection frame are integrated as the three-dimensional target detection result.

6. The method for three-dimensional object detection based on image point cloud and refined feature extraction according to claim 5, characterized in that: The detection results for each major category include a heat map representing the location and type of the target center point, the horizontal offset of the target center point, the height of the target center point, and the size and orientation of the target.

Images