A point cloud target detection method based on multimodal fusion network
By multi-modal fusion of point clouds and pictures, and using the new neural network architecture to extract features, the problem of low point cloud detection accuracy in the existing technology is solved, the accuracy and applicability of three-dimensional target detection is improved, and it is suitable for autonomous driving. and other fields.
Patent Information
- Application Number
- CN202411331451.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-24
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-09-24
AI Technical Summary
The existing three-dimensional object detection technology lacks color information when utilizing point cloud data, resulting in low detection accuracy, and it is difficult to extract unique features from irregular point clouds, making it difficult to migrate to three-dimensional object detection.
Design a point cloud object detection method based on multimodal fusion network. By fusing point cloud data with corresponding two-dimensional pictures, a new neural network architecture is adopted, including data augmentation, voxelization, feature expansion, and convolution Neural network and residual sparse convolutional network extract the features of three-dimensional point clouds and pictures, and optimize the detection results through loss functions.
The accuracy of three-dimensional object detection is improved, especially on KITTI datasets and small targets, and the universality and applicability of the network is enhanced, and suitable for autonomous driving and other 3D visual perception tasks.
Smart Images

Figure CN118864824B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of three-dimensional target detection, and specifically relates to a point cloud target detection method based on a multimodal fusion network. Background Art
[0002] With the rapid development of technologies such as autonomous driving and remote sensing, the application of three-dimensional data such as point clouds has become more extensive. Point clouds are a kind of point set based on three-dimensional space to describe the approximate shape and characteristics of objects. They can provide more information for environmental perception, such as depth and position, and can still depict the approximate shape of objects without being affected by weather factors such as light. The corresponding disadvantages are their sparsity, large number, and irregularity. Pictures can provide rich color information, and the price of obtaining pictures is relatively low compared to obtaining three-dimensional data, but there is no information such as depth, which makes it inconvenient to detect problems such as occlusion in the real world. Therefore, the combination of three-dimensional data and pictures combines the advantages of both to make up for their respective shortcomings and improve their detection accuracy. Therefore, it is of practical significance to design a point cloud target detection network based on multimodal fusion using convolutional neural networks.
[0003] Existing neural networks have taken point clouds as their input data, overcoming the problem of large point cloud data volume, but there are still some problems. If traditional neural networks, such as PointPillars and SECOND networks, are used, irregular point clouds are converted into regular voxels and input into the intermediate network layer to obtain detection results. Since only point cloud data is used, color and other information are missing, and the point cloud itself has the characteristics of near-dense and far-sparse, the detection accuracy is not high. Although the detection technology of two-dimensional images is becoming more and more mature and the detection accuracy is very high, it is difficult to apply to three-dimensional objects and difficult to migrate. How to extract unique features from irregular points and extract information from images to improve network detection accuracy has become a key issue in the task of three-dimensional target detection. Therefore, based on multimodal networks, the advantages of both are integrated, and a new neural network architecture is designed to solve the above problems and improve detection accuracy, which is of great significance in the field of point cloud target detection. Summary of the invention
[0004] In view of the above problems, the present invention proposes a point cloud target detection method based on a multimodal fusion network. By fusing two types of data, point cloud and corresponding image, a new neural network architecture is designed to solve the above problems and improve detection accuracy.
[0005] The technical solution of the present invention:
[0006] A point cloud target detection method based on a multimodal fusion network, the steps are as follows:
[0007] Step 1: Obtain the 3D point cloud data and corresponding 2D image of the target to be detected.
[0008] Step 2: Perform data enhancement on the 3D point cloud data obtained in step 1 to remove noise.
[0009] Furthermore, data enhancement methods include random rotation RandomFlip3D, scaling GlobalRotScaleTrans, filtering point cloud PointsRangeFilter and random permutation PointShuffle method.
[0010] Step 3: Convert the data-enhanced 3D point cloud data into a regular voxel form, and then expand the 3D point cloud features in the 3D point cloud data, expand the voxel features into 3D point cloud features, and finally convert the number of channels of the 3D point cloud features to 128 channels. At this time, the 3D point cloud features are recorded as point_features.
[0011] Step 3.1: Preprocess and crop the 3D point cloud data after data enhancement according to its position information, retain only the 3D point cloud data within the camera range, and discard the 3D point cloud data outside the camera range. Then uniformly encode the preprocessed and cropped 3D point cloud data, and encode the encoded 3D point cloud data along the axis, Axis and The axes are then uniformly divided into different voxels.
[0012] Furthermore, the specific method of the pre-processing and cropping is: transforming and mapping the three-dimensional point cloud coordinates in the three-dimensional point cloud data into image coordinates, and obtaining the image position corresponding to each three-dimensional point cloud, which is defined as follows:
[0013]
[0014] in, is the combined information of the camera’s internal and external parameters, is the camera standard data, is the pose change matrix from the laser radar to the camera coordinate system, and the matrix information of the pose change matrix is used to convert the 3D point cloud coordinates into image coordinates. is the 3D point cloud coordinate. is the image position corresponding to each 3D point cloud.
[0015] Then, the image boundary information is obtained using the image position corresponding to each 3D point cloud, and the 3D point cloud mask is generated using the image boundary information. The 3D point cloud mask projected within the 3D point cloud range is set to 1, and the 3D point cloud mask at other positions is set to 0. Finally, the entire 3D point cloud data is masked to obtain the pre-processed and cropped 3D point cloud data.
[0016] Step 3.2: The original features of the 3D point cloud obtained after preprocessing and cropping are recorded as ( , , , ), which represent the coordinates and reflection intensity of the original 3D point cloud respectively.
[0017] Step 3.3: Express the coordinates of the center point within the voxel as ( , , ), the difference between the coordinates of the original 3D point cloud and the coordinates of the center point in the voxel is recorded as , , , the calculation method is , , .
[0018] Step 3.4: Calculate the distance from the original 3D point cloud to the center point in the voxel as , the formula is , the difference between the coordinates and distance of the original 3D point cloud is recorded as , , , the calculation method is , , .
[0019] Step 3.5: Upgrade each 3D point cloud in the voxel to a 10-dimensional feature in the following order: , , , , , , , ), the dimension-enhanced features are encoded as vectors ( ),in, Expressed as the maximum number of non-empty voxels. Indicates the number of point clouds contained in a voxel, Represented as the final expanded feature dimension; the obtained vector is subjected to the VFE operation, where the VFE operation includes a fully connected layer, BatchNormalization, and ReLU expansion to 64 dimensions, and finally the convolution layer is used to convert the number of channels of the three-dimensional point cloud feature to 128 channels. At this time, the three-dimensional point cloud feature is recorded as point_features.
[0020] Step 4: The features of the two-dimensional image obtained in step 1 are preprocessed first, and then extracted again through a two-dimensional extraction network. The two-dimensional extraction network adopts a hierarchical construction method in a convolutional neural network to downsample the image by 4 times, 8 times, 16 times, and 32 times, respectively, to obtain the corresponding four image features, and the number of channels is correspondingly changed to 256, 512, 1024, and 2048. The four obtained image features are passed through the Neck layer, and the number of channels is uniformly changed to 256. The four obtained image features and one of the image features with 256 channels are taken as a whole, and the number of channels is changed to 128 through a convolution kernel of size 3, and uniformly converted to a feature layer with 640 channels through a convolution layer. The 640-channel feature layer is again converted to a 128-channel image feature through a linear layer, recorded as pic_features.
[0021] Furthermore, the pre-processing in step 4 is specifically as follows:
[0022] First, the pixel values of the RGB channels of the two-dimensional image are normalized, the normalized pixel values are mean-shifted, the corresponding channel offset values are subtracted from the corresponding channels, and finally the image data is scaled by standard deviation.
[0023] Step 5: After the dimensions of the image feature pic_features obtained in step 4 and the three-dimensional point cloud feature point_features obtained in step 3 have been matched, the two features are directly added together for fusion without changing the dimensions. The fused three-dimensional point cloud features are mapped to voxels again through the activation function ReLU.
[0024] Step 6: Use the residual sparse convolution network structure composed of 3D sparse convolution, submanifold sparse convolution, BatchNormalization and ReLU to extract the fused 3D point cloud features in step 5, and gradually convert the 3D point cloud features into 3D sparse features with downsampling sizes of 2, 4, and 8 times.
[0025] Step 7: The three-dimensional sparse features obtained in step 6 are transmitted through the ToDense layer. The axes are stacked to obtain the corresponding top view of the BEV.
[0026] Step 8: The BEV top view obtained in step 7 is used as the input of two sets of 2D convolutional networks. After transposed convolution, the two sets of features obtained by the two sets of 2D convolutional networks are finally spliced and fused to obtain the two-dimensional fused network features.
[0027] Furthermore, each group of the 2D convolutional networks includes 5 2D convolutional layers and 1 2D transposed convolution, and each 2D convolutional layer and 2D transposed convolution is followed by BatchNormalization and ReLU. The semantic features output by the 2D convolutional layer of a group of 2D convolutional networks are , the spatial features output by the two-dimensional convolution layer of another set of 2D convolutional networks are ,in, is the width of the input feature map, is the height of the input feature map, is the channel of the input feature map. After two-dimensional transposed convolution, the features become .
[0028] Step 9: The multi-detection head network uses three 1 1 The two-dimensional convolution kernel acts on the network feature map obtained in step 8 to obtain the number of feature channels corresponding to different categories. The number of feature channels corresponding to different categories are category classification, position regression and direction classification.
[0029] Step 10: The design of the loss function includes the category classification loss function and the position regression loss function. The category classification loss function is used to measure the accuracy of the object detection model established in steps 1-9 in classifying different categories. The position regression loss function is further divided into the angle regression loss function and the positioning regression loss.
[0030] Furthermore, the loss function is as follows:
[0031] Step 10.1: Angular regression loss function The SmoothL1 loss function is used and is defined as follows:
[0032]
[0033] in, Subscript represents the predicted value, Subscript represents the true value, represents the center coordinates of the bounding box, Represents the length, width, and height of the bounding box.
[0034] Step 10.2: The positioning regression loss function adopts the cross entropy regression method and is defined as follows:
[0035]
[0036] in, represents the predicted value, Represents the true value.
[0037] Step 10.3: The category classification loss function uses the FocalLoss function, which is defined as follows:
[0038]
[0039] in, Indicates the probability that the sample belongs to the true category, and is a hyperparameter of the focal loss function.
[0040] Beneficial effects of the present invention:
[0041] 1. Compared with the existing network architecture, the architecture of the present invention improves the detection accuracy of the KITTI dataset, and also improves the detection accuracy of small targets such as pedestrians. It can be applied not only to the KITTI dataset, but also to existing larger public large-scale autonomous driving datasets, such as the nuSences dataset. Dataset.
[0042] 2. The present invention takes the three-dimensional point cloud as input and performs data enhancement technology on the input data. It is also highly universal to other networks and is not only applicable to the network of the present invention, but also to other 3D visual perception algorithms, such as BEV detection and three-dimensional object segmentation tasks.
[0043] 3. The present invention improves the point cloud grouping method, combines the voxel structured features with the point cloud features, expands the features of the original data, provides more effective information before inputting the network architecture, and expresses each point cloud information more reasonably.
[0044] 4. The network architecture proposed in the present invention combines and merges the features of point cloud and image. The advantages of point cloud data providing depth information and image data providing rich texture information can be effectively merged, thereby improving the detection accuracy of three-dimensional objects. It can be applied to related fields such as autonomous driving and underwater robots. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 It is a schematic diagram of the detection architecture of point cloud and image based on multimodal fusion of the present invention;
[0046] Figure 2 The point cloud data is cropped and visualized;
[0047] Figure 3 It is a residual network consisting of a 3D sparse convolution and two sub-manifold sparse convolutions;
[0048] Figure 4 It is a schematic diagram of a two-dimensional feature map passing through a backbone network;
[0049] Figure 5 Qualitative results on the KITTI validation set are shown; DETAILED DESCRIPTION
[0050] In order to make the technical solution of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0051] This embodiment takes the KITTI dataset as an example, which contains 7481 training samples and 7518 test samples. The training samples are divided into a training set and a validation set. Three classes, namely Car, Cyclist and Pedestrian, are evaluated. The KITTI dataset divides the model according to the height, occlusion and truncation of the bounding box in the image plane, and the evaluation difficulty is divided into simple, medium and difficult. The model is trained on the model using 80% of the training sample data, and the remaining 20% of the data is used for verification. The present invention uses 11 recall positions to calculate the average mean precision, where the rotation IoU threshold of the Car class is 0.7, the rotation IoU threshold of the Cyclist class is 0.5, and the rotation IoU threshold of the Pedestrian class is selected as 0.5. Among them, the network training of the present invention is set to a batch size of 4, the training framework is PyTorch 2.0.0, the system environment is Ubuntu 22.04, the processor is Intel Xeon Platinum 8260M@ 2.40GHz, the acceleration card is NVIDIA GeForce RTX 3090, the setting is 80 epoches, the Adam optimizer is used, the initial learning rate is set to 0.004, the exponential decay factor is 0.8, and the decay weight of 0.01 is used. is 0.9, The schematic diagram of the detection architecture of the point cloud and image based on multimodal fusion of the present invention is as follows: Figure 1 shown.
[0052] Step 1: Obtain the 3D point cloud data and the corresponding 2D image of the target to be detected. Here, the present invention uses the KITTI dataset. To create the KITTI 3D point cloud data, it is necessary to generate data annotations containing target labels and bounding boxes from the original data. At the same time, separate training target data is generated for the 3D point cloud data in the KITTI dataset. And store it in a .bin format file, and generate a file containing data information for the training data, such as coordinate position information, rotation angle, etc.
[0053] Step 2: Perform a series of data enhancement operations on the 3D point cloud data obtained in step 1 to remove noise, which is used to improve the generalization ability of the target detection model and reduce the cost of data collection and labeling. For example, adding the random rotation RandomFlip3D method is to rotate the 3D point cloud data within the reference value along the The axis direction is randomly rotated according to a uniform distribution of 0.5 to obtain the azimuth change. The scaling GlobalRotScaleTrans method is added to perform global scaling using a random scaling factor uniformly distributed in [0.95,1.05] within the reference value. These reference values are randomly sampled into the 3D point cloud data of the original sample to simulate different scenarios. This type of method can effectively enhance the robustness of the model. The PointsRangeFilter method is used to filter the 3D point cloud data within the specified range to filter out the noise. The PointShuffle method is used to randomly shuffle the 3D point cloud data within a certain range to ensure that it does not affect its overall detection accuracy.
[0054] Step 3: Convert the data-enhanced 3D point cloud data into a regular voxel form, and then expand the 3D point cloud features in the 3D point cloud data, expand the voxel features into 3D point cloud features, and finally convert the number of channels of the 3D point cloud features to 128 channels. At this time, the 3D point cloud features are recorded as point_features.
[0055] Step 3.1: Since the data collected by the LiDAR is a 360° surround scene data, the annotation file only provides the target label information of the forward view. However, the surround scene also contains targets to be detected, but the lack of label data will interfere with the training of the detection network. Therefore, the data-enhanced 3D point cloud data is pre-processed and cropped according to its position information, and only the point cloud information under the forward view is retained, such as Figure 2 As shown. The specific method is: transform and map the 3D point cloud coordinates in the 3D point cloud data to the image coordinates, and obtain the image position corresponding to each 3D point cloud. The definition is as follows:
[0056]
[0057] in, is the combined information of the camera’s internal and external parameters, is the camera standard data, is the pose change matrix from the laser radar to the camera coordinate system, and the matrix information of the pose change matrix is used to convert the 3D point cloud coordinates into image coordinates. is the 3D point cloud coordinate. is the image position corresponding to each 3D point cloud.
[0058] Then, the image boundary information is obtained using the image position corresponding to each 3D point cloud, and the 3D point cloud mask is generated using the image boundary information. The 3D point cloud mask projected within the 3D point cloud range is set to 1, and the 3D point cloud mask at other positions is set to 0. Finally, the entire 3D point cloud data is masked to obtain the pre-processed and cropped 3D point cloud data.
[0059] Step 3.2: The original features of the 3D point cloud obtained after preprocessing and cropping are recorded as ( , , , ), respectively representing the coordinates and reflection intensity of the original 3D point cloud. The 3D point cloud data is then cropped and divided so that all categories of detection tasks are in the same target detection model. The specific approach is to The axis range is 70.4 meters to 0, The axis range is 40m to -40m, The axis range is 1 meter to -3 meters. The size of each voxel is set to 0.05 meters on the x-axis, 0.05 meters on the y-axis, and 0.1 meters on the z-axis. Therefore, the grid size of the generated voxels is 1408*1600*40. The maximum number of points in each voxel is set to 5.
[0060] Step 3.3: Express the coordinates of the center point within the voxel as ( , , ), the difference between the coordinates of the original 3D point cloud and the coordinates of the center point in the voxel is recorded as , , , the calculation method is , , .
[0061] Step 3.4: Calculate the distance from the original 3D point cloud to the center point in the voxel as , the formula is , the difference between the coordinates and distance of the original 3D point cloud is recorded as , , , the calculation method is , , .
[0062] Step 3.5: Upgrade each 3D point cloud in the voxel to a 10-dimensional feature in the following order: , , , , , , , ), the dimension-enhanced features are encoded as vectors ( ),in, Set to 16000, which represents the maximum number of non-empty voxels. Set to 5, the number of point clouds contained in a voxel, It is represented as the final expanded feature dimension, which is set to 10. The vectors are uniformly subjected to VFE operations, where the VFE operation includes fully connected layers, BatchNormalization and ReLU expansion to 64 dimensions, and finally the convolution layer is used to convert the number of channels of the three-dimensional point cloud features to 128 channels. At this time, the three-dimensional point cloud features are recorded as point_features.
[0063] Step 4.1: The features of the two-dimensional image obtained in step 1 are also preprocessed accordingly. First, the pixel values of the three RGB channels of the image are normalized, that is, each pixel value is divided by 255 and the pixel value is in the range of [0,1]. The normalized pixel values are mean shifted, and the corresponding channel offset values [0.485, 0.456, 0.406] are subtracted from the corresponding channels. Finally, the image data is scaled by standard deviation. The preprocessed image is then passed through a two-dimensional extraction network for feature extraction. The two-dimensional extraction network adopts a hierarchical construction method in a convolutional neural network to downsample the image by 4 times, 8 times, 16 times, and 32 times, respectively, to obtain the corresponding four image features, and the number of channels is correspondingly changed to 256, 512, 1024, and 2048. The four obtained image features are passed through the Neck layer, and the number of channels is uniformly changed to 256.
[0064] Step 4.2: Take the four features obtained and an image feature with 256 channels as a whole, and use a convolution kernel of size 3 to change the number of channels to 128, and then convert them into a feature layer with 640 channels through a convolution layer.
[0065] Step 4.3: The 640-channel feature layer is converted into 128-channel image features through the linear layer again, recorded as pic_features, waiting to be fused with the 3D point cloud features point_features.
[0066] Step 5: After the dimensions of the image feature pic_features and the three-dimensional point cloud feature point_features have been matched, the two features are directly added together for fusion without changing the dimensions. The fused three-dimensional point cloud features are mapped to voxels again through the activation function ReLU.
[0067] Step 6: Use 3D sparse convolution, submanifold sparse convolution, BatchNormalization and ReLU to form a residual sparse convolution network structure to extract the fused 3D point cloud features in step 5, and gradually convert the 3D point cloud features into 3D sparse features with 2, 4 and 8 times the downsampling size, such as Figure 3As shown in the figure, a 3D sparse convolution and two sub-manifold sparse convolutions form a residual network structure. The purpose is to extract the fused features. Due to the addition of the residual structure, the network becomes deeper and the convergence speed of the network is accelerated.
[0068] Table 1 Residual sparse convolutional network structure
[0069]
[0070] Step 7: The three-dimensional sparse features obtained in step 6 are transmitted through the ToDense layer. The axes are stacked to obtain the corresponding BEV top view, which is converted into a 2D feature map.
[0071] Step 8: The top view of the BEV obtained in step 7 is used as the input of two sets of 2D convolutional networks, where each set of 2D convolutional networks contains 5 2D convolutional layers and 1 2D transposed convolution. Each 2D convolutional layer and transposed convolution is followed by BatchNormalization and ReLU. The semantic features output by a set of 2D convolutional networks (CNN backbone network) are , the spatial features output by the two-dimensional convolution layer of another set of 2D convolutional networks are ,in, is the width of the input feature map, is the height of the input feature map, is the channel of the input feature map. After two-dimensional transposed convolution, the features become ,Finally, the two sets of features obtained by the two sets of 2D convolutional networks are spliced and fused to obtain the two-dimensional fused network features, such as Figure 4 shown.
[0072] Step 9: After obtaining the two-dimensional fused network features, three 1 1. The two-dimensional convolution kernel acts on the obtained network feature map to obtain the number of feature channels corresponding to different categories, namely category classification, position regression and direction classification. And use the detection tasks, such as pedestrians, bicycles, and cars to give different sizes of bounding boxes, where the threshold of the intersection over union (IoU) of the positive samples of pedestrians and bicycles is 0.35, the threshold of the intersection over union (IoU) of the positive samples of cars is 0.6, the threshold of the intersection over union (IoU) of the negative samples of pedestrians and bicycles is 0.2, and the threshold of the intersection over union (IoU) of the negative samples of cars is 0.45.
[0073] Step 10: The design of the loss function includes the category classification loss function and the position regression loss function. The category classification loss function is used to measure the accuracy of the object detection model established in steps 1-9 in classifying different categories. Among them, the position regression loss function is further divided into the positioning regression loss function and the angle regression loss.
[0074] Step 10.1: Angular regression loss function The SmoothL1Loss loss function is used, which is defined as follows:
[0075]
[0076] in, Subscript represents the predicted value, Subscript represents the true value, represents the center coordinates of the bounding box, Represents the length, width, and height of the bounding box.
[0077] Step 10.2: The positioning regression loss function uses the cross entropy regression method. The cross entropy is used to determine the degree of closeness between the actual output and the expected output, and it describes the distance between the actual output and the expected output. The smaller the cross entropy value, the closer the two outputs are. The definition is as follows:
[0078]
[0079] in, represents the predicted value, Represents the true value.
[0080] Step 10.3: The category classification loss function uses the FocalLoss function to solve the problem of extremely unbalanced number of positive and negative samples in target detection. Since in the dataset, for example, the 3D point cloud data in the KITTI scene has only a few true bounding boxes, and there are 60,000-80,000 unmatched bounding boxes (negative samples), negative samples will drown out a small number of samples that are helpful for training. The definition is as follows:
[0081]
[0082] in, Indicates the probability that the sample belongs to the true category, and is the hyperparameter of FocalLoss. After training, the hyperparameter is set to Select 0.25, Select 2.0.
[0083] Figure 5The qualitative results of the KITTI validation set are shown. From the experimental results, it can be seen that the improved network can effectively mark the objects to be detected within the target range.
Claims
1. A point cloud target detection method based on a multimodal fusion network, characterized in that: The steps are as follows: Step 1: Obtain the 3D point cloud data and corresponding 2D images of the target to be detected; Step 2: Perform data enhancement on the 3D point cloud data obtained in step 1 to remove noise; Step 3: Convert the data-enhanced 3D point cloud data into a regular voxel form, and then expand the 3D point cloud features in the 3D point cloud data, expand the voxel features into 3D point cloud features, and finally convert the number of channels of the 3D point cloud features into 128 channels. At this time, the 3D point cloud features are recorded as point_features; Step 3.1: Preprocess and crop the data-enhanced 3D point cloud data according to its position information, retain only the 3D point cloud data within the camera range, and discard the 3D point cloud data outside the camera range; then uniformly encode the preprocessed and cropped 3D point cloud data, and encode the encoded 3D point cloud data along the axis, Axis and The axes are then uniformly divided into different voxels; Step 3.2: The original features of the 3D point cloud obtained after preprocessing and cropping are recorded as ( , , , ), respectively representing the coordinates and reflection intensity of the original 3D point cloud; Step 3.3: Express the coordinates of the center point within the voxel as ( , , ), the difference between the coordinates of the original 3D point cloud and the coordinates of the center point in the voxel is recorded as , , , the calculation method is , , ; Step 3.4: Calculate the distance from the original 3D point cloud to the center point in the voxel as , the formula is , the difference between the coordinates and distance of the original 3D point cloud is recorded as , , , the calculation method is , , ; Step 3.5: Upgrade each 3D point cloud in the voxel to a 10-dimensional feature in the following order: , , , , , , , ), the dimension-enhanced features are encoded as vectors ( ),in, It is expressed as the maximum number of non-empty voxels; Indicates the number of point clouds contained in a voxel, Represented as the final expanded feature dimension; the obtained vector is subjected to VFE operation, where the VFE operation includes fully connected layer, BatchNormalization, and ReLU expansion to 64 dimensions, and finally the number of channels of the three-dimensional point cloud feature is converted to 128 channels through the convolution layer. At this time, the three-dimensional point cloud feature is recorded as point_features; Step 4: The features of the two-dimensional image obtained in step 1 are preprocessed first, and then subjected to feature extraction through a two-dimensional extraction network. The two-dimensional extraction network adopts a hierarchical construction method in a convolutional neural network to downsample the image by 4 times, 8 times, 16 times, and 32 times, respectively, to obtain four corresponding image features, and the number of channels is correspondingly changed to 256, 512, 1024, and 2048; the four obtained image features are passed through a Neck layer, and the number of channels is uniformly changed to 256; the four obtained image features and one of the image features with 256 channels are taken as a whole, and the number of channels is changed to 128 through a convolution kernel of size 3, and are uniformly converted into a feature layer of 640 channels through a convolution layer, and the 640-channel feature layer is again converted into a 128-channel image feature through a linear layer, recorded as pic_features; Step 5: After the dimensions of the image feature pic_features obtained in step 4 and the three-dimensional point cloud feature point_features obtained in step 3 have been matched, the two features are directly added together for fusion, and the dimensions remain unchanged. The fused three-dimensional point cloud features are mapped to voxels again through the activation function ReLU; Step 6: Use the residual sparse convolution network structure composed of 3D sparse convolution, submanifold sparse convolution, BatchNormalization and ReLU to extract the fused 3D point cloud features in step 5, and gradually convert the 3D point cloud features into 3D sparse features with downsampling sizes of 2, 4 and 8 times; Step 7: The three-dimensional sparse features obtained in step 6 are transmitted through the ToDense layer. The axes are stacked to obtain the corresponding top view of the BEV; Step 8: The top view of the BEV obtained in step 7 is used as the input of two sets of 2D convolutional networks. After transposed convolution, the two sets of features obtained by the two sets of 2D convolutional networks are finally spliced and fused to obtain the two-dimensional fused network features; Step 9: The multi-detection head network uses three 1 1. The two-dimensional convolution kernel acts on the network feature map obtained in step 8 to obtain the number of feature channels corresponding to different categories. The number of feature channels corresponding to different categories are category classification, position regression and direction classification respectively. Step 10: The design of the loss function includes the category classification loss function and the position regression loss function; the category classification loss function is used to measure the accuracy of the target detection model established in steps 1-9 in classifying different categories; the position regression loss function is further divided into the angle regression loss function and the positioning regression loss.
2. According to claim 1, a point cloud target detection method based on a multimodal fusion network is characterized in that: In the step 2, the data enhancement methods include random rotation RandomFlip3D, scaling GlobalRotScaleTrans, filtering point cloud PointsRangeFilter and random arrangement PointShuffle method.
3. The point cloud target detection method based on a multimodal fusion network according to claim 1 is characterized in that: In the step 3.1, the specific method of the pre-processing and cropping is: transforming and mapping the three-dimensional point cloud coordinates in the three-dimensional point cloud data into image coordinates, and obtaining the image position corresponding to each three-dimensional point cloud, which is defined as follows: in, is the combined information of the camera’s internal and external parameters, is the camera standard data, is the pose change matrix from the laser radar to the camera coordinate system, and the matrix information of the pose change matrix is used to convert the three-dimensional point cloud coordinates into image coordinates; is the three-dimensional point cloud coordinate; The image position corresponding to each 3D point cloud; Then, the image boundary information is obtained using the image position corresponding to each 3D point cloud, and a 3D point cloud mask is generated using the image boundary information; the 3D point cloud mask projected within the 3D point cloud range is set to 1, and the 3D point cloud mask at other positions is set to 0; finally, the entire 3D point cloud data is masked to obtain the pre-processed and cropped 3D point cloud data.
4. The point cloud target detection method based on a multimodal fusion network according to claim 1, characterized in that: The pre-processing in step 4 is specifically as follows: First, the pixel values of the RGB channels of the two-dimensional image are normalized, the normalized pixel values are mean-shifted, the corresponding channel offset values are subtracted from the corresponding channels, and finally the image data is scaled by standard deviation.
5. The point cloud target detection method based on a multimodal fusion network according to claim 1, characterized in that: In step 8, each group of the 2D convolutional networks includes 5 2D convolutional layers and 1 2D transposed convolution, and each 2D convolutional layer and 2D transposed convolution are followed by BatchNormalization and ReLU; the semantic features output by the 2D convolutional layer of a group of 2D convolutional networks are , the spatial features output by the two-dimensional convolution layer of another set of 2D convolutional networks are ,in, is the width of the input feature map, is the height of the input feature map, is the channel of the input feature map; after two-dimensional transposed convolution, the features become .
6. The point cloud target detection method based on a multimodal fusion network according to claim 1 is characterized in that: In step 10, the loss function is as follows: Step 10.1: Angular regression loss function The SmoothL1 loss function is used and is defined as follows: in, Subscript represents the predicted value, Subscript represents the true value, represents the center coordinates of the bounding box, Indicates the length, width, and height of the bounding box; Step 10.2: The positioning regression loss function adopts the cross entropy regression method and is defined as follows: in, represents the predicted value, represents the true value; Step 10.3: The category classification loss function uses the FocalLoss function, which is defined as follows: in, Indicates the probability that the sample belongs to the true category, and is a hyperparameter of the focal loss function.
Citation Information
Patent Citations
Three-dimensional target detection method based on multi-modal fusion and deformable attention
CN117975436A
Three-dimensional target detection method, system and device based on multi-modal feature fusion convolution and medium
CN118552717A
