A point cloud target detection method and device based on three-dimensional intersection-over-union ratio

Abstract

The application relates to a three-dimensional intersection-over-union-based point cloud target detection method and device, which uses a laser radar sensor to collect three-dimensional point cloud data; performs preliminary data processing and data enhancement on the original collected point cloud; inputs the processed point cloud data into a three-dimensional point cloud feature extraction network to obtain deep point cloud feature information; and designs a detection head using a three-dimensional intersection-over-union loss function, wherein the difference obtained by the method is returned through the back propagation of a neural network, and the detection frame is optimized and adjusted, so that the effect of accurately detecting the point cloud target is achieved. Compared with the L1 norm used as a loss function to calculate the loss in the prior art, the three-dimensional intersection-over-union loss function has stronger applicability to three-dimensional point cloud data and can achieve higher accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a point cloud target detection method and device based on three-dimensional intersection-over-union. BACKGROUND

[0002] Target detection, as an important research direction of computer vision, plays a very important role in application scenarios such as automatic driving, three-dimensional modeling, foreign object intrusion detection, and lane line recognition. Laser radar, as a sensor with strong anti-interference ability and capable of working all day long, is widely used in the field of target detection. Therefore, realizing target detection on point cloud data collected by laser radar has become a very popular research and application direction. However, compared with two-dimensional image data detection, the three-dimensional characteristics such as disorder and dispersion of three-dimensional point cloud data make it difficult to improve the accuracy of target detection, which affects the actual landing application, so a target detection method suitable for the three-dimensional characteristics of point cloud is very needed. SUMMARY

[0003] The present application relates to the technical field of computer vision, and in particular to a point cloud target detection method and device based on three-dimensional intersection-over-union.

[0004] To achieve the above-mentioned application purposes, the following technical solutions are adopted: in the first aspect, the present application provides a point cloud target detection method based on three-dimensional intersection-over-union, comprising:

[0005] Collecting three-dimensional point cloud data using a laser radar sensor, processing the collected point cloud data, and obtaining original point cloud data;

[0006] Data preprocessing is performed on the original point cloud data, and data augmentation suitable for multiple scenes is performed to obtain initial point cloud;

[0007] The initial point cloud data is input into a point cloud feature extraction network, and a cylindrical voxelization point cloud sampling module is used to encode the three-dimensional point cloud sampling to extract two-dimensional feature information; a BEVBackBone is used to extract deep features of the feature information;

[0008] The deep feature information is input into a detection head module designed using a three-dimensional intersection-over-union loss function, the deviation between the target frame and the true value frame is calculated, the target frame is regressed back through back propagation, and the target point cloud is framed;

[0009] Visualizing the scene and the framed target point cloud on a computer device.

[0010] Secondly, the present invention provides a point cloud target detection device based on three-dimensional intersection-over-union ratio, which mainly consists of a lidar sensor, a target detection system, and a computer visualization device; the target detection system includes a data processing module, a feature extraction module, and a three-dimensional target detection head module;

[0011] The data processing module sets a range threshold for the point cloud scene based on the point cloud coordinate data, thereby distinguishing discrete points and performing discrete point removal processing; it performs point drop processing on the overall point cloud data, that is, for each frame of point cloud scene, it randomly deletes part of the point cloud data, making the point cloud scene slightly incomplete; and it performs global rotation, global scaling and ceiling enhancement processing on the preprocessed point cloud data.

[0012] The feature extraction module is based on the BEVBackBone network to extract deep point cloud feature information. The BEVBackBone network mainly consists of three groups: the first group contains 3 convolutional layers, and the second and third groups contain 5 convolutional layers. The last layer of each group uses a 3*3 convolutional kernel for downsampling to extract feature information at different scales.

[0013] The 3D target detection head module calculates the forward propagation loss of the neural network based on a single anchor frame detection head and a 3D intersection-union (IU) loss function. Anchor frames are manually set and placed in a deep, high-dimensional feature map. For regions where the anchor frame intersects with the target ground truth frame, these anchor frames are retained, while all other non-intersecting regions are discarded. The 2D anchor frame information is decoded to generate a 3D target prediction box, which is then compared with the 3D ground truth frame using the IU loss function. This result is used for backpropagation of the neural network to adjust the 3D target prediction box, ultimately generating a predicted point cloud target.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: compared with the traditional use of L1 norm as the loss function to calculate the loss, the 3D intersection and union loss function is more applicable to 3D point cloud data and can achieve higher accuracy, especially for targets with more obvious height information such as pedestrians, the accuracy is improved by 1.9% compared with the original. Attached Figure Description

[0015] Figure 1 This is a diagram of a point cloud target detection device based on 3D intersection-union ratio.

[0016] Figure 2 This is a flowchart of the point cloud target detection method based on 3D intersection-union ratio.

[0017] Figure 3 This is a schematic diagram of a 3D point cloud scene.

[0018] Figure 4 This is a diagram showing the results of point cloud object detection. Detailed Implementation

[0019] This invention provides a point cloud target detection method and apparatus based on 3D intersection-union ratio (IUGR). The aim is to optimize existing target detection heads based on 2D image data, making them more adaptable to 3D point cloud data and improving the accuracy of point cloud target detection. The method primarily involves preprocessing and augmenting the point cloud data acquired by LiDAR, inputting the data into a voxel feature extraction network, and then inputting the obtained high-dimensional feature information into a 3D IUGR detection head. A loss function is then used to adjust the detection box size, and finally, the detection effect of the point cloud targets is visualized through a human-computer interaction interface.

[0020] Combination Figure 1 , Figure 2 A point cloud target detection method based on 3D intersection-union ratio (IoU) is proposed. A target detection network using the IoU loss function is presented, and the network is applied in practice using a device. The method consists of five steps:

[0021] Three-dimensional point cloud data is collected using a lidar sensor, and the collected point cloud data is processed using CAD drawing tools to obtain raw point cloud data.

[0022] The raw point cloud data is preprocessed and various data augmentations are performed to adapt to different scenarios, resulting in an initial point cloud.

[0023] The initial point cloud data is input into the point cloud feature extraction network. In this invention, the columnar voxelized point cloud sampling module is used to encode and extract the three-dimensional point cloud sampling into two-dimensional feature information. Then, BEVBackBone is used to perform deep extraction of feature information.

[0024] The deep feature information is input into the detection head module designed with the 3D intersection-union loss function. By calculating the deviation between the target box and the ground truth box, the target box is backpropagated and regressed to accurately define the target point cloud.

[0025] Visualize the scene and the defined target point cloud on a computer device.

[0026] Furthermore, the raw point cloud data undergoes data preprocessing, including:

[0027] Discrete point removal is performed on the point cloud data. The coordinates of each point in the 3D point cloud data are retrieved. Based on these coordinates, a point cloud threshold is set. The difference between the point cloud coordinates and the point cloud threshold is calculated. Based on this difference, discrete points in the point cloud image are obtained, and these discrete points are removed. The following formula can be used to remove the discrete points:

[0028]

[0029] Among them, X i ′ represents the mapping value of the initial point cloud image, Xmedian The median of a point cloud image represents the median value of all points in the image. MAD represents the absolute deviation of the median from the mean, and n represents the deviation coefficient. k X represents the mean of all point clouds. median -nMAD represents the minimum deviation value, X median +nMAD represents the maximum value of the deviation, and eliminate() represents the elimination function.

[0030] The point cloud data obtained in the above steps is subjected to random point drop processing. That is, for each frame of the point cloud scene, a small portion of the point cloud data is randomly deleted, so that the point cloud scene is slightly incomplete, and the pre-processed point cloud data is obtained.

[0031] Furthermore, data augmentation will be performed in various ways to adapt to different scenarios, including:

[0032] Using a global rotation function, the preprocessed point cloud data is augmented. The preprocessed point cloud data is randomly rotated by a certain angle within a manually set rotation range, with the z-axis as the rotation axis. The rotation matrix around the z-axis is shown in the following formula:

[0033]

[0034] Where x', y', and z' represent the three-dimensional coordinates of the point cloud after rotation, x, y, and z represent the three-dimensional coordinates of the point cloud before rotation, and θ represents the rotation angle;

[0035] The point cloud data of the global scene is scaled as a whole, that is, the coordinate values ​​of all point clouds are multiplied by a value within a set range; based on the point cloud threshold obtained in the above steps, a ceiling enhancement function is designed. A small portion of point cloud data is randomly generated from the maximum value of the z-axis downwards within a certain range to simulate a scene with an overhead occluding object; after three data enhancement functions, the initial point cloud data is finally obtained.

[0036] Furthermore, the point cloud feature extraction network includes:

[0037] The columnar voxelization point cloud sampling module is used to perform voxelization feature encoding, converting the initial 3D point cloud data into 2D voxel feature information; the 2D voxel feature information is then input into BEVBackBone for deeper feature extraction to obtain 2D deep feature information.

[0038] Furthermore, the columnar voxelization point cloud sampling module, including the columnar voxelization method and voxel feature encoding, includes:

[0039] Define the length, width, and height dimensions of each columnar voxel. The length and width parameters must be equal and much smaller than the scene's length and width range, while the height parameter must be consistent with the scene's height range. Use these parameters to divide the entire point cloud scene into columnar voxels, transforming the scattered and disordered point cloud scene into a regular and ordered voxel scene. Each columnar voxel will then contain several point cloud data points. Calculate the average of all point cloud coordinates within each voxel to obtain a data point representing the voxel's feature information. The average point cloud coordinates are calculated using the following formula:

[0040]

[0041] Where P[a, b, c] represents the average value of the point cloud, [a, b, c] represents the average coordinates of each point cloud in the voxelized point cloud image, m represents the number of points in the voxelized point cloud image, and a i b i c i Represents the three-dimensional coordinates of the point cloud in the i-th voxelized point cloud image.

[0042] Furthermore, the aforementioned method for extracting deeper point cloud feature information via BEVBackBone, a two-dimensional feature extraction network, includes:

[0043] BEVBackBone consists of three groups, each containing three to five layers of convolutional networks. Each convolutional network layer consists of a 2D CNN, a batch normalization (BN) layer, and a rule activation function. The voxel feature information obtained above is input into BEVBackBone. Through continuous downsampling via convolution, the dimensionality information is increased, ultimately obtaining sparse, high-dimensional deep feature information.

[0044] Furthermore, the design utilizes a three-dimensional intersection-union (IUU) detection head, including:

[0045] Anchor boxes with similar dimensions to the target are manually set and placed in the deep high-dimensional feature information map obtained above. At this time, there will be areas where the anchor boxes intersect with the ground truth boxes of the target. These anchor boxes are retained, and all other non-intersecting areas are removed. The two-dimensional anchor box information is decoded to generate a three-dimensional target prediction box. The intersection-union-comparison (IOU) loss function is obtained by performing an intersection-union-comparison loss function on the three-dimensional ground truth boxes. The three-dimensional target prediction boxes are then adjusted by backpropagation of the neural network to finally generate an accurate predicted point cloud target.

[0046] Furthermore, the visualization of the scene and the defined target point cloud includes:

[0047] The point cloud target detection results obtained from the above training are used to image the point cloud coordinates and the detection box of the defined target in the 3D scene using the Open3D library, so that the specific position of the set target in the point cloud scene can be observed.

[0048] This invention also provides a point cloud target detection device based on three-dimensional intersection-over-union ratio (IoU), which mainly consists of a lidar sensor, a target detection system, and a computer visualization device; the lidar sensor includes a scanning lidar, a fixed tripod, and a connecting network cable; the target detection system includes a data processing module, a feature extraction module, and a three-dimensional target detection head; the computer visualization device includes a control and display submodule.

[0049] The target detection system is designed by computer programming and uploaded to a server cluster to train on the collected and organized point cloud data. Then, the inference model is downloaded to the local machine to visualize the test data.

[0050] Furthermore, the data processing module sets a range threshold for the point cloud scene based on the point cloud coordinate data, thereby distinguishing discrete points and performing discrete point removal processing; it also performs point loss processing on the overall point cloud data, that is, for each frame of the point cloud scene, it randomly deletes a small portion of the point cloud data, making the point cloud scene slightly incomplete; and it further performs global rotation, global scaling, and ceiling enhancement processing on the preprocessed point cloud data to further enhance the fitting and robustness of the point cloud data.

[0051] Furthermore, the feature extraction module extracts deep point cloud feature information based on the BEVBackBone network. The BEVBackBone network mainly consists of three groups: the first group contains 3 convolutional layers, and the second and third groups contain 5 convolutional layers; the last layer of each group uses a 3*3 convolutional kernel for downsampling, which can extract feature information at different scales.

[0052] Furthermore, the 3D target detection head module calculates the forward propagation loss of the neural network based on the single anchor box detection head and the 3D intersection-union (IU) loss function. Anchor boxes similar in size to the target are manually set and placed in the obtained deep high-dimensional feature information map. At this point, areas where the anchor boxes intersect with the target ground truth boxes will appear. These anchor boxes are retained, while all other non-intersecting areas are discarded. The 2D anchor box information is decoded to generate a 3D target prediction box, which is then compared with the 3D ground truth box using the IU loss function. This result is used for backpropagation of the neural network to adjust the 3D target prediction box, ultimately generating an accurate predicted point cloud target.

[0053] The lidar device used is the DJI Livox Avia scanning type, with an output wavelength of 905nm, a scanning power of 9W, and a field of view (FOV) of 70.4° (horizontal) × 77.2° (vertical) when scanning non-repeating.

[0054] The server cluster computing devices are mainly GeForce RTX 2080Ti work cards with 11GB of video memory.

[0055] The technical solutions of the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments of the present invention.

[0056] Example

[0057] like Figure 1 The diagram shown is of a point cloud target detection device based on three-dimensional intersection-union ratio proposed in this invention; as follows: Figure 2 The diagram shows the flowchart of the point cloud target detection method based on three-dimensional intersection-union ratio proposed in this invention. First, the raw point cloud data collected and processed by the lidar sensor undergoes data processing, including data preprocessing and data augmentation: A point cloud range threshold is set to [-6.7, -11.3, -3.5, 14.1, 11.1, 2.5] (representing coordinates [-x, -y, -z, x, y, z] respectively), and all discrete points outside this range threshold are removed. Then, 5% of the global point cloud data is randomly deleted for point drop processing. In data augmentation, the global rotation angle is randomly selected from [-0.25, 0.25]; the global scaling enhancement range is [0.95, 1.05]; and ceiling enhancement involves randomly generating 1% of all points in the point cloud within the z-axis range of [1.5, 2.5].

[0058] The processed point cloud data is input into the feature extraction module. During the feature column voxelization encoding process, the size of each small column is set to [0.1, 0.1, 6] for division. The BEVBackBone network is divided into three groups: the first group includes 3 convolutional layers with an output dimension of 64, the second group includes 5 convolutional layers with an output dimension of 128, and the third group includes 5 convolutional layers with an output dimension of 256. Finally, the output dimensions of the three groups of convolutional networks are adjusted to a unified 128-dimensional dimension before feature aggregation, merging them into a set of 384-dimensional high-dimensional feature information output.

[0059] The aforementioned high-dimensional feature information is input into the 3D intersection-union (IU) detection head. The initial anchor box size for the pedestrian category is manually set to [0.7, 0.7, 1, 7], placed on the high-dimensional information feature map, and compared with the ground truth box. An IU values ​​greater than 0.5 are set as positive samples, and all other anchor boxes are deleted. The positive samples are decoded into 3D predicted boxes and their IU values ​​are compared with the 3D ground truth boxes to calculate the 3D IU loss function. The difference is used for backpropagation in the neural network, and the regression sets the initial anchor boxes to become the final 3D predicted boxes.

[0060] Finally, the Open3D library is used to visualize the global point cloud scene and the predicted bounding boxes, such as... Figure 3 , Figure 4 As shown, this achieves the effect of 3D point cloud target detection.

[0061] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A point cloud object detection method based on three-dimensional intersection over union, characterized in that, The application relates to a method for processing point cloud data, and a computer device. The method comprises the following steps: Collecting three-dimensional point cloud data by using a laser radar sensor, processing the collected point cloud data, and obtaining original point cloud data; Data preprocessing is performed on the original point cloud data, and various scene-adaptive data augmentations are performed to obtain initial point cloud data. The data preprocessing on the original point cloud data comprises the following steps: wherein X' = X - X i represents a mapping value of an initial point cloud image, X median represents a median value of all point clouds in the point cloud image, MAD represents an absolute deviation of the median value from the mean value, n represents a deviation coefficient, X k represents a mean value of all point clouds, X median -nMAD represents a minimum value of the deviation value, X median +nMAD represents a maximum value of the deviation value, eliminate() represents a rejection function; Discrete point elimination processing is performed on the point cloud data; the coordinate values of each point cloud in the three-dimensional point cloud data are queried, the point cloud threshold value corresponding to the point cloud is set according to the coordinate values, the difference between the point cloud coordinate value and the point cloud threshold value is calculated, and the discrete points in the point cloud image are obtained according to the difference; the discrete points are eliminated by the following formula: Random point elimination processing is performed on the point cloud data obtained in the above step; The initial point cloud data is input into a point cloud feature extraction network, three-dimensional point cloud sampling is extracted into two-dimensional feature information by using a cylindrical voxelization point cloud sampling module coding, and deep feature information is extracted by using a BEVBackBone. The deep feature information is input into a designed detection head module using a three-dimensional intersection-over-union loss function, the deviation between the target frame and the true value frame is calculated, the target frame is regressed by back propagation, and the target point cloud is framed.

2. The method of claim 1, wherein, The scene and the framed target point cloud are visualized on the computer device.

3. The method of claim 1, wherein, CAD drawing tools are used to process the collected point cloud data to obtain original point cloud data. Various scene-adaptive data augmentations are performed, including: A global rotation function is used to perform data augmentation on the point cloud data preprocessed; the point cloud data obtained by preprocessing is randomly rotated by a certain angle within an artificially set rotation range with the coordinate system z-axis as the rotation axis; and the rotation matrix around the z-axis is as follows: Wherein, x', y' and z' represent the three-dimensional coordinates of the point cloud after rotation, x, y and z represent the three-dimensional coordinates of the point cloud before rotation, and theta represents the rotation angle.

4. The method of claim 1, wherein, Overall scaling processing is performed on the point cloud data in the global scene, that is, the coordinate values of all point clouds are multiplied by a value within a set range; a ceiling augmentation function is designed according to the point cloud threshold value; point cloud data is randomly generated within a certain range from the maximum value of the z-axis to simulate a scene with an upper shielding object; and finally, the initial point cloud data is obtained through the three data augmentation functions. The point cloud feature extraction network comprises:

5. The method of claim 4, wherein, A cylindrical voxelization point cloud sampling module is used for voxelization feature coding to convert the initial three-dimensional point cloud data into two-dimensional voxel feature information; and the two-dimensional voxel feature information is input into a BEVBackBone for deeper feature extraction to obtain two-dimensional deep feature information. The cylindrical voxelization point cloud sampling module is used for voxelization feature coding, and specifically comprises the following steps: Set the length, width and height dimensions of the columnar voxel, the length and width parameters are required to be equal and much smaller than the length and width range of the scene, and the height parameter is required to be consistent with the height range of the scene. The entire point cloud scene is divided into columnar voxels with the above parameters, and the scattered and disordered point cloud scene is divided into a regular and ordered voxel scene. At this time, several point cloud data information will be contained in each columnar voxel. The mean value of all point cloud coordinates in each voxel is calculated to obtain a data representing the feature information of the voxel. The mean value of the point cloud coordinates is calculated by the following formula: wherein P[a, b, c] represents the point cloud average value, [a, b, c] represents the coordinate average value of each point cloud in the voxelized point cloud image, m represents the number of point clouds in the voxelized point cloud image, a i , b i , c i represents the three-dimensional coordinates of the point cloud of the i-th voxelized point cloud image.

6. The method of claim 4, wherein, The BEVBackBone is used to extract deeper point cloud feature information. Specifically, the BEVBackBone is composed of three groups, each group containing three to five layers of convolutional networks. Each layer of convolutional network is composed of a 2DCNN, a BN layer and a Rule activation function. The voxel feature information obtained above is input into the BEVBackBone, and the dimension information is continuously down-sampled and improved through convolution, and finally the sparse high-dimensional deep feature information is obtained.

7. The method of claim 1, wherein, A detection head using three-dimensional intersection over union is designed. Specifically, an anchor box is manually set and placed in the deep high-dimensional feature information obtained above. At this time, there will be an intersection area between the anchor box and the target true value box. The anchor box in this part is retained, and the rest of the non-intersection area is removed. The two-dimensional anchor box information in this part is decoded to generate a three-dimensional target prediction box. The three-dimensional intersection over union loss function is used to adjust the three-dimensional target prediction box through neural network back propagation, and finally the predicted point cloud target is generated.

8. The method of claim 1, wherein, The visualization of the scene and the framed target point cloud includes: using the open3d library to image the point cloud coordinates and the detection box of the framed target in the three-dimensional scene, so that the specific position of the set target in the point cloud scene can be observed. 9.A three-dimensional intersection over union based point cloud object detection device, characterized in that, The system mainly comprises a laser radar sensor, a target detection system and a computer visualization device. The target detection system comprises a data processing module, a feature extraction module and a three-dimensional target detection head module. The data processing module sets the range threshold of the point cloud scene according to the point cloud coordinate data, and then distinguishes and removes the discrete points. The whole point cloud data is processed by dropping points, that is, for each frame of point cloud scene, part of the point cloud data is randomly deleted, so that the point cloud scene is slightly incomplete. The preprocessed point cloud data is further processed by global rotation, global scaling and ceiling enhancement. The feature extraction module extracts deep point cloud feature information based on the BEVBackBone network. The BEVBackBone network mainly includes three groups. The first group contains 3 convolutional layers, and the second and third groups contain 5 convolutional layers. The last layer of each group uses a 3*3 convolution kernel for down-sampling to extract feature information of different scales. The three-dimensional target detection head module calculates the loss of neural network forward propagation based on a single anchor box detection head and a three-dimensional intersection over union loss function. An artificial anchor frame is set in a deep high-dimensional feature information graph; for the area where the anchor frame and the target true value frame intersect, the anchor frame in this part is retained, and the rest of the area without intersection is all removed; the two-dimensional anchor frame information in this part is decoded to generate a three-dimensional target prediction frame, and the three-dimensional target prediction frame is compared with the three-dimensional true value frame to obtain an intersection over union loss function, which is used for back propagation of the neural network to adjust the three-dimensional target prediction frame, and finally a predicted point cloud target is generated.

Images