Point cloud 3D target detection method, computer device and computer readable storage medium

Abstract

The invention belongs to the technical field of point cloud target detection, and particularly relates to a point cloud 3D target detection method, a computer device and a computer readable storage medium. The method comprises the following steps: acquiring 3D point cloud data and inputting the 3D point cloud data into a trained target detection model to obtain a target detection result; wherein the target detection model comprises a context aggregation module used for enriching seed point features, and the processing process of the target detection model comprises the steps of performing compression processing on features output by each set abstraction layer in a backbone network pointnet + +, splicing all the compressed features, and storing the spliced features in the context aggregation module; and after splicing, channel dimension adjustment is carried out, and a channel dimension adjustment result and output of a second feature propagation layer in pointnet + + are subjected to feature fusion processing and then output. According to the invention, the fine-grained information aggregation of the local area is realized, and the accuracy of positioning the candidate target is improved by aggregating the global information of different levels of the backbone network.

Description

Technical Field

[0001] The present invention belongs to the technical field of point cloud object detection, and particularly relates to a point cloud 3D object detection method, a computer device, and a computer-readable storage medium. Background Art

[0002] 3D object detection based on point cloud can accurately locate the center and bounding box of an object and identify its semantic category, which is indispensable in 3D scene applications such as autonomous driving, robot operation, and augmented reality. However, due to the irregularity and disorder of point cloud, it cannot directly apply the currently rapidly developing 2D object detection models.

[0003] Early 3D object detection methods usually convert point cloud into regular data structures such as voxels or bird's-eye views and then use them for detection tasks, but this usually brings expensive computational overhead or loss of geometric details. The Chinese invention patent with the application publication number CN116229452A and the application publication date of June 6, 2023 discloses a point cloud three-dimensional object detection method based on improved multi-scale feature fusion. The point cloud three-dimensional object detection model for performing three-dimensional object detection includes a dimensionality reduction module, an encoding module, a multi-scale feature fusion module, and a detection head. The dimensionality reduction module therein is used to perform voxelization dimensionality reduction and feature enhancement processing on the input 3D point cloud image to obtain a 2D pseudo-image after feature enhancement. That is, it adopts the strategy of projecting point cloud onto a two-dimensional plane. Although this processing method can simplify data processing, it will lose some important three-dimensional characteristic data of point cloud, resulting in missed detection and misdetection of targets.

[0004] To solve the above problems, with the rapid development of deep learning, it has also been widely applied in the field of 3D object detection. Among them, PointNet is a neural network model specifically designed for processing point cloud data, which is designed to directly operate on irregular point clouds without converting them into regular grids or voxels. On this basis, PointNet++ is derived, which is an updated version of PointNet and a deep learning network with better local feature learning ability, semantic segmentation ability, and robustness, and can better handle complex 3D scenes, especially scenes with complex structures or multiple parts. Furthermore, some scholars introduced the concept of Hough voting and proposed an end-to-end 3D object detection method that directly uses point clouds as input, achieving relatively excellent performance. Moreover, due to the rise of Transformer and the fact that self-attention in it has no order requirement for input data, it has a natural advantage in applications dealing with unordered point clouds. Some existing works have improved the Transformer model to adapt to 3D object detection tasks. For example, the concept of voting is abandoned, the KPS method (nearest point sampling method) is used to directly select candidate objects from seed points, and a concept of no grouping is proposed. Transformer is used to aggregate the features of object candidates from uniformly distributed seed points, achieving further performance. However, the points obtained by using the KPS method are relatively sparse on the object surface and are easily interfered by surrounding background points, misjudging background points as candidate objects, which is not conducive to subsequent global context learning of candidate objects. When using PointNet++ for processing, the obtained seed point features only have higher-level semantic information but lack fine-grained information in the local area, which is also not conducive to subsequent classification and selection of seed point features, resulting in inaccurate candidate objects finally selected. Summary of the Invention

[0005] The purpose of the present invention is to provide a point cloud 3D object detection method, a computer device, and a computer-readable storage medium to solve the problem that the finally selected candidate objects are inaccurate due to the lack of fine-grained information in the local area in the seed point features in the prior art.

[0006] To solve the above technical problems, the present invention provides a point cloud 3D object detection method, which includes:

[0007] Obtain 3D point cloud data and input it into the trained object detection model to obtain the object detection result;

[0008] Among them, the object detection model includes a backbone network for generating original seed points covering the entire scene, a context aggregation module for enriching the features of the original seed points, a candidate object generation module for discriminating the original seed points and the seed points with enriched features to generate candidate objects, and a detection head for obtaining the final detection result based on the generated candidate objects;

[0009] The backbone network adopts pointnet++; the processing process of the context aggregation module includes: compressing the features output by each set abstraction layer in pointnet++, splicing all the compressed features, adjusting the channel dimension after splicing, and copying the result after channel dimension adjustment and performing feature fusion processing with the output of the second feature propagation layer in pointnet++ and then outputting.

[0010] Furthermore, a CN module is adopted for the compression processing, and the CN module includes a multi-layer perceptron, batch normalization, an activation function, and a pooling layer connected in sequence.

[0011] Furthermore, the pooling layer is a maximum adaptive pooling layer.

[0012] Furthermore, the process of the feature fusion processing is: splicing the result after channel dimension adjustment with the output of the second feature propagation layer, and performing channel dimension adjustment through a multi-layer perceptron after splicing.

[0013] Furthermore, the candidate object generation module includes an initial candidate object generation unit for generating initial candidate objects and a candidate object enhancement and refinement unit for enhancing and refining the generated initial candidate objects.

[0014] Furthermore, the candidate object enhancement and refinement unit includes N cascaded improved Transformer models. The improved Transformer model is a model obtained by improving the traditional Transformer model. The improvement includes that the self-attention module in the model adds a channel self-attention branch on the basis of the spatial self-attention branch, and then fuses the outputs of the spatial self-attention branch and the channel self-attention branch as the output of the self-attention module; the channel self-attention branch therein is used to calculate the channel relationship matrix between all feature points of the input of the candidate object enhancement and refinement unit.

[0015] Furthermore, the way of fusing the outputs of the spatial self-attention branch and the channel self-attention branch is: splicing the outputs of the spatial self-attention branch and the channel self-attention branch, and performing channel dimension adjustment through a multi-layer perceptron after splicing.

[0016] Further, the initial candidate target generation unit is used to discriminate and generate initial candidate targets for the original seed points and the seed points after enriching the seed point features by using the K-nearest neighbor sampling method.

[0017] To solve the above technical problems, the present invention also provides a computer device, including a processor, and the processor executes a computer program to implement the steps of the 3D object detection method for point cloud introduced above.

[0018] To solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the 3D object detection method for point cloud introduced above are implemented.

[0019] The beneficial effects are as follows: The present invention is an improved invention. Considering that global features are beneficial to object classification and imply the layout information of the entire point cloud scene, therefore, the present invention adds a context aggregation module for enriching seed point features to the entire object detection model. This context aggregation module aggregates the global information in the backbone network pointnet++ to improve the accuracy of positioning candidate targets. When the context aggregation module is specifically processed, first, the features output by each set abstraction layer are compressed, after compression, feature splicing is performed, after splicing, the channel dimension is adjusted, and the result of the channel dimension adjustment is feature-fused with the output of the second feature propagation layer in pointnet++, realizing the fine-grained information aggregation of the local area. Finally, the result of the feature fusion process is input into the candidate target generation module to generate candidate targets, so as to generate relatively accurate candidate targets and improve the accuracy of 3D object detection. Description of the Drawings

[0020] Figure 1 is the overall network framework diagram of the object detection model of the present invention;

[0021] Figure 2 is the schematic diagram of the simple multi-scale context aggregation module (MSC) of the present invention;

[0022] Figure 3 is the structural diagram of the dual-branch Transformer network of the present invention. Detailed Embodiments

[0023] The main inventive concept of the present invention lies in improving the object detection model for object detection. It is improved such that in addition to the backbone network PointNet++, the candidate object generation module, and the detection head in the model, a context aggregation module for enriching the features of seed points is added. The processing process of this context aggregation module is as follows: compress the features output by each set abstraction layer in the backbone network PointNet++; splice all the compressed features; adjust the channel dimension after splicing and perform feature fusion processing on the result of the channel dimension adjustment and the output of the second feature propagation layer in PointNet++ and then output. This network can aggregate global information at different levels and improve the accuracy of positioning candidate objects. Based on this concept, a 3D point cloud object detection method, a computer device, and a computer-readable storage medium of the present invention can be realized. To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0024] An embodiment of a 3D point cloud object detection method:

[0025] The 3D point cloud object detection method of the present invention uses an object detection model improved by introducing global context and dual-branch Transformer. After training the object detection model, input the obtained 3D point cloud data into the trained object detection model to obtain the object detection result.

[0026] As Figure 1 shown, the entire object detection model includes a backbone network for generating original seed points covering the entire scene, a context aggregation module for enriching the features of the original seed points, a candidate object generation module for discriminating the original seed points and the seed points with enriched features to generate candidate objects, and a detection head for obtaining the final detection result based on the generated candidate objects. As an optimized processing solution, the candidate object generation module further includes an initial candidate object generation unit for generating initial candidate objects and a candidate object enhancement and refinement unit for enhancing and refining the generated initial candidate objects. The following will introduce each module / unit in detail.

[0027] I. Backbone network.

[0028] Preprocess the point cloud data and select 20,000 points from each sample as the input (N represents the number of point clouds), Figure 1 and the Input point in Figure 1 is the input point cloud. As Figure 1Extract the features of each point in the scene using the Backbone in []. Use pointnet++ as the backbone network to continuously downsample and generate seed points that evenly cover the entire scene. As Figure 2 shown, pointnet++ has 4 set abstraction layers (SetAbstraction) and 2 feature propagation layers (Feature Propagation) with skip connections. The set abstraction layers are downsampling layers, namely Figure 2 SA1 - SA4 in [], and the feature propagation layers are upsampling layers, namely Figure 2 FP1 - FP2 in []. The combination abstraction layer is used to continuously downsample the point set and aggregate the information of adjacent points using ball query for deep feature learning; moreover, the 1024 points selected in the second layer are used as seed points, continue to downsample and pass through the feature backpropagation layer, and attach the higher-level point features to the seed point features by interpolation, so that the seed points have richer features, and finally obtain seed points that evenly cover the entire scene with coordinates c i ∈R 3 and feature f i ∈R d .

[0029] II. Context Aggregation Module.

[0030] Subsequently, the points obtained by using KPS (k-Closest Point Sampling) are relatively sparse on the object surface and are easily interfered by surrounding background points, misjudging the background points as candidate objects, which is not conducive to subsequent global context learning of candidate objects. In addition, using the method without grouping to aggregate the features of candidate objects from all seed points only considers the attention relationship between different point features, and mainly measures the spatial similarity between point clouds. Although it can effectively refine the proposal (referring to the process of optimizing and adjusting the initially generated candidate regions in the object detection process), there are still many cases where high-attention seed points are located in the background or on other objects during this process. In order to make the finally obtained seed point features not only have higher-level semantic information but also have fine-grained information of the local area, so as to use subsequent KPS to classify and select accurate targets for the seed point features, and considering that global features are beneficial to object classification and imply the layout information of the entire point cloud scene, the present invention proposes a simple multi-scale context aggregation module (referred to as MSC, simply called the context aggregation module) to enrich the seed point features and improve the accuracy of positioning candidate targets by aggregating the global context information of different levels of the backbone network.

[0031] First, as Figure 2As shown, in order to aggregate the features of different set abstraction layers, a CN module is designed. This CN gate module is used to compress the features output by each set abstraction layer. Its network structure is as Figure 2 shown, including a multi-layer perceptron (MLP), batch normalization (BN), activation function, and pooling layer connected in sequence. In this embodiment, the activation function selects ReLU, and the pooling layer selects max adaptive pooling. Based on this structure, the output features of each set abstraction layer are fixed in the channel dimension by using the MLP, and then the number of different sampling points of different set abstraction layers is fixed to 1 through max adaptive pooling. In this way, the features output by each set abstraction layer are compressed into a one-dimensional vector with a fixed dimension:

[0032] CN(SA i ) = Max(MLP(SA i ))

[0033] In the formula, CN() represents the processing by the CN module, Max() represents max adaptive pooling, MLP() represents the processing by the multi-layer perceptron, and SA i represents the output of the i-th set abstraction layer, where i = 1, 2, 3, 4. It should be noted that batch normalization and the activation function are omitted in the above formula.

[0034] Furthermore, all the compressed features obtained by using the CN module are concatenated along the channel dimension and then passed through the CN module for channel dimension adjustment to obtain g. The specific implementation formula is:

[0035] g = CN([CN(SA i ), i = 1, 2, 3, 4])

[0036] In the formula, [] represents the concatenation process.

[0037] Then, it is replicated to the number of seed points and appended to each seed point feature, and then a MLP( Figure 2 not shown in the figure, which is the MLP in the following formula g ) is used for channel dimension adjustment, thus obtaining:

[0038] f i ' = MLP g ([f, R(g)])

[0039] In the formula, R() represents replication, that is, Repeat. When performing the Repeat operation here, it is necessary to keep the same as the Figure 2 Tensor (i.e., dimension) of FP2 in order to perform the concatenation operation; f represents the output of FP2 in pointnet++.

[0040] III. Initial candidate target generation unit.

[0041] The initial candidate target generation unit is used to discriminate and generate initial candidate targets for the seed points after enriching the seed point features by using the K-nearest neighbor sampling method, that is, the KPS (hereinafter referred to as the KPS module) as in Figure 1 , and the result obtained is Figure 1 the Proposal in

[0042] The seed point features with global information imply the distribution of the entire scene, can more accurately perceive the distribution and connection of the target objects, and are beneficial for the KPS module to select more accurate target candidate objects. However, considering that in the proposal refinement module, it is necessary to rely on the seed points to help the candidate objects learn global information, and aggregate the seed point features within the specific targets around the candidate objects in an ungrouped manner. At this time, it is necessary to aggregate the local target information as much as possible, and the seed point features with global information are instead not conducive to the learning of the candidate object features.

[0043] Therefore, the KPS module adopts a retrospective method to extract the candidate object features, which does not affect the original seed point features and does not affect the subsequent candidate target enhancement and refinement unit. The specific steps are as follows: First, aggregate the global information of the seed point features of the original features to obtain the global seed point features with richer global information, and this process is the processing process of the context aggregation module; then, use the KPS module to discriminate the global seed point features (that is, the seed points after rich features) to obtain more accurate candidate object indexes, and then revisit the original seed point features, select the candidate object features according to the indexes, and obtain the coordinates of the candidate objects from the seed point positions according to the indexes. This process is the processing process of discriminating both the original seed points and the seed points after rich features to generate initial candidate targets.

[0044] IV. Candidate target enhancement and refinement unit.

[0045] The candidate target enhancement and refinement unit is used to enhance and refine the generated initial candidate targets. In this embodiment, 6 layers of Transformers are stacked (the specific number of stacked layers can be adjusted according to actual needs) as the candidate target enhancement and refinement unit ( Figure 1 all the content with green background color in

[0046] In 3D point cloud object detection, previous Transformer multi-head self-attention modules often only consider the relationship between points. By calculating the attention weights between different point features, the point features are weighted and aggregated to obtain the updated query, i.e., the candidate object point features. Here, the relationship between different candidate objects in the point cloud space is mainly considered, which is called the point-based spatial self-attention branch. The present invention proposes another point-based channel self-attention branch and combines these two branches, which can not only make good use of the context relationship in the remote space of the point cloud scene, but also explore the channel context dependence of different candidate objects. Specifically:

[0047] 1) The point-based spatial self-attention branch, i.e., Figure 1 and 3 "Point-wise Multi-head Self-attention" in Figure 3 The point-based spatial self-attention branch mainly borrows the multi-head self-attention calculation method of the original Transformer. The candidate object features generated in the previous step are projected to generate query embedding (Q), key embedding (K), and value embedding (V). A matrix multiplication operation is performed between Q and K to generate the attention matrix, and then the matrix is normalized using softmax. The normalized matrix is multiplied by V to obtain the updated Q. As l+1 shown, the output X

[0048]

[0049] In the formula, W p represents the weight matrix, which is used to perform linear transformation and concatenation on the outputs of multiple spatial attention modules; [] represents the concatenation operation along the channel dimension; represents the output of the l-th layer of the Transformer, and l represents the number of stacked attention modules; H represents multi-head attention. Each spatial attention module is defined as follows:

[0050]

[0051] In the formula, h represents the index of the attention head; represents the spatial attention map, which measures the influence of other points on the current point; C represents the total channel dimension, and σ() represents the Softmax operation; represents the projection weight of the linear mapping, and d q , d k and d v represent the corresponding dimensions respectively.

[0052] 2) Point-based channel attention branch, namely Figure 1 , 3 "Channel-Wise Multi-head Self-attention" in. To enrich the features of proposal points and make the finally regressed and generated bounding boxes more accurate, a point-based channel self-attention branch is proposed to learn the channel context relationship between different proposal points. Similar to the point-based spatial self-attention branch, the same input is used, but when calculating the self-attention matrix, the channel relationship matrix between all points is calculated without considering the features of all points. As Figure 3 shown, the output of the point-based channel self-attention branch is expressed as follows:

[0053]

[0054] In the formula, W c represents the weight matrix, which is used for linear transformation of the concatenated features; [] represents the concatenation operation along the channel dimension; represents the channel attention map, which measures the influence of other channels on the current channel; SF() represents the Softmax operation; represents the projection weight of the linear mapping.

[0055] 3) Fuse the point-based spatial self-attention branch and the point-based channel attention branch, and the fusion result is used as the output of the self-attention module. The fusion method of the two branches is: concatenate the results of the two branches along the channel dimension, and then process them through two layers of MLP to obtain a reasonable output in a learning way to well retain the spatial context relationship and channel context dependence relationship of different points. The specific expression is:

[0056]

[0057] V. Detection head.

[0058] The detection head is used to obtain the final detection result based on the generated candidate targets. As Figure 3 shown, in order to obtain better prediction results, it aggregates the detection results of different decoding layers and sends them to the 3D NMS module (IoU threshold is 0.25) for processing to obtain the final 3D object detection result. Among them, NMS represents non-maximum suppression, which is an algorithm for extracting the window with the highest score in object detection.

[0059] An embodiment of a computer device:

[0060] An embodiment of a computer device of the present invention includes a memory, a processor, an internal bus, a computer program stored in the memory, a communication interface, an input device, and a display screen. The processor and the memory communicate and interact with each other through the internal bus.

[0061] Among them, the processor is used to provide computing and control capabilities, and execute the computer program to implement the steps of the method introduced in an embodiment of a point cloud 3D object detection method of the present invention. It can be a processing device such as a microprocessor MCU or a programmable logic device FPGA. The memory can be a non-volatile storage medium or an internal memory. The non-volatile storage medium stores an operating system and a computer program, and the internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be implemented through WIFI, a carrier network, NFC (Near Field Communication), or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device. It can also be an external keyboard, a touchpad, or a mouse, etc.

[0062] An embodiment of a computer-readable storage medium:

[0063] An embodiment of a computer-readable storage medium of the present invention stores a computer program thereon. When the program is executed by a processor, it implements the point cloud 3D object detection method proposed in the above embodiment. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (abbreviated as SRAM), electrically erasable programmable read-only memory (abbreviated as EEPROM), erasable programmable read-only memory (abbreviated as EPROM), programmable read-only memory (abbreviated as PROM), read-only memory (abbreviated as ROM), magnetic memory, or flash memory.

[0064] In summary, the present invention provides a method for 3D object detection of point clouds improved by introducing global context and a dual-branch Transformer, which utilizes rich global context information to improve the problems existing in KPS, obtains relatively accurate candidate objects, and selects the features of candidate objects by means of backtracking; and proposes a structure of a dual-branch Transformer to enhance the features of candidate objects, improving the accuracy of 3D object detection.

[0065] The specific implementation manners are given above, but the present invention is not limited to the described implementation manners. The basic idea of the present invention lies in the above basic solution. For those of ordinary skill in the art, according to the teachings of the present invention, it does not require creative labor to design various deformed models, formulas, and parameters. Changes, modifications, substitutions, and variations made to the implementation manners without departing from the principles and spirit of the present invention still fall within the protection scope of the present invention.

Claims

1. A point cloud 3D object detection method, characterized in that The method includes: Obtaining 3D point cloud data and inputting it into a trained object detection model to obtain an object detection result; Among them, the object detection model includes a backbone network for generating original seed points covering the entire scene, a context aggregation module for enriching the features of the original seed points, a candidate object generation module for discriminating the original seed points and the seed points with enriched features to generate candidate objects, and a detection head for obtaining a final detection result based on the generated candidate objects; The backbone network uses pointnet++; the processing process of the context aggregation module includes: compressing the features output by each set abstraction layer in pointnet++, splicing all the compressed features, adjusting the channel dimension after splicing, and copying the result after channel dimension adjustment and performing feature fusion processing with the output of the second feature propagation layer in pointnet++ and then outputting.

2. The point cloud 3D object detection method according to claim 1, wherein The compression processing is performed using a CN module, and the CN module includes a multi-layer perceptron, batch normalization, an activation function, and a pooling layer connected in sequence.

3. The point cloud 3D object detection method according to claim 2, wherein The pooling layer is a maximum adaptive pooling layer.

4. The point cloud 3D object detection method according to claim 1, characterized in that The process of the feature fusion processing is: splicing the result after channel dimension adjustment with the output of the second feature propagation layer, and adjusting the channel dimension after splicing through a multi-layer perceptron.

5. The point cloud 3D object detection method according to claim 1, characterized in that, The candidate object generation module includes an initial candidate object generation unit for generating initial candidate objects and a candidate object enhancement and refinement unit for enhancing and refining the generated initial candidate objects.

6. The point cloud 3D object detection method according to claim 5, wherein The candidate object enhancement and refinement unit includes N cascaded improved Transformer models. The improved Transformer model is a model obtained by improving the traditional Transformer model. The improvement includes that the self-attention module in the model adds a channel self-attention branch on the basis of the spatial self-attention branch, and then fuses the outputs of the spatial self-attention branch and the channel self-attention branch as the output of the self-attention module; the channel self-attention branch therein is used to calculate the channel relationship matrix between all feature points of the input of the candidate object enhancement and refinement unit.

7. The point cloud 3D object detection method according to claim 6, wherein The way of fusing the outputs of the spatial self-attention branch and the channel self-attention branch is: splicing the outputs of the spatial self-attention branch and the channel self-attention branch, and adjusting the channel dimension after splicing through a multi-layer perceptron.

8. The point cloud 3D object detection method according to claim 5, characterized in that The initial candidate object generation unit is used to discriminate and generate initial candidate objects from the original seed points and the seed points with enriched features by using the K-nearest neighbor sampling method.

9. A computer device, comprising a processor, characterized in that, The processor executes a computer program to implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 8.

Images