Target detection method and device based on 3D point cloud data and storage medium
By dividing 3D point cloud data into grid blocks and combining with self-attention mechanism, the problems of large amount of computing and spatial information loss in the existing technology are solved, efficient and accurate target detection is achieved, and environmental perception of autonomous vehicles is improved ability.
Patent Information
- Application Number
- CN202411130874.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-12-06
AI Technical Summary
When the existing 3D object detection methods process 3D point cloud data, the point-based method has a large amount of calculation, while the columnar method will lose spatial information when dividing point cloud data, resulting in inaccurate detection results.
The 3D point cloud data to be detected is divided into multiple grid blocks, and each grid block is extracted separately. After processing by using a convolutional neural network, a two-dimensional feature map is obtained, and the object detection is performed through the detection head, combined with the self-contained Attention mechanism calculates the degree of correlation between grid blocks to determine attention weights, reduces the amount of calculations, and preserves spatial information.
While reducing the amount of computing and memory, the accuracy of target detection results is improved, the loss of spatial information is avoided, and the environmental perception ability of autonomous driving vehicles is improved.
Smart Images

Figure CN119091407A_ABST
Abstract
Description
Technical Field
[0001] One or more embodiments of the present application relate to the field of target detection, and in particular, to a method, device, and storage medium for performing target detection based on 3D point cloud data. Background Art
[0002] With the rapid development of autonomous driving technology, 3D object detection technology, as a core component of the autonomous driving field, has become increasingly important. 3D object detection technology mainly uses 3D point cloud data obtained by laser radar scanning to detect objects, realize three-dimensional recognition of environmental objects, and thus improve the environmental perception ability of autonomous driving vehicles.
[0003] In the related art, 3D target detection methods are mainly divided into point-based methods and column-based methods. Among them, the point-based method needs to process each 3D point cloud data obtained by the lidar scan. In addition, the point cloud data is complex and unstructured, which causes the method to require a large amount of memory and calculation. The column-based method needs to divide the entire 3D point cloud data into several columns, and then send the columns to the convolutional neural network for feature extraction and classification. Since this method divides the point cloud data into columns, it weakens the connection between each point cloud data in the horizontal direction, so it will lose some spatial information of the 3D point cloud data, which will lead to inaccurate target detection results. Summary of the invention
[0004] In view of this, the present application provides a method, device and storage medium for target detection based on 3D point cloud data to address the deficiencies in the related art.
[0005] According to a first aspect of one or more embodiments of the present application, a method for object detection based on 3D point cloud data is provided, the method comprising:
[0006] Acquire 3D point cloud data to be detected, and divide the 3D point cloud data to be detected into a plurality of grid blocks;
[0007] Performing feature extraction on the 3D point cloud data to be detected contained in each grid block respectively, to obtain the point cloud features corresponding to each grid block, wherein the features in the z direction of the point cloud features are feature statistics of each 3D point cloud data to be detected contained in each grid block in the z direction;
[0008] Inputting the point cloud features corresponding to the plurality of grid blocks into a convolutional neural network to obtain a two-dimensional feature map output by the convolutional neural network;
[0009] The two-dimensional feature map is input into a detection head to obtain a category detection result and a position detection result corresponding to the 3D point cloud data to be detected output by the detection head.
[0010] Optionally, the feature extraction of the 3D point cloud data to be detected contained in any grid block to obtain the point cloud features corresponding to the any grid block includes: inputting the 3D point cloud data to be detected contained in any grid block into a fully connected layer to obtain the original point cloud features corresponding to the any grid block output by the fully connected layer; performing maximum pooling processing on the original point cloud features corresponding to the any grid block to obtain local aggregation features corresponding to the any grid block; splicing the original point cloud features with the local aggregation features, and calculating the feature statistics of the spliced point cloud features in the z direction to obtain the point cloud features corresponding to the any grid block.
[0011] Optionally, the method further includes: using a self-attention mechanism to calculate the degree of correlation between each grid block and other grid blocks, and determining the attention weight corresponding to each grid block according to the calculation result, so as to obtain the point cloud features with attention weights corresponding to each grid block; wherein the attention weights are positively correlated with the degree of correlation. Inputting the point cloud features corresponding to the plurality of grid blocks into the convolutional neural network includes: inputting the point cloud features corresponding to the plurality of grid blocks with attention weights into the convolutional neural network.
[0012] Optionally, the convolutional neural network includes a deconvolution layer and at least two parallel convolution layers with different spatial resolutions, wherein the at least two convolution layers are used to extract at least two feature maps with different resolutions from the point cloud features corresponding to the multiple grid blocks respectively; the deconvolution layer is used to adjust the sizes of the at least two feature maps with different resolutions to the same size, so as to perform feature fusion on at least two feature maps of the same size to obtain the two-dimensional feature map.
[0013] Optionally, each convolutional layer includes a dilated convolutional network layer.
[0014] Optionally, the convolutional neural network also includes an average pooling layer, which is used to perform average pooling processing on the feature map after feature fusion to output the two-dimensional feature map.
[0015] According to a second aspect of one or more embodiments of the present application, a method for training a target detection model is provided, including:
[0016] Acquire sample 3D point cloud data to be detected and a sample category and a sample position corresponding to the sample 3D point cloud data to be detected;
[0017] The sample 3D point cloud data to be detected is input into a target detection model, so that the target detection model divides the sample 3D point cloud data to be detected into a plurality of grid blocks, and extracts sample point cloud features corresponding to the sample 3D point cloud data to be detected contained in each grid block, and processes the sample point cloud features corresponding to the plurality of grid blocks respectively by a convolutional neural network layer in the target detection model to obtain a sample two-dimensional feature map corresponding to the sample 3D point cloud data to be detected, and performs target detection on the sample two-dimensional feature map by a detection head in the target detection model, and outputs a sample category detection result and a sample position detection result corresponding to the sample 3D point cloud data to be detected; wherein, the sample features in the z direction in the sample point cloud features are sample feature statistics in the z direction of each sample 3D point cloud data to be detected contained in each grid block;
[0018] A category loss value is calculated according to the sample category and the sample category detection result, and a position loss value is calculated according to the sample position and the sample position detection result, and the target detection model is iteratively trained based on the category loss value and the position loss value.
[0019] According to a third aspect of one or more embodiments of the present application, an electronic device is provided, including:
[0020] processor;
[0021] a memory for storing processor-executable instructions;
[0022] The processor implements the method described in any one of the embodiments of the first aspect or the second aspect by running the executable instructions.
[0023] According to a fourth aspect of one or more embodiments of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any embodiment of the first aspect or the second aspect are implemented.
[0024] According to the fifth aspect of one or more embodiments of the present application, a computer program product is provided, comprising a computer program and / or instructions, which, when executed by a processor, implement the steps of the method described in any embodiment of the first or second aspect above.
[0025] As can be seen from the above technical solutions, in one or more embodiments of the present application, the acquired 3D point cloud data to be detected is divided into a plurality of grid blocks. Since the grid blocks can divide the 3D point cloud data to be detected into a series of uniform small cubes or rectangular bodies, the distance between adjacent grid blocks is constant, which helps to capture the relationship between each 3D point cloud data to be detected in the horizontal direction (i.e., the z direction) and avoid spatial information loss. Then, feature extraction is performed on the 3D point cloud data to be detected contained in each grid block to obtain the point cloud features corresponding to each grid block, and the features in the z direction of the point cloud features are the feature statistics of each 3D point cloud data to be detected contained in each grid block in the z direction. This method greatly reduces the amount of data of feature information of each grid block in the z direction, thereby effectively reducing the subsequent amount of calculation and the required memory. Then, the point cloud features corresponding to each grid block are respectively input into the convolutional neural network to obtain a two-dimensional feature map output by the convolutional neural network. Finally, the detection head is used to perform target detection on the two-dimensional feature map to obtain the category detection result and position detection result of the 3D point cloud data to be detected.
[0026] The above target detection method can reduce the amount of calculation and memory while retaining the spatial information of the 3D point cloud data to be detected in the horizontal direction, thereby effectively improving the accuracy of the final target detection result.
[0027] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0029] Figure 1 The present invention is a schematic diagram of a scenario in which a vehicle performs target detection, provided by an exemplary embodiment.
[0030] Figure 2 It is a flowchart of a method for target detection based on 3D point cloud data provided by an exemplary embodiment.
[0031] Figure 3 It is a schematic diagram of dividing a grid block provided by an exemplary embodiment.
[0032] Figure 4a is a schematic diagram of a fully connected layer provided by an exemplary embodiment.
[0033] Figure 4b It is a schematic diagram of extracting point cloud features provided by an exemplary embodiment.
[0034] Figure 5It is a schematic diagram of determining the attention weights corresponding to each grid block provided by an exemplary embodiment.
[0035] Figure 6 It is a schematic diagram of a convolutional neural network provided by an exemplary embodiment.
[0036] Figure 7a It is a schematic diagram of the structure of a convolutional layer provided by an exemplary embodiment.
[0037] Figure 7b is a schematic diagram of another convolutional neural network provided by an exemplary embodiment.
[0038] Figure 8a It is a schematic diagram of a target detection model provided by an exemplary embodiment.
[0039] Figure 8b It is a flowchart of a method for training a target detection model provided by an exemplary embodiment.
[0040] Fig. 9 It is a schematic structural diagram of an electronic device provided by an exemplary embodiment.
[0041] Fig.10 It is a block diagram of a device for performing target detection based on 3D point cloud data provided by an exemplary embodiment.
[0042] Fig.11 It is a block diagram of a training device for a target detection model provided by an exemplary embodiment. DETAILED DESCRIPTION
[0043] Next, one or more embodiments of the present application are described in detail.
[0044] Figure 1 FIG. 1 is a schematic diagram of a scene in which a vehicle performs target detection according to an exemplary embodiment. Figure 1 As shown, the cloud server 12 establishes a communication connection with the vehicle 11. The vehicle 11 can be any vehicle with target detection requirements (including manned vehicles or unmanned vehicles), and the vehicle 11 is equipped with a device (such as a laser radar) that can collect 3D point cloud data of the vehicle's surrounding environment. The cloud server 12 can provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.
[0045] based on Figure 1In the scene diagram shown, the vehicle 11 can collect 3D point cloud data of the surrounding environment during driving, and upload the collected 3D point cloud data to the cloud server, so that the cloud server can perform target detection on the received 3D point cloud data based on the technical solution of the present application, and return the target detection result to the vehicle 11. It should be noted that the cloud server 12 can establish a communication connection with multiple vehicles to perform target detection on the 3D point cloud data reported by multiple vehicles respectively, and return each target detection result to the corresponding vehicle respectively.
[0046] In another embodiment, the target detection solution of the present application can be applied to a vehicle, which can be Figure 1 The vehicle 11 shown in the figure collects 3D point cloud data of the surrounding environment during driving, and implements the target detection scheme of the present application on the collected 3D point cloud data locally in the vehicle, so as to directly obtain the target detection result corresponding to the 3D point cloud data.
[0047] Figure 2 The following is a flow chart of a method for performing target detection based on 3D point cloud data provided by an exemplary embodiment. Figure 2 As shown, the method may include the following steps:
[0048] S201, obtaining 3D point cloud data to be detected, and dividing the 3D point cloud data to be detected into a plurality of grid blocks.
[0049] The 3D point cloud data to be detected can be collected by a laser radar or other equipment installed on the target vehicle, and the target vehicle can be understood as any vehicle that has a target detection requirement. Specifically, the 3D point cloud data to be detected is a collection of a large number of spatial points collected during environmental scanning, that is, each spatial point in the 3D point cloud data to be detected represents a specific location in the real world. In this embodiment, each spatial point in the 3D point cloud data to be detected can be expressed as (x, y, z, r), where x, y, and z represent the three-dimensional spatial coordinates of any spatial point, and r represents the reflection intensity of any spatial point.
[0050] The 3D point cloud data to be detected is divided into multiple grid blocks. Figure 3 FIG. 1 is a schematic diagram of dividing a grid block provided by an exemplary embodiment. Figure 3As shown, the grid block can divide the three-dimensional space where the 3D point cloud data to be detected is located into a series of uniform small cubes or rectangular bodies. In other words, rasterizing the 3D point cloud data to be detected can achieve uniform division of the 3D point cloud data to be detected. Since the 3D point cloud data to be detected is disordered, irregular and sparse, the number of spatial points contained in different grid blocks is not necessarily the same. In this embodiment, N spatial points in each grid block are randomly sampled (N is a positive integer) to perform feature encoding on the sampled spatial points. When the number of spatial points contained in the grid block is greater than or equal to N, all spatial points in the grid block are sampled; when the number of spatial points contained in the grid block is less than N, a number of spatial points with a feature of 0 are filled in the grid block so that the number of spatial points in the grid block is N. In this way, on the one hand, a certain number of spatial points are selected for feature encoding, which helps to reduce the amount of calculation in the target detection process; on the other hand, zero-filling is performed on the grid blocks whose number of spatial points is less than the preset value N, which helps to ensure that the 3D point cloud data to be detected contained in each grid block has the same length / size, avoiding the problem of data format / feature dimension mismatch between subsequent grid blocks.
[0051] Preferably, in order to extract more spatial information from the 3D point cloud data to be detected, this embodiment proposes to represent the feature of each spatial point as a ten-dimensional vector (x, y, z, r, x d ,y d ,z d ,x s ,y s ,z s ), where x, y, and z represent the three-dimensional spatial coordinates of any spatial point, r represents the reflection intensity of any spatial point, and x d ,y d ,z d Respectively represent the distance between any spatial point and the arithmetic mean of all spatial points in the grid block where it is located, x s ,y s ,z s Respectively characterize the coordinate offset between any spatial point and the center point of the grid block in which it is located. Assuming that the 3D point cloud data to be detected is evenly divided into D grid blocks, the 3D point cloud data to be detected can be converted into a tensor (D, N, 10), where D represents the number of grid blocks, N represents the number of spatial points sampled in each grid block, and 10 represents the characteristic dimension of each spatial point. It should be noted that setting the characteristic dimension of the spatial point to ten dimensions is only an example, and those skilled in the art can set the characteristic dimension of the spatial point according to actual needs, and this application does not limit this.
[0052] S202, respectively perform feature extraction on the 3D point cloud data to be detected contained in each grid block to obtain the point cloud features corresponding to each grid block, wherein the features in the z direction of the point cloud features are feature statistics in the z direction of each 3D point cloud data to be detected contained in each grid block.
[0053] The fully connected layer can be used to extract features of the 3D point cloud data to be detected contained in each grid block to obtain the original point cloud features corresponding to each grid block. Figure 4a FIG. 1 is a schematic diagram of a fully connected layer provided by an exemplary embodiment. Figure 4a As shown, the fully connected layer includes three layers of networks: a linear layer (Linear Layer), a batch normalization layer (BatchNormalization Layer, BN) and an activation function layer (Rectified Linear Unit Layer, ReLU). The input of the fully connected layer is the 3D point cloud data to be detected contained in each grid block, and the output is the original point cloud features corresponding to each grid block. In combination with the previous embodiment, after the 3D point cloud data to be detected is converted into a tensor (D, N, 10), the tensor (D, N, 10) is input into the fully connected layer, and the fully connected layer will output a tensor (D, N, M), where M represents the spatial feature dimension of the point cloud feature.
[0054] Then, the original point cloud features of each grid block in the z direction are further processed. The original point cloud features of each grid block in the z direction can be understood as the features of each 3D point cloud data to be detected contained in the grid block in the z direction. Specifically, the statistics of the original point cloud features of each grid block in the z direction are calculated to obtain feature statistics. Feature statistics may include but are not limited to: the average, median or maximum value of the original point cloud features. Then, the feature statistics are used as the features of each grid block in the z direction, thereby obtaining the point cloud features corresponding to each grid block. In this way, while retaining the feature information of the 3D point cloud data to be detected along the z direction, the amount of data that needs to be processed is greatly reduced, which helps to reduce the amount of calculation of subsequent target detection operations.
[0055] In one embodiment, the original point cloud features output by the fully connected layer may be further processed to obtain more comprehensive point cloud feature information. Figure 4b FIG. 1 is a schematic diagram of extracting point cloud features provided by an exemplary embodiment. Figure 4b, first input the 3D point cloud data to be detected contained in any grid block into the fully connected layer. The fully connected layer can be composed of a Linear layer, a BN layer and a ReLU layer. The fully connected layer will output the point cloud features corresponding to any grid block (in this embodiment, the point cloud features are called "original point cloud features"). Then input the original point cloud features corresponding to any grid block into the maximum pooling layer (MaxPooling) to perform maximum pooling processing on the original point cloud features, and output the local aggregation features corresponding to any grid block. Then, the original point cloud features corresponding to any grid block are concatenated with the local aggregation features to obtain the concatenated point cloud features corresponding to any grid block. Calculate the feature statistics of the concatenated point cloud features in the z direction, and use the feature statistics as the features of any grid block in the z direction, so as to obtain the point cloud features corresponding to any grid block.
[0056] Since the original point cloud features corresponding to any grid block reflect the global feature information of the 3D point cloud data to be detected contained in the grid block, and the local aggregate features reflect the local feature information of the 3D point cloud data to be detected contained in the grid block, this embodiment splices the local aggregate features corresponding to any grid block with the original point cloud features to obtain the point cloud features corresponding to any grid block, thereby achieving the simultaneous acquisition of feature information of the 3D point cloud data to be detected from both the global and local perspectives, which helps to obtain more comprehensive feature information to further improve the accuracy of target detection results.
[0057] In one embodiment, before the point cloud features corresponding to the multiple grid blocks are respectively input into the convolutional neural network, the attention weight of each grid block in the multiple grid blocks can be determined by using the self-attention mechanism. The self-attention mechanism is intended to enhance the feature expression of key grid blocks while weakening the feature information of less important grid blocks. Whether any grid block is a key grid block depends on the degree of correlation between it and other grid blocks. The higher the degree of correlation, the more critical the grid block is, so the attention weight corresponding to the grid block will be greater. In other words, the attention weight corresponding to each grid block is positively correlated with the aforementioned degree of correlation.
[0058] In this embodiment, the core of the self-attention mechanism is to define three sets of vectors Q, K, and V, where the query vector Q represents the question for each grid block, the key vector K represents the location information of each grid block, and the value vector V represents the weight value of each grid block. The three sets of vectors Q, K, and V can be obtained by linearly transforming the point cloud features corresponding to each grid block. The principle can be referred to the following formulas (1)-(3):
[0059] Q=XW Q (1)
[0060] K=XW K (2)
[0061] V=XW V (3)
[0062] In the above formulas (1)-(3), X represents the point cloud features corresponding to multiple grid blocks, and W Q The weight matrix representing the query vector Q, W K The weight matrix representing the key vector K, W V A weight matrix representing the value vector V.
[0063] Figure 5 FIG. 1 is a schematic diagram of determining the attention weights corresponding to each grid block provided by an exemplary embodiment. Figure 5 As shown, assume that the point cloud features corresponding to the four grid blocks are G1, G2, G3, and G4. Use the above formulas (1)-(3) to perform linear transformations on the point cloud features G1, G2, G3, and G4, respectively, to obtain the vectors Q1, K1, and V1 corresponding to the point cloud feature G1, the vectors Q2, K2, and V2 corresponding to the point cloud feature G2, the vectors Q3, K3, and V3 corresponding to the point cloud feature G3, and the vectors Q4, K4, and V4 corresponding to the point cloud feature G4. Then perform dot products on the query vector Q corresponding to any point cloud feature and the key vectors K corresponding to other point cloud features. The larger the dot product result, the higher the correlation between the two point cloud features. Combined with Figure 5 Taking the point cloud feature G1 as an example, the query vector Q1 corresponding to the point cloud feature G1 is dot-producted with the key vector K1 corresponding to itself, the key vector K2 corresponding to the point cloud feature G2, the key vector K3 corresponding to the point cloud feature G3, and the key vector K4 corresponding to the point cloud feature G4. The dot product results are g 1,1 ,g 1,2 ,g 1,3 ,g 1,4 The four dot product results are input into the Soft-max layer to normalize the four dot product results based on the Softmax function. Figure 5 In the example, the dot product result g 1,1 The corresponding normalized value is g′ 1,1 , the dot product result g 1,2 The corresponding normalized value is g′ 1,2 , the dot product result g 1,3 The corresponding normalized value is g′ 1,3 , the dot product result g 1,4 The corresponding normalized value is g′ 1,4 Then, each normalized value is multiplied by the value vector of the corresponding point cloud feature, and the four product results are added together to obtain the point cloud feature with attention weight ( Figure 5denoted by B1 in ). So far, the point cloud feature G1 corresponding to a grid block is transformed into the point cloud feature B1 with attention weight. By analogy, the point cloud feature G2 is transformed into the point cloud feature B2 with attention weight, the point cloud feature G3 is transformed into the point cloud feature B3 with attention weight, and the point cloud feature G4 is transformed into the point cloud feature B4 with attention weight.
[0064] Since the attention weight corresponding to each grid block is determined based on the degree of correlation between the grid blocks, the point cloud features with attention weights can cover the contextual information between the grid blocks, thereby helping to improve the accuracy of the target detection results.
[0065] S203: Input the point cloud features corresponding to the multiple grid blocks into a convolutional neural network to obtain a two-dimensional feature map output by the convolutional neural network.
[0066] In one embodiment, the convolutional neural network may include a deconvolution layer and at least two convolution layers with different spatial resolutions, and the at least two convolution layers are in parallel. Figure 6 FIG. 1 is a schematic diagram of a convolutional neural network provided by an exemplary embodiment. Figure 6 As shown in the figure, the convolutional neural network contains four parallel convolutional layers (represented by MBlock1 to MBlock4) and one deconvolution layer, where the deconvolution layer is connected after the four convolutional layers and the spatial resolutions of the four convolutional layers are different. Specifically, the different spatial resolutions of the convolutional layers are achieved by setting different sizes of the convolution kernels. Figure 6 In the figure, the width of the convolution kernels in the convolution layers MBlock1 to MBlock4 is the same, and the depth and height of the convolution kernels can be seen in Table 1:
[0067] Table 1
[0068]
[0069]
[0070] Combined with Table 1, assuming that the convolution step size of the convolution layer MBlock1 is S, the convolution step sizes of the convolution layers MBlock2, MBlock3, and MBlock4 are 2S, 3S, and 4S respectively. The point cloud features corresponding to the multiple grid blocks are input into the convolution layers MBlock1 to MBlock4 respectively, and the feature maps output by the convolution layers MBlock1 to MBlock4 can be obtained. Because the spatial resolutions of the convolution layers MBlock1 to MBlock4 are different, the resolutions of the four feature maps obtained are also different, and feature maps of different resolutions cannot be spliced together, so the feature maps of different resolutions need to be input into the deconvolution layer, so that the deconvolution layer can upsample the feature maps of different resolutions and adjust the sizes of the feature maps of different resolutions to the same size. Combined with Figure 6 , the deconvolution layer can output four feature maps of the same size, and then splice the four feature maps of the same size to output a spliced feature map, which is a two-dimensional feature map. In summary, the input of the convolutional neural network is the point cloud features corresponding to multiple grid blocks, and the output is a two-dimensional feature map.
[0071] In this embodiment, the point cloud features corresponding to the multiple grid blocks are convolved by at least two convolution layers with different spatial resolutions, so that the underlying features and high-level features are extracted from the point cloud features corresponding to the multiple grid blocks at the same time. The underlying features usually refer to local detail information such as contours, textures, and edges, and the high-level features usually refer to overall semantic information. Therefore, the underlying features and the high-level features are spliced, and target detection is performed based on the spliced feature map, which helps to detect targets from local details to overall semantics, thereby improving the accuracy of target detection results.
[0072] In one embodiment, each convolutional layer in the convolutional neural network may include a dilated convolutional network layer. Figure 7a FIG. 1 is a schematic diagram of a convolutional layer structure provided by an exemplary embodiment. Figure 7a As shown, the convolution layer MBlock contains three sublayers: a dilated convolutional network layer, a batch normalization layer (BN layer), and an activation function layer (ReLU layer). Among them, the dilated convolutional network layer is used to perform dilated convolution operations on the input point cloud features. The principle of the dilated convolution operation is to add holes between the convolution kernel elements, thereby increasing the actual size of the convolution kernel, but without increasing the number of parameters. Therefore, the dilated convolution operation can capture a wider range of contextual information between the 3D point cloud data to be detected with a larger receptive field, and thus can improve the accuracy of the target detection results based on a wider range of contextual information. In this embodiment, each convolutional layer may include three superimposed dilated convolutional network layers (refer to Figure 7a ).
[0073] In one embodiment, the convolutional neural network may further include an average pooling layer. Figure 7b , the concatenated feature map can be input into the average pooling layer to perform average pooling on the concatenated feature map. The output of the average pooling layer is the output of the convolutional neural network. By performing average pooling on the concatenated feature map, the dimension of the feature map can be further reduced, so as to reduce the subsequent calculation amount and improve the target detection efficiency. Of course, the maximum pooling layer can also be used instead of the average pooling layer to achieve the effect of reducing the dimension of the feature map.
[0074] S204: Input the two-dimensional feature map to a detection head to obtain a category detection result and a position detection result output by the detection head corresponding to the 3D point cloud data to be detected.
[0075] The detection head (Single Shot MultiBox Detector, SSD) is a key component for detecting objects. Specifically, the detection head is an end-to-end object detection framework that can simultaneously predict the categories and positions of multiple objects in one forward propagation, significantly improving the detection speed. Generally, the detection head detects the input two-dimensional feature map at different scales to obtain multiple layers of feature maps of different sizes. Then, a priori boxes of different scales and aspect ratios are generated on each layer of feature maps. The number and proportion of prior boxes can be adjusted according to the specific application scenario. For each prior box, the detection head predicts four offsets to adjust the position and size of the prior box according to the offsets so that the adjusted prior box can more accurately surround the object. At the same time, the detection head also predicts a category vector representing the possible category probability of each object. After obtaining all the predicted bounding boxes and categories, the detection head uses the NMS algorithm to filter out overlapping bounding boxes and retain only the most likely detection results. The NMS algorithm sorts the candidate boxes according to the confidence score of each category and removes those low-scoring boxes that highly overlap with high-scoring boxes. The detection results output by the detection head include a category detection result and a position detection result. The two detection results are obtained based on the 3D point cloud data to be detected, so the two detection results correspond to the 3D point cloud data to be detected.
[0076] In the above embodiment, the acquired 3D point cloud data to be detected is divided into a plurality of grid blocks, which helps to capture the relationship between each 3D point cloud data to be detected in the horizontal direction (i.e., the z direction) and avoid the loss of spatial information. Then, feature extraction is performed on the 3D point cloud data to be detected contained in each grid block to obtain the point cloud features corresponding to each grid block, and the features in the z direction of the point cloud features are the feature statistics of each 3D point cloud data to be detected contained in each grid block in the z direction. This method greatly reduces the amount of data of feature information in the z direction of each grid block, and can effectively reduce the subsequent amount of calculation and required memory. Target detection is then performed based on the point cloud features corresponding to each grid block, which can reduce the amount of calculation and memory while retaining the spatial information of the 3D point cloud data to be detected in the horizontal direction, thereby effectively improving the accuracy of the final target detection result.
[0077] In the above embodiments, the present invention introduces a solution for target detection based on 3D point cloud data. In the specific implementation process, a target detection model can be established and trained, so as to efficiently and accurately implement the above target detection process based on the target detection model. For example, the target detection model may include a point cloud feature processing layer, a convolutional neural network layer, and a detection head. Figure 8a ,in:
[0078] The point cloud feature processing layer is used to divide the 3D point cloud data to be detected into multiple grid blocks, and perform feature extraction on the 3D point cloud data to be detected contained in each grid block, and output the point cloud features corresponding to each grid block. The features in the z direction of the point cloud features are the feature statistics of each 3D point cloud data to be detected contained in each grid block in the z direction.
[0079] The convolutional neural network layer is used to process the point cloud features corresponding to multiple grid blocks and output a two-dimensional feature map;
[0080] The detection head is used to perform target detection on the two-dimensional feature map and output the category detection results and position detection results of the 3D point cloud data to be detected.
[0081] For the above target detection model, this application also proposes a corresponding model training scheme. Figure 8b For a detailed introduction:
[0082] Figure 8b FIG. 1 is a flowchart of a method for training a target detection model provided by an exemplary embodiment. Figure 8b As shown, the method may include the following steps:
[0083] S801. Obtain sample 3D point cloud data to be detected and the sample category and sample position corresponding to the sample 3D point cloud data to be detected.
[0084] In this embodiment, the sample category and the sample position refer to the category and position of the target contained in the sample 3D point cloud data to be detected, respectively, and the sample category and the sample position are both known and determined. The method for obtaining the sample 3D point cloud data to be detected can refer to the above embodiment S201, which will not be repeated here.
[0085] S802. Input the sample 3D point cloud data to be detected into a target detection model, so that the target detection model divides the sample 3D point cloud data to be detected into multiple grid blocks, and extracts sample point cloud features corresponding to the sample 3D point cloud data to be detected contained in each grid block, and processes the sample point cloud features corresponding to the multiple grid blocks respectively by the convolutional neural network layer in the target detection model to obtain a sample two-dimensional feature map corresponding to the sample 3D point cloud data to be detected, and performs target detection on the sample two-dimensional feature map by the detection head in the target detection model, and outputs the sample category detection result and the sample position detection result corresponding to the sample 3D point cloud data to be detected; wherein, the sample features in the z direction in the sample point cloud features are the sample feature statistics in the z direction of each sample 3D point cloud data to be detected contained in each grid block.
[0086] Input the 3D point cloud data of the sample to be detected Figure 8a The target detection model shown in the figure, the point cloud feature processing layer in the target detection model will divide the sample 3D point cloud data to be detected into multiple grid blocks, and extract the sample point cloud features corresponding to the sample 3D point cloud data to be detected contained in each grid block. The sample features in the z direction of the sample point cloud features corresponding to each grid block are the sample feature statistics in the z direction of each sample 3D point cloud data to be detected contained in the grid block. Then, the sample point cloud features corresponding to the multiple grid blocks are respectively input into the convolutional neural network layer, so that the convolutional neural network layer processes the sample point cloud features corresponding to the multiple grid blocks respectively, and outputs the sample two-dimensional feature map corresponding to the sample 3D point cloud data to be detected. Then, the sample two-dimensional feature map is subjected to target detection by the detection head in the target detection model, and the sample category detection result and the sample position detection result corresponding to the sample 3D point cloud data to be detected are output.
[0087] S803. Calculate a category loss value according to the sample category and the sample category detection result, and calculate a position loss value according to the sample position and the sample position detection result, and iteratively train the target detection model based on the category loss value and the position loss value.
[0088] In this embodiment, the category loss value can be calculated using the cross entropy loss function. The calculation principle can refer to formula (4):
[0089] Loss class=-δ a (1-p a ) β logp a (4)
[0090] In formula (4), Loss class Represents the category loss value; δ a Represents the balance factor, which is used to adjust the loss weights of different categories; p a Characterizes the sample category detection result, which is used to represent the probability of belonging to category a predicted by the target detection model; β represents the modulation factor, which is used to reduce the contribution of easy-to-classify samples and increase the contribution of difficult-to-classify samples.
[0091] The position loss value can be calculated using the SmoothL2 loss function. The calculation principle can be referred to formula (5):
[0092]
[0093] In formula (5), Loss local represents the position loss value; b represents the parameters of the prior frame, including the coordinates (x, y, z) of the center point of the prior frame, the size of the prior frame (w, l, h) and the rotation angle θ; Δb represents the difference between the sample position and the sample position detection result.
[0094] In addition, in three-dimensional object detection, the orientation of the object is an important information, so the direction loss value of the object can also be calculated. The direction loss value is used to characterize the difference between the sample direction detection result output by the object detection model and the sample direction (i.e., the actual direction of the sample), and can be calculated using the cross entropy loss or a similar loss function. In summary, the loss function in this embodiment can be designed as a weighted sum of the three losses of category loss, position loss, and direction loss. The specific principle can be referred to formula (6):
[0095]
[0096] In formula (6), L represents the total loss value, N pos Characterize the number of positive samples, Loss local Characterizes the position loss value, Loss class Characterize the category loss value, Loss head Characterizes the direction loss value, γ local The weight coefficient representing the directional loss value, γ class The weight coefficient that represents the category loss value, γ head The weight coefficient that represents the directional loss value.
[0097] After multiple iterations of training, when the total loss value is less than the preset loss value or the predefined number of iterations is reached, the training of the target detection model is completed and can be used to achieve the following Figure 2 The target detection scheme shown.
[0098] Fig. 9 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application. Fig. 9 At the hardware level, the electronic device includes a processor 902, an internal bus 904, a network interface 906, a memory 908, and a non-volatile memory 910, and may also include hardware required for other services. The processor 902 reads the corresponding computer program from the non-volatile memory 910 into the memory 908 and then runs it. Of course, in addition to the software implementation, this application does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0099] Corresponding to the above method embodiments, the present application also provides an embodiment of a device for performing target detection based on 3D point cloud data.
[0100] Fig.10 is a block diagram of a device for performing target detection based on 3D point cloud data according to an exemplary embodiment of the present application. Fig.10 The device includes an acquisition unit 1002, a feature extraction unit 1004, a convolution unit 1006 and a detection result acquisition unit 1008, wherein:
[0101] The acquisition unit 1002 is configured to acquire the 3D point cloud data to be detected and divide the 3D point cloud data to be detected into a plurality of grid blocks.
[0102] The feature extraction unit 1004 is configured to perform feature extraction on the 3D point cloud data to be detected contained in each grid block respectively, and obtain the point cloud features corresponding to each grid block, wherein the features in the z direction of the point cloud features are the feature statistics of each 3D point cloud data to be detected contained in each grid block in the z direction.
[0103] The convolution unit 1006 is configured to input the point cloud features corresponding to the multiple grid blocks into the convolutional neural network to obtain a two-dimensional feature map output by the convolutional neural network.
[0104] The detection result acquisition unit 1008 is configured to input the two-dimensional feature map into the detection head to obtain the category detection result and the position detection result corresponding to the 3D point cloud data to be detected output by the detection head.
[0105] Optionally, the feature extraction unit 1004 is specifically used to: input the 3D point cloud data to be detected contained in any one of the grid blocks into the fully connected layer to obtain the original point cloud features corresponding to any one of the grid blocks output by the fully connected layer; perform maximum pooling processing on the original point cloud features corresponding to any one of the grid blocks to obtain local aggregation features corresponding to any one of the grid blocks; splice the original point cloud features with the local aggregation features, and calculate the feature statistics of the spliced point cloud features in the z direction to obtain the point cloud features corresponding to any one of the grid blocks.
[0106] Optionally, the device further comprises:
[0107] The attention unit 1010 is configured to calculate the correlation between each grid block and other grid blocks respectively by using a self-attention mechanism, and determine the attention weight corresponding to each grid block respectively according to the calculation result, so as to obtain the point cloud features with attention weights corresponding to each grid block; wherein the attention weight is positively correlated with the correlation degree;
[0108] The convolution unit 1006 is specifically used to input the point cloud features with attention weights corresponding to the multiple grid blocks into the convolutional neural network.
[0109] Optionally, the convolutional neural network includes a deconvolution layer and at least two parallel convolution layers with different spatial resolutions, wherein the at least two convolution layers are used to extract at least two feature maps with different resolutions from the point cloud features corresponding to the multiple grid blocks respectively; the deconvolution layer is used to adjust the sizes of the at least two feature maps with different resolutions to the same size, so as to perform feature fusion on at least two feature maps of the same size to obtain the two-dimensional feature map.
[0110] Optionally, each convolutional layer includes a dilated convolutional network layer.
[0111] Optionally, the convolutional neural network also includes an average pooling layer, which is used to perform average pooling processing on the feature map after feature fusion to output the two-dimensional feature map.
[0112] Fig.11 is a block diagram of a training device for a target detection model according to an exemplary embodiment of the present application. Fig.11 , the device includes a sample acquisition unit 1102, an input unit 1104 and an iteration unit 1106, wherein:
[0113] The sample acquisition unit 1102 is configured to acquire sample 3D point cloud data to be detected and a sample category and a sample position corresponding to the sample 3D point cloud data to be detected.
[0114] The input unit 1104 is configured to input the sample 3D point cloud data to be detected into the target detection model, so that the target detection model divides the sample 3D point cloud data to be detected into multiple grid blocks, and extracts the sample point cloud features corresponding to the sample 3D point cloud data to be detected contained in each grid block, and processes the sample point cloud features corresponding to the multiple grid blocks respectively by the convolutional neural network layer in the target detection model to obtain a sample two-dimensional feature map corresponding to the sample 3D point cloud data to be detected, and the detection head in the target detection model performs target detection on the sample two-dimensional feature map, and outputs the sample category detection result and the sample position detection result corresponding to the sample 3D point cloud data to be detected; wherein, the sample features in the z direction in the sample point cloud features are the sample feature statistics of each sample 3D point cloud data to be detected contained in each grid block in the z direction.
[0115] The iteration unit 1106 is configured to calculate a category loss value according to the sample category and the sample category detection result, and to calculate a position loss value according to the sample position and the sample position detection result, and to iteratively train the target detection model based on the category loss value and the position loss value.
[0116] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.
[0117] The devices or modules described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0118] In a typical configuration, a computer includes one or more processors, including a central processing unit (CPU) and a graphics processing unit (GPU), an input / output interface, a network interface, and a memory. The CPU is used for computing simulations, and the GPU is used for outputting high-quality 3D images.
[0119] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0120] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.
[0121] Corresponding to the embodiments of the aforementioned method, the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of any embodiment of the aforementioned method are implemented.
[0122] Corresponding to the embodiments of the aforementioned method, the present application also provides a computer program product, including a computer program and / or instructions, which implement the steps of any embodiment of the aforementioned method when executed by a processor.
[0123] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.
Claims
1. A method for target detection based on 3D point cloud data, characterized in that: include: Acquire 3D point cloud data to be detected, and divide the 3D point cloud data to be detected into a plurality of grid blocks; Performing feature extraction on the 3D point cloud data to be detected contained in each grid block respectively, to obtain the point cloud features corresponding to each grid block, wherein the features in the z direction of the point cloud features are feature statistics of each 3D point cloud data to be detected contained in each grid block in the z direction; Inputting the point cloud features corresponding to the plurality of grid blocks into a convolutional neural network to obtain a two-dimensional feature map output by the convolutional neural network; The two-dimensional feature map is input into a detection head to obtain a category detection result and a position detection result corresponding to the 3D point cloud data to be detected output by the detection head.
2. The method according to claim 1, characterized in that Feature extraction is performed on the 3D point cloud data to be detected contained in any grid block to obtain point cloud features corresponding to any grid block, including: Inputting the 3D point cloud data to be detected contained in any grid block into a fully connected layer, and obtaining the original point cloud features corresponding to any grid block output by the fully connected layer; Performing maximum pooling processing on the original point cloud features corresponding to any grid block to obtain local aggregation features corresponding to any grid block; The original point cloud features are spliced with the local aggregation features, and feature statistics of the spliced point cloud features in the z direction are calculated to obtain the point cloud features corresponding to any grid block.
3. The method according to claim 1, characterized in that: Also includes: The degree of correlation between each grid block and other grid blocks is calculated by using a self-attention mechanism, and the attention weight corresponding to each grid block is determined according to the calculation result, so as to obtain a point cloud feature with an attention weight corresponding to each grid block; wherein the attention weight is positively correlated with the degree of correlation; The step of inputting the point cloud features respectively corresponding to the plurality of grid blocks into the convolutional neural network comprises: inputting the point cloud features respectively corresponding to the plurality of grid blocks with attention weights into the convolutional neural network.
4. The method according to claim 1, characterized in that The convolutional neural network comprises a deconvolution layer and at least two parallel convolutional layers with different spatial resolutions, wherein: At least two convolutional layers are used to extract at least two feature maps of different resolutions from the point cloud features corresponding to the multiple grid blocks respectively; The deconvolution layer is used to adjust the sizes of the at least two feature maps with different resolutions to the same size, so as to perform feature fusion on the at least two feature maps of the same size to obtain the two-dimensional feature map.
5. The method according to claim 4, characterized in that Each convolutional layer includes a dilated convolutional network layer.
6. The method according to claim 4, characterized in that The convolutional neural network also includes an average pooling layer, which is used to perform average pooling processing on the feature map after feature fusion to output the two-dimensional feature map.
7. A method for training a target detection model, characterized in that: include: Acquire sample 3D point cloud data to be detected and a sample category and a sample position corresponding to the sample 3D point cloud data to be detected; The sample 3D point cloud data to be detected is input into a target detection model, so that the target detection model divides the sample 3D point cloud data to be detected into a plurality of grid blocks, and extracts sample point cloud features corresponding to the sample 3D point cloud data to be detected contained in each grid block, and processes the sample point cloud features corresponding to the plurality of grid blocks respectively by a convolutional neural network layer in the target detection model to obtain a sample two-dimensional feature map corresponding to the sample 3D point cloud data to be detected, and performs target detection on the sample two-dimensional feature map by a detection head in the target detection model, and outputs a sample category detection result and a sample position detection result corresponding to the sample 3D point cloud data to be detected; wherein, the sample features in the z direction in the sample point cloud features are sample feature statistics in the z direction of each sample 3D point cloud data to be detected contained in each grid block; A category loss value is calculated according to the sample category and the sample category detection result, and a position loss value is calculated according to the sample position and the sample position detection result, and the target detection model is iteratively trained based on the category loss value and the position loss value.
8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 7 by running the executable instructions.
9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer program product, characterized in that The method comprises a computer program and / or instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.
