Deep learning-based point cloud salient target identification method

Abstract

The invention discloses a point cloud salient target recognition method based on deep learning, and relates to the technical field of point cloud salient target recognition, and the technical key points comprise the steps: (1) carrying out the preprocessing of training data, including the normalization of point cloud coordinates and the addition of noise points; (2) performing voxelization on the point cloud and obtaining sparse voxel data; (3) constructing a sparse convolution U-shaped neural network model; (4) constructing a loss function of the network by using the improved Dice Loss based on distance information weighting and Cross-Entropy Loss; (5) training the constructed network through the training data set; and (6) predicting a salient target in the test data set by using the trained network. According to the method for optimizing the salient target edge of the point cloud, the problem of edge blurring existing in a salient target recognition task in the current point cloud can be effectively solved, the salient target in the point cloud can be quickly and accurately recognized and segmented, and the segmentation precision and efficiency of the salient target of the point cloud are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud salient object recognition, and in particular to a point cloud salient object recognition method based on deep learning. Background Art

[0002] Salient Object Detection (SOD) aims to segment the most attractive object from the surrounding background. As a preprocessing procedure, SOD has wide applications in many downstream tasks, such as semantic segmentation, object detection, and visual tracking. Much research has been devoted to image-based salient object detection, and many well-known works have emerged.

[0003] Recently, with the rapid development of 3D sensors, such as LiDAR and Kinect cameras, it has become more convenient to obtain 3D data, and point cloud-based applications have become more and more common. However, there are still few works focusing on salient object detection in 3D point clouds. Due to the irregularity and sparsity of point cloud data, it is difficult to extend the methods designed for regular 2D images to point clouds. Therefore, there are still some unsolved problems in the field of salient object detection in point clouds.

[0004] Due to the irregular and sparse characteristics of point cloud data, processing point cloud data related tasks becomes extremely challenging. Traditional methods rely on manually designed features and machine learning algorithms to process point cloud data, but these methods are often limited by feature expression capabilities and algorithm complexity.

[0005] In recent years, with the development of deep learning technology, some classic methods and models have emerged. However, since point cloud data consists of a large number of irregularly distributed points, and salient object detection in point clouds needs to consider factors such as spatial distribution, density and point cloud shape, these classic methods have low accuracy in point cloud salient object recognition tasks. Summary of the invention

[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a point cloud salient target recognition method based on deep learning. By constructing a U-shaped neural network model, distance-weighted Dice Loss and Cross-Entropy Loss are introduced to supervise the model training process, so as to realize the recognition and segmentation of salient targets in point cloud data, and provide more accurate preprocessing results for many downstream tasks.

[0007] To achieve the above object, the present invention provides the following technical solution: a point cloud salient object recognition method based on deep learning, comprising the following steps:

[0008] (1) Obtain the original point cloud data and the manual annotation results of the target area from the point cloud public dataset to construct the training dataset and test dataset;

[0009] (2) Preprocessing the acquired data, including normalizing the point cloud coordinates and adding noise points;

[0010] (3) Voxelize the point cloud and obtain sparse voxel data.

[0011] (4) Network construction: Construct a sparse convolutional U-shaped neural network model that can locate the location of the salient target and segment the target and background.

[0012] (5) Construct loss function: Combine the improved Dice Loss and Cross-Entropy Loss to construct the network loss function:

[0013] Cross-Entropy Loss is used to supervise the point-level category prediction of the model output, and the improved Dice Loss is used to supervise the point-level mask of the model output. The characteristic of the improved Dice Loss is that the weight is calculated according to the distance from the point to the nearest target point, and the weight of the points around the edge of the target is increased.

[0014] (6) Model training: The training data set is input into the U-shaped network in step (4), and supervised training is performed using the loss function constructed in step (5) until the loss function value converges.

[0015] (7) Model reasoning: The test data set in step (1) is sequentially input into the U-shaped neural network trained in step (6) for reasoning to obtain the result of salient object recognition.

[0016] Preferably, the step (4) is specifically implemented as follows:

[0017] The sparse convolutional U-shaped neural network model consists of a total of 5 layers. At each level, the network includes an encoder layer, a skip connection layer, and a decoder layer. At the same time, the front end of the network contains an initial input convolution layer, and the end contains a normalization layer, an activation layer, and a fully connected layer.

[0018] Each encoder layer consists of n residual blocks and a downsampling convolution layer. The residual block includes a batch normalization layer, an activation layer, and a 3*3*3 convolution layer, which will not change the size of the feature dimension. The downsampling convolution layer increases the feature dimension and reduces the spatial resolution by half. The number of residual blocks n is set to 2.

[0019] Each decoder layer consists of an upsampling deconvolution and n residual blocks. The upsampling deconvolution layer reduces the feature dimension while doubling the spatial resolution. The residual block includes a batch normalization layer, an activation layer, and a convolution layer of size 3*3*3. The convolution layer in the first residual block reduces the feature dimension obtained by jump-connecting each layer of the decoder with the corresponding encoder output by half, and the subsequent residual blocks will not change the feature dimension size. The number of residual blocks n is set to 2. In the entire U-shaped neural network, the input of the deconvolution layer in the first decoder is the output of the fifth encoder, and the input of the remaining decoders is the output of the previous decoder.

[0020] The skip connection concatenates the features obtained from each layer output of the encoder with the upsampled results of the decoder at its corresponding layer.

[0021] Preferably, the step (5) is specifically implemented as follows:

[0022]

[0023] Among them, L CE represents Cross-Entropy Loss, G i is the result of manually annotating the i-th point, 1 represents the target point, 0 represents the background point, y i The probability that the i-th point predicted by the model belongs to the target point.

[0024]

[0025] Among them, L Dice represents the improved Dice Loss, one_hot(G i ) indicates that the result of manually annotated i-th point is one-hot encoded, background point 0 is represented by 10, and target point 1 is represented by 01; P i The predicted probability of the i-th point on the background and target predicted by the model. ω i It is the reciprocal of the distance from point i to the nearest target point.

[0026] Preferably, in step (5), the weighted loss function of Dice Loss and Cross-Entropy Loss is:

[0027] Loss = β * L CE +(1-β)*L Dice

[0028] Among them, Loss represents the total loss function of the network, L CE represents Cross-Entropy Loss, L Dice represents the improved Dice Loss, β represents the weight parameter, and β is set to 0.5;

[0029] Preferably, in step (6), after the training data set is input into the neural network model in step (4), the Adam optimizer is used and the learning rate is set to 0.001 for training.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] Compared with the existing point cloud salient target recognition method, the present invention is beneficial to sparse convolutional U-shaped neural network for salient target recognition and segmentation, and combines improved Dice Loss and Cross-Entropy Loss to supervise network training, wherein the improved Dice Loss calculates the weight of a point according to the distance from the point to the nearest target point, increases the weight of the target edge point to obtain clearer target edge segmentation, and improves the accuracy of point cloud salient target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the encoder structure in the U-shaped network structure in an embodiment of the present invention;

[0033] Figure 2 A schematic diagram of a decoder structure in a U-shaped network structure according to an embodiment of the present invention;

[0034] Figure 3 A schematic diagram of an overall U-shaped network structure according to an embodiment of the present invention;

[0035] Figure 4 Schematic diagram of Example 1 of a point cloud salient target result obtained in an embodiment of the present invention;

[0036] Figure 5 Schematic diagram of Example 2 of a point cloud salient target result obtained in an embodiment of the present invention;

[0037] Figure 6 Schematic diagram of Example 3 of point cloud salient target result obtained according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] Reference Figures 1 to 6 An embodiment of a point cloud salient object recognition method based on deep learning of the present invention is further described.

[0039] The technical solution adopted by the present invention is as follows:

[0040] (1) We obtain 2000 point cloud data containing salient objects and the corresponding manual annotation results from the point cloud public dataset to construct a training dataset A containing the original point cloud data and the manual segmentation results of salient objects. At the same time, we obtain another 872 point cloud data containing salient objects to construct a test dataset B.

[0041] (2) Perform data preprocessing on the training dataset A and the test dataset B, including normalizing the point cloud coordinates and adding noise points.

[0042] (2-a) In order to eliminate the magnitude differences between point coordinates, point cloud data normalization is to unify the coordinate values ​​of the point cloud to a specific range, including: subtracting the smallest coordinate value in the scene from each x-coordinate value, and then dividing it by the x-coordinate range to convert the original x-coordinate value to the range of 0 to 1. Similarly, the y and z coordinate normalization results are obtained, and finally the normalized results are added to the original features to enrich the original point cloud features.

[0043] (2-b) In order to increase the diversity and robustness of the data, random noise in the x, y, and z directions is generated from a normal distribution with a mean of 0 and a standard deviation of [3, 3, 3], and then added to the first three coordinate values ​​of the point cloud data points to achieve random translation of the point cloud data.

[0044] Voxelize the point cloud and obtain voxel features.

[0045] (3-a) Point cloud voxelization: Voxelization is the process of converting irregular point clouds into regular voxel form. According to the scale of the voxel, the points in the point cloud data are mapped to the corresponding voxel grid. Sparse quantization is achieved by calculating the index of the point in each dimension, and the index and coordinates of the point cloud data in each voxel are obtained, which is convenient for subsequent processing and analysis.

[0046] (3-b) Obtain voxel features: First, preprocess the point cloud features within each voxel, including calculating the mean value, normalized coordinates, and center-to-point distance of the point cloud within the voxel. Then, a simple feedforward neural network is used to perform nonlinear transformation and feature extraction on the preprocessed point cloud features to obtain the feature representation within each voxel; aggregate each voxel feature to obtain the aggregated feature of each voxel; finally, the aggregated feature and voxel coordinate information are integrated into a sparse tensor representation.

[0047] (4) Construct a sparse convolutional U-shaped neural network model that can locate the location of salient targets and segment the target and background.

[0048] The sparse convolutional U-shaped neural network model contains a total of 5 layers. At each level, the network contains an encoder layer, a skip connection layer, and a decoder layer. At the same time, the network contains an initial input convolution layer at the front end and a normalization layer, an activation layer, and a fully connected layer at the end.

[0049] Each encoder layer consists of n residual blocks and a downsampling convolution layer. The residual block includes a batch normalization layer, an activation layer, and a 3*3*3 convolution layer, which will not change the size of the feature dimension. The downsampling convolution layer increases the feature dimension and reduces the spatial resolution by half. The number of residual blocks n is set to 2.

[0050] Each decoder layer consists of an upsampling deconvolution and n residual blocks. The upsampling deconvolution layer reduces the feature dimension while doubling the spatial resolution. The residual block includes a batch normalization layer, an activation layer, and a convolution layer of size 3*3*3. The convolution layer in the first residual block reduces the feature dimension obtained by jump-connecting each layer of the decoder with the corresponding encoder output by half, and the subsequent residual blocks will not change the feature dimension size. The number of residual blocks n is set to 2. In the entire U-shaped neural network, the input of the deconvolution layer in the first decoder is the output of the fifth encoder, and the input of the remaining decoders is the output of the previous decoder.

[0051] The skip connection concatenates the features obtained from each layer output of the encoder with the upsampled results of the decoder at its corresponding layer.

[0052] (5) Combine the improved Dice Loss and Cross-Entropy Loss to construct the network loss function:

[0053] The weighted loss function of Dice Loss and Cross-Entropy Loss is:

[0054] Loss = β * L CE +(1-β)*L Dice

[0055] L CE =-∑ i G i log(y i )

[0056] Among them, G i is the result of manually annotating the i-th point, 1 represents the target point, 0 represents the background point, y i The probability that the i-th point predicted by the model belongs to the target point.

[0057]

[0058] Among them, L Dice represents the improved Dice Loss, one_hot(G i ) indicates that the result of manually annotated i-th point is one-hot encoded, background point 0 is represented by 10, and target point 1 is represented by 01; P iThe predicted probability of the i-th point on the background and target predicted by the model. ω i It is the reciprocal of the distance from point i to the nearest target point.

[0059] (6) The U-shaped network is trained using the training dataset A and supervised by the constructed loss function until the loss converges.

[0060] (7) The test data set B is sequentially input into the network trained in step (6) for inference to obtain the result of salient object recognition.

[0061] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for identifying salient objects in point clouds based on deep learning, characterized in that: The following steps are involved: (1) Obtain the original point cloud data and the manual annotation results of the target area from the point cloud public dataset to construct the training dataset and test dataset; (2) Preprocessing the acquired data, including normalizing the point cloud coordinates and adding noise points; (3) voxelize the point cloud and obtain sparse voxel data; (4) Network construction: A sparse convolutional U-shaped neural network model is constructed, which can locate the location of the salient target and segment the target and background; (5) Construct loss function: Combine the improved Dice Loss and Cross-Entropy Loss to construct the network loss function: Cross-Entropy Loss is used to supervise the point-level category prediction of the model output, and the improved Dice Loss is used to supervise the point-level mask output of the model. The improved Dice Loss calculates the weight based on the distance from the point to the nearest target point, increasing the weight of the points around the edge of the target. (6) Model training: Input the training data set into the U-shaped neural network model in step (4), and use the loss function constructed in step (5) for supervised training until the loss function value converges; (7) Model reasoning: The test data set in step (1) is sequentially input into the U-shaped neural network trained in step (6) for reasoning to obtain the result of salient target recognition.

2. The method for identifying salient point cloud objects based on deep learning according to claim 1, characterized in that: The step (4) is specifically implemented as follows: The sparse convolutional U-shaped neural network model includes a total of 5 layers. At each level, the network includes an encoder layer, a skip connection layer, and a decoder layer. At the same time, the front end of the network includes an initial input convolution layer, and the end includes a normalization layer, an activation layer, and a fully connected layer. Each encoder layer consists of n residual blocks and a downsampling convolution layer. The residual block includes a batch normalization layer, an activation layer, and a convolution layer of size 3*3*3. The number of residual blocks n is set to 2. The decoder layer consists of an upsampling deconvolution and n residual blocks. The residual block includes a batch normalization layer, an activation layer, and a convolution layer of size 3*3*3. The number of residual blocks n is set to 2. In the entire U-shaped neural network, the input of the deconvolution layer in the first decoder is the output of the fifth encoder, and the input of the remaining decoders is the output of the previous decoder. The skip connection concatenates the features obtained from each layer output of the encoder with the upsampled results of the decoder at its corresponding layer.

3. The method for identifying salient point cloud objects based on deep learning according to claim 1, characterized in that: The step (5) is specifically implemented as follows: Among them, L CE represents Cross-Entropy Loss, G i is the result of manually annotating the i-th point, 1 represents the target point, 0 represents the background point, y i The probability that the i-th point predicted by the model belongs to the target point; Among them, L Dice represents the improved Dice Loss, one_hot(G i ) indicates that the result of manually annotated i-th point is one-hot encoded, background point 0 is represented by 10, and target point 1 is represented by 01; P i is the predicted probability of the i-th point on the background and target predicted by the model, ω i It is the reciprocal of the distance from point i to the nearest target point.

4. The method for identifying salient point cloud objects based on deep learning according to claim 1, characterized in that: In step (5), the weighted loss function of Dice Loss and Cross-Entropy Loss is: Loss=β*L CE +(1-β)*L Dice Among them, Loss represents the total loss function of the network, L CE represents Cross-Entropy Loss, L Dice represents the improved Dice Loss, β represents the weight parameter, and β is set to 0.

5.

5. The method for identifying salient point cloud objects based on deep learning according to claim 1, characterized in that: In step (6), after the training data set is input into the neural network model in step (4), the Adam optimizer is used and the learning rate is set to 0.001 for training.

Images