Convolutional neural network-based point cloud fragmentation downsampling method and system

Abstract

The invention provides a point cloud fragment downsampling method and system based on a convolutional neural network, electronic equipment and a storage medium, and aims to solve the problem that the point cloud quality after sampling is poor due to the fact that the aggregation degree in a sampling algorithm slice is poor and the whole point cloud cannot be completely covered. The method comprises the following steps: acquiring three-dimensional point cloud data, setting the number of divided slices, initializing a cluster center according to the number of the slices, and dividing each piece of data; re-calculating a new cluster center, and re-dividing a part of data points according to a preset standard; repeating the above steps until the algorithm meets a termination condition, and obtaining a to-be-processed point cloud slice; sending the obtained to-be-processed point cloud slice into a down-sampling network to obtain a point cloud slice after down-sampling; and slicing the point cloud after down-sampling, and splicing into a complete down-sampling point cloud. According to the invention, a more vivid down-sampling point cloud can be generated.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a point cloud segmentation down-sampling method based on a convolutional neural network, a point cloud segmentation down-sampling system based on a convolutional neural network, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the continuous improvement of 3D point cloud capture and scanning performance and the continuous development of deep learning technology, 3D point clouds have been widely used in fields such as virtual reality and autonomous driving. At the same time, the huge 3D data and increasingly complex deep learning algorithms have also posed challenges to the storage of hardware devices and the processing of computers.

[0003] Point cloud downsampling technology is to select more robust or valuable points from the original point cloud to retain, so as to reduce the overall number of point clouds and thus reduce the complexity of point cloud processing. For example, the existing convolutional neural network-based downsampling method requires about 1 to 2GB of video memory to process 1,000 points. However, the 3D point cloud scanned in the actual scene generally has 200,000 to 2 million points, which conservatively estimates that more than 200GB of video memory is required. Directly processing 3D point clouds based on convolutional neural networks directly poses severe challenges to hardware devices, especially in the two key aspects of memory capacity and data processing speed.

[0004] At present, as the complexity of deep learning networks increases, many network architectures cannot directly process large-scale point clouds. At the same time, for a trained network model, the number of input points that can be supported is often fixed. This requires the design of a point cloud slicing algorithm to first slice the large-scale point cloud and then feed it into the neural network for processing. However, various current sampling methods, such as farthest point sampling and K-nearest neighbor point cloud slicing algorithms, have poor aggregation within the sampling algorithm slices, and cannot achieve full coverage of the entire point cloud, which will lead to worse point cloud quality after sampling. Summary of the invention

[0005] In order to at least solve the problem in the prior art that the degree of aggregation within the sampling algorithm slice is poor, the entire point cloud cannot be fully covered, and thus the point cloud quality after sampling is poor. The present invention provides a point cloud slice downsampling method based on a convolutional neural network, a point cloud slice downsampling system based on a convolutional neural network, an electronic device, and a computer-readable storage medium. The sampling process can perfectly cover the original point cloud without repetition, and the degree of aggregation within the slice is higher, so that a more realistic downsampled point cloud can be generated.

[0006] In a first aspect, the present disclosure provides a point cloud fragmentation downsampling method based on a convolutional neural network, the method comprising:

[0007] S1: Obtain 3D point cloud data, set the number of slices, initialize the cluster center according to the number of slices, and divide each data;

[0008] S2: Recalculate new cluster centers and re-divide some data points according to preset standards;

[0009] S3: Repeat step S2 until the algorithm meets the termination condition and obtains the point cloud slice to be processed;

[0010] S4: Send the obtained point cloud slices to be processed into the downsampling network to obtain downsampled point cloud slices;

[0011] S5: Slice the downsampled point cloud and stitch it into a complete downsampled point cloud.

[0012] Furthermore, the three-dimensional point cloud data is obtained, the number of slices is set, the cluster center is initialized according to the number of slices, and each data is divided, including:

[0013] For the acquired 3D point cloud data, use {p i |i=1,…,n} represents the 3D unordered point cloud to be sliced, where each point has three channel features (x, y, z);

[0014] Set the number of slices that need to be divided into k slices, each slice has n patch points;

[0015] The FPS (Farthest Point Sampling) method is used to randomly initialize k cluster centers. For each data point in the data set, its distance to each cluster center is calculated; then, the data point is assigned to the cluster with the cluster center closest to it.

[0016] Furthermore, the recalculation of the new cluster center and the re-division of some data points according to a preset standard include:

[0017] Calculate the number of points in each cluster, find the cluster with the most points, set it as cluster c, and satisfy Its cluster center Defined as:

[0018] x mid =average(x1,...,x l ) (1)

[0019] y mid =average(y1,...,y l ) (2)

[0020] z mid=average(z1,...,z l ) (3)

[0021] Then calculate the distance from each point to the cluster center Select the point with the longest distance from the cluster center satisfy:

[0022]

[0023] calculate The distance to the center of other clusters is sorted from small to large, and the cluster corresponding to the smallest distance is selected. If the number of points in the cluster is less than n patch , then move the point from cluster c to this cluster, otherwise, ignore this distance and its corresponding cluster, judge the next distance, until the first cluster that meets the conditions is found, and move the point from cluster c to this cluster.

[0024] Furthermore, the algorithm satisfies the termination condition including:

[0025] Each cluster contains n points. patch .

[0026] Furthermore, the step of sending the obtained point cloud slices to be processed to a downsampling network to obtain downsampled point cloud slices includes:

[0027] The obtained point cloud slices to be processed are sent to the preset downsampling network. The input of the network is a patch The output is a slice point cloud with m points. patch Slice point cloud of points;

[0028] The preset downsampling network is used to extract the features of the point cloud through several convolutional layers, and scores the contribution value of each point, and selects points with high scores to form the downsampled point cloud.

[0029] Furthermore, the preset downsampling network sampling process includes:

[0030] The original point cloud slices are processed in the following ways:

[0031] F1=conv1(P in ) (5)

[0032] F2=conv2(F1) (6)

[0033] F mat =conv3(F2) (7)

[0034] F4=conv4(F mat ·P in ) (8)

[0035] F5=conv5(F4) (9)

[0036] F sort =sort(F5) (10)

[0037] idx = get_index(F sort ) (11)

[0038] P sp =select_pt(idx,P in ) (12)

[0039] P in Represents the original point cloud slice, conv1 represents the first layer of convolution, F1 is the feature matrix output by this layer, conv2 represents the second layer of convolution, F2 is the feature matrix output by this layer, conv3 represents the third layer of convolution, f mat Output the transformation matrix for this layer, and then use the transformation matrix F mat And the input point cloud dot multiplication, conv4 represents the fourth layer of convolution, F4 represents the fourth layer of convolution output features, conv5 represents the fifth layer of convolution, F5 represents the fifth layer of convolution output features, sort represents the sorting of features, F sort Indicates the sorting result. get_index indicates selecting the first m with larger values. patch elements, idx represents the queried index value, select_pt represents the selection based on the index value and the original point cloud, P sp Represents a downsampled point cloud slice.

[0040] Furthermore, the method further comprises:

[0041] The preset downsampling network model is trained and tested, including:

[0042] Select a public data set and divide it into a test set and a training set;

[0043] During training and testing, the batch size and learning rate are set, and decay to 0.9 times the original value every 20 epochs. The total number of epochs is 200. The loss function of the point cloud downsampling model is shown as follows:

[0044]

[0045] Where, L emd Represents the displacement distance, P in represents the original point cloud, P sp represents the downsampled point cloud, It is a mapping relationship, that is, for each element in the downsampled point set, there is a unique and non-repeated point in the input point set corresponding to it.

[0046] In a second aspect, the present disclosure provides a point cloud fragmentation downsampling system based on a convolutional neural network, the system comprising:

[0047] A cloud data acquisition and preliminary division module, which is configured to acquire three-dimensional point cloud data, set the number of slices to be divided, initialize the cluster center according to the number of slices, and divide each data;

[0048] a re-dividing module, which is configured to recalculate new cluster centers and re-divide some data points according to a preset standard; and,

[0049] Repeat the above steps until the algorithm meets the termination condition and obtains the point cloud slice to be processed;

[0050] A point cloud downsampling module is configured to send the obtained point cloud slices to be processed into a downsampling network to obtain downsampled point cloud slices;

[0051] The point cloud stitching module is configured to slice the downsampled point cloud and stitch it into a complete downsampled point cloud.

[0052] In a third aspect, the present disclosure provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the point cloud segmentation downsampling method based on a convolutional neural network as described in any one of the first aspects.

[0053] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the point cloud segmentation downsampling method based on a convolutional neural network as described in any one of the first aspects above is implemented.

[0054] Beneficial effects:

[0055] The present invention provides a point cloud slice downsampling method based on a convolutional neural network, a point cloud slice downsampling system based on a convolutional neural network, an electronic device and a storage medium. After the acquired point cloud data is initially divided into cluster centers, new cluster centers are recalculated, and points that exceed a predetermined number and correspond to the farthest distance from the cluster center are migrated to a cluster where the cluster center that does not exceed a preset number and is closest to the point is located, thereby redistributing data and ensuring that the number of points in each cluster is consistent, so that the sampling process can perfectly cover the original point cloud without duplication, and the internal aggregation degree of the slice is higher, so that a more realistic down-sampled point cloud can be generated, thereby improving the quality of the point cloud after downsampling. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1A schematic diagram of a flow chart of a point cloud slice down-sampling method based on a convolutional neural network provided in the first embodiment of the present disclosure;

[0057] Figure 2 A schematic diagram of a specific implementation process of cloud slicing downsampling provided in an embodiment of the present disclosure;

[0058] Figure 3 A network architecture diagram of a point cloud downsampling model provided in an embodiment of the present disclosure;

[0059] Figure 4 An architecture diagram of a point cloud fragmentation downsampling system based on a convolutional neural network provided in the second embodiment of the present disclosure;

[0060] Figure 5 This is an architecture diagram of an electronic device provided in Embodiment 3 of the present disclosure. DETAILED DESCRIPTION

[0061] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the present disclosure is further described in detail below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention, rather than to limit the present invention.

[0062] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence; and, in the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be arbitrarily combined with each other.

[0063] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. The singular forms of "a", "said" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings.

[0064] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present disclosure, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0065] The following are the relevant terms that may appear in the embodiments of the present disclosure:

[0066] 1. Convolutional Neural Network: A convolutional neural network (CNN) is a type of feedforward neural network with a deep structure that includes convolutional computations and is one of the representative algorithms in deep learning. After the 21st century, with the proposal of deep learning theory and the improvement of numerical computing devices, convolutional neural networks have developed rapidly and have been applied in fields such as computer vision and natural language processing.

[0067] The difference between a convolutional neural network and an ordinary neural network is that a convolutional neural network includes a feature extractor composed of a convolutional layer and a subsampling layer (pooling layer). In the convolutional layer of a convolutional neural network, a neuron is only connected to some neighboring layer neurons. In a convolutional layer of a CNN, there are usually several feature maps, and each feature map consists of neurons arranged in a rectangle. Neurons in the same feature map share weights, and the shared weights here are the convolutional kernels. The convolutional kernels are generally initialized in the form of a random decimal matrix, and during the training process of the network, the convolutional kernels will learn to obtain reasonable weights. The direct benefit brought by sharing weights (convolutional kernels) is to reduce the connections between different layers of the network and at the same time reduce the risk of overfitting. Subsampling, also called pooling, usually has two forms: mean pooling and max pooling. Pooling can be regarded as a special convolutional process. Convolution and pooling greatly simplify the model complexity and reduce the parameters of the model.

[0068] 2. Point Cloud Slicing: For a large point cloud, several small point clouds can be obtained through point cloud slicing, and each small point cloud has the same number of points.

[0069] 3. K-means Algorithm: It is a distance-based clustering algorithm, and its core idea is to divide the data into K clusters through an iterative method, so that the data points within each cluster are as close as possible to their cluster centers, while the data points between different clusters are as far away as possible.

[0070] 4. Farthest Point Sampling (FPS): Sample M points (M < N) from N points. Then two sets A and B can be set. A represents the set formed by the selected points, and B represents the set composed of the unselected points. As the name implies, what FPS does is to select a point with the maximum distance from the points in set B to the points in set A each time.

[0071] Select the first two points: The first point is randomly selected, and the second point is the point with the farthest distance from this point. This step is relatively simple.

[0072] Select the remaining points: At this time, what needs to be determined is the distance between sets A and B. The fundamental idea of ​​farthest point sampling is that the sampling points to be obtained each time are the points in the origin set B that are the farthest away from the sampling point set A. Therefore, for the next sampling, it is necessary to traverse every point in the origin set B. Now, take any point B1, calculate the distance from B1 to each point in set A, and take the smallest distance as the distance from point B1 to set A. For each point in set B, perform a calculation to find the minimum distance from each point to set A, and then take the largest distance among these distances as the farthest distance between the current point set B and the sampling point set A. The point corresponding to the maximum distance is the farthest point of A in B, that is, the point that needs to be sampled in this calculation, and put it into set A. Repeat this process for the calculation of subsequent points.

[0073] 5. Point cloud segmentation algorithm using farthest point sampling: Point cloud segmentation is performed according to the farthest point sampling order. For example, an original point cloud containing n points needs to be divided into several point cloud slices containing m points. First, the farthest point sampling method is used to sort the sampling process of n points, and the 1st point to the mth point are selected to form the first point cloud slice, the (m+1)th point to the (2m)th point to form the second point cloud slice, ..., the (n-m+1)th point to the nth point form the last point cloud slice.

[0074] 6. K nearest neighbor: also known as KNN (K-Nearest Neighbor), for a point in the point cloud, search for the K points closest to it, including itself.

[0075] 7. Slicing algorithm of point cloud slice algorithm using farthest point sampling and K nearest neighbor: If the original point cloud containing n points needs to be divided into several point cloud slices containing m points, first use farthest point sampling to get several base points, and take K nearest neighbor (KNN) method for each base point to obtain point cloud slices containing m points.

[0076] 8. Euclidean distance, the Euclidean distance between point p and point q can be expressed as:

[0077]

[0078] 9. Modelnet40: Modelnet40 is a public dataset containing about 40 object categories, including 12,311 CAD models.

[0079] 10. Point cloud coverage: It indicates the coverage ratio of the sliced ​​point cloud to the original point cloud after splicing. The larger the coverage ratio, the better the effect. 100% means that the sliced ​​point cloud covers the original point cloud without duplication or omission.

[0080] 11. Slice aggregation degree: Indicates the compactness of the point cloud slice. The smaller the better, indicating that the point cloud slices are densely arranged. The aggregation degree of a point cloud slice is described by the Euclidean distance from the geometric center of the slice to the farthest point in the slice set.

[0081] 12. Chamfer distance: For input point cloud P in And the downsampled point cloud P sp The chamfer distance between can be expressed as follows, where x represents the point cloud P in A point in the set, y represents the point cloud P sp For a point in the set, the smaller the chamfer distance is, the better the quality of the downsampled point cloud is.

[0082]

[0083] The following is a detailed description of the technical solution of the present invention and how the technical solution of the present invention solves the problems existing in the prior art with specific embodiments. It is understandable that in the embodiments of the present application, the execution subject can perform some or all of the steps in the embodiments of the present application, and these steps or operations are only examples. The embodiments of the present application can also perform other operations or deformations of various operations. In addition, each step can be performed in a different order presented in the embodiments of the present application, and it is possible not to perform all the operations in the embodiments of the present application. And, the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0084] Figure 1 A flowchart of a point cloud segmentation downsampling method based on a convolutional neural network is provided in the first embodiment of the present disclosure, such as Figure 1 As shown, the method includes:

[0085] S1: Obtain 3D point cloud data, set the number of slices, initialize the cluster center according to the number of slices, and divide each data;

[0086] S2: Recalculate new cluster centers and re-divide some data points according to preset standards;

[0087] S3: Repeat step S2 until the algorithm meets the termination condition and obtains the point cloud slice to be processed;

[0088] S4: Send the obtained point cloud slices to be processed into the downsampling network to obtain downsampled point cloud slices;

[0089] S5: Slice the downsampled point cloud and stitch it into a complete downsampled point cloud.

[0090] Point cloud slicing is a technique that defines a plane or a series of parallel planes and intercepts the point cloud data on the plane to obtain a subset of the point cloud within a specified range. For a large point cloud, several small point clouds can be obtained through point cloud slicing, and each small point cloud has the same number of points. Point cloud downsampling technology is a method of sparse processing of point cloud data, which aims to reduce the density of point clouds, and reduce the amount of data, reduce processing complexity, and improve processing efficiency by selecting more robust or more valuable points from the original point cloud for retention.

[0091] The specific process of point cloud segmentation downsampling in the embodiment of the present disclosure is as follows: Figure 2 As shown in the figure, in order to achieve a better effect of point cloud downsampling, it is necessary to obtain more point data of the same type to obtain point cloud slices and improve the quality of the downsampled point cloud. After obtaining the three-dimensional point cloud data, the number of slices to be divided is determined, so as to determine the required initial cluster center and perform preliminary division of the data. The preliminary division can use the FPS slicing algorithm or the K nearest neighbor slicing algorithm. It is only necessary to divide the points with closer distances into the same cluster as much as possible. At this time, the data points in each cluster may be inconsistent. At this time, it is necessary to recalculate the new cluster center and re-divide some data points according to the preset standard. That is, in order to ensure that the point data of the same point cloud slice is the same, it is necessary to move some points that exceed the number. By moving the points farther from the cluster center to other clusters with a relatively longer distance or clusters with insufficient number of points, the migration of point data is completed until all clusters contain the same points and obtain the point cloud slices to be processed. Then, the obtained point cloud slices to be processed are sent to the downsampling network to obtain the downsampled point cloud slices; the downsampling network can be various existing convolutional neural network models that can realize downsampling, such as the LeNet5 network model or a self-built model. The downsampled point cloud slices are obtained through model processing to reduce the amount of data. Finally, the downsampled point cloud slices are spliced ​​into a complete downsampled point cloud for subsequent processing.

[0092] The disclosed embodiment performs a preliminary division of the acquired point cloud data by initializing the cluster centers, and then recalculates the new cluster centers, and migrates the points that exceed a predetermined number and correspond to the points with the farthest distance from the cluster center to the cluster where the cluster center that does not exceed the preset number and is closest to the point is located, thereby achieving data redistribution, ensuring that the number of points in each cluster is consistent, and enabling the sampling process to perfectly cover the original point cloud without repetition, and the internal aggregation degree of the slices is higher, so that a more realistic downsampled point cloud can be generated, thereby improving the quality of the point cloud after downsampling.

[0093] Furthermore, the three-dimensional point cloud data is obtained, the number of slices is set, the cluster center is initialized according to the number of slices, and each data is divided, including:

[0094] For the acquired 3D point cloud data, use {p i|i=1,…,n} represents the 3D unordered point cloud to be sliced, where each point has three channel features (x, y, z);

[0095] Set the number of slices that need to be divided into k slices, each slice has n patch points;

[0096] The FPS method is used to randomly initialize k cluster centers. For each data point in the data set, its distance to each cluster center is calculated; then, the data point is assigned to the cluster with the cluster center closest to it.

[0097] When performing the initial division, after setting the number of slices, randomly initialize k cluster centers through the FPS method, and calculate the distance between each data point and each cluster center using the Euclidean distance; and cluster points with close distances into the same cluster as much as possible to ensure that the sampling points are evenly distributed in space and can cover the entire point cloud. Based on this, the amount of data for each cluster point is adjusted to maximize the compactness of the point cloud slice.

[0098] Furthermore, the recalculation of the new cluster center and the re-division of some data points according to a preset standard include:

[0099] Calculate the number of points in each cluster, find the cluster with the most points, set it as cluster c, and satisfy Its cluster center Defined as:

[0100] x mid =average(x1,...,x l ) (1)

[0101] y mid =average(y1,...,y l ) (2)

[0102] z mid =average(z1,...,z l ) (3)

[0103] Then calculate the distance from each point to the cluster center Select the point with the longest distance from the cluster center satisfy:

[0104]

[0105] calculate The distance to the center of other clusters is sorted from small to large, and the cluster corresponding to the smallest distance is selected. If the number of points in the cluster is less than n patch, then move the point from cluster c to this cluster, otherwise, ignore this distance and its corresponding cluster, judge the next distance, until the first cluster that meets the conditions is found, and move the point from cluster c to this cluster.

[0106] After the initial clustering using the FPS method, the data points of each cluster may not be consistent. For clusters with more than a set number of data points, starting from the cluster with the largest number of points, move the point farthest from the cluster center to the cluster with insufficient data points and closest to the center of other clusters except the center of this cluster. Then repeat the above steps, re-determine the cluster with the most data points, and migrate the excess data points until the termination condition is met.

[0107] Preferably, after all the redundant data points of the cluster with the most data points are migrated, the cluster with the second most data points can be processed until the data points of all clusters are consistent.

[0108] Furthermore, the algorithm satisfies the termination condition including:

[0109] Each cluster contains n points. patch .

[0110] Through the migration of data points, all clusters contain n points patch After , it means that the termination condition is met. The completed point cloud slices can perfectly cover the original point cloud without duplication, and the internal aggregation degree of the slices is higher, so a more realistic down-sampled point cloud can be generated.

[0111] Furthermore, the step of sending the obtained point cloud slices to be processed to a downsampling network to obtain downsampled point cloud slices includes:

[0112] The obtained point cloud slices to be processed are sent to the preset downsampling network. The input of the network is a patch The output is a slice point cloud with m points. patch Slice point cloud of points;

[0113] The preset downsampling network is used to extract the features of the point cloud through several convolutional layers, and scores the contribution value of each point, and selects points with high scores to form the downsampled point cloud.

[0114] Among them, n patch and m patch is a value set independently, m patch Less than n patch , through the trained downsampling network, several convolutional layers in it extract the features of the input point cloud, and then score the contribution value of each point, and select the points with high scores to form the downsampled point cloud; for each point cloud slice, a slice point cloud with the same number of data points is obtained.

[0115] Furthermore, the preset downsampling network sampling process includes:

[0116] The original point cloud slices are processed in the following ways:

[0117] F1=conv1(P in ) (5)

[0118] F2=conv2(F1) (6)

[0119] F mat =conv3(F2) (7)

[0120] F4=conv4(F mat ·P in ) (8)

[0121] F5=conv5(F4) (9)

[0122] F sort =sort(F5) (10)

[0123] idx = get_index(F sort ) (11)

[0124] P sp =select_pt(idx,P in ) (12)

[0125] P in represents the original point cloud slice, conv1 represents the first layer of convolution, F1 is the feature matrix output by this layer, conv2 represents the second layer of convolution, F2 is the feature matrix output by this layer, conv3 represents the third layer of convolution, F mat Output the transformation matrix for this layer, and then use the transformation matrix F mat And the input point cloud dot multiplication, conv4 represents the fourth layer of convolution, F4 represents the fourth layer of convolution output features, conv5 represents the fifth layer of convolution, F5 represents the fifth layer of convolution output features, sort represents the sorting of features, F sort Indicates the sorting result, get_index means selecting the first m with larger values patch elements, idx represents the queried index value, select_pt represents the selection based on the index value and the original point cloud, P sp Represents a downsampled point cloud slice.

[0126] The network architecture of the point cloud downsampling model is as follows Figure 3 As shown, the input is n patch A data point containing three channel features is transformed by the conversion matrix F after multiple layers of convolution. matMultiply the point cloud with the input point cloud, output the features through the fourth and fifth convolution layers, sort the features, and select the top m with larger values patch elements, the output is m patch Each point cloud slice to be processed is processed by the downsampling network to obtain each point cloud slice.

[0127] Furthermore, the method further comprises:

[0128] The preset downsampling network model is trained and tested, including:

[0129] Select a public data set and divide it into a test set and a training set;

[0130] During training and testing, the batch size and learning rate are set, and decay to 0.9 times the original value every 20 epochs. The total number of epochs is 200. The loss function of the point cloud downsampling model is shown as follows:

[0131]

[0132] Where, L emd Represents the displacement distance, P in represents the original point cloud, P sp represents the downsampled point cloud, It is a mapping relationship, that is, for each element in the downsampled point set, there is a unique and non-repeated point in the input point set corresponding to it.

[0133] In one implementation, the batch size can be set to 8; the learning rate can be set to 0.001,

[0134] The loss function is set as the displacement distance. For the input point cloud P in And the downsampled point cloud P sp The displacement distance between them can be expressed as formula (13), where x represents the point cloud P in A point in the set, Represents the point cloud P sp A point in the set, It is a mapping relationship. Specifically, for each element in the downsampled point set, there is a unique and non-repeated point in the input point set corresponding to it. The smaller the displacement distance, the better the quality of the downsampled point cloud.

[0135] Furthermore, the downsampled point cloud is sliced ​​and spliced ​​into a complete downsampled point cloud, including:

[0136] The downsampled point cloud slices are stitched using the following formula:

[0137]

[0138] represents the i-th slice of the point cloud output by the downsampling network, It represents the complete downsampled point cloud after stitching, and concat represents the stitching operation.

[0139] The method proposed in the embodiment of the present disclosure, when the public data set Modelnet40 uses 10,000 points as input, first slices each point cloud into 10 slices, and each slice of 1,000 points is sent to the downsampling network, sampled to a point cloud slice containing 250 points, and finally restored to a sparse point cloud of 2,500 points. Since the method used in this embodiment can perfectly cover the original point cloud without repetition, and the internal aggregation degree of the slice is higher, a more realistic downsampled point cloud can be generated. The quality comparison of the point cloud slices is shown in Table 1, and the quality comparison of the downsampled point cloud is shown in Table 2.

[0140] Table 1: Quality comparison of point cloud slices

[0141] Evaluation indicators FPS FPS+KNN The method of this embodiment Point cloud coverage 100％ 83.1％ 100％ Slice Polymerization 0.61 0.23 0.24

[0142] Table 2: Comparison of downsampled point cloud quality

[0143] Evaluation indicators FPS FPS+KNN The method of this embodiment Chamfer distance 0.0042 0.0049 0.0026 Distance 311.7 1188.5 264.5

[0144] From Table 1 and Table 2, we can see that the farthest point sampling algorithm is used to slice the point cloud in the sampling order and then sample. This method has poor aggregation within the slice, resulting in a decrease in the chamfer distance and displacement distance of the entire point cloud after sampling. The farthest point sampling method obtains several base points and takes the K nearest neighbor (KNN) method for each point cannot achieve complete coverage of the entire point cloud, resulting in worse point cloud quality after sampling.

[0145] Compared with the two existing methods (as shown in Table 1 and Table 2), since the method used in the present invention can perfectly cover the original point cloud without repetition and the internal aggregation degree of the slice is higher, a more realistic down-sampled point cloud can be generated.

[0146] The second embodiment of the present disclosure also provides a point cloud segmentation downsampling system based on a convolutional neural network, such as Figure 4 As shown, the system comprises:

[0147] A cloud data acquisition and preliminary division module 11 is configured to acquire three-dimensional point cloud data, set the number of divided slices, initialize the cluster center according to the number of slices, and divide each data;

[0148] A re-dividing module 12 is configured to recalculate new cluster centers and re-divide some data points according to a preset standard; and

[0149] Repeat the above steps until the algorithm meets the termination condition and obtains the point cloud slice to be processed;

[0150] A point cloud downsampling module 13, which is configured to send the obtained point cloud slices to be processed into a downsampling network to obtain downsampled point cloud slices;

[0151] The point cloud stitching module 14 is configured to stitch the downsampled point cloud slices into a complete downsampled point cloud.

[0152] Furthermore, the cloud data acquisition and preliminary division module 11 is specifically configured as follows:

[0153] For the acquired 3D point cloud data, use {p i |i=1,…,n} represents the 3D unordered point cloud to be sliced, where each point has three channel features (x, y, z);

[0154] Set the number of slices that need to be divided into k slices, each slice has n patch points;

[0155] The FPS method is used to randomly initialize k cluster centers. For each data point in the data set, its distance to each cluster center is calculated; then, the data point is assigned to the cluster with the cluster center closest to it.

[0156] Furthermore, the re-dividing module 12 is specifically configured as follows:

[0157] Calculate the number of points in each cluster, find the cluster with the most points, set it as cluster c, and satisfy Its cluster center Defined as:

[0158] x mid =average(x1,...,x l ) (1)

[0159] y mid =average(y1,...,y l ) (2)

[0160] z mid =average(z1,...,z l ) (3)

[0161] Then calculate the distance from each point to the cluster center Select the point with the longest distance from the cluster center satisfy:

[0162]

[0163] calculate The distance to the center of other clusters is sorted from small to large, and the cluster corresponding to the smallest distance is selected. If the number of points in the cluster is less than n patch , then move the point from cluster c to this cluster, otherwise, ignore this distance and its corresponding cluster, judge the next distance, until the first cluster that meets the conditions is found, and move the point from cluster c to this cluster.

[0164] Furthermore, the algorithm satisfies the termination condition including:

[0165] Each cluster contains n points. patch .

[0166] Furthermore, the point cloud downsampling module 13 is specifically configured as follows:

[0167] The obtained point cloud slices to be processed are sent to the preset downsampling network. The input of the network is a patch The output is a slice point cloud with m points. patch Slice point cloud of points;

[0168] The preset downsampling network is used to extract the features of the point cloud through several convolutional layers, and scores the contribution value of each point, and selects points with high scores to form the downsampled point cloud.

[0169] Furthermore, the preset downsampling network sampling process includes:

[0170] The original point cloud slices are processed in the following ways:

[0171] F1=conv1(P in ) (5)

[0172] F2=conv2(F1) (6)

[0173] F mat =conv3(F2) (7)

[0174] F4=conv4(F mat ·P in ) (8)

[0175] F5=conv5(F4) (9)

[0176] F sort =sort(F5) (10)

[0177] idx = get_index(F sort ) (11)

[0178] P sp =select_pt(idx,P in) (12)

[0179] P in Represents the original point cloud slice, conv1 represents the first layer of convolution, F1 is the feature matrix output by this layer, conv2 represents the second layer of convolution, F2 is the feature matrix output by this layer, conv3 represents the third layer of convolution, f mat Output the transformation matrix for this layer, and then use the transformation matrix F mat And the input point cloud dot multiplication, conv4 represents the fourth layer of convolution, F4 represents the fourth layer of convolution output features, conv5 represents the fifth layer of convolution, F5 represents the fifth layer of convolution output features, sort represents the sorting of features, F sort Indicates the sorting result. get_index indicates selecting the first m with larger values. patch elements, idx represents the queried index value, select_pt represents the selection based on the index value and the original point cloud, P sp Represents a downsampled point cloud slice.

[0180] Furthermore, the point cloud downsampling module 13 is also configured as follows:

[0181] The preset downsampling network model is trained and tested, including:

[0182] Select a public data set and divide it into a test set and a training set;

[0183] During training and testing, the batch size and learning rate are set, and decay to 0.9 times the original value every 20 epochs. The total number of epochs is 200. The loss function of the point cloud downsampling model is shown as follows:

[0184]

[0185] Where, L emd Represents the displacement distance, P in represents the original point cloud, P sp represents the downsampled point cloud, φ is a mapping relationship, that is, for each element in the downsampled point set, there is a unique and non-repeated point in the input point set corresponding to it.

[0186] The point cloud segmentation down-sampling system based on a convolutional neural network in the embodiment of the present disclosure is used to implement the point cloud segmentation down-sampling method based on a convolutional neural network in method embodiment 1, so the description is relatively simple. For details, please refer to the relevant description in the previous method embodiment, which will not be repeated here.

[0187] In addition, if Figure 5As shown, the third embodiment of the present disclosure further provides an electronic device, including a memory 100 and a processor 200, wherein the memory 100 stores a computer program, and when the processor 200 runs the computer program stored in the memory 100, the processor 200 executes the above-mentioned various possible methods.

[0188] The memory 100 is connected to the processor 200. The memory 100 may be a flash memory, a read-only memory or other memory, and the processor 200 may be a central processing unit or a single-chip microcomputer.

[0189] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to perform the above-mentioned various possible methods.

[0190] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), Digital Video Disc (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0191] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A point cloud fragmentation downsampling method based on convolutional neural network, characterized in that: The method comprises: S1: Obtain 3D point cloud data, set the number of slices, initialize the cluster center according to the number of slices, and divide each data; S2: Recalculate new cluster centers and re-divide some data points according to preset standards; S3: Repeat step S2 until the algorithm meets the termination condition and obtains the point cloud slice to be processed; S4: Send the obtained point cloud slices to be processed into the downsampling network to obtain downsampled point cloud slices; S5: Slice the downsampled point cloud and stitch it into a complete downsampled point cloud.

2. The method according to claim 1, characterized in that The method of obtaining three-dimensional point cloud data, setting the number of slices to be divided, initializing cluster centers according to the number of slices, and dividing each data includes: For the acquired 3D point cloud data, use {p i |i=1,…,n} represents the 3D unordered point cloud to be sliced, where each point has three channel features (x, y, z); Set the number of slices that need to be divided into k slices, each slice has n patch points; The farthest point sampling (FPS) method is used to randomly initialize k cluster centers. For each data point in the data set, its distance to each cluster center is calculated; then, the data point is assigned to the cluster with the cluster center closest to it.

3. The method according to claim 2, characterized in that The recalculation of the new cluster center and re-dividing some data points according to a preset standard include: Calculate the number of points in each cluster, find the cluster with the most points, set it as cluster c, and satisfy Its cluster center Defined as: x mid =average(x1,...,x l ) (1) and mid =average(y1,...,y l ) (2) z mid =average(z1,...,z l ) (3) Then calculate the distance from each point to the cluster center Select the point with the longest distance from the cluster center satisfy: calculate The distance to the center of other clusters is sorted from small to large, and the cluster corresponding to the smallest distance is selected. If the number of points in the cluster is less than n patch , then move the point from cluster c to this cluster, otherwise, ignore this distance and its corresponding cluster, judge the next distance, until the first cluster that meets the conditions is found, and move the point from cluster c to this cluster.

4. The method according to claim 3, characterized in that The algorithm meets the termination conditions including: Each cluster contains n points. patch .

5. The method according to claim 1, characterized in that The step of sending the obtained point cloud slices to be processed into a downsampling network to obtain downsampled point cloud slices includes: The obtained point cloud slices to be processed are sent to the preset downsampling network. The input of the network is a patch The output is a slice point cloud with m points. patch Slice point cloud of points; The preset downsampling network is used to extract the features of the point cloud through several convolutional layers, and scores the contribution value of each point, and selects points with high scores to form the downsampled point cloud.

6. The method according to claim 5, characterized in that The preset downsampling network sampling process includes: The original point cloud slices are processed in the following ways: F1=conv1(P in ) (5) F2=conv2(F1) (6) <h2 style=";text-align:left;direction:ltr">F<h2 style=";text-align:left;direction:ltr"> mat <h2 style=";text-align:left;direction:ltr"> =conv3(F2) (7) F4=conv4(F mat ·P in ) (8) F5=conv5(F4) (9) F sort =sort(F5) (10) idx=get_index(F sort ) (11) P sp =select_pt(idx,P in ) (12) P in Represents the original point cloud slice, conv1 represents the first layer of convolution, F1 is the feature matrix output by this layer, conv2 represents the second layer of convolution, F2 is the feature matrix output by this layer, conv3 represents the third layer of convolution, f mat Output the transformation matrix for this layer, and then use the transformation matrix F mat And the input point cloud dot multiplication, conv4 represents the fourth layer of convolution, F4 represents the fourth layer of convolution output features, conv5 represents the fifth layer of convolution, F5 represents the fifth layer of convolution output features, sort represents the sorting of features, F sort Indicates the sorting result. get_index indicates selecting the first m with larger values. patch elements, idx represents the queried index value, select_pt represents the selection based on the index value and the original point cloud, P sp Represents a downsampled point cloud slice.

7. The method according to claim 1, characterized in that The method further comprises: The preset downsampling network model is trained and tested, including: Select a public data set and divide it into a test set and a training set; During training and testing, the batch size and learning rate are set, and decay to 0.9 times the original value every 20 epochs. The total number of epochs is 200. The loss function of the point cloud downsampling model is shown as follows: Where, L emd Represents the displacement distance, P in represents the original point cloud, P sp represents the downsampled point cloud, φ is a mapping relationship, that is, for each element in the downsampled point set, there is a unique and non-repeated point in the input point set corresponding to it.

8. A point cloud fragmentation downsampling system based on convolutional neural network, characterized in that: The system comprises: A cloud data acquisition and preliminary division module, which is configured to acquire three-dimensional point cloud data, set the number of slices to be divided, initialize the cluster center according to the number of slices, and divide each data; a re-dividing module, which is configured to recalculate new cluster centers and re-divide some data points according to a preset standard; and, Repeat the above steps until the algorithm meets the termination condition and obtains the point cloud slice to be processed; A point cloud downsampling module is configured to send the obtained point cloud slices to be processed into a downsampling network to obtain downsampled point cloud slices; The point cloud stitching module is configured to slice the downsampled point cloud and stitch it into a complete downsampled point cloud.

9. An electronic device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor runs the computer program stored in the memory, the processor executes the point cloud segmentation downsampling method based on a convolutional neural network as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the point cloud segmentation downsampling method based on a convolutional neural network as described in any one of claims 1 to 7 is implemented.

Images