A large-scale three-dimensional point cloud semantic segmentation method and system

Abstract

The application relates to a large-scale three-dimensional point cloud semantic segmentation method and system, which comprises the following steps: establishing a space grid in the distribution range of a point cloud; adopting a grid downsampling method to encode the grid space position and the point cloud in the grid as the key and value of a map, and storing all the grids in the form of an unordered_map; for any grid, establishing a domain lookup table of the grid based on the unordered_map, and performing domain lookup based on the domain lookup table; the domain lookup table of the grid is a position code difference table of n grids in the neighborhood range of the grid as the center, which is established from near to far; a network model for point cloud semantic segmentation is constructed to segment and extract the semantic features of the point cloud; the grid downsampling is high in efficiency, the unordered_map of the grid index can be established at the same time during the downsampling, the time for establishing the unordered_map is saved, and the speed of the downsampled point cloud can be improved; the unorded_map structure lookup algorithm is low in complexity, the searching speed of the neighborhood points of the point cloud can be improved, and the training and reasoning speed of the large-scale point cloud semantic segmentation is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning 3D semantic segmentation, in particular to a large-scale three-dimensional point cloud semantic segmentation method and system. BACKGROUND

[0002] Point cloud learning has received more and more attention in recent years due to its wide application in computer vision, autonomous driving, robots and other fields. As the leading technology of artificial intelligence, deep learning has been successfully used to solve various two-dimensional visual problems. However, due to the unique challenges faced by deep neural networks in processing point clouds, deep learning based on point clouds is still in its infancy. Recently, deep learning of point clouds has become more prosperous, and many methods have been proposed to solve different problems in this field:

[0003] 1) Due to the irregularity of point cloud distribution, and in order not to lose information, it is always necessary to use nearest neighbor search to find the point cloud in the neighborhood of a certain point. At present, FLANN type algorithm is generally used to search for points in the neighborhood, and for large-scale point clouds, this type of algorithm has efficiency problems, and in the process of down-sampling throughout the model, the sampling process needs to use the nearest neighbor search algorithm. Its efficiency impact on the model is generally large.

[0004] 2) The object in the physical space is naturally clustered and the contour is smooth, especially for some categories such as lane lines, the boundary smoothness is very high, and for the application of autonomous driving and other aspects, the collected point cloud categories are extremely unbalanced. When using weighting or similar weighting technology, it tends to identify the category with more quantity as the category with less quantity, which affects the boundary of the segmented category, resulting in a non-smooth boundary. SUMMARY

[0005] The present application provides a large-scale three-dimensional point cloud semantic segmentation method and system to solve the problem of slow search speed of point cloud neighborhood in the prior art, which affects the training and inference speed of large-scale point cloud semantic segmentation.

[0006] According to the first aspect of the present application, a large-scale three-dimensional point cloud semantic segmentation method is provided, comprising: step 1, establishing a space grid in the distribution range of the point cloud;

[0007] Step 2, using a grid down-sampling method, encoding the grid space position and the point cloud in the grid as the key and value of the map, and storing all the grids in the form of unordered_map;

[0008] Step 3, for any grid, a domain lookup table of the grid is established based on the unordered_map, and domain lookup is performed based on the domain lookup table; the domain lookup table of the grid is a position coding difference table of n grids in the neighborhood range of the grid as the center, which is established from near to far;

[0009] Step 4, constructing a network model for point cloud semantic segmentation, and segmenting and extracting the semantic features of the point cloud.

[0010] On the basis of the above technical solutions, the application can also be improved as follows.

[0011] Optionally, the method for establishing a space grid in step 1 comprises:

[0012] Traversing the point cloud to obtain a point cloud bounding box;

[0013] Re-traversing the point cloud to establish the space grid according to the point cloud bounding box.

[0014] Optionally, the method for grid down-sampling in step 2 comprises: replacing the values of all points in each grid with the average value or weighted average value of the points in the grid.

[0015] Optionally, the method for performing domain lookup based on the domain lookup table in step 3 comprises:

[0016] Starting from the nearest position in the domain lookup table to perform neighborhood lookup, and stopping when the lookup range or number is reached;

[0017] The lookup method comprises: nearest neighbor KNN lookup and neighborhood Radius NN lookup.

[0018] Optionally, the process of constructing a network model for point cloud semantic segmentation in step 4 comprises:

[0019] A skeleton network is established with KPConv as the core, kernel points are generated based on the skeleton of KPConv, features are extracted by point cloud convolution to realize encoding, the features are input into an MLP multi-layer perceptron to realize decoding, a U-shaped network is established to complete point cloud semantic segmentation.

[0020] Optionally, the loss function of the network model comprises: a Focal Loss loss function.

[0021] Optionally, the process of constructing a network model for point cloud semantic segmentation in step 4 further comprises: constructing a contour point loss function, the contour point loss function adds the loss of the contour boundary to the loss of the network model, different phase values are assigned according to the category label, and a multi-phase level set is used to extract the category contour predicted by the network model and the sample.

[0022] According to a second aspect of the present application, a large-scale three-dimensional point cloud semantic segmentation system is provided, comprising a spatial grid establishment module, a sampling module, a field lookup module and a semantic segmentation module.

[0023] The spatial grid establishment module is configured to establish a spatial grid within the distribution range of the point cloud.

[0024] The sampling module is configured to use a grid subsampling method, encode the grid spatial position and the point cloud within the grid as the key and value of the map, and store all the grids in the form of unordered_map.

[0025] The field lookup module is configured to, for any grid, establish a field lookup table of the grid based on the unordered_map, and perform field lookup based on the field lookup table; the field lookup table of the grid is a position encoding difference table of n grids within the neighborhood range of the grid, which is established from near to far.

[0026] The semantic segmentation module is configured to construct a network model for point cloud semantic segmentation, and extract the semantic features of the point cloud.

[0027] According to a third aspect of the present application, an electronic device is provided, comprising a memory and a processor, wherein the processor is configured to execute a computer management program stored in the memory to implement the steps of the large-scale three-dimensional point cloud semantic segmentation method.

[0028] According to a fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer management program, wherein the computer management program is executed by a processor to implement the steps of the large-scale three-dimensional point cloud semantic segmentation method.

[0029] The large-scale three-dimensional point cloud semantic segmentation method, system, electronic device and storage medium provided by the present application can greatly reduce the time complexity of neighborhood lookup, so that the time consumption is mainly concentrated in the model, and when the same model is used, the efficiency is greatly improved, and the general model speed-up method can further improve the efficiency.

[0030] Focal Loss is used as part of the loss function to improve the problem of extremely unbalanced sample labels, and the semantic type contour is used as part of the loss function to improve the smoothness of the class boundary of the segmentation, and the smoothness of the sample boundary with a small proportion is greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 A flowchart of a large-scale three-dimensional point cloud semantic segmentation method provided by the present application is shown in FIG. 1.

[0032] Figure 2 A flowchart of a method for neighborhood lookup of a point cloud based on a neighborhood lookup table provided by an embodiment of the present application is shown in FIG. 2.

[0033] Figure 3 A structural diagram of a point cloud semantic segmentation network based on KPConv provided by an embodiment of the present application is shown in FIG. 3.

[0034] Figure 4 A structural block diagram of a large-scale three-dimensional point cloud semantic segmentation system provided by the present application is shown in FIG. 4.

[0035] Figure 5 A hardware structural diagram of a possible electronic device provided by the present application is shown in FIG. 5.

[0036] Figure 6 A hardware structural diagram of a possible computer readable storage medium provided by the present application is shown in FIG. 6. DETAILED DESCRIPTION

[0037] The principles and features of the present application are described below in conjunction with the accompanying drawings, and the examples are only used to explain the present application and not to limit the scope of the present application.

[0038] Figure 1 A flowchart of a large-scale three-dimensional point cloud semantic segmentation method provided by the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the method comprises the following steps:

[0039] Step 1: Establish a spatial grid within the distribution range of the point cloud.

[0040] Step 2: Use the grid downsampling method, encode the grid spatial position and the point cloud within the grid as the key and value of the map, and store all the grids in the form of unordered_map.

[0041] Step 3: For any grid, establish a neighborhood lookup table for the grid based on unordered_map, and perform neighborhood lookup based on the neighborhood lookup table; the neighborhood lookup table for the grid is a position encoding difference table of n grids in the neighborhood range of the grid as the center, established from near to far.

[0042] Step 4: Construct a network model for point cloud semantic segmentation, and extract the semantic features of the point cloud.

[0043] In the current point cloud deep learning algorithm, the algorithm complexity of the FLANN type neighborhood lookup is about O(n 1-1 / km), and the region search complexity is about O(n). The application provides a large-scale three-dimensional point cloud semantic segmentation method, and the unorded_map structure search algorithm complexity is O(m), m is the number of searched neighborhoods, the point cloud neighborhood point search speed can be improved, and the large-scale point cloud semantic segmentation training and inference speed is greatly improved.

[0044] In the current point cloud deep learning algorithm, in order to ensure that the sampling points lose as little information as possible, the farthest point downsampling, density downsampling and the like are mainly used, and when the point cloud scale is very large, the downsampling speed is very slow. The application provides a large-scale three-dimensional point cloud semantic segmentation method, and the grid downsampling is used, the efficiency is high compared with other downsampling, and the unorder_map of the grid index can be established at the same time during the downsampling, the time for establishing the unorded_map is saved, and the speed of the down-sampled point cloud can be improved.

[0045] Embodiment 1

[0046] The embodiment 1 provided by the application is an embodiment of the large-scale three-dimensional point cloud semantic segmentation method provided by the application, which is combined with Figure 1 It can be known that the embodiment of the semantic segmentation method comprises:

[0047] Step 1, a space grid is established in the distribution range of the point cloud.

[0048] In a possible embodiment mode, the point cloud is traversed, the point cloud bounding box is obtained, and is recorded as (min_x, max_x, min_y, max_y, min_z, max_z).

[0049] The point cloud is traversed again, and the space grid is established according to the point cloud bounding box.

[0050] Step 2, the grid downsampling method is adopted, the grid space position code and the point cloud in the grid are taken as the key and the value of the map, and all the grids are stored in the form of unordered_map.

[0051] As Figure 2 The flowchart of the method for searching the neighborhood of the point cloud based on the neighborhood search table provided by the embodiment of the application is shown. In a possible embodiment mode, the grid downsampling method in step 2 comprises: the average value or the weighted average value of the points in each grid is used to replace the values of all the points in the original grid, that is, the blue points in Figure 2 The whole point cloud is converted to the key points obtained by the downsampling, and the shape characteristics of the point cloud are basically unchanged at the same time, so that the purpose of reducing the calculation amount is achieved.

[0052] Step 3, for any grid, a domain lookup table of the grid is established based on unordered_map, and domain lookup is performed based on the domain lookup table; the domain lookup table of the grid is a position coding difference table of n grids in the neighborhood range of the grid as the center, which is established from near to far.

[0053] In a possible implementation manner, the method of performing domain lookup based on the domain lookup table in step 3 includes:

[0054] The neighborhood lookup is performed from the nearest position in the domain lookup table, and is stopped when the lookup range or number is reached.

[0055] The lookup method includes KNN (k-Nearest Neighbor) lookup and Radius NN (point in a circle with a radius of r) lookup.

[0056] Step 4, a network model for point cloud semantic segmentation is constructed to segment and extract semantic features of the point cloud.

[0057] As shown in Figure 3 Fig. 1 is a schematic diagram of a structure of a point cloud semantic segmentation network provided by an embodiment of the present application, based on which a model suitable for large-scale point cloud semantic segmentation is constructed to realize semantic feature extraction of the point cloud. Specifically, in a possible implementation manner, the process of constructing the network model for point cloud semantic segmentation in step 4 includes:

[0058] A skeleton network is established with KPConv (Kernel Point Convolution) as the core or in a similar convolution manner, kernel points are generated based on the skeleton of KPConv, features are extracted by point cloud convolution to realize encoding, the features are input to a multilayer perceptron (MLP) to realize decoding, and a U-shaped network is established to complete point cloud semantic segmentation.

[0059] In a possible implementation manner, the loss function of the network model includes a Focal Loss loss function.

[0060] The Focal Loss loss function is:

[0061] Focal Loss (p t )=-α t (1-p t ) γ log(p t )。

[0062] Wherein, α tis a hyperparameter, is a constant, the more the number of general categories, the smaller t According to the unbalanced category of the point cloud data collected in the automatic driving field, the value of is the square root of 1 / 6 of the category weight. t The value of can be 2.

[0063] The Focal Loss loss function can solve the model performance problem caused by unbalanced data samples. Based on the cross-entropy loss function, through a dynamic scaling factor, the weight of the sample with a large number of easily distinguishable samples in the training process can be dynamically reduced, so as to quickly focus on the sample with a small number of difficult-to-distinguish samples.

[0064] In a possible embodiment, the process of constructing the network model for point cloud semantic segmentation in step 4 further includes: constructing an outline point loss function, the outline point loss function adds the loss of the outline boundary to the loss of the network model, assigns different phase values according to the category label, and uses a multi-phase level set (variational method) to extract the category outline of the sample and the network model prediction.

[0065] In a possible embodiment, the outline point loss function is:

[0066] Outline Loss=Length+λ·Region;

[0067]

[0068]

[0069] Wherein, λ represents a control parameter, which is used to control the influence size of Region in the loss function, i, j, k represent the coordinate index in the three-dimensional space, Ω represents the three-dimensional space, respectively represent the intensity gradient in the x, y, and z directions in the three-dimensional space, θ represents a regularization parameter, u i,, , v i,j,k respectively represent the intensity of the coordinate point (i, j, k) on different phases, c1 and c2 represent the phase boundary value, the energy minimization function in the active contour is taken as part of the loss function, and the size of the loss function accounts for the smoothness of the predicted category outline.

[0070] Embodiment 2

[0071] The embodiment 2 provided by the application is an embodiment of a large-scale three-dimensional point cloud semantic segmentation system provided by the application, Figure 4 is a large-scale three-dimensional point cloud semantic segmentation system structure diagram provided by the embodiment of the application, and in combination with Figure 4 It can be known that the embodiment of the semantic segmentation system includes a spatial grid establishment module, a sampling module, a field searching module, and a semantic segmentation module.

[0072] The spatial grid creation module is used to create a spatial grid within the distribution range of the point cloud.

[0073] The sampling module is used to employ a grid downsampling method, using grid spatial location encoding and point clouds within the grid as keys and values ​​for the map. All grids are stored in the form of unordered_map.

[0074] The neighborhood lookup module is used to build a neighborhood lookup table for any grid based on the unordered_map, and to perform a neighborhood lookup based on the neighborhood lookup table. The neighborhood lookup table of a grid is a table of positional coding differences of n grids within its neighborhood range, centered on the grid and arranged from near to far.

[0075] The semantic segmentation module is used to build a network model for point cloud semantic segmentation and to segment and extract the semantic features of point clouds.

[0076] It is understood that the large-scale three-dimensional point cloud semantic segmentation system provided by the present invention corresponds to the large-scale three-dimensional point cloud semantic segmentation method provided in the foregoing embodiments. The relevant technical features of the large-scale three-dimensional point cloud semantic segmentation system can be referred to the relevant technical features of the large-scale three-dimensional point cloud semantic segmentation method, and will not be repeated here.

[0077] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating an embodiment of the electronic device provided in this invention. For example... Figure 5 As shown, this embodiment of the invention provides an electronic device, including a memory 1310, a processor 1320, and a computer program 1311 stored in the memory 1310 and executable on the processor 1320. When the processor 1320 executes the computer program 1311, it performs the following steps: establishing a spatial grid within the distribution range of the point cloud; using a grid downsampling method, using the grid spatial location code and the point cloud within the grid as the key and value of a map, storing all grids in the form of unordered_map; for any grid, establishing a neighborhood lookup table for that grid based on the unordered_map, and performing a neighborhood lookup based on the neighborhood lookup table; the neighborhood lookup table for a grid is a table of position code differences for n grids within its neighborhood range, centered on that grid and arranged from near to far; constructing a network model for point cloud semantic segmentation, and segmenting and extracting the semantic features of the point cloud.

[0078] Please see Figure 6 , Figure 6 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by the present invention. (See diagram below.) Figure 6As shown, the embodiment provides a computer readable storage medium 1400, which stores a computer program 1411, and the computer program 1411 is executed by a processor to implement the following steps: a spatial grid is established in the distribution range of the point cloud; a grid spatial position code and a point cloud in the grid are used as a key and a value of a map, and all grids are stored in the form of an unordered_map by using a grid downsampling method; for any grid, a domain lookup table of the grid is established based on the unordered_map, and domain lookup is performed based on the domain lookup table; the domain lookup table of the grid is a position code difference table of n grids in a neighborhood range of the grid, which is established from near to far and takes the grid as a center; a network model for semantic segmentation of the point cloud is constructed, and semantic features of the point cloud are segmented and extracted.

[0079] The large-scale three-dimensional point cloud semantic segmentation method, system, electronic device and storage medium provided by the embodiment of the application can greatly reduce the time complexity of neighborhood lookup, so that the time consumption is mainly concentrated in the model, and when the same model is used, the efficiency is greatly improved, and the general model speed-up method can further improve the efficiency.

[0080] Focal Loss is used as part of the loss function to improve the problem of extremely unbalanced sample labels, and the semantic type contour is used as part of the loss function to improve the smoothness of the class boundary segmented, and the smoothness of the sample boundary with a small proportion is greatly improved.

[0081] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0082] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0083] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0084] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0086] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those of skill in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, the attached claims are intended to embrace all such variations and modifications as fall within the scope of the present application.

[0087] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A large-scale 3D point cloud semantic segmentation method, characterized in that, The semantic segmentation method includes: Step 1, establishing a spatial grid within the distribution range of the point cloud; Step 2: Using a grid downsampling method, the grid spatial location encoding and the point cloud within the grid are used as the keys and values ​​of the map. All grids are stored in the form of unordered_map. Step 3: For any grid, establish a neighborhood lookup table for that grid based on the unordered_map, and perform a neighborhood lookup based on the neighborhood lookup table; the neighborhood lookup table for a grid is a table of positional coding differences among n grids within its neighborhood range, centered on that grid and arranged from nearest to farthest. Step 4: Construct a network model for point cloud semantic segmentation to segment and extract the semantic features of the point cloud; The process of constructing a network model for point cloud semantic segmentation in step 4 includes: establishing a backbone network with KPConv as the core, generating kernel points based on the KPConv backbone, extracting features from point cloud convolution to achieve encoding, inputting the features into an MLP-based multilayer perceptron to achieve decoding, and establishing a U-shaped network to complete point cloud semantic segmentation. The process of constructing a network model for point cloud semantic segmentation in step 4 further includes: constructing a contour point loss function, which incorporates the loss of contour boundaries into the loss of the network model, assigning different phase values ​​according to the category label, and using a multiphase level set to extract the sample and the category contour predicted by the network model.

2. The semantic segmentation method according to claim 1, characterized in that, The method for establishing a spatial mesh in step 1 includes: traversing the point cloud and obtaining the point cloud bounding box; The point cloud is traversed again, and the spatial mesh is constructed based on the bounding box of the point cloud.

3. The semantic segmentation method according to claim 1, characterized in that, The method for grid downsampling in step 2 includes replacing the values ​​of all points in the grid with the average or weighted average of the points in each grid.

4. The semantic segmentation method according to claim 1, characterized in that, The method for performing a neighborhood search based on the neighborhood lookup table in step 3 includes: starting the neighborhood search from the nearest position in the neighborhood lookup table and stopping when the search range or number of items is reached; The search methods include nearest neighbor KNN search and neighborhood Radius NN search.

5. The semantic segmentation method according to claim 1, characterized in that, The loss function of the network model includes the Focal Loss function.

6. A large-scale 3D point cloud semantic segmentation system applying the large-scale 3D point cloud semantic segmentation method as described in any one of claims 1-5, characterized in that, The semantic segmentation system includes: a spatial grid establishment module, a sampling module, a neighborhood search module, and a semantic segmentation module; The spatial grid establishment module is used to establish a spatial grid within the distribution range of the point cloud; The sampling module is used to employ a grid downsampling method to encode the grid spatial location and the point cloud within the grid as the key and value of the map, and all grids are stored in the form of unordered_map; The neighborhood lookup module is used to establish a neighborhood lookup table for any grid based on the unordered_map, and to perform a neighborhood lookup based on the neighborhood lookup table; the neighborhood lookup table of the grid is a position code difference table of n grids within its neighborhood range, centered on the grid and arranged from near to far. The semantic segmentation module is used to construct a network model for point cloud semantic segmentation and to segment and extract the semantic features of the point cloud.

7. An electronic device, characterized in that, Includes a memory and a processor, wherein the processor is used to implement, as claimed in claim 1, a computer management program stored in the memory. The steps of any one of the following large-scale three-dimensional point cloud semantic segmentation methods.

8. A computer-readable storage medium, characterized in that, It stores computer management programs, which, when executed by a processor, implement as described in claim 1. The steps of any one of the following large-scale three-dimensional point cloud semantic segmentation methods.

Images