Point cloud segmentation method, point cloud segmentation device, equipment and computer storage medium

Abstract

The invention provides a point cloud segmentation method, a point cloud segmentation device, point cloud segmentation equipment and a computer storage medium. The point cloud segmentation method comprises the following steps: acquiring original point cloud data; obtaining high-dimensional voxel point features of each data point in the original point cloud data; obtaining spatial voxel features by using the high-dimensional voxel point features belonging to the same voxel grid; transforming the space voxel features into aerial view features through dimension transformation; and performing point cloud segmentation prediction based on the aerial view features. Through the point cloud segmentation method, voxel division is performed on the point cloud data, so that data points of different heights and different categories are dispersed in different voxel grids, and then the voxels in the height direction and the feature channels are recombined, so that height information is reserved, and meanwhile, a low-time-consumption 2D convolutional network is conveniently used subsequently to perform a segmentation task.

Description

Technical Field

[0001] This application relates to the technical field of point cloud processing, and particularly to a point cloud segmentation method, a point cloud segmentation device, a point cloud segmentation device, and a computer storage medium. Background Art

[0002] Existing point cloud segmentation methods are divided into traditional and deep learning methods. Traditional methods mainly rely on clustering. By extracting features designed based on prior knowledge from point cloud data, clustering algorithms are used to cluster the point cloud. Deep learning methods obtain a mapping model from point cloud data to labels by learning a large amount of labeled data, which is used to infer new data. In deep learning methods, point-based methods directly learn features from the original point cloud for point-level classification; projection / cylindrical voxel-based methods highly compress the point cloud data to obtain a 2D representation of the point cloud data, and then train a mature 2D convolutional network to segment the projected image, and finally restore the labels of the point cloud from the segmentation results of the projection; 3D voxel-based methods discretize the point cloud data in three-dimensional space and use 3D convolution to extract features to train a segmentation model.

[0003] In the above methods, point-based and 3D voxel-based methods are difficult to be used for large-scale outdoor point cloud data. In the projection / cylindrical voxel-based method, when mapping to a bird's-eye view, only the points with the maximum activation value are retained in each cylindrical voxel, losing most of the point information and forming a single-level bird's-eye view, which degrades the model performance. In addition, when there are overlapping points in the vertical direction, the representation ability of the cylindrical voxel is insufficient. Summary of the Invention

[0004] To solve the above technical problems, this application proposes a point cloud segmentation method, a point cloud segmentation device, a point cloud segmentation device, and a computer storage medium.

[0005] To solve the above technical problems, this application proposes a point cloud segmentation method, and the point cloud segmentation method includes:

[0006] Obtain the original point cloud data;

[0007] Obtain the high-dimensional voxel point features of each data point in the original point cloud data;

[0008] Utilize the high-dimensional voxel point features belonging to the same voxel grid to obtain spatial voxel features;

[0009] Transform the spatial voxel features into bird's-eye view features through dimensionality transformation;

[0010] Perform point cloud segmentation prediction based on the bird's-eye view features.

[0011] Wherein, the step of utilizing the high-dimensional voxel point features belonging to the same voxel grid to obtain spatial voxel features includes:

[0012] Obtain all high-dimensional voxel point features belonging to the same voxel grid;

[0013] Use the maximum eigenvalue of all high-dimensional voxel point features in each feature dimension to form the spatial voxel feature.

[0014] Wherein, before obtaining the spatial voxel feature by using the high-dimensional voxel point features belonging to the same voxel grid, the point cloud segmentation method further includes:

[0015] Perform downsampling of the high-dimensional voxel point features belonging to the same voxel grid with different sizes to obtain high-dimensional voxel point features at different levels;

[0016] The obtaining of the spatial voxel feature by using the high-dimensional voxel point features belonging to the same voxel grid includes:

[0017] Use the high-dimensional voxel point features at different levels belonging to the same voxel grid to obtain spatial voxel features at different levels.

[0018] Wherein, the transforming the spatial voxel feature into a bird's-eye view feature through dimension transformation includes:

[0019] Respectively transform the spatial voxel features at different levels through dimension transformation into two-dimensional voxel features at different levels;

[0020] Concatenate the two-dimensional voxel features at different levels in the channel dimension to generate the bird's-eye view feature.

[0021] Wherein, the obtaining of the high-dimensional voxel point features of each data point in the original point cloud data includes:

[0022] Obtain the voxel center point where the data point is located;

[0023] Concatenate the data point with the voxel center point to obtain the low-dimensional voxel point of the data point;

[0024] Extract the high-dimensional voxel point features of the low-dimensional voxel point.

[0025] Wherein, the point cloud segmentation method further includes:

[0026] Obtain the voxel grid size and the point cloud data range value;

[0027] Use the coordinates of the data point, the point cloud data range value, and the voxel grid size to determine the voxel grid index of the data point.

[0028] Wherein, after obtaining the original point cloud data, the point cloud segmentation method further includes:

[0029] Determine the three-dimensional coordinate range;

[0030] Determine whether the three-dimensional coordinates of the data points in the original point cloud data are within the three-dimensional coordinate range;

[0031] If not, remove the data points or update the three-dimensional coordinates of the data points according to the boundary of the exceeded three-dimensional coordinate range.

[0032] To solve the above technical problems, the present application also proposes a point cloud segmentation device, which includes: a point cloud acquisition module, a feature extraction module, a feature transformation module, and a point cloud segmentation module; wherein,

[0033] The point cloud acquisition module is used to acquire original point cloud data;

[0034] The feature extraction module is used to acquire the high-dimensional voxel point features of each data point in the original point cloud data;

[0035] The feature extraction module is used to acquire spatial voxel features by using the high-dimensional voxel point features belonging to the same voxel grid;

[0036] The feature transformation module is used to transform the spatial voxel features into bird's-eye view features through dimensional transformation;

[0037] The point cloud segmentation module is used to perform point cloud segmentation prediction based on the bird's-eye view features.

[0038] To solve the above technical problems, the present application also proposes a point cloud segmentation device, which includes a memory and a processor coupled to the memory; wherein, the memory is used to store program data, and the processor is used to execute the program data to implement the point cloud segmentation method as described above.

[0039] To solve the above technical problems, the present application also proposes a computer storage medium, which is used to store program data, and when the program data is executed by a computer, it is used to implement the above point cloud segmentation method.

[0040] Compared with the prior art, the beneficial effects of the present application are: the point cloud segmentation device acquires original point cloud data; acquires the high-dimensional voxel point features of each data point in the original point cloud data; acquires spatial voxel features by using the high-dimensional voxel point features belonging to the same voxel grid; transforms the spatial voxel features into bird's-eye view features through dimensional transformation; performs point cloud segmentation prediction based on the bird's-eye view features. Through the above point cloud segmentation method, the point cloud data is divided into voxels, so that data points of different heights and different categories are scattered in different voxel grids, and then the voxels in the height direction are recombined with the feature channels, so as to retain the height information and facilitate subsequent use of a 2D convolutional network with low time consumption for segmentation tasks. Brief Description of the Drawings

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0042] Wherein:

[0043] Figure 1 is a schematic flowchart of an embodiment of the point cloud segmentation method provided by the present application;

[0044] Figure 2 is a schematic network flowchart of the point cloud segmentation method provided by the present application;

[0045] Figure 3 is a schematic flowchart of another embodiment of the point cloud segmentation method provided by the present application;

[0046] Figure 4 is a schematic structural diagram of an embodiment of the point cloud segmentation device provided by the present application;

[0047] Figure 5 is a schematic structural diagram of an embodiment of the point cloud segmentation device provided by the present application;

[0048] Figure 6 is a schematic structural diagram of an embodiment of the computer storage medium provided by the present application. Detailed Description of the Embodiments

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0050] In the description and claims of this application and the above-mentioned drawings, terms such as "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of this application described here, for example, can be implemented in an order other than those illustrated or described here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0051] Point cloud data is a collection of a set of vectors in three-dimensional space, and each point contains three-dimensional coordinates and color information or reflection intensity information. The point cloud data collected by lidar contains the position and reflection intensity information of objects and plays an important role in autonomous driving. The point cloud segmentation task aims to assign category information to each point and is a basic task in the environmental perception of autonomous driving.

[0052] To solve the problems of a large amount of point information loss and height information loss caused by projection or columnar voxelization in existing deep learning point cloud segmentation methods based on projection or columnar voxels, this application provides a point cloud segmentation method based on a multi-level bird's-eye view. The technical problems solved are as follows:

[0053] 1. When common bird's-eye view-based laser point cloud segmentation methods map point cloud data into a bird's-eye view, max pooling is used for multiple points that may exist in a voxel, and this process causes the loss of other point feature information in the voxel.

[0054] 2. In common bird's-eye view-based laser point cloud segmentation methods, columnar voxels are used to compress the information in the height direction, resulting in the influence on the recognition accuracy when points of different categories coincide in the vertical direction.

[0055] For details, please continue to refer to Figure 1 and Figure 2 , Figure 1 is a schematic flowchart of an embodiment of the point cloud segmentation method provided by this application, Figure 2 is a schematic network flowchart of the point cloud segmentation method provided by this application.

[0056] The point cloud segmentation method of the present application is applied to a point cloud segmentation device. Among them, the point cloud segmentation device of the present application can be a server, a terminal device, or a system in which the server and the terminal device cooperate with each other. Correspondingly, each part included in the point cloud segmentation device, such as each unit, subunit, module, and submodule, can be all set in the server, all set in the terminal device, or separately set in the server and the terminal device.

[0057] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers or as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide a distributed server, or as a single software or software module, which is not specifically limited here.

[0058] As Figure 1 shown, the specific steps are as follows:

[0059] Step S11: Obtain the original point cloud data.

[0060] In the embodiment of the present application, in Figure 2 the original point cloud stage shown, the point cloud segmentation device obtains lidar point cloud data and uses a labeling tool to label the data point by point to obtain ground truth data. Each point is represented by 4D data, such as (x, y, z, i), where x, y, z represent the spatial position coordinates of the point, and i represents the reflection intensity of the point. The semantic label of the point cloud is represented as L sem ∈[0, C), where C is the number of categories.

[0061] Furthermore, in order to reduce the processing amount of the point cloud data, before performing subsequent feature extraction, the point cloud segmentation device can also perform preprocessing of the original point cloud data, that is, limit the three-dimensional coordinate range of the original point cloud data.

[0062] Specifically, the three-dimensional coordinate range processed by the point cloud segmentation device is: (x min , y min , z min , x max , y max , z max ). The point cloud segmentation device traverses the original point cloud data, removes the data points that exceed the above three-dimensional coordinate range, or limits the data points that exceed the above three-dimensional coordinate range to the boundary of the three-dimensional coordinate range.

[0063] It should be noted that before traversing, the point cloud segmentation device needs to align the center point of the original point cloud data with the center point of the three-dimensional coordinate range.

[0064] Step S12: Obtain the high-dimensional voxel point features of each data point in the original point cloud data.

[0065] In the embodiment of the present application, the point cloud segmentation device voxelizes the above three-dimensional coordinate range. Assuming the voxel grid size is (x v , y v , z v ), then W*L*H voxels can be obtained, where:

[0066]

[0067] Then, the point cloud segmentation device puts each data point of the original point cloud data into the above voxel grid.

[0068] Furthermore, the point cloud segmentation device can also perform feature augmentation on the original point cloud data.

[0069] Specifically, the point cloud segmentation device calculates the distance between the data point and the center point of the voxel grid where it is located Concatenates this distance with the data point in the feature dimension to obtain Obtain 7-dimensional features, that is, low-dimensional voxel points.

[0070] Finally, the point cloud segmentation device uses a multi-layer perceptron to extract the high-dimensional features of the data point, that is, the high-dimensional voxel point features, to obtain where N is the number of points, and C1 is the feature dimension after extraction by the multi-layer perceptron.

[0071] Step S13: Utilize the high-dimensional voxel point features belonging to the same voxel grid to obtain spatial voxel features.

[0072] In the embodiment of the present application, before the point cloud segmentation device obtains the spatial voxel features, it needs to first determine the high-dimensional voxel point features of each voxel grid. Therefore, the present application also provides a method for calculating voxel network indexes. For details, please refer to Figure 3 , Figure 3 is a schematic flowchart of another embodiment of the point cloud segmentation method provided by the present application.

[0073] As Figure 3 shown, the specific steps are as follows:

[0074] Step S21: Obtain the voxel grid size and the range value of the point cloud data.

[0075] Step S22: Determine the voxel grid index of the data point by using the coordinates of the data point, the range value of the point cloud data, and the voxel grid size.

[0076] In the embodiment of the present application, the i-th data point (x i , y i , zi , i i ) Index in the voxel grid is where floor represents rounding down.

[0077] Furthermore, the point cloud segmentation device determines all high-dimensional voxel point features belonging to the same voxel grid according to the voxel grid index, so as to map and obtain a 3D voxel representation.

[0078] Specifically, taking a certain grid v in the 3D space as an example, after calculation in step S12, it is obtained that points fall into this grid, that is, they have the same index. Let the n point features be respectively Take the maximum value of the n data points for each dimension feature to obtain the feature of this grid The rest of the non-empty grids are processed in the same way, and finally a 3D space voxel feature with a size of W×L×H is obtained.

[0079] Furthermore, in order to obtain multi-level features, the point cloud segmentation device can also downsample the data points falling into the same voxel grid, and downsample them to sizes such as 1 / 2 and 1 / 4 of the number of points in the original grid respectively. After completing the downsampling of the original point cloud data, the point cloud segmentation device executes the above step S13 to obtain multi-level 3D space voxel features, which are respectively denoted as V0, V1, V2, where i ∈ (0, 1, 2).

[0080] Step S14: Transform the spatial voxel feature through dimensional transformation into a bird's-eye view feature.

[0081] In the embodiment of the present application, the point cloud segmentation device respectively performs dimensional transformation on the 3 different levels of 3D space voxel features obtained in step S13 to obtain where i ∈ (0, 1, 2), and splices the 3 levels of 3D space voxel features in the channel dimension to obtain a multi-level bird's-eye view feature

[0082] Step S15: Perform point cloud segmentation prediction based on the bird's-eye view feature.

[0083] In the embodiment of the present application, the point cloud segmentation device inputs the multi-level bird's-eye view feature obtained in step S14 into the segmentation network to obtain a segmentation prediction result.

[0084] In this application, a point cloud segmentation device acquires original point cloud data; acquires high-dimensional voxel point features of each data point in the original point cloud data; uses the high-dimensional voxel point features belonging to the same voxel grid to acquire spatial voxel features; transforms the spatial voxel features through dimensional transformation into bird's-eye view features; and performs point cloud segmentation prediction based on the bird's-eye view features. Through the above point cloud segmentation method, the point cloud data is divided into voxels, so that data points of different heights and different categories are scattered in different voxel grids, and then the voxels in the height direction are recombined with the feature channels, achieving the purpose of retaining height information while facilitating subsequent use of a low-latency 2D convolutional network for the segmentation task.

[0085] The point cloud segmentation method of this application performs multi-level sampling on the points in the grid in order to obtain multi-level voxel features and retain more point-level feature information as much as possible.

[0086] The point cloud segmentation method of this application recombines 3D spatial voxel features into a 2D bird's-eye view, which is convenient for subsequent use of a low-latency 2D convolutional network to achieve point cloud semantic segmentation while retaining the voxel height information.

[0087] The point cloud segmentation method of this application maps the voxelized point cloud data into a bird's-eye view, uses 2D convolution to aggregate local features on the bird's-eye view, provides semantic information, and avoids performing complex local feature aggregation on the original point cloud.

[0088] Before mapping the point cloud into a bird's-eye view, the point cloud segmentation method of this application performs random sampling on the points in each voxel, downsamples them to 1 / 2 and 1 / 4 respectively, then maps to obtain three levels of bird's-eye views, and finally stitches the three levels of bird's-eye views together to retain as much point information as possible.

[0089] The point cloud segmentation method of this application divides the point cloud space according to 3D voxels. When mapping to a bird's-eye view, the voxels in the height direction are stitched to the channel dimension for subsequent use of 2D convolution, and at the same time, the purpose of retaining height information is achieved.

[0090] Those skilled in the art can understand that in the above method of the specific implementation manner, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0091] To implement the above point cloud segmentation method, this application also proposes a point cloud segmentation device. For details, please refer to Figure 4 , Figure 4 is a schematic structural diagram of an embodiment of the point cloud segmentation device provided by this application.

[0092] The point cloud segmentation device 500 of this embodiment includes: a point cloud acquisition module 51, a feature extraction module 52, a feature transformation module 53, and a point cloud segmentation module 54.

[0093] Among them, the point cloud acquisition module 51 is used to acquire original point cloud data.

[0094] The feature extraction module 52 is used to acquire the high-dimensional voxel point features of each data point in the original point cloud data.

[0095] The feature extraction module 52 is used to utilize the high-dimensional voxel point features belonging to the same voxel grid to acquire spatial voxel features.

[0096] The feature transformation module 53 is used to transform the spatial voxel features into bird's-eye view features through dimensional transformation.

[0097] The point cloud segmentation module 54 is used to perform point cloud segmentation prediction based on the bird's-eye view features.

[0098] To implement the above point cloud segmentation method, the present application also proposes a point cloud segmentation device. For details, please refer to Figure 5 , Figure 5 which is a schematic structural diagram of an embodiment of the point cloud segmentation device provided by the present application.

[0099] The point cloud segmentation device 400 of this embodiment includes a processor 41, a memory 42, an input / output device 43, and a bus 44.

[0100] The processor 41, the memory 42, and the input / output device 43 are respectively connected to the bus 44. Program data is stored in the memory 42, and the processor 41 is used to execute the program data to implement the point cloud segmentation method described in the above embodiment.

[0101] In the embodiment of the present application, the processor 41 can also be referred to as a CPU (Central Processing Unit, central processing unit). The processor 41 may be an integrated circuit chip with signal processing capabilities. The processor 41 can also be a general-purpose processor, a digital signal processor (DSP, Digital Signal Process), an application-specific integrated circuit (ASIC, Application Specific Integrated Circuit), a field-programmable gate array (FPGA, FieldProgrammable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor, or the processor 41 can also be any conventional processor, etc.

[0102] The present application also provides a computer storage medium. Please continue to refer to Figure 6 , Figure 6 FIG. Figure 6 is a schematic structural diagram of an embodiment of the computer storage medium provided by the present application. A computer program 61 is stored in the computer storage medium 600. When the computer program 61 is executed by a processor, it is used to implement the point cloud segmentation method of the above embodiment.

[0103] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0104] The above are only the embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structural or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present application.

Claims

1. A point cloud segmentation method, characterized in that: The point cloud segmentation method comprises: Get the original point cloud data; Acquire high-dimensional voxel point features of each data point in the original point cloud data; Using the high-dimensional voxel point features belonging to the same voxel grid, spatial voxel features are obtained; Transforming the spatial voxel features into bird's-eye view features through dimensional transformation; Point cloud segmentation prediction is performed based on the bird's-eye view features.

2. The point cloud segmentation method according to claim 1, characterized in that: The method of obtaining spatial voxel features by utilizing high-dimensional voxel point features belonging to the same voxel grid includes: Get all high-dimensional voxel point features belonging to the same voxel grid; The spatial voxel feature is composed by using the maximum eigenvalue of all the high-dimensional voxel point features in each feature dimension.

3. The point cloud segmentation method according to claim 2, characterized in that: Before obtaining the spatial voxel features by using the high-dimensional voxel point features belonging to the same voxel grid, the point cloud segmentation method further includes: Down-sampling high-dimensional voxel point features belonging to the same voxel grid at different sizes to obtain high-dimensional voxel point features at different levels; The method of obtaining spatial voxel features by utilizing high-dimensional voxel point features belonging to the same voxel grid includes: The spatial voxel features at different levels are obtained by utilizing the high-dimensional voxel point features at different levels belonging to the same voxel grid.

4. The point cloud segmentation method according to claim 3, characterized in that: The step of transforming the spatial voxel features into bird's-eye view features through dimensional transformation includes: The spatial voxel features at different levels are transformed into two-dimensional voxel features at different levels through dimensional transformation respectively; The two-dimensional voxel features at different levels are spliced ​​in the channel dimension to generate the bird's-eye view features.

5. The point cloud segmentation method according to claim 1, characterized in that: The obtaining of high-dimensional voxel point features of each data point in the original point cloud data includes: Obtaining the voxel center point where the data point is located; Splicing the data point with the voxel center point to obtain a low-dimensional voxel point of the data point; Extract high-dimensional voxel features of the low-dimensional voxel points.

6. The point cloud segmentation method according to claim 5, characterized in that: The point cloud segmentation method further comprises: Get the voxel grid size and point cloud data range value; The voxel grid index of the data point is determined using the coordinates of the data point, the point cloud data range value, and the voxel grid size.

7. The point cloud segmentation method according to claim 1, characterized in that: After obtaining the original point cloud data, the point cloud segmentation method further includes: Determine the three-dimensional coordinate range; Determine whether the three-dimensional coordinates of the data points of the original point cloud data are within the three-dimensional coordinate range; If not, the data point is removed, or the three-dimensional coordinates of the data point are updated according to the exceeded three-dimensional coordinate range boundary.

8. A point cloud segmentation device, characterized in that: The point cloud segmentation device comprises: a point cloud acquisition module, a feature extraction module, a feature transformation module, and a point cloud segmentation module; wherein, The point cloud acquisition module is used to acquire original point cloud data; The feature extraction module is used to obtain high-dimensional voxel point features of each data point in the original point cloud data; The feature extraction module is used to obtain spatial voxel features by utilizing high-dimensional voxel point features belonging to the same voxel grid; The feature transformation module is used to transform the spatial voxel features into bird's-eye view features through dimensional transformation; The point cloud segmentation module is used to perform point cloud segmentation prediction based on the bird's-eye view features.

9. A point cloud segmentation device, characterized in that: The point cloud segmentation device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the point cloud segmentation method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the point cloud segmentation method according to any one of claims 1 to 7.

Images