Method and device for determining semantics of three-dimensional point clouds

Abstract

The present invention discloses a method and apparatus for determining the semantics of a three-dimensional point cloud, relating to the field of computer technology. A specific implementation of the method includes: acquiring three-dimensional point cloud data of a target scene; determining multiple single-fragment features for each vertex in the three-dimensional point cloud data; determining multiple-fragment features for each vertex based on the multiple single-fragment features, wherein the multiple-fragment features are used to characterize the geometric shape characteristics of the vertex; and determining predicted semantic information for each vertex in the three-dimensional point cloud data based on the multiple-fragment features. This implementation can accurately determine the semantic information of vertices in the three-dimensional point cloud data.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for determining the semantics of a three-dimensional point cloud. Background Art

[0002] 3D point cloud data, a digital representation of real-world scenes, is a primary research topic in computer graphics. Mainstream research focuses on learning models from large-scale data to achieve semantic perception and understanding of 3D scenes, facilitating research and applications in emerging fields such as autonomous driving and intelligent robotics. However, existing technologies lack methods for accurately determining the semantic information in 3D point cloud data. Summary of the Invention

[0003] In view of this, an embodiment of the present invention provides a method and apparatus for determining semantics of a three-dimensional point cloud, which can accurately determine the semantic information of three-dimensional point cloud data.

[0004] In a first aspect, an embodiment of the present invention provides a method for determining semantics of a three-dimensional point cloud, comprising:

[0005] Obtain three-dimensional point cloud data of the target scene;

[0006] Determining multiple single-mode fragments for each vertex in the three-dimensional point cloud data according to multiple fragment segmentation methods, and respectively determining fragment features of each of the single-mode fragments to obtain multiple single-fragment features for each of the vertices;

[0007] Determining multiple fragment features of each vertex according to the multiple single fragment features of each vertex, wherein the multiple fragment features are used to characterize geometric shape features of the vertex;

[0008] Determine predicted semantic information of each vertex in the three-dimensional point cloud data based on the multiple fragment features of each vertex.

[0009] Optionally, determining a plurality of single-piece element features of each vertex in the three-dimensional point cloud data includes:

[0010] Determining multiple fragment segmentation methods and / or multiple segmentation levels;

[0011] Determining multiple fragment segmentation modes according to the fragment segmentation method and / or the segmentation level;

[0012] For each vertex in the three-dimensional point cloud data: using each of the fragment segmentation methods, the vertex is extracted and processed to obtain multiple single-mode fragments for the vertex; the fragment features of each of the single-mode fragments are determined respectively to obtain multiple single-mode fragment features of the vertex.

[0013] Optionally, determining the multiple fragment features of each vertex based on the multiple single fragment features of each vertex includes:

[0014] Determine a vertex descriptor of the vertex, where the vertex descriptor includes at least one of the following: three-dimensional coordinates of the vertex, normal information of the vertex, color information of the vertex, and curvature of the vertex;

[0015] The multiple single-fragment features of the vertex and the vertex descriptor of the vertex are input into the fragment pool network to obtain the multiple-fragment features of the vertex.

[0016] Optionally, determining the predicted semantic information of each vertex in the three-dimensional point cloud data according to the multiple fragment features of each vertex includes:

[0017] Combining the multiple fragment features of each vertex into a first point cloud feature of the three-dimensional point cloud data;

[0018] Processing the first point cloud features using a depth mapping operator to generate a depth image of the three-dimensional point cloud data;

[0019] Performing convolution and deconvolution processing on the depth image to generate depth image semantic features;

[0020] Processing the depth image semantic features using an inverse depth mapping operator to generate point cloud semantic features;

[0021] Converting the point cloud semantic features into an N*M format, where N is the number of vertices in the three-dimensional point cloud data and M is the number of semantic classification labels;

[0022] The converted point cloud semantic features are used to determine the predicted semantic information of each vertex in the three-dimensional point cloud data.

[0023] Optionally, the processing the first point cloud features using a depth mapping operator to generate a depth image of the three-dimensional point cloud data includes:

[0024] Processing the first point cloud features using a multilayer perceptron to generate second point cloud features;

[0025] performing depth mapping processing on the second point cloud features to generate a first depth image;

[0026] Performing convolution processing on the first depth image to generate a third point cloud feature;

[0027] Perform depth mapping processing on the third point cloud feature to generate a depth image of the three-dimensional point cloud data.

[0028] Optionally, performing convolution and deconvolution processing on the depth image to generate depth image semantic features includes:

[0029] Performing convolution processing on the depth image of the three-dimensional point cloud data to generate a fourth point cloud feature;

[0030] Deconvolution is performed on the fourth point cloud feature to generate a first depth image semantic feature.

[0031] Optionally, the processing of the depth image semantic features using an inverse depth mapping operator to generate point cloud semantic features includes:

[0032] Performing inverse depth mapping processing on the first depth image semantic features to generate first point cloud semantic features;

[0033] Performing deconvolution processing on the first point cloud semantic features to generate second depth image semantic features;

[0034] Performing inverse depth mapping processing on the second depth image semantic features to generate second point cloud semantic features;

[0035] The second point cloud semantic features are processed using a multi-layer perceptron to generate semantic features of the three-dimensional point cloud data.

[0036] In a second aspect, an embodiment of the present invention provides a device for determining semantics of a three-dimensional point cloud, comprising:

[0037] Point cloud acquisition module, used to obtain three-dimensional point cloud data of the target scene;

[0038] a single fragment determination module, configured to determine, based on a plurality of fragment segmentation modes, a plurality of single-mode fragments for each vertex in the three-dimensional point cloud data, and respectively determine a fragment feature of each of the single-mode fragments to obtain a plurality of single-fragment features for each of the vertices;

[0039] a multi-fragment determination module, configured to determine a multi-fragment feature of each vertex based on a plurality of single-fragment features of each vertex, wherein the multi-fragment feature is used to characterize a geometric shape feature of the vertex;

[0040] The semantic determination module is used to determine the predicted semantic information of each vertex in the three-dimensional point cloud data according to the multiple fragment features of each vertex.

[0041] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0042] one or more processors;

[0043] a storage device for storing one or more programs,

[0044] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any one of the above embodiments.

[0045] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having a computer program stored thereon, which implements the method described in any of the above embodiments when the program is executed by a processor.

[0046] One embodiment of the above invention has the following advantages or beneficial effects: determining multiple single-fragment features for each vertex in three-dimensional point cloud data, determining multiple-fragment features for each vertex based on the multiple single-fragment features for each vertex, and determining predicted semantic information for each vertex in the three-dimensional point cloud data based on the multiple-fragment features for each vertex.

[0047] Different fragment segmentation methods have different advantages and disadvantages, and different segmentation levels are used to determine different segmentation fragment sizes. Through different fragment segmentation methods and segmentation levels, a variety of fragment segmentation methods can be determined, thereby obtaining different fragment features, corresponding to the geometric shape features of fragments at different angles. In an embodiment of the present invention, multiple fragment features of a vertex are obtained using a variety of different fragment segmentation methods. Therefore, the multiple fragment features can characterize the rich geometric shape features of the vertex, and then based on the multiple fragment features of the vertex, the predicted semantic information of the vertex can be more accurately determined.

[0048] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0050] Figure 1 This is a schematic diagram of a process of a method for determining semantics of a three-dimensional point cloud provided by an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the structure of a fragment pool network provided by an embodiment of the present invention;

[0052] Figure 3 is a schematic diagram of the process of another method for determining the semantics of a three-dimensional point cloud provided by an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of a process for generating a depth image using a depth mapping operator according to an embodiment of the present invention is provided;

[0054] Figure 5 is a schematic diagram of a convolutional neural network architecture based on depth mapping provided by an embodiment of the present invention;

[0055] Figure 6 1 is a flow chart of another method for determining semantics of a three-dimensional point cloud provided by an embodiment of the present invention;

[0056] Figure 7a is a schematic diagram comparing multi-scale segmentation results provided by an embodiment of the present invention;

[0057] Figure 7b This is a schematic diagram comparing the segmentation results of multiple methods provided by an embodiment of the present invention.

[0058] Figure 8 1 is a schematic structural diagram of a device for determining semantics of a three-dimensional point cloud provided by an embodiment of the present invention;

[0059] Figure 9 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0061] Some early work on 3D point clouds did not directly use discrete point clouds as input for deep neural networks, but instead first structured the point clouds. For example, converting point cloud data into images or voxels can result in information loss in the 3D point cloud data. Therefore, research on deep learning methods that directly operate on discrete point clouds has high theoretical and practical value. Qi et al. proposed PointNet, the first deep neural network for point cloud data. This work was a milestone in point cloud deep learning research and serves as the prototype framework for most current point cloud deep learning. Subsequently, Qi et al. designed PointNet by adding sampling and grouping layers to perform deep feature computation. Engelmann et al. strengthened feature learning of the neighborhood of sampled points (adjacent vertices in 3D Euclidean space) and attempted to learn semantic features across patches. Tatarchenko et al. proposed tangent convolution, which learns surface geometric features from projected virtual tangent images. Li et al.'s PointCNN introduced an x-transformation for one-dimensional convolutional kernels, enabling hierarchical convolution of point clouds and learning deep features. Among them, the operator can order the disordered point cloud and use it to determine the order of convolution data input. However, the mapping from three-dimensional point cloud to ordered one-dimensional point sequence is sensitive to the local point cloud density and cannot cope with occlusion and other situations. These methods are all based on the PointNet framework and have similar limitations. In addition, there are technical routes such as graph neural networks and convolution expansion applied to point cloud deep learning. Landrieu et al. divided the scanned scene into similar elements and established a "superpoint" graph structure to learn the contextual relationship between object parts. Pan et al. used a parallel framework to expand the convolutional neural network in the conventional domain to a curved two-dimensional manifold. However, this method requires dense three-dimensional mesh data as input and is not suitable for three-dimensional point cloud scenes.

[0062] 3D point clouds are digital simulations of the real world. Existing construction methods and data formats pose many challenges to semantic segmentation, including:

[0063] Geometric features are difficult to quantify. Geometric features are a small-scale local feature description that does not contain plane-level information. Existing methods are far from human intuitive visual perception.

[0064] The definition and application of contextual information are complex. Contextual information refers to the common semantic combinations of an object's internal and external components and is not restricted to fixed form, quantity, or explicit semantic expression. Existing recognition methods cannot foresee the semantic content of an object, making accurate segmentation difficult. However, applying contextual rules requires a reasonable geometric segmentation structure. This creates a paradox between segmentation and recognition.

[0065] Three-dimensional point clouds are digital descriptions of the real world and are widely used in the fields of autonomous driving and intelligent robots. Compared with traditional two-dimensional images or videos, three-dimensional point clouds are more accurate, comprehensive, and conform to the natural spatial geometric structure of scenes. Three-dimensional point clouds can extract view-independent geometric features, such as rotation invariance, and can well reflect the semantics and contextual information of objects. Therefore, geometric analysis of three-dimensional point cloud data is an important research method to achieve scene semantic understanding. The flexible scene scale, uneven vertex distribution, and discrete and disordered data form of three-dimensional point clouds also bring many challenges to semantic segmentation. In order to meet these challenges, the solution of the embodiment of the present invention explores along the lines of geometric feature extraction and depth mapping, constructs a depth mapping operator to project the three-dimensional point cloud into a depth image to structure the discrete point cloud data, and then designs a fragment pooling network to condense and encode the geometric features, thereby enhancing the feature information of the vertices.

[0066] Figure 1 This is a schematic diagram of the process of a semantic determination method of a three-dimensional point cloud provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0067] Step 101: Acquire 3D point cloud data of the target scene.

[0068] The target scene is the scene of interest for analysis, and can be a scene depicting a person, a location, a road surface, or an obstacle. 3D point cloud data is a specialized form of information representation for the target scene. 3D point cloud data is characterized by the three dimensions of space it represents. Compared to 2D image data, 3D point cloud data, leveraging the third dimension, allows for a natural decoupling of objects from their background and provides greater uniformity.

[0069] Step 102: Determine multiple single-mode fragments for each vertex in the three-dimensional point cloud data based on multiple fragment segmentation methods, and determine the fragment features of each single-mode fragment to obtain multiple single-fragment features for each vertex.

[0070] Because each vertex contains very limited semantic information, similar to superpixels in image analysis, embodiments of the present invention extract local geometric fragments to provide richer information. Three-dimensional point cloud data can be segmented into multiple fragments based on its normals, color, and texture. Fragment segmentation is a common method for extracting geometric features from surface point clouds. Semantic annotation in embodiments of the present invention is performed at the point level, while fragment segmentation is only used to provide additional geometric information.

[0071] Since each fragment segmentation method has its own advantages and disadvantages, and most methods have adjustable segmentation levels. The segmentation level is used to determine the size of the fragment after segmentation (coarse or fine). The segmentation level can be determined according to specific needs, such as 1, 2, 4, 5, etc. Therefore, the method of the embodiment of the present invention adopts multiple fragment segmentation methods and multiple segmentation levels for segmentation. The fragment segmentation methods adopted may include: (1) Normal-based region growing (NRG) algorithm, the segmentation result is a plane-like object with almost consistent normal information. (2) Color-based region growing (CRG) algorithm, the segmentation result is a block of almost the same color. (3) Euclidean cluster extraction (ECE) algorithm, clustering the point cloud according to Euclidean distance, and the shape of the clustering result is uncertain.

[0072] Single-fragment features represent the fragment features determined using a single-fragment segmentation method. A single-fragment segmentation method corresponds to a fixed fragment segmentation method and segmentation level. The single-fragment features of a target vertex can be determined by extracting and processing the target vertex from the 3D point cloud data using a preset fragment segmentation method, obtaining a single-method fragment for the target vertex, and determining the fragment features corresponding to the single-method fragment, i.e., obtaining the single-fragment features of the target vertex for the preset fragment segmentation method.

[0073] Determine multiple fragment segmentation methods and / or multiple segmentation levels; determine multiple fragment segmentation methods based on the fragment segmentation methods and / or segmentation levels; for each vertex in the three-dimensional point cloud data: use each fragment segmentation method to extract and process the vertex to obtain multiple single-mode fragments for the vertex; determine the fragment features of each single-mode fragment respectively to obtain multiple single-mode fragment features of the vertex.

[0074] Step 103: Determine the multiple fragment features of each vertex based on the multiple single fragment features of each vertex.

[0075] Multiple fragment features for a vertex can be generated in a variety of ways. Multiple fragment features can be directly combined to generate multiple fragment features for that vertex. Alternatively, multiple fragment features can be directly combined and then processed using a multilayer perceptron to generate multiple fragment features for that vertex. Alternatively, multiple fragment features can be generated by summing or averaging the corresponding elements in each of the vertex's single fragment features.

[0076] Multiple fragment features are used to characterize the geometric shape features of vertices. The multiple fragment features of vertices are obtained using a variety of different fragment segmentation methods. Each fragment segmentation method corresponds to a different fragment segmentation method and / or segmentation level. Compared with the use of a single fragment segmentation method or segmentation level, the multiple fragment features in the embodiment of the present invention segment the three-dimensional point cloud data through a variety of different fragment segmentation methods, so that the multiple fragment features incorporate geometric shape features of multiple different angles, which can characterize richer geometric shape features of the vertices.

[0077] Step 104: Determine predicted semantic information of each vertex in the three-dimensional point cloud data based on the multiple fragment features of each vertex.

[0078] Semantics refers to the category information of real-world objects corresponding to 3D point cloud data. Predicting semantic information involves learning and analyzing information about each vertex in 3D point cloud data, such as coordinates, color, depth, and normal, to predict pixel- or vertex-level semantic classification labels.

[0079] Specifically, after learning and analyzing 3D point cloud data, the corresponding point cloud semantic features are obtained. These point cloud semantic features are in the form of N*M, where N is the number of vertices in the 3D point cloud data and M is the number of semantic classification labels. Each element in the point cloud semantic features represents the confidence score between the target vertex and M categories of objects. The category with the highest score is used as the final semantics for the target vertex.

[0080] In an embodiment of the present invention, multiple single-fragment features are determined for each vertex in the 3D point cloud data, and multiple fragment features are determined for each vertex based on the multiple single-fragment features. Based on the multiple fragment features of each vertex, predicted semantic information for each vertex in the 3D point cloud data is determined.

[0081] Different fragment segmentation methods have different advantages and disadvantages, and different segmentation levels are used to determine different segmentation fragment sizes. Through different fragment segmentation methods and segmentation levels, fragment features of different angles can be obtained, corresponding to different fragment geometric shape features. In an embodiment of the present invention, the multiple fragment features of a vertex are obtained using a variety of different fragment segmentation methods. Each fragment segmentation method corresponds to a different fragment segmentation method and / or segmentation level. Therefore, the multiple fragment features can characterize the rich geometric shape features of the vertex, and then based on the multiple fragment features of the vertex, the predicted semantic information of the vertex can be determined more accurately.

[0082] It should be noted that the aforementioned semantic determination method for 3D point clouds can also be applied, but not limited to, in the fields of autonomous driving, high-precision mapping, smart cities (e.g., unmanned vehicle delivery processes), smart security, smart transportation, and smart healthcare. For example, in the field of autonomous driving, 3D point cloud data within a certain range around the vehicle can be acquired. Using the aforementioned semantic determination method for 3D point clouds, all obstacles, traffic signs, and traffic lights within that range can be detected and identified, enabling the development of rational path planning solutions and ensuring the safety of autonomous driving operations.

[0083] For example, in the field of smart healthcare, three-dimensional point cloud data of various organs or parts of the human body or animals can be obtained, and the semantic determination method of the above-mentioned three-dimensional point cloud can be used to identify areas such as redness, swelling, bleeding or lesions, thereby providing assistance to subsequent doctors in diagnosing and treating patients.

[0084] In one embodiment of the present invention, multiple fragment features of each vertex are determined based on multiple single fragment features of each vertex, including: determining a vertex descriptor of the vertex, the vertex descriptor including at least one of the following: three-dimensional coordinates of the vertex, normal information of the vertex, color information of the vertex, and curvature of the vertex; inputting the multiple single fragment features of the vertex and the vertex descriptor of the vertex into the fragment pool network to obtain multiple fragment features of the vertex.

[0085] Assume that the 3D point cloud data contains N vertices. i |i=1,2,..,N} represents the vertex set of the point cloud, label i is the semantic annotation corresponding to each vertex. i With a 10-dimensional vertex descriptor:

[0086] Where (x, y, z) is the three-dimensional coordinate of the point; (n x , n y , n z ) is the normal information of each point estimated by principal component analysis using the neighborhood; (r, g, b) are the three channels of color information: red, green, and blue (optional); c is the curvature defined using the PCL standard library.

[0087] Use L different fragment segmentation methods for each point P i L fragments are extracted, and L fragment features are obtained: F1, F2, ..., F L and point descriptors However, not all segmentation results are meaningful for the fragments around a given point. Using all information is not only redundant, but also the mis-segmentation results caused by specific segmentation algorithms and control parameters can mislead model learning and cause the learning rate to drop.

[0088] In order to obtain a simplified and meaningful feature representation, an embodiment of the present invention proposes a novel pooling neural network. L is the number of fragment segmentation methods used, and each fragment segmentation method corresponds to a different fragment segmentation method and / or segmentation level.

[0089] Figure 2 This is a schematic diagram of the structure of a fragment pool network provided by an embodiment of the present invention. Figure 2 As shown, for each vertex, the input includes a fragment descriptor F j (j=1, 2, ..., L)F j (j=1, 2, ..., L) and point descriptors The output is the MPF (Multi-method Patch Feature) feature. Score is the confidence score for each category. MLP stands for Multilayer Perceptron, and the number in parentheses indicates the number of neurons. FC stands for Fully Connected Layer.

[0090] The fragment pool network firstly transforms all the fragment features F j (j=1, 2, ..., L) as input, and then perform two stages of MLP (Multi-Layer Perceptron) to independently transform the fragment features of the same level, thereby obtaining a feature representation of L×252 dimensions. In order to extract the most informative features, the fragment pooling network uses a maximum pooling layer to reduce the dimension to 252 as the high-level fragment feature. This feature is further compared with the point descriptor Combined and multi-fragment features MPF obtained by another multi-layer perceptron i During the training process, for each vertex in the training set, the network input includes point descriptors and fragment features, and uses the vertex label information label i To supervise, an M-dimensional semantic score Score is predicted. Therefore, even without higher-level context, the fragment pooling network can be used to predict point cloud semantics. In experiments testing the accuracy of semantic determination methods for 3D point clouds, the semantic prediction results of the fragment pooling network are used as a baseline for parallel comparison.

[0091] Figure 3 FIG. 1 is a schematic diagram of the process of another method for determining the semantics of a three-dimensional point cloud provided by an embodiment of the present invention. Figure 3 As shown, the method includes:

[0092] Step 301: Acquire three-dimensional point cloud data of the target scene.

[0093] Step 302: Determine multiple single-piece features of each vertex in the three-dimensional point cloud data.

[0094] Step 303: Determine a multi-fragment feature of each vertex based on the multiple single-fragment features of each vertex. The multi-fragment feature is used to characterize the geometric shape features of the vertex.

[0095] Step 304: Combine the multiple fragment features of each vertex into a first point cloud feature of the three-dimensional point cloud data.

[0096] Step 305: Perform convolution and deconvolution processing on the depth image to generate depth image semantic features.

[0097] Step 306: Use the inverse depth mapping operator to process the depth image semantic features to generate point cloud semantic features.

[0098] Step 307: Convert the point cloud semantic features into the form of N*M, where N is the number of vertices in the three-dimensional point cloud data and M is the number of semantic classification labels.

[0099] Step 308: Using the converted point cloud semantic features, determine the predicted semantic information of each vertex in the three-dimensional point cloud data.

[0100] The depth mapping operator is used to generate multiple local depth maps of point cloud data. The depth mapping operator DM(p,q) transforms the upper point cloud (containing p vertices P1, P2, ..., P p ) and the sampling point cloud (q sampling vertices ) as input and generate a local depth image based on the sampling point neighborhood. For each sampling point Use the k-nearest neighbor algorithm to search for its neighboring points and project them to In order to improve the learning effect, it is necessary to always ensure that the depth map is consistent with The depth image with a resolution of K×K is then sampled in the tangent plane, and the depth (the absolute distance between adjacent points and the tangent plane) is used as an additional information channel. Similarly, the vertex features of adjacent points are projected to generate a two-dimensional feature image. This operation aims to learn the rich local structural features of the points. The value of K can be set according to specific needs, for example, K can be set to 16.

[0101] Since the points are discrete, the method of the embodiment of the present invention performs interpolation between adjacent projected points to obtain a complete depth feature image. Figure 4 FIG. 1 is a flow chart of generating a depth image using a depth mapping operator according to an embodiment of the present invention. Figure 4 As shown, such depth images are insensitive to sampling density and provide compact and rich geometric information for subsequent learning.

[0102] After the convolution layer, the method of the embodiment of the present invention introduces an inverse depth operation operator. The inverse depth operation operator is used to convert the semantic features of the depth image into the semantic features of the upper point cloud. The inverse depth mapping operator RDM(q, p) takes the semantic features of the depth image output by the convolutional neural network as input and outputs points P1, P2, ..., P with semantic label information. p , from the closest P i of Extract P from the associated depth image i Depth mapping and inverse depth mapping are key operators in building an end-to-end convolutional neural network structure.

[0103] Convolutional neural networks are an excellent method for machine learning on structured data. However, the three-dimensional reconstructed scene point cloud data is usually unstructured, and traditional convolutional neural network methods are difficult to apply. Therefore, the method of the embodiment of the present invention designs novel depth mapping and inverse depth mapping operators to build a bridge between convolutional neural networks and point cloud data. The depth mapping operator maps the point cloud into several depth images, and then the two-dimensional convolutional neural network can be easily applied to the depth image. The inverse depth mapping operator is the inverse operation of the depth mapping operator. It remaps the sampled depth image back to the upper point cloud with deconvolution features to complete the feature return.

[0104] The framework of the method of the embodiment of the present invention consists of two main network models. The first is the fragment pooling network, which can learn to extract the geometric features of the fragments. First, multiple methods of fragment segmentation are tried on the point cloud, where each fragment segmentation method has different segmentation levels and / or fragment segmentation methods. Second, the geometric features of these fragment results are extracted and the fragment feature descriptor F is calculated. i Finally, the multi-fragment feature descriptor F i With vertex descriptor The fragment pooling network is used as input to jointly generate the MPF (Multiple Fragment Features) features for each vertex in the point cloud. The second network is a convolutional neural network based on depth mapping. The classification network takes the MPF of the 3D point cloud as input and uses an embedded convolutional neural network for learning to output the semantic label of each point.

[0105] Figure 5 FIG is a schematic diagram of a convolutional neural network architecture based on depth mapping provided by an embodiment of the present invention. Figure 5 As shown, the network consists of N vertices and their geometric features MPF i As input, for the i-th vertex, the previously introduced fragment pooling network is used to extract point features. The network output is the scores of N vertices for M semantic categories, and the final semantic classification is determined by the maximum score.

[0106] exist Figure 5 In the method, a depth mapping operator is used to process the first point cloud feature to generate a depth image of the three-dimensional point cloud data, including: using a multi-layer perceptron to process the first point cloud feature to generate a second point cloud feature; performing depth mapping processing on the second point cloud feature to generate a first depth image; performing convolution processing on the first depth image to generate a third point cloud feature; and performing depth mapping processing on the third point cloud feature to generate a depth image of the three-dimensional point cloud data.

[0107] Performing convolution and deconvolution processing on the depth image to generate depth image semantic features, including: performing convolution processing on the depth image of the three-dimensional point cloud data to generate a fourth point cloud feature; performing deconvolution processing on the fourth point cloud feature to generate a first depth image semantic feature.

[0108] The inverse depth mapping operator is used to process the semantic features of the depth image to generate point cloud semantic features, including: performing inverse depth mapping processing on the semantic features of the first depth image to generate first point cloud semantic features; performing deconvolution processing on the first point cloud semantic features to generate second depth image semantic features; performing inverse depth mapping processing on the semantic features of the second depth image to generate second point cloud semantic features; and using a multilayer perceptron to process the second point cloud semantic features to generate semantic features of three-dimensional point cloud data.

[0109] FPS(x, y) represents the farthest point sampled, with x and y representing the size of the point cloud before and after sampling. This method performs two phases of farthest point sampling, sampling N1 and N2 points respectively. N1 and N2 can be set as needed, for example, N1 = N / 32 and N2 = N1 / 32.

[0110] DM(x, y) represents a depth mapping operation, which projects the neighborhood of each sample point in the upper point cloud onto the tangent plane in Euclidean space, generating a depth image with a resolution of K*K around each sample point. x and y are the number of input point clouds and the number of sampled point clouds output, respectively. Depth information is added to the vertex features as an additional input channel and combined with the vertex features. The structured data form of the depth image can support two-dimensional convolution and deconvolution operations, thereby extracting deep semantic features. RDM(x, y) records the back projection of the DM(x, y) mapping relationship, passing the depth image and the learned deep-level features back to the upper point cloud.

[0111] Conv2d(x, y) and DeConv2d(x, y) refer to convolution / deconvolution operators on the depth image, where x and y represent the channel sizes of the input and output. A skip connection layer is added after the deconvolution layer DeConv2d(x, y) and introduced between points of the same color, passing detailed information to the later stages of the network and avoiding information loss. The final output is a score of dimension N*M. That is, for N points, confidence scores are obtained for M object classes, and the class with the highest score is used as the final semantics of the vertex.

[0112] The solution of this method embodiment designs a deep convolutional neural network, which takes an unstructured discrete point cloud as input and aims to output the semantic information of each point. Figure 6 FIG. 1 is a flow chart of another method for determining the semantics of a three-dimensional point cloud provided by an embodiment of the present invention. Figure 6 As shown in the figure, DM and RDM represent the transformation of point cloud (a) (e) and depth map (b) using depth mapping operator / inverse depth mapping operator; MPF (Multi-method Patch Feature) represents the multi-method patch feature generated by the patch pooling network; CNN combines MPF and depth map structure to generate semantic features (d).

[0113] In an embodiment of the present invention, how the number of fragment segmentation methods and the selection of segmentation scales affect the performance of the fragment pool network is specifically evaluated. Since the input of the fragment pool network is the fragment segmentation results of different methods, the performance of the fragment pool network at different segmentation levels is evaluated experimentally. The evaluation index uses the overall accuracy of the fragment pool network to evaluate the robustness of features using multiple parameter methods. In addition, the effectiveness of the combination of different segmentation methods is further evaluated. Experiments have shown that with the increase of segmentation levels and the combination of more fragment segmentation methods, the accuracy is improved. This verifies the effectiveness of the fragment pool network for feature extraction, indicating that it can condense the geometric features of many fragment segmentation methods.

[0114] Figure 7a 3 is a schematic diagram for comparing multi-scale segmentation results provided by an embodiment of the present invention. Figure 7b : This is a schematic diagram comparing the segmentation results of multiple methods provided by an embodiment of the present invention. Among them, CRG is a color-based region growing algorithm, ECE is a Euclidean cluster extraction algorithm, NRG is a normal-based region growing algorithm, and CRG is a color-based region growing algorithm. Figure 7a and 7b It can be clearly seen that the more segmentation levels and fragment segmentation methods used, the more reliable the extracted geometric features and the higher the semantic segmentation accuracy.

[0115] The fragment segmentation method and segmentation level can be determined manually or randomly. During implementation, try to choose multiple segmentation methods and more segmentation levels to make the semantic segmentation results more accurate.

[0116] Figure 8 FIG. 1 is a schematic diagram of a structure of a semantic determination device for a three-dimensional point cloud provided by an embodiment of the present invention. Figure 8 As shown, the device includes:

[0117] Point cloud acquisition module 801, used to acquire three-dimensional point cloud data of the target scene;

[0118] a single fragment determining module 802 for determining, based on a plurality of fragment segmentation modes, a plurality of single-mode fragments for each vertex in the three-dimensional point cloud data, and determining fragment features of each of the single-mode fragments to obtain a plurality of single-fragment features for each vertex;

[0119] A multi-fragment determination module 803 is configured to determine a multi-fragment feature of each vertex based on multiple single-fragment features of each vertex, wherein the multi-fragment feature is used to characterize the geometric shape features of the vertex;

[0120] The semantic determination module 804 is configured to determine predicted semantic information of each vertex in the three-dimensional point cloud data according to the multiple fragment features of each vertex.

[0121] Optionally, the single-chip element determination module 802 is specifically configured to:

[0122] Determining multiple fragment segmentation methods and / or multiple segmentation levels;

[0123] Determining multiple fragment segmentation modes according to the fragment segmentation method and / or the segmentation level;

[0124] For each vertex in the three-dimensional point cloud data: using each of the fragment segmentation methods, the vertex is extracted and processed to obtain multiple single-mode fragments for the vertex; the fragment features of each of the single-mode fragments are determined respectively to obtain multiple single-mode fragment features of the vertex.

[0125] Optionally, the multi-fragment determination module 803 is specifically configured to:

[0126] Determine a vertex descriptor of the vertex, where the vertex descriptor includes at least one of the following: three-dimensional coordinates of the vertex, normal information of the vertex, color information of the vertex, and curvature of the vertex;

[0127] The multiple single-fragment features of the vertex and the vertex descriptor of the vertex are input into the fragment pool network to obtain the multiple-fragment features of the vertex.

[0128] Optionally, the semantic determination module 804 is specifically configured to:

[0129] Combining the multiple fragment features of each vertex into a first point cloud feature of the three-dimensional point cloud data;

[0130] Processing the first point cloud features using a depth mapping operator to generate a depth image of the three-dimensional point cloud data;

[0131] Performing convolution and deconvolution processing on the depth image to generate depth image semantic features;

[0132] Processing the depth image semantic features using an inverse depth mapping operator to generate point cloud semantic features;

[0133] Converting the point cloud semantic features into an N*M format, where N is the number of vertices in the three-dimensional point cloud data and M is the number of semantic classification labels;

[0134] The converted point cloud semantic features are used to determine the predicted semantic information of each vertex in the three-dimensional point cloud data.

[0135] Optionally, the semantic determination module 804 is specifically configured to:

[0136] Processing the first point cloud features using a multilayer perceptron to generate second point cloud features;

[0137] performing depth mapping processing on the second point cloud features to generate a first depth image;

[0138] Performing convolution processing on the first depth image to generate a third point cloud feature;

[0139] Perform depth mapping processing on the third point cloud feature to generate a depth image of the three-dimensional point cloud data.

[0140] Optionally, the semantic determination module 804 is specifically configured to:

[0141] Performing convolution processing on the depth image of the three-dimensional point cloud data to generate a fourth point cloud feature;

[0142] Deconvolution is performed on the fourth point cloud feature to generate a first depth image semantic feature.

[0143] Optionally, the semantic determination module 804 is specifically configured to:

[0144] Performing inverse depth mapping processing on the first depth image semantic features to generate first point cloud semantic features;

[0145] Performing deconvolution processing on the first point cloud semantic features to generate second depth image semantic features;

[0146] Performing inverse depth mapping processing on the second depth image semantic features to generate second point cloud semantic features;

[0147] The second point cloud semantic features are processed using a multi-layer perceptron to generate semantic features of the three-dimensional point cloud data.

[0148] An embodiment of the present invention provides an electronic device, including:

[0149] one or more processors;

[0150] a storage device for storing one or more programs,

[0151] When one or more programs are executed by one or more processors, the one or more processors implement the method of any of the above embodiments.

[0152] Reference below Figure 9 , which shows a schematic structural diagram of a computer system 900 of a terminal device suitable for implementing an embodiment of the present invention. Figure 9 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0153] like Figure 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage unit 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the system 900 are also stored in the RAM 903. The CPU 901, ROM 902, and RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0154] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, and the like; an output section 907 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 908 including a hard disk and the like; and a communication section 909 including a network interface card such as a LAN card or a modem. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 910 as needed, so that computer programs read therefrom can be installed into the storage section 908 as needed.

[0155] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from a removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above-mentioned functions defined in the system of the present invention are performed.

[0156] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0157] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0158] The modules described in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, they may be described as: a point cloud acquisition module, a single-fragment determination module, a multi-fragment determination module, and a semantic determination module. The names of these modules do not, in some cases, limit the modules themselves. For example, the point cloud acquisition module may also be described as a "module for acquiring three-dimensional point cloud data of a target scene."

[0159] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiments, or may exist independently without being incorporated into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the device, the device includes:

[0160] Obtain three-dimensional point cloud data of the target scene;

[0161] Determining multiple single-mode fragments for each vertex in the three-dimensional point cloud data according to multiple fragment segmentation methods, and respectively determining fragment features of each of the single-mode fragments to obtain multiple single-fragment features for each of the vertices;

[0162] Generating a multiple fragment feature of each vertex according to the multiple single fragment features of each vertex, wherein the multiple fragment features are used to characterize the geometric shape features of the vertex;

[0163] Determine predicted semantic information of each vertex in the three-dimensional point cloud data based on the multiple fragment features of each vertex.

[0164] According to the technical solution of an embodiment of the present invention, multiple single-fragment features are determined for each vertex in the three-dimensional point cloud data, and multiple fragment features are determined for each vertex based on the multiple single-fragment features. Based on the multiple fragment features of each vertex, predicted semantic information for each vertex in the three-dimensional point cloud data is determined.

[0165] Different fragment segmentation methods have different advantages and disadvantages, and different segmentation levels are used to determine different segmentation fragment sizes. Through different fragment segmentation methods and segmentation levels, fragment features of different angles can be obtained, corresponding to different fragment geometric shape features. In an embodiment of the present invention, the multiple fragment features of a vertex are obtained using a variety of different fragment segmentation methods. Each fragment segmentation method corresponds to a different fragment segmentation method and / or segmentation level. Therefore, the multiple fragment features can characterize the rich geometric shape features of the vertex, and then based on the multiple fragment features of the vertex, the predicted semantic information of the vertex can be determined more accurately.

[0166] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for determining the semantics of a three-dimensional point cloud, characterized in that: include: Obtain three-dimensional point cloud data of the target scene; Determining multiple single-mode fragments for each vertex in the three-dimensional point cloud data according to multiple fragment segmentation methods, and respectively determining fragment features of each of the single-mode fragments to obtain multiple single-fragment features for each of the vertices; Generating a multiple fragment feature of each vertex according to the multiple single fragment features of each vertex, wherein the multiple fragment features are used to characterize the geometric shape features of the vertex; Determining predicted semantic information of each vertex in the three-dimensional point cloud data according to the multiple fragment features of each vertex; Determining the predicted semantic information of each vertex in the three-dimensional point cloud data based on the multiple fragment features of each vertex includes: combining the multiple fragment features of each vertex into a first point cloud feature of the three-dimensional point cloud data; Using a depth mapping operator, the first point cloud features are processed to generate a depth image of the three-dimensional point cloud data; convolution and deconvolution are performed on the depth image to generate depth image semantic features; using an inverse depth mapping operator, the depth image semantic features are processed to generate point cloud semantic features; the point cloud semantic features are converted into an N*M form, where N is the number of vertices in the three-dimensional point cloud data and M is the number of semantic classification labels; and using the converted point cloud semantic features, the predicted semantic information of each vertex in the three-dimensional point cloud data is determined.

2. The method according to claim 1, characterized in that The step of determining a plurality of single-mode fragments for each vertex in the three-dimensional point cloud data according to the plurality of fragment segmentation modes, and respectively determining a fragment feature of each of the single-mode fragments to obtain a plurality of single-fragment features for each of the vertices, includes: Determining multiple fragment segmentation methods and / or multiple segmentation levels; Determining multiple fragment segmentation modes according to the fragment segmentation method and / or the segmentation level; For each vertex in the three-dimensional point cloud data: using each of the fragment segmentation methods, the vertex is extracted and processed to obtain multiple single-mode fragments for the vertex; the fragment features of each single-mode fragment are determined respectively to obtain multiple single-fragment features of the vertex.

3. The method according to claim 1, characterized in that Generating multiple fragment features of each vertex according to the multiple single fragment features of each vertex includes: Determine a vertex descriptor of the vertex, where the vertex descriptor includes at least one of the following: three-dimensional coordinates of the vertex, normal information of the vertex, color information of the vertex, and curvature of the vertex; The multiple single-fragment features of the vertex and the vertex descriptor of the vertex are input into the fragment pool network to obtain the multiple-fragment features of the vertex.

4. The method according to claim 1, wherein The processing of the first point cloud feature by using a depth mapping operator to generate a depth image of the three-dimensional point cloud data includes: Processing the first point cloud features using a multilayer perceptron to generate second point cloud features; performing depth mapping processing on the second point cloud features to generate a first depth image; Performing convolution processing on the first depth image to generate a third point cloud feature; Perform depth mapping processing on the third point cloud feature to generate a depth image of the three-dimensional point cloud data.

5. The method according to claim 1, wherein The performing convolution and deconvolution processing on the depth image to generate depth image semantic features includes: Performing convolution processing on the depth image of the three-dimensional point cloud data to generate a fourth point cloud feature; Deconvolution is performed on the fourth point cloud feature to generate a first depth image semantic feature.

6. The method according to claim 5, characterized in that The method of processing the depth image semantic features using an inverse depth mapping operator to generate point cloud semantic features includes: Performing inverse depth mapping processing on the first depth image semantic features to generate first point cloud semantic features; Performing deconvolution processing on the first point cloud semantic features to generate second depth image semantic features; Performing inverse depth mapping processing on the second depth image semantic features to generate second point cloud semantic features; The second point cloud semantic features are processed using a multi-layer perceptron to generate semantic features of the three-dimensional point cloud data.

7. A semantic determination device for a three-dimensional point cloud, characterized in that: include: Point cloud acquisition module, used to obtain three-dimensional point cloud data of the target scene; a single fragment determination module, configured to determine, based on a plurality of fragment segmentation modes, a plurality of single-mode fragments for each vertex in the three-dimensional point cloud data, and respectively determine a fragment feature of each of the single-mode fragments to obtain a plurality of single-fragment features for each of the vertices; a multi-fragment determination module, configured to determine a multi-fragment feature of each vertex based on a plurality of single-fragment features of each vertex, wherein the multi-fragment feature is used to characterize a geometric shape feature of the vertex; a semantic determination module, configured to generate predicted semantic information of each vertex in the three-dimensional point cloud data based on multiple fragment features of each vertex; The semantic determination module is specifically used to: combine the multiple fragment features of each vertex into the first point cloud feature of the three-dimensional point cloud data; use the depth mapping operator to process the first point cloud feature to generate a depth image of the three-dimensional point cloud data; perform convolution and deconvolution on the depth image to generate depth image semantic features; use the inverse depth mapping operator to process the depth image semantic features to generate point cloud semantic features; convert the point cloud semantic features into N*M form, where N is the number of vertices in the three-dimensional point cloud data and M is the number of semantic classification labels; use the converted point cloud semantic features to determine the predicted semantic information of each vertex in the three-dimensional point cloud data.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images