Point cloud semantic segmentation method, device and electronic device

Abstract

The present disclosure provides a point cloud semantic segmentation method, device and electronic device, which relate to the field of artificial intelligence. The method includes: obtaining an original point cloud, encoding the point cloud data corresponding to the original point cloud through a coding network model to obtain point cloud coding features; inputting the point cloud coding features into a decoding network model, obtaining edge features corresponding to the original point cloud according to the point cloud coding features through the edge module branch in the decoding network model, and obtaining point features corresponding to the original point cloud according to the point cloud coding features and the edge features through the point module branch in the decoding network model; performing classification prediction based on the point features and the edge features to obtain classification information corresponding to each point in the original point cloud. The present disclosure can increase the scope of information transmission and better utilize local environmental information through the mutual communication and coordination of points and edges, thereby improving the accuracy and efficiency of point cloud semantic segmentation.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a point cloud semantic segmentation method, a point cloud semantic segmentation device and an electronic device. Background Art

[0002] With the development of computer technology, computer vision and machine learning researchers are becoming more and more interested in the problem of image semantic segmentation. More and more application scenarios require accurate and efficient segmentation technology, such as autonomous driving, indoor navigation, and even virtual reality and augmented reality.

[0003] At present, the technical solutions for segmenting three-dimensional application scenes based on deep learning mainly include voxel-based solutions and point cloud-based solutions. However, due to the sparsity of the spatial distribution of three-dimensional objects, directly voxelizing the three-dimensional space will have problems such as low utilization of storage space; due to the irregularity and disorder of point clouds, the convolutional layer used to extract local areas cannot be applied to point clouds.

[0004] In view of this, there is an urgent need in this field to develop a new point cloud semantic segmentation method and device.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The embodiments of the present disclosure provide a point cloud semantic segmentation method, a point cloud semantic segmentation device and an electronic device, which can accurately and efficiently perform point cloud semantic segmentation to at least a certain extent, thereby helping computers understand three-dimensional environments.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to one aspect of an embodiment of the present disclosure, a point cloud semantic segmentation method is provided, including: obtaining an original point cloud, encoding point cloud data corresponding to the original point cloud through an encoding network model to obtain point cloud encoding features; inputting the point cloud encoding features into a decoding network model, obtaining edge features corresponding to the original point cloud according to the point cloud encoding features through an edge module branch in the decoding network model, and obtaining point features corresponding to the original point cloud according to the point cloud encoding features and the edge features through a point module branch in the decoding network model; performing classification prediction based on the point features and the edge features to obtain classification information corresponding to each point in the original point cloud.

[0009] According to one aspect of an embodiment of the present disclosure, a point cloud semantic segmentation device is provided, including: an encoding module, used to obtain an original point cloud, and encode point cloud data corresponding to the original point cloud through an encoding network model to obtain point cloud encoding features; a decoding module, used to input the point cloud encoding features into a decoding network model, and obtain edge features corresponding to the original point cloud according to the point cloud encoding features through an edge module branch in the decoding network model, and obtain point features corresponding to the original point cloud according to the point cloud encoding features and the edge features through a point module branch in the decoding network model; a classification module, used to perform classification prediction based on the point features and the edge features to obtain classification information corresponding to each point in the original point cloud.

[0010] In some embodiments of the present disclosure, based on the aforementioned scheme, the encoding module is configured to: downsample the original point cloud multiple times to generate multiple levels of point cloud layers containing different numbers of points; perform feature extraction on the point cloud data corresponding to each level of the point cloud layer to obtain point cloud coding features corresponding to each level of the point cloud layer.

[0011] In some embodiments of the present disclosure, based on the aforementioned scheme, the point cloud coding features include intermediate point cloud coding features and zero-level point cloud coding features, wherein the zero-level point cloud coding features are generated by extracting features from the last-level point cloud layer generated by downsampling, and the intermediate point cloud coding features are generated by extracting features from the point cloud layer before the last-level point cloud layer.

[0012] In some embodiments of the present disclosure, the intermediate point cloud coding features include multi-level intermediate point cloud coding features; the decoding network model includes multi-level decoding network sub-models, and each level of the decoding network sub-model includes a point module and an edge module; based on the aforementioned scheme, the decoding module includes: a feature extraction unit, used to input M-level point features and M+1-level intermediate point cloud coding features into the M+1-level decoding network sub-model, obtain M-level edge features according to the M-level point features and M-1-level edge features through the edge module in the M+1-level decoding network sub-model, and obtain M+1-level point features according to the M+1-level intermediate point cloud coding features, the M-level point features and the M-1-level edge features through the point module in the M+1-level decoding network sub-model; wherein M is a positive integer less than the total number of decoding network sub-models.

[0013] In some embodiments of the present disclosure, based on the aforementioned scheme, the decoding module is configured to: input the zero-level point cloud coding features and the first-level intermediate point cloud coding features into the first-level decoding network sub-model, obtain the zero-level edge features according to the zero-level point cloud coding features through the edge module in the first-level decoding network sub-model, and obtain the first-level point features and zero-level edge features according to the first-level intermediate point cloud coding features, the zero-level point cloud coding features and the zero-level edge features through the point module in the first-level decoding network sub-model.

[0014] In some embodiments of the present disclosure, based on the aforementioned scheme, the feature extraction unit includes: an edge feature acquisition unit, used to perform feature extraction on the position information of each point in the M-level point features, the M-1-level graph structure and the M-1-level edge features through the edge module to obtain the M-level edge features; a point feature acquisition unit, used to perform feature extraction on the M-level edge features, the M-level point features and the M+1-level intermediate point cloud coding features through the point module to obtain the M+1-level point features.

[0015] In some embodiments of the present disclosure, based on the aforementioned scheme, the edge feature acquisition unit includes: a graph structure construction unit, used to construct an M-level graph structure according to the position information of each point in the M-level point features and the M-1-level graph structure; an upsampling unit, used to upsample the edges according to the M-1-level edge features and the M-level graph structure to obtain upsampled edge features; and an encoding unit, used to encode the M-level graph structure, the M-level point features and the upsampled edge features to obtain the M-level edge features.

[0016] In some embodiments of the present disclosure, based on the aforementioned scheme, the graph structure construction unit is configured as follows: determining any point in the M-level point cloud layer as a first target point, obtaining a first neighbor point corresponding to the first target point in the M-level point cloud layer, and the distance between the first neighbor point and the first target point does not exceed a preset distance threshold; connecting each of the first target points in the M-level point cloud layer and the first neighbor point corresponding to each of the first target points to obtain an M-level initial graph structure; determining any edge in the M-level initial graph structure as a first target edge, and obtaining two endpoints corresponding to the first target edge. ; Determine the neighboring points corresponding to each of the endpoints in the M-1 level point cloud layer, determine a target neighboring point from the neighboring points corresponding to each of the endpoints, connect the target neighboring points to form a detection edge, and determine whether the detection edge is an edge in the M-1 level graph structure; when the detection edge is not an edge in the M-1 level graph structure, delete the target edge from the M-level initial graph structure; repeatedly determine whether the detection edges corresponding to all edges in the M-1 level initial graph structure are edges in the M-1 level graph structure, and update all edges according to the determination result to obtain the M-level graph structure.

[0017] In some embodiments of the present disclosure, based on the aforementioned scheme, the upsampling unit is configured to: determine any edge in the M-level graph structure as an edge to be processed; determine a neighboring edge corresponding to the edge to be processed in the edges corresponding to the M-1-level edge feature according to the two endpoints corresponding to the edge to be processed; take the intersection of the neighboring edge and the edge corresponding to the M-1-level edge feature, and interpolate the edge feature corresponding to the edge in the intersection according to the distance between the two endpoints corresponding to the edge to be processed, so as to obtain the upsampled edge feature.

[0018] In some embodiments of the present disclosure, based on the aforementioned scheme, the encoding unit is configured to: take any edge in the M-level graph structure as the second target edge, and obtain two endpoints corresponding to the second target edge; obtain point features corresponding to each of the endpoints from the M-level point features, connect the point features corresponding to each of the endpoints, and perform feature extraction on the connected point features to obtain connection point features; concatenate the connection point features and the upsampled edge features corresponding to the second target edge, and perform feature extraction on the concatenated features to obtain M-level edge sub-features corresponding to the second target edge; repeat the above steps until the M-level edge sub-features corresponding to each edge in the M-level graph structure are obtained, and determine the M-level edge features based on the M-level edge sub-features corresponding to each edge.

[0019] In some embodiments of the present disclosure, the last-level point cloud layer is a zero-level point cloud layer; based on the aforementioned scheme, the point cloud semantic segmentation device can also be configured to: determine any point in the zero-level point cloud layer as a second target point, and determine the neighboring point corresponding to the second target point according to the relationship between the distance between the second target point and other points in the zero-level point cloud layer and a preset distance; construct the zero-level graph structure based on the second target point and the second neighboring point.

[0020] In some embodiments of the present disclosure, based on the aforementioned scheme, the decoding module is configured as follows: performing maximum pooling processing on the M-level edge sub-features corresponding to the edges connecting each point in the M-level point cloud layer to obtain the regional features corresponding to each point in the M-level point cloud layer; connecting the point features corresponding to each point in the M-level point cloud layer with the regional features, and performing interpolation processing on the connected features to obtain interpolation point features; connecting the interpolation point features with the M+1-level intermediate point cloud coding features, and performing feature extraction on the connected features to obtain the M+1-level point features.

[0021] In some embodiments of the present disclosure, based on the aforementioned scheme, the classification module is configured as follows: performing maximum pooling processing on the edge sub-features in the edge features corresponding to each point in the original point cloud to obtain the regional features corresponding to each point in the original point cloud; connecting the point features with the regional features corresponding to each point in the original point cloud, and performing classification prediction on the connected features through a multi-layer perceptron to obtain classification information corresponding to each point in the original point cloud.

[0022] According to one aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: one or more processors and a storage device, wherein the storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors execute the point cloud semantic segmentation method as described in the above embodiments.

[0023] In the technical solutions provided by some embodiments of the present disclosure, the point cloud data corresponding to the original point cloud is encoded through the encoding network model to obtain the point cloud encoding features; then the edge features corresponding to the original point cloud are obtained according to the point cloud encoding features through the edge module branch in the decoding network model, and the point features corresponding to the original point cloud are obtained according to the point cloud encoding features and the edge features through the point module branch in the decoding network model; finally, classification prediction is performed based on the point features and the edge features to obtain the classification information corresponding to each point in the original point cloud. The technical solution of the present disclosure can increase the scope of information transmission and better utilize local environmental information through the mutual communication and coordination of points and edges, thereby improving the accuracy and efficiency of point cloud semantic segmentation.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0026] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of the embodiments of the present disclosure can be applied;

[0027] Figure 2 The flowchart of the point cloud semantic segmentation method according to one embodiment of the present disclosure is schematically shown;

[0028] Figure 3The schematic diagram shows a process flow of decoding an original point cloud by a decoding network model according to an embodiment of the present disclosure;

[0029] Figure 4 A schematic diagram of a decoding process of a decoding network model according to an embodiment of the present disclosure is shown;

[0030] Figure 5 The schematic diagram of the structure of the edge module according to one embodiment of the present disclosure is shown;

[0031] Figure 6 The schematic diagram shows a flow chart of obtaining M-level edge features through an edge module according to an embodiment of the present disclosure;

[0032] Figure 7 A flowchart for constructing a graph structure according to an embodiment of the present disclosure is schematically shown;

[0033] Figure 8 A flowchart for constructing an M-level graph structure according to an embodiment of the present disclosure is schematically shown;

[0034] Figures 9A-9C A flowchart of filtering edge sets in an M-level graph structure according to an embodiment of the present disclosure is schematically shown;

[0035] Fig.10 A schematic diagram of a process for obtaining up-sampled edge features according to an embodiment of the present disclosure is shown;

[0036] Fig.11 A schematic diagram of a structure for detecting an intersection between an edge and an edge corresponding to an M-1 level edge feature according to an embodiment of the present disclosure is shown;

[0037] Fig.12 The schematic diagram of the structure of the edge coding layer according to an embodiment of the present disclosure is shown;

[0038] Fig.13 The following schematically shows a flow chart of obtaining M-level edge features according to an embodiment of the present disclosure;

[0039] Fig.14 The schematic diagram of the structure of the point module according to one embodiment of the present disclosure is shown;

[0040] Fig.15 The following schematically shows a flow chart of obtaining point features through a point module according to an embodiment of the present disclosure;

[0041] Fig.16 A schematic diagram of the structure of a point cloud semantic segmentation model according to an embodiment of the present disclosure is shown;

[0042] Fig.17 A schematic diagram of the semantic segmentation effect of a large indoor point cloud according to an embodiment of the present disclosure is shown;

[0043] Fig.18 A schematic diagram of the semantic segmentation effect of a large indoor point cloud according to an embodiment of the present disclosure is shown;

[0044] Fig.19 A block diagram of a point cloud semantic segmentation device according to an embodiment of the present disclosure is schematically shown;

[0045] Fig. 20 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0046] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more comprehensive and complete and will fully convey the concept of the example embodiments to those skilled in the art.

[0047] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0048] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0049] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0050] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of the embodiments of the present disclosure can be applied.

[0051] like Figure 1As shown, the system architecture 100 may include a terminal device 101, a network 102, and a server 103. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as a wired communication link, a wireless communication link, and the like.

[0052] It should be understood that Figure 1 The number of terminal devices, networks and servers in the example is only illustrative. Any number of terminal devices, networks and servers may be provided according to actual needs. For example, the server 103 may be a server cluster composed of multiple servers. The terminal device 101 may be a photographing device such as a photographic scanner or a laser radar, through which an image may be acquired, and then a point cloud corresponding to the image may be determined according to the image.

[0053] In one embodiment of the present disclosure, after the terminal device 101 obtains the original point cloud, it can send the point cloud data corresponding to the original point cloud to the server 103 through the network 102. When the server 103 obtains the point cloud data corresponding to the original point cloud, it can perform multi-level encoding on the point cloud data through the encoding network model, that is, down-sample the point cloud data multiple times to gradually reduce the number of points in the point cloud. In the process of downsampling, intermediate point cloud coding features and zero-level point cloud coding features can be obtained, wherein the zero-level point cloud coding features are the coding features corresponding to the last level point cloud layer output by the encoding network model. At the same time, the zero-level point cloud coding features are also the input features input into the decoding network model. Then, the edge features corresponding to the original point cloud are obtained according to the point cloud coding features through the edge module branch in the decoding network model, and the point features corresponding to the original point cloud are obtained according to the point cloud coding features and edge features through the point module branch in the decoding network model. Specifically, the decoding network model includes multiple decoding network sub-models, and each decoding network sub-model includes a point module and an edge module, wherein the point module can determine the point features corresponding to the current point cloud layer according to the point features and edge features corresponding to the previous point cloud layer and the intermediate point cloud coding features corresponding to the current point cloud layer, and the edge module can determine the edge features corresponding to the current point cloud layer according to the point features corresponding to the current point cloud layer and the edge features corresponding to the previous point cloud layer, and can also determine the zero-level edge features according to the zero-level point cloud features; finally, classification prediction is performed according to the point features and edge features corresponding to the original point cloud to obtain classification information corresponding to each point in the original point cloud. The technical solution of the disclosed embodiment can combine point features and edge features, increase the scope of information transmission, and better utilize local environmental information, thereby improving the accuracy and efficiency of point cloud semantic segmentation.

[0054] It should be noted that the point cloud semantic segmentation method provided in the embodiment of the present disclosure is generally executed by a server, and accordingly, the point cloud semantic segmentation device is generally set in the server. However, in other embodiments of the present disclosure, the point cloud semantic segmentation method provided in the embodiment of the present disclosure can also be executed by a terminal device, and the point cloud semantic segmentation method provided in the embodiment of the present disclosure can segment three-dimensional scene data with rich environmental structure information and obtain good segmentation effect.

[0055] In the related technologies in this field, the existing technical solutions for scene segmentation based on deep learning mainly include voxel-based solutions and point cloud-based solutions. The voxel-based solution divides the three-dimensional space into regularly arranged small squares (voxels) and uses convolutional neural networks widely used for two-dimensional segmentation for feature learning. The point cloud-based solution only considers the occupied area in the three-dimensional space, directly uses the three-dimensional coordinates of the points as the input of the network, and extracts features through a multi-layer perceptron (MLP).

[0056] Accordingly, the related technologies have drawbacks. For voxel-based solutions, due to the sparse spatial distribution of three-dimensional objects, unlike two-dimensional images where objects are densely distributed, directly voxelizing the three-dimensional space will result in problems such as low utilization of storage space. For point cloud-based solutions, due to the irregularity and disorder of point clouds, the convolution layer used to extract local areas cannot be applied to point clouds. Although in order to solve this problem, researchers have proposed some solutions, such as voxel-based solutions that use octree representation, sparse convolution and other methods to save space; extract local information by applying the maximum pooling layer (MaxPooling) in local areas; utilize contextual information by mapping local areas, etc. However, the effects of these solutions are not very ideal, and the accuracy and efficiency of segmentation are low.

[0057] At the same time, although the voxel-based solution can directly use deep convolutional neural networks for feature learning, the accuracy of three-dimensional objects will be lost during the voxelization process. One problem that these solutions always face is the trade-off between object accuracy and network training efficiency. When the size of the voxel is reduced to ensure the accuracy of the object, the memory required by the network and the time required for training will increase significantly. Therefore, the present disclosure chooses to perform three-dimensional scene segmentation based on point clouds. The point cloud-based solution needs to design a new method different from the convolution layer to extract local information. Directly applying the maximum pooling layer (MaxPooling) in the local area to obtain local information lacks consideration of the relationship between points, and directly taking the maximum value may ignore some important information. Other local mapping-based solutions consider the information transfer between points, but these solutions are separate maps for each level of point cloud, or directly fix the point cloud size for mapping, without considering the transmission of edge information between different levels.

[0058] In view of the problems existing in the related art, the embodiments of the present disclosure provide a point cloud semantic segmentation method, which is implemented based on machine learning. Machine learning is a type of artificial intelligence. Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0059] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0060] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphics processing so that the computer processes the images into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0061] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0062] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0063] The solution provided by the embodiments of the present disclosure relates to an image semantic understanding technology of artificial intelligence, which is specifically described by the following embodiments:

[0064] The embodiment of the present disclosure first proposes a point cloud semantic segmentation method, which can be applied to three-dimensional application scenarios such as augmented reality, virtual reality, and household robots. The implementation details of the technical solution of the embodiment of the present disclosure are described in detail below:

[0065] Figure 2 The flowchart of the point cloud semantic segmentation method according to an embodiment of the present disclosure is schematically shown. The point cloud semantic segmentation method can be executed by a server, which can be Figure 1 The server 103 shown in FIG. Figure 2 As shown, the point cloud semantic segmentation method at least includes steps S210 to S230, which are described in detail as follows:

[0066] In step S210, the original point cloud is obtained, and the point cloud data corresponding to the original point cloud is encoded through the encoding network model to obtain point cloud coding features.

[0067] In one embodiment of the present disclosure, the target scene can be photographed by the terminal device 101 to obtain a set of point data constituting the target scene, that is, an original point cloud; it can also be an original point cloud corresponding to the target scene downloaded from the network by the terminal device 101, or it can be an original point cloud corresponding to the target scene locally stored by the terminal device 101, etc., which is not specifically limited in the present embodiment. After obtaining the original point cloud sent by the terminal device 101, the original point cloud can be input into the point cloud semantic segmentation model to extract features of the points in the original point cloud and the edges connected to the points to obtain point features and edge features. Furthermore, classification prediction can be performed based on the point features and edge features to obtain classification information corresponding to each point.

[0068] In one embodiment of the present disclosure, the point cloud semantic segmentation model includes an encoding network model and a decoding network model, wherein the encoding network model is used to encode the point cloud data corresponding to the original point cloud, that is, to downsample the original point cloud level by level to obtain the point cloud coding features corresponding to each level of point cloud layer. Specifically, Figure 3 FIG. 4 shows a schematic diagram of the process of encoding the point cloud data corresponding to the original point cloud by the encoding network model, such as Figure 3 As shown in FIG. 1 , the original point cloud is encoded twice, and the process of each encoding process is the same, including: a sampling stage, a grouping stage, and a feature extraction stage. In the sampling stage, first, any point is selected as the target point from the original point cloud or the feature points obtained in the previous encoding process, and then the point farthest from the target point is selected from other points, and the farthest point and the target point form a point set, and then the point farthest from the point set is iteratively selected, and the point set is updated until the number of points in the point set reaches a preset number; repeat the above steps until multiple point sets are obtained, that is, R points are selected from the R points of the original point cloud. 1 Center points. Grouping stage: Take each center point as a circle point, and group the circle point and all points within the preset radius from the circle point into a sub-point set. Feature extraction stage: The R obtained in the previous layer 1 ×k×(C+D) lattice, where R 1 is the number of center points, k is the number of points in each sub-point set, C is the feature dimension of each point, and D is the coordinate dimension of each point; for each group of points k×(C+D), the center point is first normalized to obtain the local coordinates, and then feature extraction is performed to obtain 1×(C 1 +D) feature vector; by extracting features from each group of points, we get R 1 (C 1 The feature points obtained from each encoding process are used as the input feature points for the next encoding process, and then the above process is performed to obtain the required number of feature points. The dimension shown in the figure is (R 2 , C2 +D), these feature points constitute a point cloud layer, that is, by encoding the point cloud data corresponding to the original point cloud multiple times in sequence, multi-level point cloud layers can be obtained. For example, encoding the original point cloud can obtain a first-level point cloud layer, encoding the first-level point cloud layer can obtain a second-level point cloud layer, and so on, until the last-level point cloud layer is obtained. Since the last-level point cloud layer is the input feature of the decoding network model, for convenience, it can be defined as a zero-level point cloud layer. Furthermore, the point cloud coding features obtained by extracting features from the zero-level point cloud layer can be marked as zero-level point cloud coding features, and the point cloud coding features obtained by extracting features from the non-zero-level point cloud layer can be marked as intermediate point cloud coding features. Figure 3 The dimensions shown in (R 1 , C 1 +D) is the intermediate point cloud encoding feature corresponding to the first-level point cloud layer formed by downsampling the original point cloud, and its dimension is (R 2 , C 2 +D) is the zero-level point cloud coding feature corresponding to the secondary point cloud layer formed by downsampling the primary point cloud layer. It is worth noting that the coding network model can perform multi-level coding processing on the original point cloud to form zero-level point cloud coding features and multi-level intermediate point cloud coding features, wherein each level of intermediate point cloud coding features corresponds to different levels of point cloud layers containing different numbers of points. For example, the original point cloud contains 1000 points, and the coding network model encodes the point cloud data corresponding to the original point cloud 4 times, then it can generate a third-level intermediate point cloud coding feature corresponding to a third-level point cloud layer containing 500 points, a second-level intermediate point cloud coding feature corresponding to a corresponding second-level point cloud layer containing 250 points, a first-level intermediate point cloud coding feature corresponding to a first-level point cloud layer containing 100 points, and a zero-level point cloud coding feature corresponding to a zero-level point cloud layer containing 50 points. The decoding network model is used to upsample the zero-level point cloud encoding features step by step to obtain point features and edge features corresponding to each level of point cloud layer, and finally obtain point features and edge features corresponding to the original point cloud.

[0069] In step S220, the point cloud coding features are input into the decoding network model, and the edge features corresponding to the original point cloud are obtained according to the point cloud coding features through the edge module branch in the decoding network model, and the point features corresponding to the original point cloud are obtained according to the point cloud coding features and the edge features through the point module branch in the decoding network model.

[0070] In one embodiment of the present disclosure, after obtaining the intermediate point cloud coded features and the zero-level point cloud coded features, they can be input as input features into the decoding network model for decoding to obtain point features and edge features corresponding to the original point cloud. The intermediate point cloud coded features at each level are connected to the point features at the corresponding level in the decoding process by skipping. For example, if you want to obtain M-level point features, you can input the M-level intermediate point cloud coded features into the point module that generates M-level point features, so that the point module generates M-level point features based on the M-level intermediate point cloud coded features.

[0071] In one embodiment of the present disclosure, the decoding network model includes a multi-level decoding network sub-model, and each level of the decoding network sub-model includes an edge module and a point module. Further, the edge modules in each level of the decoding network sub-model constitute an edge module branch, and the point modules constitute a point module branch, wherein the edge module branch can obtain edge features corresponding to the original point cloud according to the point cloud coding features, and the point module branch can obtain point features corresponding to the original point cloud according to the point cloud coding features and the edge features. In the embodiment of the present disclosure, the hierarchical edge module branch and the point module branch cooperate with each other to perform point cloud semantic segmentation, and the graph is initialized at the level where the points are the sparsest, that is, the last level of the point cloud layer, and the graph is gradually expanded with the upsampling decoding process of the points, so that the edge features are transmitted between different levels through the upsampling of the edges, and the point features of different levels are received to improve the edge features. At the same time, the edge features of different layers are also used to provide rich contextual information for the learning of point features, so that the edge module branches and the point module branches constitute mutual transmission and communication of information.

[0072] When using decoding network sub-models at each level for hierarchical decoding, the specific process is as follows: the zero-level edge feature is obtained according to the zero-level point cloud coding feature through the edge module in the first-level decoding network sub-model, and the first-level point feature is obtained according to the zero-level point cloud coding feature, the zero-level edge feature and the first-level intermediate point cloud coding feature through the point module therein; the M-level edge feature is obtained according to the M-level point feature and the M-1-level edge feature through the edge module in the M-level decoding network sub-model, and the M+1-level point feature is obtained according to the M+1-level intermediate point cloud coding feature, the M-level point feature and the M-level edge feature through the point module therein, where M is a positive integer less than the total number of decoding network sub-models, so that the decoding network model can obtain point features of the same size as the original point cloud after multiple upsampling.

[0073] Figure 4 A schematic diagram of the process of decoding by a decoding network model is shown, such as Figure 4As shown, the decoding network model 400 includes a first-level decoding network sub-model, a second-level decoding network sub-model, ..., an M+1-level decoding network sub-model, ..., an N-level decoding network sub-model, and the zero-level point cloud coding features and the first-level intermediate point cloud data are input into the first-level decoding network sub-model as features, and the first-level decoding network sub-model is used to decode according to the zero-level point cloud coding features and the first-level intermediate point cloud coding features to obtain the first-level point features and the zero-level edge features; then the first-level point features and the second-level intermediate point cloud data are input into the second-level decoding network sub-model as features, and the second ... The coding network submodel decodes according to the first-level point features and the second-level intermediate point cloud coding features to obtain the second-level point features and the first-level edge features; ...; the M-level point features and the M+1-level intermediate point cloud data are input into the M+1-level decoding network submodel, and the M+1-level decoding network submodel decodes according to the M-level point features and the M+1-level intermediate point cloud coding features to obtain the M+1-level point features and the M-level edge features, and repeats the above steps until the N-level point features and the N-level edge features are obtained, wherein the number of points corresponding to the N-level point features is the same as the number of points in the original point cloud.

[0074] In one embodiment of the present disclosure, when obtaining M-level edge features based on M-level point features and M-1-level edge features through the edge module, feature extraction can be performed based on the position information of each point in the M-level point features, the M-1-level graph structure and the M-1-level edge features to obtain M-level edge features.

[0075] Figure 5 The schematic diagram of the structure of the edge module is shown in FIG. Figure 5 As shown, the edge module 500 includes a graph construction layer 501, an edge upsampling layer 502 and an edge coding layer 503, wherein the graph construction layer 501 can obtain an M-level graph structure according to M-level point features and an M-1-level graph structure; the edge upsampling layer 502 can obtain upsampled edge features according to M-level graph structures and M-1-level edge features; and the edge coding layer 503 can obtain M-level edge features according to M-level point features, M-level graph structures and upsampled edge features. Figure 5 The edge module shown can receive point features at different levels to improve the edge features and increase the accuracy of the edge features.

[0076] In one embodiment of the present disclosure, Figure 6 FIG. 4 shows a flow chart of obtaining M-level edge features through edge modules, as shown in FIG. Figure 6 As shown, the process of obtaining M-level edge features includes at least steps S601-S603, specifically:

[0077] In step S601, an M-level graph structure is constructed based on the position information of each point in the M-level point features and the M-1-level graph structure.

[0078] In one embodiment of the present disclosure, Figure 7 The flowchart of building the graph structure is shown as Figure 7 As shown, after obtaining the zero-level point cloud coding features, the position information of the points (N 0 ×3) Initialize the graph to form a zero-level graph structure G 0 ; Then, after obtaining the first-level point features according to the zero-level point cloud coding features through the point module, the graph construction module can be used to obtain the first-level point features according to the position information of the midpoints and G 0 Generate a first-level graph structure G 1 ; ...; After obtaining the M+1 level point features according to the M level point features through the point module, the graph construction module is used to obtain the M+1 level point features according to the position information of the midpoints and G M Generate M+1 level graph structure G M+1 , repeat the above steps until the graph structure G generated according to the N-level point features is obtained.

[0079] In one embodiment of the present disclosure, for a given point cloud P = {p1, p2, ..., pN}, a directed graph G = (V, E) can be constructed, where the point cloud can correspond to a three-dimensional space, and the points in the point cloud have three coordinate values, V = P is a vertex set, and E is an edge set connecting each vertex with its surrounding area points. In the embodiment of the present disclosure, after obtaining the zero-level point cloud coding feature, the graph can be initialized according to the position information of the point in the zero-level point cloud coding feature. Specifically, all points V in the zero-level point cloud layer corresponding to the zero-level point cloud coding feature are initialized. 0 As the point set of the zero-level point cloud layer; then determine any point in the zero-level point cloud layer as the target point, and determine the neighboring points corresponding to the target point according to the relationship between the distance between the target point and other points in the zero-level point cloud layer and the preset distance. When the distance between the target point and other points is less than or equal to the preset distance, other points can be used as the neighboring points corresponding to the target point; finally, construct a zero-level graph structure based on the target point and the neighboring points. Specifically, the target point can be connected to the neighboring points to form edges, E 0 is the edge set consisting of the edges connecting each point in the zero-level point cloud layer with its nearest neighbor point, and then according to the point set V 0 and edge set E 0 The zero-level graph structure G can be determined 0 =(V 0 , E 0 ).

[0080] In one embodiment of the present disclosure, the construction of the M-level graph structure can be constructed based on the position information of each point in the M-level point features and the M-1-level graph structure. Figure 8 The construction flow chart of the M-level graph structure is shown as Figure 8As shown, in step S801, any point in the M-level point cloud layer is determined as the first target point, and the first neighbor point corresponding to the first target point in the M-level point cloud layer is obtained, and the distance between the first neighbor point and the first target point does not exceed the preset distance threshold; in step S802, each first target point in the M-level point cloud layer is connected to the first neighbor point corresponding to each first target point to obtain an M-level initial graph structure; in step S803, any edge in the M-level initial graph structure is used as the first target edge, and the first target edge pair is obtained. corresponding two endpoints; in step S804, the neighboring points corresponding to each endpoint are determined in the M-1 level point cloud layer, a target neighboring point is determined from the neighboring points corresponding to each endpoint, and the target neighboring points are connected to form a detection edge, and it is determined whether the detection edge is an edge in the M-1 level graph structure; in step S805, when the detection edge is not an edge in the M-1 level graph structure, the target edge is deleted from the M level initial graph structure; in step S806, steps S803-S805 are repeated to obtain the M level graph structure.

[0081] As a further explanation of steps S804-S805, Figures 9A-9C The screening flow chart of edge sets in M-level graph structure is shown. For the M-level initial graph structure G M (0) =(V M , E M (0) ) ij , the corresponding two endpoints are i and j, and the neighboring points whose distances from the two endpoints do not exceed the preset distance threshold are determined from the M-1 level point cloud layer, such as Fig.9A As shown, the neighboring points of endpoint i are A, B, and C, and the neighboring points of endpoint j are D, E, and F; then the edge e is formed based on the neighboring points of the two endpoints. ij The corresponding neighbor edge set is Fig. 9B As shown; then the neighbor edge set and Fig. 9C Take the intersection of the edge sets in the M-1 level graph structure shown. If the intersection is not empty, it means that edge e ij The two endpoints of are close to each other and are connected in the M-1 level point cloud layer, so the edge e ij Save in the final edge set E M and M-level graph structure G M If the intersection is empty, it means that edge e ij The two endpoints of are so far apart that there is no connection in the M-1 level point cloud layer, then e ij From the initial edge set E M (0) Repeat the above operation for each edge in the M-level initial graph structure until the final edge set E is obtained. M and M-level graph structure G M.

[0082] from Figure 7 It can be seen that the graph structure construction process in the embodiment of the present disclosure is different from the solution of building a graph for each layer separately, but rather constructs the graph structure in a hierarchical manner, thereby increasing the scope of information transmission.

[0083] In step S602, edges are upsampled according to the M-1 level edge features and the M level graph structure to obtain upsampled edge features.

[0084] In one embodiment of the present disclosure, as a further explanation of step S602, Fig.10 A schematic diagram of the process of obtaining up-sampled edge features is shown, such as Fig.10 As shown, in step S1001, any edge in the M-level graph structure is determined as an edge to be processed; in step S1002, the neighboring edge corresponding to the edge to be processed is determined in the edges corresponding to the M-1-level edge features according to the two endpoints corresponding to the edge to be processed; in step S1003, the intersection of the neighboring edge and the edge corresponding to the M-1-level edge feature is taken, and the edge features corresponding to the edges in the intersection are interpolated according to the distances corresponding to the endpoints of the edge to be processed to obtain the upsampled edge features. The specific implementation method of taking the intersection in steps S1002 and S1003 is the same as Figures 9A-9C The process shown is the same as that shown in FIG. 2 . Further, when it is determined that there is an intersection, that is, when there is a neighboring edge corresponding to the edge in the M-1 level graph structure among the edges corresponding to the M-1 level edge features, as shown in FIG. Fig.11 As shown in the dark directed edges AD, BD, BF and CF in the figure, the corresponding edge features can be interpolated according to the distance between the two endpoints i and j to obtain the upsampled edge features, as shown in Fig.11 As shown in FIG. 1 , the upsampled edge feature is the directed edge ij, which is obtained by upsampling the edge features corresponding to the directed edges AD, BD, BF, and CF. The upsampling of the edge enables the transmission of edge features between different levels, so that the edge has richer high and low level information.

[0085] In step S603, the M-level graph structure, the M-level point features and the upsampled edge features are encoded to obtain the M-level edge features.

[0086] In one embodiment of the present disclosure, the edge coding layer 503 performs encoding according to the M-level graph structure, the M-level point features and the upsampled edge features, so as to obtain the M-level edge features. Fig.12 The schematic diagram of the structure of the edge coding layer is shown in FIG. Fig.12 As shown, the edge coding layer 503 includes an index layer 1201, a point feature connection layer 1202, a first multi-layer perceptron layer 1203, a point-edge feature connection layer 1204, and a second multi-layer perceptron layer 1205. The edge coding layer can process the upsampled edge feature corresponding to any edge in the M-level graph structure to obtain the M-level edge feature.

[0087] based on Fig.12 The schematic diagram of the structure of the edge coding layer is shown in FIG. Fig.13 FIG. 4 shows a schematic diagram of the process of obtaining M-level edge features, such as Fig.13 As shown, the process of obtaining M-level edge features includes at least steps S1301-S1304, specifically:

[0088] In step S1301, any edge in the M-level graph structure is used as a second target edge, and two endpoints corresponding to the second target edge are obtained.

[0089] In one embodiment of the present disclosure, after obtaining the M-level graph structure according to the M-level point features and the M-1-level graph structure, any edge e can be selected from the M-level graph structure through the index layer 1201. ij as the second target edge, and determine the second target edge e ij The corresponding two endpoints i and j.

[0090] In step S1302, point features corresponding to two endpoints are obtained from the M-level point features, the point features corresponding to the two endpoints are connected, and feature extraction is performed on the connected point features to obtain connection point features.

[0091] In one embodiment of the present disclosure, the index layer 1201 can be used to obtain the point features of the endpoint i from the M-level point features according to the two endpoints i and j corresponding to the second target edge. and the point features of endpoint j At the same time, obtain the upsampled edge features corresponding to the second target edge eij from the upsampled edge features Then, the point feature of endpoint i can be connected through the point feature connection layer 1202. and the point features of endpoint j Connect to form the connected feature f edge , the corresponding channel number is given by becomes Finally, the first multi-layer perceptron layer 1203 is used to connect the features f edge Perform feature extraction to obtain connection point features.

[0092] In step S1303, the connection point feature and the upsampled edge feature corresponding to the second target edge are concatenated, and feature extraction is performed on the concatenated features to obtain M-level edge sub-features corresponding to the second target edge.

[0093] In one embodiment of the present disclosure, after obtaining the connection point features, they can be combined with the upsampled edge features. The concatenated features are connected through the point-edge feature connection layer 1204, and then the concatenated features can be extracted through the second multi-layer perceptron layer 1205 to obtain the features related to the second target edge eij The corresponding M-level edge features.

[0094] In step S1304, steps S1301-S1303 are repeated until the M-level edge sub-features corresponding to each edge in the M-level graph structure are obtained, and the M-level edge features are determined according to the M-level edge sub-features corresponding to each edge.

[0095] In one embodiment of the present disclosure, steps S1301-S1303 are repeated to obtain M-level edge sub-features corresponding to all edges in the M-level graph structure. Then determine the M-level edge feature based on all the M-level edge sub-features The number of edges corresponding to this M-level edge feature is |E M |, the number of channels is K M .

[0096] It should be noted that for the generation of zero-level edge features, it is only necessary to Fig.12 The edge coding layer shown can perform feature extraction on the zero-level point cloud coding features. Specifically, the zero-level point cloud coding features can be input into the edge coding layer. The edge coding layer first takes any point in the zero-level point cloud layer corresponding to the zero-level point cloud coding features as the target point, and then searches for neighboring points in the zero-level point cloud layer whose distance from the target point does not exceed a preset distance threshold, and simultaneously obtains the point features corresponding to the target point and the neighboring points, and connects them; then, the connected point features are extracted through a multi-layer perceptron to obtain zero-level edge features.

[0097] Fig.14 The schematic diagram of the structure of the point module is shown in FIG. Fig.14 As shown, the point module 1400 includes a maximum pooling layer 1401, a connection layer 1402, an interpolation layer 1403, a connection layer 1404 and a multi-layer perceptron output layer 1405, wherein (|E M |×K M ) is an M-level edge feature, which can be obtained by H EM Simplified representation, where |E M | is the number of edges in the M-level edge feature, K M is the number of channels of M-level edge features; (N M ×C M , N M ×3) is an M-level point feature, which can be obtained by F VL Simplified representation, where N M is the number of points in the M-level feature, C M is the number of channels of M-level point features; (N M+1 ×C' M+1 ) is the M+1 level intermediate point cloud encoding feature, where N M+1 is the number of points in the M+1 level intermediate point cloud encoding feature, C' M+1is the number of channels of the M+1 level intermediate point cloud encoding features; (N M ×(C M +K M )) is the connected feature output by the connection layer 1402; (N M+1 ×(C M +K M )) is the interpolation point feature; (N M+1 ×C M+1 , N M+1 ×3) is the M+1 level point feature, which can be used by F V(M+1) Simplified representation, where N M+1 is the number of points in the M+1 level feature, C M+1 is the number of channels of M+1 level point features. Fig.14 The structure of the point module shown, Fig.15 A schematic diagram of the process of obtaining point features through the point module is shown, such as Fig.15 As shown, the process of obtaining point features includes at least steps S1501-S1503, specifically:

[0098] In step S1501, maximum pooling processing is performed on the M-level edge sub-features corresponding to the edges connecting each point in the M-level point cloud layer to obtain the regional features corresponding to each point in the M-level point cloud layer.

[0099] In one embodiment of the present disclosure, the maximum pooling layer 1401 receives the M-level edge features H EM The edge sub-feature H corresponding to the edge connected to each point EM ( pi) The maximum pooling process is performed to generate regional features corresponding to each point in the M-level point cloud layer by aggregating features. After the regional features corresponding to all points in the M-level point cloud layer are obtained, the regional features are output to the connection layer 1402.

[0100] In step S1502, the point features and regional features corresponding to each point in the M-level point cloud layer are connected, and the connected features are interpolated to obtain interpolated point features.

[0101] In one embodiment of the present disclosure, after obtaining the regional features corresponding to all points in the M-level point cloud layer, the point features and regional features corresponding to all points can be connected to obtain the connected features. Then, the connected features can be input to the interpolation layer 1403, and interpolated by the interpolation layer 1403 to obtain the interpolated point features.

[0102] In step S1503, the interpolation point features and the M+1-level intermediate point cloud coding features are connected, and feature extraction is performed on the connected features to obtain the M+1-level point features.

[0103] In one embodiment of the present disclosure, after obtaining the interpolation point features, the interpolation point features can be connected to the M+1 level intermediate point cloud coding features through the connection layer 1404, and then the connected features are extracted through the multi-layer perceptron output layer 1405 to obtain the M+1 level point features. It is worth noting that the connection of features in the embodiment of the present disclosure is all serial, so that the information of the points and the information of the edges can be retained to the greatest extent.

[0104] In the disclosed embodiment, the point module performs feature extraction based on M-level edge features, M-level point features and M+1-level intermediate point cloud coding features to obtain M+1-level point features, and can transmit contextual environment information for the points through edge features, thereby improving the accuracy of the obtained point features.

[0105] In step S230, classification prediction is performed based on point features and edge features to obtain classification information corresponding to each point in the original point cloud.

[0106] In one embodiment of the present disclosure, after obtaining point features and edge features corresponding to the point cloud layer having the same number of points as the original point cloud by upsampling the zero-level point cloud coding features multiple times, classification prediction can be performed based on the point features and edge features to obtain classification information of the points. Specifically, the edge sub-features corresponding to each point in the original point cloud in the edge features can be processed by maximum pooling to obtain the regional features corresponding to each point in the original point cloud; then the point features are connected with the regional features corresponding to each point in the original point cloud (for example, in series, etc.), and the connected features are classified and predicted by a classifier to obtain the classification information corresponding to each point in the original point cloud, wherein the classifier can specifically be a multi-layer perceptron, and the point features corresponding to the original point cloud are extracted by the multi-layer perceptron to obtain the semantic label of each point, which is the classification information of each point, used to indicate the type of object to which each point belongs, such as which points belong to roads, which points belong to buildings, which points belong to road signs, and so on.

[0107] Taking a household sweeping robot as an example, the scanning device in the sweeping robot can scan the spatial layout and obtain a point cloud map of the spatial layout. The scanning device can then input the obtained point cloud information into a data processing device, and the data processing device processes the received point cloud information to obtain classification information for each point in the point cloud, such as which point in the spatial layout is a table leg, which point is a carpet, and so on. Fig.16 The structural diagram of the point cloud semantic segmentation model is shown in FIG. Fig.16 As shown, the point cloud data N×C′ corresponding to the spatial layout is first multi-level encoded by the encoding network model 1601 to obtain the multi-level intermediate point cloud encoding features N i ×C' i (i is a positive integer) and zero-level point cloud encoding features (N 0×C 0 , N 0 ×3), where N is the number of points in the original point cloud, C' is the number of channels of the points in the original point cloud, and N i is the number of points in the i-level point cloud layer, C' i N is the number of channels of the point features output by the encoding network model. 0 is the number of points in the zero-level point cloud layer, C 0 N is the number of channels of the zero-level point cloud encoding features output by the encoding network model. 0 ×3 is the position information of the midpoint of the zero-level point cloud layer; then the zero-level edge feature |E is obtained according to the zero-level point cloud coding feature by decoding the edge module in the network model 1602. 0 |×K 0 , where |E 0 | is the number of edges in the zero-order edge feature, K 0 is the number of channels of the zero-level edge feature; the point module obtains the first-level point feature (N) according to the zero-level point cloud coding feature, the zero-level edge feature and the first-level intermediate point cloud coding feature. 1 ×C 1 , N 1 ×3). Since there are multiple repeated decoding network sub-models in the decoding network model 1602, these decoding network sub-models can obtain i-level edge features based on i-level point features, and obtain i+1-level point features based on i-level point features, i-level edge features and i+1-level intermediate point cloud coding features. After upsampling, the decoding network model 1602 can output point features (N×C, N×3) and edge features |E|×K. Then, the edge features are processed by maximum pooling to obtain the regional features corresponding to each point in the point cloud of the spatial layout, and the point features are connected in series with the regional features corresponding to each point to obtain the series features; finally, a multi-layer perceptron is used to perform classification prediction based on the series features to output the classification information (N×class) corresponding to each point in the spatial layout.

[0108] In one embodiment of the present disclosure, when generating the initial graph structure corresponding to each level of point cloud layer, not only the neighboring points of each point in each level of point cloud layer in the point cloud layer can be considered, but also edges can be randomly built according to balls of different radii. For example, the points in the point cloud layer at each level are used as the center of the circle, and multiple balls are formed with different radii, and several points are selected from each ball, and then the edges in the initial graph structure are formed according to the center point of the circle and the selected several points.

[0109] In one embodiment of the present disclosure, before using the point cloud semantic segmentation model to process the point cloud to obtain the semantic information of the point, it is necessary to train the point cloud semantic segmentation model based on the point cloud samples and the corresponding semantic samples. During the training process, since there is a loss in the point module branch corresponding to the point module and there is also a loss in the edge module branch corresponding to the edge module, the loss of the point module branch can be calculated according to the type of object to which the point belongs, and the loss of the edge module branch can be calculated according to the semantic consistency of the two endpoints of the edge, where the semantic consistency refers to whether the two endpoints of the edge correspond to the same type of object. The loss function used to calculate the loss can be a cross entropy loss function, etc., and return to Fig.16 , the edge features of the final output are extracted through the multi-layer perceptron to obtain the classification of the edge features |E|. According to the classification, the loss of the edge module branch can be determined. Similarly, the loss of the point module branch can be determined according to the classification information corresponding to the point. After obtaining the loss of the point module branch and the loss of the edge module branch, the two losses can be weighted and summed to determine the loss corresponding to the point cloud semantic segmentation model. By continuously adjusting the parameters of the model to minimize the loss function of the model, the model training is completed.

[0110] The point cloud semantic segmentation in the disclosed embodiment processes the point cloud data through an encoding network model and a decoding network model including a point module and an edge module to obtain the classification information corresponding to each point in the point cloud, and in the decoding process, through a hierarchical mapping process and upsampling of edges, it is possible to transfer edge features between different levels, so that the edge has richer high and low level information, and also increases the transmission range of point information. At the same time, in the process of guiding the learning of edge features through the semantic consistency of the two endpoints of the edge, the discrimination of point features belonging to different categories is also enhanced. Fig.17 and Fig.18 The semantic segmentation effect of a large indoor point cloud is shown in Figure 2. Fig.17 and 18 As shown, the input image is an image obtained by photographing the indoor layout through a photographing device, the ground truth is the annotation data determined based on the input image, and the image corresponding to this solution is the effect image generated after the indoor point cloud corresponding to the input image is segmented according to the point cloud semantic segmentation method in this disclosure. Fig.17 and Fig.18 It can be seen that the effect is not much different from the ground truth, that is to say, the point cloud semantic segmentation method disclosed in the present invention can improve the accuracy of point cloud segmentation and achieve a good segmentation effect.

[0111] The following describes an apparatus embodiment of the present disclosure, which can be used to execute the point cloud semantic segmentation method in the above-mentioned embodiment of the present disclosure. For details not disclosed in the apparatus embodiment of the present disclosure, please refer to the above-mentioned embodiment of the point cloud semantic segmentation method of the present disclosure.

[0112] Fig.19 A block diagram of a point cloud semantic segmentation device according to an embodiment of the present disclosure is schematically shown.

[0113] Reference Fig.19 As shown, a point cloud semantic segmentation device 1900 according to an embodiment of the present disclosure includes: an encoding module 1901, a decoding module 1902 and a classification module 1903.

[0114] Among them, the encoding module 1901 is used to obtain the original point cloud, and encode the point cloud data corresponding to the original point cloud through the encoding network model to obtain the point cloud coding features; the decoding module 1902 is used to input the point cloud coding features into the decoding network model, and obtain the edge features corresponding to the original point cloud according to the point cloud coding features through the edge module branch in the decoding network model, and obtain the point features and edge features corresponding to the original point cloud according to the point cloud coding features and the edge features through the point module branch in the decoding network model; the classification module 1903 is used to perform classification prediction based on the point features and the edge features to obtain the classification information corresponding to each point in the original point cloud.

[0115] In one embodiment of the present disclosure, the encoding module 1901 is configured to: perform multiple downsampling on the original point cloud to generate multiple levels of point cloud layers containing different numbers of points; perform feature extraction on the point cloud data corresponding to each level of the point cloud layer to obtain point cloud coding features corresponding to each level of the point cloud layer.

[0116] In one embodiment of the present disclosure, the point cloud coding features include intermediate point cloud coding features and zero-level point cloud coding features, wherein the zero-level point cloud coding features are generated by extracting features from the last-level point cloud layer generated by downsampling, and the intermediate point cloud coding features are generated by extracting features from the point cloud layer before the last-level point cloud layer.

[0117] In one embodiment of the present disclosure, the intermediate point cloud coding features include multi-level intermediate point cloud coding features; the decoding network model includes multi-level decoding network sub-models, and each level of the decoding network sub-model includes a point module and an edge module; the decoding module 1902 includes: a feature extraction unit, used to input M-level point features and M+1-level intermediate point cloud coding features into the M+1-level decoding network sub-model, obtain M-level edge features according to the M-level point features and M-1-level edge features through the edge module in the M+1-level decoding network sub-model, and obtain M+1-level point features according to the M+1-level intermediate point cloud coding features, the M-level point features and the M-1-level edge features through the point module in the M+1-level decoding network sub-model; wherein M is a positive integer less than the total number of decoding network sub-models.

[0118] In one embodiment of the present disclosure, the decoding module 1902 is configured to: input the zero-level point cloud coding features and the first-level intermediate point cloud coding features into the first-level decoding network sub-model, obtain the zero-level edge features according to the zero-level point cloud coding features through the edge module in the first-level decoding network sub-model, and obtain the first-level point features according to the first-level intermediate point cloud coding features, the zero-level point cloud coding features and the zero-level edge features through the point module in the first-level decoding network sub-model.

[0119] In one embodiment of the present disclosure, the feature extraction unit includes: an edge feature acquisition unit, which is used to extract features from the position information of each point in the M-level point features, the M-1-level graph structure and the M-1-level edge features to obtain the M-level edge features.

[0120] In one embodiment of the present disclosure, the edge feature acquisition unit includes: a graph structure construction unit, used to construct an M-level graph structure according to the position information of each point in the M-level point feature and the M-1-level graph structure; an upsampling unit, used to upsample the edge according to the M-1-level edge feature and the M-level graph structure to obtain the upsampled edge feature; and an encoding unit, used to encode the M-level graph structure, the M-level point feature and the upsampled edge feature to obtain the M-level edge feature.

[0121] In one embodiment of the present disclosure, the graph structure construction unit is configured to: determine any point in the M-level point cloud layer as a first target point, obtain a first neighbor point corresponding to the first target point in the M-level point cloud layer, and the distance between the first neighbor point and the first target point does not exceed a preset distance threshold; connect each of the first target points in the M-level point cloud layer and the first neighbor point corresponding to each of the first target points to obtain an M-level initial graph structure; use any edge in the M-level initial graph structure as a first target edge, and obtain two endpoints corresponding to the first target edge; in M- Determine the neighboring points corresponding to each of the endpoints in the level 1 point cloud layer, determine a target neighboring point from the neighboring points corresponding to each of the endpoints, connect the target neighboring points to form a detection edge, and determine whether the detection edge is an edge in the M-1 level graph structure; when the detection edge is not an edge in the M-1 level graph structure, delete the target edge from the M-level initial graph structure; repeatedly determine whether the detection edges corresponding to all edges in the M-1 level initial graph structure are edges in the M-1 level graph structure, and update all edges according to the determination result to obtain the M-level graph structure.

[0122] In one embodiment of the present disclosure, the upsampling unit is configured to: determine any edge in the M-level graph structure as an edge to be processed; determine a neighboring edge corresponding to the edge to be processed in the edges corresponding to the M-1-level edge feature according to the two endpoints corresponding to the edge to be processed; take the intersection of the neighboring edge and the edge corresponding to the M-1-level edge feature, and interpolate the edge feature corresponding to the edge in the intersection according to the distance between the two endpoints corresponding to the edge to be processed, so as to obtain the upsampled edge feature.

[0123] In one embodiment of the present disclosure, the encoding unit is configured to: take any edge in the M-level graph structure as a second target edge, and obtain two endpoints corresponding to the second target edge; obtain point features corresponding to each of the endpoints from the M-level point features, connect the point features corresponding to each of the endpoints, and perform feature extraction on the connected point features to obtain connection point features; concatenate the connection point features and the upsampled edge features corresponding to the second target edge, and perform feature extraction on the concatenated features to obtain M-level edge sub-features corresponding to the second target edge; repeat the above steps until the M-level edge sub-features corresponding to each edge in the M-level graph structure are obtained, and determine the M-level edge features based on the M-level edge sub-features corresponding to each edge.

[0124] In one embodiment of the present disclosure, the last-level point cloud layer is a zero-level point cloud layer; the point cloud semantic segmentation device 1900 can also be configured to: determine any point in the zero-level point cloud layer as a second target point, and determine the neighboring point corresponding to the second target point based on the relationship between the distance between the second target point and other points in the zero-level point cloud layer and a preset distance; construct the zero-level graph structure based on the second target point and the second neighboring point.

[0125] In one embodiment of the present disclosure, the decoding module 1902 is configured to: perform maximum pooling processing on the M-level edge sub-features corresponding to the edges connecting each point in the M-level point cloud layer to obtain the regional features corresponding to each point in the M-level point cloud layer; connect the point features corresponding to each point in the M-level point cloud layer with the regional features, and perform interpolation processing on the connected features to obtain interpolation point features; connect the interpolation point features with the M+1-level intermediate point cloud coding features, and perform feature extraction on the connected features to obtain the M+1-level point features.

[0126] In one embodiment of the present disclosure, the classification module 1903 is configured to: perform maximum pooling processing on the edge sub-features in the edge features corresponding to each point in the original point cloud to obtain the regional features corresponding to each point in the original point cloud; connect the point features with the regional features corresponding to each point in the original point cloud, and classify and predict the connected features through a multi-layer perceptron to obtain classification information corresponding to each point in the original point cloud.

[0127] Fig. 20 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present disclosure is shown.

[0128] It should be noted that Fig. 20 The computer system 2000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0129] like Fig. 20 As shown, the computer system 2000 includes a central processing unit (CPU) 2001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 2002 or the program loaded from the storage part 2008 to the random access memory (RAM) 2003 to implement the image annotation method described in the above embodiment. In the RAM 2003, various programs and data required for system operation are also stored. The CPU 2001, the ROM 2002 and the RAM 2003 are connected to each other through the bus 2004. The input / output (I / O) interface 2005 is also connected to the bus 2004.

[0130] The following components are connected to the I / O interface 2005: an input section 2006 including a keyboard, a mouse, etc.; an output section 2007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 2008 including a hard disk, etc.; and a communication section 2009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 2009 performs communication processing via a network such as the Internet. A drive 2010 is also connected to the I / O interface 2005 as needed. A removable medium 2011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 2010 as needed so that a computer program read therefrom is installed into the storage section 2008 as needed.

[0131] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 2009, and / or installed from a removable medium 2011. When the computer program is executed by a central processing unit (CPU) 2001, various functions defined in the system of the present disclosure are executed.

[0132] It should be noted that the computer-readable medium shown in the embodiment of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0133] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0134] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0135] As another aspect, the present disclosure further provides a computer-readable medium, which may be included in the image processing device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0136] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0137] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0138] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0139] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A point cloud semantic segmentation method, characterized in that: include: Obtaining an original point cloud, encoding point cloud data corresponding to the original point cloud through a coding network model to obtain point cloud coding features; the point cloud coding features include intermediate point cloud coding features and zero-level point cloud coding features, wherein the zero-level point cloud coding features are generated by extracting features from the last level point cloud layer generated by downsampling, and the intermediate point cloud coding features are generated by extracting features from the point cloud layer before the last level point cloud layer; the intermediate point cloud coding features include multi-level intermediate point cloud coding features; Inputting the point cloud coding features into a decoding network model, wherein the decoding network model includes a multi-level decoding network sub-model, and each level of the decoding network sub-model includes a point module and an edge module; Inputting the M-level point features and the M+1-level intermediate point cloud coding features into the M+1-level decoding network sub-model, extracting the position information of each point in the M-level point features, the M-1-level graph structure and the M-1-level edge features through the edge module in the M+1-level decoding network sub-model to obtain the M-level edge features, and obtaining the M+1-level point features according to the M+1-level intermediate point cloud coding features, the M-level point features and the M-level edge features through the point module in the M+1-level decoding network sub-model; wherein M is a positive integer less than the total number of decoding network sub-models; Perform classification prediction based on the point features and the edge features to obtain classification information corresponding to each point in the original point cloud; The extracting the position information of each point in the M-level point features, the M-1-level graph structure and the M-1-level edge features to obtain the M-level edge features includes: Take the points in the M-level point cloud layer as the center of the circle, and form multiple spheres with different radii, select several points from each sphere, and form an M-level initial graph structure based on the center point of the circle and the selected points; Any edge in the M-level initial graph structure is used as the first target edge, and two endpoints corresponding to the first target edge are obtained; neighboring points corresponding to each of the endpoints are determined in the M-1 level point cloud layer, a target neighboring point is determined from the neighboring points corresponding to each of the endpoints, and the target neighboring points are connected to form a detection edge, and it is determined whether the detection edge is an edge in the M-1 level graph structure; when the detection edge is not an edge in the M-1 level graph structure, the target edge is deleted from the M-level initial graph structure; it is repeatedly determined whether the detection edges corresponding to all edges in the M-level initial graph structure are edges in the M-1 level graph structure, and all edges are updated according to the determination result to obtain the M-level graph structure; The edges are upsampled according to the M-1-level edge features and the M-level graph structure to obtain upsampled edge features; the M-level graph structure, the M-level point features and the upsampled edge features are encoded to obtain the M-level edge features; wherein the zero-level edge feature is obtained by taking any point in the zero-level point cloud layer corresponding to the zero-level point cloud encoding feature as the target point, searching for neighboring points in the zero-level point cloud layer whose distance from the target point does not exceed a preset distance threshold, obtaining point features corresponding to the target point and the neighboring points at the same time, and performing feature extraction on the connected point features.

2. The point cloud semantic segmentation method according to claim 1, characterized in that: The step of encoding the point cloud data corresponding to the original point cloud by means of the encoding network model to obtain point cloud encoding features includes: Downsampling the original point cloud multiple times to generate multiple levels of point cloud layers containing different numbers of points; Feature extraction is performed on the point cloud data corresponding to each level of the point cloud layer to obtain point cloud coding features corresponding to each level of the point cloud layer.

3. The point cloud semantic segmentation method according to claim 1, characterized in that: The method further comprises: The zero-level point cloud coding features and the first-level intermediate point cloud coding features are input into the first-level decoding network sub-model, and the zero-level edge features are obtained according to the zero-level point cloud coding features through the edge module in the first-level decoding network sub-model, and the first-level point features are obtained according to the first-level intermediate point cloud coding features, the zero-level point cloud coding features and the zero-level edge features through the point module in the first-level decoding network sub-model.

4. The point cloud semantic segmentation method according to claim 1, characterized in that: The upsampling of edges according to the M-1 level edge features and the M level graph structure to obtain upsampled edge features includes: Determine any edge in the M-level graph structure as an edge to be processed; Determine, according to the two endpoints corresponding to the edge to be processed, the neighboring edge corresponding to the edge to be processed among the edges corresponding to the M-1 level edge features; The intersection of the neighbor edge and the edge corresponding to the M-1 level edge feature is taken, and the edge feature corresponding to the edge in the intersection is interpolated according to the distance between the two endpoints corresponding to the edge to be processed, so as to obtain the upsampled edge feature.

5. The point cloud semantic segmentation method according to claim 1, characterized in that: The encoding of the M-level graph structure, the M-level point features, and the upsampled edge features to obtain the M-level edge features includes: Taking any edge in the M-level graph structure as a second target edge, and obtaining two endpoints corresponding to the second target edge; Acquire point features corresponding to each of the endpoints from the M-level point features, connect the point features corresponding to each of the endpoints, and perform feature extraction on the connected point features to obtain connection point features; Concatenate the connection point feature and the upsampled edge feature corresponding to the second target edge, and perform feature extraction on the concatenated features to obtain M-level edge sub-features corresponding to the second target edge; The above steps are repeated until the M-level edge sub-features corresponding to each edge in the M-level graph structure are obtained, and the M-level edge feature is determined according to the M-level edge sub-features corresponding to each edge.

6. The point cloud semantic segmentation method according to claim 1, characterized in that: The last level point cloud layer is a zero-level point cloud layer; the method further includes: Determine any point in the zero-level point cloud layer as a second target point, and determine a neighboring point corresponding to the second target point according to a relationship between the distance between the second target point and other points in the zero-level point cloud layer and a preset distance; A zero-level graph structure is constructed according to the second target point and neighboring points corresponding to the second target point.

7. The point cloud semantic segmentation method according to claim 1, characterized in that: The step of obtaining the M+1 level point features according to the M+1 level intermediate point cloud coding features, the M level point features and the M level edge features through the point module in the M+1 level decoding network sub-model includes: Performing maximum pooling processing on the M-level edge sub-features corresponding to the edges connected to each point in the M-level point cloud layer to obtain the regional features corresponding to each point in the M-level point cloud layer; Connecting the point features corresponding to each point in the M-level point cloud layer with the regional features, and performing interpolation processing on the connected features to obtain interpolation point features; The interpolation point feature and the M+1 level intermediate point cloud coding feature are connected, and feature extraction is performed on the connected features to obtain the M+1 level point feature.

8. The point cloud semantic segmentation method according to claim 1, characterized in that: The performing classification prediction according to the point features and the edge features to obtain classification information corresponding to each point in the original point cloud includes: Performing maximum pooling processing on the edge sub-features in the edge features corresponding to each point in the original point cloud to obtain regional features corresponding to each point in the original point cloud; The point features are connected with the regional features corresponding to each point in the original point cloud, and the connected features are classified and predicted by a multi-layer perceptron to obtain the classification information corresponding to each point in the original point cloud.

9. A point cloud semantic segmentation device, characterized in that: include: A coding module, used to obtain an original point cloud, and encode the point cloud data corresponding to the original point cloud through a coding network model to obtain intermediate point cloud coding features and zero-level point cloud coding features; the point cloud coding features include intermediate point cloud coding features and zero-level point cloud coding features, wherein the zero-level point cloud coding features are generated by extracting features from the last level point cloud layer generated by downsampling, and the intermediate point cloud coding features are generated by extracting features from the point cloud layer before the last level point cloud layer; the intermediate point cloud coding features include multi-level intermediate point cloud coding features; A decoding module, used for inputting the point cloud coding features into a decoding network model, wherein the decoding network model includes a multi-level decoding network sub-model, and each level of the decoding network sub-model includes a point module and an edge module; The decoding module includes: a feature extraction unit, which is used to input the M-level point features and the M+1-level intermediate point cloud coding features into the M+1-level decoding network sub-model, extract the position information of each point in the M-level point features, the M-1-level graph structure and the M-1-level edge features through the edge module in the M+1-level decoding network sub-model to obtain the M-level edge features, and obtain the M+1-level point features according to the M+1-level intermediate point cloud coding features, the M-level point features and the M-level edge features through the point module in the M+1-level decoding network sub-model; wherein M is a positive integer less than the total number of decoding network sub-models; The feature extraction unit includes: an edge feature acquisition unit, and the edge feature acquisition unit includes: a graph structure construction unit, which is used to take a point in the M-level point cloud layer as the center of a circle and form multiple spheres with different radii, select a number of points from each sphere, and form an M-level initial graph structure according to the center point and the selected points; take any edge in the M-level initial graph structure as a first target edge, and obtain two endpoints corresponding to the first target edge; determine the neighboring points corresponding to each of the endpoints in the M-1 level point cloud layer, determine a target neighboring point from the neighboring points corresponding to each of the endpoints, and connect the target neighboring points to form a detection edge, and determine whether the detection edge is an edge in the M-1 level graph structure; when the detection edge is not an edge in the M-1 level graph structure, remove the target edge from the M-level initial graph structure. The method comprises the following steps: deleting the detected edges corresponding to all the edges in the M-level initial graph structure; repeatedly judging whether the detected edges corresponding to all the edges in the M-1-level graph structure are the edges in the M-1-level graph structure, and updating all the edges according to the judgment result to obtain the M-level graph structure; an upsampling unit is used to upsample the edges according to the M-1-level edge features and the M-level graph structure to obtain the upsampled edge features; an encoding unit is used to encode the M-level graph structure, the M-level point features and the upsampled edge features to obtain the M-level edge features; wherein the zero-level edge feature is to take any point in the zero-level point cloud layer corresponding to the zero-level point cloud encoding feature as the target point, find the neighboring points in the zero-level point cloud layer whose distance from the target point does not exceed the preset distance threshold, obtain the point features corresponding to the target point and the neighboring points at the same time, and extract the features of the connected point features; The classification module is used to perform classification prediction based on the point features and the edge features to obtain classification information corresponding to each point in the original point cloud.

10. An electronic device, characterized in that: include: one or more processors; A storage device, wherein the storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors execute the point cloud semantic segmentation method as described in any one of claims 1 to 8.

11. A computer readable medium, characterized in that The computer-readable medium carries one or more programs, and when the one or more programs are executed by an electronic device, the electronic device implements the point cloud semantic segmentation method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that It comprises a computer program carried on a computer-readable medium, the computer program containing a program code for executing the method according to any one of claims 1 to 8.

Images