Point cloud segmentation method and device based on adaptive graph convolution, equipment and storage medium

Abstract

The invention discloses a point cloud segmentation method and device based on adaptive graph convolution, equipment and a storage medium. The method comprises the following steps: acquiring point cloud information corresponding to a target object, and determining a neighborhood edge point corresponding to each point cloud in the point cloud information; based on the point cloud information, neighborhood edge points corresponding to the point clouds and a preset multi-layer perceptron model, determining an adaptive convolution kernel corresponding to each point cloud to construct a convolution kernel library; obtaining a weight score vector corresponding to each convolution kernel in a convolution kernel library based on a position offset between point cloud position information in the point cloud information and neighborhood edge point position information in the neighborhood edge points and a preset multilayer perceptron model; and determining enhanced edge features according to each weight score vector and the convolution kernel library, and aggregating each enhanced edge feature to obtain a target output feature. According to the technical scheme, the relation between different points can be flexibly captured, and the accuracy of three-dimensional point cloud classification and segmentation is improved.

Description

Technical Field

[0001] The present invention relates to the field of point cloud analysis technology, and in particular to a point cloud segmentation method, device, equipment and storage medium based on adaptive graph convolution. Background Art

[0002] Point cloud analysis is one of the important fields in computer vision, especially the point cloud classification and segmentation problem, which is widely used in fields such as autonomous driving, face recognition and pattern recognition, and has become one of the classic problems in the industry. With the rapid development of deep learning methods, there are more and more research methods based on point cloud analysis and more and more standard point cloud data sets that can be used for training. These methods and data sets have greatly promoted the progress of point cloud analysis.

[0003] In the existing technology, kernel weights are usually predicted, each parameter in the kernel is assumed to imply a fixed positional relationship, and an attention mechanism is introduced to assign different weights to each edge to distinguish important points. However, these methods still cannot essentially distinguish the relationship between each point pair, because the features between each point pair are still generated by the same convolution kernel, which makes some distinctive features impossible to be identified and the accuracy is extremely low. Summary of the invention

[0004] In view of this, the present invention provides a point cloud segmentation method, apparatus, device and storage medium based on adaptive graph convolution, which can flexibly capture the connection between different points and improve the accuracy of three-dimensional point cloud classification and segmentation.

[0005] According to one aspect of the present invention, an embodiment of the present invention provides a point cloud segmentation method based on adaptive graph convolution, the method comprising:

[0006] Obtaining point cloud information corresponding to the target object, and determining the neighborhood edge points corresponding to each point cloud in the point cloud information;

[0007] Determine the adaptive convolution kernel Kernel corresponding to each point cloud based on the point cloud information, the neighborhood edge points corresponding to the point cloud, and a preset multi-layer perceptron model to construct a convolution kernel library;

[0008] Determine the position offsets between the point cloud position information in the point cloud information and the neighborhood edge point position information in the neighborhood edge points, and obtain the weight score vectors corresponding to each convolution kernel in the convolution kernel library based on the position offsets and the preset multi-layer perceptron model;

[0009] The enhanced edge features are determined according to each of the weight score vectors and the convolution kernel library, and the enhanced edge features are aggregated to obtain target output features, and the target output features are used as the result of point cloud segmentation.

[0010] According to another aspect of the present invention, an embodiment of the present invention further provides a point cloud segmentation device based on adaptive graph convolution, the device comprising:

[0011] An information acquisition module, used to acquire point cloud information corresponding to the target object, and determine the neighborhood edge points corresponding to each point cloud in the point cloud information;

[0012] A convolution kernel library construction module is used to determine the adaptive convolution kernel Kernel corresponding to each point cloud based on the point cloud information, the neighborhood edge points corresponding to the point cloud, and a preset multi-layer perceptron model to construct a convolution kernel library;

[0013] A weight coefficient determination module, used to determine the position offset between the point cloud position information in the point cloud information and the neighborhood edge point position information in the neighborhood edge point, and obtain the weight score vector corresponding to each convolution kernel in the convolution kernel library based on the position offset and a preset multi-layer perceptron model;

[0014] The feature determination module is used to determine the enhanced edge features according to each of the weight score vectors and the convolution kernel library, and to aggregate each of the enhanced edge features to obtain a target output feature, and to use the target output feature as a result of point cloud segmentation.

[0015] According to another aspect of the present invention, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the point cloud segmentation method based on adaptive graph convolution described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a processor to implement the point cloud segmentation method based on adaptive graph convolution described in any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, an embodiment of the present invention further provides a computer program product, characterized in that the computer program product includes a computer program, and when the computer program is executed by a processor, it implements the point cloud segmentation method based on adaptive graph convolution described in any embodiment of the present invention.

[0021] The technical effect of the present invention is that, by determining the neighborhood edge points corresponding to each point cloud in the point cloud information, the adaptive convolution kernels corresponding to each point cloud are determined based on the point cloud information, the neighborhood edge points corresponding to the point cloud, and a preset multi-layer perceptron model to construct a convolution kernel library. On this basis, the weight score vectors corresponding to each convolution kernel in the convolution kernel library are obtained according to the position offsets between the point cloud position information in the point cloud information and the neighborhood edge point position information in the neighborhood edge points, and the preset multi-layer perceptron model. Then, the enhanced edge features are determined according to each weight score vector and the convolution kernel library, and the enhanced edge features are aggregated to obtain the target output features. The target output features are used as the result of point cloud segmentation, so that different convolution kernels can be generated according to the positional relationship between the point pairs. By generating weight coefficients and constructing corresponding convolution kernels from the dynamic kernel library according to the weight coefficients, the connection between different points can be flexibly captured, thereby improving the accuracy of three-dimensional point cloud classification and segmentation.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A flowchart of a point cloud segmentation method based on adaptive graph convolution provided by an embodiment of the present invention;

[0025] Figure 2 A flowchart of another point cloud segmentation method based on adaptive graph convolution provided by an embodiment of the present invention;

[0026] Figure 3 A schematic diagram of a point cloud segmentation architecture based on adaptive graph convolution provided by an embodiment of the present invention;

[0027] Figure 4 A structural block diagram of a point cloud segmentation device based on adaptive graph convolution provided by an embodiment of the present invention;

[0028] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] In one embodiment, Figure 1 A flowchart of a point cloud segmentation method based on adaptive graph convolution is provided for one embodiment of the present invention. This embodiment is applicable to the situation when three-dimensional point clouds in images are segmented. The method can be performed by a point cloud segmentation device based on adaptive graph convolution. The point cloud segmentation device based on adaptive graph convolution can be implemented in the form of hardware and / or software. The point cloud segmentation device based on adaptive graph convolution can be configured in an electronic device.

[0032] like Figure 1 As shown, the point cloud segmentation method based on adaptive graph convolution in this embodiment specifically includes the following steps:

[0033] S110, obtaining point cloud information corresponding to the target object, and determining the neighborhood edge points corresponding to each point cloud in the point cloud information.

[0034] The point cloud information includes the point cloud features and point cloud position information corresponding to each point cloud; the neighborhood edge points include the neighborhood edge point features and neighborhood edge point position information. The point cloud position information in this embodiment can be understood as the point cloud 3D coordinate information, and the neighborhood edge point position information is the coordinate information of the neighborhood edge points.

[0035] In this embodiment, the corresponding point cloud information is obtained by scanning the object through a laser radar scanner as input, and each point cloud in the point cloud information corresponds to a corresponding neighborhood edge point. In this embodiment, the neighborhood edge point corresponding to each point cloud is generally a plurality of edge points, and the neighborhood edge point corresponding to each point cloud in the point cloud information can be determined by a preset nearest neighbor algorithm, farthest point sampling, and ball query method. In some embodiments, when the ball query method is used for determination, each center point can be used as the center of the sphere, and a point cloud neighborhood is constructed for all point clouds within a preset radius. All point clouds in the point cloud neighborhood are sorted in ascending order according to the distance between the center point and the point cloud neighborhood, and the first n point clouds are taken as the neighbor points of the center point, where n is the number of neighbor points in each neighborhood that is preset. In other embodiments, the preset nearest neighbor algorithm can be K nearest neighbor, which means k nearest neighbors, and each sample can be represented by its closest k neighbors. The algorithm assumes that all instances correspond to points in N-dimensional Euclidean space. By calculating the distance between a point and all other points, taking out the K points closest to the point, and then counting the points with the largest proportion of categories among these K points, the point belongs to that category.

[0036] S120, determining the adaptive convolution kernel Kernel corresponding to each point cloud based on the point cloud information, the neighborhood edge points corresponding to the point cloud, and the preset multi-layer perceptron model to construct a convolution kernel library.

[0037] Among them, the preset multilayer perceptron model can be understood as the multilayer perceptron model in the prior art. The multilayer perceptron model is a deep neural network model, including an input layer, an output layer and a hidden layer. The activation function corresponding to the multilayer perceptron model is generally nonlinear, exemplarily, the rectified linear unit ReLU, ReLU provides a very simple nonlinear transformation.

[0038] In this embodiment, the adaptive convolution kernel Kernel corresponding to each point cloud can be determined through point cloud information, the neighborhood edge points corresponding to the point cloud, and the preset multi-layer perceptron model, so as to construct a convolution kernel library through the adaptive convolution kernel Kernel corresponding to each point cloud. In some embodiments, the point cloud features corresponding to each point cloud in the point cloud information and the edge point features corresponding to the neighborhood edge points corresponding to the point cloud can be extracted, and each point is taken as the center point, and the features of the center point and the edge point features of the neighborhood edge points around the center point are input into the dynamic graph edge convolution network DGCNN network in matrix form to obtain the target edge features after the standard edge convolution. On this basis, the target edge features are input into the preset multi-layer perceptron model for feature extraction to obtain the convolution kernel corresponding to each feature, so as to construct a convolution kernel library through the convolution kernel corresponding to each feature.

[0039] It can be understood that in standard shared convolution, the input of the convolution kernel is input sequentially, so the parameter adjustment is based on all inputs. In this embodiment, each input edge feature generates an independent convolution kernel, which is different from the standard convolution random initialization convolution kernel and generates an independent convolution kernel based on all input optimization parameters. Parameter optimization is only related to the unique input edge feature.

[0040] S130, determining the position offsets between the point cloud position information in the point cloud information and the neighborhood edge point position information in the neighborhood edge points, and obtaining the weight score vectors corresponding to each convolution kernel in the convolution kernel library based on the position offsets and a preset multi-layer perceptron model.

[0041] Among them, the position offset is the difference between the point cloud position information in the point cloud information and the neighborhood edge point position information in the neighborhood edge point. The weight score vector can be understood as the weight ratio coefficient corresponding to each convolution kernel.

[0042] In this embodiment, the position offset between the point cloud position information in the point cloud information and the position information of the neighborhood edge points in the neighborhood edge points can be determined by calculating the difference between the point cloud position information in the point cloud information and the position information of the neighborhood edge points in the neighborhood edge points, and then the position offset is input into the preset multi-layer perceptron model to obtain the weight score vector. It can be understood that in the center point and the surrounding neighborhood point set, when a specific position parameter based on the two is given, the corresponding relationship coefficient can be obtained according to the position parameter to adaptively allocate different weights in different dynamic kernels, thereby preparing for the subsequent generation of a convolution kernel with independent discriminability.

[0043] S140, determining the enhanced edge features according to each weight score vector and the convolution kernel library, and aggregating each enhanced edge feature to obtain a target output feature, and using the target output feature as a result of point cloud segmentation.

[0044] Among them, the enhanced edge feature can be understood as the enhanced edge feature obtained by weighting the weight score vector and the convolution kernel library, and then convolving with the offset of the neighborhood edge point; the target output feature can be understood as the output feature obtained by aggregating each enhanced edge feature.

[0045] In this embodiment, the target adaptive convolution kernel is determined according to the weight score vector and the convolution kernel library, and the target adaptive convolution kernel is convolved with the offset of the neighborhood edge point to obtain the enhanced edge feature, and then all the enhanced edge features are aggregated to obtain the target output feature. Specifically, the generated weight score vector is weighted and summed with each convolution kernel in the convolution kernel library to obtain an independent adaptive convolution kernel, and then the adaptive convolution kernel is convolved with the offset of the neighboring node to obtain the enhanced edge feature. In this convolution operation, the shape of the convolution kernel will change according to the positional relationship between each node and its neighbor to adapt to the spatial structure between different nodes. Finally, all enhanced edge features are aggregated through a symmetric operation function to obtain the final output feature.

[0046] The technical solution of the embodiment of the present invention determines the neighborhood edge points corresponding to each point cloud in the point cloud information, determines the adaptive convolution kernel corresponding to each point cloud based on the point cloud information, the neighborhood edge points corresponding to the point cloud, and a preset multi-layer perceptron model to construct a convolution kernel library. On this basis, the weight score vector corresponding to each convolution kernel in the convolution kernel library is obtained according to the position offset between the point cloud position information in the point cloud information and the neighborhood edge point position information in the neighborhood edge points, and the preset multi-layer perceptron model. Then, the enhanced edge features are determined according to each weight score vector and the convolution kernel library, and the enhanced edge features are aggregated to obtain the target output features. The target output features are used as the result of point cloud segmentation, so that different convolution kernels can be generated according to the positional relationship between the point pairs. By generating weight coefficients and constructing corresponding convolution kernels from the dynamic kernel library according to the weight coefficients, the connection between different points can be flexibly captured, and the accuracy of three-dimensional point cloud classification and segmentation can be improved.

[0047] In one embodiment, Figure 2 A flowchart of another point cloud segmentation method based on adaptive graph convolution provided by one embodiment of the present invention. Based on the above embodiments, this embodiment determines the neighborhood edge points corresponding to each point cloud in the point cloud information; determines the adaptive convolution kernel Kernel corresponding to each point cloud based on the point cloud information, the neighborhood edge points corresponding to the point cloud, and a preset multi-layer perceptron model to construct a convolution kernel library; determines the position offset between the point cloud position information in the point cloud information and the neighborhood edge point position information in the neighborhood edge points, and obtains the weight score vector corresponding to each convolution kernel in the convolution kernel library based on the position offset and the preset multi-layer perceptron model; and determines the enhanced edge features according to each weight score vector and the convolution kernel library, and further refines the target output features by aggregating each enhanced edge feature.

[0048] like Figure 2As shown, the point cloud segmentation method based on adaptive graph convolution in this embodiment may specifically include the following steps:

[0049] S210, obtaining point cloud information corresponding to the target object, and taking each point cloud as a center point, searching for all neighborhood edge points around the center point according to a preset nearest neighbor algorithm.

[0050] Among them, the preset nearest neighbor algorithm can be understood as the k-nearest neighbor method, which is a machine learning algorithm. It is a classification algorithm that determines its own category based on the classification of its K nearest samples. When calculating the distance, the straight-line distance between them, namely the Euclidean distance, or the sum of the absolute values ​​of the coordinates, namely the Manhattan distance, can be used. This embodiment is not limited here.

[0051] In this embodiment, for all point cloud information, each point cloud is taken as the center point, and the center point is used as the reference to search for all neighborhood edge points around the center point according to the preset nearest neighbor algorithm. It can be understood that the k nearest neighbors around the center point are searched through the nearest neighbor algorithm as all neighborhood edge points around the center point. Specifically, the distance between the current center point and all points is calculated, and the obtained distances are arranged in ascending order. The K points with the closest distances are selected from the sorting results, and then the frequency of occurrence of the categories of these K points is counted, and the category with the highest frequency among these K points is used as the predicted classification, thereby obtaining the k nearest neighbors around the center point.

[0052] S220, extracting point cloud features corresponding to each point cloud in the point cloud information, and edge point features corresponding to the neighborhood edge points corresponding to the point cloud.

[0053] In this embodiment, feature extraction is performed on each point cloud in the point cloud information to obtain point cloud features corresponding to each point cloud, and feature extraction is performed on the neighborhood edge points corresponding to the point cloud to obtain edge point features corresponding to the neighborhood edge points corresponding to the point cloud.

[0054] S230, taking each point as the center point respectively, and inputting the features of the center point and the edge point features of the neighborhood edge points around the center point into the dynamic graph edge convolution network DGCNN network in matrix form to obtain the target edge features after standard edge convolution.

[0055] In this embodiment, the features of the center point and the edge point features of the edge points in the neighborhood around the center point are input into the dynamic graph edge convolution network DGCNN network in matrix form to obtain the target edge features after standard edge convolution. The dynamic graph edge convolution network DGCNN network formula is expressed as: h(x i ,x j )=h θ (x i ,x j-x i ), where x i Represented as the feature of the center point, x j Represented as edge point features of the neighborhood edge points; x j -x i Represents local area information; θ represents the learning parameter.

[0056] S240, inputting the target edge features into the multi-layer perceptron model for feature extraction, and the target edge features are processed by a 1×1 convolution layer and a Relu activation function in the multi-layer perceptron model to obtain the convolution kernel corresponding to each feature.

[0057] In this embodiment, the target edge features are input into the multi-layer perceptron model for feature extraction, and the target edge features are processed by a 1×1 convolution layer and a Relu activation function in the multi-layer perceptron model to obtain the convolution kernel corresponding to each feature. The convolution kernel is represented as: Where m represents the number of convolution kernels; Δx ij Represented as x j -x i ;x i Represented as the feature of the center point, x j g is the edge point feature represented by the neighborhood edge point; m <·,·> is implemented by multi-layer perceptron, Represented as f ij The target edge features after standard edge convolution; p j Represents the location information of the neighborhood edge points; N i It is expressed as the number of edge points in the area corresponding to the center point.

[0058] In this embodiment, the ReLU function is defined as the maximum value of the element and . It can be expressed as: ReLU(x) = max(x, 0). In this embodiment, the reason for using ReLU is that it performs particularly well in derivation: either making the parameters disappear or passing through, which makes the optimization performance better, and ReLU alleviates the gradient vanishing problem that plagued previous neural networks.

[0059] S250, the convolution kernels corresponding to the features are combined into a convolution kernel library.

[0060] In this embodiment, the convolution kernels corresponding to the features are combined into a convolution kernel library, where the convolution kernel library can be expressed as: Among them, W k Represented as the kth convolution kernel.

[0061] S260, calculating the difference between the point cloud position information in the point cloud information and the neighborhood edge point position information in the neighborhood edge point, and using the difference as the position offset.

[0062] In this embodiment, the difference between the point cloud position information in the point cloud information and the neighborhood edge point position information in the neighborhood edge point is calculated, and the difference is used as the position offset.

[0063] S270, inputting the position offset into a preset multi-layer perceptron model to obtain a weight score vector.

[0064] In this embodiment, the position offset is input into a preset multi-layer perceptron model to obtain a weight score vector. The weight score vector is represented as: ij，1 ,s ij，2 , ..., s ij，k ; Among them, s ij，k It is represented as the position information of the jth neighborhood edge point of the ith center point corresponding to the kth convolution kernel.

[0065] S280, determining a target adaptive convolution kernel according to each weight score vector and the convolution kernel library, and convolving the target adaptive convolution kernel with the offset of the neighborhood edge point to obtain an enhanced edge feature.

[0066] In this embodiment, the target adaptive convolution kernel is determined according to each weight score vector and the convolution kernel library, and the target adaptive convolution kernel is convolved with the offset of the neighborhood edge point to obtain enhanced edge features. Specifically, the generated weight coefficient is weighted and summed with the convolution kernel in the dynamic library to obtain an independent adaptive convolution kernel, and then the adaptive convolution kernel is convolved with the offset of the neighboring node to obtain enhanced edge features. In this convolution operation, the shape of the convolution kernel changes according to the positional relationship between each node and its neighbors to adapt to the spatial structure between different nodes. Finally, all enhanced edge features are aggregated through a symmetric operation function to obtain the final output feature.

[0067] In one embodiment, determining a target adaptive convolution kernel according to each weight score vector and a convolution kernel library includes:

[0068] The weight score vector is normalized using the preset Softmax function to obtain a normalized weight score vector;

[0069] The normalized weight score vector is weighted and summed with the corresponding convolution kernel in the convolution kernel library to obtain the target adaptive convolution kernel.

[0070] Among them, the preset Softmax function, also known as the normalized exponential function, is a commonly used function in machine learning. It can compress a K-dimensional vector containing any real number into another K-dimensional real vector so that the range of each element is between (0,1) and the sum of all elements is 1.

[0071] In this embodiment, the weight score vector is normalized using a preset Softmax function to obtain a normalized weight score vector, and the normalized weight score vector is weighted summed with the corresponding convolution kernel in the convolution kernel library to obtain the target adaptive convolution kernel.

[0072] S290, aggregating all enhanced edge features according to a preset symmetric operation function, and obtaining target output features.

[0073] In this embodiment, all enhanced edge features are aggregated according to a preset symmetry operation function to obtain a target output feature, wherein the preset symmetry operation function may also be referred to as a symmetry function, and the preset symmetry operation function is used to map the enhanced edge features to a feature vector. It can be understood that a set of functions can be defined for the enhanced edge features, and then the enhanced edge features are mapped to the function vector.

[0074] The technical solution of the embodiment of the present invention extracts the point cloud features corresponding to each point cloud in the point cloud information, and the edge point features corresponding to the neighborhood edge points corresponding to the point cloud, and takes each point as the center point respectively, and inputs the features of the center point and the edge point features of the neighborhood edge points around the center point into the dynamic graph edge convolution network DGCNN network in matrix form to obtain the target edge features after standard edge convolution, and inputs the target edge features into the multi-layer perceptron model for feature extraction. The target edge features are processed by a 1×1 convolution layer and a Relu activation function in the multi-layer perceptron model to obtain the convolution kernel corresponding to each feature, and the convolution kernels corresponding to each feature are composed of a convolution kernel library. On this basis, the difference between the point cloud position information in the point cloud information and the neighborhood edge point position information in the neighborhood edge points is calculated, and the difference is used as the position offset. The position offset is input into the preset multi-layer perceptron model to obtain a weight score vector. Then, the target adaptive convolution kernel is determined according to each weight score vector and the convolution kernel library, and the target adaptive convolution kernel is convolved with the offset of the neighborhood edge point to obtain the enhanced edge feature. All enhanced edge features are aggregated according to the preset symmetric operation function to obtain the target output feature. The above technical scheme can design an independent adaptive kernel for each point pair in the point cloud data to capture the significant relationship, generate different convolution kernels according to the positional relationship between the point pairs, generate weight coefficients, and construct the corresponding adaptive convolution kernel from the dynamic kernel library according to the weight coefficients. The adaptive convolution kernel is convolved with the offset of the neighboring node to obtain the enhanced edge feature, further flexibly capture the connection between different points, and improve the accuracy of three-dimensional point cloud classification and segmentation.

[0075] For example, to better understand the point cloud segmentation method based on adaptive graph convolution, Figure 3A schematic diagram of a point cloud segmentation architecture based on adaptive graph convolution provided by an embodiment of the present invention, Figure 3 Δp k Expressed as the position offset of the kth center point; Δf k Represented as the edge point feature of the kth center point; S k It represents the position offset after normalization of the k-th position offset.

[0076] In this embodiment, the neighborhood edge points corresponding to each point cloud in the point cloud information are first determined, and all edge features in the neighborhood are processed through a 1×1 convolution layer and a Relu activation function to obtain a convolution kernel library. At the same time, the position offset between the point cloud position information in the point cloud information and the neighborhood edge point position information in the neighborhood edge point is determined. The position relationship is used as input to generate a relationship weight coefficient through a multi-layer perceptron to use the position information to model the relationship between the nodes in the graph to capture the spatial structure between them. Then, in the convolution kernel construction module, the weight coefficient generated in the position perception module is weighted and summed with the convolution kernel in the dynamic library to obtain an independent adaptive convolution kernel. Then, this adaptive convolution kernel is convolved with the offset of the neighboring node to obtain the enhanced edge feature. In this convolution operation, the shape of the convolution kernel will change according to the position relationship between each node and its neighbor to adapt to the spatial structure between different nodes. Finally, all enhanced edge features are aggregated through a symmetric operation function to obtain the final output feature. The process representation can be expressed as: h i =S(K(p i , p j )x ij |x j ∈N i ).

[0077] In one embodiment, Figure 4 This is a structural block diagram of a point cloud segmentation device based on adaptive graph convolution provided in one embodiment of the present invention. The device is suitable for segmenting a three-dimensional point cloud in an image. The device can be implemented by hardware / software. It can be configured in an electronic device to implement a point cloud segmentation method based on adaptive graph convolution in an embodiment of the present invention.

[0078] like Figure 4 As shown, the device includes: an information acquisition module 410, a convolution kernel library construction module 420, a weight coefficient determination module 430 and a feature determination module 440;

[0079] The information acquisition module 410 is used to acquire point cloud information corresponding to the target object and determine the neighborhood edge points corresponding to each point cloud in the point cloud information;

[0080] A convolution kernel library construction module 420 is used to determine the adaptive convolution kernel Kernel corresponding to each point cloud based on the point cloud information, the neighborhood edge points corresponding to the point cloud, and a preset multi-layer perceptron model to construct a convolution kernel library;

[0081] A weight coefficient determination module 430 is used to determine the position offset between the point cloud position information in the point cloud information and the neighborhood edge point position information in the neighborhood edge point, and obtain the weight score vector corresponding to each convolution kernel in the convolution kernel library based on the position offset and a preset multi-layer perceptron model;

[0082] The feature determination module 440 is used to determine the enhanced edge features according to each of the weight score vectors and the convolution kernel library, and to aggregate each of the enhanced edge features to obtain a target output feature, and to use the target output feature as a result of point cloud segmentation.

[0083] In an embodiment of the present invention, a convolution kernel library construction module is used to determine the adaptive convolution kernel corresponding to each point cloud based on point cloud information, the neighborhood edge points corresponding to the point cloud, and a preset multi-layer perceptron model to construct a convolution kernel library. On this basis, a weight coefficient determination module obtains a weight score vector corresponding to each convolution kernel in the convolution kernel library according to the position offset between the point cloud position information in the point cloud information and the neighborhood edge point position information in the neighborhood edge points, and a preset multi-layer perceptron model. Then, a feature determination module determines the enhanced edge features according to each weight score vector and the convolution kernel library, and aggregates each enhanced edge feature to obtain a target output feature. The target output feature is used as the result of point cloud segmentation, so that different convolution kernels can be generated according to the positional relationship between the point pairs. By generating weight coefficients and constructing corresponding convolution kernels from the dynamic kernel library according to the weight coefficients, the connection between different points can be flexibly captured, and the accuracy of three-dimensional point cloud classification and segmentation can be improved.

[0084] In one embodiment, the point cloud information includes point cloud features and point cloud position information corresponding to each point cloud; the neighborhood edge points include neighborhood edge point features and neighborhood edge point position information; accordingly, the information acquisition module 410 includes:

[0085] The edge determination unit is used to take each point cloud as a center point and search for all neighborhood edge points around the center point according to a preset nearest neighbor algorithm.

[0086] In one embodiment, the convolution kernel library construction module 420 includes:

[0087] A feature extraction unit, used to extract point cloud features corresponding to each point cloud in the point cloud information, and edge point features corresponding to the neighborhood edge points corresponding to the point cloud;

[0088] A target feature determination unit is used to take each point as a center point, and input the features of the center point and the edge point features of the neighborhood edge points around the center point into a dynamic graph edge convolution network DGCNN network in a matrix form to obtain target edge features after standard edge convolution;

[0089] A convolution kernel determination unit is used to input the target edge feature into a multi-layer perceptron model for feature extraction. The target edge feature is processed by a 1×1 convolution layer and a Relu activation function in the multi-layer perceptron model to obtain a convolution kernel corresponding to each feature;

[0090] The convolution kernel library determination unit is used to form a convolution kernel library with the convolution kernels corresponding to each feature.

[0091] In one embodiment, the weight coefficient determination module 430 includes:

[0092] An offset determination unit, used to calculate the difference between the point cloud position information in the point cloud information and the neighborhood edge point position information in the neighborhood edge points, and use the difference as the position offset;

[0093] The weight score determination unit is used to input the position offset into a preset multi-layer perceptron model to obtain a weight score vector.

[0094] In one embodiment, the feature determination module 440 includes:

[0095] An enhanced feature determination unit, used to determine a target adaptive convolution kernel according to each of the weight score vectors and the convolution kernel library, and convolve the target adaptive convolution kernel with the offset of the neighborhood edge point to obtain an enhanced edge feature;

[0096] An aggregation unit is used to aggregate all enhanced edge features according to a preset symmetric operation function and obtain a target output feature; wherein the preset symmetric operation function is used to map the enhanced edge features to a feature vector.

[0097] In one embodiment, determining the target adaptive convolution kernel according to each of the weight score vectors and the convolution kernel library includes:

[0098] A normalization unit, used for normalizing the weight score vector using a preset Softmax function to obtain a normalized weight score vector;

[0099] The target convolution kernel determination unit is used to perform weighted summation on the normalized weight score vector and the corresponding convolution kernel in the convolution kernel library to obtain a target adaptive convolution kernel.

[0100] The point cloud segmentation device based on adaptive graph convolution provided in an embodiment of the present invention can execute the point cloud segmentation method based on adaptive graph convolution provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0101] In one embodiment, Figure 5 A schematic diagram of the structure of an electronic device provided for an embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0102] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0103] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0104] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as a point cloud segmentation method based on adaptive graph convolution.

[0105] In some embodiments, the point cloud segmentation method based on adaptive graph convolution may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the point cloud segmentation method based on adaptive graph convolution described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the point cloud segmentation method based on adaptive graph convolution in any other appropriate manner (e.g., by means of firmware).

[0106] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] The computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable adaptive graph convolution-based point cloud segmentation device, so that when the computer program is executed by the processor, the functions / operations specified in the flowchart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0108] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0109] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0110] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0111] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0112] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0113] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A point cloud segmentation method based on adaptive graph convolution, characterized in that: The method comprises: Obtaining point cloud information corresponding to the target object, and determining the neighborhood edge points corresponding to each point cloud in the point cloud information; Determine the adaptive convolution kernel Kernel corresponding to each point cloud based on the point cloud information, the neighborhood edge points corresponding to the point cloud, and a preset multi-layer perceptron model to construct a convolution kernel library; Determine the position offsets between the point cloud position information in the point cloud information and the neighborhood edge point position information in the neighborhood edge points, and obtain the weight score vectors corresponding to each convolution kernel in the convolution kernel library based on the position offsets and the preset multi-layer perceptron model; The enhanced edge features are determined according to each of the weight score vectors and the convolution kernel library, and the enhanced edge features are aggregated to obtain target output features, and the target output features are used as the result of point cloud segmentation.

2. The method according to claim 1, characterized in that The point cloud information includes point cloud features and point cloud position information corresponding to each point cloud; the neighborhood edge points include neighborhood edge point features and neighborhood edge point position information; accordingly, determining the neighborhood edge point information corresponding to each point cloud in the point cloud information includes: Each point cloud is taken as a center point, and all neighborhood edge points around the center point are searched according to a preset nearest neighbor algorithm.

3. The method according to claim 1, characterized in that The step of determining the adaptive convolution kernel Kernel corresponding to each point cloud based on the point cloud information, the neighborhood edge points corresponding to the point cloud, and the preset multi-layer perceptron model to construct a convolution kernel library includes: Extracting point cloud features corresponding to each point cloud in the point cloud information, and edge point features corresponding to the neighborhood edge points corresponding to the point cloud; Each point is taken as a center point, and the features of the center point and the edge point features of the neighborhood edge points around the center point are input into the dynamic graph edge convolution network DGCNN network in matrix form to obtain the target edge features after standard edge convolution; The target edge features are input into a multi-layer perceptron model for feature extraction. The target edge features are processed by a 1×1 convolution layer and a Relu activation function in the multi-layer perceptron model to obtain a convolution kernel corresponding to each feature; The convolution kernels corresponding to each feature are combined into a convolution kernel library.

4. The method according to claim 1, characterized in that: The determining of the position offsets between the point cloud position information in the point cloud information and the neighborhood edge point position information in the neighborhood edge points, and obtaining the weight score vectors corresponding to each convolution kernel in the convolution kernel library based on the position offsets and a preset multi-layer perceptron model, includes: Calculate the difference between the point cloud position information in the point cloud information and the neighborhood edge point position information in the neighborhood edge point, and use the difference as the position offset; The position offset is input into a preset multi-layer perceptron model to obtain a weight score vector.

5. The method according to claim 1, characterized in that: Determining the enhanced edge features according to each of the weight score vectors and the convolution kernel library, and aggregating each of the enhanced edge features to obtain a target output feature, includes: Determine a target adaptive convolution kernel according to each of the weight score vectors and the convolution kernel library, and convolve the target adaptive convolution kernel with the offset of the neighborhood edge point to obtain an enhanced edge feature; All enhanced edge features are aggregated according to a preset symmetric operation function to obtain target output features; wherein the preset symmetric operation function is used to map the enhanced edge features to a feature vector.

6. The method according to claim 5, characterized in that The step of determining a target adaptive convolution kernel according to each of the weight score vectors and the convolution kernel library comprises: Using a preset Softmax function to normalize the weight score vector to obtain a normalized weight score vector; The normalized weight score vector is weighted and summed with the corresponding convolution kernel in the convolution kernel library to obtain the target adaptive convolution kernel.

7. A point cloud segmentation device based on adaptive graph convolution, characterized in that: The device comprises: An information acquisition module, used to acquire point cloud information corresponding to the target object, and determine the neighborhood edge points corresponding to each point cloud in the point cloud information; A convolution kernel library construction module is used to determine the adaptive convolution kernel Kernel corresponding to each point cloud based on the point cloud information, the neighborhood edge points corresponding to the point cloud, and a preset multi-layer perceptron model to construct a convolution kernel library; A weight coefficient determination module, used to determine the position offset between the point cloud position information in the point cloud information and the neighborhood edge point position information in the neighborhood edge point, and obtain the weight score vector corresponding to each convolution kernel in the convolution kernel library based on the position offset and a preset multi-layer perceptron model; The feature determination module is used to determine the enhanced edge features according to each of the weight score vectors and the convolution kernel library, and to aggregate each of the enhanced edge features to obtain a target output feature, and to use the target output feature as a result of point cloud segmentation.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the point cloud segmentation method based on adaptive graph convolution as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the point cloud segmentation method based on adaptive graph convolution according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the point cloud segmentation method based on adaptive graph convolution according to any one of claims 1 to 6.

Images