Graph neural network training method, point cloud feature extraction method, device and medium

Abstract

The application provides a graph neural network training method, a point cloud feature extraction method, equipment and a storage medium. The graph neural network is applied to point cloud feature extraction. The training method comprises the following steps: obtaining a to-be-trained point cloud, constructing graph structure data according to the to-be-trained point cloud, the graph structure data comprising a plurality of nodes and edges formed between the plurality of nodes; constructing a to-be-trained graph neural network based on a plurality of aggregation functions and weight parameters of each aggregation function; inputting the graph structure data into the to-be-trained graph neural network, obtaining an aggregation result of each aggregation function, fusing the aggregation result of each aggregation function according to the weight parameters of each aggregation function, and obtaining a prediction output of the to-be-trained graph neural network; and training the weight parameters of each aggregation function based on the prediction output until a final graph neural network is obtained. Through the above method, the network model architecture is searched for different scenes and data, and the scene analysis capability of the graph neural network model is improved.

Description

Technical Field

[0001] This application relates to the field of point cloud data processing, and in particular to a neural network training method, terminal, and medium for point cloud feature extraction. Background Technology

[0002] Compared to two-dimensional images, 3D point cloud information contains richer descriptive information, overcoming the limitations of single-view projection. 3D point cloud information contains abundant information about real objects, and extracting this information can address problems that are difficult to solve with traditional machine vision. Due to the massive amount of information in 3D point clouds, processing requires sampling and information extraction. The data features of point clouds are well-suited for analysis using Graph Neural Networks (GNNs).

[0003] Generative Neural Networks (GNNs) sample points from a point cloud as nodes in a graph, connect them according to rules, and learn the corresponding point cloud information by aggregating information from nodes and their neighbors. Current GNN-based point cloud learning methods, on the one hand, are prone to losing a large amount of useful information, and on the other hand, lack feature representations of point clouds for specific application scenarios. Summary of the Invention

[0004] This application provides a graph neural network training method, a point cloud feature extraction method, an apparatus, and a storage medium.

[0005] One technical solution adopted in this application is to provide a graph neural network training method for point cloud feature extraction, wherein the graph neural network training method includes:

[0006] Obtain the point cloud to be trained, and construct graph structure data based on the point cloud to be trained, wherein the graph structure data includes several nodes and edges formed between the several nodes;

[0007] A graph neural network to be trained is constructed based on several aggregation functions and the weight parameters of each aggregation function;

[0008] The graph structure data is input into the graph neural network to be trained, the aggregation result of each aggregation function is obtained, and the aggregation results of each aggregation function are fused according to the weight parameters of each aggregation function to obtain the prediction output of the graph neural network to be trained.

[0009] The weight parameters of each aggregation function are trained based on the predicted output until the final graph neural network is obtained.

[0010] The step of training the weight parameters of each aggregation function based on the predicted output until the final graph neural network is obtained includes:

[0011] The weight parameters of each aggregation function are trained based on the predicted output until the final combination of weight parameters is obtained.

[0012] Based on the weight parameter combination, the influence factor of each aggregation function is calculated;

[0013] The aggregation function with the largest influence factor is retained to form the final graph neural network.

[0014] The graph neural network to be trained consists of multiple aggregation layers, and each aggregation layer contains several aggregation functions connected in parallel.

[0015] The step of retaining the aggregation function with the largest influence factor to form the final graph neural network includes:

[0016] The aggregation function with the largest influence factor in each aggregation layer is retained as the unique aggregation function of that aggregation layer, and the trained multi-layer aggregation layers constitute the final graph neural network.

[0017] The aggregation function includes at least one of the following: mean aggregation function, max pooling aggregation function, local graph convolution aggregation function, attention aggregation function, high-dimensional aggregation function, and chained aggregation function.

[0018] The step of constructing graph structure data based on the point cloud to be trained includes:

[0019] All points in the point cloud to be trained are used as nodes in the graph structure data, wherein each node contains the three-dimensional coordinate information of the corresponding point;

[0020] Obtain the distance between any two nodes, and connect the edges between two nodes whose distance is less than a preset threshold.

[0021] The graph structure data is constructed based on all nodes and the edges that have been created.

[0022] The acquisition of the point cloud to be trained includes:

[0023] Obtain the original point cloud;

[0024] The original point cloud is divided into several point clouds to be trained by a preset clustering algorithm;

[0025] The step of fusing the aggregation results of each aggregation function according to the weight parameters of each aggregation function to obtain the prediction output of the graph neural network to be trained includes:

[0026] The aggregation results of each aggregation function are fused according to the weight parameters of each aggregation function to obtain the first predicted point cloud feature of each point cloud to be trained.

[0027] The first predicted point cloud features of all the point clouds to be trained are fused together to obtain the second predicted point cloud features of the original point cloud, which are used as the prediction output of the graph neural network to be trained.

[0028] The graph neural network training method further includes, after dividing the original point cloud into several point clouds to be trained using a preset clustering algorithm, the latter method further includes:

[0029] Select several starting points from the aforementioned point clouds to be trained, as well as the centroid of each point cloud to be trained;

[0030] Based on the distance between each starting point and the centroid of all point clouds to be trained, the secondary point cloud to be trained corresponding to each starting point is obtained.

[0031] The step of obtaining the secondary point cloud to be trained corresponding to each starting point based on the distance between each starting point and the centroids of all point clouds to be trained includes:

[0032] Based on the first distance between each starting point and the centroid of all the point clouds to be trained, obtain the first sampling point in the point cloud to be trained with the largest first distance, and form a sampling point set with the starting point and the first sampling point;

[0033] Calculate the second distance between the sampling point set and the centroids of all the point clouds to be trained, obtain the second sampling point in the point cloud to be trained with the largest second distance, add the second sampling point to the sampling point set, until the number of points in the sampling point set reaches a preset number, and use the final sampling point set as the secondary point cloud to be trained corresponding to the starting point.

[0034] Another technical solution adopted in this application is to provide a point cloud feature extraction method, which includes: acquiring a point cloud to be identified; inputting the point cloud to be identified into a pre-trained graph neural network to acquire target point cloud features of the point cloud to be identified; wherein, the graph neural network is trained by the above-mentioned graph neural network training method.

[0035] Another technical solution adopted in this application is to provide a terminal device, the terminal device including a processor and a memory connected to the processor, wherein the memory stores program instructions; the processor is used to execute the program instructions stored in the memory to implement the graph neural network training method and / or the point cloud feature extraction method described above.

[0036] Another technical solution adopted in this application is to provide a computer storage medium for storing program data, which, when executed by a computer, is used to implement the graph neural network training method and / or the point cloud feature extraction method described above.

[0037] The beneficial effects of this application are: it provides a graph neural network training method, which includes acquiring a point cloud to be trained; constructing graph structure data based on the point cloud, wherein the graph structure data includes several nodes and edges formed between the nodes; constructing a graph neural network to be trained based on several aggregation functions and the weight parameters of each aggregation function; inputting the graph structure data into the graph neural network to be trained, obtaining the aggregation result of each aggregation function, and fusing the aggregation results of each aggregation function according to the weight parameters of each aggregation function to obtain the prediction output of the graph neural network to be trained; training the weight parameters of each aggregation function based on the prediction output until the final graph neural network is obtained. This application retains more original point cloud information through graph sampling methods and finds the most suitable graph model for the current scene through differentiable graph neural network search technology, and performs farthest point sampling after dividing the point cloud region, increasing the applicability to point cloud information with uneven density. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating an embodiment of the graph neural network training method provided in this application;

[0040] Figure 2 yes Figure 1 A flowchart illustrating the sub-steps of S100;

[0041] Figure 3 yes Figure 2 A flowchart illustrating the steps following S120;

[0042] Figure 4 yes Figure 3 A flowchart illustrating the sub-steps of S140 in the middle section;

[0043] Figure 5 yes Figure 3 A flowchart illustrating the steps following S140;

[0044] Figure 6 yes Figure 1 A flowchart illustrating the sub-steps of the S300 process;

[0045] Figure 7 yes Figure 1 A flowchart illustrating the steps from S100 to S400.

[0046] Figure 8 yes Figure 1 A flowchart illustrating the sub-steps of the S400 system;

[0047] Figure 9 yes Figure 8 Flow diagram of S410-S430;

[0048] Figure 10 This is a flowchart illustrating an embodiment of the point cloud feature extraction method provided in this application.

[0049] Figure 11 This is a schematic flowchart of an embodiment of the 3D point cloud feature extraction method and application provided in this application.

[0050] Figure 12 This is a schematic diagram of the structure of an embodiment of the terminal device provided in this application;

[0051] Figure 13 This is a schematic diagram of the structure of an embodiment of the computer storage medium provided in this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0053] 3D point cloud data is a very important type of spatial geometric data. It is a non-hierarchical and irregular data form, generally used to construct the surface shape of three-dimensional geometric objects. At the same time, point cloud data is also the original representation of data generated by devices such as LiDAR, 3D sensors, and stereo scanners.

[0054] In recent years, deep learning technology has demonstrated significant advantages in fields such as image recognition and natural language processing, propelling end-to-end machine learning and pattern recognition technologies to new heights. Deep learning algorithms possess powerful feature extraction and representation capabilities, overcoming the cumbersome manual feature engineering preprocessing required in traditional machine learning. They automatically identify and extract latent feature information from training data and learn its patterns for modeling. However, traditional deep learning methods for 3D point cloud modeling fail to effectively capture the spatial geometric information between data points, and the modeling process is complex and consumes significant memory resources. Furthermore, 3D point cloud data belongs to the non-Euclidean domain and cannot be easily converted into tensor representations. Therefore, traditional deep learning algorithms have limitations in modeling 3D point cloud data, resulting in unsatisfactory processing performance.

[0055] Graphs, as a special type of data structure, possess rich application value. Graph structure data mainly consists of nodes and edges connecting the nodes. Nodes contain the attribute characteristics of the data, and edges represent the dependencies between nodes. For the effective representation of non-Euclidean domain data, graph structures demonstrate great applicability and application value. Therefore, point cloud data is suitable for parsing using Graph Neural Networks (GNNs).

[0056] GNNs typically sample some points from a point cloud as nodes in the graph and connect them according to rules. By aggregating information from nodes and their neighborhoods, they learn the corresponding point cloud features. However, if GNNs perform too many aggregations, some point cloud data may become blurred or be lost. Furthermore, different aggregation methods have a significant impact on the final representation of node features.

[0057] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the graph neural network training method provided in this application.

[0058] like Figure 1 As shown, the graph neural network training method of this application embodiment may include the following steps:

[0059] S100: Obtain the point cloud to be trained, and construct graph structure data based on the point cloud to be trained. The graph structure data includes several nodes and the edges formed between the nodes.

[0060] In one embodiment, a point cloud to be trained is obtained, each point in the point cloud to be trained is taken as a node in the graph structure data, and each node is connected to its nearest neighbor to form an edge, wherein the edge formed between nodes is a directed edge.

[0061] Specifically, each point in the point cloud to be trained and its neighboring points are constructed into several local directed graphs, realizing the transformation of unstructured point cloud data to be trained into structured graph structure data.

[0062] For details on how to obtain the training point cloud in S100, please refer to [link / reference needed]. Figure 2 , Figure 2 yes Figure 1 A flowchart illustrating the sub-steps of S100.

[0063] like Figure 2 As shown, the subsequent flowchart of S100 may include:

[0064] S110, acquire the original point cloud.

[0065] Specifically, raw point cloud data is obtained by automatically measuring the surface of an object using 3D scanning equipment, such as 2D / 3D LiDAR, stereo cameras, and time-of-flight cameras (TOFCamera).

[0066] S120 divides the original point cloud into several point clouds to be trained using a preset clustering algorithm.

[0067] In one embodiment, the original point cloud is voxelized using a uniformly sized voxel grid to obtain a downsampled point cloud. Specifically, voxelizing the original point cloud reduces redundant point cloud data while preserving the shape feature information of the point cloud. The voxel grid size in this embodiment is not limited.

[0068] Optionally, the i-th point in the downsampled point cloud is represented as P. i In one embodiment, a preset clustering algorithm is used to divide the downsampled point cloud into N regions, i.e., N training point clouds. The similarity of point features within each training point cloud is greater than the similarity of point features between training point clouds.

[0069] For post-processing methods involving dividing the original point cloud into several training point clouds, please refer to [link to relevant documentation]. Figure 3 , Figure 3 yes Figure 2 A flowchart illustrating the steps following S120.

[0070] like Figure 3 As shown, the steps following S120 may include:

[0071] S130, select several starting points from several point clouds to be trained, and the centroid of each point cloud to be trained.

[0072] Among them, the centroids {Q1, Q2, ..., Q} of the N point clouds to be trained are calculated. N}. And randomly select a point P from several point clouds to be trained. x As the starting point.

[0073] S140: Based on the distance between each starting point and the centroid of all point clouds to be trained, obtain the secondary point cloud to be trained corresponding to each starting point.

[0074] Specifically, after clustering point clouds, regions are divided, and sampling of the farthest point is achieved on a region-by-region basis. The resulting multiple sub-point clouds to be trained are processed to retain more original point cloud information and reduce sampling loss caused by the problem of denser points in the near and sparser points in the far during point cloud sampling.

[0075] For details on how to obtain the training sub-point cloud corresponding to the starting point, please refer to [link / reference]. Figure 4 , Figure 4 yes Figure 3 A flowchart illustrating the sub-steps of S140.

[0076] like Figure 4 As shown, the sub-step of S140 may include:

[0077] S141, based on the first distance between each starting point and the centroid of all the point clouds to be trained, obtain the first sampling point in the point cloud to be trained with the largest first distance, and form a sampling point set with the starting point and the first sampling point.

[0078] In one embodiment, the starting point P is calculated. x With the centroids {Q1, Q2, ..., Q} of N point clouds to be trained N Find the first distance between} and the starting point P. x The first training point cloud region with the furthest distance is selected, and a point is randomly sampled from this region as the first sampling point, which is then compared with the starting point P. x This constitutes the sampling point set A.

[0079] Optionally, the first distance can be the Euclidean distance between the starting point and the centroid of the point cloud to be trained.

[0080] S142, calculate the second distance between the sampling point set and the centroids of all the point clouds to be trained, obtain the second sampling point in the point cloud to be trained with the largest second distance, add the second sampling point to the sampling point set, until the number of points in the sampling point set reaches the preset number, and use the final sampling point set as the secondary point cloud to be trained corresponding to the starting point.

[0081] Among them, the centroids {Q1, Q2, ..., Q} of the sampling point set A and the training point cloud are calculated. N Find the second distance between the sampling point set A and the sampling point set A. Find the training point cloud that is farthest from the second distance between the sampling point set A and the sampling point set A. Randomly sample a point from the training point cloud and use it as the second sampling point. Add the second sampling point to the sampling point set A.

[0082] After N samplings, a set of N sampling points A is finally obtained, which is the secondary point cloud C1 to be trained.

[0083] Update the centroid of the training point cloud after removing the sampling points to {Q′1,Q′2,…,Q′} N}

[0084] Repeated sampling is performed on the secondary point cloud to be trained until the secondary point cloud to be trained reaches a preset number or there are no uncollected points in a certain area. The final set of sampled points is then used as the secondary point cloud to be trained corresponding to the starting point.

[0085] Optionally, the second distance between the sampled point set and the centroids of all the point clouds to be trained can be the Euclidean distance between the sampled point set and the centroids of all the point clouds to be trained.

[0086] By acquiring multiple training sub-point cloud data, information from weaker point cloud locations is preserved, improving the sampling capability for point cloud information with uneven density.

[0087] For details on how to construct graph structure data in S100, please refer to [link / reference]. Figure 5 , Figure 5 yes Figure 3 A flowchart illustrating the steps following S140.

[0088] like Figure 5 As shown, the steps following S140 may include:

[0089] S150 uses all points in the point cloud to be trained as nodes in the graph structure data, where each node contains the three-dimensional coordinate information of the corresponding point.

[0090] The point information constructed from each point cloud to be trained is added to the node set v of the graph structure data. For each individual node v in the node set... i It contains the three-dimensional coordinate information of the corresponding points in the point cloud to be trained.

[0091] S160: Obtain the distance between any two nodes and connect the edges between two nodes whose distance is less than a preset threshold.

[0092] Calculate the distance between any two nodes in the node set. If the distance between any two nodes is less than a preset threshold, then connect the two nodes and add the edge connecting the two nodes whose distance is less than the preset threshold to the edge set E of the graph structure data.

[0093] Optionally, the distance between any two nodes is the Euclidean distance between any two nodes.

[0094] Optionally, the edges e between nodes satisfy the following relationship:

[0095] e = {(v a ,v b )‖v a -v b || <r}

[0096] Where r is the connection distance threshold between two nodes.

[0097] Optionally, the edge e between nodes and the edge set E of the graph structure data satisfy the following relationship:

[0098] e∈E

[0099] S170, based on all nodes and the edges already created, constructs the graph structure data.

[0100] For the sampled point cloud set to be trained {C1, C2, ..., C...} nBased on the vertex set v and edge set E of the graph structure data corresponding to the point cloud set to be trained, construct the corresponding graph structure data.

[0101] Among them, graph structure data G i satisfy:

[0102] G i =(v,E)

[0103] S200 constructs a graph neural network to be trained based on several aggregation functions and the weight parameters of each aggregation function.

[0104] In one embodiment, a search space for a graph neural network is constructed, and different aggregation functions and weight parameters of each aggregation function are used to evaluate the structural features of the graph neural network. Aggregation functions that are not applicable to the current features are removed, and the aggregation function with the best effect is selected to achieve aggregation and update of graph node information.

[0105] Optionally, the aggregation function includes, but is not limited to, the following functions: mean aggregation function, max pooling aggregation function, local graph convolution aggregation function, attention aggregation function, high-dimensional aggregation function, and chained aggregation function.

[0106] In one embodiment, when using the mean aggregation function, the information of all neighboring nodes in the graph structure is fed into a single-layer perceptron, the mean is calculated, and then concatenated with the target node.

[0107] In one embodiment, when using the max pooling aggregation function, the information of all neighboring nodes in the graph structure is passed into a single-layer perceptron, the max pooling value is calculated, and then concatenated with the target node.

[0108] In one embodiment, when using the local graph convolution aggregation function, a fixed number of points are sampled from the neighborhood of each node in the graph structure, and after calculating and constructing the adjacency matrix, the local graph convolution operation is performed.

[0109] In one embodiment, when using the attention aggregation function, the information of all neighboring nodes in the graph structure is input into a single-layer perceptron and then the features of adjacent nodes are concatenated. The node weighting aggregation is performed through the attention mechanism.

[0110] In one embodiment, when using a high-dimensional aggregation function, a multilayer perceptron is used to map the node features in the graph structure to a high dimension, and the aggregation is completed by the node features in the neighborhood of the graph structure.

[0111] In one embodiment, when using a chained aggregation function, the neighboring nodes in the graph structure are randomly sorted and then input into a Long Short-Term Memory (LSTM) artificial neural network to aggregate the outputs of each node.

[0112] S300: Input the graph structure data into the graph neural network to be trained, obtain the aggregation result of each aggregation function, and fuse the aggregation results of each aggregation function according to the weight parameters of each aggregation function to obtain the prediction output of the graph neural network to be trained.

[0113] In one embodiment, by obtaining the aggregation result and weight coefficient of each aggregation function and fusing the aggregation results, a network structure suitable for the current task in the graph neural network to be trained is searched.

[0114] Please refer to the method for predicting the results of the graph neural network under training based on the aggregation function and its weight parameters for graph structure data. Figure 6 , Figure 6 yes Figure 1 A flowchart illustrating the sub-steps of the S300 process.

[0115] like Figure 6 As shown, S300 may include the following steps:

[0116] S310, the aggregation results of each aggregation function are fused according to the weight parameters of each aggregation function to obtain the first predicted point cloud feature of each point cloud to be trained.

[0117] The weight parameters of each aggregation function in the graph neural network to be trained are obtained, and the aggregation results of each aggregation function on the graph structure data are fused to update the graph node information of the graph neural network to be trained. The first predicted point cloud features of each point cloud to be trained include graph neural network features with representational capabilities.

[0118] Optionally, the aggregation function includes at least one of the following: mean aggregation function, max pooling aggregation function, local graph convolution aggregation function, attention aggregation function, high-dimensional aggregation function, and chained aggregation function.

[0119] In one embodiment, when using the mean aggregation function, the information of all neighboring nodes in the graph structure is fed into a single-layer perceptron, the mean is calculated, and then concatenated with the target node.

[0120] In one embodiment, when using the max pooling aggregation function, the information of all neighboring nodes in the graph structure is passed into a single-layer perceptron, the max pooling value is calculated, and then concatenated with the target node.

[0121] In one embodiment, when using the local graph convolution aggregation function, a fixed number of points are sampled from the neighborhood of each node in the graph structure, and after calculating and constructing the adjacency matrix, the local graph convolution operation is performed.

[0122] In one embodiment, when using the attention aggregation function, the information of all neighboring nodes in the graph structure is input into a single-layer perceptron and then the features of adjacent nodes are concatenated. The node weighting aggregation is performed through the attention mechanism.

[0123] In one embodiment, when using a high-dimensional aggregation function, a multilayer perceptron is used to map the node features in the graph structure to a high dimension, and the aggregation is completed by the node features in the neighborhood of the graph structure.

[0124] In one embodiment, when using a chained aggregation function, the neighboring nodes in the graph structure are randomly sorted and then input into a Long Short-Term Memory (LSTM) artificial neural network to aggregate the outputs of each node.

[0125] S320: The first predicted point cloud features of all the point clouds to be trained are fused to obtain the second predicted point cloud features of the original point cloud, which are used as the prediction output of the graph neural network to be trained.

[0126] Specifically, each point cloud to be trained contains some features of the original point cloud. By fusing all graph neural network information representing the point cloud to be trained, that is, fusing the first predicted point cloud features of all point clouds to be trained, the second predicted point cloud features of the original point cloud are obtained.

[0127] Specifically, the second predicted point cloud feature of the original point cloud is the graph neural network information that characterizes the original point cloud.

[0128] Please see Figure 7 , Figure 7 yes Figure 1 A flowchart illustrating the steps from S100 to S400. The steps from S100 to S400 include:

[0129] Region partitioning. The original point cloud is divided into several point clouds to be trained according to a preset clustering algorithm.

[0130] Farthest point sampling and graph construction. Downsample the point cloud to be trained to obtain a secondary point cloud containing the feature information of the point cloud to be trained. Construct a graph structure based on the set of points in the point cloud to be trained and the set of edges that meet the requirements.

[0131] Graph neural network multi-layer aggregation. Information is updated multiple times for nodes in a single layer of the graph, a suitable aggregation function is searched, and the optimal network structure is found layer by layer.

[0132] Point cloud feature extraction. Multiple point cloud sampling images to be trained are aggregated to obtain the features of each node in each sub-point cloud to be trained. The features of the sub-point clouds to be trained are then fused to obtain the features of the original point cloud.

[0133] Graph neural network prediction. In specific tasks, it is necessary to retrain the graph neural network and the weights of the post-processing layer. The resulting complete graph neural network model can be used for analysis and prediction of specific scenarios.

[0134] Please continue reading. Figure 1 , Figure 1 Following S300 of the graph neural network training method, it also includes:

[0135] S400 trains the weight parameters of each aggregation function based on the predicted output until the final graph neural network is obtained.

[0136] In one embodiment, under a specific task scenario, it is necessary to train the weight coefficients of each aggregation function in each layer, search for the optimal weight coefficients of all aggregation functions in that layer, and obtain the graph neural network structure most suitable for the current application scenario through layer-by-layer aggregation function search.

[0137] Optionally, the specific task scenario can be 3D point cloud segmentation.

[0138] Optionally, the specific task scenario can be classified as a 3D point cloud.

[0139] Optionally, the aggregation function includes at least one of the following: mean aggregation function, max pooling aggregation function, local graph convolution aggregation function, attention aggregation function, high-dimensional aggregation function, and chained aggregation function.

[0140] In one embodiment, when using the mean aggregation function, the information of all neighboring nodes in the graph structure is fed into a single-layer perceptron, the mean is calculated, and then concatenated with the target node.

[0141] In one embodiment, when using the max pooling aggregation function, the information of all neighboring nodes in the graph structure is passed into a single-layer perceptron, the max pooling value is calculated, and then concatenated with the target node.

[0142] In one embodiment, when using the local graph convolution aggregation function, a fixed number of points are sampled from the neighborhood of each node in the graph structure, and after calculating and constructing the adjacency matrix, the local graph convolution operation is performed.

[0143] In one embodiment, when using the attention aggregation function, the information of all neighboring nodes in the graph structure is input into a single-layer perceptron and then the features of adjacent nodes are concatenated. The node weighting aggregation is performed through the attention mechanism.

[0144] In one embodiment, when using a high-dimensional aggregation function, a multilayer perceptron is used to map the node features in the graph structure to a high dimension, and the aggregation is completed by the node features in the neighborhood of the graph structure.

[0145] In one embodiment, when using a chained aggregation function, the neighboring nodes in the graph structure are randomly sorted and then input into a Long Short-Term Memory (LSTM) artificial neural network to aggregate the outputs of each node.

[0146] For the detailed process of training the weight parameters of each aggregation function in the training graph neural network, please refer to [link to training instructions]. Figure 8 , Figure 8 yes Figure 1 A flowchart illustrating the sub-steps of the S400.

[0147] S410 trains the weight parameters of each aggregation function based on the predicted output until the final combination of weight parameters is obtained.

[0148] In one embodiment, if the graph neural network needs to aggregate N times, an aggregation function weight parameter training network with N layers is constructed. Each layer of the aggregation function weight parameter training network includes multiple branches of all aggregation functions. The multiple node features obtained from aggregation are arranged according to the search weights {α1, α2, ..., α...} n The weighted sums are then used to obtain the output of this layer.

[0149] Optionally, the aggregation function weight parameters can be used to train a network such as SuperNet.

[0150] S420 calculates the influence factor of each aggregation function based on the combination of weight parameters.

[0151] Among them, a normalization function is used to assign search weights {α1, α2, ..., α} to each layer. n The influence factor γ of each aggregation function is calculated by normalization.

[0152] Alternatively, the normalization function can be Softmax (normalization exponential function).

[0153] Optionally, the influence factor γ satisfies:

[0154]

[0155] S430 retains the aggregation function with the largest influence factor to form the final graph neural network.

[0156] In one embodiment, the aggregation function with the largest influence factor γ is selected as the retained structure of the graph neural network, while other aggregation functions are removed. By searching layer by layer for the optimal aggregation function, the graph neural network structure most suitable for the current task scenario is obtained.

[0157] Please see Figure 9 , Figure 9 yes Figure 8 A flowchart of the S410-S430 process.

[0158] In one embodiment, if the graph neural network needs to aggregate N times, then an aggregation function weight parameter training network with N hidden layers is constructed.

[0159] Each layer contains multiple function branches of all aggregation functions, and the aggregated node features are arranged according to search weights {α1, α2, ..., α...} n The weighted sums are used to obtain the output of this layer. A normalization function is applied to the search weights {α1, α2, ..., α} of each layer. n The influence factor γ of each aggregation function is calculated by normalization.

[0160] Each layer retains the aggregation function containing the largest influence factor γ, which serves as the retained structure of the graph neural network, gradually forming the final graph neural network.

[0161] In this embodiment, the graph neural network method includes acquiring a point cloud to be trained; constructing graph structure data based on the point cloud, wherein the graph structure data includes several nodes and edges between the nodes; constructing a graph neural network to be trained based on several aggregation functions and weight parameters of each aggregation function; inputting the graph structure data into the graph neural network to be trained, obtaining the aggregation result of each aggregation function, and fusing the aggregation results of each aggregation function according to the weight parameters of each aggregation function to obtain the prediction output of the graph neural network to be trained; training the weight parameters of each aggregation function based on the prediction output until the final graph neural network is obtained. This application retains more original point cloud information through graph sampling and finds the most suitable graph model for the current scene through differentiable graph neural network search technology. Furthermore, it performs farthest point sampling after dividing the point cloud region, increasing the applicability to point cloud information with uneven density.

[0162] The graph neural network trained in the above embodiments can be applied to feature extraction, feature recognition and feature prediction in various fields. For example, the trained graph neural network can be applied to target recognition in autonomous driving scenarios to accurately identify traffic elements such as obstacles, vehicles and pedestrians on the road.

[0163] Please refer to the details. Figure 10 , Figure 10 This is a flowchart illustrating an embodiment of the point cloud feature extraction method provided in this application.

[0164] like Figure 10 As shown, Figure 10 This may include the following steps:

[0165] S1, acquire the point cloud to be identified.

[0166] In one embodiment, 3D measurement equipment such as lidar or millimeter-wave radar is used to acquire point cloud information to be identified by scanning.

[0167] S2, input the point cloud to be identified into a pre-trained graph neural network to obtain the target point cloud features.

[0168] In one embodiment, the acquired point cloud information is input into a trained graph neural network, which extracts feature information from the point cloud data.

[0169] Optionally, after completing the point cloud data feature extraction step, the feature information of the point cloud data can be used to perform tasks including but not limited to: point cloud target recognition; and / or point cloud target tracking; and / or point cloud classification; and / or point cloud segmentation.

[0170] In one embodiment, the lidar on the autonomous vehicle acquires point cloud information of the surrounding environment, extracts point cloud feature information of the surrounding environment through the pre-trained graph neural network, and identifies, tracks, and segments target information such as vehicles, pedestrians, and non-motorized vehicles in the surrounding environment based on the point cloud feature information.

[0171] The above embodiments are merely one common example of this application and do not constitute any limitation on the technical scope of this application. Therefore, any minor modifications, equivalent changes, or alterations made to the above content based on the substance of the solution of this application shall still fall within the scope of the technical solution of this application.

[0172] Please see Figure 11 , Figure 11 This is a schematic flowchart of an embodiment of the 3D point cloud feature extraction method and application provided in this application.

[0173] The steps of an embodiment of a 3D point cloud feature extraction method and application may include:

[0174] Point cloud sampling. The target point cloud is voxelized using a uniformly sized voxel network while preserving the features of the point cloud.

[0175] Sparse sample graph construction. Construct graphs from the obtained point clouds, where the vertex information of the graph contains the 3D information of the corresponding points in the point cloud, and the edges of the graph satisfy the condition that the distance between vertices is less than a predetermined threshold.

[0176] Point cloud network model training includes constructing a search space and searching for a graph network architecture. The search space includes, but is not limited to, using the mean aggregation function, max pooling aggregation function, local graph convolution aggregation function, attention aggregation function, high-dimensional aggregation function, and chained aggregation function. The graph network architecture search finds the optimal optimization function layer by layer to find the graph neural network architecture most suitable for the current task, thus completing the training of the network model.

[0177] Point cloud feature extraction. The trained graph neural network model is used to aggregate multiple point cloud sampling images to obtain the feature values ​​of each node in each sub-point cloud to be trained. The sub-point cloud features are then fused to obtain the features of the original point cloud.

[0178] 3D vision application scenarios. In real-world environments, the trained graph networks can perform tasks including but not limited to: semantic segmentation, instance segmentation, object recognition, and object detection.

[0179] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of an embodiment of the terminal device provided in this application. The terminal device 500 of this application embodiment includes a processor 51 and a memory 52.

[0180] The memory 52 stores program data, and the processor 51 executes the program data to implement the graph neural network training method and / or point cloud feature extraction method described in the above embodiments.

[0181] In this embodiment, processor 51 can also be referred to as a CPU (Central Processing Unit). Processor 51 may be an integrated circuit chip with signal processing capabilities. Processor 51 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 51 can be any conventional processor.

[0182] This application also provides a computer storage medium; please refer to the following: Figure 13 , Figure 13 This is a schematic diagram of a computer storage medium according to an embodiment of the present application. The computer storage medium 600 stores program data 61, which, when executed by a processor, is used to implement the graph neural network training method and / or point cloud feature extraction method of the above embodiments.

[0183] When the embodiments of this application are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0184] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A graph neural network training method, wherein the graph neural network is applied to point cloud feature extraction, characterized in that, The graph neural network training method includes: Obtain the point cloud to be trained, and construct graph structure data based on the point cloud to be trained, wherein the graph structure data includes several nodes and edges formed between the several nodes; A graph neural network to be trained is constructed based on several aggregation functions and the weight parameters of each aggregation function; The graph structure data is input into the graph neural network to be trained, the aggregation result of each aggregation function is obtained, and the aggregation results of each aggregation function are fused according to the weight parameters of each aggregation function to obtain the prediction output of the graph neural network to be trained. The weight parameters of each aggregation function are trained based on the predicted output until the final graph neural network is obtained. The step of training the weight parameters of each aggregation function based on the predicted output until the final graph neural network is obtained includes: The weight parameters of each aggregation function are trained based on the predicted output until the final combination of weight parameters is obtained. Based on the weight parameter combination, the influence factor of each aggregation function is calculated; The aggregation function with the largest influence factor is retained to form the final graph neural network.

2. The graph neural network training method according to claim 1, characterized in that, The graph neural network to be trained consists of multiple aggregation layers, each of which contains several aggregation functions connected in parallel. The step of retaining the aggregation function with the largest influence factor to form the final graph neural network includes: The aggregation function with the largest influence factor in each aggregation layer is retained as the unique aggregation function of that aggregation layer, and the trained multi-layer aggregation layers constitute the final graph neural network.

3. The graph neural network training method according to claim 1 or 2, characterized in that, The aggregation function includes at least one of the following: mean aggregation function, max pooling aggregation function, local graph convolution aggregation function, attention aggregation function, high-dimensional aggregation function, and chained aggregation function.

4. The graph neural network training method according to claim 1, characterized in that, The step of constructing graph structure data based on the point cloud to be trained includes: All points in the point cloud to be trained are used as nodes in the graph structure data, wherein each node contains the three-dimensional coordinate information of the corresponding point; Obtain the distance between any two nodes, and connect the edges between two nodes whose distance is less than a preset threshold. The graph structure data is constructed based on all nodes and the edges that have been created.

5. The graph neural network training method according to claim 1, characterized in that, The acquisition of the point cloud to be trained includes: Obtain the original point cloud; The original point cloud is divided into several point clouds to be trained by a preset clustering algorithm; The step of fusing the aggregation results of each aggregation function according to the weight parameters of each aggregation function to obtain the prediction output of the graph neural network to be trained includes: The aggregation results of each aggregation function are fused according to the weight parameters of each aggregation function to obtain the first predicted point cloud feature of each point cloud to be trained. The first predicted point cloud features of all the point clouds to be trained are fused together to obtain the second predicted point cloud features of the original point cloud, which are used as the prediction output of the graph neural network to be trained.

6. The graph neural network training method according to claim 5, characterized in that, After dividing the original point cloud into several point clouds to be trained using a preset clustering algorithm, the graph neural network training method further includes: Select several starting points from the aforementioned point clouds to be trained, as well as the centroid of each point cloud to be trained; Based on the distance between each starting point and the centroid of all point clouds to be trained, the secondary point cloud to be trained corresponding to each starting point is obtained.

7. The graph neural network training method according to claim 6, characterized in that, The step of obtaining the secondary point cloud to be trained corresponding to each starting point based on the distance between each starting point and the centroids of all point clouds to be trained includes: Based on the first distance between each starting point and the centroid of all the point clouds to be trained, obtain the first sampling point in the point cloud to be trained with the largest first distance, and form a sampling point set with the starting point and the first sampling point; Calculate the second distance between the sampling point set and the centroids of all the point clouds to be trained, obtain the second sampling point in the point cloud to be trained with the largest second distance, add the second sampling point to the sampling point set, until the number of points in the sampling point set reaches a preset number, and use the final sampling point set as the secondary point cloud to be trained corresponding to the starting point.

8. A point cloud feature extraction method, characterized in that, The point cloud feature extraction method includes: Obtain the point cloud to be identified; The point cloud to be identified is input into a pre-trained graph neural network to extract the target point cloud features. The graph neural network is trained by the graph neural network training method according to any one of claims 1 to 7.

9. A terminal device, characterized in that, The terminal device includes a processor and a memory connected to the processor, wherein... The memory stores program instructions; The processor is used to execute program instructions stored in the memory to implement the graph neural network training method as described in any one of claims 1 to 7, and / or the point cloud feature extraction method as described in claim 8.

10. A computer-readable storage medium, characterized in that, The storage medium stores program instructions, which, when executed, implement the graph neural network training method as described in any one of claims 1 to 7, and / or the point cloud feature extraction method as described in claim 8.

Images