Point cloud feature extraction method, system and architecture based on residual natural attention
Through the point cloud feature extraction method based on residual natural attention, natural neighbor search and self-attention network are used to solve the problem of noise data error and neighborhood definition impact in point cloud feature extraction, efficient and robust feature extraction is achieved, and the accuracy of point cloud recognition and reconstruction is improved.
Patent Information
- Application Number
- CN202211180084.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-26
- Publication Date
- 2025-08-12
- Estimated Expiration
- 2042-09-26
AI Technical Summary
The existing point cloud feature extraction methods are prone to large errors when processing noisy data, and the impact of different neighborhood definitions on feature extraction results has not been fully explored. How to effectively define and utilize the local geometric structural characteristics of each data point and its neighbors in the point cloud is a challenge.
A point cloud feature extraction method based on residual natural attention is adopted. A feature extraction network is built by stacking multiple residual natural attention layers with offsets and multi-layer perception mechanisms in series. Local features are extracted using natural neighbor search and ball query methods, and feature aggregation and offset are combined with self-attention networks to reduce learning degradation and gradient disappearance during training.
Effectively extract local and global features of point clouds, reduce model calculations, avoid feature waste and overfitting, improve the accuracy of point cloud recognition, segmentation and reconstruction tasks, and enhance the robustness of noise data.
Smart Images
Figure CN115512201B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of point cloud feature extraction, and relates to a point cloud feature extraction method, system and architecture based on residual natural attention. Background Art
[0002] 3D data is typically obtained by scanning the visible surfaces of real-world objects using 3D sensors, which can capture richer geometric information, depth information, and scale information. Common 3D data formats include implicit surfaces, voxel grids, meshes, and point clouds.
[0003] A point cloud is a set of values with specified x, y, and z coordinates. Because it doesn't require a large number of data nodes to represent a three-dimensional model structure, it's easy and quick to obtain, and it takes up little storage space. It's widely used in fields as diverse as autonomous driving, robotics, drones, and the film and television industry. Unlike data with rasterized structures like audio and image signals, point clouds have a non-Euclidean data structure. The processing and extraction of local geometric relationships between data points within a point cloud, as well as the analysis and use of global geometric features representing the shape of the point cloud itself, form the foundation for subsequent application tasks such as point cloud recognition, keypoint detection, classification and segmentation, object detection, and 3D reconstruction. Therefore, for non-Euclidean structured data such as point clouds, the rational and efficient extraction of topological features from complex geometric structures that can be used for other tasks holds great research value in computer vision.
[0004] In existing research on point cloud feature extraction, neighbors, as a crucial component of the geometric structure of local point cloud features, have long been a crucial source of feature extraction data. To date, nearly all deep learning point cloud feature extraction methods involving neighbor search algorithms have defined neighbors as either k-nearest neighbors or sphere query neighbors.
[0005] Although different neighbor search algorithms will result in different scopes of neighborhood definition, the impact of these neighborhoods carrying different characteristics on the results of point cloud feature extraction is a question that has not been specifically explored. In addition, the features carried by different data points have different effects on the entire point cloud, and the features in different neighborhoods also have different effects on the overall point cloud features. How to define, select, focus on and utilize each data point and its neighbors in the point cloud, further extract the features of the local geometric structure based on these neighbors, and finally efficiently integrate them into the overall features of the point cloud is a challenging hot issue. In addition, the point cloud data collected in real life usually has noise. Directly using existing methods on these noisy data will produce large errors when using features for tasks such as point cloud recognition, segmentation, and reconstruction. Summary of the Invention
[0006] The purpose of the present invention is to provide a point cloud feature extraction method, system and architecture based on residual natural attention, which reduces learning degradation and gradient disappearance during training while ensuring feature extraction capability.
[0007] In order to achieve the above object, the basic scheme of the present invention is: a point cloud feature extraction method based on residual natural attention, comprising the following steps:
[0008] Input original 3D point cloud;
[0009] A feature extraction network model is constructed using a series of stacked residual natural attention layers with offsets and a transformation layer composed of multi-layer perceptrons.
[0010] The original 3D point cloud is input into the first offset residual natural attention layer of the feature extraction network model. The input of the i-th offset residual natural attention layer of the feature extraction network model is the output features of the layers before the i-th offset residual natural attention layer.
[0011] The output features of multiple residual natural attention layers with offsets are concatenated and input into the transformation layer. The transformation layer performs fitting transformation on the input features and finally obtains the point cloud features existing in the high-dimensional space.
[0012] The feature extraction network model is trained to learn the weight parameters of the network and obtain an optimized feature extraction network model.
[0013] The working principle and beneficial effects of this basic solution are: multiple offset residual natural attention layers are stacked in series, so that the latter layer in the feature extraction network model is directly connected to the previous layer, and the feature maps extracted from all the previous layers are reused. Without wasting the features of the existing calculation results, the overall feature dimension size of each layer in the feature extraction network model can also be designed to be relatively small, reducing redundant data.
[0014] The offset residual natural attention layer calculates offsets on features to capture relationship features between data points, allowing for more neighborhood features while maintaining the robustness of the original features. While the residual network structure reasonably increases network depth, it can also, to a certain extent, avoid network degradation during the learning process. Multiple offset residual natural attention layers are stacked using a dense network-like architecture, preserving and processing the features of each layer. This network structure reduces model computational complexity, avoids feature waste and redundancy, and effectively avoids overfitting. While maintaining feature extraction capabilities, it also reduces learning degradation and gradient vanishing during training.
[0015] Furthermore, the original three-dimensional point cloud is N in ×din Point cloud feature vector, where N in is the total number of data points in the point cloud data, d in is the dimension of the metric space where the input point cloud is located; in the three-dimensional metric space, each data point x in the point cloud data i All meet the requirements Represents three-dimensional space.
[0016] Set the original 3D point cloud vector to facilitate subsequent point cloud feature extraction.
[0017] Furthermore, the residual natural attention layer with offset is composed of a natural attention layer with a self-attention network as its core, a feature offset structure, and a residual structure.
[0018] The structure is simple and easy to use.
[0019] Furthermore, the method to obtain the output features of the natural attention layer is as follows:
[0020] The original 3D point cloud is N in ×d in Point cloud feature vector, where N in is the total number of data points in the point cloud data, d in is the dimension of the metric space where the input point cloud is located; the original 3D point cloud is input into the feature extraction network model;
[0021] Sampling the farthest point of the input vector, we get N out ×d in Point cloud feature vector, N out is the number of samples;
[0022] Using the ball query method and combining it with the natural neighbor search method, we search for the nearest neighbor of the input vector and get N out ×(λ+k)×d in The point cloud feature vector, λ, k are the number of neighbors obtained by natural neighbor and ball query search respectively;
[0023] N out ×d in The point cloud feature vector is repeated λ+k times and stacked to get N out ×(λ+k)×d in The new vector is the same as the N obtained by farthest point sampling and neighbor search. out ×(λ+k)×d in The point cloud feature vectors are subtracted according to a fixed dimension and then concatenated to obtain N out ×(λ+k)×2d in The point cloud feature vector of
[0024] Then it is sent to the self-attention network, and the self-attention network takes the data point’s own features f i Offset to neighbor features Parameter learning is performed as a new local feature of the data point itself, the features of all the neighbors of the data point are aggregated to the data point itself, and different data points are given discriminative attention according to different weights to extract and aggregate local features:
[0025]
[0026] Among them, f i Represents the data point x i Characteristics carried; x i,j Represents x i The jth neighbor of φ(·) and ψ(·) are multi-layer perceptron linear projection modules with different weights; ρ() represents the soft-max regularization operation; Θ(·) is a small positional encoding network with trainable parameters;
[0027] Finally, through the average pooling and maximum pooling operations, the main features of Φ(f) are retained and the parameters and calculation amount are reduced to obtain N out ×d out The point cloud feature vector, d out is the dimension of the output feature vector.
[0028] Furthermore, a natural neighbor search is performed on the input vector. The specific method is as follows:
[0029] For x i For a data set ∈X, the formal description of natural neighbors is as follows:
[0030] Among them, NN(x i ) is the point x i The natural neighbor set of λ (x i ) is the point x i The set of λ nearest neighbors, KNN λ (x j ) is the point x j The set of λ nearest neighbors; λ is the eigenvalue of the natural neighbor. When the natural neighbor search process reaches a stable state, the eigenvalue of the natural neighbor is the search round r, which is defined as follows:
[0031]
[0032] The search round r ranges from [1,λ]; the range of λ satisfies 2≤λ≤n, where n is a constant.
[0033] Define two auxiliary functions as follows:
[0034] knn k (x i )=findKNN(x i ,r,X), whose function is to find the data point x i Among the k nearest neighbors in the set X, to x i The index of the neighbor node with the closest Euclidean distance to r;
[0035] set.count(item), which counts the number of times the element item appears in the set;
[0036] Based on two auxiliary functions, all natural neighbor searches are performed. By defining and selecting the neighborhood, the features of the neighbors are aggregated to the central data point to obtain the local features of the central data point.
[0037] The method of parallelizing the calculation of natural neighbors is used to speed up the entire natural neighbor search.
[0038] The natural neighbor search algorithm is applied to point cloud feature extraction. Natural neighbor and sphere queries are used to search for the neighbors of each data point. Data points within these two types of neighborhoods are then used to extract and aggregate features within the local neighborhood. When using natural neighbors to construct a neighborhood graph, complex geometric structures are not connected to data points of different components, effectively preserving the local characteristics of each distinct topological structure.
[0039] By determining the number of natural neighbors a data point has, outliers can be quickly identified. These outliers are likely to be noise points in the noisy point cloud. In subsequent processing, noisy data points can be ignored or assigned a lower attention weight, thereby filtering out noise or reducing the level of trust in them.
[0040] Furthermore, the input of the self-attention network is the coarse local features of the point cloud feature vector Where N is the number of valid data points left after sampling at the farthest point; c is the number of attention channels, c = λ + k, which means that the width of the attention channel is the sum of the natural neighbor feature value and the limit on the number of nearest neighbors of the ball query; d in is the number of feature dimensions input to the self-attention network; the point cloud feature vector input to the self-attention network is linearly projected by the multi-layer perceptron to obtain Q (Query), K (Key) and V (Value). The rules are:
[0041]
[0042] in, These are all learnable parameters that exist in the multilayer perceptron;
[0043] At the same time, the self-attention network uses a multi-head attention mechanism to learn weights through multiple self-attention networks, and finally averages the attention of each head through an average pooling operation;
[0044] The final output is a unique attention feature matrix h of the neighborhood local features of each data point in the receptive field after point cloud downsampling. A , the process is formalized as follows:
[0045]
[0046] The number of attention heads, i.e., the number of self-attention networks, is h, and A is the abbreviation of Attention, which is integrated with h. Represents the transposition of the second and third dimensions of the high-dimensional matrix K; the multilayer perceptron includes a learnable parameter matrix In multi-layer perceptrons with different attention heads, the learnable parameter matrices are independent of each other and are not shared.
[0047] Using the attention mechanism and combining it with the natural neighbor search algorithm, the features carried by the data points in the neighborhood will be assigned different weights according to the attention mechanism, focusing on the parts with obvious geometric features to ensure the effectiveness of extracting local features of the point cloud.
[0048] Furthermore, the feature offset structure of the residual natural attention layer with offset is:
[0049] N out ×d out Perform matrix subtraction operation on the point cloud feature vector of the original point cloud feature to obtain the feature offset of the output feature of the natural attention layer and the original point cloud feature:
[0050]
[0051] Among them, f i and f j They are the local natural attention features after the natural attention layer and f i The characteristics of the data points in the corresponding origin cloud; is a function symbol representing the intermediate process, representing the fitting function; Θ=(θ1,θ2,…,θ M ), each θ m With vector (f j -f i ) have the same dimensions, θ m It is a multi-layer perceptron composed of M linear transformation layers, batch normalization layers and ReLU activation function layers.
[0052] Feature shifting can obtain more neighborhood features and maintain the robustness of the original features.
[0053] Furthermore, the residual structure of the residual natural attention layer with offset is:
[0054] The point cloud features after the natural attention layer and feature offset are The original input point cloud features will be used as the features f that directly participate in the residual calculation, and the function that finally passes through the residual network layer structure is Where f is the point cloud feature input to the residual natural attention layer with offset;
[0055] Get the residual natural attention layer to extract new features The formal expression of:
[0056]
[0057] Among them, f i It is the input of the residual natural attention layer with offset to represent the point cloud feature vector; and They are fitting objective functions with different learnable weights, corresponding to different fitting objective functions h in the natural attention layer and the residual natural attention layer with offset Φ With h Θ ; Θ and Φ are Θ=(θ1, θ2,…,θ D ) and Φ=(φ1,φ2,…,φ D ), encoding the weights of the multilayer perceptron composed of D different linear transformation layers, batch normalization operations and ReLU activation function layers. Each θ d and φ d Both have and f i The same dimension; finally, the vector representing the output feature is obtained Its dimension is the same as f i same.
[0058] Use learnable weights to train feature extraction network models and optimize model performance.
[0059] Furthermore, the multi-layer perceptron constituting the transformation layer is composed of a fully connected layer, a batch normalization layer, and an activation function layer.
[0060] The structure is simple and easy to use.
[0061] Furthermore, the calculation of stacking multiple residual natural attention layers with offsets in series is as follows:
[0062] x l =H l ([x1,x2,…,x l-1 ])
[0063] where X i Represents the feature map carried by the i-th layer; [x1,x2,…,x l-1 ] is the vector concatenated from the feature maps carried by layer 0 to layer l-1; H l is the learnable function to be fitted in the lth layer, which has the same dimension as the concatenated feature vector.
[0064] Multiple residual natural attention layers with offsets are stacked in series to fully consider the local geometric structure and features between each data point in the point cloud and its neighboring data points in its neighborhood, so as to extract the local neighborhood features of the point cloud data as much as possible.
[0065] The present invention also provides a point cloud feature extraction system based on residual natural attention, comprising a processing unit, which executes the method of the present invention to perform point cloud feature extraction.
[0066] The system extracts local features of data points based on the potential relationships between data points in the point cloud, ensuring the effectiveness of the extracted local features of the point cloud.
[0067] The present invention also provides a point cloud extraction network architecture based on residual natural attention, comprising at least four layers of residual natural attention layers with offsets, and a transformation layer composed of a multi-layer perceptron;
[0068] The offset residual natural attention layer is stacked in series and the output is connected to the transformation layer. The core of the offset residual natural attention layer is the natural attention layer. The natural attention layer includes a farthest point sampling module, a ball query module, a natural neighbor search module, a subtraction module, a repeated operation module, a splicing operation module, a high-dimensional attention module, an average pooling module and a maximum pooling operation module. The offset residual natural attention layer is used to extract the feature offset of the original point cloud feature and input it into the transformation layer. The transformation layer performs a fitting transformation on the feature offset of the original point cloud feature to obtain the point cloud feature existing in the high-dimensional quantitative space.
[0069] The point cloud extraction network architecture calculates feature offsets to capture the relationship features between data points. Using residual networks and dense networks, the network depth of the natural attention layer is stacked to more than four layers. This reduces learning degradation and vanishing gradients during training while maintaining feature extraction capabilities. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] Figure 1 Schematic diagram of the structure of the point cloud feature extraction method based on residual natural attention of the present invention;
[0071] Figure 2 2 is a schematic structural diagram of the natural attention layer of the point cloud feature extraction method based on residual natural attention of the present invention;
[0072] Figure 3 2 is a schematic structural diagram of the natural attention layer of the point cloud feature extraction method based on residual natural attention of the present invention;
[0073] Figure 4 It is a structural diagram of the residual natural attention layer with offset in the point cloud feature extraction method based on residual natural attention of the present invention. DETAILED DESCRIPTION
[0074] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.
[0075] In the description of the present invention, it should be understood that the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention.
[0076] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.
[0077] In order to better extract the features of point cloud data, explore the influence of neighborhood features on the overall features of point cloud, and improve the robustness of network models to noise, the present invention discloses a point cloud feature extraction method based on residual natural attention. This solution is based on the attention mechanism widely used in the field of natural language processing, combined with the natural neighbor search algorithm, and proposes a point cloud feature extraction network architecture to better extract the global geometric features and local relative features of point cloud data, and directly apply the features to point cloud classification and point cloud semantic segmentation tasks. A good feature extraction method also solves the problem of inaccurate classification results of noisy point clouds in point cloud classification tasks. Figure 1 As shown, this method includes the following steps:
[0078] Input the original 3D point cloud, the original 3D point cloud is N in ×din Point cloud feature vector, where N in is the total number of data points in the point cloud data, d in is the dimension of the metric space where the input point cloud is located; in the three-dimensional metric space, each data point x in the point cloud data i All meet the requirements Therefore, for the problem of using 3D point cloud data as the original input, d in =3.
[0079] The feature extraction network model is constructed using a series of stacked residual natural attention layers (RNALs) with offsets and a multilayer perceptron (MLP) consisting of a transformation layer (LBR, Linear + Batch Normalization + ReLU). The multilayer perceptron (MLP(·)) that constitutes the transformation layer consists of a fully-connected layer (FC), a batch normalization layer (BN), and an activation function layer. The activation function can be a Relu activation function.
[0080] Multiple residual natural attention layers with offsets are stacked in series, similar to a dense network architecture, and the RNAL of each layer is spliced with the output of all RNALs before it and used as input. Figure 4 As shown in , each RNAL contains a small network that has undergone feature offset and residual network. The core of this small network is the Natural Attention Layer (NAL), as shown in Figure 2 As shown in the figure, the residual natural attention layer with offsets consists of a natural attention layer with a self-attention network at its core, a feature offset structure, and a residual structure. Each time point cloud data passes through RNAL, the resulting point cloud features in a high-dimensional space are concatenated by LBR through a concatenation operation. This ensures that the output features of any RNAL layer are ultimately added to the overall features of a higher-dimensional space and then pass through a transformation layer composed of a multi-layer perceptron, using more learnable weight parameters to fit the final target feature extraction.
[0081] The original 3D point cloud is input into the first offset residual natural attention layer of the feature extraction network model. The input of the i-th offset residual natural attention layer of the feature extraction network model is the output features of the layers before the ii-th offset residual natural attention layer.
[0082] The output features of multiple residual natural attention layers with offsets are concatenated and input into the transformation layer. The transformation layer performs fitting transformation on the input features and finally obtains the point cloud features existing in the high-dimensional space.
[0083] The feature extraction network model is trained to learn the weight parameters of the network and obtain an optimized feature extraction network model.
[0084] In a preferred embodiment of the present invention, the feature transformation operation (BN-ReLU-Conv) of each layer takes into account all the features of the previous layer and propagates them to the next layer for calculation. The purpose of this is to further improve the information flow between layers and avoid the gradient vanishing phenomenon during training. Using a direct connection mode from any layer to all subsequent layers, the calculation of multiple residual natural attention layers with offsets stacked in series is as follows:
[0085] x l =H l ([x1,x2,…,x l-1 ])
[0086] where X i Represents the feature map carried by the i-th layer; [x1,x2,…,x l-1 ] is the vector concatenated from the feature maps carried by layer 0 to layer l-1; H l is the learnable function to be fitted in the lth layer, which has the same dimension as the concatenated feature vector.
[0087] Multiple residual natural attention layers with offsets are stacked in series to fully consider the local geometric structure and features between each data point in the point cloud and its neighboring data points in its neighborhood, so as to extract the local neighborhood features of the point cloud data as much as possible.
[0088] In a preferred embodiment of the present invention, the method for obtaining the output features of the natural attention layer is as follows:
[0089] The original 3D point cloud is N in ×d in The point cloud feature vector is obtained and the original 3D point cloud is input into the feature extraction network model;
[0090] Perform farthest point sampling (FPS, Farthest Point Sampling, Pointne++ method can be used) on the input vector to obtain N out ×d in Point cloud feature vector, N out is the number of samples;
[0091] Using the ball query method (BNN, Ball Nearest Neighbor), combined with the natural neighbor search method, the nearest neighbor of the input vector is searched to obtain N out ×(λ+k)×d in The point cloud feature vector, λ, k are the number of neighbors obtained by natural neighbor and ball query search respectively;
[0092] N out ×d in The point cloud feature vector is repeated λ+k times and stacked (R, Repeat, is a repetitive operation, which repeats the vector a fixed number of times and stacks it to form a new vector with one dimension higher than the original vector) to obtain N out ×(λ+k)×d in The new vector is the same as the N obtained by farthest point sampling and neighbor search. out ×(λ+k)×d in The point cloud feature vectors are subtracted in a fixed dimension (the + and - symbols in the figure represent matrix addition and subtraction respectively) and then concatenated (i.e., C, Concatenate, concatenation operation, concatenating two vectors in a fixed dimension to form a new vector with the same dimension as the original vector) to obtain N out ×(λ+k)×2d in The point cloud feature vector of
[0093] Then it is sent to the self-attention network, and the self-attention network takes the data point’s own features f i Offset to neighbor features Parameter learning is performed as a new local feature of the data point itself, the features of all the neighbors of the data point are aggregated to the data point itself, and different data points are given discriminative attention according to different weights to extract and aggregate local features:
[0094]
[0095] Among them, f i Represents the data point x i Characteristics carried; x i,j Represents x i The jth neighbor of φ(·) and ψ(·) are multi-layer perceptron linear projection modules with different weights; ρ() represents the soft-max regularization operation; Θ(·) is a small positional encoding network with trainable parameters;
[0096] Finally, through the average pooling operation (avg, average-pooling) and the maximum pooling operation (max, max-pooling), the main features of Φ(f) are retained and the parameters and calculation amount are reduced to obtain N out ×d out The point cloud feature vector, d out is the dimension of the output feature vector.
[0097] Perform a natural neighbor search on the input vector as follows:
[0098] For x i For a data set ∈X, the formal description of natural neighbors is as follows:
[0099] Among them, NN(x i ) is the point x i The natural neighbor set of λ (x i ) is the point x i The set of λ nearest neighbors, KNN λ (x j ) is the point x j The set of λ nearest neighbors; λ is the eigenvalue of the natural neighbor. When the natural neighbor search process reaches a stable state, the eigenvalue of the natural neighbor is the search round r, which is defined as follows:
[0100]
[0101] The search round r ranges from [1,λ]; the range of λ satisfies 2≤λ≤n, where n is a constant.
[0102] Define two auxiliary functions as follows:
[0103] knn k (x i )=findKNN(x i ,r,X), whose function is to find the data point x i Among the k nearest neighbors in the set X, to x i The index of the neighbor node with the closest Euclidean distance to r;
[0104] set.count(item), which counts the number of times the element item appears in the set;
[0105] Based on two auxiliary functions, a full natural neighbor search is performed. When using natural neighbors to construct a neighbor graph, complex geometric structures do not result in data points connecting different components, effectively preserving the local characteristics of each distinct component structure. As the natural neighbors of each data point are connected, a natural local neighborhood is created. By defining and selecting the neighborhood, the characteristics of the neighbors are aggregated to the central data point, resulting in a local feature of the central data point.
[0106] Because the natural neighbor search space is slightly larger, the basic natural neighbor search process takes a certain amount of time. In actual applications, it is found that the natural neighbor search process takes a long time, so the method of parallelizing the calculation of natural neighbors is used to speed up the entire natural neighbor search. According to the theorem of λ≤30, the findKNN(x i The ,r,X) function directly calculates the 30-nearest neighbors of all elements in the data point set X and saves this data as known data, which can be used as a query data source. This parallel processing process will accelerate as the number of elements in X increases, and its speedup is limited only by the maximum number of processes in the computer during parallel processing.
[0107] In the parallel calculation process of natural neighbors, the synchronization data needs to be added to the critical section for locking to avoid data inconsistency caused by parallel processing; before the entire search process begins, the 30-nearest neighbors of all data points in X are obtained and a neighbor index graph G is constructed. k , G k Element i in r,c Represents the data point x r The index number of the data point with the smallest Euclidean distance c among the 30 nearest neighbors in X. More specifically, OpenACC is used to parallelize and optimize the entire natural neighbor search process.
[0108] In order to make the natural neighbor indexes representable using tensors and embedding them into deep learning frameworks such as Pytorch or TensorFlow, it is necessary to find a way to tensorize the natural neighbor index relations represented by irregular data structures. This paper adopts a time complexity of O(λn 2 ) filling algorithm, fill the index into a regular index tensor, O(λn 2 ), where λ is the natural neighbor eigenvalue and n is the number of data points in the point cloud. This results in a padded natural neighbor index tensor of size n × λ, where the padded indices are the indices of the data points themselves. In other words, multiple self-loop edges are used for padding.
[0109] When calculating the new local features of the point cloud itself, this paper uses the offset between the data point's own features and the features of its neighbors as the new local features of the data point itself and provides them to the SAL (Self-Attention Layer) for parameter learning. The features of all the neighbors of the data point are aggregated to the data point itself, and different data points are given different attention according to different weights. The input of the 3D SAL is a coarse local feature composed of the natural neighbor graph, ball query and nearest neighbor. like Figure 3 As shown in the figure, S (SoftMax) is the normalized exponential function, and X is the matrix multiplication operation. The input of the self-attention network is the coarse local features of the point cloud feature vector Where N is the number of valid data points left after sampling at the farthest point; c is the number of attention channels, c = λ + k, which means that the width of the attention channel is the sum of the natural neighbor feature value and the limit on the number of nearest neighbors of the ball query; d in is the number of feature dimensions input to the self-attention network; the point cloud feature vector input to the self-attention network is linearly projected by the multi-layer perceptron to obtain Q (Query), K (Key) and V (Value). The rules are:
[0110]
[0111] in, These are all learnable parameters that exist in the multilayer perceptron;
[0112] At the same time, the self-attention network uses a multi-head attention mechanism (Multi-HeadAttention), which learns weights through multiple self-attention networks and finally averages the attention of each head through an average pooling operation;
[0113] The final output is a unique attention feature matrix h of the neighborhood local features of each data point in the receptive field after point cloud downsampling. A , the process is formalized as follows:
[0114]
[0115] Among them, the number of attention heads, that is, the number of self-attention networks, is h, h A is a symbol representing the attention feature matrix, A is the abbreviation of Attention, and it is integrated with h; Represents the transposition of the second and third dimensions of the high-dimensional matrix K; the multilayer perceptron includes a learnable parameter matrix In multi-layer perceptrons with different attention heads, the learnable parameter matrices are independent of each other and are not shared.
[0116] This technical solution applies the natural neighbor search algorithm to point cloud feature extraction: natural neighbors and ball queries are used to search for neighbors of each data point, and data points within these two types of neighborhoods are used to extract and aggregate features in the local neighborhood. When using natural neighbors to construct the neighborhood graph, parts of complex geometric structures do not generate connections between data points of different components, effectively preserving the local features of each different topological structure. Compared to using K-nearest neighbors, in natural neighbors, noisy data points only establish neighbor relationships with other noisy data points, and the features of valid data points are not aggregated to the noise, avoiding the problems faced by K-nearest neighbors.
[0117] By determining the number of natural neighbors a data point has, outliers can be quickly identified. These outliers are likely to be noise points in the noisy point cloud. In subsequent processing, noisy data points can be ignored or assigned a lower attention weight, thereby filtering out noise or reducing the level of trust in them.
[0118] Through average pooling (avg, average-pooling) and maximum pooling operations (max, max-pooling), the attention feature matrix h is retained A The main features of N are obtained by reducing parameters and calculation amount. out ×d out The point cloud feature vector. Average pooling and maximum pooling operations are also necessary operations to meet translation invariance, rotation invariance, and scale invariance in feature extraction of point cloud data.
[0119] In a preferred embodiment of the present invention, based on NAL and combined with residual and feature offset, the residual natural attention layer with offset proposed in this article is obtained. The point cloud features after NAL are further subjected to matrix subtraction operation with the original point cloud features to obtain a feature offset (Feature Offset) from the attention features to the original point cloud features; a shortcut operation is performed with the original point cloud to increase the depth of the overall network model while avoiding network degradation. LBR (Linear, BatchNormlization and ReLU) as a multi-layer perceptron represents the sequential connection of three operations; in order to further enhance the feature extraction capability of multi-layer stacked RNAL and avoid gradient vanishing or overfitting during training, the idea of dense network is also combined when stacking deep RNAL, taking all the features of each previous layer into account and propagating them to the next layer for calculation, reducing the amount of model calculation and avoiding feature waste and redundancy.
[0120] The feature offset structure of the residual natural attention layer with offset is:
[0121] N out ×dout Perform matrix subtraction operation on the point cloud feature vector of the original point cloud feature to obtain the feature offset of the output feature of the natural attention layer and the original point cloud feature:
[0122]
[0123] Among them, f i and f j They are the local natural attention features after the natural attention layer and f i The characteristics of the data points in the corresponding origin cloud; is a function symbol representing the intermediate process, representing the fitting function; Θ=(θ1,θ2,…,θ M ), each θ m With vector (f j -f i ) have the same dimensions, θ m The model is a multilayer perceptron consisting of M linear transformation layers, batch normalization layers, and ReLU activation function layers. Feature shifting can obtain more neighborhood features while maintaining the robustness of the original features.
[0124] In a preferred embodiment of the present invention, the residual structure of the residual natural attention layer with offset is:
[0125] The point cloud features after the natural attention layer and feature offset are The original input point cloud features will be used as the features f that directly participate in the residual calculation, and the function that finally passes through the residual network layer structure is Where f is the point cloud feature input to the residual natural attention layer with offset;
[0126] Get the residual natural attention layer to extract new features The formal expression of:
[0127]
[0128] Among them, f i It is the input of the residual natural attention layer with offset to represent the point cloud feature vector; and They are fitting objective functions with different learnable weights, corresponding to different fitting objective functions h in the natural attention layer and the residual natural attention layer with offset Φ With h Θ ; Θ and Φ are Θ=(θ1, θ2,…,θ D ) and Φ=(φ1,φ2,…,φ D ), encoding the weights of the multilayer perceptron composed of D different linear transformation layers, batch normalization operations and ReLU activation function layers. Each θ dand φ d Both have and f i The same dimension; finally, the vector representing the output feature is obtained Its dimension is the same as f i same.
[0129] The present invention also provides a residual natural attention-based point cloud feature extraction system, comprising a processing unit that executes the method described herein to extract point cloud features. The system extracts local features of data points based on their potential relationships within the point cloud, ensuring the effectiveness of the extracted local features.
[0130] The present invention also provides a point cloud extraction network architecture based on residual natural attention, including at least four layers of residual natural attention layers with offsets, and a transformation layer composed of a multi-layer perceptron.
[0131] The offset residual natural attention layer is stacked in series and the output is connected to the transformation layer. The core of the offset residual natural attention layer is the natural attention layer. The natural attention layer includes the farthest point sampling module, the ball query module, the natural neighbor search module, the subtraction module, the repeated operation module, the splicing operation module, the high-dimensional attention module, the average pooling module and the maximum pooling operation module. The offset residual natural attention layer is used to extract the feature offset of the original point cloud feature and input it into the transformation layer. The transformation layer performs a fitting transformation on the feature offset of the original point cloud feature to obtain the point cloud feature in the high-dimensional quantity space.
[0132] This paper uses an attention mechanism in the point cloud feature extraction process, combined with a natural neighbor search algorithm, to propose a natural attention layer. The attention mechanism assigns different weights to the features carried by the data points within the neighborhood, focusing on areas with prominent geometric features to ensure the effectiveness of extracting local features from the point cloud. The feature extraction and model design utilize the concepts of feature offset, residual networks, and dense networks.
[0133] Feature offset can obtain more neighborhood features while maintaining the robustness of the original features. The residual network structure can reasonably increase the network depth while also avoiding network degradation during the learning process to a certain extent. The dense network can retain the features of each layer and continue to process them. This network structure reduces the model's computational load, avoids feature waste and redundancy, and effectively avoids overfitting. The features are offset to obtain the relationship characteristics between data points. Using residual networks and dense networks, the network depth of the natural attention layer is stacked to more than 4 layers. While ensuring feature extraction capabilities, it reduces learning degradation and gradient disappearance during training.
[0134] To address the learning degradation or gradient vanishing problems that occur when the point cloud feature extraction network is too deep due to excessive stacking, the overall network architecture of the present invention combines the ideas of feature offset, residual network and dense network to solve various problems that may arise during deep network training, laying the foundation for deep point cloud feature extraction network models.
[0135] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
[0136] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.
Claims
1. A point cloud feature extraction method based on residual natural attention, characterized in that: The steps include: Input original 3D point cloud; A feature extraction network model is constructed using a series of stacked residual natural attention layers with offsets and a transformation layer composed of multi-layer perceptrons. The original three-dimensional point cloud is input into the first layer of the offset residual natural attention layer of the feature extraction network model. The input of the i-th layer of the offset residual natural attention layer of the feature extraction network model is the output features of the layers before the i-th layer of the offset residual natural attention layer. The offset residual natural attention layer consists of a natural attention layer with a self-attention network as the core, a feature offset structure, and a residual structure. The natural attention layer includes a farthest point sampling module, a ball query module, a natural neighbor search module, a subtraction module, a repeated operation module, a splicing operation module, a high-dimensional attention module, an average pooling module, and a maximum pooling operation module. The output features of multiple residual natural attention layers with offsets are concatenated and input into the transformation layer. The transformation layer performs fitting transformation on the input features and finally obtains the point cloud features existing in the high-dimensional space. The feature extraction network model is trained to learn the weight parameters of the network and obtain an optimized feature extraction network model.
2. The point cloud feature extraction method based on residual natural attention according to claim 1, characterized in that The original three-dimensional point cloud is N in ×d in Point cloud feature vector, where N in is the total number of data points in the point cloud data, d in is the dimension of the metric space where the input point cloud is located; in the three-dimensional metric space, each data point x in the point cloud data i All meet the requirements X={x1,x2,…,x n }, Represents three-dimensional space.
3. The point cloud feature extraction method based on residual natural attention according to claim 1, characterized in that The method to obtain the output features of the natural attention layer is as follows: The original 3D point cloud is N in ×d in Point cloud feature vector, where N in is the total number of data points in the point cloud data, d in is the dimension of the metric space where the input point cloud is located; the original 3D point cloud is input into the feature extraction network model; Sampling the farthest point of the input vector, we get N out ×d in Point cloud feature vector, N out is the number of samples; Using the ball query method and combining it with the natural neighbor search method, we search for the nearest neighbor of the input vector and get N out ×(λ+k)×d in The point cloud feature vector, λ, k are the number of neighbors obtained by natural neighbor and ball query search respectively; N out ×d in The point cloud feature vector is repeated λ+k times and stacked to get N out ×(λ+k)×d in The new vector is the same as the N obtained by farthest point sampling and neighbor search. out ×(λ+k)×d in The point cloud feature vectors are subtracted according to a fixed dimension and then concatenated to obtain N out ×(λ+k)×2d in The point cloud feature vector of Then it is sent to the self-attention network, and the self-attention network takes the data point’s own features f i Offset to neighbor features Parameter learning is performed as a new local feature of the data point itself, the features of all the neighbors of the data point are aggregated to the data point itself, and different data points are given discriminative attention according to different weights to extract and aggregate local features: Among them, f i Represents the data point x i Characteristics carried; x i,j Represents x i The jth neighbor of φ(·) and ψ(·) are multi-layer perceptron linear projection modules with different weights; ρ() represents the soft-max regularization operation; Θ(·) is a small positional encoding network with trainable parameters; Finally, through the average pooling and maximum pooling operations, the main features of Φ(f) are retained and the parameters and calculation amount are reduced to obtain N out ×d out The point cloud feature vector, d out is the dimension of the output feature vector.
4. The point cloud feature extraction method based on residual natural attention according to claim 3, characterized in that Perform a natural neighbor search on the input vector as follows: For x i For a data set ∈X, the formal description of natural neighbors is as follows: Among them, NN(x i ) is the point x i The natural neighbor set of λ (x i ) is the point x i The set of λ nearest neighbors, KNN λ (x j ) is the point x j The set of λ nearest neighbors; λ is the eigenvalue of the natural neighbor. When the natural neighbor search process reaches a stable state, the eigenvalue of the natural neighbor is the search round r, which is defined as follows: The search round r ranges from [1,λ]; the range of λ satisfies 2≤λ≤n, where n is a constant. Define two auxiliary functions as follows: knn k (x i )=findKNN(x i ,r,X), whose function is to find the data point x i Among the k nearest neighbors in the set X, to x i The index of the neighbor node with the closest Euclidean distance to r; set.count(item), which counts the number of times the element item appears in the set; Based on two auxiliary functions, all natural neighbor searches are performed. By defining and selecting the neighborhood, the features of the neighbors are aggregated to the central data point to obtain the local features of the central data point. The method of parallelizing the calculation of natural neighbors is used to speed up the entire natural neighbor search.
5. The point cloud feature extraction method based on residual natural attention according to claim 3, characterized in that The input of the self-attention network is the coarse local features of the point cloud feature vector Where N is the number of valid data points left after sampling at the farthest point; c is the number of attention channels, c = λ + k, which means that the width of the attention channel is the sum of the natural neighbor feature value and the limit on the number of nearest neighbors of the ball query; d in is the number of feature dimensions input to the self-attention network; the point cloud feature vector input to the self-attention network is linearly projected by the multi-layer perceptron to obtain Q (Query), K (Key) and V (Value). The rules are: in, These are all learnable parameters that exist in the multilayer perceptron; At the same time, the self-attention network uses a multi-head attention mechanism to learn weights through multiple self-attention networks, and finally averages the attention of each head through an average pooling operation; The final output is a unique attention feature matrix h of the neighborhood local features of each data point in the receptive field after point cloud downsampling. A , the process is formalized as follows: The number of attention heads, i.e., the number of self-attention networks, is h, and A is the abbreviation of Attention, which is integrated with h. Represents the transposition of the second and third dimensions of the high-dimensional matrix K; the multilayer perceptron includes a learnable parameter matrix In multi-layer perceptrons with different attention heads, the learnable parameter matrices are independent of each other and are not shared.
6. The point cloud feature extraction method based on residual natural attention according to claim 1, characterized in that The feature offset structure of the residual natural attention layer with offset is: N out ×d out Perform matrix subtraction on the point cloud feature vector of the original point cloud feature to obtain the feature offset of the output feature of the natural attention layer and the original point cloud feature: Among them, f i and f j They are the local natural attention features after the natural attention layer and f i The characteristics of the data points in the corresponding origin cloud; is a function symbol representing the intermediate process, representing the fitting function; Θ=(θ1,θ2,…,θ M ), each θ m With vector (f j -f i ) have the same dimensions, θ m It is a multi-layer perceptron composed of M linear transformation layers, batch normalization layers and ReLU activation function layers.
7. The point cloud feature extraction method based on residual natural attention according to claim 1, characterized in that The residual structure of the residual natural attention layer with offset is: The point cloud features after the natural attention layer and feature offset are The original input point cloud features will be used as the features f that directly participate in the residual calculation, and the function that finally passes through the residual network layer structure is Where f is the point cloud feature input to the residual natural attention layer with offset; Get the residual natural attention layer to extract new features The formal expression of: Among them, f i It is the input of the residual natural attention layer with offset to represent the point cloud feature vector; and They are fitting objective functions with different learnable weights, corresponding to different fitting objective functions h in the natural attention layer and the residual natural attention layer with offset Φ With h Θ ; Θ and Φ are Θ=(θ1, θ2,…,θ D ) and Φ=(φ1,φ2,…,φ D ), encoding the weights of the multilayer perceptron composed of D different linear transformation layers, batch normalization operations and ReLU activation function layers. Each θ d and φ d Both have and f i The same dimension; finally, the vector representing the output feature is obtained Its dimension is the same as f i same.
8. The point cloud feature extraction method based on residual natural attention according to claim 1, characterized in that The multi-layer perceptron that constitutes the transformation layer consists of a fully connected layer, a batch normalization layer, and an activation function layer.
9. The point cloud feature extraction method based on residual natural attention according to claim 1, characterized in that The calculation of stacking multiple residual natural attention layers with offsets in series is as follows: where X i Represents the feature map carried by the i-th layer; From level 0 to level A vector formed by concatenating the feature maps carried by the layer; It is The function to be fitted that the layer can learn has the same dimension as the concatenated feature vector.
10. A point cloud feature extraction system based on residual natural attention, characterized in that: The method comprises a processing unit, wherein the processing unit executes the method according to any one of claims 1 to 9 to perform point cloud feature extraction.