Point cloud data processing method based on point cloud Transform network

Abstract

The invention discloses a point cloud data processing method based on a point cloud Transform network, and the method comprises the steps: carrying out the local sampling and normalization processing of point cloud data, and generating an enhanced point cloud feature sequence; then, the data are input into a point cloud Transform network into which an adapter module is embedded; gating vectors generated by geometric conditions are introduced into an adapter module, and cross-layer collaborative updating is executed among different Transform coding layers by using cross-layer memory vectors, so that deep modulation of features and continuous transmission of interlayer information are realized; and finally, decoding the final point feature sequence through a downstream task head to output a classification or segmentation result. According to the method, the fixed trunk parameter set does not participate in updating in the training stage, only the adapter and task head parameters are updated, the training cost is reduced, and meanwhile the precision of the point cloud task is effectively improved.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent processing of 3D point clouds and deep learning, and in particular to a point cloud data processing method based on point cloud Transformer networks, which can be applied to various 3D scenarios including industrial 3D inspection, autonomous driving environmental perception, and 3D point cloud recognition in orthopedic surgery. Background Technology

[0002] Point cloud data is a digital representation of the shape and structure of objects in three-dimensional space, and it is widely used in fields such as autonomous driving, robot vision, geographic information systems, industrial inspection, and orthopedic surgical navigation. However, point cloud data is characterized by sparsity, irregularity, and heterogeneity, which poses many challenges to traditional deep learning network models.

[0003] Existing point cloud processing methods typically rely on voxelization or meshing to transform point cloud data, which often results in the loss of the original geometric information of the point cloud. Although some methods use graph-based models to process point clouds, these methods still face problems such as huge computational costs and model complexity due to the sparsity of point clouds.

[0004] In recent years, Transformer models have achieved significant success in natural language processing and computer vision tasks. Their self-attention mechanism, by capturing long-range dependencies, has also shown potential in processing point cloud data. However, standard Transformer models suffer from the following problems when processing point clouds: a lack of effective geometric information and topological modeling makes it difficult to effectively identify local shapes and global structures within point clouds. Furthermore, standard Transformers typically rely on fully connected self-attention mechanisms, which tend to overlook local details and overall layout of point clouds when processing sparse point cloud data, thus reducing the model's classification and recognition performance.

[0005] To improve point cloud processing performance, researchers have proposed several methods, including Fourier transform coding, geometric level adapters, and adaptive conditional modulation. However, these methods still fall short of fully addressing the performance limitations of point clouds in complex scenes, particularly when dealing with regions of varying scales and geometric complexities. Therefore, introducing multi-layer adapters that adapt to the point cloud structure into the Transformer model, enabling the network to better capture local details, global structure, and cross-layer information, remains a critical technical challenge in the field of point cloud processing. Summary of the Invention

[0006] The purpose of this invention is to provide a point cloud data processing method based on point cloud Transformer networks. By introducing multi-layer adapter modules into the point cloud Transformer network, the geometric structure, topological information, sample-level conditions and other features of the point cloud are explicitly modeled. This improves network performance while avoiding excessive increase in computational complexity, and can effectively improve the accuracy and robustness of point cloud classification, recognition and other tasks.

[0007] To achieve the above objectives, the technical solution provided by this invention is: a point cloud data processing method based on a point cloud Transformer network, comprising the following steps:

[0008] S1: Acquire point cloud data, perform local sampling, rotation transformation and scale normalization on the point cloud data, and generate an enhanced point cloud feature sequence for deep learning modeling;

[0009] S2: Input the enhanced point cloud feature sequence into the point cloud Transformer network. The point cloud Transformer network consists of multiple Transformer coding layers connected in sequence, and an adapter module is embedded in each Transformer coding layer, so that the adapter modules in each Transformer coding layer together constitute a multi-layer adapter module.

[0010] S3: In the current Transformer coding layer, the point feature sequence from the previous Transformer coding layer is processed by layer normalization and then multi-head self-attention calculation is performed. The attention fusion point feature sequence of the Transformer coding layer is obtained by residual fusion.

[0011] S4: Input the attention fusion point feature sequence into the corresponding adapter module in the current Transformer coding layer, and sequentially perform ascending projection, nonlinear activation, feature fusion and dimensionality reduction projection to perform feature modulation on the attention fusion point feature sequence, and obtain the adaptation enhancement point feature sequence through residual fusion;

[0012] S5: Input the adapted and enhanced point feature sequence into the feedforward network of the current Transformer coding layer for nonlinear mapping, and obtain the coding layer output point feature sequence of the Transformer coding layer through residual fusion; use the coding layer output point feature sequence as the point feature sequence of the next Transformer coding layer, and repeat steps S3 to S5 until the final point feature sequence of the last Transformer coding layer is obtained.

[0013] S6: Input the final point feature sequence into the downstream task head for feature decoding, and generate point cloud classification results or point cloud segmentation results as point cloud data processing results output.

[0014] Furthermore, in step S1, when performing rotation transformation and scale normalization on the point cloud data, the first... The coordinates of the points are as follows The geometric center of the point cloud is denoted as The rotation matrix is ​​denoted as The scaling factor is denoted as And the point coordinates are standardized as follows:

[0015] ;

[0016] In the formula, The maximum radius of the rotated point cloud is used to ensure that the standardized point coordinates fall within a uniform scale range; subsequently, the... Standardized point coordinates of each point By combining and mapping the attribute features of points, an enhanced point cloud feature sequence is formed. ,in, This indicates the total number of points contained in the point cloud data, in the sequence. Indicates the first Enhanced feature vectors of points, The range of values ​​is to .

[0017] Furthermore, in step S2, the point cloud Transformer network is composed of... The system consists of several sequentially connected Transformer coding layers. Within each Transformer coding layer, intra-layer feature updates are performed in the order of "multi-head self-attention update - adapter module modulation - feedforward network update". Residual fusion is used at the multi-head self-attention update, adapter module modulation, and feedforward network update points to ensure that the enhanced point cloud feature sequence is progressively passed between each Transformer coding layer to obtain the final point feature sequence. .

[0018] Furthermore, in step S3, an attention fusion point feature sequence is generated. In multi-head self-attention calculations, a relative position bias matrix generated by the relative displacement between any two points in the point cloud is introduced. , making the first The attention weights of each Transformer coding layer are superimposed on the relative position bias matrix during calculation. This ensures that the single-head attention output satisfies:

[0019] ;

[0020] In the formula, This indicates that the point-pair relevance scores are exponentially normalized to obtain the attention weight matrix, which is row-normalized and has non-negative elements. This indicates that the attention weight matrix is ​​based on the pair The attention output obtained by weighted summation , , These are the query matrix, key matrix, and value matrix, respectively. The feature dimension is single-headed; then the attention output is fused with the residual of the point feature sequence input to the current Transformer coding layer to obtain the attention fused point feature sequence. .

[0021] Furthermore, in step S4, the adapter module performs a process on the first feature in the attention fusion point sequence. Feature vectors of attention fusion points The first step is to perform rising projection and nonlinear activation to obtain the second step. The intermediate feature vector of the adapter and based on the first Geometric condition vectors Generate the first gate vectors For the first The intermediate feature vector of the adapter With the first Feature vectors of attention fusion points Perform element-wise weighted fusion to obtain the first fusion feature vectors , is represented as:

[0022] ;

[0023] In the formula, This represents element-wise multiplication. Indicates and A vector of all 1s with the same dimension; then the first... fusion feature vectors The dimensionality reduction projection yields the first Each adapter output vector ; will the first Each adapter output vector By scaling factor With the Feature vectors of attention fusion points Add the residuals to obtain the first... Each adaptation enhancement point feature vector , is represented as:

[0024] ;

[0025] From all Constructing the feature sequence of the adaptation enhancement point .

[0026] Furthermore, in step S5, the multi-layer adapter module performs cross-layer collaborative updates between different Transformer coding layers, for the first... Layer adaptation enhancement point feature sequence Hierarchical aggregation yields hierarchical vectors. , is represented as:

[0027] ;

[0028] And based on the hierarchical vector Exponential recursion is used to form a cross-layer memory vector. , is represented as:

[0029] ;

[0030] In the formula, For memory update coefficients; subsequently, in the... In the layer adapter module, the cross-layer memory vector As one of the input conditions for gating generation, and the first Layer Geometric condition vector of points They work together to generate the gated vector and make the gated vector Participating in the Layer-to-adapter intermediate feature vector Feature vectors fused with attention points The fusion process is used to obtain the fused feature vector. Thus, the final point feature sequence Maintain the continuity of cross-layer information during progressive updates between layers.

[0031] Furthermore, in step S6, the downstream task head receives the final point feature sequence. It also outputs the prediction results, where in the point cloud classification task, the classification prediction results are output by the downstream task head. ,in, This represents the mapping function of the downstream task head; in the point cloud segmentation task, the downstream task head maps the final point feature sequence to the first... Feature vector of points Output point-by-point prediction results Furthermore, during the training phase, the backbone parameter set of the Transformer encoding layer in the point cloud Transformer network is fixed. It does not participate in the update; it only updates the parameter set of the multi-layer adapter module. With downstream task header parameter set Training is completed by minimizing the task loss function, so that the optimization objective satisfies:

[0032] ;

[0033] In the formula, The mapping function for the downstream task header, denoted as [function name] in the point cloud classification task. In the point cloud segmentation task, it is denoted as ; For monitoring labels, This is the task loss function.

[0034] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0035] 1. Controllable parameter increment and lower training cost: By embedding adapter modules in each Transformer encoding layer, task adaptation is mainly focused on the parameter updates of the adapter module and the downstream task head, which in most cases helps to reduce the computation and storage overhead caused by large-scale fine-tuning of the backbone network.

[0036] 2. Maintain stability of the main feature flow and make the adaptation process more robust: The adapter module output is superimposed back to the attention fusion point feature by using a scaling residual fusion method, so that the magnitude of the adaptation enhancement is controllable. This allows feature modulation to be completed layer by layer without destroying the original feature expression, reducing the risk of feature drift during training.

[0037] 3. Geometric condition-driven gated fusion is more suitable for point cloud structures: Geometric conditions are introduced into the adapter module to generate gated vectors and perform gated fusion of intermediate features and input features. This makes it easier for the network to take into account both local geometric details and global semantic consistency in most point cloud scenarios, and improves its adaptability to different data distributions.

[0038] 4. Cross-layer collaboration enhances semantic progression consistency: Through hierarchical convergence and cross-layer memory recursion, adaptation enhancement information of different Transformer coding layers is continuously transmitted, making the semantic updates of deep and shallow layers more coherent, thereby helping to reduce the expression inconsistency problem caused by the fragmentation of information between layers.

[0039] 5. Versatile structure, easy engineering integration and migration: The adapter module is embedded in the Transformer encoding layer in a pluggable manner, and can be used in combination with different point cloud Transformer backbones and different downstream task heads. In most cases, it helps to reuse and migrate in tasks such as point cloud classification and point cloud segmentation, as well as in application scenarios including medical 3D point clouds. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention.

[0041] Figure 2 This is an internal flowchart of a single Transformer encoding layer.

[0042] Figure 3 This is a flowchart of the cross-layer collaborative update process of the present invention. Detailed Implementation

[0043] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0044] like Figures 1 to 3 As shown in the figure, this embodiment discloses a point cloud data processing method based on a point cloud Transformer network, the specific details of which are as follows:

[0045] S1: Acquire point cloud data, perform local sampling, rotation transformation, and scale normalization on the point cloud data to generate an enhanced point cloud feature sequence for deep learning modeling; wherein, when performing rotation transformation and scale normalization on the point cloud data, the first... The coordinates of the points are as follows The geometric center of the point cloud is denoted as The rotation matrix is ​​denoted as The scaling factor is denoted as And the point coordinates are standardized as follows:

[0046] ;

[0047] In the formula, The maximum radius of the rotated point cloud is used to ensure that the standardized point coordinates fall within a uniform scale range; subsequently, the... Standardized point coordinates of each point By combining and mapping the attribute features of points, an enhanced point cloud feature sequence is formed. ,in, This indicates the total number of points contained in the point cloud data, in the sequence. Indicates the first Enhanced feature vectors of points, The range of values ​​is to .

[0048] S2: Input the enhanced point cloud feature sequence into the point cloud Transformer network. The point cloud Transformer network consists of multiple Transformer coding layers connected in sequence, and an adapter module is embedded in each Transformer coding layer, so that the adapter modules in each Transformer coding layer together constitute a multi-layer adapter module.

[0049] Specifically, the point cloud Transformer network is composed of The system consists of several sequentially connected Transformer coding layers. Within each Transformer coding layer, intra-layer feature updates are performed in the order of "multi-head self-attention update - adapter module modulation - feedforward network update". Residual fusion is used at the multi-head self-attention update, adapter module modulation, and feedforward network update points to ensure that the enhanced point cloud feature sequence is progressively passed between each Transformer coding layer to obtain the final point feature sequence. .

[0050] S3: In the current Transformer coding layer, the point feature sequence from the previous Transformer coding layer is processed by layer normalization and then multi-head self-attention calculation is performed. The attention fusion point feature sequence of the Transformer coding layer is obtained by residual fusion.

[0051] Generate attention fusion point feature sequence In multi-head self-attention calculations, a relative position bias matrix generated by the relative displacement between any two points in the point cloud is introduced. , making the first The attention weights of each Transformer coding layer are superimposed on the relative position bias matrix during calculation. This ensures that the single-head attention output satisfies:

[0052] ;

[0053] In the formula, This indicates that the point-pair relevance scores are exponentially normalized to obtain the attention weight matrix, which is row-normalized and has non-negative elements. This indicates that the attention weight matrix is ​​based on the pair The attention output obtained by weighted summation , , These are the query matrix, key matrix, and value matrix, respectively. The feature dimension is single-headed; then the attention output is fused with the residual of the point feature sequence input to the current Transformer coding layer to obtain the attention fused point feature sequence. .

[0054] S4: Input the attention fusion point feature sequence into the corresponding adapter module in the current Transformer coding layer, and sequentially perform ascending projection, nonlinear activation, feature fusion, and dimensionality reduction projection to modulate the attention fusion point feature sequence. Then, obtain the adaptation enhancement point feature sequence through residual fusion, as follows:

[0055] The adapter module selects the first feature sequence of attention fusion points. Feature vectors of attention fusion points The first step is to perform rising projection and nonlinear activation to obtain the second step. The intermediate feature vector of the adapter and based on the first Geometric condition vectors Generate the first gate vectors For the first The intermediate feature vector of the adapter With the first Feature vectors of attention fusion points Perform element-wise weighted fusion to obtain the first fusion feature vectors , is represented as:

[0056] ;

[0057] In the formula, This represents element-wise multiplication. Indicates and A vector of all 1s with the same dimension; then the first... fusion feature vectors The dimensionality reduction projection yields the first Each adapter output vector ; will the first Each adapter output vector By scaling factor With the Feature vectors of attention fusion points Add the residuals to obtain the first... Each adaptation enhancement point feature vector , is represented as:

[0058] ;

[0059] From all Constructing the feature sequence of the adaptation enhancement point .

[0060] S5: Input the adapted and enhanced point feature sequence into the feedforward network of the current Transformer coding layer for nonlinear mapping, and obtain the output point feature sequence of the Transformer coding layer through residual fusion; use the output point feature sequence of the coding layer as the point feature sequence of the next Transformer coding layer, and repeat steps S3 to S5 until the final point feature sequence of the last Transformer coding layer is obtained; specifically as follows:

[0061] The multi-layer adapter module performs cross-layer collaborative updates between different Transformer coding layers, for the first... Layer adaptation enhancement point feature sequence Hierarchical aggregation yields hierarchical vectors. , is represented as:

[0062] ;

[0063] And based on the hierarchical vector Exponential recursion is used to form a cross-layer memory vector. , is represented as:

[0064] ;

[0065] In the formula, For memory update coefficients; subsequently, in the... In the layer adapter module, the cross-layer memory vector As one of the input conditions for gating generation, and the first Layer Geometric condition vector of points They work together to generate the gated vector and make the gated vector Participating in the Layer-to-adapter intermediate feature vector Feature vectors fused with attention points The fusion process is used to obtain the fused feature vector. Thus, the final point feature sequence Maintain the continuity of cross-layer information during progressive updates between layers.

[0066] S6: Input the final point feature sequence into the downstream task head for feature decoding, and generate point cloud classification results or point cloud segmentation results as the output of point cloud data processing results, as follows:

[0067] The downstream task head receives the final point feature sequence. It also outputs the prediction results, where in the point cloud classification task, the classification prediction results are output by the downstream task head. ,in, This represents the mapping function of the downstream task head; in the point cloud segmentation task, the downstream task head maps the final point feature sequence to the first... Feature vector of points Output point-by-point prediction results Furthermore, during the training phase, the backbone parameter set of the Transformer encoding layer in the point cloud Transformer network is fixed. It does not participate in the update; it only updates the parameter set of the multi-layer adapter module. With downstream task header parameter set Training is completed by minimizing the task loss function, so that the optimization objective satisfies:

[0068] ;

[0069] In the formula, The mapping function for the downstream task header, denoted as [function name] in the point cloud classification task. In the point cloud segmentation task, it is denoted as ; For monitoring labels, This is the task loss function.

[0070] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A point cloud data processing method based on point cloud Transformer networks, characterized in that, Includes the following steps: S1: Acquire point cloud data, perform local sampling, rotation transformation and scale normalization on the point cloud data, and generate an enhanced point cloud feature sequence for deep learning modeling; S2: Input the enhanced point cloud feature sequence into the point cloud Transformer network. The point cloud Transformer network consists of multiple Transformer coding layers connected in sequence, and an adapter module is embedded in each Transformer coding layer, so that the adapter modules in each Transformer coding layer together constitute a multi-layer adapter module. S3: In the current Transformer coding layer, the point feature sequence from the previous Transformer coding layer is processed by layer normalization and then multi-head self-attention calculation is performed. The attention fusion point feature sequence of the Transformer coding layer is obtained by residual fusion. S4: Input the attention fusion point feature sequence into the corresponding adapter module in the current Transformer coding layer, and sequentially perform ascending projection, nonlinear activation, feature fusion and dimensionality reduction projection to perform feature modulation on the attention fusion point feature sequence, and obtain the adaptation enhancement point feature sequence through residual fusion; S5: Input the adapted and enhanced point feature sequence into the feedforward network of the current Transformer coding layer for nonlinear mapping, and obtain the coding layer output point feature sequence of the Transformer coding layer through residual fusion; use the coding layer output point feature sequence as the point feature sequence of the next Transformer coding layer, and repeat steps S3 to S5 until the final point feature sequence of the last Transformer coding layer is obtained. S6: Input the final point feature sequence into the downstream task head for feature decoding, and generate point cloud classification results or point cloud segmentation results as point cloud data processing results output.

2. The point cloud data processing method based on point cloud Transformer networks according to claim 1, characterized in that, In step S1, when performing rotation transformation and scale normalization on the point cloud data, the first... The coordinates of the points are as follows The geometric center of the point cloud is denoted as The rotation matrix is ​​denoted as The scaling factor is denoted as And the point coordinates are standardized as follows: ； In the formula, The maximum radius of the rotated point cloud is used to ensure that the standardized point coordinates fall within a uniform scale range; subsequently, the... Standardized point coordinates of each point By combining and mapping the attribute features of points, an enhanced point cloud feature sequence is formed. ,in, This indicates the total number of points contained in the point cloud data, in the sequence. Indicates the first Enhanced feature vectors of points, The range of values ​​is to .

3. The point cloud data processing method based on point cloud Transformer networks according to claim 2, characterized in that, In step S2, the point cloud Transformer network is composed of The system consists of several sequentially connected Transformer coding layers. Within each Transformer coding layer, feature updates are performed in the order of "multi-head self-attention update - adapter module modulation - feedforward network update". Residual fusion is used at the multi-head self-attention update, adapter module modulation, and feedforward network update points to ensure that the enhanced point cloud feature sequence is progressively passed between each Transformer coding layer to obtain the final point feature sequence. .

4. The point cloud data processing method based on point cloud Transformer networks according to claim 3, characterized in that, In step S3, an attention fusion point feature sequence is generated. In multi-head self-attention calculations, a relative position bias matrix generated by the relative displacement between any two points in the point cloud is introduced. , making the first The attention weights of each Transformer coding layer are superimposed on the relative position bias matrix during calculation. This ensures that the single-head attention output satisfies: ； In the formula, This indicates that the point-pair relevance scores are exponentially normalized to obtain the attention weight matrix, which is row-normalized and has non-negative elements. This indicates that the attention weight matrix is ​​based on the pair The attention output obtained by weighted summation , , These are the query matrix, key matrix, and value matrix, respectively. The feature dimension is single-headed; then the attention output is fused with the residual of the point feature sequence input to the current Transformer coding layer to obtain the attention fused point feature sequence. .

5. The point cloud data processing method based on point cloud Transformer networks according to claim 4, characterized in that, In step S4, the adapter module performs a process on the first feature in the attention fusion point sequence. Feature vectors of attention fusion points The first step is to perform rising projection and nonlinear activation to obtain the second step. The intermediate feature vector of the adapter and based on the first Geometric condition vectors Generate the first gate vectors For the first The intermediate feature vector of the adapter With the first Feature vectors of attention fusion points Perform element-wise weighted fusion to obtain the first fusion feature vectors , represented as: ； In the formula, This represents element-wise multiplication. Indicates and A vector of all 1s with the same dimension; then the first... fusion feature vectors The dimensionality reduction projection yields the first Each adapter output vector ; will the first Each adapter output vector By scaling factor With the Feature vectors of attention fusion points Add the residuals to obtain the first... Each adaptation enhancement point feature vector , represented as: ； From all Constructing the feature sequence of the adaptation enhancement point .

6. The point cloud data processing method based on point cloud Transformer networks according to claim 5, characterized in that, In step S5, the multi-layer adapter module performs cross-layer collaborative updates between different Transformer coding layers, for the first... Layer adaptation enhancement point feature sequence Hierarchical aggregation yields hierarchical vectors. , represented as: ； And based on the hierarchical vector Exponential recursion is used to form a cross-layer memory vector. , represented as: ； In the formula, For memory update coefficients; subsequently, in the... In the layer adapter module, the cross-layer memory vector As one of the input conditions for gating generation, and the first Layer Geometric condition vector of points They work together to generate the gated vector and make the gated vector Participating in the Layer-to-adapter intermediate feature vector Feature vectors fused with attention points The fusion process is used to obtain the fused feature vector. Thus, the final point feature sequence Maintain the continuity of cross-layer information during progressive updates between layers.

7. The point cloud data processing method based on a point cloud Transformer network according to claim 6, characterized in that, In step S6, the downstream task head receives the final point feature sequence. It also outputs the prediction results, where in the point cloud classification task, the classification prediction results are output by the downstream task head. ,in, This represents the mapping function of the downstream task head; in the point cloud segmentation task, the downstream task head maps the final point feature sequence to the first... Feature vector of points Output point-by-point prediction results Furthermore, during the training phase, the backbone parameter set of the Transformer encoding layer in the point cloud Transformer network is fixed. It does not participate in the update; it only updates the parameter set of the multi-layer adapter module. With downstream task header parameter set Training is completed by minimizing the task loss function, so that the optimization objective satisfies: ； In the formula, The mapping function for the downstream task header, denoted as [function name] in the point cloud classification task. In the point cloud segmentation task, it is denoted as ; For monitoring labels, This is the task loss function.

Images