A 3D point cloud analysis method and device based on a Transformer and enhanced with local semantic learning capability

Abstract

This invention discloses a 3D point cloud analysis method and apparatus based on Transformer that enhances local semantic learning capabilities. The method constructs a deep learning model, including a local semantic self-attention learning module, which can perceive global context information and acquire finer-grained local semantic features in parallel, thereby enhancing the perception capability of the entire network framework. Local and global features of the point cloud are obtained from different modules through four local semantic learning modules, and these features are converged to obtain global joint features. The joint features obtained in the learning stage are then sent to subsequent point cloud classification and segmentation stages to obtain semantic classification results and final segmentation results. This invention can acquire key local geometric semantic information in 3D data and has significant advantages in various 3D point cloud analysis applications, such as point cloud classification, point cloud segmentation, and semantic segmentation of large indoor scenes.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a 3D point cloud analysis method and apparatus based on Transformer that enhances local semantic learning capabilities. Background Technology

[0002] In recent years, 3D models have received increasing attention. 3D point clouds are an important method for representing 3D data and are also the simplest representation of 3D data. A point cloud is mainly composed of a set of discrete points distributed in space. Compared with other 3D data representations such as Voxel and Mesh, 3D point cloud data is not only smaller in scale but can also be directly acquired using 3D scanning equipment. With continuous breakthroughs in 3D sensors and point cloud data acquisition equipment, such as 3D scanners, LiDAR, and RGB-D cameras, point cloud processing methods can bypass traditional, expensive mesh reconstruction or denoising techniques and directly process point clouds. This significantly improves the processing efficiency of point cloud data and reduces data loss during conversion. Therefore, with advancements in hardware and point cloud processing technology in recent years, 3D point cloud data processing has been applied in various fields such as robotics, autonomous driving, and industrial manufacturing.

[0003] However, the discrete and unordered data structure of point clouds makes their processing a significant challenge. While some researchers have proposed 3D point cloud processing techniques that have greatly improved processing efficiency and quality, the complex geometric and semantic information contained in point cloud data remains a key research focus and a challenge. Influenced by 2D visual processing, some methods focus on how to use conventional convolution to process conventional 3D meshes converted from 3D point clouds. Clearly, using 3D meshes to process point cloud data not only introduces errors due to data conversion but also incurs a huge computational burden. In this work, we focus on applying deep learning techniques to directly process 3D point cloud data.

[0004] PointNet pioneered the use of deep learning for end-to-end direct processing of point cloud data, achieving satisfactory results in point cloud classification and segmentation tasks. However, PointNet employed global aggregation without considering local structure, resulting in a weaker ability to perceive fine-grained details. Subsequent researchers proposed various deep learning-based point cloud learning methods. PointCNN designed the X-transform to associate points with kernels, thus "regularizing" the point cloud data and enabling the use of standard operators. Atzmon et al. proposed PCNN: a new framework applying convolutional neural networks to point clouds. DGCNN proposed an edge convolution operation to learn local geometric information by aggregating features of the local shape of the point cloud while maintaining permutation invariance. KPConv designed a kernel function that transforms point features using a sphere as a local learning region. PAConv proposed using a dynamic weight matrix during point cloud learning to construct local point cloud geometric information.

[0005] While some recent point cloud processing methods utilize local features to improve performance, their ability to perceive global long-range dependencies is insufficient. Meanwhile, network models based on the Transformer framework, compared to the aforementioned methods, are better able to perceive global information in long-range dependencies, and have therefore been transferred to vision tasks, achieving breakthrough results on multiple vision datasets. In 3D point cloud processing, Guo et al. proposed Point Cloud Transformer (PCT) to learn point cloud features using the Transformer framework. Zhao et al. proposed Point Transformer (PT), which designed a self-attention network and also achieved good results on point cloud tasks. However, in complex 3D vision tasks, local information is key to understanding 3D global semantics, regardless of whether it's a convolutional network or the Transformer framework. Directly applying the Transformer framework to disordered and irregular point cloud data, merely perceiving global feature information is insufficient.

[0006] Therefore, it can be seen that the existing methods have poor point cloud classification and segmentation results due to insufficient mining of point cloud geometric semantic information and insufficient shape perception ability of 3D point cloud objects. Summary of the Invention

[0007] This invention provides a 3D point cloud analysis method and apparatus based on Transformer that enhances local semantic learning capabilities, in order to solve or at least partially solve the technical problem of poor classification and segmentation performance in the prior art.

[0008] To address the aforementioned technical problems, the first aspect of this invention provides a 3D point cloud analysis method based on Transformer that enhances local semantic learning capabilities, comprising:

[0009] Acquire 3D point cloud data and perform preprocessing;

[0010] A model for point cloud semantic classification and segmentation is constructed, comprising an input feature embedding module, a local semantic self-attention learning module, an aggregation module, a pooling module, a classification module, and a segmentation module. The input feature embedding module performs affine transformations on 3D point cloud targets and learns high-dimensional point cloud patch vectors. The local semantic self-attention learning module learns global and local semantic information of the point cloud patches based on the high-dimensional point cloud patch vectors. The local semantic self-attention learning module includes multiple local semantic self-attention learning units. The first local semantic self-attention learning unit uses the high-dimensional vectors obtained from the input feature embedding module... The point cloud patch vector of degree is used as input for computation. Each subsequent local semantic self-attention learning unit is calculated using the output of the previous local semantic self-attention learning unit as input. The aggregation module is used to aggregate the local features obtained by each local semantic self-attention learning unit to obtain aggregate features. The pooling module is used to reduce the dimensionality of the aggregate features through pooling operations to obtain global aggregate features. The semantic classification module is used to optimize the loss function based on the global aggregate features and 3D point cloud data classification to obtain semantic classification results. The semantic segmentation module is used to optimize the loss function based on the global aggregate features and 3D point cloud data segmentation to obtain the segmentation results for each part.

[0011] We use the constructed point cloud semantic classification and segmentation model to perform semantic classification and segmentation of 3D point cloud data.

[0012] In one implementation, the processing procedure of the feature embedding module includes:

[0013] Initialize the raw point cloud data The input is fed into the input feature embedding module, and T-Net is used to... Perform an affine transformation to obtain the point cloud dataset after the affine transformation.

[0014] The candidate point set is obtained by uniformly sampling the point cloud surface using the farthest point sampling algorithm.

[0015] Use the K-nearest neighbors algorithm to obtain the candidate point set. The k points that are most semantically relevant to each point locally;

[0016] For each candidate point set, local point cloud semantic information is learned in the local semantic region. Then, all local semantic information is aggregated into features, and a multilayer perceptron is used to learn and increase the dimensionality of the features. The specific aggregation relationship is as follows:

[0017]

[0018] in This represents the point cloud of a local region obtained through the K-nearest neighbor algorithm, where Max represents the max pooling operation, and F... out The feature vector is the final output of the input feature embedding module.

[0019] In one implementation, the number of local semantic self-attention learning units is four, and the processing procedure of the local semantic self-attention learning module includes:

[0020] The first local semantic self-attention learning unit takes the feature vector obtained from the input feature embedding module as input and calculates to obtain the first local feature;

[0021] The second local semantic self-attention learning unit uses the first local features and the feature vector obtained by the input feature embedding module as input to calculate the second local features;

[0022] The third local semantic self-attention learning unit uses the second local features and the feature vector obtained from the input feature embedding module as input to calculate the third local features;

[0023] The fourth local semantic self-attention learning unit uses the feature vector obtained from the third local feature and the input feature embedding module as input to calculate the fourth local feature.

[0024] In one implementation, the computation process for each local semantic self-attention learning unit includes:

[0025] After adding the original coordinate information to the high-dimensional point cloud patch vector obtained by the input feature embedding module, it is used as the Q, K, V feature vectors in the self-attention mechanism. The attention weight feature A is calculated using the feature vectors Q and K, and then normalized using Softmax.

[0026] The feature vector V is learned by the function f through local semantic features. agg Feature information learning is performed to obtain a feature vector V′ containing local semantic information;

[0027] The feature vector V′ containing local semantic information is combined with the attention weight feature A to obtain the self-attention feature vector F. Sa ;

[0028] The original input features F of each local semantic self-attention learning unit in With self-attention feature vector F Sa The following approach combines this with a biased attention strategy:

[0029] F′ out =LBR(F in -F Sa )+F in ;

[0030] Among them, F′ out Features for self-attention output;

[0031] A set of linear normalization layers is used to learn the self-attention output feature F′. out and the original input features F of the current layer in The features are used to obtain the corresponding local features, as follows:

[0032]

[0033] in This represents the local feature output by the i-th Local Semantic Self-Attention Learning Unit (LSL-SA).

[0034] In one implementation, the aggregation process of the aggregation module is as follows:

[0035]

[0036] in, F represents the first to fourth local features. So The global aggregate feature is a 1024-dimensional global feature vector.

[0037] In one implementation, the semantic classification module includes three fully connected linear layers, and the processing procedure of the semantic classification module is as follows:

[0038] The feature vectors obtained from the pooling module are input into 521-dimensional and 256-dimensional linear fully connected layers, respectively.

[0039] Based on the 3D point cloud classification task, the last linear fully connected layer outputs different judgment results for k candidate classes.

[0040] In one implementation, the semantic segmentation module includes four linear fully connected layers, and the processing procedure of the semantic segmentation module is as follows:

[0041] The feature vectors obtained from the pooling module and the original input data The semantic categories are combined with the 64-dimensional category encoding features obtained by upscaling the MLP features to obtain the combined feature vector;

[0042] The combined feature vectors are then input into 512-dimensional, 256-dimensional, and 128-dimensional linear fully connected layers, respectively.

[0043] Based on the 3D point cloud segmentation task, the final segmentation result is obtained by segmenting N points into M semantic subcategories through the last linear fully connected layer, where N is the number of point clouds and M is the number of semantic categories.

[0044] Based on the same inventive concept, a second aspect of the present invention provides a 3D point cloud analysis device based on Transformer with enhanced local semantic learning capabilities, comprising:

[0045] The data acquisition and preprocessing module is used to acquire 3D point cloud data and perform preprocessing.

[0046] The model building module is used to construct a model for point cloud semantic classification and segmentation. This model includes an input feature embedding module, a local semantic self-attention learning module, an aggregation module, a pooling module, a classification module, and a segmentation module. The input feature embedding module performs affine transformations on the 3D point cloud targets and learns high-dimensional point cloud patch vectors. The local semantic self-attention learning module learns global features and local semantic information of the point cloud patches based on the high-dimensional point cloud patch vectors. The local semantic self-attention learning module includes multiple local semantic self-attention learning units, with the first unit being the input feature embedding module. The high-dimensional point cloud patch vector is used as input for computation. Each subsequent local semantic self-attention learning unit uses the output of the previous local semantic self-attention learning unit as input for computation. The aggregation module aggregates the local features obtained by each local semantic self-attention learning unit to obtain aggregated features. The pooling module reduces the dimensionality of the aggregated features through pooling operations to obtain global aggregated features. The semantic classification module optimizes the loss function based on the global aggregated features and 3D point cloud data classification to obtain semantic classification results. The semantic segmentation module optimizes the loss function based on the global aggregated features and 3D point cloud data segmentation to obtain the segmentation results for each part.

[0047] The semantic classification and segmentation module is used to perform semantic classification and segmentation of 3D point cloud data using the constructed point cloud semantic classification and segmentation model.

[0048] Based on the same inventive concept, a third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the method described in the first aspect.

[0049] Based on the same inventive concept, a fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0050] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0051] This invention constructs a model for point cloud semantic classification and segmentation. The model includes an input feature embedding module, a local semantic self-attention learning module, an aggregation module, a pooling module, a classification module, and a segmentation module. The input feature embedding module performs affine transformations on 3D point cloud targets and learns high-dimensional point cloud patch vectors. The local semantic self-attention learning module learns global features and local semantic information of point cloud patches based on the high-dimensional point cloud patch vectors. Overall, this invention learns the geometric semantic information of point clouds through a deep model and integrates the local semantic self-attention mechanism into a deep network model, learning robust features for 3D point cloud shape recognition. This overcomes uncertainties such as noise and spatial deformation, improving the accuracy of 3D point cloud shape recognition. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating the 3D point cloud analysis method based on Transformer with enhanced local semantic learning capabilities in an embodiment of the present invention.

[0054] Figure 2 This is a schematic diagram of the structure of the model for point cloud semantic classification and segmentation constructed in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of the workflow of the input feature embedding module (IFE) in an embodiment of the present invention;

[0056] Figure 4 This is an architecture diagram of the Local Semantic Learning Self-Attention Module (LSL-SA) in an embodiment of the present invention. Detailed Implementation

[0057] This invention discloses a 3D point cloud analysis method and apparatus based on Transformer with enhanced local semantic learning capabilities. The method includes: a novel Local Semantic Learning Self-attention (LSL-SA) network framework for point clouds that can be deployed on a local computing device. This framework incorporates the proposed Local Semantic Learning Self-attention (LSL-SA) mechanism, which understands the semantic information of 3D data holistically while enhancing local feature learning. This mechanism is better suited for handling disordered and irregular point clouds, particularly for tasks with high-value geometric information, such as point clouds. The mechanism is integrated into a specially designed LSL-SA module, enabling parallel perception of global context information and acquisition of finer-grained local semantic features, thereby enhancing the overall perceptual capabilities of the network framework. This method can be integrated and compatible with typical point cloud deep learning network structures in an end-to-end manner. The invention utilizes four... Each local semantic learning unit obtains local and global features of the point cloud from different units and performs feature aggregation to obtain global joint features. Then, the joint features obtained in the learning stage are sent to the subsequent point cloud classification and point cloud segmentation stages. In the 3D point cloud classification task, there are three fully connected layers with 521, 256, and different candidate categories, respectively. In the point cloud segmentation task, the 1024-dimensional global features with two different pooling methods and the LSL-SA features obtained from the previous layer are combined to obtain (1024+1024+64)-dimensional local and global mixed features, which are then passed through four fully connected layers to obtain the final segmentation result.

[0058] The 3D point cloud analysis method based on Transformer with enhanced local semantic learning capability of the present invention can be used in 3D point cloud processing in various CAX fields. It can obtain key local geometric semantic information in 3D data and has obvious advantages in various 3D point cloud analysis applications such as point cloud classification, point cloud segmentation and large indoor scene semantic segmentation.

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] Example 1

[0061] This invention provides a 3D point cloud analysis method based on Transformer with enhanced local semantic learning capabilities, comprising:

[0062] S1: Acquire 3D point cloud data and perform preprocessing;

[0063] S2: Construct a model for point cloud semantic classification and segmentation. This model includes an input feature embedding module, a local semantic self-attention learning module, an aggregation module, a pooling module, a classification module, and a segmentation module. The input feature embedding module performs affine transformations on the 3D point cloud targets and learns high-dimensional point cloud patch vectors. The local semantic self-attention learning module learns global and local semantic information of the point cloud patches based on the high-dimensional point cloud patch vectors. The local semantic self-attention learning module includes multiple local semantic self-attention learning units. The first local semantic self-attention learning unit uses the feature vectors obtained from the input feature embedding module... The high-dimensional point cloud patch vector is used as input for computation. Each subsequent local semantic self-attention learning unit is calculated using the output of the previous local semantic self-attention learning unit as input. The aggregation module aggregates the local features obtained by each local semantic self-attention learning unit to obtain aggregated features. The pooling module reduces the dimensionality of the aggregated features through pooling operations to obtain global aggregated features. The semantic classification module optimizes the loss function based on the global aggregated features and 3D point cloud data classification to obtain semantic classification results. The semantic segmentation module optimizes the loss function based on the global aggregated features and 3D point cloud data segmentation to obtain the segmentation results for each part.

[0064] S3: Perform semantic classification and segmentation of 3D point cloud data using the constructed point cloud semantic classification and segmentation model.

[0065] Figure 1 This is a flowchart of a 3D point cloud analysis method based on Transformer with enhanced local semantic learning capability according to an embodiment of the present invention. The following is an example... Figure 1 To illustrate some specific implementation processes of the present invention, we will use examples, such as... Figure 1 As shown, the 3D point cloud analysis method based on Transformer with enhanced local semantic learning capabilities and its applications in point cloud classification and segmentation include the following steps:

[0066] Step S1: Input 3D point cloud data information into the computing device and preprocess the data. When training on the ModelNet40 dataset, first load the data according to the set batch size, and use jitter, rotation, and translation to perform data augmentation.

[0067] Step S2: The proposed input feature embedding module performs an affine transformation on the 3D point cloud object and learns to obtain a high-dimensional point cloud patch vector, which is then input into the subsequent network module.

[0068] Step S3: Use the proposed LSL-SA to learn the global features and local semantic information of the point cloud patch.

[0069] Step S4: Aggregate the features of all kernel-related learning modules in the point feature learning stage.

[0070] Step S5, aggregate the global features F from the different stages. So Dimensionality reduction is achieved through pooling operations;

[0071] Step S6: Optimize the loss function based on global aggregated features and 3D point cloud data classification to obtain the classification result;

[0072] Step S7: Based on the loss function optimization of global aggregated features and 3D point cloud data segmentation, the segmentation result of each part is obtained, and the segmentation task is finally completed.

[0073] In step S1, in one embodiment of the present invention, the hardware device is a central processing unit (CPU): Intel i5 10500 (4.3GHz); a graphics processing unit (GPU): NVIDIA GeForce RTX 3090 (24GB VRAM); a memory storage: 64GB; and a physical data storage: SSD (500GB). After the hardware device is debugged and found to be normal, the software is installed: the design and implementation of the PyTorch platform, the programming language Python 3.8, the torch 1.8 software compilation platform, etc. After completion, the code of the present invention is deployed and data is input.

[0074] Figure 2 This is a structural diagram of the overall point cloud data processing model according to an embodiment of the present invention.

[0075] In one implementation, the processing procedure of the feature embedding module includes:

[0076] S21: Initialize the raw point cloud data The input is fed into the input feature embedding module, and T-Net is used to... Perform an affine transformation to obtain the point cloud dataset after the affine transformation.

[0077] S22: The candidate point set is obtained by uniformly sampling the point cloud surface using the farthest point sampling algorithm.

[0078] S23: Use the K-nearest neighbor algorithm to obtain the candidate point set. The k points that are most semantically relevant to each point locally;

[0079] S24: For each candidate point set, learn local point cloud semantic information in the local semantic region, then aggregate all local semantic information features, and use a multilayer perceptron to learn and increase the feature dimensionality. The specific aggregation relationship is as follows:

[0080]

[0081] in This represents the point cloud of a local region obtained through the K-nearest neighbor algorithm, where Max represents the max pooling operation, and F... out The feature vector is the final output of the input feature embedding module.

[0082] Figure 3 This is a flowchart of the input embedding module structure according to an embodiment of the present invention. The following is an example of... Figure 3 The following example illustrates the point cloud data processing process in the IFE module of this invention.

[0083] In step S21, each 3D point cloud object contains 1024 points after preprocessing, and an affine transformation factor A is learned using a regression network T-net consisting of multiple hidden layers. θ It can be represented as follows:

[0084]

[0085] Where A θ It is composed of element θ 11 ~θ 33 The affine factor matrix is ​​composed of 3×3, and then the affine transformation factor A is... θ With the original input point set Combined

[0086] In step S22, the number of uniform sampling points is set to 512 using the FPS algorithm to obtain a set of candidate points for uniform sampling.

[0087] During the implementation of step S23, the local semantic range K is set to 16.

[0088] In the implementation of step S24, feature aggregation is performed for each point using max pooling. At the same time, MLP is used to increase the feature dimension to 256 dimensions during the feature dimensionality upscaling process. Therefore, after passing through the IFE module, each point learns 256-dimensional feature information.

[0089] In one implementation, the number of local semantic self-attention learning units is four, and the processing procedure of the local semantic self-attention learning module includes:

[0090] The first local semantic self-attention learning unit takes the feature vector obtained from the input feature embedding module as input and calculates to obtain the first local feature;

[0091] The second local semantic self-attention learning unit uses the first local features and the feature vector obtained by the input feature embedding module as input to calculate the second local features;

[0092] The third local semantic self-attention learning unit uses the second local features and the feature vector obtained from the input feature embedding module as input to calculate the third local features;

[0093] The fourth local semantic self-attention learning unit uses the feature vector obtained from the third local feature and the input feature embedding module as input to calculate the fourth local feature.

[0094] In one implementation, the computation process for each local semantic self-attention learning unit includes:

[0095] S31: After adding the original coordinate information to the high-dimensional point cloud patch vector obtained by the input feature embedding module, it is used as the Q, K, V feature vectors in the self-attention mechanism. The attention weight feature A is calculated using the feature vectors Q and K, and then normalized using Softmax.

[0096] S32: Learn the function f from the feature vector V using local semantic features. agg Feature information learning is performed to obtain a feature vector V′ containing local semantic information;

[0097] S33: Combine the feature vector V′ containing local semantic information with the attention weight feature A to obtain the self-attention feature vector F. Sa ;

[0098] S34: The original input features F of each local semantic self-attention learning unit in With self-attention feature vector F Sa The following approach combines this with a biased attention strategy:

[0099] F′ out =LBR(F in -F Sa )+F in ;

[0100] Among them, F′ out Features for self-attention output;

[0101] S35: A set of linear normalization layers is used to learn the self-attention output feature F′. out and the original input features F of the current layer inThe features are used to obtain the corresponding local features, as follows:

[0102]

[0103] in This represents the local feature output by the i-th Local Semantic Self-Attention Learning Unit (LSL-SA).

[0104] Specifically Figure 4 This is a data flow structure diagram of a local semantic self-attention learning module according to an embodiment of the present invention.

[0105] Specifically, step S3 includes the following steps:

[0106] In step S31, the 256-dimensional feature vector obtained from IFE is used as Q, K, and V respectively for self-attention calculation, and the attention weight feature A = (α) is obtained first. i,j =Q·K T Then, using Softmax for normalization, it can be represented as follows:

[0107]

[0108] Where α i,j Represents relative to the current center point x i and its corresponding semantic relation neighborhood point x j The resulting self-attention weights are derived from the attention distribution weights. and weighted average weight And after a series of operations, such as weighted summation ∑, the result is obtained;

[0109] During the implementation of step S32, a local semantic information learning function was designed. in For local neighborhood features, x i Let x be the center point of the current location. j It means that for x j The semantic relation neighborhood points of the current center point are represented by a set of learnable parameters θ, which are used to learn local semantic information. Max is used as the channel symmetric function.

[0110] In step S33, the feature vector V′ containing local semantic information is combined with the attention weight feature A to obtain the self-attention feature vector F. Sa It can be represented as: F Sa =A·V′;

[0111] During the implementation of step S34:

[0112] Self-attention can be summarized as: These are the parameters that the network model learns from the input data.

[0113] During the implementation of step S35:

[0114] Four groups of LSL-SA were used, thus obtaining respectively The LSL-SA feature vectors, except for the first group of LSL-SA, are the feature inputs of the previous group of LSL-SA and the original input feature vectors after IFE.

[0115] In one implementation, the aggregation process of the aggregation module is as follows:

[0116]

[0117] in, F represents the first to fourth local features. So The global aggregate feature is a 1024-dimensional global feature vector.

[0118] Furthermore, in step S4, the features of all kernel-related learning modules in the point feature learning stage are aggregated.

[0119] In one embodiment of the present invention, the point feature learning stage aggregates the kernel correlation learning modules of the previous layer into 256, 256, 256, 256-dimensional feature vectors, which can be represented as:

[0120]

[0121] The output characteristics of each group of LSL-SA modules are as follows: The aggregation yields the overall aggregated feature vector F. So .

[0122] Step S5, aggregate the global features F from the different stages. So Dimensionality reduction is achieved through pooling operations.

[0123] Furthermore, the pooling operation in step S5 is represented as follows:

[0124] F g =Max(F So )

[0125] In one embodiment of the present invention, after the max pooling operation, the originally learned aggregated features are max pooled into a unified 1024-dimensional feature vector. Additionally, during the segmentation task, an average pooling operation Avg, denoted as F, is also employed. g =Avg(F Sp This yields another set of 1024-dimensional feature vectors.

[0126] In one implementation, the semantic classification module includes three fully connected linear layers, and the processing procedure of the semantic classification module is as follows:

[0127] The feature vectors obtained from the pooling module are input into 521-dimensional and 256-dimensional linear fully connected layers, respectively.

[0128] Based on the 3D point cloud classification task, the last linear fully connected layer outputs different judgment results for k candidate classes.

[0129] Specifically, in step S6, the classification result is obtained by optimizing the loss function based on the global aggregated features and the 3D point cloud data classification.

[0130] In the specific implementation process, step S6 includes the following steps:

[0131] Step S61: Input the 1024-dimensional point feature vector obtained in S5 into 521-dimensional and 256-dimensional linear fully connected layers respectively;

[0132] Step S62: After passing through the first two linear fully connected layers, output the different judgment results of k candidate classes according to the 3D point cloud classification task;

[0133] In one embodiment of the present invention, in a 3D shape recognition application, the model needs to determine its semantic category in the input original point cloud. The current instance is evaluated on the ModelNet40 dataset, and the candidate class K is determined to be 40. The result obtained is the final discrimination result, thus realizing the 3D shape recognition classification application.

[0134] In one implementation, the semantic segmentation module includes four linear fully connected layers, and the processing procedure of the semantic segmentation module is as follows:

[0135] The feature vectors obtained from the pooling module and the original input data The semantic categories are combined with the 64-dimensional category encoding features obtained by upscaling the MLP features to obtain the combined feature vector;

[0136] The combined feature vectors are then input into 512-dimensional, 256-dimensional, and 128-dimensional linear fully connected layers, respectively.

[0137] Based on the 3D point cloud segmentation task, the final segmentation result is obtained by segmenting N points into M semantic subcategories through the last linear fully connected layer, where N is the number of point clouds and M is the number of semantic categories.

[0138] Step S7: Based on the loss function optimization of global aggregated features and 3D point cloud data segmentation, the segmentation result of each part is obtained, and the segmentation task is finally completed.

[0139] Further, step S7 includes the following steps:

[0140] Step S71: Combine the features obtained from the two different pooling methods in S5 with the category features to obtain F for segmentation. seg Feature vector;

[0141] In the specific implementation process, the feature vector aggregation in step S71 is represented as follows:

[0142] F seg =cat(F g1 F g2 F c )

[0143] Where cat represents aggregation operation, F g1 F represents the global feature vector obtained after max pooling operation. g2 F represents the global feature vector obtained after average pooling. c This represents the one-hot categorical encoding feature.

[0144] In the specific implementation process, step S7 is achieved through the following steps:

[0145] Step S72: Input the vectors from the previous layers into 512-dimensional, 256-dimensional, and 128-dimensional linear fully connected layers respectively;

[0146] Step S73: After passing the feature vectors of the first three linear fully connected layers, the N points are segmented into M semantic subcategories according to the 3D point cloud segmentation task to obtain the final segmentation result. For details, please refer to the audit reference materials, which correspond to the semantic segmentation of point cloud components and the semantic segmentation of large indoor scenes, respectively.

[0147] Specifically, one of the accompanying drawings in the empirical reference materials includes an effect diagram of a point cloud component segmentation application task according to an embodiment of the present invention. The following uses this drawing as an example to illustrate the processing procedure when applying point cloud component semantic segmentation according to the present invention. In one embodiment of the present invention, in the 3D point cloud component segmentation application task, it is necessary to complete the semantic segmentation of different parts of a 3D point cloud target. In this example, the component semantic segmentation is evaluated on the ShapeNet dataset, with 50 component semantic categories. According to step S73, where N is 2048 and M is 50, the semantic label of each point is finally obtained. Based on the semantic segmentation of different components in different 3D point cloud models, the result is the final semantic result of the discrimination.

[0148] Specifically, another accompanying figure in the empirical reference is a diagram illustrating the effect of semantic segmentation of a large indoor scene according to an embodiment of the present invention. The following uses this figure as an example to illustrate the processing procedure of the present invention in the semantic segmentation of a large indoor scene. In one embodiment of the present invention, in the semantic segmentation of a large indoor scene, the S3DIS dataset is used for evaluation and testing. Based on the semantic label of each point in the 3D point cloud, the semantic labels of all 13 categories of point cloud labels in 272 indoor scenes are semantically discriminated. Each 3D point cloud object contains 4096 points, and each point contains different semantic information. According to step S73, where N is 4096 and M is 13, the semantic label of each point is finally obtained.

[0149] Publicly available 3D point cloud shape datasets were used as test subjects, such as ModelNet40, ShapeNet, and S3DIS datasets. This invention presents a method and application for classifying and segmenting 3D point clouds based on learning point cloud features using the Transformer framework of deep learning. A complete model framework was designed and implemented, and deployed on a computing device, improving the accuracy of 3D point cloud shape recognition. This demonstrates the effectiveness of the method.

[0150] Example 2

[0151] Based on the same inventive concept, this embodiment provides a 3D point cloud analysis device based on Transformer with enhanced local semantic learning capabilities, comprising:

[0152] The data acquisition and preprocessing module is used to acquire 3D point cloud data and perform preprocessing.

[0153] The model building module is used to construct a model for point cloud semantic classification and segmentation. This model includes an input feature embedding module, a local semantic self-attention learning module, an aggregation module, a pooling module, a classification module, and a segmentation module. The input feature embedding module performs affine transformations on the 3D point cloud targets and learns high-dimensional point cloud patch vectors. The local semantic self-attention learning module learns global features and local semantic information of the point cloud patches based on the high-dimensional point cloud patch vectors. The local semantic self-attention learning module includes multiple local semantic self-attention learning units, with the first unit being the input feature embedding module. The high-dimensional point cloud patch vector is used as input for computation. Each subsequent local semantic self-attention learning unit uses the output of the previous local semantic self-attention learning unit as input for computation. The aggregation module aggregates the local features obtained by each local semantic self-attention learning unit to obtain aggregated features. The pooling module reduces the dimensionality of the aggregated features through pooling operations to obtain global aggregated features. The semantic classification module optimizes the loss function based on the global aggregated features and 3D point cloud data classification to obtain semantic classification results. The semantic segmentation module optimizes the loss function based on the global aggregated features and 3D point cloud data segmentation to obtain the segmentation results for each part.

[0154] The semantic classification and segmentation module is used to perform semantic classification and segmentation of 3D point cloud data using the constructed point cloud semantic classification and segmentation model.

[0155] Since the apparatus described in Embodiment 2 of this invention is the same apparatus used to implement the 3D point cloud analysis method based on Transformer with enhanced local semantic learning capability in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this apparatus based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All apparatuses used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0156] Example 3

[0157] Based on the same inventive concept, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the method described in Embodiment 1.

[0158] Since the computer-readable storage medium described in Embodiment 3 of this invention is the same computer-readable storage medium used in implementing the 3D point cloud analysis method based on Transformer with enhanced local semantic learning capability in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer-readable storage medium based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer-readable storage media used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0159] Example 4

[0160] Based on the same inventive concept, this application also provides a computer device, including storage, a processor, and a computer program stored in the storage and executable on the processor, wherein the processor executes the program to implement the method in Embodiment 1.

[0161] Since the computer device described in Embodiment 4 of this invention is the same computer device used to implement the 3D point cloud analysis method based on Transformer with enhanced local semantic learning capability in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer device based on the method described in Embodiment 1 of this invention, and therefore will not be described again here. All computer devices used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0162] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0163] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0164] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0165] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations to the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A 3D point cloud analysis method based on Transformer with enhanced local semantic learning capabilities, characterized in that, include: Acquire 3D point cloud data and perform preprocessing; A model for point cloud semantic classification and segmentation is constructed, comprising an input feature embedding module, a local semantic self-attention learning module, an aggregation module, a pooling module, a classification module, and a segmentation module. The input feature embedding module performs affine transformations on the 3D point cloud targets and learns high-dimensional point cloud patch vectors. The local semantic self-attention learning module learns global features and local semantic information of the point cloud patches based on the high-dimensional point cloud patch vectors. The local semantic self-attention learning module includes multiple local semantic self-attention learning units, with the first unit taking the high-dimensional point cloud patch vectors obtained from the input feature embedding module as input. The computation process involves each subsequent local semantic self-attention learning unit (LSI) using the output of the previous LSI as input. The aggregation module aggregates the local features obtained from each LSI to obtain aggregated features. The pooling module reduces the dimensionality of the aggregated features through pooling operations to obtain global aggregated features. The semantic classification module optimizes the loss function based on the global aggregated features and 3D point cloud data classification to obtain semantic classification results. The semantic segmentation module optimizes the loss function based on the global aggregated features and 3D point cloud data segmentation to obtain segmentation results for each part. Specifically, the feature embedding module's processing includes: initializing the original point cloud data... The input is fed into the input feature embedding module, and T-Net is used to... Perform an affine transformation to obtain the point cloud dataset after the affine transformation. The candidate point set is obtained by uniformly sampling the point cloud surface using the farthest point sampling algorithm. Use the K-nearest neighbors algorithm to obtain the candidate point set. For each point in the local semantic region, select the k most relevant points. For each candidate point set, learn local point cloud semantic information in the local semantic region. Then, aggregate all local semantic information features and use a multilayer perceptron to learn and increase the feature dimensionality. The specific aggregation relationship is as follows: in This represents the point cloud of a local region obtained through the K-nearest neighbor algorithm. This indicates a max pooling operation. The feature vector is the final output of the input feature embedding module; We use the constructed point cloud semantic classification and segmentation model to perform semantic classification and segmentation of 3D point cloud data.

2. The 3D point cloud analysis method based on Transformer with enhanced local semantic learning capability as described in claim 1, characterized in that, The number of local semantic self-attention learning units is 4, and the processing of the local semantic self-attention learning module includes: The first local semantic self-attention learning unit takes the feature vector obtained from the input feature embedding module as input and calculates to obtain the first local feature; The second local semantic self-attention learning unit uses the first local features and the feature vector obtained by the input feature embedding module as input to calculate the second local features; The third local semantic self-attention learning unit uses the second local features and the feature vector obtained from the input feature embedding module as input to calculate the third local features; The fourth local semantic self-attention learning unit uses the feature vector obtained from the third local feature and the input feature embedding module as input to calculate the fourth local feature.

3. The 3D point cloud analysis method based on Transformer with enhanced local semantic learning capability as described in claim 2, characterized in that, The computational process for each local semantic self-attention learning unit includes: After adding the original coordinate information to the high-dimensional point cloud patch vector obtained from the input feature embedding module, it serves as the Q, K, and V feature vectors in the self-attention mechanism. The attention weight features are then calculated using the feature vectors Q and K. And normalize using Softmax; Learning a function from local semantic features for feature vector V By learning feature information, feature vectors containing local semantic information are obtained. ; Feature vectors containing local semantic information With attention weight features By combining these, we obtain the self-attention feature vector. ; The original input features of each local semantic self-attention learning unit With self-attention feature vector The following approach combines this with a biased attention strategy: ； in, Features for self-attention output; A set of linear normalization layers is used to learn self-attention output features. and the original input features of the current layer The features are used to obtain the corresponding local features, as follows: in Indicates the first Local features output by a Local Semantic Self-Attention Learning Unit (LSL-SA).

4. The 3D point cloud analysis method based on Transformer with enhanced local semantic learning capability as described in claim 2, characterized in that, The aggregation process of the aggregation module is as follows: in, ~ Representing the first to fourth local features, The global aggregate feature is a 1024-dimensional global feature vector.

5. The 3D point cloud analysis method based on Transformer with enhanced local semantic learning capability as described in claim 1, characterized in that, The semantic classification module consists of three fully connected linear layers. The processing procedure of the semantic classification module is as follows: The feature vectors obtained from the pooling module are input into 521-dimensional and 256-dimensional linear fully connected layers, respectively. Based on the 3D point cloud classification task, the output is obtained through the last linear fully connected layer. Different judgment results for each candidate class.

6. The 3D point cloud analysis method based on Transformer with enhanced local semantic learning capability as described in claim 1, characterized in that, The semantic segmentation module consists of four fully connected linear layers. The processing procedure of the semantic segmentation module is as follows: The feature vectors obtained from the pooling module and the original input data The semantic categories are combined with the 64-dimensional category encoding features obtained by upscaling the MLP features to obtain the combined feature vector; The combined feature vectors are then input into 512-dimensional, 256-dimensional, and 128-dimensional linear fully connected layers, respectively. Based on the 3D point cloud segmentation task, the last linear fully connected layer is used to... Each point The final segmentation result is obtained by segmenting each semantic subcategory. For the number of point clouds, The number of semantic categories.

7. A 3D point cloud analysis device based on Transformer with enhanced local semantic learning capabilities, characterized in that, include: The data acquisition and preprocessing module is used to acquire 3D point cloud data and perform preprocessing. The model building module is used to construct a model for point cloud semantic classification and segmentation. This model includes an input feature embedding module, a local semantic self-attention learning module, an aggregation module, a pooling module, a classification module, and a segmentation module. The input feature embedding module performs affine transformations on the 3D point cloud targets and learns high-dimensional point cloud patch vectors. The local semantic self-attention learning module learns global features and local semantic information of the point cloud patches based on the high-dimensional point cloud patch vectors. The local semantic self-attention learning module includes multiple local semantic self-attention learning units. The first local semantic self-attention learning unit uses the high-dimensional point cloud patch vectors obtained from the input feature embedding module as input. The system calculates the input from the previous local semantic self-attention learning unit. Each subsequent local semantic self-attention learning unit uses the output of the previous unit as input. The aggregation module aggregates the local features obtained from each unit to obtain aggregated features. The pooling module reduces the dimensionality of the aggregated features through pooling operations to obtain global aggregated features. The semantic classification module optimizes the loss function based on the global aggregated features and 3D point cloud data classification to obtain semantic classification results. The semantic segmentation module optimizes the loss function based on the global aggregated features and 3D point cloud data segmentation to obtain segmentation results for each part. Specifically, the feature embedding module's processing includes: initializing the original point cloud data... The input is fed into the input feature embedding module, and T-Net is used to... Perform an affine transformation to obtain the point cloud dataset after the affine transformation. The candidate point set is obtained by uniformly sampling the point cloud surface using the farthest point sampling algorithm. Use the K-nearest neighbors algorithm to obtain the candidate point set. For each point in the local semantic region, select the k most relevant points. For each candidate point set, learn local point cloud semantic information in the local semantic region. Then, aggregate all local semantic information features and use a multilayer perceptron to learn and increase the feature dimensionality. The specific aggregation relationship is as follows: in This represents the point cloud of a local region obtained through the K-nearest neighbor algorithm. This indicates a max pooling operation. The feature vector is the final output of the input feature embedding module; The semantic classification and segmentation module is used to perform semantic classification and segmentation of 3D point cloud data using the constructed point cloud semantic classification and segmentation model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1 to 6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 6.

Images