Three-dimensional point cloud task analysis method fusing local features and global context information

Abstract

A three-dimensional point cloud task analysis method fusing local features and global context information belongs to the field of three-dimensional point cloud task analysis, and comprises the following steps: step 1, constructing a point cloud data set: step 1.1), defining the data set; step 1.2), preprocessing the point cloud data set; step 2, point cloud data set sampling: down-sampling the point cloud number of the original point cloud data set to a fixed value through a farthest point sampling FPS mode; step 3, grouping point cloud data sets; 4, extracting local features; step 5, neighborhood feature propagation; step 6, constructing a space shape position code; step 7, constructing channel attention; step 8, local feature enhancement; step 9, global context feature extraction; and step 10, performing three-dimensional point cloud classification and segmentation tasks. According to the invention, the precision of the three-dimensional point cloud task is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional point cloud task analysis, and specifically relates to a three-dimensional point cloud task analysis method that integrates local features and global context information. Background Art

[0002] In recent years, 3D point cloud technology has been widely used in many fields such as autonomous driving and intelligent robots. Unlike traditional 2D image data, point cloud data consists of a set of disordered and irregular points. 3D point cloud task analysis is a key component in the practical application of this technology and is also a hot topic in environmental intelligent perception research. 3D point cloud task analysis plays a vital role in achieving the understanding and perception of 3D space.

[0003] As a perception technology for fine processing of three-dimensional space, the effect of 3D point cloud task analysis requires high efficiency, stability and accuracy. As deep learning technology has shown remarkable stability, reliability and automatic feature extraction capabilities in tasks such as image recognition, generation and segmentation, many studies have gradually applied deep learning to 3D point cloud tasks. However, due to the disorder and sparsity of point cloud data itself, deep learning faces many challenges in processing 3D point clouds.

[0004] Inspired by the successful application of convolutional neural networks in image tasks, researchers have tried to convert irregular point cloud data into regular forms such as images or voxels in order to apply convolutional neural networks for feature extraction. The main research directions include multi-view based and voxel based methods. Multi-view based methods project point clouds onto a two-dimensional plane and use mature two-dimensional convolution for processing; while voxel based methods first divide the three-dimensional space into regular voxels and then apply three-dimensional convolution to the voxel data. Although these methods perform well, they cannot well preserve the three-dimensional geometric information of the point cloud due to the possible loss of position information and lack of fine-grained features caused by the voxelization process.

[0005] To address these problems, researchers have developed point-based methods that directly use point features and location information to maintain the integrity of location information. This idea has spawned different feature aggregation methods for learning high-level semantic features. For example, PointNet introduced a multi-layer perceptron (MLP)-based method to handle three-dimensional tasks, but its feature extraction is still limited to local information. To this end, PointNet++ improves the capabilities of PointNet by introducing a hierarchical sampling strategy. DenseKPNet, on the other hand, establishes a complementary relationship between local features and high-level geometric information through multi-scale convolution and density connection to improve the accuracy of point cloud tasks and the ability to extract fine-grained features.

[0006] Recently, Transformer has gradually become a mainstream deep learning model, which solves the problem of long-term dependency and achieves excellent performance in most computer vision tasks. Inspired by the significant achievements of Transformer in two-dimensional vision tasks, researchers have developed a large number of methods for three-dimensional point cloud task analysis. Transformer has a natural advantage in learning the semantic information of geometric features. PCT, as a pure Transformer network, first proposed neighborhood embedding to map point clouds to a high-dimensional architectural feature space. PU-Transformer is the first Transformer model introduced for point cloud upsampling, and a new multi-head self-attention structure variant is developed to enhance the point-by-point and channel relationship of feature maps. OctForme inherits the order in the octation process, similar to the z-order, providing scalability, reducing the amount of computation while also having good performance, but it is still constrained by the octree structure itself. On the other hand, FlatFormer adopts a window-based sorting strategy to group points, similar to window partitioning. However, this design lacks scalability in receptive field. However, these methods only take the relative position relationship between point clouds as local geometric features, ignoring the shape relationship between point clouds, and the refinement of the rich shape semantic position information of point clouds is still limited. Summary of the invention

[0007] In order to overcome the shortcomings of the existing technology, the present invention proposes a three-dimensional point cloud task analysis method that integrates local features and global context information. First, a group self-attention mechanism (Local Group Self-Attention, LGA) is designed to capture the local interaction information in each area. In order to capture the separated local area feature relationship, a group propagation module (Local Group Propagation, LGP) is proposed to transmit information between different regions through query points to allow features to propagate between neighbors and obtain more fine-grained feature information. In order to further expand the effective receptive field, a global context feature extraction module (Global Shape Feature Module, GSFM) is proposed to learn global context information through key shape points. Finally, in order to solve the position information clues between global contexts, spatial shape position encoding (Spatial-Shape Relative Position Encoding, SS-RPE) is introduced to obtain the position relationship between points.

[0008] The technical solution adopted by the present invention to solve the technical problem is: A three-dimensional point cloud task analysis method integrating local features and global context information, the method comprising the following steps: Step 1: Build a point cloud dataset. The process is as follows: Step 1.1), define the data set; Step 1.2), point cloud dataset preprocessing: for the point cloud dataset P, random rotation, scaling and noise addition preprocessing operations are performed to generate point cloud datasets under various conditions, thereby improving the method's ability to resist interference with data changes; Step 2: Sampling of point cloud datasets. In order to alleviate the computational burden caused by the huge number of point clouds in the point cloud dataset, the number of point clouds in the original point cloud dataset is downsampled to a fixed value through the farthest point sampling (FPS) method. Step 3: Grouping of point cloud data sets: In three-dimensional geometry, shapes can be represented by a set of closely spaced three-dimensional coordinates. For the same target geometry, the point cloud data will be similar in spatial features and semantic expression. Therefore, when performing point cloud tasks, extracting the semantic information of the target geometry depends on the correlation between each point and its surrounding neighborhood points. KNN (K-Nearest Neighbors) is a commonly used algorithm for determining the neighborhood points of each point. Calculating the distance between each point and its nearest K neighbors can better understand the correlation between point cloud data. Grouping point cloud data using the KNN algorithm can effectively capture this local spatial relationship, which helps to extract the semantic information of point cloud data. Step 4: Local feature extraction: first construct a neighborhood set, then extract neighborhood set features; Step 5: Neighborhood feature propagation: In order to capture the separated local area feature relationship, Establish neighborhood relationships between query points in different areas to allow point cloud features to propagate between neighbors and obtain more fine-grained feature information; Step 6: Construct spatial shape position coding, first construct attention position coding and azimuth position coding, and then perform feature mixing; Step 7: Construct channel attention; Step 8: Local feature enhancement Step 9: extracting global context features, using the Global Shape Feature Module (GSFM) for global context feature extraction; Step 10: 3D point cloud classification and segmentation tasks. The process is as follows: Step 10.1), constructing the classification task: the present invention proposes a shape feature propagation module (SAPFormer Block) consisting of steps 1 to 8. For the three-dimensional point cloud classification task, the primary point cloud is obtained by point cloud embedding. The feature encoder consists of a downsampling layer and a SAPFormer module. The classification result is obtained by global average pooling (GAP) and MLP. The downsampling layer is a layer used in the deep learning model to reduce the spatial dimension of the input data. Global average pooling is a layer used in deep learning models. Unlike traditional pooling layers, the purpose of the global average pooling layer is to convert the entire feature map into a single average value, which helps to simplify the model and directly generate class labels after feature extraction. Point cloud embedding is used to extract primary features of point clouds. Step 10.2), construct the segmentation task: For the 3D point cloud segmentation task, the U-Net architecture is used. The encoder is the same as the classification task in step 10.1). After each stage is fed to the corresponding upsample, the segmentation result is accumulated; U-Net is a deep learning architecture that effectively processes a small amount of training data and can generate high-resolution segmented images. It is mainly used to restore the spatial dimension of the feature map and increase its height and width. Upsampling can convert low-resolution feature maps into high-resolution outputs.

[0009] Furthermore, in step 1.1), the point cloud data is composed of a series of three-dimensional coordinates, and each coordinate point contains the coordinate information of the point cloud in three-dimensional space. And the information f reflecting the texture characteristics of the point cloud data. The point cloud data set containing N points is denoted as ,in is the coordinate information of the i-th point. The texture feature set corresponding to P is recorded as , is the color information of the i-th point.

[0010] The process of step 1.2) is: Step 1.2.1), random rotation, randomly generate a rotation angle , and according to the angle Constructing a 3D coordinate rotation matrix , its expression is, ; Combine the point cloud dataset P with the rotation matrix Multiply them together to get the original point cloud rotated around the z axis Point cloud dataset after angle , the expression is, ; Step 1.2.2), random scaling, randomly generate scaling ratio , the zoom ratio Multiply with the rotated point cloud dataset Generate scaled data , the expression is, ; Step 1.2.3), add noise and randomly generate a set of point cloud datasets Three-dimensional coordinates of the same dimension , The value of is small, the scaled data set and noise Adding, adding tiny noise interference to the original point cloud, .

[0011] The process of step 2 is as follows: Step 2.1), determine the sampling initial point, and Traverse and calculate the average value of all points in the three coordinate dimensions, and use the calculated average value as the initial point for sampling the farthest point , and at the same time, add it to the sampling point set ; Step 2.2), calculate the distance matrix, the farthest point sampling is to select the distance from the point cloud data set The farthest point, therefore, in order to obtain point cloud data With sampling point set The distance is obtained through the distance matrix Record each point with The Euclidean distance between the newly added points; Step 2.3), update the sampling point set, The point corresponding to the maximum value is denoted by and add it to ,along with Update sampling points, The Euclidean distance between the newly added sampling points and the point cloud dataset will be recalculated, and the point clouds whose Euclidean distance is less than the original value will be updated at the same time; finally, the above operations are repeated to increase the number of point clouds in the sampling point set to a fixed value.

[0012] The process of step 4 is as follows: Step 4.1) Build the neighborhood set: According to the point cloud set P constructed in step 1, KPconv is used to extract local features. KPconv is a mature point cloud network architecture that proposes kernel point convolution and simulates deformation convolution to obtain local information of point clouds. Then, the FPS algorithm in step 2 is used to obtain The center point is obtained local area, and then use the point cloud sampled by the KNN algorithm in step 3 Divide into N / K local area blocks, each area contains K points, and group the K nearest neighbor points around the center point of each area, that is, , where C is the dimension of the point cloud feature, is the number of points after sampling; Step 4.2) Extract neighborhood set features: Point cloud after sampling grouping After that, LGA is used to extract the grouped local neighborhood features, and each local neighborhood point is input into MLP (MultiLayer Perceptron) and then generated in turn , each local region feature is ,remember for , the calculation formula is as follows, ; ; ; in, MLP is a feed-forward neural network with an input layer, a hidden layer, and an output layer. It generates a description of the input data in the global expression space through multiple calculations of the hidden layer; by calculating the query vector With all key vectors The similarity between them is measured by using the Softmax function to obtain a weight distribution for weighted summation of associated numerical vectors ; Softmax function is a function that converts data to between 0 and 1. C is the dimension of the point cloud feature to prevent gradient explosion. Gradient explosion means that during back propagation, the gradient value increases exponentially with the increase of the number of network layers, resulting in a significant update of the network weights and making the network unstable. is the attention weight vector, which is the fusion weight ordinal number adaptively adjusted according to the significance of global features and local features; finally, the numerical vector After multiplying with the weight vector, the grouped point cloud features are obtained through MLP .

[0013] The process of step 5 is as follows: Step 5.1) Build a neighborhood index and group the point cloud features obtained in step 4 And the grouped point cloud set Use the KNN algorithm in step 3 to get the index map of neighbor points, which is expressed as follows: ; ; in, yes of The index mapping of the nearest point is obtained by querying different indexes o The nearest point area ; Step 5.2), neighborhood propagation, through index mapping To calculate the inverse index mapping , backtrack to another area , query point pass and Get all the elements in two sets and create a new set , ,in is the coordinate information of the lth point, is the feature of the lth point, k_n is The number of all points that can be contacted (including itself), given a point , indicating that the local area to which it belongs is , g represents the number of associated regions.

[0014] The process of step 6 is as follows: Step 6.1), attention position encoding, the query point obtained in step 5.2) and its associated points Input into MLP to upgrade the feature dimension to , then, by And the associated points Calculate the geometric relative position information , is the number of distances between points; finally, the point features in step 5.2) Q, K, V are obtained through MLP in turn, and the position relationship Pos and geometric representation G are obtained by the following formula: ; ; ; in, is a geometric representation learnable matrix, is the position relationship learnable matrix; Step 6.2), azimuth position encoding, the query point obtained in step 5.2) and its associated points And features Input to the azimuth position encoding APE (Azimuth Position Encoding) for encoding. APE represents the position relationship between point clouds in the polar coordinate system. ; ; ; ; in, is a linear transformation and normalization operation. In the local area constructed in step 5.2), is the radius between the point and the center of the circle, as well as It shows the angle relationship with the center of the circle, enhancing the position information; Step 6.3), feature mixing, spatial shape position encoding ss-RPE is composed of the attention position encoding of step 6.1) and the azimuth position encoding of step 6.2. ss-RPE combines the position relationship Pos obtained in step 6.1) and the features obtained by APE in step 6.2) And point cloud features The final features are accumulated and expressed as follows: ; ; ; in, is the feature obtained after encoding, To query the relationship between the locations of points, The weights encoding the attention positions.

[0015] In step 7, different channels usually show strong geometric relationships in the point cloud. Therefore, reweighting is performed according to the different semantic information represented between the point channels. Specifically, the features obtained in step 6.3 are Projection generation and as well as , embed the query into and key embedding The weight matrix is ​​obtained by matrix product of , then the matrix is ​​split and repeated operations are performed to obtain Channel Matrix , propagate the spatial information of the point to each channel, and obtain the information relationship between the points, After the softmax normalization function, the weight of a single channel is Element level with key embedding Multiply to maintain channel differences, and then update The characteristics of this process are expressed as, ; ; ; ; in, is a linear projection. Linear projection is an important concept in linear algebra and mathematical analysis. It maps a vector space to its subspace. is a repeated operation function, according to Copy D copies and join them together .

[0016] In step 8, after the point cloud set P passes through steps 4, 5, 6 and 7, it is extracted again through step 4 to obtain the local final point cloud fine-grained features. , ; Among them, LGA(*) means to perform step 4 again. is the encoding feature obtained after step 6, is the point after encoding in step 6.

[0017] In step 9, according to the point cloud set P constructed in step 1, a few key shape points (Key Shapepoint, KSP) are used to capture the key shape areas of different objects. The positions of KSP points change dynamically according to the shapes of the objects. Therefore, several KSP points are used to represent the specific deformation of such objects. The formula is as follows: ; ; ; in, is the offset of the point, is the calculated KSP point, The linear layer is a basic layer in the neural network, also known as the fully connected layer or dense layer. The role of the linear layer in the neural network is to perform linear transformation on the input data. GSFM receives the coordinate information of the grouped points after step 4.1) and the coordinate information P of all points as well as the features. For the convenience of representation, the grouped points are taken as the neighborhood bounding sphere and all points are taken as the entire point cloud bounding sphere. The volume ratio of the neighborhood bounding sphere and the point cloud bounding sphere is calculated respectively. As a slight deformation of the entire object, and then combined with the specific deformation of the object, a more effective global context feature is obtained from the shape information in the 3D point. ; ; ; in, yes The volume of the corresponding neighborhood bounding sphere, is the volume of the point cloud bounding sphere, It's splicing. is the coordinate of the KSP point, It is used to indicate the location of the local neighborhood. is the original point cloud feature, is the feature of KSP point, is the final point cloud feature.

[0018] The beneficial effects of the present invention are mainly manifested in: 1. The present invention proposes a method for extracting local features of point clouds. The method first extracts features from local areas of point clouds and reduces the amount of calculation by grouping point clouds. Then, query points are used to transmit information between different areas to allow features to propagate between neighbors, thereby obtaining contextual information of the point clouds.

[0019] 2. The present invention designs a new global shape feature extraction module. In order to further expand the effective receptive field, the module uses a few Key Shape Points (KSP) to capture the key shape areas of different objects to expand the receptive field and obtain global context features.

[0020] 3. The present invention designs a new position encoding: Spatial Shape Position Encoding (SS-RPE) captures the position information between points, promotes communication between points, and interacts with current features by calculating the difference in relative position information of spatial shapes. Therefore, it can simultaneously process the disordered features of point clouds and adaptively enhance position information.

[0021] 4. The present invention can effectively improve the accuracy of three-dimensional point cloud tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a network structure diagram.

[0023] Figure 2 This is the SS-RPE structure diagram. DETAILED DESCRIPTION

[0024] The present invention will be further described below in conjunction with the accompanying drawings.

[0025] Reference Figure 1 and Figure 2 , a 3D point cloud task analysis method integrating local features and global context information, the method comprising the following steps: Step 1: Build a point cloud dataset. The process is as follows: Step 1.1), define the data set; In step 1.1), the point cloud data is composed of a series of three-dimensional coordinates, each of which contains the coordinate information of the point cloud in three-dimensional space. As well as the information f reflecting the texture features of the point cloud data, the point cloud data set containing N points is recorded as ,in is the coordinate information of the i-th point. The texture feature set corresponding to P is recorded as , is the color information of the i-th point.

[0026] Step 1.2), point cloud dataset preprocessing: For the point cloud dataset P, random rotation, scaling and noise addition preprocessing operations are performed to generate point cloud datasets under various conditions, thereby improving the method's ability to resist interference with data changes. The process is as follows: Step 1.2.1), random rotation, randomly generate a rotation angle , and according to the angle Constructing a 3D coordinate rotation matrix , its expression is, ; Combine the point cloud dataset P with the rotation matrix Multiply them together to get the original point cloud rotated around the z axis Point cloud dataset after angle , the expression is, ; Step 1.2.2), random scaling, randomly generate scaling ratio , the zoom ratio Multiply with the rotated point cloud dataset Generate scaled data , the expression is, ; Step 1.2.3), add noise and randomly generate a set of point cloud datasets Three-dimensional coordinates of the same dimension , The value of is small, the scaled data set and noise Adding, adding tiny noise interference to the original point cloud, .

[0027] Step 2: Sampling of point cloud datasets. In order to alleviate the computational burden caused by the huge number of point clouds in the point cloud dataset, the number of point clouds in the original point cloud dataset is downsampled to a fixed value through the farthest point sampling (FPS) method. The process of step 2 is as follows: Step 2.1), determine the sampling initial point, and Traverse and calculate the average value of all points in the three coordinate dimensions, and use the calculated average value as the initial point for sampling the farthest point , and at the same time, add it to the sampling point set ; Step 2.2), calculate the distance matrix, the farthest point sampling is to select the distance from the point cloud data set The farthest point, therefore, in order to obtain point cloud data With sampling point set The distance is obtained through the distance matrix Record each point with The Euclidean distance between the newly added points; Step 2.3), update the sampling point set, The point corresponding to the maximum value is denoted by and add it to ,along with Update sampling points, The Euclidean distance between the newly added sampling points and the point cloud dataset will be recalculated, and the point clouds whose Euclidean distance is less than the original value will be updated at the same time; finally, the above operations are repeated to increase the number of point clouds in the sampling point set to a fixed value.

[0028] Step 3, grouping of point cloud data sets: In three-dimensional geometry, shapes can be represented by a set of closely spaced three-dimensional coordinates. For the same target geometry, their point cloud data will be similar in spatial features and semantic expression. Therefore, when performing point cloud tasks, extracting the semantic information of the target geometry depends on the correlation between each point and its surrounding neighborhood points. KNN (K-Nearest Neighbors) is a commonly used algorithm for determining the neighborhood points of each point. Calculating the distance between each point and its nearest K neighbors can better understand the correlation between point cloud data. Grouping point cloud data through the KNN algorithm can effectively capture this local spatial relationship, which helps to extract the semantic information of point cloud data;

[0029] Step 4: Local feature extraction: first construct a neighborhood set, then extract neighborhood set features; The process of step 4 is as follows: Step 4.1) Build the neighborhood set: According to the point cloud set P constructed in step 1, KPconv is used to extract local features. KPconv is a mature point cloud network architecture that proposes kernel point convolution and simulates deformation convolution to obtain local information of point clouds. Then, the FPS algorithm in step 2 is used to obtain The center point is obtained local area, and then use the point cloud sampled by the KNN algorithm in step 3 Divide into N / K local area blocks, each area contains K points, and group the K nearest neighbor points around the center point of each area, that is, , where C is the dimension of the point cloud feature, is the number of points after sampling; Step 4.2) Extract neighborhood set features: Point cloud after sampling grouping After that, LGA is used to extract the grouped local neighborhood features, and each local neighborhood point is input into MLP (MultiLayer Perceptron) and then generated in turn , each local region feature is ,remember for , the calculation formula is as follows, ; ; ; in, MLP is a feed-forward neural network with an input layer, a hidden layer, and an output layer. It generates a description of the input data in the global expression space through multiple calculations of the hidden layer; by calculating the query vector With all key vectors The similarity between them is measured by using the Softmax function to obtain a weight distribution for weighted summation of associated numerical vectors ; Softmax function is a function that converts data to between 0 and 1. C is the dimension of the point cloud feature to prevent gradient explosion. Gradient explosion means that during back propagation, the gradient value increases exponentially with the increase of the number of network layers, resulting in a significant update of the network weights and making the network unstable. is the attention weight vector, which is the fusion weight ordinal number adaptively adjusted according to the significance of global features and local features; finally, the numerical vector After multiplying with the weight vector, the grouped point cloud features are obtained through MLP .

[0030] Step 5: Neighborhood feature propagation: In order to capture the separated local area feature relationship, Establish neighborhood relationships between query points in different areas to allow point cloud features to propagate between neighbors and obtain more fine-grained feature information; The process of step 5 is as follows: Step 5.1) Build a neighborhood index and group the point cloud features obtained in step 4 And the grouped point cloud set Use the KNN algorithm in step 3 to get the index map of neighbor points, which is expressed as follows: ; ; in, yes of The index mapping of the nearest point is obtained by querying different indexes o The nearest point area ; Step 5.2), neighborhood propagation, through index mapping To calculate the inverse index mapping , backtrack to another area , query point pass and Get all the elements in two sets and create a new set , ,in is the coordinate information of the lth point, is the feature of the lth point, k_n is The number of all points that can be contacted (including itself), given a point , indicating that the local area to which it belongs is , g represents the number of associated regions.

[0031] Step 6: Construct spatial shape position coding, first construct attention position coding and azimuth position coding, and then perform feature mixing; The process of step 6 is as follows: Step 6.1), attention position encoding, the query point obtained in step 5.2) and its associated points Input into MLP to upgrade the feature dimension to , then, by And the associated points Calculate the geometric relative position information , is the number of distances between points; finally, the point features in step 5.2) Q, K, V are obtained through MLP in turn, and the position relationship Pos and geometric representation G are obtained by the following formula: ; ; ; in, is a geometric representation learnable matrix, is the position relationship learnable matrix; Step 6.2), azimuth position encoding, the query point obtained in step 5.2) and its associated points And features Input to the azimuth position encoding APE (Azimuth Position Encoding) for encoding. APE represents the position relationship between point clouds in the polar coordinate system. ; ; ; ; in, is a linear transformation and normalization operation. In the local area constructed in step 5.2), is the radius between the point and the center of the circle, as well as It shows the angle relationship with the center of the circle, enhancing the position information; Step 6.3), feature mixing, spatial shape position encoding ss-RPE is composed of the attention position encoding of step 6.1) and the azimuth position encoding of step 6.2). ss-RPE combines the position relationship Pos obtained in step 6.1) and the features obtained by APE in step 6.2) And point cloud features The final features are accumulated and expressed as follows: ; ; ; in, is the feature obtained after encoding, To query the relationship between the locations of points, The weights encoding the attention positions.

[0032] Step 7: Construct channel attention; In step 7, different channels usually show strong geometric relationships in the point cloud. Therefore, the features obtained in step 6.3) are reweighted according to the different semantic information represented between the point channels. Projection generation and as well as , embed the query into and key embedding The weight matrix is ​​obtained by matrix product of , then the matrix is ​​split and repeated operations are performed to obtain Channel Matrix , propagate the spatial information of the point to each channel, and obtain the information relationship between the points, After the softmax normalization function, the weight of a single channel is Element level with key embedding Multiply to maintain channel differences, and then update The characteristics of this process are expressed as, ; ; ; ; in, is a linear projection. Linear projection is an important concept in linear algebra and mathematical analysis. It maps a vector space to its subspace. is a repeated operation function, according to Copy D copies and join them together .

[0033] Step 8: Local feature enhancement In step 8, after the point cloud set P passes through steps 4, 5, 6 and 7, it is extracted again through step 4 to obtain the local final point cloud fine-grained features. , ; Among them, LGA(*) means to perform step 4 again. is the encoding feature obtained after step 6, is the point after encoding in step 6.

[0034] Step 9: extracting global context features, using the Global Shape Feature Module (GSFM) for global context feature extraction; In step 9, according to the point cloud set P constructed in step 1, a few key shape points (Key Shapepoint, KSP) are used to capture the key shape areas of different objects. The positions of KSP points change dynamically according to the shapes of the objects. Therefore, several KSP points are used to represent the specific deformation of such objects. The formula is as follows: ; ; ; in, is the offset of the point, is the calculated KSP point, The linear layer is a basic layer in the neural network, also known as the fully connected layer or dense layer. The role of the linear layer in the neural network is to perform linear transformation on the input data. GSFM receives the coordinate information of the grouped points after step 4.1) and the coordinate information P of all points as well as the features. For the convenience of representation, the grouped points are taken as the neighborhood bounding sphere and all points are taken as the entire point cloud bounding sphere. The volume ratio of the neighborhood bounding sphere and the point cloud bounding sphere is calculated respectively. As a slight deformation of the entire object, and then combined with the specific deformation of the object, a more effective global context feature is obtained from the shape information in the 3D point. ; ; ; in, yes The volume of the corresponding neighborhood bounding sphere, is the volume of the point cloud bounding sphere, It's splicing. is the coordinate of the KSP point, It is used to indicate the location of the local neighborhood. is the original point cloud feature, is the feature of KSP point, is the final point cloud feature.

[0035] In the Transformer model, the Feed-Forward Neural Network (FFN) refers to one of the core components of the Transformer. It is located after each encoder and decoder layer of the Transformer. The Feed-Forward Neural Network is a fully connected feed-forward neural network consisting of two linear transformations (fully connected layers) and a nonlinear activation function.

[0036] Step 10: 3D point cloud classification and segmentation tasks. The process is as follows: Step 10.1) Build a classification task: Figure 1 As shown on the right, the present invention proposes a shape feature propagation module (SAPFormer Block) consisting of steps 1 to 8, such as Figure 1 As shown in the upper part, for the 3D point cloud classification task, the primary point cloud is obtained by point cloud embedding. The feature encoder consists of a downsampling layer and a SAPFormer module. The classification result is obtained by global average pooling (GAP) and MLP. The downsampling layer is a layer used in deep learning models to reduce the spatial dimensions (such as height and width) of the input data. Global average pooling is a method used in deep learning models. Unlike traditional pooling layers (such as maximum pooling or average pooling), the purpose of the global average pooling layer is to convert the entire feature map into a single average value, which helps to simplify the model and directly generate class labels after feature extraction. Point Embedding is used to extract primary features of point clouds, such as MLP, Kpconv network architecture;

[0037] Step 10.2), build segmentation task: Figure 1As shown in the lower part, for the 3D point cloud segmentation task, the U-Net architecture is used. The encoder is the same as the classification task in step 10.1). After each stage is fed to the corresponding upsample, the segmentation result is accumulated. U-Net is a deep learning architecture that effectively processes a small amount of training data and can generate high-resolution segmented images. It is mainly used to restore the spatial dimension of the feature map and increase its height and width. Upsampling can convert low-resolution feature maps into high-resolution outputs.

[0038] This embodiment uses the ShapeNetPart dataset and the ModelNet40 dataset to perform performance testing on the present invention. The testing process includes the following steps: Step 1: Define the comparison method as follows: PointNet: Introduced an MLP-based approach to solve 3D point cloud tasks.

[0039] Kpconv: proposed kernel point convolution, which simulated deformation convolution to obtain local information of point cloud.

[0040] PAconv: proposed position adaptive convolution to obtain point cloud information and simulate complex 3D position changes.

[0041] Step 2, define evaluation indicators: This paper uses Instance IoU (ins. mIoU) and class IoU (cls.mIoU) as evaluation indicators for part segmentation tasks, and uses the average accuracy of each category (mAcc) and the overall accuracy of all categories (OA) as evaluation indicators for shape classification. ; ; ; ; in, is the total number of shape categories in the dataset (in the ShapeNet dataset = 16), represents the number of instances of the i-th class, N represents the number of all shape classes, is the number of correct predictions for the ith class, is the number of samples in the ith class, is the number of classes in the dataset, is the total number of samples in the dataset; Step 3, data set introduction, is as follows: 3.1) Introduction to part segmentation dataset; ShapeNetPart is a dataset for point cloud part segmentation, which consists of 16,880 3D models from 16 different shape categories (such as airplanes and chairs), of which 14,006 models are used for training and 2,874 for testing. The number of parts in each category ranges from 2 to 6, with a total of 50 different parts.

[0042] 3.2) Introduction to shape classification dataset; ModelNet40 is a shape classification dataset, which consists of 9843 training models and 2468 testing CAD models belonging to 40 categories, and it contains 15,000 objects from 15 classes.

[0043] 3.3) Evaluation of part segmentation experimental results; Tables 1 and 2 list the part segmentation results of the present invention on ShapeNetPart. It can be seen from the table that the method is about 0.7% higher than the method based on point convolution paconv and other methods. Moreover, the present invention achieves the best and suboptimal performance in Instance IoU and classIoU respectively. In addition, the part segmentation effect on objects such as car, lamp, laptop, mug, and table is optimal, which verifies the effectiveness of the present invention in point cloud part segmentation tasks.

[0044] Table 1 shows the part segmentation results of the present invention on ShapeNetPart; method Cls.mIoU Ins.mIoU Airplane Bag Cap Car Chair EarPhone Guitar PointNet 80.4 83.7 83.4 78.7 82.5 74.9 89.6 73.0 91.5 paconv 84.6 86.1 84.3 85.0 90.4 79.7 90.6 80.8 92.0 Kpconv 85.1 86.4 84.6 86.3 87.2 81.1 91.1 77.8 92.6 SAPFormer 85.0 86.8 84.9 86.0 89.9 81.3 91.8 75.4 92.1 Table 2 shows the part segmentation results of the present invention on ShapeNetPart (Table 2 is a continuation of Table 1); method Knife Lamp Laptop MotorBike Mug Pistol Rocket SkateBoard table PointNet 85.9 80.8 95.3 65.2 93.0 81.2 57.9 72.8 80.6 paconv 88.7 82.2 95.9 73.9 94.7 84.7 65.9 81.4 84.0 Kpconv 88.4 82.7 96.2 78.1 95.8 85.4 69.0 82.0 83.6 SAPFormer 87.9 86.4 96.3 76.7 95.8 84.5 68.0 78.7 84.4 3.4) Evaluation of shape classification experimental results; Table 3 lists the shape classification results of the present invention on ModelNet40. It can be seen from the table that compared with the advanced methods in recent years, the present invention has achieved suboptimal performance in both OA and mAcc, which verifies the effectiveness of the present invention in point cloud classification tasks.

[0045] Table 3 shows the shape classification results of the present invention on ModelNet40; method mAcc OA PointNet 86.2 89.2 paconv - 93.6 kpconv - 92.9 SAPFormer <![CDATA[ 90.9 ]]> <![CDATA[ 93.9 ]]> The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A 3D point cloud task analysis method integrating local features and global context information, characterized in that: The method comprises the following steps: Step 1: Build a point cloud dataset. The process is as follows: Step 1.1), define the data set; Step 1.2), point cloud dataset preprocessing: for the point cloud dataset P, random rotation, scaling and noise addition preprocessing operations are performed to generate point cloud datasets under various conditions; Step 2: Sampling the point cloud dataset: downsampling the number of point clouds in the original point cloud dataset to a fixed value by sampling the farthest point FPS. Step 3: Grouping of point cloud data sets: grouping of point cloud data sets using the KNN algorithm; Step 4: Local feature extraction: first construct a neighborhood set, then extract neighborhood set features; Step 5: Neighborhood feature propagation: In order to capture the separated local area feature relationship, Establish neighborhood relationships between query points in different areas to allow point cloud features to propagate between neighbors and obtain more fine-grained feature information; Step 6: Construct spatial shape position coding, first construct attention position coding and azimuth position coding, and then perform feature mixing; Step 7: Construct channel attention; Step 8: Local feature enhancement Step 9: extracting global context features, using a global shape feature extraction module GSFM for global context feature extraction; Step 10: 3D point cloud classification and segmentation tasks. The process is as follows: Step 10.1, construct the classification task: For the 3D point cloud classification task, the primary point cloud is obtained by point cloud embedding. The feature encoder consists of a downsampling layer and a SAPFormer module. The classification result is obtained by global average pooling and MLP. Step 10.2, construct segmentation task: For the 3D point cloud segmentation task, the U-Net architecture is used. The encoder is the same as the classification task in step 10.

1. After each stage is fed to the corresponding upsampling, the segmentation result is accumulated.

2. The three-dimensional point cloud task analysis method for integrating local features and global context information according to claim 1, characterized in that: In step 1.1), the point cloud data is composed of a series of three-dimensional coordinates, each of which contains the coordinate information of the point cloud in three-dimensional space. As well as the information f reflecting the texture features of the point cloud data, the point cloud data set containing N points is recorded as ,in is the coordinate information of the i-th point, and the texture feature set corresponding to P is recorded as , is the color information of the i-th point.

3. The three-dimensional point cloud task analysis method for integrating local features and global context information according to claim 1 or 2, characterized in that: The process of step 1.2) is: Step 1.2.1), random rotation, randomly generate a rotation angle , and according to the angle Constructing a 3D coordinate rotation matrix , its expression is, ； Combine the point cloud dataset P with the rotation matrix Multiply them together to get the original point cloud rotated around the z axis Point cloud dataset after angle , the expression is, ； Step 1.2.2), random scaling, randomly generate scaling ratio , the zoom ratio Multiply with the rotated point cloud dataset Generate scaled data , the expression is, ； Step 1.2.3), add noise and randomly generate a set of point cloud datasets Three-dimensional coordinates of the same dimension , The value of is small, the scaled data set and noise Adding, adding tiny noise interference to the original point cloud, 。 4. The three-dimensional point cloud task analysis method for integrating local features and global context information according to claim 3, characterized in that: The process of step 2 is as follows: Step 2.1), determine the sampling initial point, and Traverse and calculate the average value of all points in the three coordinate dimensions, and use the calculated average value as the initial point for sampling the farthest point , and at the same time, add it to the sampling point set ; Step 2.2), calculate the distance matrix, the farthest point sampling is to select the distance from the point cloud data set The farthest point, therefore, in order to obtain point cloud data With sampling point set The distance is obtained through the distance matrix Record each point with The Euclidean distance between the newly added points; Step 2.3), update the sampling point set, The point corresponding to the maximum value is denoted by and add it to ,along with Update sampling points, The Euclidean distance between the newly added sampling points and the point cloud dataset will be recalculated, and the point clouds whose Euclidean distance is less than the original value will be updated at the same time; finally, the above operations are repeated to increase the number of point clouds in the sampling point set to a fixed value.

5. The three-dimensional point cloud task analysis method for fusing local features and global context information according to claim 1 or 2, characterized in that: The process of step 4 is as follows: Step 4.1) Build the neighborhood set: According to the point cloud set P constructed in step 1, KPconv is used to extract local features. KPconv is a mature point cloud network architecture that proposes kernel point convolution and simulates deformation convolution to obtain local information of point clouds. Then, the FPS algorithm in step 2 is used to obtain The center point is obtained local area, and then use the point cloud sampled by the KNN algorithm in step 3 Divide into N / K local area blocks, each area contains K points, and group the K nearest neighbor points around the center point of each area, that is, , where C is the dimension of the point cloud feature, is the number of points after sampling; Step 4.2) Extract neighborhood set features: Point cloud after sampling grouping After that, LGA is used to extract the grouped local neighborhood features, and each local neighborhood point is input into MLP to generate , each local region feature is ,remember for , the calculation formula is as follows, ； ； ； in, MLP is a feed-forward neural network with an input layer, a hidden layer, and an output layer. It generates a description of the input data in the global expression space through multiple calculations of the hidden layer; by calculating the query vector With all key vectors The similarity between them is measured by using the Softmax function to obtain a weight distribution for weighted summation of associated numerical vectors ; Softmax function is a function that converts data to between 0 and 1. C is the dimension of the point cloud feature to prevent gradient explosion. Gradient explosion means that during back propagation, the gradient value increases exponentially with the increase of the number of network layers, resulting in a significant update of the network weights and making the network unstable. is the attention weight vector, which is the fusion weight ordinal number adaptively adjusted according to the significance of global features and local features; finally, the numerical vector After multiplying with the weight vector, the grouped point cloud features are obtained through MLP .

6. The three-dimensional point cloud task analysis method for integrating local features and global context information according to claim 5, characterized in that: The process of step 5 is as follows: Step 5.1) Build a neighborhood index and group the point cloud features obtained in step 4 And the grouped point cloud set Use the KNN algorithm in step 3 to get the index map of neighbor points, which is expressed as follows: ； ； in, yes of The index mapping of the nearest point is obtained by querying different indexes o The nearest point area ; Step 5.2), neighborhood propagation, through index mapping To calculate the inverse index mapping , backtrack to another area , query point pass and Get all the elements in two sets and create a new set , ,in is the coordinate information of the lth point, is the feature of the lth point, k_n is The number of all points that can be contacted, given a point , indicating that the local area to which it belongs is , g represents the number of associated regions.

7. The three-dimensional point cloud task analysis method for fusing local features and global context information according to claim 6, characterized in that: The process of step 6 is as follows: Step 6.1), attention position encoding, the query point obtained in step 5.2) and its associated points Input into MLP to upgrade the feature dimension to , then, by And the associated points Calculate the geometric relative position information , is the number of distances between points; finally, the point features in step 5.2) Q, K, and V are obtained through MLP in turn. The position relationship Pos and the geometric representation G are obtained by the following formula: ； ； ； in, is a geometric representation learnable matrix, is the position relationship learnable matrix; Step 6.2), azimuth position encoding, the query point obtained in step 5.2) and its associated points And features Input to the azimuth position encoding APE for encoding. APE represents the position relationship between point clouds in the polar coordinate system. ； ； ； ； in, is a linear transformation and normalization operation. In the local area constructed in step 5.2), is the radius between the point and the center of the circle, as well as It shows the angle relationship with the center of the circle, enhancing the position information; Step 6.3), feature mixing, spatial shape position encoding ss-RPE is composed of the attention position encoding of step 6.1) and the azimuth position encoding of step 6.2). ss-RPE combines the position relationship Pos obtained in step 6.1) and the features obtained by APE in step 6.2) And point cloud features The final features are accumulated and expressed as follows: ； ； ； in, is the feature obtained after encoding, To query the relationship between the locations of points, The weights encoding the attention positions.

8. The three-dimensional point cloud task analysis method for fusing local features with global context information according to claim 7, characterized in that: In step 7, the features obtained in step 6.3) Projection generation , as well as , embed the query into and key embedding The weight matrix is ​​obtained by matrix product of , then the matrix is ​​split and repeated operations are performed to obtain Channel Matrix , propagate the spatial information of the point to each channel, and obtain the information relationship between the points, After the softmax normalization function, the weight of a single channel is Element level with key embedding Multiply to maintain channel differences, and then update The characteristics of this process are expressed as, ； ； ； ； in, is a linear projection. Linear projection is an important concept in linear algebra and mathematical analysis. It maps a vector space to its subspace. is a repeated operation function, according to Copy D copies and join them together .

9. The three-dimensional point cloud task analysis method for fusing local features and global context information according to claim 8, characterized in that: In step 8, after the point cloud set P passes through steps 4, 5, 6 and 7, it is extracted again through step 4 to obtain the local final point cloud fine-grained features. , ； Among them, LGA(*) means to perform step 4 again. is the encoding feature obtained after step 6, is the point after encoding in step 6.

10. The three-dimensional point cloud task analysis method for fusing local features with global context information according to claim 9, characterized in that: In step 9, according to the point cloud set P constructed in step 1, a few key shape points KSP are used to capture the key shape areas of different objects. The positions of the KSP points change dynamically according to the shapes of the objects. Therefore, several KSP points are used to represent the specific deformation of such objects. The formula is as follows: ； ； ； in, is the offset of the point, is the calculated KSP point, is the feature of the input point; GSFM receives the coordinate information of the grouped points after step 4.1) and the coordinate information P of all points as well as the features. For the convenience of representation, the grouped points are taken as the neighborhood bounding sphere and all points are taken as the entire point cloud bounding sphere. The volume ratio of the neighborhood bounding sphere and the point cloud bounding sphere is calculated respectively. As a slight deformation of the entire object, and then combined with the specific deformation of the object, a more effective global context feature is obtained from the shape information in the 3D point. ； ； ； in, yes The volume of the corresponding neighborhood bounding sphere, is the volume of the point cloud bounding sphere, It's splicing. is the coordinate of the KSP point, It is used to indicate the location of the local neighborhood. is the original point cloud feature, is the feature of KSP point, is the final point cloud feature.

Images