Point cloud classification and segmentation method based on point cloud channel attention feature fusion mechanism

Abstract

The invention discloses a point cloud classification and segmentation method based on a point cloud channel attention feature fusion mechanism, and the method builds a deep learning model, comprises a down-sampling module which carries out the re-sampling of points through a farthest point sampling algorithm, employs K neighbors for each sampling point, and carries out the segmentation of the points. Aggregation is carried out through maximum pooling, and local points are converted into normal distribution through a geometric mapping module; a double-branch input structure is constructed, one branch inputs a multi-layer perceptron layer with residual connection to perform weight learning on neighborhood features of each point of a point cloud, the other branch inputs a multi-scale channel attention fusion module, and channel attention is extracted through two branches with different scales. A multi-scale channel attention mechanism model is designed, local and global features are combined, and an attention mechanism is adopted in a feature space to fuse multi-scale information of point cloud features. According to the method, a channel attention fusion model is designed to solve the problem that the capability of capturing a remote dependency relationship by using convolution in a traditional point cloud classification and segmentation task is limited, a double-branch input network structure is adopted in a feature extraction stage, and dimension reduction operation is performed on a channel during input; the method has the advantages that the accuracy of classification and segmentation tasks can be improved, and meanwhile, the calculation amount is reduced. The method has great advantages in point cloud classification and point cloud segmentation application tasks.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional point cloud analysis and computer vision technology, and involves a point cloud classification and segmentation method based on a point cloud channel attention feature fusion mechanism. Background Art

[0002] In recent years, point cloud processing has become an important research direction in the field of 3D computer vision. Point cloud is a data format used to depict the surface of an object or scene in 3D space. Unlike traditional pixel-based 2D data sets, point cloud contains a set of unordered points. Point cloud data points contain spatial position information and other information such as color and normal vector. However, due to the disordered and irregular nature of point cloud data, it is impossible to directly apply image processing methods to point cloud analysis.

[0003] In order to effectively deal with the special properties of point clouds, researchers have proposed many methods, which are mainly divided into voxel-based and original point cloud-based methods. The voxel-based method is to voxelize the point cloud and then use deep convolution to achieve deep learning tasks. The voxel-based method uses the means of regularizing point cloud data to equally separate the space within the point cloud range along the x, y, and z axes, thereby obtaining equal-sized voxel grids as a regular representation of the original point cloud. The number of point clouds in the grid is used as the basic feature of the voxel grid. After that, the 2D convolution is expanded to 3D, and the convolution kernel becomes 3D. The point cloud can be processed according to a model and method similar to the 2D convolutional neural network. The original point cloud-based method does not need to convert the point cloud into a regular method through special means, and directly inputs the point cloud data into the network.

[0004] Many existing deep neural networks for point cloud analysis can more effectively expand the receptive field of point cloud features by constructing deeper convolutional networks and increasing the number of input neighborhoods, but this approach significantly increases the computational cost, and the ability of convolution to capture long-range dependencies in point cloud analysis is limited. Because understanding the global shape context is crucial for real-world data, it leads to poor performance on tasks related to point cloud analysis. Summary of the invention

[0005] The purpose of this invention is to provide a point cloud classification and segmentation method based on a point cloud channel attention feature fusion mechanism, aiming to overcome the problem of limited ability of using convolution to capture long-range dependencies in traditional point cloud classification and segmentation tasks, while effectively reducing the amount of computation.

[0006] The present invention provides a method for extracting features of point clouds at different scales by constructing a multi-layer perceptron layer with residual connections and two input branches of multi-scale channel attention. The multi-layer perceptron layer mainly learns the shared weights of point cloud features, and the introduction of a multi-scale channel attention module can focus on the connection between channels in point cloud features.

[0007] The main purpose of constructing a dual branch is to enable the network to focus on the original information of the point cloud while paying attention to the differences between the point cloud features. When constructing a dual-branch network, the channel dimension is reduced, which reduces the amount of calculation and increases the accuracy and robustness of the model.

[0008] The specific technical solutions of the present invention are as follows:

[0009] Step S1: In the downsampling stage, the given point cloud data N is downsampled by using the farthest point sampling algorithm. S Point downsampling.

[0010] Step S2: Use spherical query to obtain the neighboring points of each downsampled point in the point cloud to form a local neighborhood.

[0011] Step S3: The local neighborhood is transformed into a normal distribution through a geometric mapping module while maintaining the original geometric properties.

[0012] Step S4: Construct a dual-branch input structure, where one branch inputs a multi-layer perceptron layer with residual connections to learn weights for the neighborhood features of each point in the point cloud, and then aggregates the features through a maximum pooling operation. The other branch inputs a multi-scale channel attention fusion module to extract channel attention through two branches of different scales.

[0013] Step S5: Feature extraction at different scales is achieved by repeatedly executing the channel attention feature fusion network structure.

[0014] Step S6: Output the global shape descriptor and input it into the classifier to perform classification and segmentation tasks of the three-dimensional point cloud.

[0015] In steps S1 and S2, the farthest point sampling algorithm is used to resample the points of the point set, K center points are selected from the input point set, and a certain radius is defined. A spherical query is used to find N points within the radius sphere of the sampling point as neighboring points to form a local feature neighborhood.

[0016] In step S3, the local neighborhood point set is transformed into a normal distribution through geometric mapping. Let {f i,j} is the center point f i The sampled and grouped neighbor point set, each neighbor point is a d-dimensional vector, we call each neighbor point set {f i,j} is mapped using the following formula:

[0017]

[0018] Among them, α and β are learnable parameters, σ is a scalar that describes the feature differences between all neighbor point sets. represents the Hadamard product, It is a numerically stable decimal value that prevents the denominator from being zero.

[0019] In step S4, the multilayer perceptron layer with residual connection is connected by two serially connected MLP+BN+RELU layers, where the multilayer perceptron layer uses forward residual connection to alleviate the problem of gradient disappearance and gradient explosion. The specific design is as follows:

[0020]

[0021] Where A represents the aggregation operation of feature points using the maximum pooling operation.

[0022] The multi-scale channel attention model in step S4 uses convolution to focus on the dependencies between channels and extracts channel attention through two branches of different scales. One branch obtains the global feature weight and the other branch obtains the local feature weight. The local feature extraction input formula is specifically designed as follows:

[0023] L(X)=β(conv1(δ(β(conv2(X)))))

[0024] The β above represents the batch normalization operation and δ represents the activation function.

[0025] The formula for obtaining the global feature weight of another branch is specifically designed as follows:

[0026] G(X)=β(conv1(δ(β(conv2(g(X))))))

[0027] The β above represents the batch normalization operation, δ represents the activation function, and g represents the global maximum pooling operation.

[0028] The present invention combines local and global features by designing a multi-scale channel attention mechanism model, and adopts the attention mechanism in the feature space to fuse the multi-scale information of point cloud features.

[0029] In step S5, feature extraction at different scales is subsequently achieved by repeatedly executing a dual-branch model constructed by a multi-layer perceptron layer and a channel attention fusion module.

[0030] In step S6, when performing the classification task, the global shape is output and the classification result is obtained through the fully connected layer:

[0031]

[0032] When performing the segmentation task, a layered propagation strategy based on distance-based reverse interpolation and cross-layer skip links are used to obtain the point features of all original points:

[0033]

[0034]

[0035] For each point x in P, find the k points closest to it in the original point cloud coordinate space. Use the above formula to weight the features of the k points of x and get the features of x. This weight is inversely related to the distance between x and k points. The farther the distance, the smaller the contribution of the point to the features of x.

[0036] From the above, we can get the point features of each original point, and finally calculate the segmentation result.

[0037] The advantages of the present invention are:

[0038] 1. A channel attention fusion model is designed to overcome the limited ability of convolution to capture long-range dependencies in traditional point cloud classification and segmentation tasks.

[0039] 2. In the feature extraction stage, a dual-branch input network structure is used to reduce the dimension of the channel during input. This has the advantage of improving the accuracy of classification and segmentation tasks while reducing the amount of computation. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a structural diagram of the present invention.

[0041] Figure 2 It is a structural diagram of a downsampling stage of the present invention.

[0042] Figure 3 A dual-branch structure diagram consisting of a residual multi-layer perceptron and a channel attention fusion mechanism of the present invention.

[0043] Figure 4 A schematic diagram of a process of the present invention. DETAILED DESCRIPTION

[0044] The following describes the point cloud classification and segmentation method based on the point cloud channel attention feature fusion mechanism provided by an embodiment of the present invention with reference to the accompanying drawings. First, the overall process architecture, downsampling architecture diagram, and dual-branch structure diagram provided by the design example of the present invention will be described with reference to the accompanying drawings.

[0045] like Figure 1 As shown, the method comprises the following steps

[0046] Step S1: In the downsampling stage, the given point cloud data N is downsampled by using the farthest point sampling algorithm. S Point downsampling.

[0047] Step S2: Use spherical query to obtain the neighboring points of each downsampled point in the point cloud to form a local neighborhood.

[0048] Step S3: The local neighborhood is transformed into a normal distribution through a geometric mapping module while maintaining the original geometric properties.

[0049] Step S4: Construct a dual-branch input structure, where one branch inputs a multi-layer perceptron layer with residual connections to learn weights for the neighborhood features of each point in the point cloud, and then aggregates the features through a maximum pooling operation. The other branch inputs a multi-scale channel attention fusion module to extract channel attention through two branches of different scales.

[0050] Step S5: Feature extraction at different scales is achieved by repeatedly executing the channel attention feature fusion network structure.

[0051] Step S6: Upsample the low-level point cloud features to generate a feature representation that integrates high-level information.

[0052] Step S7: Output the global shape descriptor and input it into the classifier to perform classification and segmentation tasks of the three-dimensional point cloud.

[0053] Step S1 downsampling architecture: Input point cloud data P = {P1, P2, ...., P N},P i Represents the i-th point, the farthest point sampling algorithm gradually selects the point farthest from the currently selected point set:

[0054] FP(P,k)={P i1 ,P i2 ,....,P ik}

[0055] Among them, P ik is the point selected for the kth time, and k is the number of points after downsampling.

[0056] The selection criterion is to maximize the distance of each newly selected point to the nearest already selected point.

[0057] In step S2, the K center points after downsampling are delimited with a certain radius, and a spherical query is used to find N points within the radius sphere of the sampling point as neighboring points to form a local feature neighborhood.

[0058] In step S3, the local neighborhood point set is transformed into a normal distribution through geometric mapping, and {fi,j} is the distribution of the center point f i The sampled and grouped neighbor point set, each neighbor point is a d-dimensional vector, we map each neighbor point set {fi,j} through the following formula:

[0059]

[0060] Among them, α and β are learnable parameters, σ is a scalar that describes the feature differences between all neighbor point sets. represents the Hadamard product, It is a numerically stable decimal value that prevents the denominator from being zero.

[0061] In step S4, the multilayer perceptron layer with residual connection is connected by two serially connected MLP+BN+RELU layers, where the multilayer perceptron layer uses forward residual connection to alleviate the problem of gradient disappearance and gradient explosion. The specific design is as follows:

[0062]

[0063] Where A represents the aggregation operation of feature points using the maximum pooling operation.

[0064] The multi-scale channel attention model in step S4 uses convolution to focus on the dependencies between channels and extracts channel attention through two branches of different scales. One branch obtains the global feature weight and the other branch obtains the local feature weight. The local feature extraction input formula is specifically designed as follows:

[0065] L(X)=β(conv1(δ(β(conv2(X)))))

[0066] The β above represents the batch normalization operation and δ represents the activation function.

[0067] The formula for obtaining the global feature weight of another branch is specifically designed as follows:

[0068] G(X)=β(conv1(δ(β(conv2(g(X))))))

[0069] The β above represents the batch normalization operation, δ represents the activation function, and g represents the global maximum pooling operation.

[0070] The present invention combines local and global features by designing a multi-scale channel attention mechanism model, and adopts the attention mechanism in the feature space to fuse the multi-scale information of point cloud features.

[0071] In step S5, feature extraction at different scales is subsequently achieved by repeatedly executing a dual-branch model constructed by a multi-layer perceptron layer and a channel attention fusion module.

[0072] In step S6, in the upsampling stage, the high-level feature representation is interpolated according to the coordinate information of the low-level point cloud data to obtain the feature values ​​of the corresponding positions, and these interpolated feature values ​​are concatenated with the low-level feature representation to generate a low-level feature representation that integrates the high-level information.

[0073] In step S7, when performing the classification task, the global shape is output and the classification result is obtained through the fully connected layer:

[0074]

[0075] When performing the segmentation task, a layered propagation strategy based on distance-based reverse interpolation and cross-layer skip links are used to obtain the point features of all original points:

[0076]

[0077]

[0078] For each point x in P, find the k points closest to it in the original point cloud coordinate space. Use the above formula to weight the features of the k points of x and get the features of x. This weight is inversely related to the distance between x and k points. The farther the distance, the smaller the contribution of the point to the features of x.

[0079] From the above, we can get the point features of each original point, and finally calculate the segmentation result:

[0080] y i =argmax(pi)

[0081] where y i Corresponding to the category of each point, the argmax function returns the index corresponding to the maximum value in the probability vector, that is, the predicted category label.

Claims

1. A point cloud classification and segmentation method based on point cloud channel attention feature fusion mechanism, characterized in that: The following steps are involved: Step S1: In the downsampling stage, the given point cloud data N is downsampled by using the farthest point sampling algorithm. S Point downsampling. Step S2: Use spherical query to obtain the neighboring points of each downsampled point in the point cloud to form a local neighborhood. Step S3: The local neighborhood is transformed into a normal distribution through a geometric mapping module while maintaining the original geometric properties. Step S4: Construct a dual-branch input structure, where one branch inputs a multi-layer perceptron layer with residual connections to learn weights for the neighborhood features of each point in the point cloud, and then aggregates the features through a maximum pooling operation. The other branch inputs a multi-scale channel attention fusion module to extract channel attention through two branches of different scales. Step S5: Feature extraction at different scales is achieved by repeatedly executing the channel attention feature fusion network structure. Step S6: Upsample the low-level point cloud features to generate a feature representation that integrates high-level information. Step S7: Output the global shape descriptor and input it into the classifier to perform classification and segmentation tasks of the three-dimensional point cloud.

2. According to claim 1, a point cloud classification and segmentation method based on point cloud channel attention feature fusion mechanism is characterized in that: The step S1 includes inputting point cloud data P={P1, P2, ...., P N },P i Represents the i-th point. The farthest point sampling algorithm gradually selects the point farthest from the currently selected point set: FP(P,k)={P i1 ,P i2 ,....,P ik }, where P ik is the kth selected point, k is the number of points after downsampling. The selection criterion is to maximize the distance from each newly selected point to the nearest selected point.

3. According to claim 1, a point cloud classification and segmentation method based on point cloud channel attention feature fusion mechanism is characterized in that: In step S2, the K center points after downsampling are delimited with a certain radius, and a spherical query is used to find N points within the radius sphere of the sampling point as neighboring points to form a local feature neighborhood.

4. According to claim 1, a point cloud classification and segmentation method based on point cloud channel attention feature fusion mechanism is characterized in that: In step S3, the local neighborhood point set is transformed into a normal distribution through geometric mapping. Let {f i,j } is the center point f i The sampled and grouped neighbor point set, each neighbor point is a d-dimensional vector, we call each neighbor point set {f i,j } is mapped using the following formula: Among them, α and β are learnable parameters, σ is a scalar that describes the feature differences between all neighbor point sets. represents the Hadamard product, It is a numerically stable decimal value that prevents the denominator from being zero.

5. According to claim 1, a point cloud classification and segmentation method based on point cloud channel attention feature fusion mechanism is characterized in that: The multi-layer perceptron layer with residual connection in step S4 is connected by two serially connected MLP+BN+RELU layers, wherein the multi-layer perceptron layer uses a forward residual connection to alleviate the problem of gradient disappearance and gradient explosion, and the specific design is as follows: Where A represents the aggregation operation of feature points using the maximum pooling operation. The multi-scale channel attention model in step S4 uses convolution to focus on the dependencies between channels and extracts channel attention through two branches of different scales. One branch obtains the global feature weight and the other branch obtains the local feature weight. The local feature extraction input formula is specifically designed as follows: L(X)=β(conv1(δ(β(conv2(X))))) The β above represents the batch normalization operation and δ represents the activation function. The formula for obtaining the global feature weight of another branch is specifically designed as follows: G(X)=β(conv1(δ(β(conv2(g(X)))))) The β above represents the batch normalization operation, δ represents the activation function, and g represents the global maximum pooling operation. The present invention combines local and global features by designing a multi-scale channel attention mechanism model, aiming to use the attention mechanism in the feature space to fuse the multi-scale information of point cloud features.

6. According to claim 1, a point cloud classification and segmentation method based on point cloud channel attention feature fusion mechanism is characterized in that: In step S5, feature extraction of different scales is achieved by repeatedly executing a dual-branch model constructed by a multi-layer perceptron layer and a channel attention fusion module.

7. The point cloud classification and segmentation method based on the point cloud channel attention feature fusion mechanism according to claim 1 is characterized in that: In step S6, in the upsampling stage, the high-level feature representation is interpolated according to the coordinate information of the low-level point cloud data to obtain the feature values ​​of the corresponding positions, and these interpolated feature values ​​are concatenated with the low-level feature representation to generate a low-level feature representation that integrates the high-level information.

8. The point cloud classification and segmentation method based on the point cloud channel attention feature fusion mechanism according to claim 1 is characterized in that: In step S7, when performing the classification task, the global shape is output and then the classification result is obtained through the fully connected layer: The above formula is the standard form of the Softmax function, which is used to convert the output score of the neural network into a probability distribution. i represents the score of the i-th category. i It is the sum of the exponential functions of all category scores, which is used for normalization so that the sum of the probabilities of all categories is 1. When performing the segmentation task, a layered propagation strategy based on distance-based reverse interpolation and cross-layer skip links are used to obtain the point features of all original points: For each point x in P, find the k points closest to it in the original point cloud coordinate space. Use the above formula to weight the features of the k points of x and get the features of x. This weight is inversely related to the distance between x and k points. The farther the distance, the smaller the contribution of the point to the features of x. From the above, we can get the point features of each original point, and finally calculate the segmentation result: y i =argmax(p i ) where y i Corresponding to the category of each point, the argmax function returns the index corresponding to the maximum value in the probability vector, that is, the predicted category label, thereby obtaining the segmentation result of the test point cloud data.

Images