CNN-Transform-based three-dimensional object classification method and device

Abstract

The invention discloses a CNN-Transform-based three-dimensional object classification method and device, and belongs to the technical field of three-dimensional object classification, and the method comprises the steps: carrying out the sampling of point cloud data based on an FPS algorithm, and obtaining the neighborhood point cloud data; performing convolution processing on the neighborhood point cloud data based on a SENet attention mechanism to obtain local features; performing global learning on the local features through Transform to obtain global features; and inputting the global features into a classifier to obtain a classification result. According to the method, an SENet attention mechanism is introduced in a point cloud local feature aggregation stage, dynamic re-calibration of the features is realized through adaptive learning of importance of different channel features, a dependency relationship between channels can be effectively captured, important geometric features can be highlighted, irrelevant noise information can be suppressed, the expression ability of a model to a local structure is improved, and the robustness of the model is improved. The method can better maintain the local geometric structure information of the point cloud, improves the discrimination of the features, and is of great significance for improving the performance of tasks such as point cloud classification and segmentation.

Description

Technical Field

[0001] This application belongs to the field of three-dimensional object classification technology, specifically relating to a three-dimensional object classification method and apparatus based on CNN-Transformer. Background Technology

[0002] With the rapid development of 3D sensing technologies (such as LiDAR, structured light, and depth cameras), 3D data is increasingly being used in scenarios such as autonomous driving, intelligent manufacturing, robot navigation, AR / VR, and smart cities. How to efficiently and accurately extract features from disordered and unstructured 3D data and achieve object classification has become a core research direction for both academia and industry.

[0003] Different object recognition methods have emerged to address different 3D data representation methods:

[0004] The multi-view method projects 3D point cloud data onto a 2D plane from different angles, and then performs 2D convolution on the 2D plane image. However, due to occlusion issues, a significant amount of inherent information in the 3D point cloud is lost during the 3D-to-2D projection process. Therefore, the multi-view method performs poorly when applied to large-scale scenes or when studying the intrinsic geometric relationships between 3D data.

[0005] Voxel-based methods convert 3D point cloud models into voxel forms that approximate the shape of an object. Each voxel block contains a set of related points, and then a 3D CNN is used to classify the voxels. Compared to multi-view methods, voxel-based methods focus on the relationships between 3D data and group points with internal connections into sets of points, thus constructing voxels. While voxel-based models solve the problems of unordered and unstructured point clouds, they are limited by resolution. Low-resolution methods are faster to train but suffer from significant loss of point cloud information; high-resolution methods offer strong shape description capabilities but significantly increase spatial complexity.

[0006] Therefore, how to better utilize the information of each point in point cloud data and reduce noise and errors caused by computational complexity is a current research hotspot. Summary of the Invention

[0007] The purpose of this application is to provide a 3D object classification method and apparatus based on CNN-Transformer to solve the problem of low accuracy in existing 3D object classification methods based on CNN-Transformer.

[0008] According to a first aspect of the embodiments of this application, a three-dimensional object classification method based on CNN-Transformer is provided, comprising:

[0009] Neighborhood point cloud data is obtained by sampling point cloud data based on the FPS algorithm;

[0010] Local features are obtained by performing convolution processing on the neighborhood point cloud data based on the SENet attention mechanism.

[0011] Global features are obtained by performing global learning on the local features using a Transformer.

[0012] The global features are input into the classifier to obtain the classification result.

[0013] In some optional embodiments of this application, sampling of point cloud data based on the FPS algorithm to obtain neighborhood point cloud data includes:

[0014] A preset number of center points are selected from the point cloud data by sampling at the furthest distance;

[0015] Based on sphere queries, construct neighborhood point cloud data with the central point as the center of the sphere.

[0016] In some optional embodiments of this application, constructing neighborhood point cloud data with the center point as the sphere based on sphere query includes:

[0017] Using the center point as the center, a spherical neighborhood is constructed using multiple spheres of different radii to obtain the neighborhood point cloud data.

[0018] In some optional embodiments of this application, local features are obtained by convolutional processing of the neighborhood point cloud data based on the SENet attention mechanism, including:

[0019] The neighborhood point cloud data is convolutionally processed based on the SENet attention mechanism to obtain initial local features;

[0020] The local features are obtained by multi-scale fusion of multiple initial local features corresponding to each center point.

[0021] In some optional embodiments of this application, local features are obtained by convolutional processing of the neighborhood point cloud data based on the SENet attention mechanism, including:

[0022] The neighborhood point cloud data is convolved to obtain initial local features;

[0023] The initial local features are obtained by applying attention weights based on the SENet attention mechanism.

[0024] In some optional embodiments of this application, convolution processing is performed on the neighborhood point cloud data to obtain initial local features, including:

[0025] A local coordinate system is constructed with the center point as the origin;

[0026] Within the local coordinate system, the relative coordinates are obtained by subtracting the coordinates of the neighborhood point cloud data from the coordinates of the center point.

[0027] The neighborhood point cloud data containing the relative coordinates is used to obtain a feature tensor through a multilayer perceptron.

[0028] Through function The local feature is obtained by taking the maximum value of the feature tensor in the neighborhood point cloud data corresponding to each center point.

[0029] In some alternative embodiments of this application, the multilayer perceptron includes multiple fully connected layers, batch normalization, and activation functions.

[0030] In some optional embodiments of this application, global features are obtained by globally learning the local features using a Transformer, including:

[0031] A self-attention graph of the neighborhood point cloud data is generated based on a point self-attention mechanism;

[0032] The point self-attention features are obtained by weighted summation of the local features based on the self-attention map.

[0033] The local features are mapped onto the query matrix and the key-value matrix to obtain the channel similarity matrix;

[0034] Perform the channel similarity matrix The operation extracts the maximum value for each channel;

[0035] The maximum value is expanded to the same dimension as the channel similarity matrix to obtain the channel maximum value matrix;

[0036] The channel attention weights are obtained based on the difference between the channel maximum value matrix and the channel similarity matrix;

[0037] pass The channel attention matrix is ​​obtained by normalizing the function.

[0038] The local features are weighted based on the channel attention matrix to obtain the channel attention features;

[0039] The global feature is obtained by adding the point self-attention feature and the channel attention feature.

[0040] According to a second aspect of the embodiments of this application, a 3D object classification device based on CNN-Transformer is provided, comprising:

[0041] The sampling module is used to sample point cloud data based on the FPS algorithm to obtain neighborhood point cloud data;

[0042] The feature aggregation module is used to perform convolution processing on the neighborhood point cloud data based on the SENet attention mechanism to obtain local features;

[0043] The global feature learning module is used to perform global learning on the local features through Transformer to obtain global features;

[0044] The classification module is used to input the global features into the classifier to obtain the classification result.

[0045] According to a third aspect of the embodiments of this application, an electronic device is provided, which may include:

[0046] processor;

[0047] Memory used to store processor-executable instructions;

[0048] The processor is configured to execute instructions to implement the CNN-Transformer-based 3D object classification method as described in any embodiment of the first aspect.

[0049] The above-mentioned technical solution of this application has the following beneficial technical effects:

[0050] This application provides a CNN-Transformer-based 3D object classification method. It introduces the SENet attention mechanism during the point cloud local feature aggregation stage, achieving dynamic feature recalibration through adaptive learning of the importance of features from different channels. This design effectively captures inter-channel dependencies, highlights important geometric features, and suppresses irrelevant noise, thereby improving the model's ability to represent local structures. Compared to existing local feature extraction methods, this invention better preserves the local geometric structure information of point clouds and improves feature discriminativity, which is significant for enhancing the performance of point cloud classification and segmentation tasks. Attached Figure Description

[0051] Figure 1 This is a flowchart of a 3D object classification method based on CNN-Transformer in an exemplary embodiment of this application;

[0052] Figure 2 This is a schematic diagram of a 3D object classification model based on CNN-Transformer in an exemplary embodiment of this application;

[0053] Figure 3 This is an exemplary embodiment of the LFA local feature aggregation module architecture diagram in this application;

[0054] Figure 4 This is a schematic diagram of multi-scale feature stitching in an exemplary embodiment of this application;

[0055] Figure 5 This is a PSA architecture diagram in an exemplary embodiment of this application;

[0056] Figure 6 This is a CSA architecture diagram in an exemplary embodiment of this application;

[0057] Figure 7 This is an exemplary embodiment of the present application, showing a global feature learning architecture diagram.

[0058] Figure 8 This is a sample point cloud data diagram of a bathtub in an exemplary embodiment of this application;

[0059] Figure 9 This is a point cloud data map of a bed sample in an exemplary embodiment of this application;

[0060] Figure 10 This is a point cloud data diagram of a bookshelf sample in an exemplary embodiment of this application;

[0061] Figure 11 This is a sample point cloud data map of a car in an exemplary embodiment of this application;

[0062] Figure 12 This is a guitar sample point cloud data map in an exemplary embodiment of this application;

[0063] Figure 13 This is a piano sample point cloud data map in an exemplary embodiment of this application;

[0064] Figure 14 This is a point cloud data map of a plant sample in an exemplary embodiment of this application;

[0065] Figure 15 This is a staircase sample point cloud data map in an exemplary embodiment of this application;

[0066] Figure 16 This is a schematic diagram of a 3D object classification device based on CNN-Transformer in an exemplary embodiment of this application;

[0067] Figure 17 This is a schematic diagram of an electronic device structure according to an exemplary embodiment of this application. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to specific embodiments and accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of this application. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0069] The accompanying drawings illustrate layer structure diagrams according to embodiments of this application. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0070] Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0071] In the description of this application, it should be noted that the terms "first", "second", and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0072] Furthermore, the technical features involved in the different embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0073] In related technologies, static multilayer perceptrons (MLPs) are typically used to extract neighborhood features during local feature extraction. Their weights are fixed after training, which makes it impossible to dynamically adapt to the geometric characteristics (such as edges, planes, curvature) of different input data. This results in the same MLP failing to effectively capture multi-scale local structures (such as edge details of small-radius neighborhoods and the overall shape of large-radius neighborhoods) in sparse or dense point cloud scenarios, leading to a decrease in feature discriminativeness. At the same time, fixed weights cause an imbalance between the contributions of noise-sensitive channels and key feature channels, reducing the model's generalization ability.

[0074] Furthermore, existing related technical solutions typically employ offset attention mechanisms and learnable positional encoding of relative coordinates during global feature learning. Offset attention adjusts attention weights by incorporating positional information to better capture the relationship between global and local features, while positional encoding injects positional information into features by calculating the relative distances between points, helping the model utilize spatial information.

[0075] However, the Transformer based on offset attention mainly learns features in the spatial dimension, ignoring the interactions within the feature channels. This single-dimensional attention mechanism limits the model's ability to model complex feature relationships, resulting in limited performance when processing multi-dimensional feature data.

[0076] To address the two core problems of dynamic feature adaptation deficiency and insufficient multi-dimensional feature interaction in 3D point cloud feature learning, this invention proposes a multi-scale feature fusion method based on dynamic channel attention guidance and a spatial-channel dual-path attention network architecture to improve feature representation capabilities.

[0077] The following description, in conjunction with the accompanying drawings, details a CNN-Transformer-based three-dimensional object classification method and apparatus provided in this application through specific embodiments and application scenarios.

[0078] like Figure 1 As shown in Embodiment 1 of this application, a 3D object classification method based on CNN-Transformer is provided, including the following steps:

[0079] Step S101: Sample the point cloud data based on the FPS algorithm to obtain neighborhood point cloud data;

[0080] Step S102: Perform convolution processing on the neighborhood point cloud data based on the SENet attention mechanism to obtain local features;

[0081] Step S103: Use Transformer to perform global learning on the local features to obtain global features;

[0082] Step S104: Input the global features into the classifier to obtain the classification result.

[0083] Existing technical solutions use static multilayer perceptrons (MLPs) to aggregate neighborhood features. The network weights are fixed after training, which makes it impossible to dynamically adjust the parameters to adapt to the dynamic changes of local geometric features (edges / planes / curvature) of point clouds. For example, when capturing high-frequency edge details in a small radius neighborhood, the fixed weights of the MLP cannot enhance the response of edge-sensitive channels; while when representing the overall shape in a large radius neighborhood, the same MLP is difficult to suppress the interference of noise channels.

[0084] To address this issue, this embodiment constructs a dynamic channel attention-guided multi-scale feature extraction mechanism to achieve dynamic adaptation of local feature extraction. This invention embeds a SENet (Squeeze-and-Excitation Net) attention module into the LFA module of existing technical solutions, capturing channel importance through global average pooling and dynamically adjusting the weights of each channel in the MLP.

[0085] This embodiment introduces the SENet (Squeeze-and-Excitation Network) attention mechanism during the point cloud local feature aggregation stage. By adaptively learning the importance of features from different channels, it achieves dynamic feature recalibration. This design effectively captures the dependencies between channels, highlights important geometric features, and suppresses irrelevant noise, thereby improving the model's ability to express local structures. Compared to existing local feature extraction methods, this invention better preserves the local geometric structure information of the point cloud and improves feature discriminativity, which is of great significance for improving the performance of tasks such as point cloud classification and segmentation.

[0086] In some embodiments, such as Figure 2 As shown, step S103 includes:

[0087] A self-attention map of the neighborhood point cloud data is generated based on a point self-attention mechanism; the local features are weighted and summed based on the self-attention map to obtain point self-attention features; the local features are mapped to a query matrix and a key-value matrix to obtain a channel similarity matrix; the channel similarity matrix is ​​then processed... The operation involves extracting the maximum value for each channel; expanding the maximum value to the same dimension as the channel similarity matrix to obtain a channel maximum value matrix; and obtaining the channel attention weight based on the difference between the channel maximum value matrix and the channel similarity matrix. The channel attention is normalized by a function to obtain a channel attention matrix; the local features are weighted based on the channel attention matrix to obtain channel attention features; the point self-attention features and the channel attention features are added together to obtain the global features.

[0088] Related technical solutions use a Transformer based on offset attention when modeling global features. This method only adjusts the attention weights by spatial position offset, ignores the dependencies between feature channels, and makes it difficult to capture local geometric details and global context information at the same time.

[0089] To address the shortcomings of related technical solutions, this embodiment utilizes point self-attention to study spatial correlation and channel self-attention to study channel correlation. These two attention mechanisms enhance feature representation from the spatial and channel dimensions, respectively, enabling the model to simultaneously capture local geometric patterns (such as edges) and global semantic information (such as overall shape).

[0090] This embodiment proposes a global feature learning strategy based on a dual attention mechanism in the global feature modeling stage. It integrates point self-attention (PSA) and channel self-attention (CSA) to enhance the expressive power of point cloud features from both spatial and channel dimensions, and significantly improve the model's ability to understand and discriminate three-dimensional shape structures.

[0091] In terms of technical details, the point self-attention mechanism introduced in this embodiment aims to model the spatial dependencies between points. Unlike traditional convolutional operations that only perform feature interactions in local neighborhoods, the point self-attention module, based on the Transformer structure, takes a global perspective. By calculating the spatial similarity between points and combining it with learnable positional encoding, it achieves long-distance dependency modeling across regions and structures, capturing the overall morphological features and geometric structural differences of the point cloud. For example, for 3D objects with symmetrical or repetitive spatial structures (such as pianos, bookshelves, etc.), the point self-attention mechanism can establish explicit connections between point pairs at greater distances, thereby preserving global contour information.

[0092] On the other hand, the channel self-attention mechanism models the potential semantic or geometric dependencies between each feature channel. In deep networks, different channels may correspond to different spatial patterns or semantic information. However, traditional feature fusion methods fail to distinguish the importance of channels, easily leading to redundant features interfering with effective expression. This embodiment calculates the similarity and difference between channels through the CSA module, adaptively allocates attention weights, enhances the response to key channels (such as feature channels representing edges, curvature, local mutations, etc.), and suppresses redundant channels, thereby improving the discriminativeness and generalization ability of features.

[0093] After constructing the two attention mechanisms in parallel, their outputs are fused to form the "spatial-channel dual attention feature representation" proposed in this embodiment. This design achieves synergistic enhancement of the spatial and semantic domains, enabling the model to not only accurately perceive spatial structures (such as the distribution relationship of points and symmetrical structures), but also effectively highlight discriminative feature dimensions (such as detail edges and semantic regions), significantly improving the overall feature representation capability.

[0094] like Figure 2 As shown in Embodiment 2 of this application, a 3D object classification model based on CNN-Transformer is provided, which combines local feature aggregation and global feature learning. The core feature of the model is to extract features from the neighborhood of the point cloud through a convolutional neural network and to learn the global features of the point cloud through a Transformer, thereby achieving accurate point cloud classification.

[0095] The model mainly consists of two parts: an LFA local feature aggregation module and a GFL global feature learning module. LFA local feature aggregation captures local structures at different scales through multi-scale grouping. Each scale uses different radii and sample sizes for neighborhood sampling to better capture local information at different scales. After neighborhood sampling, convolution and batch normalization are used to process the local features. GFL global feature learning uses a Transformer to learn global features and capture long-range dependencies.

[0096] For the input raw point cloud data, the local feature aggregation module mainly addresses three issues: First, the input raw point cloud data is often numerous, leading to excessive computation and reduced algorithm speed. Therefore, the local feature aggregation module needs to use appropriate methods to select some relatively important points from the input raw point cloud data as center points. Second, after obtaining the center points, in order to extract local features of the point cloud, it is necessary to define what "local" means for this point, that is, which points are included in the local area of ​​the center point. This involves dividing the point cloud into different regions and obtaining the local features of different regions. Finally, unlike image data, which is distributed on a regular pixel grid with uniform data density, point cloud data is irregularly and unevenly distributed in space. When the point cloud is uneven, if the same sphere radius is used when partitioning each sub-region, fewer point cloud data will be selected within the "local" area of ​​the center point in some sparse regions. The local feature aggregation module needs to address the problem of inconsistent point cloud distribution density.

[0097] like Figure 3 As shown, to address the three issues mentioned above, the local feature aggregation module is designed with three parts: a sampling layer, a grouping layer, and a feature extraction layer. The sampling layer uses the FPS (Farthest Point Sampling) algorithm to sample the input points, selecting several center points from all the original point cloud input points. The grouping layer uses the center points obtained from the sampling layer and a ball query algorithm to divide the point set into several local point regions. The feature extraction layer consists of convolutional layers and an introduced attention mechanism (SENet) layer. The convolutional layers apply convolutional kernels to each local point region to extract local features, perform normalization, and introduce activation functions to obtain high-dimensional features for each region in the grouping layer.

[0098] Sampling layer: given the original point cloud data as input This model uses the Farthest Point Sampling (FPS) algorithm to select the set of center points. .

[0099] The algorithm works as follows: First, a point is randomly selected as the initial center point. In each subsequent step, the algorithm calculates the relationship between each point in the unselected original point set and the selected center point set. The distance between them will be the largest distance between the unselected original points. Add the selected center point set, continue calculating the distance based on the new center point set, and keep iterating until the target number of center points is obtained.

[0100] Through the processing of the sampling layer, a small set of center points is extracted from the large amount of raw input point cloud data. The center point set obtained by the farthest point sampling algorithm ensures that the points in the set have relatively large distances, avoiding the problem of sampling extensively in one area of ​​the overall point cloud data while hardly sampling in other areas, thus ensuring a good uniformity of sampling.

[0101] Grouping layer: Raw point cloud data (size: ,in, This represents the total number of points in the original point cloud. The three-dimensional coordinate dimensions of the original point cloud data. Additional feature dimensions for each point cloud data point, and a sampled set of center points (size: ,in, The number of points within the set of central points. (The three-dimensional coordinates of the center point) and the number of points in the local area of ​​each center point. As input.

[0102] For each sampled center point, the task of the grouping layer is to define the "local" of the center point, that is, to find the point cloud near the center point as a local neighborhood, which facilitates further feature extraction.

[0103] There are two algorithms for defining neighborhoods: Ball Query and K-Nearest Neighbors (KNN). Ball Query uses a center point as the sphere, and all points within that radius constitute the neighborhood. Its advantage lies in scale consistency; the neighborhood's size is determined by a fixed radius, not a fixed number of points, avoiding inconsistencies in neighborhood scale due to density variations. KNN, on the other hand, always selects the nearest neighbor to the center point. In a high-density region, the neighborhood may contain too many redundant points, while in a low-density region, the neighborhood may lose details due to insufficient points.

[0104] This model uses the ball query algorithm to select neighboring points of the center point. This is because this model is designed for point cloud data in real-world scenarios (such as LiDAR scanning), where the density is usually uneven (e.g., dense points near the sensor and sparse points in the distance). The ball query defines the neighborhood by a fixed spatial scale, which can generate more generalized local features.

[0105] Feature extraction layer: Receives the local neighborhood point cloud generated by the grouping layer, and transforms the unordered set of local point clouds into feature vectors through a multilayer perceptron (MLP) and feature aggregation operations. Its input is... The local neighborhood of each center point has a size of ,in, The number of center points, and the neighborhood of each center point contains There are points, each point has 3D coordinates and Additional features.

[0106] The feature extraction layer first translates the coordinates of the neighboring points to a local coordinate system with the center point as the origin. Specifically, it subtracts the coordinates of the center point from the coordinates of each neighboring point to obtain the relative coordinates, as shown in Equation (3.1):

[0107]

[0108] Local coordinate systems can eliminate global positional biases, allowing the network to focus on local geometric relationships (such as relative distances between points). The original point coordinates, The coordinates of the center point are given.

[0109] After obtaining the local coordinate system, the neighborhood point cloud data is fed into a multilayer perceptron (MLP). The MLP consists of multiple fully connected layers (Linear), batch normalization (BatchNorm), and activation functions (ReLU). The features of each point are independently mapped to a higher-dimensional space, achieving nonlinear feature transformation of the points.

[0110] Because point cloud data is unordered—meaning the order of points is arbitrary and changing the order doesn't alter the 3D shape represented by the point cloud—it differs from data structures like images with fixed pixel arrangements. If features are directly extracted from each point in the point cloud and then simply concatenated, changes in the point order will alter the concatenation result, leading to inconsistent model output and impacting subsequent processing. Therefore, a symmetric function is used in the feature extraction layer. Aggregating the features of all points within the neighborhood ensures that the model remains invariant to the disorder of the point cloud input. For the neighborhood of each center point, the feature tensor size of the MLP output is... ,pass Operation in Taking the maximum value in the dimension, we get a The eigenvectors represent the local features of the center point.

[0111] In addition, point cloud data is characterized by uneven density. In real-world scenarios, some areas of a point cloud may be very dense, while others may be relatively sparse. If only a fixed radius is used... Defining the neighborhood of the center point may cause problems due to the uneven density of point cloud data, such as when the selected radius... When the radius is small, the neighborhood around the center point is also small, which may not be able to capture enough contextual information, resulting in insufficient feature representation. However, when the selected radius is small... When the size is large, too many irrelevant points may be introduced, leading to feature dilution and affecting the model's feature representation ability. To address this issue, this model uses multiple spherical neighborhoods with different radii for each center point, extracts features separately, and then fuses the multi-scale features. This ensures that local details are captured while also obtaining a wider range of contextual information.

[0112] Specifically, different radii are given for each sampling point as the center. , , Performing a ball query yields neighborhoods with three different radii, such as Figure 4 As shown, features are extracted from neighborhoods obtained at different scales using feature extraction layers, resulting in multiple sets of features. These features are then processed... The operations are concatenated to obtain a multi-scale feature representation of the center point. By inputting multiple radii at once for sphere querying, the model has the ability to cope with uneven point cloud data density. In areas with high and dense point cloud data density, small radius neighborhoods can capture details, while in areas with low and sparse point cloud data density, large radius neighborhoods can supplement the context. This design solves the problem of inconsistent point cloud distribution density and improves the model's adaptability to different geometric structures.

[0113] Because point cloud data is sparse and has uneven density distribution, the features of different channels contribute differently to the final classification task. Different channels may correspond to different geometric features or spatial relationships. In order to enable the model to automatically focus on the feature channels that are most useful to the current task, after the neighborhood point cloud data is processed by convolutional layers, this model introduces the SENet (Squeeze-and-Excitation) network.

[0114] The core idea of ​​SENet (Squeeze-and-Excitation) is to model the dependencies between channels and adaptively assign weights to each channel, thereby improving the feature representation capability.

[0115] SENet consists of two parts: Squeeze (channel compression) and Excitation (channel excitation). The Squeeze operation uses global average pooling to compress the spatial distribution of each channel into a single value, forming a channel description vector, as shown in Equation (3.2):

[0116] in, It is the description vector of channel c. It is the value of the c-th channel of the feature map at position (i,j). This represents the spatial dimension of each channel. The excitation operation applies nonlinear activation and sigmoid compression to the channel description vector through two fully connected layers (FC layers) to generate a channel attention vector, as shown in Equation (3.3):

[0117]

[0118] in, and It is the weight matrix of the fully connected layer. yes function, It is the attention weight for each channel, and its value ranges from... After obtaining the channel attention weights, multiply them channel by channel by the original feature map to obtain new feature maps, as shown in Equation (3.4):

[0119] Through compression and excitation mechanisms, the model can automatically focus on more discriminative channel features and suppress redundant or useless information.

[0120] With the introduction of SENet, the output of each convolutional layer passes through the SE attention module, which enhances the network's adaptability by dynamically adjusting the response intensity of each channel. Meanwhile, multi-scale feature concatenation itself integrates spatial features at different scales. SENet further performs channel recalibration after each scale and each convolutional layer, making the fusion of multi-scale features more reasonable and efficient, and avoiding the "noise" effect of certain scales or channels on the final features.

[0121] To accurately classify point cloud data, it's crucial to consider both the global shape and structure of the point cloud and its local geometric details. Current methods using Transformers (such as PointTransformer and Point Cloud Transformer) struggle to efficiently balance these two aspects. The core of the Point Transformer method is a local self-attention layer, where attention calculations are limited to the K nearest neighbors of each point. While this design effectively captures local geometric details, it lacks attention to the overall shape and structure of the point cloud. The Point Cloud Transformer method performs global self-attention calculations on all point cloud data. When point cloud density is uneven, the attention weights for distant points are significantly affected by noise, impacting classification performance. This model uses a Transformer based on dual attention for global feature learning.

[0122] Point Self Attention: Some structures in point clouds (such as the symmetry of 3D objects) need to be associated with features over long distances. Transformers can overcome the local limitations of traditional convolutions and achieve global or large-scale spatial information interaction. Based on this idea, this model introduces point self attention in the global feature learning module to handle the spatial relationships between point cloud data.

[0123] The core idea of ​​point self-attention is to study the spatial correlation between points, combining a single point with its local neighborhood, and achieving efficient fusion of global and local information through self-attention computation, such as... Figure 5 As shown.

[0124] PSA attention uses local features output by the LFA local feature aggregation module. As input, firstly, the aggregated features from the LFA block are... Projecting onto two different feature spaces generates a query matrix and a key matrix, as shown in equations (3.5) and (3.6):

[0125]

[0126] in, and It is a learnable weight matrix.

[0127] After obtaining the query and key matrix, through Function computation attention graph As shown in equation (3.7):

[0128]

[0129] in, and These are the query and key matrix, respectively. It is a learnable location encoding matrix used to incorporate location information. Scaling the attention weights prevents the values ​​from becoming too large.

[0130] In the local feature aggregation module, the input raw point cloud data is sampled using the FPS algorithm. Then, by performing sphere queries at different scales with different radii, a multi-scale neighborhood is constructed for each sampled point. These neighborhoods at different scales correspond to different receptive fields, capable of capturing contextual information from local details to medium-range information. In PSA, this mechanism is inherited and extended, such as... Figure 5 As shown, local features at each neighborhood scale As a value branch, utilize After generating the value matrix, a weighted summation is performed using the attention map to obtain the output neighborhood feature set. As shown in equation (3.8):

[0131]

[0132] in, It is a value matrix. It is the attention weight matrix. The operation is used to aggregate feature information within the block.

[0133] Ultimately, utilizing Multiscale The feature representations are concatenated to generate the final PSA feature.

[0134] In deep neural networks, each channel often corresponds to different semantic or geometric attributes. CSA dynamically adjusts the weights of each channel through a self-attention mechanism. After point cloud features pass through multiple layers of networks, some channels may contain redundancy or noise, contributing little to the classification task. Channel Self-Attention is used to automatically select the channels most useful for the current task, improving the effectiveness of point cloud features.

[0135] The CSA module captures the contextual dependencies between feature channels, achieving feature aggregation along the channel dimension by calculating the affinity between channels. Utilizing a self-attention mechanism, the CSA module generates attention weights for each channel, thereby highlighting important channels and suppressing redundant information. Its architecture diagram is shown below. Figure 6 As shown.

[0136] The CSA module uses the local features output by the LFA local feature aggregation module. As inputs, see equations (3.9), (3.10), and (3.11):

[0137]

[0138]

[0139]

[0140] The input features are mapped to a query, key, and value matrix, where, , and It is a learnable parameter matrix used for queries, keys, and values.

[0141] After obtaining the query and key matrix, use the formula Obtain the similarity matrix It is used to measure the similarity between channels and reflects the degree of correlation between different channels.

[0142] Furthermore, in Based on this, as shown in equation (3.12)

[0143]

[0144] The channel attention matrix is ​​calculated. First of all, conduct The operation extracts the maximum value of each channel, forming a... The vector, then through The operation expands this vector to the same level. The same dimensions form a The matrix. Through this matrix and... The difference is used to obtain the attention weight, which is then used in the final process. Function normalization generates the final channel attention matrix. In this step, we introduce... Operation and The purpose of this operation is to enhance the differentiation between channels, allowing the channel self-attention mechanism to focus on channels with significant features that contribute greatly to the classification task, reducing the weights of channels with redundancy and noise, and avoiding the aggregation of redundant information. Specifically, The operation extracts the maximum value of each channel, reflecting the importance of that channel. The operation extends these maximum values ​​to the entire matrix, compared with the original. The matrix comparison highlights the differences between the various channels, guiding the attention mechanism to focus on the prominent channels.

[0145] Finally, as shown in equation (3.13), through

[0146]

[0147] The final CSA feature is obtained.

[0148] Dual Attention: The global feature learning module adds the output features of PSA and CSA, as shown in Equation (3.14), to generate the final global features.

[0149]

[0150] The global feature learning module includes two different types of attention, such as Figure 7 As shown: Point SelfAttention and Channel SelfAttention, the combination of these two mechanisms enables the model to better capture key information in both the spatial domain and the feature domain.

[0151] Furthermore, a residual connection is used between the global feature learning module and the local feature aggregation module, as shown in Equation (3.15):

[0152] This design allows the model to retain local features while incorporating global features to enhance its capabilities, enabling it to effectively capture both global and local features in point cloud classification tasks.

[0153] In the global feature learning module, further processing of local features depends on both the spatial relationships between points and the channel weights. Point self-attention and channel self-attention address two core issues in point cloud global feature learning, starting from the spatial domain and feature domain respectively: point self-attention enhances the expressive power of spatial information, enabling the model to understand the complex relationships between points and capture the local and global spatial structure; channel self-attention enhances the expressive power of channel information, enabling the model to automatically select the most useful feature channels and improve feature discriminativeness. The combination of these two approaches allows the model to both locate key regions and highlight key semantics.

[0154] While point self-attention can capture spatial structure, it cannot dynamically adjust weights across feature channels, making it susceptible to interference from redundant channels and resulting in less refined feature representation. Conversely, channel self-attention, while highlighting key channels, remains limited to local neighborhoods in spatial structure modeling, failing to capture long-distance dependencies and global structure. The complexity of point cloud data dictates that a single-dimensional attention mechanism is insufficient for comprehensive modeling. Only by combining spatial and channel-based attention mechanisms can a deeper understanding of point cloud data be achieved, enhancing the model's expressive power and generalization performance.

[0155] To analyze the performance and efficiency of different point cloud classification models in depth, this study compares seven mainstream point cloud classification models with the model used in this graduation project in terms of overall accuracy, average accuracy, and model size. The comparison results are shown in Table 7.1.

[0156] As can be seen from the table, PointNet, as a pioneering point cloud deep learning model, although employing symmetric functions (such as...), It solves the problem of disorder in raw point cloud data and has good overall performance, but it has a bottleneck when dealing with complex object shapes because it cannot explicitly model the geometric relationship between points.

[0157] Table 7.1 Comparison of Classification Results of Various Models

[0158]

[0159] 1. A method based on local relation modeling

[0160] Methods based on local relation modeling, such as PointNet++, DGCNN, and PointCNN, focus on modeling and learning local geometric structures and emphasize the hierarchical extraction of local features.

[0161] PointNet++ introduces local sampling and aggregation mechanisms on top of PointNet. By constructing neighborhoods for center points, it significantly enhances the model's ability to model local shapes, resulting in substantial improvements in both overall and average accuracy compared to PointNet. Notably, while enhancing classification performance, PointNet++ reduces the number of parameters required by the model. This optimization is likely due to the large number of fully connected layers in PointNet, which must learn numerous redundant weights to achieve global modeling. PointNet++, by constructing neighborhoods, distributes the classification task across multiple local networks. Each local network processes fewer points and fewer feature dimensions, thus distributing the total number of parameters across these local networks and reducing the overall parameter count.

[0162] The DGCNN model primarily operates on the EdgeConv operator, which considers not only the features of the center point but also the differences between neighboring points and the center point. Each layer's EdgeConv operation reconstructs the graph structure based on features obtained from the previous layer. This mechanism allows the model to dynamically adjust the neighborhood relationships of each point in the feature space, enabling it to capture the nonlinear changes in local structure within the high-dimensional feature space. Its average class accuracy improved to 90.2%, demonstrating its superior ability to capture local geometric features between classes. Among the three local relationship-based modeling methods tested, DGCNN achieved the best average accuracy, indicating that the model can effectively extract features from different types of point clouds and possesses good generalization ability.

[0163] To address the issue of unordered point clouds, the PointCNN model presents a method based on χ-Conv convolution. The core idea of ​​this method is to learn a transformation matrix χ to transform and weight the order of local point sets, thereby converting the unordered point sets into structured data before performing convolution. The specific steps are as follows: first, features are extracted from the local point sets; then, the points are ordered according to the learned sorting transformation matrix χ; finally, the transformed point sets are input into a regular convolution kernel for local feature extraction. This method requires the fewest parameters; however, its average accuracy is only 88.1%, which is relatively low compared to other methods.

[0164] Observing the accuracy and model size of methods based on local relationships, we can conclude that these methods typically require fewer parameters and achieve higher classification accuracy. However, their limitation lies in their inability to process global dependency information of point clouds. When the object shape is complex, they cannot effectively integrate overall information, which can negatively impact classification accuracy.

[0165] 2. A method based on global relation modeling

[0166] Global relation modeling methods, represented by Point Transformer and Point Cloud Transformer, are characterized by capturing long-distance dependencies and global contextual information in point cloud data through self-attention mechanisms, thereby improving the model's ability to understand complex spatial structures.

[0167] The Point Transformer introduces the Transformer architecture into point clouds, directly modeling long-distance dependencies between points. This model employs a learnable positional encoding mechanism to capture geometric differences between points, making the attention mechanism more adaptable to the unstructured nature of point cloud data. This mechanism helps the model achieve an accuracy of 92.8%. However, due to the complexity of attention computation, the model requires a large number of parameters, reaching 13.86 MB, significantly higher than other models.

[0168] To address the issue of the large number of parameters required by Transformers in point cloud classification, Point Cloud Transformer optimizes the attention calculation process. Typically, applying Transformers requires attention calculations on a global point set, resulting in a complexity of O(n log n). ,in, The complexity stems from the number of point clouds; such complexity would obviously lead to high overall model complexity when the number of point clouds is large. The Point Cloud Transformer's computation method involves calculating attention only for each point's K nearest neighbors, thus reducing the complexity to O(log n). This reduces the number of parameters required for the entire model. As shown in Table 7.1, this design has achieved significant results, reducing the number of model parameters to 2.80MB while maintaining the accuracy of the model in point cloud classification.

[0169] Compared to methods based on local relationships, methods based on global relationships achieve higher accuracy. However, the data in the table also clearly shows that methods based on global relationships require a larger number of parameters.

[0170] 3. A method combining local relation modeling and global relation modeling

[0171] Combining local and global modeling methods leverages the complementary advantages of both by simultaneously modeling the local geometric features and global structural information of point cloud data, and has gradually become an important development direction in the field of point cloud classification in recent years. Represented by 3DCTN and the model proposed in this application, these methods demonstrate superior overall performance in terms of classification accuracy and robustness.

[0172] As shown in Table 7.1, the 3DCTN model achieves an overall accuracy of 93.2% and an average accuracy of 92.0%, placing it among the leading models in the comparison. With only 4.22 MB of parameters, it maintains relatively high efficiency. The design philosophy of 3DCTN is to learn local structural information and global contextual relationships separately, and then fuse them to enhance the model's ability to perceive complex shapes and multi-scale features. This strategy overcomes the limitations of relying solely on local or global relationship modeling.

[0173] The proposed model further optimizes this approach. While retaining multi-scale local feature extraction (LFA module) and Transformer global relation modeling (GFL module), it introduces SENet, point self-attention (PSA), and channel self-attention (CSA) mechanisms to enhance the model's responsiveness to key regions and key semantic channels. On the ModelNet40 dataset, the overall accuracy and average accuracy reach 93.4% and 92.1%, respectively, demonstrating strong class discrimination and generalization capabilities.

[0174] In summary, the method combining local and global modeling fully extracts local detail features from the point cloud through a local feature extraction module, while enhancing the model's understanding of the global structure through a global feature learning module. The synergy of these two approaches improves both overall and average accuracy, resulting in higher classification accuracy. Simultaneously, the number of parameters required by the model does not increase significantly, making it a method that balances classification accuracy and model size.

[0175] (II) Ablation Test

[0176] This ablation experiment focuses on point cloud classification tasks. Four comparative schemes were designed to examine the impact of local feature aggregation, global feature learning (based solely on point self-attention), global feature learning (dual attention), and their combinations on model performance and parameter count. The specific schemes are as follows:

[0177] Option 1: Only use the local feature aggregation module, focusing on the local structure modeling of point clouds, to verify the effectiveness of local feature aggregation and serve as a benchmark model for subsequent comparisons.

[0178] Option 2: After local feature aggregation, the original point cloud data undergoes global feature aggregation (based solely on point self-attention). A Transformer based solely on point self-attention is then added to the local feature aggregation. This is compared with Option 1 to investigate whether the introduction of the Transformer can enhance the accuracy of the model's classification.

[0179] Option 3: After local feature aggregation of the original point cloud data, global feature learning (based on dual attention) is performed. Based on the local feature aggregation, a Transformer based on dual attention is added. Compared with Option 2, the introduction of channel self-attention can be used to study whether it can further enhance the capabilities of the entire model.

[0180] Option 4: The final solution of this model, compared with the other solutions mentioned above.

[0181] For each experiment, this study recorded the overall accuracy, mean accuracy, and model size (parameters) to analyze whether each module contributed to the classification accuracy and whether it affected the number of model parameters. The overall experimental results are shown in Table 7.2.

[0182] Table 7.2 Comparison Results of Ablation Experiments

[0183]

[0184] Note: LFA stands for Local Feature Aggregation Module, GFL (PSA) stands for Global Feature Learning Module with Point Self-Attention Only, GFL (PSA+CSA) stands for Global Feature Learning Module with Dual Attention, and LFA (SENet) stands for Local Feature Aggregation Module with SENet Introduced.

[0185] 1. Analysis of Scheme 1: Independent Performance of the Local Feature Aggregation Module

[0186] Scheme 1 uses a basic model that only includes a local feature aggregation module. The purpose of this model is to verify the role of multi-scale local structure extraction mechanisms in point cloud classification.

[0187] This scheme uses the farthest point sampling (FPS) algorithm to obtain the center point, constructs a local neighborhood through ball query, and extracts features at different scales. Finally, the local features from multiple scales are stitched together and fused to form the features of each sampling point. This scheme does not introduce any global modeling mechanism; the model only captures the local geometric details of the point cloud.

[0188] Experimental results show that the proposed scheme achieves an overall accuracy of 92.5% and an average accuracy of 90.4% on the ModelNet40 dataset, demonstrating that multi-scale local feature aggregation can improve the model's ability to model complex geometric structures. Even without a global modeling mechanism, good local features can achieve high classification accuracy. Furthermore, the scheme requires only 1.74 MB of parameters, indicating a relatively small number of parameters.

[0189] 2. Analysis of Option 2: The Effect of Introducing a Point Self-Attention Mechanism

[0190] In Scheme 2, this application introduces a global feature learning module based on Point Self Attention (PSA) on the basis of the local feature aggregation module, so as to improve the model's ability to model long-distance dependencies.

[0191] Experimental results show that the proposed scheme improves the overall accuracy to 92.9% and the average accuracy to 91.8%, representing improvements of 0.4% and 1.4% respectively compared to Scheme 1. This performance improvement indicates that by introducing the point self-attention mechanism, the model can comprehensively consider global information, utilizing information from distant regions compared to Scheme 1, thus enhancing the model's overall expressive power. Compared to Scheme 1, Scheme 2 demonstrates that introducing a Transformer based on point self-attention on top of local feature aggregation is more effective in improving the model's classification accuracy than simply using CNN for local feature aggregation. However, the introduction of the Transformer inevitably increases the number of parameters required by the entire model significantly, from 1.74 MB to 4.18 MB, an increase of approximately 2.4 MB.

[0192] 3. Analysis of Option 3: The effect of introducing dual attention (point self-attention + channel self-attention)

[0193] Scheme 3 further introduces Channel SelfAttention (CSA) on the basis of Scheme 2, and constructs a complete dual attention global feature learning module to simultaneously model spatial relationships and semantic dependencies between channels.

[0194] The results show that the overall accuracy further improved to 93.2%, and the average accuracy reached 92.1%, indicating that the introduction of the channel self-attention mechanism can further enhance the model's classification ability. It can dynamically select the most discriminative feature channels, improving the effectiveness of feature representation. Simultaneously, dual attention enhances information in both spatial and channel dimensions. After introducing the channel self-attention mechanism, the total number of model parameters further increased to 5.71 MB, an increase of approximately 1.5 MB compared to Scheme 2, indicating that while introducing channel self-attention improves model performance, it also further increases the computational burden.

[0195] 4. Analysis of Scheme 4: Performance of the final scheme

[0196] Scheme 4 represents the final form of the model proposed in this study, combining a global modeling module with local feature aggregation (introducing SENet) and a dual attention mechanism (PSA+CSA). This model maintains the same average accuracy as Scheme 3, while further improving the overall accuracy to 93.4%, making it the best performing scheme among all schemes.

[0197] Compared to Scheme 3, Scheme 4, by introducing the SENet attention mechanism, effectively models the importance of feature channels during local feature aggregation, improves the semantic selection capability at the channel level, suppresses redundant parts in multi-scale local features, and amplifies key geometric structural information. In terms of parameter size, this model is 5.78 MB, only 0.07 MB more than Scheme 3, resulting in very small additional computational overhead.

[0198] 5. Summary

[0199] Based on the results of the four sets of experiments, the following conclusions can be drawn: Scheme 1, which relies solely on local feature aggregation, has achieved high basic performance, indicating that high-quality local structure modeling is a fundamental prerequisite for point cloud classification. Schemes 2 and 3, by adding point self-attention and dual attention to local features respectively, significantly improve overall performance, demonstrating that global modeling capabilities play a crucial role in improving point cloud classification accuracy. Scheme 4 shows that the introduction of SENet improves overall accuracy with a relatively small increase in required parameters, making it a cost-effective improvement method.

[0200] (III) Qualitative Analysis

[0201] In this section, we use PointNet as the baseline method to examine its classification performance on 2468 samples of the ModelNet40 test set. Experiments show that PointNet misclassified 237 samples out of the 2468, representing approximately 9.6% of the total samples. For these misclassified samples, we used the model developed in this graduation project. Our model successfully corrected 103 of the 237 misclassified samples, achieving a correction rate of 43.46%.

[0202] In this section, this application selects eight samples that were misclassified by PointNet but were successfully corrected by the model of this application, including a bathtub sample (misclassified as a bowl), a bed sample (misclassified as a bowl), a bookshelf sample (misclassified as a curtain), a car sample (misclassified as a tent), a guitar sample (misclassified as a keyboard), a piano sample (misclassified as a range hood), a plant sample (misclassified as a flower pot), and a staircase sample (misclassified as a flower pot). The advantages of the model of this application are further demonstrated by analyzing the shape characteristics of these samples.

[0203] 1. Bathtub sample

[0204] Figure 8 The displayed 3D point cloud data structure roughly resembles an ellipsoid. Because the baseline method PointNet ignores local geometric features when extracting global features, the elliptical bowl-like shape of this sample was incorrectly classified as a bowl by PointNet.

[0205] The LFA local feature aggregation module in this application model applies multi-scale feature extraction, which can effectively capture the characteristics of the bathtub edge being thicker than other areas, as well as the features of the cavity in the center of the bathtub. At the same time, the point self-attention PSA in the GFL global feature learning module further models the long-distance spatial dependence between points, which helps to identify the closed opening structure that is more recognizable in the overall shape of the bathtub. Therefore, the model successfully corrects the classification results.

[0206] 2. Bed sample

[0207] like Figure 9 As shown, bed-type samples have a headboard, resulting in point cloud data with one end protruding. The point cloud shown in the figure exhibits a structure where one side is slightly higher, which is a typical feature of bed-type samples. PointNet ignores this local point cloud feature of one side being slightly higher and judges it based solely on the overall outline, thus confusing it with a bowl.

[0208] The LFA local feature aggregation module of this invention uses multi-scale neighborhood feature extraction. This method constructs neighborhoods for sampling points with three different radii, making it more sensitive to local protrusion information and able to identify the headboard region. Simultaneously, after the convolution output, the CSA channel self-attention mechanism of the GFL global feature learning module further enhances the response of feature channels related to geometry, achieving accurate classification.

[0209] 3. Bookshelf Sample

[0210] like Figure 10As shown, samples such as bookshelves, doors, and curtains all appear as several parallel rows of points in the 3D point cloud. PointNet only uses max pooling to extract global features when processing point cloud data, which cannot effectively distinguish different spatial planar structures in cases where spatial planar structures are unclear.

[0211] In the model of this invention, the PSA self-attention mechanism focuses on the relationship between spaces, while the CSA self-attention mechanism is used to enhance the model's response to parallel planar features, and to more clearly resolve the unique multi-layer partition features of the bookshelf sample in three-dimensional space, thereby effectively distinguishing between curtains and bookshelves.

[0212] 4. Car Samples

[0213] like Figure 11 As shown in the 3D point cloud data of a car, car samples often lack distinctive features such as wheels and windows that clearly differentiate them from other samples in 3D space due to the angle of the acquisition device. In this image, the most obvious features are a wide base and a relatively low height, which also match the characteristics of tent samples. Therefore, PointNet misclassifies it. Because car and tent samples are quite similar in overall shape and outline, the model needs to be able to identify point cloud details.

[0214] This graduation project model extracts multi-scale features using spherical neighborhoods of different radii. In this sample example, the model effectively identifies vehicle body features through multi-scale feature aggregation. Simultaneously, the PSA module in the model calculates long-distance dependencies between points, enhancing the model's ability to discriminate the overall vehicle body shape. Furthermore, the CSA module effectively suppresses irrelevant feature channels, preventing certain regions in the point cloud from being incorrectly identified as tent-like pointed features.

[0215] 5. Guitar Sample

[0216] like Figure 12 As shown, the point cloud of the guitar sample is rectangular in shape, relatively thin overall, and exhibits slender geometric features. In this image, the dense points on the right represent the guitar's soundbox, while the remaining parts represent the guitar's neck structure. Due to the slender overall shape of this sample, and the presence of some interfering points on the left, the PointNet model identifies it as a keyboard.

[0217] This graduation project's multi-scale local feature extraction model can extract features from three regions of the point cloud, from smallest to largest, using different radii. For example, in the resonator region, the model can capture its complete outline through a large-radius neighborhood. For the slender neck region, the model can perceive the slender structure and its expansion to both sides using a small radius. Since one side of the guitar has significantly fewer point clouds than the other, this model uses the PSA module to capture long-distance dependencies and model the spatial relationship between the headstock and tailstock.

[0218] 6. Piano Sample

[0219] like Figure 13 As shown, a piano in a point cloud typically appears as a structure that is high on one side and low on the other, with a protruding top. Similarly, a range hood also has irregular protrusions. PointNet only captures the overall outline, which can easily confuse samples with such protruding structures, leading to misclassification.

[0220] The PSA module of this invention establishes contextual relationships between points through global attention, enabling it to model the top protrusion structure and the structure of the lower keyboard position in the shape of a piano. The model utilizes multi-scale neighborhood feature extraction to effectively identify geometric information in different regions. Simultaneously, through the CSA module, this model further enhances the response to the asymmetrically distributed channel at the top of the piano shape, avoiding the misclassification of complex geometries as simple abstract contours. This successfully corrects the errors of PointNet and distinguishes between a piano and a range hood.

[0221] 7. Plant samples

[0222] Figure 14 The image shows a flower with its stem on the right and layers of petals on the left, making it highly recognizable. In ModelNet40's classification system, it should be categorized as a plant; however, PointNet incorrectly identifies it as a flower pot, indicating PointNet's insufficient ability to perceive complex structures.

[0223] The model of this invention learns the overall point cloud structure through the PSA module. It learns that this point cloud sample only contains plants and does not contain more point cloud information. Therefore, it is not a flower pot with plants. The CSA module improves the channel response to the layered features of the petals, so that the model can more accurately identify this type of object as a plant rather than a flower pot.

[0224] 8. Staircase Sample

[0225] Figure 15 This image shows the point cloud shape of a staircase. In real life, staircases typically have a regular, stepped outline, but the staircase in this image not only has uneven point cloud density, but its perspective is also not the angle seen in everyday life; it has been rotated. Therefore, PointNet may have only captured a portion of the densely distributed point cloud area, incorrectly classifying it as an object like a flowerpot, which concentrates a large amount of point cloud in a localized area.

[0226] This graduation project model utilizes multi-size feature extraction, which can extract the feature of height difference between continuous steps using different radii. The PSA module establishes hierarchical dependencies between points in the horizontal and vertical directions globally, while the CSA module enhances the responsiveness of the channel to represent steps. The design of these modules helps the model accurately determine that the point cloud data is stairs rather than flower pots with dense point clouds at the bottom.

[0227] 9. Summary

[0228] Analysis of eight representative examples from 103 successfully corrected samples reveals that this graduation project possesses several advantages over the baseline method PointNet: Leveraging the multi-scale feature aggregation of the LFA local feature aggregation module, the model can flexibly adapt to density variations in point clouds, extracting detailed structures (such as bed heads, piano heads, steps, etc.), thus enhancing its ability to model local details. Furthermore, the PSA module in the GFL global feature learning module establishes long-distance dependencies between points through the Transformer mechanism, enabling the model to understand the overall outline of an object, shape symmetry, and the relationships between regional hierarchies, such as recognizing the overall outline of a car or the top of a piano. Additionally, the CSA module effectively improves the responsiveness of key geometric semantic channels, allowing the model to highlight discriminative features, particularly excelling in handling structurally repetitive or semantically dense regions such as overlapping plant petals or multi-layered shelves in a bookshelf. Through the design of these modules, while maintaining an overall perception of the shape of 3D objects, the comprehensive modeling ability for geometric details, hierarchical structures, and semantic channels is further enhanced, effectively correcting a large number of PointNet misclassified samples.

[0229] like Figure 16 As shown, based on the same inventive concept, the third embodiment of this application provides a 3D object classification device based on CNN-Transformer, including:

[0230] Sampling module 11 is used to sample point cloud data based on the FPS algorithm to obtain neighborhood point cloud data;

[0231] Feature aggregation module 12 is used to perform convolution processing on the neighborhood point cloud data based on the SENet attention mechanism to obtain local features;

[0232] The global feature learning module 13 is used to perform global learning on the local features through Transformer to obtain global features;

[0233] The classification module 14 is used to input the global features into the classifier to obtain the classification result.

[0234] Optionally, such as Figure 17 As shown, this application embodiment also provides an electronic device 1100, including a processor 1101, a memory 1102, and a program or instructions stored in the memory 1102 and executable on the processor 1101. When the program or instructions are executed by the processor 1101, they implement the various processes of the above-described CNN-Transformer-based three-dimensional object classification method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0235] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0236] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described CNN-Transformer-based 3D object classification method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0237] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0238] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described embodiments of the CNN-Transformer-based 3D object classification method, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0239] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0240] It should be noted that, in this embodiment, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0241] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0242] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A 3D object classification method based on CNN-Transformer, characterized in that, include: Neighborhood point cloud data is obtained by sampling point cloud data based on the FPS algorithm; Local features are obtained by performing convolution processing on the neighborhood point cloud data based on the SENet attention mechanism. Global features are obtained by performing global learning on the local features using a Transformer. The global features are input into the classifier to obtain the classification result.

2. The 3D object classification method based on CNN-Transformer according to claim 1, characterized in that, Neighborhood point cloud data is obtained by sampling point cloud data based on the FPS algorithm, including: A preset number of center points are selected from the point cloud data by sampling at the furthest distance; Based on sphere queries, construct neighborhood point cloud data with the central point as the center of the sphere.

3. The 3D object classification method based on CNN-Transformer according to claim 2, characterized in that, Constructing neighborhood point cloud data with the central point as the center of the sphere based on sphere query includes: Using the center point as the center, a spherical neighborhood is constructed using multiple spheres of different radii to obtain the neighborhood point cloud data.

4. The 3D object classification method based on CNN-Transformer according to claim 3, characterized in that, The neighborhood point cloud data is convolutionally processed based on the SENet attention mechanism to obtain local features, including: The neighborhood point cloud data is convolutionally processed based on the SENet attention mechanism to obtain initial local features; The local features are obtained by multi-scale fusion of multiple initial local features corresponding to each center point.

5. A 3D object classification method based on CNN-Transformer according to claim 2, characterized in that, The neighborhood point cloud data is convolutionally processed based on the SENet attention mechanism to obtain local features, including: The neighborhood point cloud data is convolved to obtain initial local features; The initial local features are obtained by applying attention weights based on the SENet attention mechanism.

6. The 3D object classification method based on CNN-Transformer according to claim 5, characterized in that, The neighborhood point cloud data is convolved to obtain initial local features, including: A local coordinate system is constructed with the center point as the origin; Within the local coordinate system, the relative coordinates are obtained by subtracting the coordinates of the neighborhood point cloud data from the coordinates of the center point. The neighborhood point cloud data containing the relative coordinates is used to obtain a feature tensor through a multilayer perceptron. Through function The local feature is obtained by taking the maximum value of the feature tensor in the neighborhood point cloud data corresponding to each center point.

7. A 3D object classification method based on CNN-Transformer according to claim 6, characterized in that, The multilayer perceptron includes multiple fully connected layers, batch normalization, and activation functions.

8. A 3D object classification method based on CNN-Transformer according to claim 1, characterized in that, Global features are obtained by performing global learning on the local features using a Transformer, including: A self-attention graph of the neighborhood point cloud data is generated based on a point self-attention mechanism; The point self-attention features are obtained by weighted summation of the local features based on the self-attention map. The local features are mapped onto the query matrix and the key-value matrix to obtain the channel similarity matrix; Perform the channel similarity matrix The operation extracts the maximum value for each channel; The maximum value is expanded to the same dimension as the channel similarity matrix to obtain the channel maximum value matrix; The channel attention weights are obtained based on the difference between the channel maximum value matrix and the channel similarity matrix; pass The channel attention matrix is ​​obtained by normalizing the function. The local features are weighted based on the channel attention matrix to obtain the channel attention features; The global feature is obtained by adding the point self-attention feature and the channel attention feature.

9. A 3D object classification device based on CNN-Transformer, characterized in that, include: The sampling module is used to sample point cloud data based on the FPS algorithm to obtain neighborhood point cloud data; The feature aggregation module is used to perform convolution processing on the neighborhood point cloud data based on the SENet attention mechanism to obtain local features; The global feature learning module is used to perform global learning on the local features through Transformer to obtain global features; The classification module is used to input the global features into the classifier to obtain the classification result.

10. An electronic device, characterized in that, include: A processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement a CNN-Transformer-based 3D object classification method as described in any one of claims 1-9.

Images