A point cloud semantic segmentation method and system based on multi-scale feature fusion and attention mechanism

Abstract

The application discloses a point cloud semantic segmentation method and system based on multi-scale feature fusion and an attention mechanism, which comprises the following steps: preprocessing input point clouds, generating multi-scale features through progressive down-sampling and cross-scale feature interaction; constructing the multi-scale features into a hierarchical graph structure, connecting nodes in a layer through spatial topology, and connecting cross-scale nodes through scale mapping; calculating node geometric features based on the hierarchical graph structure, dynamically adjusting the attention neighborhood according to the regional geometric complexity, realizing cross-scale feature fusion through geometric perception attention; and learning and interacting the weights of the channel, space and scale dimensions of the fused features through multi-dimensional joint attention, and outputting a semantic segmentation result. The application explicitly models the topological association of multi-scale features through a hierarchical graph structure, overcomes the scale information breakage problem, realizes a good balance between efficiency and accuracy through a geometric perception attention mechanism and dynamic receptive field adjustment, and enhances the feature discriminability through multi-dimensional joint attention.

Description

Technical Field

[0001] This invention belongs to the field of point cloud data processing technology, and in particular relates to a point cloud semantic segmentation method and system based on multi-scale feature fusion and attention mechanism. Background Technology

[0002] Point clouds, as a form of three-dimensional spatial data representation, can accurately describe the geometric shape and spatial distribution of objects, and have wide applications in fields such as autonomous driving, robot navigation, and 3D reconstruction. Point cloud semantic segmentation, as the core task of point cloud understanding, aims to assign semantic labels (such as vehicles, pedestrians, roads, etc.) to each point in the point cloud, and its performance directly affects the accuracy of subsequent high-level tasks.

[0003] Existing point cloud semantic segmentation methods mainly face the following challenges: First, there is a bottleneck in multi-scale feature fusion. Existing methods (such as PointNet++ or voxelization methods based on hierarchical sampling) often suffer from information loss or redundancy when fusing features at different scales due to fixed sampling or interpolation operations. Furthermore, they fail to effectively model the semantic relationships between scales and struggle to balance detailed features with global context.

[0004] Second, the attention mechanism has limitations. Existing attention mechanisms mostly focus on a single dimension (such as space or channel), failing to fully utilize the geometric characteristics of point clouds (such as normal vectors and curvature). Furthermore, global attention has high computational complexity and is difficult to adapt to large-scale point clouds; local attention, on the other hand, tends to ignore long-distance semantic dependencies, resulting in limited segmentation accuracy.

[0005] Third, the integration of fusion and attention is not tight. In existing methods, multi-scale fusion and attention mechanisms are mostly simple stacking, failing to achieve deep synergy between the two. They cannot dynamically adjust the fusion strategy and attention range according to geometric features, making it difficult to adapt to the scale differences and geometric complexity of different objects in complex scenes.

[0006] Based on this, the present invention proposes a point cloud semantic segmentation method and system based on multi-scale feature fusion and attention mechanism to solve the above problems. Summary of the Invention

[0007] To address the above technical problems, this invention provides a point cloud semantic segmentation method and system based on multi-scale feature fusion and attention mechanism, in order to solve the technical problems of poor multi-scale feature fusion effect, mismatch between attention mechanism and geometric features, and difficulty in balancing efficiency and accuracy in the prior art.

[0008] The technical solution adopted by this invention to solve its technical problem is: A point cloud semantic segmentation method based on multi-scale feature fusion and attention mechanism includes the following steps: S100: Preprocesses the input point cloud and performs progressive downsampling through multiple cascaded feature extraction subnetworks. In each downsampling stage, cross-scale feature interaction is introduced through skip connections to generate point cloud features at different scale levels. S200: Construct a hierarchical graph structure from point cloud features at different scale levels. Each scale level corresponds to a layer of nodes in the graph structure. The node features are the point cloud features at that scale. Nodes within a layer are connected through spatial topological relationships, and nodes across scale layers are connected through scale mapping relationships. S300: Based on a hierarchical graph structure, it calculates the geometric features between nodes and dynamically adjusts the neighborhood range of attention calculation according to the geometric complexity of the region where the node is located. Through a geometrically aware attention mechanism, it realizes feature fusion between nodes across scales and within layers, and outputs fused features. S400: Through multi-dimensional joint attention, attention weights are learned for the fused features in the channel dimension, spatial dimension and scale dimension respectively. The attention weights of the three dimensions are then interacted with the fused features to obtain refined semantic features and output point cloud semantic segmentation results.

[0009] Preferably, each feature extraction subnetwork in S100 extracts features in the following manner: S110: For each point in the current scale level, select its K nearest neighbors, calculate the difference between the relative coordinates and normal vectors of the point and each of its nearest neighbors, and form local geometric features. The local geometric features are then mapped to a high-dimensional feature space through a multilayer perceptron to obtain the local feature vector of each point. S120: Perform global max pooling on the local feature vectors of all points in the current scale layer to obtain the global feature vector. Concatenate the global feature vector with the local feature vector of each point to obtain the preliminary feature representation. Then, refine the preliminary feature representation through convolutional layers and activation functions to output the point cloud features of the current scale layer. S130: An adaptive sampling strategy combining geometric features is adopted to determine the point set of the next scale level from the point set of the current scale level; wherein, the adaptive sampling strategy includes: calculating the curvature and semantic boundary score of each point based on the point cloud features of the current scale level, sorting them from high to low scores, and selecting a preset number of points as retained points to achieve downsampling. S140: The local geometric features of the previous scale level are interpolated and mapped onto the point set of the current scale level, and then fused with the local and global features extracted by the current scale level itself to generate the point cloud features of the current scale level.

[0010] Preferably, S200 includes: S210: For each scale level, construct an intra-layer subgraph, which includes a node set, an intra-layer edge set, and a node feature set. Each node in the node set corresponds to a point at that scale level, and the node feature set represents the point cloud features at that scale level. The intra-layer edge set is constructed using a combination of dynamic K-nearest neighbors and density filtering, and includes: S211: For each node in the node set, dynamically adjust the number of K nearest neighbors according to the density of its surrounding points. In areas where the point density is higher than a preset threshold, set the K value to the first value. In areas where the point density is lower than the preset threshold, set the K value to the second value. The first value is less than the second value. S212: For each node, select its K nearest neighbors as neighboring nodes, and establish edges between the node and each neighboring node to form a preliminary set of in-layer edges; S213: Perform density filtering on the initial intra-layer edge set to remove edges whose distance exceeds a preset distance threshold or whose feature difference exceeds a preset difference threshold, thus obtaining the optimized intra-layer edge set. S220: Construct a set of cross-scale edges between adjacent scale levels. The set of cross-scale edges is used to connect the nodes of scale level L and scale level L+1. The establishment of cross-scale edges is based on the sampling correspondence between the upper and lower level point sets. A many-to-many mapping method between parent nodes and multiple child nodes is adopted. In this method, one parent node in scale level L+1 corresponds to multiple child nodes in scale level L. The number of multiple child nodes is within a preset range. S230: Set an initial weight for each cross-scale edge in the cross-scale edge set. The initial weight is determined based on the spatial distance and feature similarity between the parent and child nodes connected by the cross-scale edge.

[0011] Preferably, the attention mechanism for geometry perception in S300 includes: S310: For each pair of nodes connected by the edge in the hierarchical graph structure, calculate the Euclidean distance, the angle between the normal vectors and the curvature difference between each pair of nodes, and convert them into Euclidean distance mapping value, normal vector angle mapping value and curvature difference mapping value respectively through the corresponding mapping function. Then, fuse the three mapping values ​​into a comprehensive geometric weight through the fusion function. S320: Combine the feature similarity of nodes with the comprehensive geometric weight to calculate the attention weights of nodes across scales and within layers; S330: Dynamically adjust the neighborhood range of attention calculation based on the geometric complexity of the region where each node is located. Geometric complexity is measured by the curvature value and local entropy value of the corresponding node. By comparing the curvature value with a preset curvature threshold and the local entropy value with a preset local entropy threshold, the nodes are divided into multiple geometric complexity categories. Each geometric complexity category corresponds to a preset neighborhood range value, and the higher the geometric complexity of the category, the smaller the neighborhood range value. S340: Based on attention weights and a defined neighborhood range, the features of each node's neighboring nodes are weighted and aggregated. The original features are then added to the aggregated features through residual connections to output the fusion features at the finest scale level.

[0012] Preferably, S310 includes: S311: For any two nodes in the hierarchical graph structure G and Calculate the Euclidean distance between them. And through the mapping function The Euclidean distance mapping value is obtained, where These are learnable parameters; S312: Computation Node and The angle between the normal vectors and map it to ,in, and They are nodes and The corresponding normal vector; S313: Computation Node and Curvature difference ,in and They are nodes and The curvature of the corresponding point, and through the mapping function The curvature difference mapping value is obtained, where This represents the maximum curvature of all points at this scale level. S314: Through the fusion function The Euclidean distance mapping value, the normal vector angle mapping value, and the curvature difference mapping value are integrated into a comprehensive geometric weight, where, , , These are learnable fusion weights.

[0013] Preferably, the formula for calculating the attention weight in S320 is: in, For nodes For nodes Attention weights For nodes With nodes Feature similarity is calculated by passing node features through the query projection matrix. Bond projection matrix The result is obtained by performing a dot product operation after mapping, i.e. , For nodes With nodes Geometric features, For geometric feature mapping function, These are the geometric feature weighting coefficients. For feature dimension, To Dimensional normalization operation; Weighted aggregation in S340 is achieved using the following formula: in, The features after fusion for node i Let i be the original feature of node i. Let be the attention weight of node i to node j. The projection matrix is ​​the value. This is a layer normalization operation; the original features are added to the aggregated features through residual connections, preserving the original feature information.

[0014] Preferably, S330 includes: Obtain the curvature value of each node at the current scale level. and local entropy And preset curvature threshold and local entropy threshold ; When node i satisfies > and > When this occurs, it indicates that the node is located in a region that is both geometrically and semantically complex, and the neighborhood range of the node is adjusted to the number of its first nearest neighbors. ; When node i satisfies > and ≤ When this occurs, it indicates that the node is located in a geometrically complex but semantically clear region, and the node's neighborhood is adjusted to the second nearest neighbor number. ; When node i satisfies and > When this occurs, it indicates that the node is located in a geometrically simple but semantically confusing region, and the node's neighborhood is adjusted to the third nearest neighbor number. ; When node i satisfies and ≤ When this occurs, it indicates that the node is located in a region that is both geometrically and semantically simple, and the neighborhood range of the node is adjusted to the fourth nearest neighbor. ; in, , , , The values ​​increase sequentially.

[0015] Preferably, semantic refinement in S400 is achieved through multi-dimensional joint attention, specifically including: S410: Perform global average pooling on the fused features in the spatial dimension to obtain global statistical information for each channel. Input the global statistical information into the multilayer perceptron and output the channel dimension attention weight for each channel. At the same time, perform global average pooling on the fused features in the channel dimension to obtain global statistical information for each node. Input the global statistical information into the multilayer perceptron and output the spatial dimension attention weight for each node. S420: The fused features are concatenated with the upsampled features of other scale levels. The concatenated features are then input into a multilayer perceptron, and the scale dimension attention weights are output. The number of scale dimension attention weights is the same as the number of scale levels, and the sum of the attention weights of each scale dimension is 1. S430: The fused features are multiplied element-wise with the channel dimension attention weights, spatial dimension attention weights and scale dimension attention weights to obtain the refined semantic features; S440: Input the refined semantic features into the classifier, output the semantic label probability distribution of each point, take the label with the highest probability as the semantic label of that point, and obtain the semantic segmentation result at the finest scale level.

[0016] A point cloud semantic segmentation system based on multi-scale feature fusion and attention mechanism is provided to execute a point cloud semantic segmentation method based on multi-scale feature fusion and attention mechanism, comprising: The multi-scale feature extraction module is used to preprocess the input point cloud and perform progressive downsampling through multiple cascaded feature extraction sub-networks. In each downsampling stage, cross-scale feature interaction is introduced through skip connections to generate point cloud features at different scale levels. The hierarchical graph construction module is used to construct the point cloud features of different scale levels into a hierarchical graph structure. Each scale level corresponds to a layer of nodes in the graph structure. The node features are the point cloud features at that scale. Nodes within a layer are connected through spatial topological relationships, and nodes across scale layers are connected through scale mapping relationships. The geometric perception cross-scale attention fusion module is used to achieve feature fusion of nodes across scales and within layers through a geometric perception attention mechanism based on the hierarchical graph structure, and output fused features. The semantic refinement module is used to learn attention weights for the fused features in the channel dimension, spatial dimension, and scale dimension through multi-dimensional joint attention. The attention weights of the three dimensions are then interacted with the fused features to obtain refined semantic features and output point cloud semantic segmentation results.

[0017] The aforementioned point cloud semantic segmentation method and system based on multi-scale feature fusion and attention mechanisms firstly constructs a hierarchical graph structure from point cloud features at different scale levels, explicitly modeling the topological relationships between multi-scale features. This maintains continuous topological relationships during feature fusion, overcoming the problem of scale information fragmentation and improving segmentation accuracy. Secondly, a geometry-aware attention mechanism is employed for cross-scale feature fusion, incorporating the geometric characteristics of the point cloud into the attention weight calculation. This makes the feature fusion more consistent with the physical geometry of 3D data, resolving the mismatch between existing attention mechanisms and geometric features. Furthermore, the neighborhood range of attention calculation is dynamically adjusted based on the geometric complexity of the node's region. This allows for focusing on details in complex regions and acquiring global context in simple regions, achieving an effective balance between computational efficiency and segmentation accuracy. Finally, multi-dimensional joint attention refinement of the fused features filters key features from multiple dimensions, enhancing the discriminativeness and robustness of semantic features and further improving the overall performance of point cloud semantic segmentation. Attached Figure Description

[0018] Figure 1 This is a flowchart of a point cloud semantic segmentation method based on multi-scale feature fusion and attention mechanism in one embodiment of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0020] This invention provides a point cloud semantic segmentation method and system based on multi-scale feature fusion and attention mechanisms. Specific embodiments are described below, such as... Figure 1 The technical solution of the present invention will be described in detail below.

[0021] (a) S100: Preprocess the input point cloud and perform progressive downsampling through multiple cascaded feature extraction subnetworks. In each downsampling stage, cross-scale feature interaction is introduced through skip connections to generate point cloud features at different scale levels.

[0022] Specifically, the input point cloud is the original 3D point cloud data. ,in 3 represents the number of points, and 3 represents the 3D coordinates of each point. The input point cloud is preprocessed to calculate the normal vector for each point. and curvature This geometric information will play a role in subsequent feature extraction and attention computation. The preprocessed point cloud is progressively downsampled through multiple cascaded feature extraction sub-networks, and cross-scale feature interactions are introduced through skip connections at each downsampling stage to generate point cloud features at different scale levels. Taking three scale levels (L1, L2, L3) as an example, the specific implementation is as follows: S110: For each point in the current scale level, select its K nearest neighbors (e.g., K=16), calculate the difference between the relative coordinates and normal vectors of the point and each of its nearest neighbors to form local geometric features. Map the local geometric features to a high-dimensional feature space through a multilayer perceptron (MLP) to obtain the local feature vector of each point. S120: Perform global max pooling on the local feature vectors of all points in the current scale layer to obtain the global feature vector. Concatenate the global feature vector with the local feature vector of each point to obtain the preliminary feature representation. Then, refine the preliminary feature representation through convolutional layers and activation functions to output the point cloud features of the current scale layer. S130: An adaptive sampling strategy combining geometric features is adopted to determine the point set of the next scale level from the point set of the current scale level; wherein, the adaptive sampling strategy includes: calculating the curvature and semantic boundary score of each point based on the point cloud features of the current scale level, sorting them from high to low scores, and selecting a preset number of points as retained points to achieve downsampling. S140: The local geometric features of the previous scale level are interpolated and mapped onto the point set of the current scale level, and then fused with the local and global features extracted by the current scale level itself to generate the point cloud features of the current scale level.

[0023] Specifically, the first feature extraction subnetwork (finest scale L1) receives the original point cloud. It uses the normal vector and curvature information to output L1-scale point cloud features. ,in The number of points at the L1 scale (obtained through downsampling, such as...) ), For feature dimensions (e.g.) =64). Subsequent feature extraction sub-networks (corresponding to coarser scale levels L2, L3, etc.) have similar structures to the first feature extraction sub-network, but their inputs are the point cloud downsampled from the previous scale level and the features from the previous scale. In each feature extraction sub-network, an adaptive sampling strategy combining geometric features is used for downsampling, prioritizing the retention of high curvature points and semantic boundary points to avoid the loss of key geometric information during the sampling process. For example, when downsampling from L1 scale to L2 scale, the curvature and semantic boundary scores of each point are calculated (which can be obtained based on preliminary feature calculations), and points with higher scores are selected and retained to obtain the point set at the L2 scale. ,in .

[0024] Simultaneously, to achieve cross-scale feature interaction, each feature extraction subnetwork, when generating features at the current scale, introduces local geometric features from the previous scale level through skip connections. For example, when generating L2 scale features... In this process, the local geometric features at the L1 scale are interpolated and mapped onto the point set at the L2 scale. These features are then fused with the local and global features extracted at the L2 scale itself. This allows the L2 scale features to contain both global semantic information and geometric detail information from the L1 scale, laying the foundation for subsequent cross-scale feature fusion.

[0025] By analogy, point cloud features at multiple scale levels can be obtained, such as features at the L3 scale. wait.

[0026] (ii) S200: Construct point cloud features of different scale levels into a hierarchical graph structure, where each scale level corresponds to a layer of nodes in the graph structure, the node features are the point cloud features at that scale, the nodes within the layer are connected by spatial topological relationships, and the nodes across scale layers are connected by scale mapping relationships.

[0027] In one embodiment, S200 includes: S210: For each scale level, construct the intra-layer subgraph. The subgraph within a layer includes a set of nodes. Inner edge set and node feature set Each node in the node set corresponds to a point at that scale level, and the node feature set represents the point cloud features at that scale level. The intra-layer edge set is constructed using a combination of dynamic K-nearest neighbors and density filtering, including: S211: For each node in the node set, dynamically adjust the number of K-nearest neighbors based on the density of its surrounding points. In areas where the point density is higher than a preset threshold (such as the surface of an object in a dense point cloud), set the K value to the first value (e.g., K=8) to avoid excessive redundant connections; in areas where the point density is lower than a preset threshold (such as a distant scene in a sparse point cloud), set the K value to the second value (e.g., K=24) to ensure that each node can obtain sufficient contextual information. The first value is less than the second value. S212: For each node, select its K nearest neighbors as neighboring nodes, and establish edges between the node and each neighboring node to form a preliminary set of in-layer edges; S213: Perform density filtering on the initial intra-layer edge set to remove edges whose distance exceeds a preset distance threshold or whose feature difference exceeds a preset difference threshold, thus obtaining the optimized intra-layer edge set. S220: Construct a set of cross-scale edges between adjacent scale levels. The set of cross-scale edges is used to connect nodes in scale level L and scale level L+1. The establishment of cross-scale edges is based on the sampling correspondence between the upper and lower level point sets, and adopts a many-to-many mapping method between parent nodes and multiple child nodes. In this way, one parent node in scale level L+1 corresponds to multiple child nodes in scale level L. The number of multiple child nodes is a preset range (4-8 in this embodiment, based on the region division during downsampling). In this way, information can flow bidirectionally between different scale levels. S230: Assign initial weights to each cross-scale edge in the cross-scale edge set. The initial weights are determined based on the spatial distance and feature similarity between the parent and child nodes connected by the cross-scale edge. For example, nodes that are closer in distance and more similar in features will have larger initial weights. These initial weights can be adaptively adjusted during subsequent training to better reflect the association strength between cross-scale nodes.

[0028] Through the above process, a complete hierarchical graph structure G is constructed. This structure can effectively preserve the topological relationships between features at different scales, providing a good foundation for subsequent cross-scale feature fusion.

[0029] (III) S300: Based on the hierarchical graph structure, calculate the geometric features between nodes, and dynamically adjust the neighborhood range of attention calculation according to the geometric complexity of the region where the node is located. Through the geometrically perceptive attention mechanism, feature fusion of nodes across scales and within layers is achieved, and fused features are output.

[0030] In one embodiment, the attention mechanism for geometry perception in S300 includes: S310: For all edges in the hierarchical graph structure (including intra-layer edges and cross-scale edges), calculate the Euclidean distance, normal vector angle, and curvature difference between connected nodes, and encode them into comprehensive geometric weights through a fusion function.

[0031] In one embodiment, S310 includes: S311: For any two nodes in the hierarchical graph structure G and Calculate the Euclidean distance between them. And through the mapping function The Euclidean distance mapping value is obtained, where For learnable parameters; further, Euclidean distance ,in and They are nodes and The three-dimensional coordinates of the corresponding points are mapped in this way, so that the geometric weight of the nearest points is naturally higher.

[0032] S312: Computation Node and The angle between the normal vectors and map it to ;in, and They are nodes and The corresponding normal vector; It can reflect the consistency of the orientation of the surfaces on which two points are located; the smaller the included angle, the better. The larger the value, the stronger the geometric relationship between the two points; S313: Computation Node and Curvature difference ,in and They are nodes and The curvature of the corresponding points is normalized to the range [0,1] using a mapping function. The curvature difference mapping value is obtained, where This represents the maximum curvature of all points at this scale level; the larger this mapping value, the smaller the curvature difference between the two points, and the more likely they belong to the same object or the same surface.

[0033] S314: Through the fusion function The above three mapping values ​​are merged into a comprehensive geometric weight, where, , , Learnable fusion weights (initial values ​​can be set to...) =0.4、 =0.3、 =0.3), and can be adaptively adjusted according to the characteristics of the actual data during training to balance the contribution of different geometric features in attention calculation.

[0034] S320: Combining node feature similarity and comprehensive geometric weights, calculate the attention weights for nodes across scales and within layers; in one embodiment, the formula for calculating the attention weights in S320 is: in, For nodes For nodes Attention weights For nodes With nodes Feature similarity is calculated by passing node features through the query projection matrix. Bond projection matrix (All are learnable matrices) are obtained by mapping and then performing a dot product operation, i.e. , For nodes With nodes Geometric features, For geometric feature mapping function, This is the geometric feature weight coefficient (initial value can be set to 0.5), used to adjust the importance of geometric features in attention calculation. It can be dynamically adjusted during training, for example, automatically increased in complex scenes to enhance the influence of geometric features. For feature dimension, To The dimension normalization operation ensures that the sum of the attention weights is 1.

[0035] For nodes within a layer, the intra-layer attention weight is calculated according to the above formula to strengthen the association between nodes within the same scale. For nodes across scales (such as nodes between L1 and L2, or L2 and L3), the cross-scale attention weight is also calculated according to the above formula to realize information transfer between different scale levels. For example, nodes at the L1 level transfer detailed information to nodes at the L2 level, and nodes at the L2 level transfer semantic information to nodes at the L1 level.

[0036] S330: Dynamically adjust the neighborhood range of attention calculation based on the geometric complexity of the region where each node is located. Geometric complexity is measured by the curvature value and local entropy value of the corresponding node. By comparing the curvature value with a preset curvature threshold and the local entropy value with a preset local entropy threshold, the nodes are divided into multiple geometric complexity categories. Each geometric complexity category corresponds to a preset neighborhood range value, and the higher the geometric complexity of the category, the smaller the neighborhood range value. The S330 includes: Obtain the curvature value of each node at the current scale level. and local entropy And preset curvature threshold (like =0.1 (curvature value normalized to the range [0,1]) and local entropy threshold (like =0.6); When node i satisfies > and > When this occurs, it indicates that the node is located in a region that is both geometrically and semantically complex (such as the edge of an object or a small structure), and the neighborhood range of the node is adjusted to the number of its first nearest neighbors. (like =4), to focus on local details; When node i satisfies > and ≤ When this occurs, it indicates that the node is located in a geometrically complex but semantically clear region, and the node's neighborhood is adjusted to the second nearest neighbor number. (like =8); When node i satisfies and > When this occurs, it indicates that the node is located in a geometrically simple but semantically confusing region, and the node's neighborhood is adjusted to the third nearest neighbor number. (like =16); When node i satisfies and ≤ When this occurs, it indicates that the node is located in a region that is both geometrically and semantically simple (such as a flat surface or the interior of a large object), and the node's neighborhood is adjusted to the fourth nearest neighbor. (like =32), to obtain more global context information; in, , , , The values ​​increase sequentially.

[0037] Specifically, the curvature value reflects the degree of curvature of the surface where the point is located. The larger the curvature value, the more complex the geometric structure of the region. The local entropy value is calculated based on the preliminary classification results of the node and reflects the degree of confusion of the semantic categories around the node. The larger the local entropy value, the more complex the semantic structure of the region.

[0038] In this way, each node can automatically select a suitable neighborhood range based on the actual situation of its own region and filter out edges that exceed the selected neighborhood range. Thus, when calculating attention weights, only relevant neighborhood nodes are considered, which improves computational efficiency while ensuring the effective use of key information.

[0039] S340: Based on attention weights and a defined neighborhood range, the features of neighborhood nodes are weighted and aggregated to output the fused features at the finest scale level.

[0040] In one embodiment, the weighted aggregation in S340 is achieved by the following formula: in, The features after fusion for node i Let i be the original feature of node i. Let be the attention weight of node i to node j. Value projection matrix (learnable matrix). The neighborhood features are the weighted aggregation results. Layer normalization is used to stabilize the training process. Adding the original features to the aggregated features via residual connections helps preserve original feature information and alleviates the gradient vanishing problem. For intra-layer feature fusion, the above aggregation operation is performed independently on nodes at each scale level, strengthening the feature correlation within the same scale. For cross-scale feature fusion, taking L1 and L2 scale levels as an example, the features of nodes at L1 scale are aggregated with the features of related nodes at L2 scale (to obtain global semantic information), and the features of nodes at L2 scale are also aggregated with the features of related nodes at L1 scale (to obtain local detail information), achieving complementary fusion of features at different scales.

[0041] Finally, the feature aggregation unit outputs fused features. To preserve fine-scale resolution for accurate semantic segmentation, the features from the finest-scale layer (such as L1) after cross-scale fusion are typically selected as the final fused features. .

[0042] (iv) S400: Through multi-dimensional joint attention, attention weights are learned for the fusion features in the channel dimension, spatial dimension and scale dimension respectively, and the attention weights of the three dimensions are interacted with the fusion features to obtain refined semantic features and output point cloud semantic segmentation results.

[0043] Specifically, the core of this step is multi-dimensional joint attention operation, which can simultaneously learn attention weights for the channel dimension, spatial dimension and scale dimension of features, thereby filtering out key features, enhancing useful information and suppressing noise and redundant information.

[0044] In one embodiment, semantic refinement in S400 is achieved through multi-dimensional joint attention, specifically including: S410: Perform global average pooling on the fused features in the spatial dimension to obtain global statistical information for each channel. Input the global statistical information into the multilayer perceptron and output the channel dimension attention weight for each channel. At the same time, perform global average pooling on the fused features in the channel dimension to obtain global statistical information for each node. Input the global statistical information into the multilayer perceptron and output the spatial dimension attention weight for each node. S420: The fused features are concatenated with the upsampled features of other scale levels. The concatenated features are then input into a multilayer perceptron, and the scale dimension attention weights are output. The number of scale dimension attention weights is the same as the number of scale levels, and the sum of the attention weights of each scale dimension is 1. S430: The fused features are multiplied element-wise with the channel dimension attention weights, spatial dimension attention weights and scale dimension attention weights to obtain the refined semantic features; S440: Input the refined semantic features into the classifier, output the semantic label probability distribution of each point, take the label with the highest probability as the semantic label of that point, and obtain the semantic segmentation result at the finest scale level.

[0045] Specifically, spatial and channel-level attention is used to identify the importance of different spatial and channel features. Scale-level attention is used to adjust the contribution of features at different scales to the final semantic segmentation. The fused features are concatenated with upsampled features from other scale levels (such as L2 and L3), input into an MLP, and the output is the attention weight for each scale. =[ , , (The sum of the weights is 1), where Weights corresponding to L1 scale features Weights corresponding to L2 scale features The weights corresponding to L3 scale features. For example, for small target regions, The weight will automatically increase to highlight fine-scale details; for large areas, and The weights will be increased accordingly to highlight the coarse-scale global semantic features.

[0046] The multidimensional joint attention unit multiplies the attention weights of the three dimensions of channel, space, and scale to obtain refined semantic features: ,in, Represents element-wise multiplication; expands the dimension to enable and and Dimensional matching.

[0047] Finally, the refined semantic features are input into a classifier (such as a classifier composed of multiple MLP layers), which outputs the semantic label probability distribution of each point. The label with the highest probability is taken as the final semantic label for that point, thus obtaining the semantic segmentation result at the finest scale level. If a semantic segmentation result with the same resolution as the original point cloud is required, an upsampling operation can be performed based on the mapping relationship from the finest scale level to the original point cloud.

[0048] In one embodiment, a point cloud semantic segmentation system based on multi-scale feature fusion and attention mechanism is also provided for performing the above method, including: The multi-scale feature extraction module is used to preprocess the input point cloud and perform progressive downsampling through multiple cascaded feature extraction sub-networks. In each downsampling stage, cross-scale feature interaction is introduced through skip connections to generate point cloud features at different scale levels. The hierarchical graph construction module is used to construct the point cloud features of different scale levels into a hierarchical graph structure. Each scale level corresponds to a layer of nodes in the graph structure. The node features are the point cloud features at that scale. Nodes within a layer are connected through spatial topological relationships, and nodes across scale layers are connected through scale mapping relationships. The geometric perception cross-scale attention fusion module is used to achieve feature fusion of nodes across scales and within layers through a geometric perception attention mechanism based on the hierarchical graph structure, and output fused features. The semantic refinement module is used to learn attention weights for the fused features in the channel dimension, spatial dimension, and scale dimension through multi-dimensional joint attention. The attention weights of the three dimensions are then interacted with the fused features to obtain refined semantic features and output point cloud semantic segmentation results.

[0049] For specific limitations regarding the point cloud semantic segmentation system based on multi-scale feature fusion and attention mechanisms, please refer to the limitations of the point cloud semantic segmentation method based on multi-scale feature fusion and attention mechanisms mentioned above, which will not be repeated here. Each module in the aforementioned point cloud semantic segmentation system based on multi-scale feature fusion and attention mechanisms can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0050] Furthermore, the specific embodiments of the present invention are not limited to the above description, and there may be various alternative solutions, all of which fall within the protection scope of the present invention. For example: The feature extraction sub-network in the multi-scale feature extraction module can adopt different network structures such as PointTransformer and PCT.

[0051] The construction of the set of edges within a layer in the hierarchical graph construction module can be achieved using a DBSCAN clustering-based method.

[0052] The geometric feature coding unit can incorporate other geometric features, such as geodesic distance and occlusion relationship.

[0053] The dynamic receptive field adjustment unit can determine the geometric complexity based on curvature value or local entropy value alone, and the specific value of the neighborhood range can also be adjusted according to the actual application scenario.

[0054] The multi-dimensional joint attention unit in the semantic refinement module can use additive interaction instead of multiplicative interaction. Compared with the prior art, the present invention has the following technical effects: (1) The present invention constructs a hierarchical graph structure through a hierarchical graph construction module, which explicitly models the topological relationship between point cloud features at different scales, so that multi-scale features can maintain continuous topological association during the fusion process. This overcomes the problem of scale information breakage caused by fixed sampling or interpolation operations in the prior art, reduces the loss of edge information, and improves the accuracy of point cloud semantic segmentation.

[0055] (2) The geometric perception cross-scale attention fusion module of the present invention explicitly encodes the geometric features of point cloud (such as Euclidean distance, angle between normal vectors, curvature difference, etc.) into the calculation of attention weights, so that the allocation of attention weights not only considers the similarity of features, but also fully reflects the geometric correlation strength between points, which is in line with the physical geometric characteristics of point cloud data and solves the problem of mismatch between existing attention mechanisms and geometric features, especially showing higher accuracy in the segmentation of complex surfaces.

[0056] (3) The dynamic receptive field adjustment unit can dynamically adjust the neighborhood range of attention calculation according to the geometric complexity of the region where the node is located. In regions with high geometric complexity, a smaller neighborhood range is used to focus on details, while in regions with low geometric complexity, a larger neighborhood range is used to obtain more global context information, thus achieving a good balance between computational efficiency and segmentation accuracy, and is suitable for large-scale point cloud scenarios.

[0057] (4) The multi-dimensional joint attention unit in the semantic refinement module simultaneously learns the attention weights of the channel dimension, spatial dimension and scale dimension of the features. It can effectively filter key features, strengthen the internal association of the same object, and adjust the contribution of features at different scales according to the characteristics of different regions, thereby further improving the discriminativeness and robustness of semantic features.

[0058] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a point cloud semantic segmentation method based on multi-scale feature fusion and attention mechanism.

[0059] In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of a point cloud semantic segmentation method based on multi-scale feature fusion and attention mechanisms.

[0060] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0061] The present invention provides a detailed description of a point cloud semantic segmentation method and system based on multi-scale feature fusion and attention mechanism. Specific examples have been used to illustrate the principles and implementation methods of the invention, and the descriptions of these embodiments are merely for the purpose of helping to understand the core ideas of the invention. It should be noted that those skilled in the art can make various improvements and modifications to the invention without departing from its principles, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A point cloud semantic segmentation method based on multi-scale feature fusion and attention mechanism, characterized in that, Includes the following steps: S100: Preprocesses the input point cloud and performs progressive downsampling through multiple cascaded feature extraction subnetworks. In each downsampling stage, cross-scale feature interaction is introduced through skip connections to generate point cloud features at different scale levels. S200: Construct a hierarchical graph structure from point cloud features at different scale levels. Each scale level corresponds to a layer of nodes in the graph structure. The node features are the point cloud features at that scale. Nodes within a layer are connected through spatial topological relationships, and nodes across scale layers are connected through scale mapping relationships. S300: Based on a hierarchical graph structure, it calculates the geometric features between nodes and dynamically adjusts the neighborhood range of attention calculation according to the geometric complexity of the region where the node is located. Through a geometry-aware attention mechanism, it achieves feature fusion between nodes across scales and within layers, outputting fused features. The geometry-aware attention mechanism in S300 includes: S310: For each pair of nodes connected by the edge in the hierarchical graph structure, calculate the Euclidean distance, the angle between the normal vectors and the curvature difference between each pair of nodes, and convert them into Euclidean distance mapping value, normal vector angle mapping value and curvature difference mapping value respectively through the corresponding mapping function. Then, fuse the three mapping values ​​into a comprehensive geometric weight through the fusion function. S320: Combine the feature similarity of nodes with the comprehensive geometric weight to calculate the attention weights of nodes across scales and within layers; S330: Dynamically adjust the neighborhood range of attention calculation based on the geometric complexity of the region where each node is located. Geometric complexity is measured by the curvature value and local entropy value of the corresponding node. By comparing the curvature value with a preset curvature threshold and the local entropy value with a preset local entropy threshold, the nodes are divided into multiple geometric complexity categories. Each geometric complexity category corresponds to a preset neighborhood range value, and the higher the geometric complexity of the category, the smaller the neighborhood range value. S340: Based on attention weights and a defined neighborhood range, the features of each node's neighboring nodes are weighted and aggregated, and the original features are added to the aggregated features through residual connections to output the fusion features at the finest scale level. S310 includes: S311: For any two nodes in the hierarchical graph structure G and Calculate the Euclidean distance between them. And through the mapping function The Euclidean distance mapping value is obtained, where These are learnable parameters; S312: Computation Node and The angle between the normal vectors and map it to ,in, and They are nodes and The corresponding normal vector; S313: Computation Node and Curvature difference ,in and They are nodes and The curvature of the corresponding point, and through the mapping function The curvature difference mapping value is obtained, where This represents the maximum curvature of all points at this scale level. S314: Through the fusion function The Euclidean distance mapping value, the normal vector angle mapping value, and the curvature difference mapping value are integrated into a comprehensive geometric weight, where, , , These are learnable fusion weights; The formula for calculating attention weights in S320 is: in, For nodes For nodes Attention weights For nodes With nodes Feature similarity is calculated by passing node features through a query projection matrix. Bond projection matrix The result is obtained by performing a dot product operation after mapping, i.e. , For nodes With nodes Geometric features, For geometric feature mapping function, These are the geometric feature weighting coefficients. For feature dimension, To Dimensional normalization operation; Weighted aggregation in S340 is achieved using the following formula: in, The features of node i after fusion Let i be the original feature of node i. Let be the attention weight of node i to node j. The projection matrix is ​​the value. This is a layer normalization operation; the original features are added to the aggregated features through residual connections, preserving the original feature information. S400: Through multi-dimensional joint attention, attention weights are learned for the fused features in the channel dimension, spatial dimension and scale dimension respectively. The attention weights of the three dimensions are then interacted with the fused features to obtain refined semantic features and output point cloud semantic segmentation results.

2. The method according to claim 1, characterized in that, Each feature extraction subnetwork in S100 extracts features in the following way: S110: For each point in the current scale level, select its K nearest neighbors, calculate the difference between the relative coordinates and normal vectors of the point and each of its nearest neighbors, and form local geometric features. The local geometric features are then mapped to a high-dimensional feature space through a multilayer perceptron to obtain the local feature vector of each point. S120: Perform global max pooling on the local feature vectors of all points in the current scale layer to obtain the global feature vector. Concatenate the global feature vector with the local feature vector of each point to obtain the preliminary feature representation. Then, refine the preliminary feature representation through convolutional layers and activation functions to output the point cloud features of the current scale layer. S130: An adaptive sampling strategy combining geometric features is adopted to determine the point set of the next scale level from the point set of the current scale level; wherein, the adaptive sampling strategy includes: calculating the curvature and semantic boundary score of each point based on the point cloud features of the current scale level, sorting them from high to low scores, and selecting a preset number of points as retained points to achieve downsampling. S140: The local geometric features of the previous scale level are interpolated and mapped onto the point set of the current scale level, and then fused with the local and global features extracted by the current scale level itself to generate the point cloud features of the current scale level.

3. The method according to claim 2, characterized in that, S200 includes: S210: For each scale level, construct an intra-layer subgraph, which includes a node set, an intra-layer edge set, and a node feature set. Each node in the node set corresponds to a point at that scale level, and the node feature set represents the point cloud features at that scale level. The intra-layer edge set is constructed using a combination of dynamic K-nearest neighbors and density filtering, and includes: S211: For each node in the node set, dynamically adjust the number of K nearest neighbors according to the density of its surrounding points. In areas where the point density is higher than a preset threshold, set the K value to the first value. In areas where the point density is lower than the preset threshold, set the K value to the second value. The first value is less than the second value. S212: For each node, select its K nearest neighbors as neighboring nodes, and establish edges between the node and each neighboring node to form a preliminary set of in-layer edges; S213: Perform density filtering on the initial intra-layer edge set to remove edges whose distance exceeds a preset distance threshold or whose feature difference exceeds a preset difference threshold, thus obtaining the optimized intra-layer edge set. S220: Construct a set of cross-scale edges between adjacent scale levels. The set of cross-scale edges is used to connect the nodes of scale level L and scale level L+1. The establishment of cross-scale edges is based on the sampling correspondence between the upper and lower level point sets. A many-to-many mapping method between parent nodes and multiple child nodes is adopted. In this method, one parent node in scale level L+1 corresponds to multiple child nodes in scale level L. The number of multiple child nodes is within a preset range. S230: Set an initial weight for each cross-scale edge in the cross-scale edge set. The initial weight is determined based on the spatial distance and feature similarity between the parent and child nodes connected by the cross-scale edge.

4. The method according to claim 3, characterized in that, The S330 includes: Obtain the curvature value of each node at the current scale level. and local entropy And preset curvature threshold and local entropy threshold ; When node i satisfies > and > When this occurs, it indicates that the node is located in a region that is both geometrically and semantically complex, and the neighborhood range of the node is adjusted to the number of its first nearest neighbors. ; When node i satisfies > and ≤ When this occurs, it indicates that the node is located in a geometrically complex but semantically clear region, and the node's neighborhood is adjusted to the second nearest neighbor number. ; When node i satisfies and > When this occurs, it indicates that the node is located in a geometrically simple but semantically confusing region, and the node's neighborhood is adjusted to the third nearest neighbor number. ; When node i satisfies and ≤ When this occurs, it indicates that the node is located in a region that is both geometrically and semantically simple, and the neighborhood range of the node is adjusted to the fourth nearest neighbor. ; in, , , , The values ​​increase sequentially.

5. The method according to claim 4, characterized in that, Weighted aggregation in S340 is achieved using the following formula: in, The features of node i after fusion Let i be the original feature of node i. Let be the attention weight of node i to node j. The projection matrix is ​​the value. This is a layer normalization operation; the original features are added to the aggregated features through residual connections, preserving the original feature information.

6. The method according to claim 5, characterized in that, In S400, semantic refinement is achieved through multi-dimensional joint attention, specifically including: S410: Perform global average pooling on the fused features in the spatial dimension to obtain global statistical information for each channel. Input the global statistical information into the multilayer perceptron and output the channel dimension attention weight for each channel. At the same time, perform global average pooling on the fused features in the channel dimension to obtain global statistical information for each node. Input the global statistical information into the multilayer perceptron and output the spatial dimension attention weight for each node. S420: The fused features are concatenated with the upsampled features of other scale levels. The concatenated features are then input into a multilayer perceptron, and the scale dimension attention weights are output. The number of scale dimension attention weights is the same as the number of scale levels, and the sum of the attention weights of each scale dimension is 1. S430: The fused features are multiplied element-wise with the channel dimension attention weights, spatial dimension attention weights and scale dimension attention weights to obtain the refined semantic features; S440: Input the refined semantic features into the classifier, output the semantic label probability distribution of each point, take the label with the highest probability as the semantic label of that point, and obtain the semantic segmentation result at the finest scale level.

7. A point cloud semantic segmentation system based on multi-scale feature fusion and attention mechanism, used to perform the steps of the method according to any one of claims 1 to 6, characterized in that, include: The multi-scale feature extraction module is used to preprocess the input point cloud and perform progressive downsampling through multiple cascaded feature extraction sub-networks. In each downsampling stage, cross-scale feature interaction is introduced through skip connections to generate point cloud features at different scale levels. The hierarchical graph construction module is used to construct the point cloud features of different scale levels into a hierarchical graph structure. Each scale level corresponds to a layer of nodes in the graph structure. The node features are the point cloud features at that scale. Nodes within a layer are connected through spatial topological relationships, and nodes across scale layers are connected through scale mapping relationships. The geometric perception cross-scale attention fusion module is used to achieve feature fusion of nodes across scales and within layers through a geometric perception attention mechanism based on the hierarchical graph structure, and output fused features. The semantic refinement module is used to learn attention weights for the fused features in the channel dimension, spatial dimension, and scale dimension through multi-dimensional joint attention. The attention weights of the three dimensions are then interacted with the fused features to obtain refined semantic features and output point cloud semantic segmentation results.

Images