Local geometry affine and dynamic scale fusion point cloud segmentation method and storage medium thereof

Abstract

The application discloses a local geometric affine and dynamic scale fusion point cloud segmentation method, comprising the following steps: step 1, obtaining and preprocessing point cloud data, performing initial mapping on input point features, and constructing a local neighborhood through farthest point sampling and ball query; step 2, constructing difference features of a center point and neighborhood points in each local neighborhood, estimating a local normalized scale, and performing residual geometric affine modulation; step 3, performing maximum pooling and attention pooling under multiple neighborhood scales respectively, and extracting local significant responses and neighborhood context information; step 4, generating dynamic weights according to response strengths of different scale branches, and performing point-by-point adaptive weighted fusion on multi-scale features; step 5, combining local adaptive geometric affine, multi-scale hybrid pooling and dynamic scale fusion into a stackable feature extraction unit, and repeatedly performing at multiple levels of an encoder; step 6, in the decoding stage, restoring point-level resolution layer by layer through interpolation upsampling and skip connection, and outputting semantic class labels of each point. The application realizes collaborative modeling of local geometric changes, neighborhood context and scale selection processes, thereby improving the semantic segmentation accuracy in complex scenes and multi-scale structure regions.

Description

Technical Field

[0001] This invention relates to the field of intelligent sensing, and more particularly to the field of 3D scene understanding and point cloud segmentation technology. Background Technology

[0002] Point cloud semantic segmentation, a key technology in 3D scene understanding, is of great significance in fields such as autonomous driving, robot perception, indoor scene understanding, and 3D digital modeling. Point cloud data can directly preserve the spatial geometry of targets, but its inherent characteristics—sparse, disordered, non-uniform distribution, and susceptibility to occlusion and noise—pose significant challenges to point-by-point semantic recognition tasks. Early methods relied heavily on manually generated geometric features and traditional classifiers, which struggled to meet the high-precision segmentation requirements of large-scale point clouds in complex scenes.

[0003] With the development of deep learning, point-based methods, represented by PointNet, PointNet++, and PointNeXt, can directly process raw point clouds and improve segmentation performance through local neighborhood sampling and hierarchical feature aggregation. For example, the published patent document CN115527032A, filed on December 27, 2022, entitled "A 3D Point Cloud Classification and Segmentation Method Integrating Dense Connection Mechanism and PointNet++," discloses a data processing method that integrates PointNet and cloud classification and segmentation. However, similar existing methods still have significant shortcomings:

[0004] On the one hand, many methods use a uniform or coarse-grained feature transformation method in the local neighborhood, which makes it difficult to accurately characterize the differences in the intensity of feature changes in different local regions, resulting in insufficient expressive power for boundary regions and non-uniformly distributed regions.

[0005] On the other hand, single-scale neighborhood aggregation cannot simultaneously take into account local boundary details and large-scale context structure, while fixed-weight multi-scale fusion methods cannot adaptively select more appropriate scale information according to the structural needs of different regions, thus limiting the segmentation accuracy and robustness in complex scenes. Summary of the Invention

[0006] This invention addresses the shortcomings of existing point cloud semantic segmentation methods, such as insufficient characterization of local feature changes, difficulty in combining boundary details and contextual information with single-scale neighborhood aggregation, and lack of adaptability in fixed-weight fusion. It proposes a point cloud semantic segmentation method based on local geometric affine transformation and dynamic scale fusion. This method achieves collaborative modeling of local geometric changes, neighborhood context, and scale selection processes through steps such as local adaptive geometric affine transformation, multi-scale hybrid pooling, and dynamic scale fusion, thereby improving the semantic segmentation accuracy in complex scenes and multi-scale structural regions.

[0007] To achieve the above objectives, the technical solution adopted by this invention is: a point cloud segmentation method based on local geometric affine transformation and dynamic scale fusion, comprising the following steps:

[0008] Step 1: Acquire and preprocess point cloud data, perform initial mapping on input point features, and construct local neighborhoods through farthest point sampling and ball query;

[0009] Step 2: Construct the difference features between the center point and the neighboring points in each local mail neighborhood, estimate the local normalized scale, and perform residual geometric affine modulation.

[0010] Step 3: Perform max pooling and attention pooling at multiple neighborhood scales to extract local salient responses and neighborhood context information;

[0011] Step 4: Generate dynamic weights based on the response intensity of branches at different scales, and perform point-by-point adaptive weighted fusion of multi-scale features;

[0012] Step 5: Combine local adaptive geometric affine transformation, multi-scale hybrid pooling, and dynamic scale fusion into stackable feature extraction units, and repeat the process at multiple levels of the encoder.

[0013] Step 6: In the decoding stage, point-level resolution is restored layer by layer through interpolation upsampling and skip connections, and the semantic category label of each point is output.

[0014] Step 1 includes the following steps:

[0015] The original point cloud with consistent data format and number of features is used as input to the encoder network;

[0016] The input point features are initially mapped using a multilayer perceptron, projecting them onto a unified feature space.

[0017] The set of center points is obtained by sampling from the farthest point, and a ball query is used to construct a local region consisting of K neighboring points for each center point;

[0018] The normalized scale is adaptively estimated within each local group by using a local adaptive geometric affine module, and the differential representation between the center point and the neighboring points is enhanced by residual affine correction to obtain the enhanced neighborhood features.

[0019] Step 2 includes the following steps:

[0020] For any center point and its neighboring points in its local neighborhood, first construct the difference features of the neighboring points relative to the center point to explicitly describe the feature change trend in the local region.

[0021] The normalized scaling factor is estimated independently for each local group, and the local scale can be obtained from the mean square statistics on the channel dimension and the neighborhood dimension.

[0022] Based on the local scale, normalization is performed on the differential features;

[0023] Learnable channel-level scaling and bias parameters are introduced to perform affine modulation on the normalized differential features;

[0024] A residual enhancement strategy is adopted to write back the affine correction term into the original neighborhood features to obtain the enhanced neighborhood features;

[0025] The enhanced neighborhood features obtained are then used to perform local neighborhood aggregation through a multi-scale hybrid pooling module.

[0026] Step 3 includes the following steps:

[0027] Multiple local sub-neighborhoods of different scales are constructed within the same layer, and features at each scale are aggregated respectively;

[0028] Attention pooling is introduced, which generates neighborhood attention scores for the current scale through a shared mapping function and performs normalization on the neighborhood dimension;

[0029] The attention-aggregated features are obtained by weighted summation of the neighborhood features.

[0030] A preset dynamic gating fusion strategy is used to concatenate the two types of aggregation results and their differences and then feed them into the gating function to obtain the channel-wise fusion coefficients, thereby obtaining the hybrid pooling features at the current scale.

[0031] By using a multi-scale hybrid pooling module, local salient responses and contextual information are modeled simultaneously at each scale, resulting in hybrid features of multiple scale branches.

[0032] The mixed features of multiple scale branches are used to achieve scale-level adaptive selection through a dynamic scale fusion module.

[0033] Step 4 includes the following steps:

[0034] For each scale branch, the scale scoring function is used to generate the response score at each center point of the current scale;

[0035] The response scores at each scale are concatenated along the scale dimension and normalized using the Softmax function to obtain the dynamic weights for each scale.

[0036] After obtaining the dynamic scale weights, the features of each scale are summed point by point to obtain the scale-adaptive feature representation after fusion.

[0037] After scale fusion, a projection mapping consisting of 1×1 convolution, batch normalization, and nonlinear activation functions is introduced to renormalize and transform the fusion result.

[0038] In step 5, the local adaptive geometric affine, multi-scale hybrid pooling and dynamic scale fusion operations are repeatedly performed in multiple layers of the encoder. The encoder consists of multiple ensemble abstraction layers, with the number of center points in each layer gradually decreasing and the number of feature channels gradually increasing, for hierarchical feature extraction from local to global.

[0039] In step 6, the decoder adopts a layer-by-layer correspondence structure corresponding to the encoder, recovers the number of points through nearest neighbor interpolation or inverse distance weighted interpolation, and combines skip connections to fuse shallow spatial details and deep semantic information. Finally, the semantic category label of each point is output through a shared multilayer perceptron to complete end-to-end point cloud semantic segmentation.

[0040] The storage medium is a computer-readable storage medium for storing software program code, which is used to execute the local geometric affine and dynamic scale fusion point cloud segmentation method.

[0041] The storage medium operates in a computer device containing a graphics processing unit, video memory, and a processor.

[0042] This invention has the following advantages:

[0043] 1. Point cloud semantic segmentation methods can more accurately characterize the intensity of local geometric changes between the center point and neighboring points by independently estimating the normalized scale for each local group and performing residual affine modulation on the neighborhood difference features, thereby enhancing the model's ability to perceive boundary regions, non-uniformly distributed regions and complex structural regions.

[0044] 2. The point cloud semantic segmentation method jointly performs max pooling and attention pooling at multiple neighborhood scales and uses a dynamic gating mechanism to adaptively fuse the two types of aggregation results. This can simultaneously preserve local salient responses and neighborhood context information, thereby improving the stability and discriminativeness of local feature aggregation.

[0045] 3. Point cloud semantic segmentation methods learn the response intensity of branches at different scales and perform point-by-point adaptive weighted fusion of multi-scale features, enabling the model to automatically select a more suitable neighborhood range based on the differences in local structure, thereby improving the segmentation effect in complex boundary regions and scale-changing scenes. Attached Figure Description

[0046] The following is a brief explanation of the content represented by each figure in this specification:

[0047] Figure 1This is a flowchart illustrating the point cloud semantic segmentation method proposed in this invention;

[0048] Figure 2 This is a network structure diagram of the point cloud semantic segmentation method proposed in this invention;

[0049] Figure 3 This is a schematic diagram of the local adaptive geometric affine module structure proposed in this invention;

[0050] Figure 4 This is a schematic diagram of the multi-scale hybrid pooling module structure proposed in this invention;

[0051] Figure 5 This is a schematic diagram of the dynamic scale fusion module structure proposed in this invention;

[0052] Figure 6 This is a visualization of the method of the present invention on the S3DIS dataset;

[0053] Figure 7 This is a visualization of the method of the present invention on the ScanNetV2 dataset. Detailed Implementation

[0054] The following description, with reference to the accompanying drawings, details the specific implementation of the present invention, including the shape and structure of each component, the relative positions and connections between the parts, the function and working principle of each part, the manufacturing process, and the operation and use methods, to help those skilled in the art to have a more complete, accurate, and in-depth understanding of the inventive concept and technical solution of the present invention.

[0055] This invention relates to a scheme for 3D scene understanding and point cloud segmentation. It is a point cloud segmentation method that fuses local geometric affine transformations and dynamic scales. The following is a detailed description of the point cloud segmentation method of this invention based on the accompanying drawings:

[0056] Figure 1This is a flowchart illustrating the point cloud semantic segmentation method proposed in this invention. It mainly includes the following steps: point cloud data input and initialization, local region construction, local adaptive geometric affine, multi-scale hybrid pooling, dynamic scale fusion, multi-level feature extraction by the encoder, layer-by-layer upsampling by the decoder, and final semantic label output. First, preliminary feature mapping is performed on the input raw point cloud, and the center point of the current layer is obtained by sampling the farthest point. Then, a local neighborhood is constructed for each center point using sphere query. Subsequently, local adaptive geometric affine enhancement is performed on the local neighborhood features, followed by hybrid pooling at multiple neighborhood scales. Finally, local feature representation is completed through dynamic scale fusion. After repeating this process in each layer of the encoder, the number of points is recovered and features are refined through interpolation and skip connections in the decoder, ultimately outputting the corresponding semantic category label for each point in the original point cloud.

[0057] Figure 2 The overall network architecture is defined as follows. Taking raw point clouds with consistent data format and feature counts as input, the point cloud data can be represented as... ,in Represents the three-dimensional coordinates of the i-th point. Additional attribute features can include color information, reflectance intensity, or other sensory attributes. After inputting the original point cloud into the encoder network, a multilayer perceptron is first used to perform an initial mapping of the input point features, projecting them onto a unified feature space. Subsequently, a set of center points is obtained through farthest point sampling, and a sphere query is used to construct a local region consisting of K neighboring points for each center point, thereby jointly modeling the semantic information and geometric structure of the points at a local scale. To enhance the model's ability to perceive local geometric changes and boundary differences, this invention uses a local adaptive geometric affine module to adaptively estimate the normalized scale within each local group, and uses residual affine correction to strengthen the difference representation between the center point and its neighboring points, obtaining enhanced neighborhood features.

[0058] Figure 3 This is a structural diagram of a locally adaptive geometric affine module. For any center point and its neighboring points in its local neighborhood, we first construct the difference features of the neighboring points relative to the center point to explicitly describe the feature change trend in the local region. The difference features can be expressed as:

[0059]

[0060] in, This represents the characteristics of the j-th neighboring point of the i-th center point. The central point features are represented. To avoid the problem of insufficient local adaptability caused by using a uniform global scale, this invention further estimates a normalized scale factor independently for each local group. This local scale can be obtained based on the mean square statistics of the channel dimension and the neighborhood dimension, and its expression can be written as:

[0061]

[0062] Where C represents the number of feature channels and K represents the number of neighborhood points. To prevent numerical instability caused by small constants, unlike the globally shared scale, the normalized scale is adaptively estimated from the current local neighborhood, thus more accurately reflecting the intensity of feature fluctuations in that local region. Based on this local scale, normalization can be performed on the differential features:

[0063]

[0064] After local normalization, learnable channel-level scaling and bias parameters are introduced to perform affine modulation on the normalized differential features. The expression for this modulation can be written as:

[0065]

[0066] To enhance local differential representation while preserving the original neighborhood semantic information, this invention employs a residual enhancement strategy, writing back the affine correction term into the original neighborhood features to obtain the enhanced neighborhood features:

[0067]

[0068] After obtaining the enhanced neighborhood features, the present invention further performs local neighborhood aggregation through a multi-scale hybrid pooling module.

[0069] Figure 4 This is a diagram of the multi-scale hybrid pooling module structure. Since point cloud scenes simultaneously contain fine-grained boundary information and a large-scale contextual structure, different regions have significantly different requirements for the receptive field of their neighborhoods. Therefore, multiple local sub-neighborhoods of different scales are constructed within the same layer, and features at each scale are aggregated separately. For a local sub-neighborhood corresponding to scale s, it can be represented as:

[0070]

[0071] Where K_s represents the number of neighborhood points participating in aggregation at the current scale. For each scale, max pooling is first used to extract salient response features from the local neighborhood:

[0072]

[0073] Since max pooling relies on only a single extremum, it easily overlooks the contextual information contained in the remaining neighboring points. Therefore, this invention further introduces attention pooling. By generating a neighborhood attention score for the current scale through a shared mapping function and performing normalization on the neighborhood dimension, we can obtain:

[0074]

[0075] Then, the neighborhood features are weighted and summed to obtain the attention aggregation features:

[0076]

[0077] Since max pooling and attention pooling emphasize local salient responses and neighborhood context information respectively, they are highly complementary. To enable the model to adaptively balance the two aggregation methods according to different local structures, this invention designs a dynamic gating fusion strategy. The two aggregation results and their differences are concatenated and fed into the gating function to obtain channel-wise fusion coefficients.

[0078]

[0079] This yields the hybrid pooling characteristics at the current scale:

[0080]

[0081] Through the above process, the multi-scale hybrid pooling module can simultaneously model local salient responses and contextual information at each scale, thereby obtaining hybrid features of multiple scale branches.

[0082] After obtaining the mixed features of multiple scale branches, the present invention further completes the scale-level adaptive selection through a dynamic scale fusion module.

[0083] Figure 5 This is a structural diagram of the dynamic scale fusion module. Since boundary regions typically rely more on small-scale details, while flat and large-structure regions require larger-scale context, fixed-weight scale fusion methods are ill-suited to the diverse needs of complex point cloud scenarios. Therefore, for each scale branch, a scale scoring function is first used to generate the response score at each center point for the current scale:

[0084]

[0085] Subsequently, the response scores at each scale are concatenated along the scale dimension and normalized using the Softmax function to obtain the dynamic weights for each scale:

[0086]

[0087] After obtaining the dynamic scale weights, the features at each scale are summed point by point to obtain the fused scale-adaptive feature representation:

[0088]

[0089] To further enhance the stability and expressive power of the fused features, a projection mapping consisting of 1×1 convolution, batch normalization, and a nonlinear activation function is introduced after scale fusion to renormalize and transform the fusion result. Its expression can be written as:

[0090]

[0091] Through the above process, the dynamic scale fusion module can achieve point-by-point dynamic weight allocation at the scale level, so that the model no longer relies on a fixed scale fusion strategy, but adaptively allocates the fusion ratio according to the response intensity of different scale branches, thereby improving the feature representation ability in complex boundary regions and scale-changing scenarios.

[0092] The aforementioned local adaptive geometric affine transformation, multi-scale hybrid pooling, and dynamic scale fusion operations are repeatedly performed across multiple layers of the encoder. In one embodiment, the encoder consists of multiple ensemble abstraction layers, with the number of center points gradually decreasing and the number of feature channels gradually increasing in each layer to achieve hierarchical feature extraction from local to global. The decoder employs a layer-by-layer upsampling structure corresponding to the encoder, recovering the number of points through nearest neighbor interpolation or inverse distance weighted interpolation, and combining skip connections to fuse shallow spatial details with deep semantic information. Finally, a shared multilayer perceptron outputs the semantic category label for each point, completing end-to-end point cloud semantic segmentation.

[0093] The system configuration used for point cloud semantic segmentation in this invention can be an NVIDIA GeForce RTX 4090 graphics processing unit, 24.0 GB of video memory, and a 13th generation Intel Core i9-13900KF processor. In one embodiment, the invention can be trained and validated on the S3DIS dataset and the ScanNetV2 dataset. The invention achieves mIoU of 70.6% and 75.6% in the Area-5 and 6-fold evaluation settings of the S3DIS dataset, respectively, and 72.1% and 72.4% in the validation and test sets of the ScanNetV2 dataset, respectively, indicating that the invention has good semantic segmentation performance.

[0094] Figure 6 The figure shows the semantic segmentation performance of this method on the S3DIS dataset. As can be seen from the figure, the present invention can obtain more complete and clearer segmentation boundaries in typical object regions such as columns, door panels, blackboards, and sofas, especially showing better class discrimination ability in edge regions with obvious local structural changes and multi-class adjacent regions.

[0095] Figure 7The results show the semantic segmentation performance of this method on the ScanNetV2 dataset. The segmentation results are closer to the baseline values. For areas where walls and furniture meet, areas near tables and chairs, and areas with strong local occlusion, this invention can better preserve the local structural contours and obtain clearer class boundaries, indicating that there is a good synergistic effect among local adaptive geometric affine transformation, multi-scale hybrid pooling, and dynamic scale fusion.

[0096] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other occasions without modification, are all within the protection scope of the present invention.

Claims

1. A point cloud segmentation method based on local geometric affine transformation and dynamic scale fusion, characterized in that, Includes the following steps: Step 1: Acquire and preprocess point cloud data, perform initial mapping on input point features, and construct local neighborhoods through farthest point sampling and ball query; Step 2: Construct the difference features between the center point and the neighboring points in each local mail neighborhood, estimate the local normalized scale, and perform residual geometric affine modulation. Step 3: Perform max pooling and attention pooling at multiple neighborhood scales to extract local salient responses and neighborhood context information; Step 4: Generate dynamic weights based on the response intensity of branches at different scales, and perform point-by-point adaptive weighted fusion of multi-scale features; Step 5: Combine local adaptive geometric affine transformation, multi-scale hybrid pooling, and dynamic scale fusion into stackable feature extraction units, and repeat the process at multiple levels of the encoder. Step 6: In the decoding stage, point-level resolution is restored layer by layer through interpolation upsampling and skip connections, and the semantic category label of each point is output.

2. The point cloud segmentation method based on local geometric affine transformation and dynamic scale fusion according to claim 1, characterized in that, Step 1 includes the following steps: The original point cloud with consistent data format and number of features is used as input to the encoder network; The input point features are initially mapped using a multilayer perceptron, projecting them onto a unified feature space. The set of center points is obtained by sampling from the farthest point, and a ball query is used to construct a local region consisting of K neighboring points for each center point; The normalized scale is adaptively estimated within each local group by using a local adaptive geometric affine module, and the differential representation between the center point and the neighboring points is enhanced by residual affine correction to obtain the enhanced neighborhood features.

3. The point cloud segmentation method based on local geometric affine transformation and dynamic scale fusion according to claim 1, characterized in that, Step 2 includes the following steps: For any center point and its neighboring points in its local neighborhood, first construct the difference features of the neighboring points relative to the center point to explicitly describe the feature change trend in the local region. The normalized scaling factor is estimated independently for each local group, and the local scale can be obtained from the mean square statistics on the channel dimension and the neighborhood dimension. Based on the local scale, normalization is performed on the differential features; Learnable channel-level scaling and bias parameters are introduced to perform affine modulation on the normalized differential features; A residual enhancement strategy is adopted to write back the affine correction term into the original neighborhood features to obtain the enhanced neighborhood features; The enhanced neighborhood features obtained are then used to perform local neighborhood aggregation through a multi-scale hybrid pooling module.

4. The point cloud segmentation method based on local geometric affine transformation and dynamic scale fusion according to claim 1, characterized in that, Step 3 includes the following steps: Multiple local sub-neighborhoods of different scales are constructed within the same layer, and features at each scale are aggregated respectively; Attention pooling is introduced, which generates neighborhood attention scores for the current scale through a shared mapping function and performs normalization on the neighborhood dimension; The attention-aggregated features are obtained by weighted summation of the neighborhood features. A preset dynamic gating fusion strategy is used to concatenate the two types of aggregation results and their differences and then feed them into the gating function to obtain the channel-wise fusion coefficients, thereby obtaining the hybrid pooling features at the current scale. By using a multi-scale hybrid pooling module, local salient responses and contextual information are modeled simultaneously at each scale, resulting in hybrid features of multiple scale branches. The mixed features of multiple scale branches are used to achieve scale-level adaptive selection through a dynamic scale fusion module.

5. The point cloud segmentation method based on local geometric affine transformation and dynamic scale fusion according to claim 1, characterized in that, Step 4 includes the following steps: For each scale branch, the scale scoring function is used to generate the response score at each center point of the current scale; The response scores at each scale are concatenated along the scale dimension and normalized using the Softmax function to obtain the dynamic weights for each scale. After obtaining the dynamic scale weights, the features of each scale are summed point by point to obtain the scale-adaptive feature representation after fusion. After scale fusion, a projection mapping consisting of 1×1 convolution, batch normalization, and nonlinear activation functions is introduced to renormalize and transform the fusion result.

6. The point cloud segmentation method based on local geometric affine transformation and dynamic scale fusion according to any one of claims 1-5, characterized in that, In step 5, the local adaptive geometric affine, multi-scale hybrid pooling and dynamic scale fusion operations are repeatedly performed in multiple layers of the encoder. The encoder consists of multiple ensemble abstraction layers, with the number of center points in each layer gradually decreasing and the number of feature channels gradually increasing, for hierarchical feature extraction from local to global.

7. The point cloud segmentation method based on local geometric affine transformation and dynamic scale fusion according to claim 6, characterized in that, In step 6, the decoder adopts a layer-by-layer correspondence structure corresponding to the encoder, recovers the number of points through nearest neighbor interpolation or inverse distance weighted interpolation, and combines skip connections to fuse shallow spatial details and deep semantic information. Finally, the semantic category label of each point is output through a shared multilayer perceptron to complete end-to-end point cloud semantic segmentation.

8. A storage medium, wherein the storage medium is a computer-readable storage medium for storing software program code, characterized in that: The software program code is used to execute the local geometric affine and dynamic scale fusion point cloud segmentation method as described in any one of claims 1-7.

9. The storage medium according to claim 8, characterized in that, The storage medium operates in a computer device containing a graphics processing unit, video memory, and a processor.

Images