A sparse point cloud semantic segmentation method based on mask modulation noise regression

Abstract

The present application relates to the technical field of three-dimensional point cloud, and discloses a sparse point cloud semantic segmentation method based on mask modulation noise regression.The method is as follows: training a point cloud semantic segmentation model based on mask modulation noise regression; in the training stage, a double-path joint optimization strategy is adopted to jointly optimize the semantic segmentation loss of the semantic prior segmentation network and the mask modulation noise regression loss of the mask guided diffusion branch; in the inference stage: collecting the original point cloud of the target region, preprocessing and segmenting the original point cloud to obtain a plurality of point cloud subblocks; extracting features of the point cloud subblocks to obtain point cloud subblock features, down-sampling the point cloud subblock features to obtain bottleneck features, up-sampling the bottleneck features to obtain semantic probability labels, and forming a point-level semantic segmentation probability graph according to the semantic probability labels.The present application solves the technical problem of insufficient robustness of the existing point cloud segmentation method to small targets and thin boundaries in real sparse and noisy point cloud scenes.

Description

Technical Field

[0001] This invention relates to the field of 3D point cloud technology, and in particular to a sparse point cloud semantic segmentation method based on mask modulation noise regression. Background Technology

[0002] 3D point cloud semantic segmentation is a crucial step in 3D scene understanding tasks such as remote sensing mapping and smart city construction. Its goal is to assign semantic category labels to each point in an unordered, unstructured point set, supporting applications such as city-level 3D modeling and environmental monitoring. Depending on the acquisition method, point clouds can originate from various sensors, including LiDAR, photogrammetric oblique 3D reconstruction, and depth cameras. Among these, oblique photogrammetric point clouds, in addition to their 3D geometric coordinates, also carry RGB color information reconstructed from multi-view images, which is beneficial for jointly utilizing geometry and texture for semantic discrimination. Therefore, it is widely used in large-scale urban and mountainous scenes.

[0003] However, point clouds in real-world production environments commonly suffer from sparse, uneven sampling, and measurement noise. On the one hand, due to viewpoint occlusion, reconstruction failures, or differences in the scale of the features themselves, many areas exhibit highly uneven density, with some areas exhibiting either excessively dense point clouds or severe undersampling. On the other hand, areas with blurred textures, strong reflections, and severe occlusion are prone to reconstruction artifacts and incorrect clustering, causing both geometric and color information to be interfered with by noise. For slender or tall targets such as power lines and power towers, as well as fine-scale structures such as alleyways, narrow roads, and slopes, the proportion of points is extremely low, and the spatial distribution is fragmented. Small targets and object boundaries often become the main sources of semantic segmentation errors.

[0004] For point cloud semantic segmentation, existing deep learning methods can be broadly categorized into several types, including direct point set methods based on Multi-Level Processing (MLP), local neighborhood aggregation methods based on convolution, and self-attention methods based on Transformer. MLP-based methods, represented by the PointNet series, model the permutation invariance of points by applying a shared MLP to each point and combining it with symmetric aggregation. They are structurally simple and have high inference efficiency, but their ability to characterize complex local geometric structures and fine-grained boundaries is limited. Convolution-based methods aggregate features within the neighborhood of points or voxels using kernel-point convolution or sparse convolution. They can effectively extract local structural information in the initial layer and have been validated in large-scale 3D scene segmentation. However, the receptive field of the convolution kernel itself is limited, and without stacking complex multi-scale or pyramidal structures, it is difficult to capture a wide range of contextual semantics. With the development of self-attention mechanisms, Transformer-based point cloud methods have gradually become a research hotspot. These methods explicitly model long-range dependencies through inter-point self-attention, combined with strategies such as relative position encoding, hierarchical sampling, and sequential attention. This achieves integrated modeling of local geometry and global semantics without relying on regular voxels or image projection, resulting in segmentation accuracy superior to traditional MLPs and convolutional networks on various indoor and outdoor benchmark datasets. However, the quadratic complexity of standard self-attention in terms of the number of points and its high memory overhead make end-to-end training and inference costly on large-scale sparse point clouds. Furthermore, lacking explicit local inductive biases similar to convolution, without additional geometrically aware position encoding or neighborhood constraints, they often exhibit oversmoothing of features and semantic confusion at small targets and object boundaries, and their robustness to noise and annotation uncertainties remains insufficient.

[0005] To improve discrimination capabilities under noisy and sparse conditions, recent research has introduced the "stepwise noise addition-reverse denoising" paradigm of diffusion models into 3D tasks, treating it as a process of explicitly modeling noise distribution and generating data priors. For point cloud semantic segmentation, existing work constructs two branches within a conditional-noise framework: a conditional network and a diffusion network. The conditional network performs semantic prediction, while the diffusion network injects multi-level noise into the feature space and learns denoising mappings. During training, it uses the diffusion noise construction system to enhance robustness, while during inference, it degenerates into a single-step forward pass to control overhead, thus balancing noise resistance and efficiency to some extent. However, these methods generally only use the time step as the condition for the diffusion process. The noise generated by the diffusion network is spatially approximately "indiscriminate perturbation," making it difficult to adaptively adjust the perturbation location and intensity according to the difficulty of the current semantic prediction: high-confidence regions are unnecessarily and repeatedly perturbed, while truly difficult boundaries and small target regions lack targeted enhancement signals, making it difficult to maximize the gain of the diffusion branch on the backbone network. Regarding the structure of diffusion networks, existing diffusion-based point cloud methods mostly adopt the global self-attention structure from the image domain to construct diffusion networks, aiming to maintain the ability to model the global scene under multi-level noise conditions. However, for large-scale sparse point clouds, global attention not only incurs huge computational costs but also suffers from significant low-frequency bias, with feature representations tending towards smooth overall shapes and insufficient sensitivity to local high-frequency information such as boundaries, edges, and slender structures. For land cover types such as power lines, curbs, and railings, which are mainly characterized by fine lines or boundary variations, diffusion networks that rely solely on global attention struggle to maintain high-resolution representation of details under multi-level noise environments.

[0006] Therefore, this invention proposes a sparse point cloud semantic segmentation method based on mask modulation noise regression to solve the problem that existing point cloud segmentation methods are not robust enough to small targets and fine boundaries in real sparse and noisy point cloud scenarios. Summary of the Invention

[0007] To address the technical problem that existing point cloud segmentation methods lack robustness to small targets and fine boundaries in real sparse and noisy point cloud scenarios, this invention provides a sparse point cloud semantic segmentation method and system based on mask-modulated noise regression.

[0008] To achieve the above objectives, the present invention adopts the following technical solution, including: A sparse point cloud semantic segmentation method based on mask modulation noise regression is proposed. The point cloud semantic segmentation model is trained based on mask modulation noise regression, and the training method is as follows: S1, Collect dense point cloud, preprocess the dense point cloud to obtain multiple point cloud sub-blocks; S2, set the semantic prior segmentation network as the main path, extract features from the point cloud sub-blocks, obtain the point cloud sub-block features, downsample the point cloud sub-block features to obtain the bottleneck features, upsample the bottleneck features to obtain the semantic probability labels, and form a point-level semantic segmentation probability map based on the semantic probability labels. S3, compress the point-level semantic segmentation probability map into a low-dimensional mask channel to obtain mask features; S4. Based on a preset noise schedule, Gaussian noise is injected into the feature of the point cloud sub-block at each time step to obtain noisy features. S5, set the mask-guided diffusion branch as the training auxiliary path, fuse the noisy features, time step embedding and mask features to obtain the fused features, downsample the fused features to obtain the noise regression features, upsample the noise regression features to obtain the predicted noise; S6. A mask modulation feature injector is set in the bottleneck layer of the semantic prior segmentation network and the mask-guided diffusion branch to perform cross-attention calculation on the bottleneck features and noise regression features. The calculated mask modulation features are unidirectionally injected into the semantic prior segmentation network, fused with the bottleneck features, and output semantic probability labels. S7, train the point cloud semantic segmentation model by minimizing the noise prediction loss and semantic segmentation loss; The trained point cloud semantic segmentation model is used to perform semantic segmentation on point clouds and generate semantic probability labels.

[0009] Preferably, the trained point cloud semantic segmentation model performs the following steps for semantic segmentation of point clouds: Step 1: Collect real point cloud data of the target area, preprocess and segment the real point cloud data to obtain multiple point cloud sub-blocks; Step 2: Extract features from the point cloud sub-blocks to obtain point cloud sub-block features, downsample the point cloud sub-block features to obtain bottleneck features, upsample the bottleneck features to obtain semantic probability labels, and form a point-level semantic segmentation probability map based on the semantic probability labels.

[0010] Preferably, the specific process of step S1 is as follows: S11, collect dense point clouds of the target area; S12, the dense point cloud is voxelized and sampled according to a fixed voxel size to control the local point density and remove obvious outliers or noise points; the coordinates and colors of the point cloud are decentered and normalized respectively to obtain the preprocessed point cloud. S13, the preprocessed point cloud is divided into overlapping sub-blocks according to a fixed spatial size to obtain multiple point cloud sub-blocks.

[0011] Preferably, the specific process of step S3 is as follows: S31, Obtain the point-level semantic segmentation probability map; S32 compresses the point-level semantic segmentation probability map into a mask feature vector through a multilayer perceptron. S33 uses a nonlinear function to restrict the mask feature vector to a fixed numerical range to obtain the mask features.

[0012] Preferably, the specific process of step S4 is as follows: S41, Preset Noise Schedule

[0013] ; in, Let T be the noise intensity, T be the total time step, and t be the current time step. The signal retention coefficients at the t-th diffusion step are... The cumulative signal retention coefficients from step 1 to step t are... This represents the signal retention coefficient at the s-th time step; S42, based on a preset noise schedule, performs feature analysis on point cloud sub-blocks at each time step. Injecting Gaussian noise yields noisy features. .

[0014] Preferably, the specific process of step S6 is as follows: S61, perform feature projection on the bottleneck features through MLP to obtain the query matrix, and perform feature projection on the noisy regression features through MLP to obtain the key matrix and value matrix; S62 filters noise information from bottleneck features and noise regression features through a cross-attention mechanism, and calculates mask modulation features based on the query matrix, key matrix, and value matrix. S63 performs learnable filtering on mask modulation features and injects the mask modulation features selected by the self-mask-guided diffusion branch into the semantic prior segmentation network through unidirectional injection.

[0015] This invention also provides a sparse point cloud semantic segmentation system based on mask modulation noise regression, applicable to the aforementioned sparse point cloud semantic segmentation method based on mask modulation noise regression. The system includes a point cloud semantic segmentation model and a mask-guided diffusion branch, as detailed below: Point cloud semantic segmentation model: Point cloud processing module: Collects dense point clouds of the target area, preprocesses and segments the dense point clouds to obtain multiple point cloud sub-blocks; Semantic prior segmentation network: Extract features from point cloud sub-blocks to obtain point cloud sub-block features, downsample point cloud sub-block features to obtain bottleneck features, upsample bottleneck features to obtain semantic probability labels, and form a point-level semantic segmentation probability map based on semantic probability labels. Mask-guided diffusion branching: Mask construction module: compresses the point-level semantic segmentation probability map into a low-dimensional mask channel to obtain mask features; Feature space noise injection module: Based on a preset noise schedule, Gaussian noise is injected into the feature of the point cloud sub-block at each time step to obtain noisy features; Mask-modulated noise regression network: Noisy features, time step embeddings and mask features are fused to obtain fused features. The fused features are downsampled to obtain noise regression features. The noise regression features are upsampled to obtain predicted noise. Mask Modulation Feature Injector: Performs cross-attention calculation on bottleneck features and noise regression features to generate mask modulation features, and injects the mask modulation features unidirectionally into the semantic prior segmentation network. The semantic prior segmentation network then completes subsequent upsampling, decoding, and classification output to generate semantic probability labels.

[0016] Preferably, the mask modulation noise regression network is a Transformer-based U-Net encoder-decoder structure; The encoder includes: Mesh feature aggregation module: transforms disordered point clouds into regular local region features; Random point discarding module: randomly shuffles the point cloud order; Feature encoding module: Converts point cloud features with location encoding into noisy regression features; Furthermore, the feature encoding module includes multiple self-attention blocks and a boundary sparse bridging path unit.

[0017] Preferably, the self-attention block includes: xCPE unit: Transforms unordered point cloud features into a feature sequence with a clear spatial address and distinguishable location; Mask embedding unit: forces the encoder to infer global and local structure from partially visible points; Two residual connection normalization units: standardize the input data; Window area attention unit: (1) Divide the feature maps of each scale in the mask modulation noise regression network into several non-overlapping local window regions according to the preset window region side length; (2) Perform multi-head self-attention computation independently within each window region to obtain the local features of window region attention; Boundary Sparse Bridging Path Unit: Constructs a sparse adjacency graph at the window domain boundary points, and applies a depthwise separable convolution to the sparse adjacency graph to output boundary enhancement features.

[0018] The present invention also provides a computer program product comprising a computer program / instruction that, when executed by a processor, implements the above-described sparse point cloud semantic segmentation method based on mask modulation noise regression.

[0019] The advantages of this invention are: 1. This invention proposes a sparse point cloud semantic segmentation method based on mask modulation noise regression. Under the prior-noise regression collaborative framework, the semantic prior segmentation network, which undertakes single-step semantic prediction, works in collaboration with the mask-guided diffusion branch, which only constructs the diffusion noise system during the training phase. During the training phase, the diffusion process is used to explicitly simulate multi-level noise perturbations and learn the noise regression mapping. During the inference phase, the auxiliary path during the training phase is closed, and only the single-step forward pass of the semantic prior segmentation network is retained, thereby achieving a balance between robustness and efficiency.

[0020] 2. This invention utilizes a point-level semantic segmentation probability generation mask output by a semantic prior segmentation network to achieve spatial selective modulation of noise, making the perturbation position and intensity adaptively aligned with the current prediction confidence: suppressing unnecessary perturbations in high-confidence regions, and strengthening perturbations on object boundaries, slender small targets, and noise-dominated regions, thereby more effectively improving the representation ability of difficult-to-segment regions, and thus significantly improving the segmentation accuracy of small targets and fine boundaries in sparse and noisy point clouds.

[0021] 3. In the mask-guided diffusion branch, the present invention adopts window-domain self-attention combined with a boundary sparse bridging structure. On the one hand, it significantly reduces the computation and memory overhead of global self-attention on large-scale sparse point clouds. On the other hand, it strengthens local high-frequency geometric details and introduces necessary cross-window-domain context consistency, making up for the problem that traditional global attention structures are insufficient in modeling slender targets and complex boundaries.

[0022] 4. This invention aims to achieve stable and precise segmentation of small targets and fine-scale structures such as narrow alleys in real sparse and noisy point cloud scenes such as large-scale cities and mountainous areas, providing a technical path that balances accuracy, robustness and engineering deployability for 3D scene understanding in remote sensing mapping and smart city construction.

[0023] 5. In the training phase of this invention, the diffusion branch guided by the mask is used to explicitly model perturbations and noise regression. In the inference phase, the semantic prior segmentation network is maintained in a single step, thereby achieving a balance between robustness and real-time performance.

[0024] 6. This invention employs window-domain self-attention and incorporates relative geometric bias in the training-phase noisy regression path. At the same time, it combines boundary sparse bridging to introduce necessary cross-window-domain context, so that attention computation is focused on the local neighborhood. Under controllable computational overhead, it highlights high-frequency details and complex geometric structures, effectively alleviates the excessive smoothing problem caused by global attention, and further enhances the coherence and separability of slender structures and complex boundaries.

[0025] 7. The semantic prior segmentation network backbone used in this invention can be flexibly selected from any point cloud segmentation network. The mask-guided diffusion branch and the mask-modulated feature injector are integrated in a modular manner, which is convenient for migration and reuse, and demonstrates good compatibility and scalability. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the modules of the sparse point cloud semantic segmentation system in this embodiment.

[0027] Figure 2 This is a schematic diagram of the structure of a self-attention block.

[0028] Figure 3 A visualization comparison of the experimental results for the Sensaturban dataset.

[0029] Figure 4 A visualization comparison of the experimental results for the Campus3D dataset.

[0030] Figure 5 A visual comparison of the experimental results for the QuBei dataset.

[0031] Figure 6 This is a flowchart of the point cloud semantic segmentation method in Example 2. Detailed Implementation

[0032] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Example 1

[0034] Please see Figure 1 , Figure 1 This is a schematic diagram of the sparse point cloud semantic segmentation system based on mask modulation noise regression provided in this embodiment. The point cloud semantic segmentation system includes a point cloud semantic segmentation model and a mask-guided diffusion branch. The point cloud semantic segmentation model includes: a point cloud processing module and a semantic prior segmentation network; the mask-guided diffusion branch includes: a mask construction module, a feature space noise injection module, a mask modulation noise regression network, and a mask modulation feature injector. Details are as follows: I. Point Cloud Processing Module.

[0035] Collect dense point clouds of the target area, preprocess and segment the dense point clouds to obtain multiple point cloud sub-blocks.

[0036] II. Semantic Prior Segmentation Network.

[0037] Feature extraction is performed on the point cloud sub-blocks to obtain point cloud sub-block features. The point cloud sub-block features are downsampled to obtain bottleneck features. The bottleneck features are upsampled to obtain semantic probability labels. A point-level semantic segmentation probability map is formed based on the semantic probability labels.

[0038] III. Mask Construction Module.

[0039] The point-level semantic segmentation probability map is compressed into a low-dimensional mask channel to obtain mask features.

[0040] IV. Feature Space Noise Injection Module.

[0041] Based on a preset noise schedule, Gaussian noise is injected into the feature of the point cloud sub-blocks at each time step to obtain noisy features.

[0042] V. Mask-modulated noise regression network.

[0043] Noisy features, time-step embeddings, and mask features are fused to obtain fused features. The fused features are downsampled to obtain noise regression features, and the noise regression features are upsampled to obtain predicted noise.

[0044] VI. Mask Modulation Feature Injector.

[0045] Cross-attention computation is performed on bottleneck features and noise regression features to generate mask modulation features. These mask modulation features are then unidirectionally injected into the semantic prior segmentation network, which performs subsequent upsampling, decoding, and classification output to generate semantic probability labels.

[0046] This embodiment 1 addresses the technical problem of insufficient robustness of existing point cloud segmentation methods to small targets and fine boundaries in realistic sparse and noisy point cloud scenarios. It proposes a sparse point cloud semantic segmentation system based on mask-modulated noise regression. While maintaining the single-step inference path and meeting engineering real-time requirements, this invention introduces a mask-guided diffusion branch that is only activated during the training phase and a window self-attention and boundary sparse bridging mechanism into the prior-noise regression collaborative framework. This enables spatially selective modeling of noise distribution and enhanced representation of local geometric details, thereby improving the overall segmentation accuracy and adaptability to complex geometric structures. At the same time, the semantic prior segmentation network that undertakes single-step semantic prediction and the mask-guided diffusion branch that only constructs the diffusion noise system during the training phase work together to achieve a balance between robustness and real-time performance.

[0047] The following will provide a detailed introduction to each module: I. Point Cloud Processing Module.

[0048] This module is used to acquire dense point clouds of a target area, preprocess and segment the dense point clouds to obtain multiple point cloud sub-blocks. The point cloud processing module includes: Point cloud acquisition module: Acquires the original point cloud of the target area through oblique photogrammetry; specifically, it completes flight path planning, joint calibration and aerial triangulation by using multi-view UAV oblique imagery and airborne GNSS / IMU trajectory, reconstructs a dense three-dimensional point cloud under a unified projection coordinate system, and retains the three-dimensional coordinates and RGB color attributes of each point, thereby obtaining the original point cloud of the target area.

[0049] Preprocessing module: The dense point cloud is voxelized and sampled according to a fixed voxel size to control the local point density and remove obvious outliers or noise points; the coordinates and colors of the point cloud are decentered and normalized respectively to obtain the preprocessed point cloud; in this embodiment, the dense point cloud is preprocessed to obtain the preprocessed point cloud. Specifically, the original point cloud is voxelized and sampled according to a fixed voxel size to control the local point density and remove obvious outliers or noise points; then the coordinates and colors of the point cloud are decentered and normalized respectively to unify the numerical scale and stabilize subsequent network training, thereby obtaining the preprocessed point cloud.

[0050] Point cloud sub-block partitioning module: Divides the preprocessed point cloud into overlapping sub-blocks according to a fixed spatial size, resulting in multiple point cloud sub-blocks. In this embodiment, the preprocessed point cloud is divided into overlapping sub-blocks according to a fixed spatial size, resulting in multiple point cloud sub-blocks. If the number of points in a sub-block exceeds the upper limit, it is downsampled to unify the scale. A k-nearest neighbor or spherical neighborhood index is pre-constructed for each point cloud sub-block, and optionally, geometric properties such as normal and curvature of the point cloud sub-block are calculated based on this index. These geometric properties are used as additional input features and fed into the subsequent network along with the coordinates of the points.

[0051] The point cloud sub-blocks are divided into training, validation, and test sets according to the scene or region, and the point-level semantic annotations in the training and validation data are sampled and verified to ensure the reliability of the supervised data.

[0052] II. Semantic Prior Segmentation Network.

[0053] Feature extraction is performed on the point cloud sub-blocks to obtain point cloud sub-block features. The point cloud sub-block features are downsampled to obtain bottleneck features. The bottleneck features are upsampled to obtain semantic probability labels. A point-level semantic segmentation probability map is formed based on the semantic probability labels.

[0054] In this embodiment, the semantic prior segmentation network is a Transformer-based U-shaped encoder-decoder structure, which includes multi-stage downsampling, upsampling and symmetric skip connections, outputs a point-level semantic segmentation probability map, and its output point-level semantic category probability is used to generate the mask vector.

[0055] The semantic prior segmentation network can use any three-dimensional segmentation backbone structure, such as point cloud Transformer, to achieve single-step forward semantic prediction.

[0056] III. Mask Construction Module.

[0057] The point-level semantic segmentation probability map is compressed into a low-dimensional mask channel to obtain mask features.

[0058] The specific steps are as follows: S31, Obtain the point-level semantic segmentation probability map.

[0059] S32 compresses the point-level semantic segmentation probability map into a mask feature vector through a multilayer perceptron.

[0060] S33 uses a nonlinear function to restrict the mask feature vector to a fixed numerical range.

[0061] In this embodiment, the point-level semantic segmentation probability map output by the semantic prior segmentation network is compressed into a low-dimensional mask channel using an MLP, resulting in a mask feature vector with the same number of points but a smaller number of channels. This mask feature vector reflects the class confidence and semantic uncertainty of each point. The mask feature vector is restricted to a fixed numerical range by a sigmoid function and, together with the guiding features and time step embedding, serves as the conditional input for the mask guiding diffusion branch.

[0062] IV. Feature Space Noise Injection Module.

[0063] Based on a preset noise schedule, Gaussian noise is injected into the feature of the point cloud sub-blocks at each time step to obtain noisy features.

[0064] The specific steps are as follows: S41, Pre-set noise schedule

[0065] ; in, Let T be the noise intensity, T be the total time step, and t be the current time step. The signal retention coefficients at the t-th diffusion step are... The cumulative signal retention coefficients from step 1 to step t are... This represents the signal retention coefficient at the s-th time step.

[0066] S42, based on a preset noise schedule, performs feature analysis on point cloud sub-blocks at each time step. Injecting Gaussian noise yields noisy features. .

[0067] In this embodiment, the preset noise follows a standard normal distribution. During the training phase, the same scene can be perturbed at multiple noise levels, allowing the mask-guided diffusion branch to learn robust representations under different noise intensities.

[0068] V. Mask-modulated noise regression network.

[0069] Noisy features, time-step embeddings, and mask features are fused to obtain fused features. The fused features are downsampled to obtain noise regression features, and the noise regression features are upsampled to obtain predicted noise.

[0070] The mask-modulated noise regression network is a Transformer-based U-Net encoder-decoder structure, comprising multi-stage downsampling, upsampling, and symmetrical skip connections. The encoder includes: Mesh feature aggregation module: used to transform disordered point clouds into regular local region features; Random point discarding module: Randomly shuffle the point cloud order and take the first A points as input; Feature encoding module: Transforms point features with positional encoding into a deep representation that encompasses the global context.

[0071] The feature encoding module includes multiple self-attention blocks, and each self-attention block includes an xCPE unit, a mask embedding unit, two residual connection normalization units, and a window attention unit, such as... Figure 2 As shown. Wherein: xCPE unit: Transforms unordered point cloud features into a feature sequence with a clear spatial address and distinguishable location; Mask embedding unit: Forces the encoder to infer global and local structure from partially visible points.

[0072] Two residuals are connected to normalization units: standardized input data.

[0073] Window area attention unit: (1) Divide the feature maps of each scale in the mask modulation noise regression network into several non-overlapping local window regions according to the preset window region side length. Each window region contains a fixed number of point features, and the set of window regions is as follows: .

[0074] (2) Multi-head self-attention computation is performed independently within each window region to obtain the local features WA of the window region attention. Relative position encoding is introduced into the attention scoring, making the attention weights depend simultaneously on feature similarity and local geometric relationships. This highlights high-frequency geometric information such as boundary shapes and slender structures within the window region, while limiting the computational load to a local range. ; in, Let Q be the local feature matrix of the k-th window region, and let Q be the query matrix. For the transpose of the k-th window region, d is the scaling factor, and d is the feature dimension inside the attention head. V is the value matrix, representing the positional encoding corresponding to the k-th window region. For linear transformation, Let be the input feature matrix for the k-th window region.

[0075] Multilayer perceptron: performs nonlinear transformation and fusion on the updated feature matrix.

[0076] The mask-modulated noise regression network also includes a boundary sparse bridging path unit, and all window attention units in the mask-modulated noise regression network are connected to the boundary sparse bridging path unit.

[0077] Boundary Sparse Bridging Path Unit: A sparse adjacency graph is constructed at the window boundary points, and a depthwise separable convolution is applied to this graph to output boundary enhancement features. In this embodiment, the boundary sparse bridging path unit compensates for the lack of cross-window information caused by simple windowing. A sparse adjacency graph is constructed at the window boundary points, considering only local neighbor relationships near the boundary. A depthwise separable convolution is applied to this sparse graph to achieve cross-window feature propagation and information aggregation, thereby introducing the necessary boundary enhancement feature B without significantly increasing computational cost. ; in, Let D be the point feature matrix from the input to the boundary sparse bridging path unit, representing the boundary sparse bridging operator. A graph representing the adjacency relationships of the boundary points of the window region. , For the set of boundary points, This is the set of edges that connect across the window domain boundary.

[0078] This design greatly reduces the complexity of cross-window computation while significantly improving the ability to represent high-frequency local features.

[0079] For each window region, the local features output by the window region attention unit and the boundary enhancement features output by the boundary sparse bridging path unit are concatenated element-wise along the channel dimension and then linearly transformed to obtain the updated noise regression features. These updated features are then fed into subsequent layers of the mask-modulated noise regression network, ultimately outputting the noise. .

[0080] ; in, For the noise regression features of the k-th window region, This is the boundary enhancement feature for the k-th window region.

[0081] VI. Mask Modulation Feature Injector.

[0082] Cross-attention computation is performed on bottleneck features and noise regression features to generate mask modulation features. These mask modulation features are then unidirectionally injected into the semantic prior segmentation network, which performs subsequent upsampling, decoding, and classification output to generate point-level semantic segmentation results.

[0083] In this embodiment, cross-attention computation is performed based on bottleneck features and noise regression features to generate mask modulation features; the mask modulation features are unidirectionally injected into the semantic prior segmentation network, which then performs subsequent upsampling, decoding, and classification output to generate point-level semantic segmentation results.

[0084] The mask modulation feature injector is located at the bottleneck of the semantic prior segmentation network and the mask-guided diffusion branch. It is used to perform learnable filtering of mask modulation features and inject them unidirectionally into the semantic prior segmentation network to avoid excessive perturbation that could damage the semantic discrimination ability of the main path.

[0085] The specific operating steps are as follows: S61, using MLP to identify bottleneck features Perform feature projection to obtain the query matrix Q, and then use MLP to regress features on the noise. Perform feature projection to obtain the key matrix K and the value matrix V.

[0086] ; ; The number of points representing the bottleneck feature. The number of channels before bottleneck feature projection. This represents the number of points in the noise regression feature. C represents the number of channels before the noise regression feature is projected, and C represents the unified channel dimension after the bottleneck feature and the noise regression feature are projected.

[0087] S62 filters bottleneck features through a cross-attention mechanism. and noise regression features The noise information is obtained, and the mask modulation features are calculated based on the query matrix Q, the key matrix K, and the value matrix V; ; ; Where O represents the intermediate fusion feature, H represents the mask modulation feature, and W represents the cross-attention weight matrix. , This is the transpose of the key matrix. This is the scaling factor.

[0088] S63 performs learnable filtering on mask modulation features and injects the mask modulation features selected by the self-mask-guided diffusion branch into the semantic prior segmentation network through unidirectional injection.

[0089] This embodiment employs a dual-path joint optimization strategy to train a sparse point cloud semantic segmentation system based on mask modulation noise regression. The dual-path joint optimization strategy includes the semantic segmentation loss of the semantic prior segmentation network. Mask modulation noise regression loss with mask-guided diffusion branch .

[0090] For semantic prior segmentation networks, a combination of cross-entropy loss and Lovász-Softmax loss is used: ; Where CE(·) represents the cross-entropy loss, These are the original semantic tags. To predict semantic labels, As a weighting factor, =1.

[0091] For the mask-guided diffusion branch, the noise regression loss is calculated using a standard noise regression target: ; in, For standard noise samples, Let I be a multidimensional standard normal distribution with a mean of 0 and a covariance of I, where 0 is the zero vector and I is the identity matrix. For the predicted noise, Let m be the noisy feature corresponding to time step t, where t is the current diffusion step, m is the mask modulation feature vector, and R is the condition set.

[0092] overall : ; The implementation of this method was verified and analyzed through model accuracy.

[0093] To verify the effectiveness of this invention in real-world scenarios, experiments were conducted on three datasets: SensatUrban, Campus3D, and the self-built dataset QuBei. To ensure comparability, all three datasets employed the same preprocessing: first, the original point cloud was voxelized to a size of 0.2m to alleviate density unevenness and memory overhead; then, the coordinates and colors were decentered and normalized, respectively. The same sampling and blockization strategies were maintained during the training and inference phases, and the official training, validation, and testing partitions of each dataset were strictly followed.

[0094] The datasets SensatUrban, Campus3D, and QuBei will be described in detail below: SensatUrban: This dataset is a large-scale urban point cloud collection, generated using fixed-wing drones, containing approximately 3 billion points covering an area of ​​7.64 square kilometers, encompassing parts of three UK cities: Birmingham, Cambridge, and York. Each point in this dataset includes coordinates in a unified projected coordinate system and RGB color information. The semantic annotations are categorized into 13 classes, including: ground, vegetation, buildings, cars, railways, bicycles, etc. All experiments in this paper follow the official classification to ensure comparability.

[0095] Campus3D: This dataset, reconstructed by the National University of Singapore using UAV imagery through photogrammetry, covers a 1.58 square kilometer outdoor campus scene. It provides a high-density point cloud of 937 million points, along with hierarchical semantic and instance annotations at five levels of granularity. It is divided into FASS, FOE, PGP, RA, UCC, and YIH categories based on function and architectural style. This paper follows the C2 hierarchy and adheres to the official classification to ensure comparability.

[0096] QuBei: This dataset was collected from a mountainous region in a province of China. The terrain is predominantly mountainous with some farmland and village clusters. The data features highly undulating terrain and includes extremely thin power lines and tall, sparse power towers. Each point contains coordinates and RGB color information in a unified projected coordinate system, and is semantically labeled into five categories: ground, vegetation, buildings, power lines, and power towers. Power lines and power towers are extremely rare, accounting for only 0.16% and 0.1% of the total points, respectively. The number and percentage of each category are shown in Table 1. The dataset is divided into 55 point cloud blocks, with 40 blocks used as the training set, 8 as the validation set, and 7 as the test set. There are 15 power towers in total, with 9 used as the training set, 3 as the validation set, and 3 as the test set.

[0097] Table 1: Percentage of each category in the QuBei dataset.

[0098]

[0099] Experimental Details: All experiments were conducted on the PyTorch platform, with training and testing performed on a single NVIDIA 3090 GPU. The AdamW optimizer was used, with a weight decay of 0.005, a learning rate of 0.001, and a batch size of 2.

[0100] Evaluation metrics: To quantitatively analyze the performance of the proposed method, the Intersection over Union (IoU), mean Intersection over Union (mIoU), and overall accuracy (OA) were used as evaluation metrics.

[0101] in, ; ; ; Where TP, FP, TN, and FN represent true positives, false positives, true negatives, and false negatives, respectively, and n represents the number of classes in the dataset.

[0102] On SensatUrban, our method shows a stable improvement in mIoU compared to CDSegNet, which only considers time steps. Class-wise comparisons show significant improvements in smaller classes like Rail and FootPath, with a 3.5% improvement in Rail and a 2.1% improvement in FootPath. Improvements are also seen in Ground, Parking, and Traffic Road classes. Visual comparisons on the SensatUrban dataset are shown below. Figure 3 As shown in Tables 2 and 3, the quantitative comparison results are as follows: Ground, Vegetation, Building, Wall, Bridge, Parking, Rail, Car, Foot Path, Bike, Water, Traffic, and Street.

[0103] Table 2: Experimental results of the Sensaturban dataset (Part 1).

[0104]

[0105] Table 3: Experimental results of the Sensaturban dataset (Part 2).

[0106]

[0107] PointNet is a network that directly processes unordered point sets. It extracts features from each point independently by sharing an MLP and then uses a symmetric aggregation function to aggregate global information, thereby achieving modeling of the invariance of point set permutation. It can be used for classification, component segmentation, and semantic segmentation.

[0108] PointNet++: Based on PointNet, it introduces hierarchical sampling, grouping, and local feature aggregation, and recursively learns multi-scale local context in the metric space, thereby enhancing the ability to model fine-grained geometric structures and non-uniformly sampled point clouds.

[0109] RandLA-Net: A lightweight network for semantic segmentation of large-scale point clouds. It uses random sampling to reduce computation and storage overhead, and gradually expands the receptive field through a local feature aggregation module to balance high efficiency and geometric detail preservation.

[0110] KPConv: A convolution operator that operates directly on point clouds. It parameterizes convolution weights through kernel points in Euclidean space and performs feature aggregation in the neighborhood of a point. Its deformable version can also learn the kernel point positions to better adapt to local geometry.

[0111] MVP-Net: A multi-scale voxel-point adaptive fusion network that jointly models geometric and semantic information under different receptive fields through a multi-scale voxel fusion module and a geometric self-attention module, and is used for semantic segmentation of point clouds in urban scenes.

[0112] LACV-Net: A network for semantic segmentation of large-scale point clouds. It models local context through a local adaptive feature enhancement module and uses a combination of VLAD / global description vectors to fuse multi-layer, multi-scale, and multi-resolution features to take into account both local details and global semantics.

[0113] EyeNet: A point cloud semantic segmentation network inspired by human peripheral vision mechanisms. It expands the effective coverage through multi-contour input and parallel branching structure, and fuses contextual information at different scales by using inter-branch connecting blocks.

[0114] SADNet: A Space-aware DeepLab network for city-level point cloud semantic segmentation, designed to alleviate the problems of information loss caused by sampling and insufficient perception of spatial relationships between points, thereby improving segmentation performance in complex urban scenes.

[0115] PTv3: Point Transformer V3 expands the receptive field through serialized point cloud representation and efficient serialized neighborhood mapping / attention mechanism, while maintaining strong 3D understanding and segmentation capabilities, improving speed and memory efficiency.

[0116] CDSegNet: A robust point cloud semantic segmentation network based on a conditional-noise framework. The conditional network is used for semantic prediction, and the noise network is used as a learnable noise feature generator. Multi-level feature perturbation is performed during training, and single-step output is maintained during inference.

[0117] For the Campus3D dataset, this paper conducts experiments according to the C2 hierarchical partitioning, which divides the data into three categories: natural objects, man-made objects, and buildings. Compared to the SensatUrban dataset, the Campus3D dataset has lower and sparser point density, poorer point cloud quality, more incomplete data, and more noise. Compared to other methods such as PointNet++, our proposed method performs best, especially compared to the non-diffusion method of Point TransformerV3, where our method outperforms by 3 percentage points in mIou, further demonstrating that the proposed diffusion method is more adaptable to sparse point clouds and point clouds with more noise. Visual comparisons on the Campus3D dataset are shown below. Figure 4 As shown in Table 4, the quantitative comparison results are as follows: Natural, Man-Made, and Construction.

[0118] Table 4: Test results on Campus3D It means no.

[0119]

[0120] Among them, DGCNN (Dynamic Graph Convolutional Neural Network) dynamically constructs a local graph through K-nearest neighbors, and applies edge convolution to aggregate neighborhood features on the graph structure to achieve efficient point cloud classification and segmentation.

[0121] For the QuBei dataset, our proposed method achieves an mIoU of 94.7% and an OA of 97.2%, significantly outperforming methods such as PTV3 and CDSegNet. By class, our method shows significant improvement in the few-sample category of Building, and also demonstrates a marked improvement in the small vegetation on drainage ditches between farmlands, with a 2.3% improvement compared to CDSegNet for the vegetation category. For the power line and power tower categories with extremely limited sample sizes, our method shows significant improvement, exhibiting good segmentation ability. Visualization results and accuracy of different methods on the QuBei dataset are shown below. Figure 5 As shown in Table 5, the components are: Ground, Vegetation, Building, Power Lines, and Power Towers.

[0122] Table 5: Experimental results on the QuBei validation set.

[0123]

[0124] In summary, this invention achieves leading mIoU on three datasets with significant differences in scale and scenario, with particularly significant advantages in elongated, small-scale, and few-sample categories, while maintaining a single-step inference path, meeting the real-time and deployability requirements of engineering projects.

[0125] Example 2

[0126] This embodiment 2 proposes a sparse point cloud semantic segmentation method based on mask-modulated noise regression. Please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is a flowchart of the point cloud semantic segmentation method in Example 2. The operation steps of the point cloud semantic segmentation method are as follows: S1, Collect dense point cloud, preprocess the dense point cloud to obtain multiple point cloud sub-blocks; S2, set the semantic prior segmentation network as the main path, extract features from the point cloud sub-blocks, obtain the point cloud sub-block features, downsample the point cloud sub-block features to obtain the bottleneck features, upsample the bottleneck features to obtain the semantic probability labels, and form a point-level semantic segmentation probability map based on the semantic probability labels. S3, compress the point-level semantic segmentation probability map into a low-dimensional mask channel to obtain mask features; S4. Based on a preset noise schedule, Gaussian noise is injected into the feature of the point cloud sub-block at each time step to obtain noisy features. S5, set the mask-guided diffusion branch as the training auxiliary path, fuse the noisy features, time step embedding and mask features to obtain the fused features, downsample the fused features to obtain the noise regression features, upsample the noise regression features to obtain the predicted noise; S6. A mask modulation feature injector is set in the bottleneck layer of the semantic prior segmentation network and the mask-guided diffusion branch to perform cross-attention calculation on the bottleneck features and noise regression features. The calculated mask modulation features are unidirectionally injected into the semantic prior segmentation network, fused with the bottleneck features, and output semantic probability labels. S7, train the point cloud semantic segmentation model by minimizing the noise prediction loss and semantic segmentation loss; The trained point cloud semantic segmentation model is used to perform semantic segmentation on point clouds and generate semantic probability labels.

[0127] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A sparse point cloud semantic segmentation method based on mask-modulated noise regression, characterized in that, The point cloud semantic segmentation model is trained based on mask-modulated noise regression, and the training method is as follows: S1, Collect dense point cloud, preprocess the dense point cloud to obtain multiple point cloud sub-blocks; S2, set the semantic prior segmentation network as the main path, extract features from the point cloud sub-blocks, obtain the point cloud sub-block features, downsample the point cloud sub-block features to obtain the bottleneck features, upsample the bottleneck features to obtain the semantic probability labels, and form a point-level semantic segmentation probability map based on the semantic probability labels. S3, compress the point-level semantic segmentation probability map into a low-dimensional mask channel to obtain mask features; S4. Based on a preset noise schedule, Gaussian noise is injected into the feature of the point cloud sub-block at each time step to obtain noisy features. S5, set the mask-guided diffusion branch as the training auxiliary path, fuse the noisy features, time step embedding and mask features to obtain the fused features, downsample the fused features to obtain the noise regression features, upsample the noise regression features to obtain the predicted noise; S6. A mask modulation feature injector is set in the bottleneck layer of the semantic prior segmentation network and the mask-guided diffusion branch to perform cross-attention calculation on the bottleneck features and noise regression features. The calculated mask modulation features are unidirectionally injected into the semantic prior segmentation network, fused with the bottleneck features, and output semantic probability labels. S7, train the point cloud semantic segmentation model by minimizing the noise prediction loss and semantic segmentation loss; The trained point cloud semantic segmentation model is used to perform semantic segmentation on point clouds and generate semantic probability labels.

2. The sparse point cloud semantic segmentation method based on mask modulation noise regression according to claim 1, characterized in that, The trained point cloud semantic segmentation model performs the following steps for semantic segmentation of point clouds: Step 1: Collect real point cloud data of the target area, preprocess and segment the real point cloud data to obtain multiple point cloud sub-blocks; Step 2: Extract features from the point cloud sub-blocks to obtain point cloud sub-block features, downsample the point cloud sub-block features to obtain bottleneck features, and upsample the bottleneck features. Obtain semantic probability labels and form a point-level semantic segmentation probability map based on the semantic probability labels.

3. The sparse point cloud semantic segmentation method based on mask modulation noise regression according to claim 1, characterized in that, The specific process of step S1 is as follows: S11, collect dense point clouds of the target area; S12, the dense point cloud is voxelized and sampled according to a fixed voxel size to control the local point density and remove obvious outliers or noise points; the coordinates and colors of the point cloud are decentered and normalized respectively to obtain the preprocessed point cloud. S13, the preprocessed point cloud is divided into overlapping sub-blocks according to a fixed spatial size to obtain multiple point cloud sub-blocks.

4. The sparse point cloud semantic segmentation method based on mask modulation noise regression according to claim 1, characterized in that, The specific process of step S3 is as follows: S31, Obtain the point-level semantic segmentation probability map; S32 compresses the point-level semantic segmentation probability map into a mask feature vector through a multilayer perceptron. S33 uses a nonlinear function to restrict the mask feature vector to a fixed numerical range to obtain the mask features.

5. The sparse point cloud semantic segmentation method based on mask modulation noise regression according to claim 1, characterized in that, The specific process of step S4 is as follows: S41, Preset Noise Schedule ； in, Let T be the noise intensity, T be the total time step, and t be the current time step. The signal retention coefficients at the t-th diffusion step are... The cumulative signal retention coefficients from step 1 to step t. This represents the signal retention coefficient at the s-th time step; S42, based on a preset noise schedule, performs feature analysis on point cloud sub-blocks at each time step. Injecting Gaussian noise yields noisy features. .

6. The sparse point cloud semantic segmentation method based on mask modulation noise regression according to claim 1, characterized in that, The specific process of step S6 is as follows: S61, perform feature projection on the bottleneck features through MLP to obtain the query matrix, and perform feature projection on the noisy regression features through MLP to obtain the key matrix and value matrix; S62 filters noise information from bottleneck features and noise regression features through a cross-attention mechanism, and calculates mask modulation features based on the query matrix, key matrix, and value matrix. S63 performs learnable filtering on mask modulation features and injects the mask modulation features selected by the self-mask-guided diffusion branch into the semantic prior segmentation network through unidirectional injection.

7. A sparse point cloud semantic segmentation system based on mask-modulated noise regression, characterized in that, A sparse point cloud semantic segmentation method based on mask-modulated noise regression, applicable to any one of claims 1-6, wherein the system comprises: a point cloud semantic segmentation model and a mask-guided diffusion branch, as detailed below: Point cloud semantic segmentation model: Point cloud processing module: Collects dense point clouds of the target area, preprocesses and segments the dense point clouds to obtain multiple point cloud sub-blocks; Semantic prior segmentation network: Extract features from point cloud sub-blocks to obtain point cloud sub-block features, downsample point cloud sub-block features to obtain bottleneck features, upsample bottleneck features to obtain semantic probability labels, and form a point-level semantic segmentation probability map based on semantic probability labels. Mask-guided diffusion branching: Mask construction module: compresses the point-level semantic segmentation probability map into a low-dimensional mask channel to obtain mask features; Feature space noise injection module: Based on a preset noise schedule, Gaussian noise is injected into the feature of the point cloud sub-block at each time step to obtain noisy features; Mask-modulated noise regression network: Noisy features, time step embeddings and mask features are fused to obtain fused features. The fused features are downsampled to obtain noise regression features. The noise regression features are upsampled to obtain predicted noise. Mask Modulation Feature Injector: Performs cross-attention calculation on bottleneck features and noise regression features to generate mask modulation features, and injects the mask modulation features unidirectionally into the semantic prior segmentation network. The semantic prior segmentation network then completes subsequent upsampling, decoding, and classification output to generate semantic probability labels.

8. A sparse point cloud semantic segmentation system based on mask modulation noise regression according to claim 7, characterized in that, The mask-modulated noise regression network is a Transformer-based U-Net encoder-decoder structure; The encoder includes: Mesh feature aggregation module: transforms disordered point clouds into regular local region features; Random point discarding module: randomly shuffles the point cloud order; Feature encoding module: Converts point cloud features with location encoding into noisy regression features; Furthermore, the feature encoding module includes multiple self-attention blocks and a boundary sparse bridging path unit.

9. A sparse point cloud semantic segmentation system based on mask modulation noise regression according to claim 8, characterized in that, The self-attention block includes: xCPE unit: Transforms unordered point cloud features into a feature sequence with a clear spatial address and distinguishable location; Mask embedding unit: forces the encoder to infer global and local structure from partially visible points; Two residual connection normalization units: standardize the input data; Window area attention unit: (1) Divide the feature maps of each scale in the mask modulation noise regression network into several non-overlapping local window regions according to the preset window region side length; (2) Perform multi-head self-attention computation independently within each window region to obtain the local features of window region attention; Boundary Sparse Bridging Path Unit: Constructs a sparse adjacency graph at the window domain boundary points, and applies a depthwise separable convolution to the sparse adjacency graph to output boundary enhancement features.

10. A computer program product, characterized in that, It includes a computer program / instruction that, when executed by a processor, implements a sparse point cloud semantic segmentation method based on mask modulation noise regression as described in any one of claims 1-6.

Images