Point cloud semantic segmentation method and system suitable for multi-module splicing scene, medium and electronic equipment
By using spherical projection and distance-weighted aggregation, the problems of overlapping artifacts and missed detection of small targets in point cloud segmentation in multi-module stitching scenarios are solved, achieving high-precision and efficient point cloud semantic segmentation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUNNIWELL AIOT TECH LTD
- Filing Date
- 2026-02-04
- Publication Date
- 2026-05-19
AI Technical Summary
In multi-module stitching scenarios, the overlapping artifacts of point cloud data, the loss of local geometric information in projection, and the imbalance of small target categories pose challenges to the accuracy and real-time performance of existing point cloud segmentation methods. In particular, in scenarios such as security monitoring, segmentation edges are prone to blurring and missed detections.
A two-dimensional feature grid is constructed using spherical projection, and feature mean fusion and semantic soft voting are performed. Local geometric features are extracted by combining a distance-weighted aggregation module, and end-to-end training is carried out through dual-stream feature fusion and a hybrid loss function to improve the segmentation effect.
It effectively solves the problems of overlapping artifacts and missed detection of small targets in multi-module stitching scenarios, improves segmentation accuracy and robustness, and maintains high computational performance.
Smart Images
Figure CN122066947A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of electrical data processing technology, specifically relating to point cloud semantic segmentation methods, systems, media, and electronic devices suitable for multi-module splicing scenarios. Background Technology
[0002] With the development of 3D perception technology, LiDAR and RGB-D cameras are widely used in fields such as autonomous driving, security monitoring, and human-computer interaction. Point clouds, as the main form of 3D spatial data, are characterized by sparsity, disorder, and unstructured nature. To expand the perception range, modern perception systems often use multiple sensor modules to stitch together the field of view (FOV) to obtain a wider coverage area.
[0003] After acquiring point cloud data, semantic segmentation is typically required to analyze subsequent behavior and trajectory tracking, determining whether each point belongs to a "human body" or "background." Currently, mainstream point cloud segmentation methods fall into two main categories: one directly processes 3D point clouds (such as PointNet++), which, while highly accurate, suffers from quadratic computational complexity with the number of points, struggles with large-scale scenes, and has poor real-time performance; the other is a projection-based method, projecting the 3D point cloud into a 2D image and then processing it using a mature convolutional neural network (CNN). This method boasts extremely high computational efficiency and is suitable for engineering applications.
[0004] Despite the efficiency advantages of projection-based methods, they face at least the following significant challenges in practical multi-module splicing scenarios:
[0005] Multi-module stitching leads to "overlap artifacts"; due to mechanical installation errors, sensor time synchronization errors and ranging noise, point clouds from different modules often cannot be perfectly aligned in the overlapping area of the field of view, resulting in "double skin" or misaligned interlacing phenomena on the same physical surface. Directly projecting such data into the network will introduce a large amount of geometric noise, resulting in blurred or even incorrect segmentation edges.
[0006] Projection loses local geometric information; when a 3D point cloud is forcibly projected onto a 2D mesh (such as a spherical projection), points that are far apart in physical space (such as the edges of objects that are occluded in front or behind) may be mapped to adjacent 2D pixels. Conventional 2D convolution cannot distinguish such "pseudo-neighborhoods", leading to confusion in feature extraction.
[0007] Small target category imbalance; in scenarios such as security monitoring, human point clouds usually only account for a very small proportion of the entire field of view. Conventional cross-entropy loss functions are easily dominated by a large number of background samples, causing the model to tend to predict human bodies as background, resulting in missed detections or low recall. Summary of the Invention
[0008] In view of this, on the one hand, some embodiments disclose a point cloud semantic segmentation method applicable to multi-module stitching scenarios, including the following steps:
[0009] S1. Perform spherical projection on the original point cloud obtained by multi-sensor stitching to construct a two-dimensional feature grid, and perform feature mean fusion and semantic soft voting on multi-source points falling into the same grid.
[0010] S2. On the two-dimensional feature grid, a distance-weighted aggregation module is constructed based on depth differences to extract local geometric features with three-dimensional geometric perception capabilities;
[0011] S3. The absolute coordinate features of the original point cloud are used as the global flow, and the extracted local geometric features are used as the local flow. The two-flow feature fusion is performed and input into the encoder-decoder network for multi-scale feature extraction and segmentation prediction.
[0012] S4. The network is trained end-to-end using a hybrid loss function that includes Focal Loss and Dice Loss, and the semantic segmentation results are output.
[0013] Furthermore, in some embodiments of the point cloud semantic segmentation method suitable for multi-module stitching scenarios, step S1, constructing a two-dimensional feature mesh, includes:
[0014] First, determine the resolution as H×W;
[0015] The horizontal field of view is divided into H parts and the vertical field of view is divided into W parts, resulting in H×W minimum windows; each minimum window constitutes a grid, and the point cloud is assigned to each grid.
[0016] Some embodiments disclose point cloud semantic segmentation methods applicable to multi-module stitching scenarios. In step S2, the weight calculation method for distance-weighted aggregation is as follows:
[0017] ;
[0018] Where Δd represents the depth difference between the neighboring points and the center point.
[0019] Some embodiments disclose point cloud semantic segmentation methods suitable for multi-module stitching scenarios. In step S3, an SE-Block channel attention mechanism is introduced in the skip connection between the encoder and the decoder.
[0020] Some embodiments disclose a point cloud semantic segmentation method suitable for multi-module splicing scenarios. In step S3, the encoder-decoder network includes a four-layer downsampling encoder and a four-layer upsampling decoder.
[0021] Some embodiments disclose a point cloud semantic segmentation method suitable for multi-module stitching scenarios. In step S4, the expression for the hybrid loss function is as follows:
[0022] ;
[0023] Where λ1 = 1.0, λ2 = 0.75;
[0024] ;
[0025] In the formula P t γ represents the model's predicted probability of the true class, and γ is the focusing parameter;
[0026] ;
[0027] Where, p i and g i ε = 1e-5 represents the values of the predicted probability map and the true label map at the i-th pixel, respectively, and ε = 1e-5 is the smoothing term.
[0028] On the other hand, some embodiments disclose point cloud semantic segmentation systems suitable for multi-module stitching scenarios, used to execute the point cloud semantic segmentation methods disclosed in the embodiments of the present invention. The system includes:
[0029] The preprocessing module is configured to perform spherical projection on the raw point cloud obtained by multi-sensor stitching, construct a two-dimensional feature grid, and perform feature mean fusion and semantic soft voting on multi-source points falling into the same grid;
[0030] The feature extraction and fusion module is configured to construct a distance-weighted aggregation module based on depth differences on a two-dimensional feature grid to extract local geometric features with three-dimensional geometric perception capabilities; the absolute coordinate features of the original point cloud are used as the global stream, and the extracted local geometric features are used as the local stream to perform dual-stream feature fusion, and then input into an encoder-decoder network for multi-scale feature extraction and segmentation prediction;
[0031] The training and inference module is configured to perform end-to-end training of the network using a hybrid loss function that includes Focal Loss and Dice Loss, and output semantic segmentation results.
[0032] Furthermore, some embodiments of the point cloud semantic segmentation system suitable for multi-module stitching scenarios also include:
[0033] A sensor module is used to acquire point cloud data from multiple perspectives; the sensor module includes at least two lidar or RGB-D cameras whose fields of view overlap.
[0034] In another aspect, some embodiments disclose a computer-readable storage medium storing a computer program that, when executed by a processor, implements the point cloud semantic segmentation method for multi-module splicing scenarios disclosed in the embodiments of the present invention.
[0035] On another front, some embodiments disclose an electronic device including a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program, it implements the point cloud semantic segmentation method for multi-module splicing scenarios disclosed in the embodiments of the present invention.
[0036] This invention discloses a point cloud semantic segmentation method applicable to multi-module stitching scenarios. It proposes a mean feature smoothing and label soft voting method based on spherical projection meshing, effectively solving the problems of "overlap artifacts" and "double skin" caused by registration errors and installation deviations in the stitching region of multi-sensor fields of view. This achieves the conversion of disordered point clouds into high signal-to-noise ratio regular tensors. A local feature aggregation module based on linear decay of mesh distance is designed, effectively solving the problems of "pseudo-neighborhood interference" and "geometric feature distortion" caused by spatial dimensionality reduction in the 2D projection process. While maintaining the computational efficiency of 2D convolution, it restores accurate perception of 3D local topology. A dual-stream coding architecture integrating absolute coordinates and relative geometry, along with a hybrid loss strategy, is constructed, effectively solving the problems of "feature submersion" and "high false negative rate" caused by the low foreground proportion of sparse human targets in large scenes. This significantly improves the recall and robustness of the algorithm in scenarios with extremely imbalanced categories. Attached Figure Description
[0037] Figure 1 , one Flowcharts of point cloud semantic segmentation methods disclosed in some embodiments;
[0038] Figure 2 , one Schematic diagram of the point cloud semantic segmentation system disclosed in these embodiments;
[0039] Figure 3 , one Some embodiments disclose real-time acquired point cloud data maps;
[0040] Figure 4 , one Some embodiments disclose schematic diagrams of the direct stitching effect of real-time acquired point cloud data;
[0041] Figure 5 The probability prediction map is obtained by processing the point cloud data collected in real time according to the methods disclosed in some embodiments of the present invention. Detailed Implementation
[0042] The term "embodiment" used herein, as an example, is not necessarily to be construed as superior to or better than other embodiments. Performance testing in these embodiments of the invention, unless otherwise specified, employs conventional testing methods in the art. It should be understood that the terminology used in these embodiments is merely for describing particular implementations and is not intended to limit the scope of the disclosure of these embodiments.
[0043] Unless otherwise stated, the technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this invention pertain; other experimental methods and technical means not specifically noted in the embodiments of this invention refer to experimental methods and technical means commonly used by one of ordinary skill in the art.
[0044] The terms “basic” and “approximately” as used herein are used to describe small fluctuations. For example, they can mean less than or equal to ±5%, such as less than or equal to ±2%, such as less than or equal to ±1%, such as less than or equal to ±0.5%, such as less than or equal to ±0.2%, such as less than or equal to ±0.1%, such as less than or equal to ±0.05%. Numerical data presented or expressed in range format herein are used for convenience and brevity only, and should therefore be interpreted flexibly to include not only the explicitly listed values that define the range, but also all independent values or subranges contained within that range. For example, a numerical range of “1–5%” should be interpreted to include not only the explicitly listed values from 1% to 5%, but also the independent values and subranges within the indicated range. Thus, this numerical range includes independent values such as 2%, 3.5%, and 4%, and subranges such as 1%–3%, 2%–4%, and 3%–5%, etc. This principle also applies to ranges that list only one value. Furthermore, this interpretation applies regardless of the width of the range or the characteristics described.
[0045] In this document, including in the claims, conjunctions such as "comprising," "including," "with," "having," "containing," "involving," and "accommodating" are understood to be open-ended, meaning "including but not limited to." Only the conjunctions "consisting of" and "composed of" are closed conjunctions.
[0046] To better illustrate the content of this invention, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that the invention can be practiced even without certain specific details. In the embodiments, some methods, means, instruments, and devices well-known to those skilled in the art are not described in detail, in order to highlight the main points of the invention.
[0047] Without conflict, the technical features disclosed in the embodiments of the present invention can be combined arbitrarily, and the resulting technical solution belongs to the content disclosed in the embodiments of the present invention.
[0048] In some implementations, the steps of a point cloud semantic segmentation method applicable to multi-module stitching scenarios are as follows:
[0049] S1. Perform spherical projection on the original point cloud obtained by multi-sensor stitching to construct a two-dimensional feature grid, and perform feature mean fusion and semantic soft voting on multi-source points falling into the same grid.
[0050] Typically, constructing a two-dimensional feature mesh involves: first determining the resolution as H×W; dividing the horizontal field of view into H parts to obtain H×W minimum windows; each minimum window constitutes a mesh, and the point cloud is assigned to each mesh;
[0051] Typically, the specific method of semantic soft voting is as follows: for points falling into the same grid, if their semantic label is foreground, add +1; if it is background, add -1. The semantic category of the grid is determined based on the sign of the accumulated result.
[0052] S2. On the two-dimensional feature mesh, a distance-weighted aggregation module is constructed based on depth differences to extract local geometric features with three-dimensional geometric perception capabilities; typically, the weights for distance-weighted aggregation are calculated as follows:
[0053] ;
[0054] Where Δd represents the depth difference between the neighboring points and the center point;
[0055] S3. The absolute coordinate features of the original point cloud are used as the global stream, and the extracted local geometric features are used as the local stream. Dual-stream feature fusion is performed, and the input is fed into the encoder-decoder network for multi-scale feature extraction and segmentation prediction. In some embodiments, the encoder-decoder network includes a four-layer downsampling encoder and a four-layer upsampling decoder. An SE-Block channel attention mechanism is introduced in the skip connection between the encoder and the decoder.
[0056] S4. The network is trained end-to-end using a hybrid loss function that includes Focal Loss and Dice Loss, and the semantic segmentation results are output.
[0057] In some embodiments, the expression for the hybrid loss function is:
[0058] ;
[0059] Where λ1 = 1.0, λ2 = 0.75;
[0060] ;
[0061] In the formula P t γ represents the model's predicted probability of the true class, and γ is the focusing parameter;
[0062] ;
[0063] Where, p i and g i ε = 1e-5 represents the values of the predicted probability map and the true label map at the i-th pixel, respectively, and ε = 1e-5 is the smoothing term.
[0064] Some embodiments disclose point cloud semantic segmentation systems suitable for multi-module stitching scenarios, such as Figure 2 As shown, the point cloud semantic segmentation method disclosed in the embodiments of the present invention includes:
[0065] A sensor module is used to acquire point cloud data from multiple perspectives; the sensor module includes at least two lidar or RGB-D cameras whose fields of view overlap.
[0066] The preprocessing module is configured to perform spherical projection on the raw point cloud obtained by multi-sensor stitching, construct a two-dimensional feature grid, and perform feature mean fusion and semantic soft voting on multi-source points falling into the same grid;
[0067] The feature extraction and fusion module is configured to construct a distance-weighted aggregation module based on depth differences on a two-dimensional feature grid to extract local geometric features with three-dimensional geometric perception capabilities; the absolute coordinate features of the original point cloud are used as the global stream, and the extracted local geometric features are used as the local stream to perform dual-stream feature fusion, and then input into an encoder-decoder network for multi-scale feature extraction and segmentation prediction;
[0068] The training and inference module is configured to perform end-to-end training of the network using a hybrid loss function that includes Focal Loss and Dice Loss, and output semantic segmentation results.
[0069] Some embodiments disclose a computer-readable storage medium storing a computer program that, when executed by a processor, implements the point cloud semantic segmentation method for multi-module splicing scenarios disclosed in the embodiments of the present invention.
[0070] Some embodiments disclose an electronic device, including a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program, it implements the point cloud semantic segmentation method for multi-module splicing scenarios disclosed in the embodiments of the present invention.
[0071] In some implementations, the point cloud semantic segmentation method applicable to multi-module stitching scenarios includes the following steps:
[0072] S1: Data preprocessing and overlapping area fusion;
[0073] Typically, the first step is to perform a coordinate transformation, converting the original Cartesian coordinate point cloud (x, y, z) into a spherical coordinate system with a horizontal azimuth angle θ. xy and vertical azimuth θ xz Then, based on the sensor's field of view, a two-dimensional feature grid of H×W is constructed; finally, during the filling of the two-dimensional feature grid, the overlapping areas are fused using mean fusion.
[0074] For multiple points falling within the same grid cell, since they may come from overlapping data from different modules, the arithmetic mean of their geometric attributes (x, y, z, h) is calculated as the feature of the grid cell to smooth out splicing noise; the semantic labels of the points falling within the same grid are subject to soft label voting, and the semantic category of the grid is determined based on the final symbol to eliminate edge label jitter.
[0075] S2, Local geometric feature extraction;
[0076] Typically, the process begins with neighborhood construction. On the generated H×W feature map, an Unfolded K×K sliding window is used to construct a local neighborhood for each pixel. Next, relative feature calculation is performed, calculating the 3D relative coordinates of each point within the neighborhood relative to the center point. Finally, distance-weighted aggregation is performed to construct a depth-based linear decay mask.
[0077] ;
[0078] The neighborhood features are weighted; typically, the weights decrease linearly with the depth difference, thereby filtering out interference points that are physically far apart but adjacent to each other, such as the edges of the background wall and the foreground human body.
[0079] S3. Dual-stream network architecture and feature fusion: This includes constructing a dual-stream coding architecture that fuses absolute coordinates and relative geometric features. The "global stream" inputs the original normalized absolute coordinates (x, y, z) to preserve the spatial position information of the human body in the scene, while the "local stream" inputs the weighted relative geometric features extracted in step S2 to provide translation-invariant morphological details. The two features are fused along the channel dimension and then input into a U-Net-like backbone network for multi-scale feature extraction. Typically, an SE-Block channel attention mechanism is introduced at skip connections during this process to adaptively select key feature channels for human body segmentation and suppress background noise.
[0080] S4. Hybrid loss optimization;
[0081] Typically, the combined loss function L is used for end-to-end joint training of the model.
[0082] The combined loss function L is expressed as:
[0083] ;
[0084] This loss function combines the advantages of Focal Loss and Dice Loss. On the one hand, Focal Loss dynamically adjusts sample weights to reduce the influence of a large number of easily classified background samples, thereby focusing on difficult-to-classify samples such as human body edges. On the other hand, Dice Loss is introduced to directly optimize the geometric overlap between the predicted region and the real label, solving the class imbalance problem caused by the sparse proportion of human targets in the scene from a global perspective. The two complement each other to significantly improve segmentation accuracy.
[0085] The technical details are further illustrated below with reference to the embodiments.
[0086] In some embodiments, the core of data processing and overlapping region fusion lies in transforming the original point cloud into a regular feature tensor and eliminating stitching artifacts, specifically including:
[0087] The raw point cloud data, stitched together from the left and right sensor modules, is acquired. Then, invalid and outlier points are removed, and the geometric properties of each valid point are calculated, including the Euclidean distance d and the horizontal azimuth angle θ. xy Vertical azimuth θ xz And the height h relative to the ground;
[0088] Where the Euclidean distance is d:
[0089] ;
[0090] Horizontal azimuth θ xy for:
[0091] ;
[0092] The vertical azimuth angle θxz is:
[0093] ;
[0094] The relative height h to the ground is:
[0095] .
[0096] Invalid points are generally defined as points whose distance to the origin is less than 0.2 meters; outliers, also called fly points, are calculated as follows:
[0097] Calculate the xyz mean and standard deviation of all points in the entire frame. If a point deviates from the mean in any dimension by more than the standard deviation multiplied by a factor, the xyz coordinates of that point are set to 0.
[0098] Subsequently, a spherical projection meshing operation is performed to construct a two-dimensional feature mesh. The resolution is set to H×W, and all points are mapped to the corresponding two-dimensional mesh coordinates (u, v) based on the horizontal field of view (HFOV) and vertical field of view (VFOV). Typically, the specific process includes: first, determining the resolution as H×W; dividing the horizontal field of view into H parts and the vertical field of view into W parts, resulting in H×W minimum windows; each minimum window constitutes a mesh, and then the point cloud is assigned to each of these meshes; finally, the number of points falling into each mesh and their corresponding labels are counted. Constructing a two-dimensional feature mesh using the field of view effectively avoids the problem of stitching together point clouds from multiple modules.
[0099] Typically, for conflict points arising from overlapping regions of multiple modules—that is, when multiple points are mapped to the same grid—a mean fusion strategy is employed to calculate the mean f of each feature of all points falling within that grid. ave As the final feature, f ave Represented as:
[0100] ;
[0101] Among them, f ave It refers to the mean of each feature, such as: x, y, z, h, etc.
[0102] The mode of semantic labels is counted, i.e., semantic soft voting, to determine the category of the grid. This results in a feature tensor of dimensions [H, W, C] as input to the neural network; where C represents the number of channels, the same as the number of features. For example, if the final selected features are "x, y, z, h", then C in the feature tensor would equal 4.
[0103] In some embodiments, the dual-stream network architecture and feature fusion stage mainly include local feature extraction and dual-stream backbone network design;
[0104] First, the local feature extraction module receives [B, C] in The coordinate feature map of [H,W] is used to expand the local neighborhood block by performing an Unfold operation using a K×K convolution kernel, and the depth difference between the center point and the neighborhood points is calculated to generate a linear decay weight w;
[0105] ;
[0106] Where w is the linear decay weight; Δd is the depth difference between the neighboring point and the center point;
[0107] Geometrically perceptive local features are extracted through weighted aggregation and MLP mapping;
[0108] Subsequently, the original coordinate features are used as a global stream, and the extracted local geometric features are used as a local stream. After concatenation along the channel dimension, they are input into the U-Net-like backbone network. The backbone network consists of a four-layer downsampling encoder and a four-layer upsampling decoder. In particular, an SE-Block channel attention module is connected in series on the skip connection path from encoder to decoder to adaptively select key feature channels and suppress background noise, thereby outputting high-precision segmentation results.
[0109] Typically, local feature extraction involves calculating the local features at each point's location. The calculation method is as follows: Given [B, C...]... in The coordinate feature map of [H,W] is used to expand the local neighborhood block using an Unfold operation with a K×K convolution kernel. On the original point cloud matrix, all points in the K×K region centered at a given point are taken as its neighbors. Local features are then calculated using these neighbors. The specific calculation method is as follows:
[0110] The neighboring points are extracted and unfolded using PyTorch's unfold method. Typically, unfold is the core operation in PyTorch used to extract sliding local blocks, essentially implementing the "sliding window" mechanism in convolution operations.
[0111] Calculate the relative coordinates of these neighboring points to the center point, i.e., |△d|;
[0112] Calculate the linear decay weights: w = ReLU(1-|Δd|); ReLU usually stands for Rectified Linear Unit, which is the mainstream activation function in deep learning and is also a ready-made operator in PyTorch that can be used directly;
[0113] The calculated decay weights are used to weight and aggregate neighborhood points, which are then input into an MLP neural network to obtain a local feature tensor. The Multilayer Perceptron (MLP) is the most basic feedforward neural network architecture in deep learning, composed of multiple fully connected layers stacked together. It achieves its ability to fit complex functions through a non-linear activation function. The weighted aggregation method involves multiplying the absolute and relative coordinates of the neighborhood points by the linear decay weights, and then concatenating the results.
[0114] Typically, if we consider the local feature extraction module as a whole, then we have: Input [B, C in [,H,W]→output [B,C out [,H,W]; where B represents the batch size, i.e., the number of samples processed simultaneously in a single forward propagation during model training or inference; C in C is the number of channels in the input feature tensor. outThis refers to the number of channels in the feature tensor extracted from the local feature extraction module; [B,C] out [H,W] represents the dimension of the extracted local feature tensor, which is a fourth-order tensor.
[0115] In some embodiments, the AdamW optimizer is used during the training and inference phases, with an initial learning rate set to 1e-3, along with a hybrid loss function L. total End-to-end optimization is performed. The hybrid loss function is composed of Focal Loss and Dice Loss weighted by λ1 = 1.0 and λ2 = 0.75, i.e.:
[0116] ;
[0117] Among them, L focal Focal Loss aims to solve the imbalance between easy and difficult samples, and its calculation formula is as follows:
[0118] ;
[0119] In the formula, P t L represents the model's predicted probability of the true class, γ is the focusing parameter; its value is 2.0; dice DiceLoss is used to optimize the overlap of segmented regions, and its calculation formula is:
[0120] ;
[0121] Where, p i and g i ε = 1e-5 represents the values of the predicted probability map and the true label map at the i-th pixel, respectively, with ε = 1e-5 being a smoothing term used to prevent the denominator from being zero and to maintain numerical stability. This combined strategy balances the mining of difficult and easy samples with class imbalance correction.
[0122] In some embodiments, during the semantic segmentation of the actually acquired point cloud, the real-time acquired point cloud is preprocessed and subjected to overlap region fusion, feature extraction, and a two-stream network architecture and feature fusion to be transformed into a feature map. This feature map is then input into a trained network model to obtain a probability prediction map of [H, W]. Finally, a binary mask is generated by setting a threshold to obtain the final human semantic segmentation result. Typically, in semantic segmentation, foreground is represented by 1 and background by 0. The threshold is generally set to 0.5, meaning that a value greater than 0.5 in the probability map represents a point as foreground, which is represented by 1. If higher accuracy is desired, the threshold can be increased, for example, to 0.6. Generally, the specific setting of the threshold should consider accuracy, recall, and the final actual effect; if the threshold is too high, it may lead to a lower recall. Figure 3The image shown is a real-time point cloud data map. When a human body is located in the area where the two modules are spliced together, if the two modules are simply spliced together, the labels seen on the feature map will be fragmented, as shown below. Figure 4 To achieve the desired effect of directly stitching together real-time collected point cloud data, this invention utilizes a point cloud semantic segmentation method suitable for multi-module stitching scenarios. This method performs point cloud semantic segmentation based on spherical projection and distance-weighted aggregation, resulting in the following final effect. Figure 5 As shown, the human point cloud is accurately segmented from the background.
[0123] The point cloud semantic segmentation method disclosed in this invention, applicable to multi-module stitching scenarios, has the following beneficial technical effects:
[0124] Strong resistance to splicing distortion: Through innovative spherical grid mean fusion and soft voting mechanism, the "double skin" and "misalignment" problems in the overlapping area of multi-module sensors are effectively solved, making the algorithm more robust to mechanical installation errors and time synchronization errors.
[0125] Balancing computational efficiency and geometric accuracy: Compared to pure 3D methods such as PointNet++, this scheme achieves extremely high inference speed using a 2D convolutional architecture; compared to conventional Range Image methods, the introduced distance-weighted aggregation module effectively recovers lost 3D topological relationships and significantly improves the sharpness of segmentation edges.
[0126] Improving recall of small targets: The combination of dual-stream feature fusion and hybrid loss strategy enables the model to maintain a high recall rate even in monitoring scenarios where the proportion of human point clouds is extremely low (<5%), which greatly reduces the risk of missed detections in security scenarios.
[0127] The technical solutions and technical details disclosed in the embodiments of this invention are merely illustrative of the inventive concept of this invention and do not constitute a limitation on the technical solutions of the embodiments of this invention. Any conventional changes, substitutions, or combinations made to the technical details disclosed in the embodiments of this invention have the same inventive concept as this invention and are within the protection scope of the claims of this invention.
Claims
1. A point cloud semantic segmentation method suitable for multi-module stitching scenarios, characterized in that, Includes the following steps: S1. Perform spherical projection on the original point cloud obtained by multi-sensor stitching to construct a two-dimensional feature grid, and perform feature mean fusion and semantic soft voting on multi-source points falling into the same grid. S2. On the two-dimensional feature grid, a distance-weighted aggregation module is constructed based on depth differences to extract local geometric features with three-dimensional geometric perception capabilities; S3. The absolute coordinate features of the original point cloud are used as the global flow, and the extracted local geometric features are used as the local flow. The two-flow feature fusion is performed and input into the encoder-decoder network for multi-scale feature extraction and segmentation prediction. S4. The network is trained end-to-end using a hybrid loss function that includes Focal Loss and Dice Loss, and the semantic segmentation results are output.
2. The point cloud semantic segmentation method applicable to multi-module stitching scenarios according to claim 1, characterized in that, In step S1, constructing the two-dimensional feature mesh includes: First, determine the resolution as H×W; The horizontal field of view is divided into H parts and the vertical field of view is divided into W parts, resulting in H×W minimum windows; each minimum window constitutes a grid, and the point cloud is assigned to each grid.
3. The point cloud semantic segmentation method applicable to multi-module stitching scenarios according to claim 1, characterized in that, The weight calculation method for distance-weighted aggregation in step S2 is as follows: ； Where Δd represents the depth difference between the neighboring points and the center point.
4. The point cloud semantic segmentation method applicable to multi-module stitching scenarios according to claim 1, characterized in that, In step S3, an SE-Block channel attention mechanism is introduced in the skip connection between the encoder and the decoder.
5. The point cloud semantic segmentation method applicable to multi-module stitching scenarios according to claim 1, characterized in that, In step S3, the encoder-decoder network includes a four-layer downsampling encoder and a four-layer upsampling decoder.
6. The point cloud semantic segmentation method applicable to multi-module stitching scenarios according to claim 1, characterized in that, The expression for the hybrid loss function in step S4 is: ； Where λ1 = 1.0, λ2 = 0.75; ； In the formula P t γ represents the model's predicted probability of the true class, and γ is the focusing parameter; ； Where, p i and g i ε = 1e-5 represents the values of the predicted probability map and the true label map at the i-th pixel, respectively, and ε = 1e-5 is the smoothing term.
7. A point cloud semantic segmentation system suitable for multi-module stitching scenarios, used to execute the point cloud semantic segmentation method according to any one of claims 1 to 6, characterized in that, include: The preprocessing module is configured to perform spherical projection on the raw point cloud obtained by multi-sensor stitching, construct a two-dimensional feature grid, and perform feature mean fusion and semantic soft voting on multi-source points falling into the same grid; The feature extraction and fusion module is configured to construct a distance-weighted aggregation module based on depth differences on the two-dimensional feature grid, extract local geometric features with three-dimensional geometric perception capabilities; use the absolute coordinate features of the original point cloud as the global stream and the extracted local geometric features as the local stream, perform dual-stream feature fusion, and input the input into the encoder-decoder network for multi-scale feature extraction and segmentation prediction; The training and inference module is configured to perform end-to-end training of the network using a hybrid loss function that includes Focal Loss and Dice Loss, and output semantic segmentation results.
8. The point cloud semantic segmentation system for multi-module stitching scenarios according to claim 7, characterized in that, Also includes: A sensor module is used to acquire point cloud data from multiple perspectives; the sensor module includes at least two lidar or RGB-D cameras whose fields of view overlap.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 6.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 6.