An outdoor cleaning robot-oriented cross-view fusion point cloud segmentation method

Abstract

This invention discloses a cross-view fusion point cloud segmentation method for outdoor cleaning robots. It constructs a backbone network comprising a dual-branch encoder, feature fusion, and a dual-branch decoder. Based on point cloud data, dual-view projections of a spherical distance map and a polar coordinate bird's-eye view are performed. The dual-branch encoder then performs multi-scale feature extraction and fusion. Gating weight allocation is applied using the cross-view difference signal between the two views to enhance the fused features. The enhanced fused features are then processed by the dual-branch decoder and concatenated with the normalized coordinates of each point to output point-by-point fused features. On the unlabeled point cloud, spatial masks are independently sampled for each of the two views to represent the grid positions of the views. These decoded features are then processed by the backbone network to predict grid occupancy. The backbone network is optimized for point cloud segmentation by incorporating grid occupancy loss, grid displacement residual loss, and dual-view co-azimuth angle consistency loss.

Description

Technical Field

[0001] This invention belongs to the field of intelligent cleaning equipment and robot perception technology, specifically relating to a cross-view fusion point cloud segmentation method for outdoor cleaning robots. Background Technology

[0002] Outdoor cleaning robots need to identify cleanable areas, avoid dynamic obstacles, and recognize fixed environmental structures in real time during operation. LiDAR, with its all-weather, high-precision, and direct output of three-dimensional coordinates, has become the primary sensor for perception. However, the implementation of existing point cloud semantic segmentation models on cleaning robot platforms faces many challenges.

[0003] While existing point cloud semantic segmentation methods improve accuracy, they still face limitations in balancing performance and efficiency. First, although raw point-based processing strategies can preserve fine geometric structures, their irregular memory access patterns are severely mismatched with the parallel computing requirements of edge-side NPUs and dedicated accelerators, making it difficult to meet real-time processing requirements. Second, while projection-based methods can utilize the efficiency of convolutional neural networks, single-viewpoint methods suffer from inherent information representation deficiencies. For example, distance views (RV) exhibit severe target deformation, while bird's-eye views (BEV) face trade-offs between spatial utilization and long-distance modeling, resulting in a single projection surface failing to capture both global semantics and local details. Finally, existing multi-view fusion strategies generally rely on complex reprojection, voxelization, or high-order interpolation algorithms when handling information flow between heterogeneous coordinate systems. These operations not only disrupt feature continuity but also generate significant intermediate computational overhead and memory usage, becoming a major obstacle to the efficient deployment of multi-view fusion technology on computationally limited edge platforms.

[0004] Furthermore, edge computing platforms face strict constraints on operator library compatibility. Many existing complex fusion mechanisms or custom operators are difficult to accelerate at the hardware level on resource-constrained embedded chips. This operator-level compatibility bottleneck makes it difficult to deploy high-performance algorithms in real time on edge devices with low power consumption and high performance requirements.

[0005] In the algorithm training and generalization stages, the scarcity of high-quality labeled data and sampling efficiency issues are equally prominent. Labeling cross-scene point cloud data is time-consuming and labor-intensive, and under the constraint of low labeling budget, existing technologies struggle to achieve efficient sample selection through active learning. Traditional methods often rely on random sampling or a single model uncertainty indicator, failing to comprehensively consider the representativeness of the samples, class balance, and temporal feature redundancy, resulting in significant sampling bias in the selected samples. Furthermore, existing self-supervised pre-training methods such as masked autoencoders (MAEs) often struggle to fully capture deep geometric structural features when processing sparse point clouds, limiting the model's generalization ability and convergence speed in downstream tasks.

[0006] Therefore, it is necessary to design a point cloud semantic segmentation method that can support self-supervised pre-training, has strong operator compatibility, low annotation requirements, is computationally lightweight, and can maintain high segmentation accuracy. Summary of the Invention

[0007] To address the shortcomings of existing technologies and improve the point cloud semantic segmentation accuracy of outdoor cleaning robots, reduce computational overhead, and enhance generalization capabilities across different scenarios, this invention adopts the following technical solution:

[0008] A cross-view fusion point cloud segmentation method for outdoor cleaning robots includes the following steps:

[0009] A backbone network comprising a dual-branch encoder, feature fusion, and dual-branch decoder is constructed. Based on point cloud data, dual-view projections of a spherical distance map and a polar coordinate bird's-eye view are performed to obtain two two-dimensional tensors sharing the same width in the azimuth dimension, providing a foundation for subsequent cross-view alignment. The two tensors are then processed by the dual-branch encoder for multi-scale feature extraction, and depth-layered cross-view azimuth alignment is performed at each fusion scale to execute the feature fusion. The cross-view difference signal, formed by the difference in the mean features within the same depth band of the same azimuth column in both views, participates in the gating weight allocation to enhance the fused features. The enhanced fused features are then processed by the dual-branch decoder and concatenated with the normalized coordinates of each point to output point-by-point fused features.

[0010] On the unlabeled point cloud, spatial masks are independently sampled for each of the two views to represent the grid positions of the views. The input features of the masked positions are replaced with learnable mask tokens. Through the input replacement mechanism, the model is prevented from directly reading the original feature information of the masked positions, thereby forcing the model to rely on the context information of the surrounding visible areas to recover the geometric structure and semantic content of the masked positions. The input features after masking are decoded by the backbone network. At the masked positions, a cross-entropy loss is constructed to predict grid occupancy through the decoded features, and a displacement regression loss is constructed to predict the displacement residual of the mean center of the visible occupied grid. The displacement regression loss is preferentially applied to the real occupied grids in the neighborhood without visible occupied grid areas. A co-azimuth column consistency loss for the two views is constructed, and the backbone network is jointly optimized to obtain pre-trained parameters with geometric and semantic priors.

[0011] A semantic segmentation model is constructed based on the optimized backbone network to segment point clouds according to the point-by-point fusion features.

[0012] Furthermore, the spherical distance map calculates spherical coordinate parameters based on point cloud data. According to the elevation and azimuth angles, points are mapped to a spherical grid. For multiple points mapped to a unified grid, the features of the closest point are retained based on the distance in the spherical coordinate parameters, and basic features are constructed at the grid. For each grid position, three-dimensional angle features are calculated based on its corresponding elevation and azimuth angles, filling all pixels in the corresponding grid. This corrects the loss of positional information caused by zeroing empty pixels in the basic features, allowing the model to still obtain complete directional information in blank areas, while also expanding the total number of feature channels. For each grid position, the three-dimensional coordinates of its four neighboring valid points are taken, and two tangent vectors are calculated along the central difference direction. The cross product of these two tangent vectors is taken and normalized to obtain the surface normal vector. Only when the grid position... When there are effective laser points in all four neighboring regions (up, down, left, and right), the surface normal vector is filled into the basic feature. Otherwise, set to zero; this calculation is equivalent to the second-order central difference approximation of the spherical projection Jacobian matrix, which can extract local surface direction information from a single frame point cloud without adding any additional sensors, and has an auxiliary role in distinguishing planar roads, edge structures and three-dimensional obstacles. At the same time, it further expands the total number of channels.

[0013] Furthermore, the feature fusion process involves dividing the dual-view radial dimension into multiple corresponding depth bands, performing average pooling and max pooling within each depth band to obtain column-direction statistical descriptors for the dual-view, calculating the difference signal between the two views and combining it with the descriptors to generate gating weights for the dual-view, broadcasting the gating weights to the corresponding depth band regions, and performing element-wise modulation on the original features to generate fusion increments, thereby obtaining enhanced fusion features through residual methods.

[0014] Furthermore, the spherical distance map branch is divided into multiple elevation zones in terms of its height dimension, with each elevation zone covering its corresponding elevation angle range, corresponding to different distance layers from near to far in physical space; the polar coordinate bird's-eye view branch is divided into multiple radial zones in terms of its radial dimension, with each radial zone covering its corresponding radial distance range; the two branches have the same number of zones to form corresponding zones of the same depth.

[0015] In the dual-view branch, the depth band extracts the corresponding local feature regions from the feature maps of the corresponding scales based on the grid row intervals of the image. Average pooling and max pooling operations are performed along the height dimension within the local feature regions to obtain two statistical descriptors representing average response features and significant response features. This results in consistent scale column direction feature representations of the spherical distance map and the polar coordinate bird's-eye view on each depth band, providing a foundation for subsequent cross-view alignment and fusion.

[0016] Within each depth band, the difference in average pooling results between the two views on the same azimuth column is calculated. This difference signal is used as an observable diagnostic measure of the consistency of observations between the two views on the corresponding azimuth column within that depth band. When the observations of the same physical region by the two views are consistent... If the difference is smaller, the difference signal directly participates in the generation of subsequent gating weights, enabling the model to perceive the magnitude of the divergence between the two views and adjust the fusion strategy accordingly.

[0017] Furthermore, the descriptors of the dual-view branches corresponding to the depth band are concatenated with the difference signal in the channel dimension to form a joint context. The joint context is then subjected to feature compression and nonlinear mapping through an aggregated convolutional layer, followed by channel-by-channel mapping to obtain the gating coefficients for the dual-view branches.

[0018] The gating coefficients are broadcast along the height dimension to the corresponding grid row intervals in the feature map of the dual view, and multiplied element by element with the features in that interval to obtain the local fusion increment of the depth band on the dual view branch.

[0019] For the local fusion increments of all depth bands, the corresponding radial dimensions are stitched together and restored to obtain the complete incremental features;

[0020] The complete incremental features are residually fused with the original features to obtain the enhanced bi-branch features.

[0021] The entire DS-CAFF module consists only of static structural operators, including average pooling, max pooling, 1×1 convolution, batch normalization, ReLU, sigmoid, element-wise multiplication and addition, as well as tensor concatenation and broadcasting operations. All of the above operators belong to regular tensor operations and do not rely on dynamic indexing or self-attention mechanisms. Therefore, they can be directly mapped to the standard operator set of mainstream embedded NPUs. In deployment toolchains such as RKNN, there is no need to introduce custom operators or approximate replacements, thus ensuring high compatibility and high running efficiency of edge deployment.

[0022] Furthermore, the dual-branch decoder first performs bilinear upsampling on the fused features, then concatenates them with the corresponding encoded features, and fuses them through depthwise separable convolution to recover the decoded features; using the normalized coordinates of each point on the dual views, bilinear interpolation sampling is performed on the corresponding decoded features to obtain point-by-point features;

[0023] A point-by-point view confidence gating mechanism is employed. After stitching together the point-by-point features of the two views, a linear transformation is performed, and an activation function is used to obtain the channel-by-channel gating coefficients of the spherical distance map branch. These coefficients are then used to generate the channel-by-channel gating coefficients of the complementary polar coordinate bird's-eye view.

[0024] The weighted features are obtained by multiplying the channel-by-channel gating coefficients with the corresponding point-by-point features, and then by adding the gated weighted features of the two views together to output a weighted fused feature.

[0025] Furthermore, a multi-scale occupancy reconstruction subtask is constructed, the decoded features are input into the occupancy prediction head, and the occupancy prediction results at multiple scales are output. The existence of a real occupied grid in the receptive field at the corresponding scale of the dual views is taken as the occupancy ground truth. A binary cross-entropy loss with positive sample weights is applied to each scale.

[0026] A residual center displacement regression subtask is constructed. For the masked occupied grid, a three-dimensional displacement residual is constructed relative to the mean center of the visible occupied grid in the current frame. This prevents the model from taking the shortcut of directly copying the absolute coordinates through position encoding, and forces the masked position to rely on the visual context to recover the local geometric offset. A two-stage regression approach is used for two-stage residual prediction. In the first stage, the center regression head predicts the initial residual based on the decoded features. In the second stage, the real residual is used for supervision at the visible position, and the initial residual predicted in the first stage is used to fill the masked position to obtain the fused context features. The decoded features and the context features are concatenated and passed through the second-stage regression head to obtain the final predicted residual. For each supervised grid, a displacement regression loss is used for residual prediction regression learning, and a weighting coefficient based on distance and positive samples is introduced.

[0027] A sparse neighborhood-first supervision strategy is constructed. A neighborhood visibility indicator map is built for the visible occupied area. Based on the neighborhood visibility, the masked grid is divided into dense areas with visible occupied areas and sparse areas without visible occupied areas. The displacement regression supervision position is limited to the set of real occupied grids in the sparse area, and the displacement regression loss is applied. By applying displacement regression supervision only in the sparse area, the model's ability to infer the structure of distant low-density areas is enhanced. When there is no effective supervision position in a training frame, it automatically falls back to all masked occupied areas for supervision to avoid zero loss leading to training instability.

[0028] Mean pooling is performed along the spatial dimension on the decoded features of the two views to obtain azimuth column descriptors. Visual weights are defined based on whether the azimuth column is completely masked in the two views. A similarity-based two-view structure is then constructed based on the azimuth column descriptors and the visual weights. Figure 1 Consistency loss; this loss is used to constrain the feature representations of RV and PB to maintain directional consistency in the same azimuth sector, thereby promoting the model to learn a geometrically consistent cross-view shared semantic space without the need for manual annotation.

[0029] Furthermore, the parameters of the jointly trained backbone network are loaded into the semantic segmentation model, and a classification head corresponding to the target classification is adopted. A small learning rate is used for the dual-branch encoder, a medium learning rate is used for the feature fusion and dual-molecule decoder, and a large learning rate is used for the classification head. Through a group learning rate strategy, while preserving the geometric and semantic priors of the pre-training stage, the classification head is allowed to converge quickly to the target domain distribution, thereby improving the rapid adaptation ability of the classification head to the target scene. A supervised loss function is constructed, and the jointly trained backbone network is subjected to supervised fine-tuning training using labeled frames. During the training process, a learning rate scheduling strategy combining linear learning rate warm-up and cosine annealing is adopted, and data augmentation methods such as random rotation, random flipping, random scale perturbation, and random point discarding can be enabled according to the actual scene. After training, the checkpoints of the model with the best performance are saved according to the mIoU index of the validation set to obtain the final semantic segmentation model weights.

[0030] The supervised loss function uses cross-entropy loss. With Lovász-Softmax loss The weighted combination is used as the main monitoring target:

[0031]

[0032] in, Represents cross-entropy loss; This represents the Lovász-Softmax loss; This indicates the optional RV branch to assist pixel-level supervision loss; , and To correspond to the loss weight coefficients, different class weights are applied to different classes to alleviate the problem of imbalanced class distribution. RV branch auxiliary supervision constructs additional supervision signals through point-to-point index mapping to RV pixels to enhance the stability of backbone feature learning.

[0033] Furthermore, the semantic segmentation model is explicitly split into a preprocessing stage, a backbone inference stage, and a postprocessing stage when deployed on the edge. The preprocessing stage and the postprocessing stage run on a general-purpose central processing unit (CPU), while the backbone inference stage is exported as a statically shaped open neural network exchange format (ONNX) and, after being converted by the RKNN toolchain, runs on an embedded neural network processor (NPU). This allows the CPU and the NPU to process different stages of different frames within the same time window, forming a three-stage pipeline.

[0034] The preprocessing stage includes point cloud reading, dual-view projection, coordinate normalization, and angle position encoding calculation; this stage includes operations that are not suitable for NPU execution, such as dynamic point indexing, irregular scattering with an uncertain number of points per frame, and dynamic tensor construction.

[0035] The main inference stage includes a dual-branch encoder, feature fusion and dual-branch decoder, as well as RV Stem convolution and PB Stem convolution. This stage only includes regular tensor operations such as static shape convolution, depthwise separable convolution, pooling, element-wise addition and broadcast multiplication.

[0036] The post-processing stage includes bilinear sampling of point-by-point fusion features, view confidence gating calculation, and forward inference of the classification head for point cloud segmentation; this stage involves dynamic point counting and irregular sampling operations.

[0037] Furthermore, the CPU preprocessing, NPU backbone inference, and CPU postprocessing stages process different stages of different frames within the same time window, thus forming a heterogeneous three-stage pipeline. This three-stage pipeline operates at steady-state conditions. Simultaneous execution: CPU thread A on the... Frame pre-processing output Embedded Neural Processing Unit (NPU) for the first Frame execution backbone inference output ; Central Processing Unit (CPU) thread B on the first Frame post-processing output In a three-stage pipeline at time T+1: CPU thread A performs a process on the first... Frame preprocessing; Embedded neural network processor (NPU) for the first... Frame execution backbone inference; CPU thread B on the first Frame post-processing and output of final result The above pipeline mechanism enables parallel collaborative execution of the CPU and NPU. A first-in-first-out (FIFO) buffer queue is set between the preprocessing stage and the main inference stage, and a buffer queue of equal length is set between the main inference stage and the postprocessing stage. After the upstream stage completes the calculation, it writes the result into the corresponding queue, and the downstream stage reads the data from the queue. When the queue is full, the upstream stage blocks and waits; when the queue is empty, the downstream stage blocks and waits. Through the double buffer queue mechanism, the three-stage steady-state parallel operation is achieved, and data race and frame misalignment problems are avoided.

[0038] The CPU / NPU heterogeneous three-stage pipelined inference deployment scheme of the present invention divides preprocessing, backbone inference and postprocessing into independent stages, and realizes parallel pipelined execution through dual buffer queues, thereby improving the steady-state throughput of the system and meeting the real-time operation requirements of outdoor cleaning robots on the embedded NPU platform.

[0039] The advantages and beneficial effects of this invention are as follows:

[0040] The invention proposes a depth-layered cross-view azimuth alignment feature fusion mechanism, which fuses dual-view features in layers according to depth bands and introduces cross-view difference signals to participate in gating weight generation, enhancing the feature representation ability of different distance regions and improving the semantic segmentation accuracy of distant sparse targets. This process consists only of static shape operators, making it easy to deploy on embedded NPU platforms. The invention also proposes a residual-based center displacement regression self-supervised task, defining the center regression target as the displacement residual relative to the mean center of the visible occupied grid. Residual modeling reduces prior position interference, and combined with a two-stage predicted structure, guides the network to recover local geometric structure information from the context, thereby improving the semantic discriminative ability of the backbone features. Furthermore, the invention proposes a sparse neighborhood-priority supervision strategy, applying displacement regression supervision only to sparse regions where no visible occupied grid exists in the neighborhood, thereby strengthening the model's ability to infer the structure of distant low-density regions and ensuring training stability through an empty set back-off mechanism. Finally, the invention proposes a dual-view occupancy-displacement joint self-supervised pre-training scheme, which jointly executes multi-scale occupancy reconstruction, residual-based displacement regression, and cross-view... Figure 1 Consistency constraints enable the network to learn geometrically consistent shared semantic representations without human annotation, thereby reducing the dependence of downstream tasks on labeled data; the method of this invention can be transferred to different types of LiDAR platforms and has good compatibility. Attached Figure Description

[0041] Figure 1 This is a flowchart of the method in an embodiment of the present invention.

[0042] Figure 2 This is a diagram of the DS-CAFF module architecture in an embodiment of the present invention.

[0043] Figure 3 This is a flowchart of the dual-view occupancy-displacement joint self-supervised pre-training method in an embodiment of the present invention. Detailed Implementation

[0044] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0045] like Figure 1 As shown, a cross-view fusion point cloud segmentation method for outdoor cleaning robots includes a dual-view fusion architecture, self-supervised pre-training, few-shot supervised learning, and lightweight edge deployment. It is suitable for rapidly deploying semantic perception models for outdoor cleaning scenarios under low annotation budgets and edge computing platform constraints. Specifically, it includes the following steps:

[0046] Step 1: Dual-view feature construction and dual-branch trunk forward pass;

[0047] Projecting a single frame of LiDAR point cloud onto a spherical range view (RV) and a polar bird's-eye view (PB) yields two two-dimensional tensors sharing the same width W in the azimuth dimension. These two tensors are then fed into a dual-branch encoder for multi-scale feature extraction. At each fusion scale, Depth-Stratified Cross-view Azimuth-aligned Feature Fusion (DS-CAFF) is performed. The cross-view difference signal, formed by the difference between the feature mean values ​​of RV and PB within the same depth band in the same azimuth column, participates in the gating weight allocation. After the fused features are processed by the dual-branch decoder, they are sampled with the normalized coordinates of each point to output point-by-point fused features. The specific steps include the following:

[0048] Step 1.1: Acquire lidar point cloud data;

[0049] Acquiring single-frame LiDAR point cloud data Each point is represented as:

[0050]

[0051] in, , , These represent the three-dimensional spatial coordinates of the point, This indicates the intensity of laser reflection.

[0052] Step 1.2: Construct the spherical distance view RV;

[0053] For each lidar point Calculate the spherical coordinate parameters:

[0054]

[0055]

[0056]

[0057] According to the angle of elevation With azimuth Discretize the points and map them to spherical grid:

[0058]

[0059]

[0060] in, and These represent the upper and lower field of view of the lidar, respectively.

[0061] For multiple points mapped to the same raster, according to distance Sort by distance, retaining only the features corresponding to the nearest point; set all grid positions not matched by the point cloud to zero. Construct at least six-dimensional basic features at each RV grid, including:

[0062]

[0063] Where R represents the maximum sensing distance.

[0064] Furthermore, three-dimensional angular position coding can be added. , , and three-dimensional surface normal vectors This results in extended multi-channel RV input features, as detailed below:

[0065] When angular position encoding is enabled, a three-dimensional angular channel is added after the above six-dimensional basic features: for each grid position According to its corresponding elevation angle With azimuth calculate , , 3D angular features are calculated and filled directly for all pixels according to grid coordinates, fixing issues in the basic features. The absence of positional information caused by handling zeros in empty pixels allows the model to still obtain complete orientation information in blank areas, while expanding the total number of channels to nine dimensions.

[0066] When surface normal vectors are further enabled, a three-dimensional normal vector channel is added after the aforementioned nine-dimensional channel: for each grid position (i,j), the three-dimensional coordinates of its four neighboring valid points are taken, and two tangent vectors are calculated along the central difference direction:

[0067]

[0068] The surface normal vector is obtained by taking the cross product of the two tangent vectors and normalizing it.

[0069]

[0070] Only when Fill normal vector when there are valid laser points in all four neighborhoods (up, down, left, and right). Otherwise, set to zero; this calculation is equivalent to the second-order central difference approximation of the spherical projection Jacobian matrix, which can extract local surface orientation information from a single frame point cloud without adding any additional sensors, and has an auxiliary role in distinguishing planar roads, edge structures and three-dimensional obstacles; at this time, the total number of channels is finally expanded to twelve dimensions.

[0071] Step 1.3: Construct a polar coordinate bird's-eye view PB;

[0072] Each lidar point Projected onto the horizontal plane XY, and then in polar coordinates on the horizontal plane XY. Discretization The formula for the polar coordinate parameters of the raster is as follows:

[0073]

[0074]

[0075] The polar ring boundary is divided along the radial distance using a square root spacing, resulting in a denser grid near the edge and a sparser grid at a distance, aligning with the law of lidar point density attenuation with distance.

[0076]

[0077] satisfy:

[0078]

[0079] The corresponding azimuth column index is:

[0080]

[0081] Statistical aggregation is performed on all points falling within the same PB grid to construct a nine-dimensional grid feature, including: normalized mean coordinates of points within the grid (3D); normalized height of the lowest and highest points within the grid (2D); mean intensity within the grid (1D); logarithmic statistic of the number of points within the grid (1D); mean normalized distance (1D); and occupancy indicator (1D).

[0082] When there is no point cloud within the raster, all corresponding channels are set to zero.

[0083] Step 1.4: Two-branch multi-scale feature encoding;

[0084] A symmetrical bi-branch encoder is constructed by inputting RV and PB feature tensors into the structure, respectively. Each encoding branch includes: a Stem convolutional layer with a stride of 1 and four downsampling stages, each consisting of an inverse residual block formed by stacked depthwise separable convolutions. Each stage is implemented using stacked inverse residual blocks of depthwise separable convolutions. In the s-th encoding stage, the RV branch outputs the feature map as follows: The PB branch output feature map is The two branches at each scale Share the same azimuth dimension This provides a foundation for subsequent cross-view alignment.

[0085] Step 1.5: DS-CAFF depth-layered cross-view azimuth alignment feature fusion;

[0086] At each scale Above, RV features Features of PB Perform the DS-CAFF fusion operation to obtain the fused output at this scale. Both have the same shape as the input.

[0087] Specifically, such as Figure 2 As shown, firstly, K corresponding depth bands are divided along the height dimension of RV and the radial dimension of PB respectively; then, average pooling and max pooling are performed in each depth band to obtain the column direction statistical descriptors of RV and PB.

[0088] Furthermore, the difference signal between the two views is calculated:

[0089]

[0090] Then, it is input together with the pooling descriptor into the gating generation module to obtain the gating weights of the RV and PB branches: , ;

[0091] Subsequently, the gating weights are broadcast to the corresponding depth band regions, and element-wise modulation is performed on the original features to generate the fusion increment:

[0092]

[0093] Finally, the enhanced fusion features are obtained through residual methods:

[0094]

[0095]

[0096] The specific steps for DS-CAFF depth-layered cross-view azimuth alignment feature fusion are as follows:

[0097] Step 1.5.1: Depth layering;

[0098] RV branch at scale The height dimension Divided into equal parts The first elevation angle band, the first Coverage of elevation angle range This corresponds to different distance layers in physical space, from near to far; the PB branch is scaled... The radial dimension is also equally divided into The radial band, the first Covered radial distance range The number of bands in the two branches The same, causing RV and PB to form For each pair of corresponding depth bands, observations are performed within a similar distance range in physical space. All band boundary indices are calculated and fixed using Python integers during module initialization, and eliminated by constant folding during ONNX export, without introducing dynamic shapes, thus meeting the requirements of embedded NPUs for static computation graphs.

[0099] Step 1.5.2: Inner azimuth pooling;

[0100] For the One depth band ( In the RV branch, based on the pre-divided intervals... From the stage Feature map Extract the corresponding local feature regions.

[0101] Along the height dimension within this local feature region Performing average pooling and max pooling operations respectively yields two statistical descriptors:

[0102]

[0103] in, and These represent the average response characteristics and significant response characteristics on the RV branch within this depth band, respectively.

[0104] For the PB branch, a consistent pooling operation is performed under the same depth band partitioning strategy to obtain the corresponding descriptor:

[0105]

[0106] Thus in each depth band The consistent scale column direction feature representations of the RV and PB views are obtained, providing a foundation for subsequent cross-view alignment and fusion.

[0107] Step 1.5.3: Cross-view difference signal;

[0108] Calculate the difference between the average pooling results of RV and PB in the same azimuth column within each depth band:

[0109]

[0110] As an observable diagnostic measure of the consistency of observations between the two views on the corresponding azimuth column: when the observations of the same physical area by the two views are consistent. The difference is smaller if the difference is smaller, and vice versa; this difference signal directly participates in the generation of subsequent gating weights, enabling the model to perceive the magnitude of the divergence between the two views and adjust the fusion strategy accordingly.

[0111] Step 1.5.4: Joint context construction and gating generation;

[0112] The first The four descriptors corresponding to each depth band are concatenated with the difference signal at the channel dimension to form a joint context representation. Its dimensions are :

[0113]

[0114] in, This indicates a splicing operation along the channel dimension. .

[0115] For joint context By aggregating convolutional layers This layer performs feature compression and nonlinear mapping, consisting of 1×1 convolutions, batch normalization (BN), and... Activation function composition:

[0116]

[0117] Subsequently, separate gating generation modules were introduced for the RV and PB branches respectively. Perform channel-by-channel mapping to generate gating coefficients:

[0118]

[0119]

[0120] in, and All dimensions are .

[0121] Step 1.5.5: Bidirectional in-band residual modulation;

[0122] For the Each depth band will have a gating factor. Along the height dimension Broadcast to the corresponding row interval in the RV feature map Then, it is multiplied element-wise with the features in that region to obtain the fusion increment of that depth band on the RV branch:

[0123]

[0124] Similarly, Broadcasting to the corresponding radial interval of the PB branch yields:

[0125]

[0126] in, This represents the radial dimension index of the PB branch.

[0127] For all depth bands The local increments are concatenated and restored along the corresponding radial dimension to obtain the complete feature increments:

[0128]

[0129] Step 1.5.6: Residual fusion output;

[0130] Finally, the incremental features and the original features are residually fused to obtain the enhanced bi-branch feature representation:

[0131]

[0132]

[0133] The entire DS-CAFF module consists of only static structure operators, including average pooling, max pooling, 1×1 convolution, batch normalization, ReLU, Sigmoid, element-wise multiplication and addition, as well as tensor concatenation and broadcasting operations.

[0134] The operators described above all belong to regularized tensor operations, which do not rely on dynamic indexing or self-attention mechanisms. Therefore, they can be directly mapped to the standard operator set of mainstream embedded NPUs. In deployment toolchains such as RKNN, there is no need to introduce custom operators or approximate replacements, thus ensuring high compatibility and high operational efficiency for edge deployments.

[0135] Step 1.6: Dual-branch decoding and point-by-point feature recovery;

[0136] Dual-branch decoding: fusing features from four scales The inputs are given to the RV decoder and PB decoder respectively. Each decoding stage first performs bilinear upsampling, then concatenates it with the features from the corresponding encoding stage, and finally fuses them using depthwise separable convolutions to restore the resolution to the same level as the first stage of the encoder. and The decoder feature map is recovered as follows:

[0137]

[0138]

[0139] Point-by-point feature sampling and aggregation: For each LiDAR point Use its normalized coordinates on RV With normalized coordinates on PB , respectively in and Perform bilinear interpolation sampling to obtain point-by-point features. and Furthermore, the point-by-point view confidence gating module is used to calculate the fusion weight of RV and PB, i.e., the point-by-point view confidence gating weight. The weighted fusion of the two point-by-point features is then concatenated with the normalized coordinates of the point to form the final fused feature for that point. Finally, the final fused features of each point are fed into the classification head, which outputs the probability distribution of each LiDAR point belonging to each semantic category.

[0140] Specifically, the execution process of the point-by-point view confidence gating module is as follows:

[0141] Pointwise features of RV and PB branches and The vectors are concatenated to form a joint input vector:

[0142]

[0143] in, This indicates a splicing operation along the channel dimension.

[0144] Will Input to linear transformation layer And using the Sigmoid activation function, the channel-wise gating coefficients of the RV branch are obtained:

[0145]

[0146] The gating coefficient of the PB branch is defined as its complementary form:

[0147]

[0148] This satisfies the channel-by-channel normalization constraint:

[0149]

[0150] Based on this, the RV and PB features are gated and weighted to obtain the final fused features:

[0151]

[0152] in, This indicates element-wise multiplication.

[0153] The constant 2 in the above coefficients is used to ensure that during the parameter initialization phase, when When initialized to a zero matrix, we have At this point, the fusion result degenerates into:

[0154]

[0155] This is equivalent to equal-weighted feature fusion, thus maintaining consistency with the ungated baseline and facilitating the loading of pre-trained models and stable fine-tuning.

[0156] Step 2: Dual-view occupancy-displacement joint self-supervised pre-training;

[0157] On the unlabeled point cloud, spatial masks are independently sampled for RV and PB, and the input features at the masked locations are replaced with learnable mask tokens. The model simultaneously performs two complementary reconstruction subtasks at the mask locations: a multi-scale occupancy reconstruction subtask (occupancy head) and a residual center displacement regression subtask (center head). The center head defines the prediction target as the displacement residual relative to the mean center of the visible occupancy grid in the current frame, and applies supervision only to far-end hard cases where no visible occupancy grid exists in the neighborhood. Simultaneously, a dual-view cosine consistency loss is superimposed, and joint optimization yields pre-trained weights with geometric and semantic priors, such as... Figure 3 As shown, the specific steps include the following:

[0158] Step 2.1: Generate independent masks for both views;

[0159] Spatial masks are generated independently for the RV view and the PB view respectively. and The mask value is either 0 or 1, with 1 indicating that the corresponding grid position is masked. Mask generation employs a weighted combination of four strategies: random mask, rectangular block mask, row and column strip mask, and hierarchical coarse-fine mixed mask. The hierarchical coarse-fine mixed mask first generates a low-resolution coarse-grained random mask with a preset stride of 8. After restoring the original resolution through nearest-neighbor upsampling, a fine-grained random perturbation is then superimposed to simultaneously maintain spatial block continuity and boundary randomness. Furthermore, an incremental scheduling strategy is used for the overall mask rate: a lower mask rate is used in the early stages of training, and the mask ratio is gradually increased in the later stages, allowing the model to progressively learn reconstruction tasks from easy to difficult.

[0160] Step 2.2: Replace the input mask token;

[0161] For the masked location, the original input features are replaced with learnable mask tokens, and all input channels are replaced with learnable mask tokens for the corresponding view. and :

[0162]

[0163]

[0164] in, and These represent the learnable mask tokens corresponding to the RV and PB views, respectively.

[0165] By using an input substitution mechanism, the model is prevented from directly reading the original feature information of the masked location, thereby forcing the model to rely on the context information of the surrounding visible area to recover the geometric structure and semantic content of the masked location.

[0166] Step 2.3: Forward propagation of the encoding / decoding backbone;

[0167] Input features after masking and Input the dual-branch encoder-decoder backbone network from step 1 to obtain the decoded output features:

[0168]

[0169]

[0170] The subsequent prediction head and the center displacement regression head share the above-mentioned backbone output features, without introducing an additional independent coding network.

[0171] Step 2.4: Multi-scale occupation reconstruction subtask (occupation head);

[0172] The question "Does the corresponding grid have a LiDAR return point?" from the original input is taken as the true value y. The following are the values ​​of the input cells: and Input occupies prediction head Output occupancy prediction results at multiple (S) scales. The true value of occupancy at the corresponding scale. The following is obtained by max pooling the original occupancy map according to the corresponding receptive field: when there is any real occupied grid in the receptive field, the ground value of that position is recorded as 1.

[0173] Apply binary cross-entropy loss with positive sample weights to each scale:

[0174]

[0175] in, Represents the set of effective monitoring locations. This indicates that at least one visible occupied raster exists within the raster's neighborhood. Supervision is applied only to locations that are "masked and have visible occupied neighbors" to avoid gradient degradation caused by completely empty regions; positive sample weights. The ratio of positive to negative samples within the current batch is adaptively estimated and updated using an exponential moving average (EMA) method, and is limited to a preset interval. The residual center displacement regression subtask (center head) regresses the 3D displacement residual relative to the "mean center of all visible occupied grids in the current frame" for the position of the masked occupied grid. This is to prevent the model from taking the shortcut of directly copying absolute coordinates through position encoding, so that the masked position must rely on visual context to recover the local geometric offset.

[0176] Step 2.5: Residualized center displacement regression subtask (center head);

[0177] For the masked occupied grid, a three-dimensional displacement residual regression task is constructed relative to the mean center of the visible occupied area in the current frame. Specifically, this task regresses the three-dimensional displacement residual relative to the mean center of all visible occupied grids in the current frame by regressing the position of the masked occupied grid. This approach aims to prevent the model from taking the shortcut of directly copying absolute coordinates through position encoding, forcing the masked position to rely on visual context to recover the local geometric offset. Specifically, it includes the following steps:

[0178] Step 2.5.1: Define the residual target;

[0179] For each grid Let the geometric center of the actual laser point within the grid be denoted as . Define the mean center of all visible grid cells in the current frame:

[0180]

[0181] in, This represents the set of grid cells that are not masked and are actually occupied.

[0182] Based on this, the regression learning objective is defined as the relative offset residual:

[0183]

[0184] Step 2.5.2: Use a two-stage regression method to predict the two-stage residuals.

[0185] The first phase involves the center returning to its head. Based on trunk features Predict the initial residual:

[0186]

[0187] And apply a stopping gradient operation to it. This ensures that the learning of backbone features is not affected during backpropagation.

[0188] Subsequently, the second-stage fusion context features are constructed:

[0189]

[0190] in, Indicates the position of the mask. This indicates the visible locations. Specifically, visible locations are supervised using the true residuals, while masked locations are filled with the first-stage prediction results. The backbone features are then used. With context features Channel splicing was performed, followed by a second-stage regression head. The final prediction result is obtained:

[0191]

[0192] Step 2.5.3: Weighted loss for close proximity and positive samples;

[0193] For each supervised grid The regression learning is performed using Smooth L1 loss, and a weighting coefficient based on distance and positive samples is introduced:

[0194]

[0195] The corresponding regression loss uses weighted Smooth L1 loss:

[0196]

[0197] in, The weights are the joint weights of distance and positive samples. Indicates the near-distance threshold. Supervisory set The sparse neighborhood priority sampling strategy determined in step 2.6 is used to improve the regression accuracy of key regions while ensuring computational efficiency.

[0198] Step 2.6: Sparse neighborhood priority supervision strategy;

[0199] The masked rasters are divided into two categories based on their neighborhood visibility: "dense neighborhood masked rasters" and "sparse neighborhood masked rasters." Displacement regression loss is applied only to the latter, which is equivalent to a hard case mining mechanism based on geometric neighborhood visibility. The steps include:

[0200] Step 2.6.1: Construct a neighborhood visibility indicator map for the visible occupied area;

[0201] With a preset neighborhood radius Visible occupied set conduct Local max pooling operation to construct visible neighborhood indicator graph :

[0202]

[0203] in, Indicates Centered on, with radius A local window area. When When, it indicates that there is at least one visible occupied grid cell within the grid's neighborhood, belonging to a dense neighborhood; when When the value is 0, it indicates that the region is a sparse neighborhood.

[0204] Step 2.6.2: Filtering difficult examples;

[0205] The mask grid will be divided into two categories based on neighborhood visibility:

[0206]

[0207]

[0208] in, This indicates the existence of a densely occupied region within the neighborhood, the displacement of which can be approximated by the local structure; This represents a sparse region that is not visible within the neighborhood. This type of region relies on global context modeling and is considered a hard case region. It typically corresponds to key perception areas such as distant low obstacles, roadside extension structures, and sparse dynamic targets.

[0209] Step 2.6.3: Define the supervised set;

[0210] Limit the displacement regression supervision location to the set of truly occupied grid cells in sparse hard examples:

[0211]

[0212] This means that displacement regression supervision is applied only to sparse and hard example regions to enhance the model's ability to infer structures in distant, low-density regions. When there are no effective supervision locations in a training frame, the model automatically falls back to the entire masked region for supervision to avoid zero loss and unstable training.

[0213] Step 2.7: Consistency constraint of co-azimuth column in both views;

[0214] Decoding features of RV and PB respectively and Perform mean pooling along the spatial dimension to obtain the azimuth column descriptors:

[0215]

[0216]

[0217] in, This indicates the azimuth column index.

[0218] Define a visible weight for each azimuth column:

[0219]

[0220] When the azimuth column is not completely masked in both RV and PB, take Otherwise take .

[0221] Based on this, a dual-view structure based on cosine similarity is constructed. Figure 1 Induced loss:

[0222]

[0223] in, Represents the cosine similarity function.

[0224] This loss constraint is used to constrain the feature representations of RV and PB to maintain directional consistency in the same azimuth sector, thereby promoting the model's learning of a geometrically consistent cross-view shared semantic space without manual annotation. Essentially, this consistency constraint can be viewed as a weakly supervised cross-view alignment mechanism to enhance the structural consistency and semantic alignability between the RV and PB views.

[0225] Step 2.8: Joint optimization training;

[0226] The final total pre-training loss is defined as:

[0227]

[0228] Among them, each The weight coefficients of the corresponding loss terms are represented, allowing asymmetric configuration of the RV and PB branches to adapt to the differences in point density between the two views. During training, the AdamW optimizer is used, combined with linear learning rate warm-up and cosine annealing scheduling strategies, and self-supervised training is performed on the full unlabeled point cloud dataset. After training converges, the backbone network parameters are saved as the initial weights for downstream semantic segmentation tasks.

[0229] Step 3: Supervised fine-tuning. Supervised fine-tuning is performed on the pre-trained backbone using a small number of labeled frames. Pre-trained weights are loaded onto the backbone with a small learning rate, while the classification head is randomly initialized with a large learning rate. A weighted loss of cross-entropy and Lovász-Softmax is used to obtain the final semantic segmentation model. This includes the following steps:

[0230] Step 3.1: Load pre-trained weights;

[0231] The self-supervised pre-trained backbone network parameters (weights) obtained in step 2 are loaded into the semantic segmentation model, including the RV branch encoder, PB branch encoder, DS-CAFF fusion module, and dual-branch decoder. The classification head is reconstructed using randomly initialized fully connected layers corresponding to the number of target categories, without loading the occupancy prediction head and center displacement regression head parameters from the pre-training stage.

[0232] Step 3.2: Setting the learning rate for each group;

[0233] The model parameters are grouped according to the functions of the network modules, and different learning rate ratios are applied.

[0234] The encoder portion, which loads pre-trained weights, includes RV Stem, PB Stem, RV Encoder, and PB Encoder, and uses a relatively small learning rate (typically...). The DS-CAFF fusion module and dual-branch decoder employ a moderate learning rate (typically...). The randomly initialized classification head (including the pointwise aggregation head and optional auxiliary head) uses the base learning rate (). By employing a grouped learning rate strategy, the geometric and semantic priors from the pre-training phase are preserved while allowing the classification head to converge quickly to the target domain distribution, thereby improving the classification head's ability to adapt rapidly to the target scene.

[0235] Step 3.3: Supervised loss function;

[0236] Using cross-entropy loss With Lovász-Softmax loss The weighted combination is used as the main monitoring target:

[0237]

[0238] in, Represents cross-entropy loss; This represents the Lovász-Softmax loss; This indicates the optional RV branch to assist pixel-level supervision loss; , and This corresponds to the loss weighting coefficient.

[0239] Furthermore, to alleviate the problem of imbalanced class distribution, different class weights are applied to different classes. RV branch-assisted supervision constructs additional supervision signals through point-to-point index mapping to RV pixels to enhance the stability of backbone feature learning.

[0240] Step 3.4: Supervise and fine-tune the training process;

[0241] Supervised fine-tuning training is performed on a small number of labeled frames over multiple rounds (typically 120 epochs). During training, a learning rate scheduling strategy combining linear learning rate warm-up and cosine annealing is employed, and data augmentation methods such as random rotation, random flipping, random scale perturbation, and random point dropping can be enabled depending on the specific scenario. After training, the checkpoints of the best-performing model are saved based on the validation set mIoU metric to obtain the final semantic segmentation model weights.

[0242] Step 4: CPU / NPU Heterogeneous Three-Stage Pipeline Inference Deployment. When deploying the trained network on the edge, it is explicitly split into three stages: preprocessing, backbone inference, and post-processing. Preprocessing and post-processing run on a general-purpose central processing unit (CPU). The backbone inference is exported as a static shape ONNX (Open Neural Network Exchange) and, after conversion by the RKNN (Rockchip Neural Network) toolchain, runs on an embedded neural network processor (NPU). The CPU and NPU process different stages of different frames within the same time window, forming a three-stage pipeline; including the following steps:

[0243] Step 4.1: Deploy the model in three stages;

[0244] The entire network is divided into a preprocessing section, a backbone inference section, and a postprocessing section according to the characteristics of the operators.

[0245] The preprocessing stage is deployed on the CPU and includes: raw point cloud reading, RV spherical projection, PB polar coordinate projection, coordinate normalization, and angle position encoding calculation. This stage includes operations that are not suitable for NPU execution, such as dynamic point indexing, irregular scattering with an uncertain number of points per frame, and dynamic tensor construction.

[0246] The backbone inference section is deployed on an embedded NPU and includes: RV Stem convolution, PB Stem convolution, a dual-branch four-stage encoder, DS-CAFF fusion modules at various scales, and a dual-branch four-stage decoder; this stage only includes regular tensor operations such as static shape convolution, depthwise separable convolution, pooling, element-wise addition, and broadcast multiplication; the network uses a fixed input size ( and Static batch size The ONNX model is exported and then deployed to the embedded NPU after quantization transformation via the RKNN toolchain.

[0247] The post-processing stage is deployed on the CPU and includes: point-by-point feature bilinear sampling, view confidence gating calculation, and classification head forward inference; this stage involves dynamic point counting and irregular sampling operations.

[0248] Step 4.2: Three-stage pipeline timing execution;

[0249] The CPU preprocessing, NPU backbone inference, and CPU postprocessing stages each process different stages of different frames within the same time window, thus forming a heterogeneous three-stage pipeline.

[0250] Let the first Frame preprocessing output is ;No. The frame backbone NPU inference output is ;No. The final semantic result output by the post-frame processing semantic tag is .

[0251] At steady state Simultaneous execution: CPU thread A on the first Frame pre-processing output NPU for the first Frame execution backbone inference output CPU thread B on the first Frame post-processing output .

[0252] Next time step (T+1): CPU thread A performs a task on the first... Frame preprocessing; NPU for the first Frame execution backbone inference; CPU thread B performs inference on the first... Frame post-processing and output of final result .

[0253] The above pipeline mechanism enables parallel collaborative execution between the CPU and NPU.

[0254] Step 4.3: Inter-stage double-buffered queue;

[0255] A first-in-first-out buffer queue is set up between the CPU preprocessing stage and the NPU backbone inference stage. Set up a buffer queue of equal length between the NPU backbone inference segment and the CPU post-processing segment. After the upstream stage completes its calculation, it writes the result to the corresponding queue. The downstream stage reads data from the queue. When the queue is full, the upstream stage blocks and waits; when the queue is empty, the downstream stage blocks and waits. Through a double-buffered queue mechanism, the three-stage steady-state parallel operation is achieved, avoiding data races and frame misalignment issues.

[0256] Step 4.4: End-to-end latency and throughput analysis;

[0257] The end-to-end latency of a single frame degrades from the sum of the three serial stages to the sum of the pipeline setup stages:

[0258]

[0259] in, Indicates preprocessing time. Indicates the time of the main reasoning. The post-processing time is determined by the longest of the three stages, i.e., the pipeline steady-state throughput is:

[0260]

[0261] During the main reasoning time Significantly greater than and On a typical embedded platform, since the NPU backbone inference time is usually significantly longer than the preprocessing and postprocessing times, the steady-state throughput is approximately:

[0262]

[0263] Compared to the serial execution method:

[0264]

[0265] The throughput improvement factor is approximately: .

[0266] This heterogeneous three-stage pipeline deployment method can significantly improve the system's real-time throughput performance and meet the deployment requirements of outdoor cleaning robots for real-time semantic perception.

[0267] Based on the above method, this invention also proposes a cross-view fusion point cloud segmentation system for outdoor cleaning robots, including a point cloud dual-view construction module, a dual-branch feature encoding module, a DS-CAFF deep hierarchical cross-view feature fusion module, a dual-view joint decoding module, a point-by-point feature recovery module, a self-supervised pre-training module, and a heterogeneous pipeline deployment module.

[0268] The input LiDAR point cloud is first projected to generate a spherical distance view (RV) and a polar coordinate bird's-eye view (PB). Then, the features of the two views are input into a dual-branch encoder for multi-scale feature extraction. At each scale, the DS-CAFF module performs cross-view azimuth alignment and depth hierarchical feature fusion. The fused features are decoded to restore the spatial resolution, and point-level fusion features are obtained through point-by-point bilinear sampling, which finally completes the semantic classification of the point cloud.

[0269] In the pre-training phase, multi-scale occupancy reconstruction subtask, residualized center displacement regression subtask, and dual-view subtask are further introduced. Figure 1 Consistency constraints are implemented to enhance the backbone network's ability to model geometric structures and cross-view semantic relationships, and to reduce the reliance of downstream tasks on manually labeled data.

[0270] During the deployment phase, a CPU / NPU heterogeneous three-stage pipeline structure is adopted, dividing the overall inference process into three stages: preprocessing, backbone inference, and postprocessing, in order to improve the real-time inference throughput on the edge side.

[0271] The dual-view joint decoding module comprises a dual-view construction module, a dual-view mask generation module, a Mask Token replacement module, a shared encoding and decoding network, an occupation reconstruction branch, a center displacement regression branch, and cross-view... Figure 1 The consistency constraint module executes the above-mentioned dual-view occupancy-displacement joint self-supervised pre-training method.

[0272] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-view fusion point cloud segmentation method for outdoor cleaning robots, characterized in that: A backbone network consisting of a dual-branch encoder, feature fusion, and dual-branch decoder is constructed. Based on point cloud data, dual-view projections of spherical distance map and polar coordinate bird's-eye view are performed to obtain two tensors that share the same width in the azimuth dimension. The two tensors obtained are processed by the dual-branch encoder for multi-scale feature extraction, and azimuth alignment is performed at each scale to perform feature fusion. The cross-view difference signal formed by the difference of feature mean values ​​in the same depth band at the same azimuth angle of the two views participates in the gating weight allocation to enhance the fused features. The enhanced fused features are then processed by the dual-branch decoder and concatenated with the normalized coordinates of each point to output the point-by-point fused features. On the unlabeled point cloud, spatial masks are independently sampled for each of the two views to represent the grid position of the view, and the input features of the masked position are replaced with learnable mask tokens. The input features after masking are then passed through the backbone network to obtain the decoded features. A cross-entropy loss is constructed to predict grid occupancy using the decoded features. A displacement regression loss is constructed to predict the displacement residual of the mean center of the visible occupied grid. The displacement regression loss is applied preferentially to the real occupied grid in the neighborhood where there are no visible occupied grids. A dual-view co-azimuth angle consistency loss is constructed and the backbone network is jointly optimized. A semantic segmentation model is constructed based on the optimized backbone network to segment point clouds according to the point-by-point fusion features.

2. The cross-view fusion point cloud segmentation method for outdoor cleaning robots according to claim 1, characterized in that: The spherical distance map is calculated based on point cloud data to obtain spherical coordinate parameters. According to the elevation and azimuth angles, the points are mapped to a spherical grid. For multiple points mapped to a unified grid, the features of the closest point are retained according to the distance in the spherical coordinate parameters, and basic features are constructed at the grid. For each grid position, calculate the three-dimensional angle feature according to its corresponding elevation angle and azimuth angle, and fill all pixels in the corresponding grid; for each grid position, take the three-dimensional coordinates of its effective neighboring points, calculate two tangent vectors according to the central difference direction, take the cross product of the two tangent vectors and normalize them to obtain the surface normal vector, and fill the surface normal vector into the basic feature only when there are effective points in the neighborhood of the grid position; otherwise, set it to zero.

3. The cross-view fusion point cloud segmentation method for outdoor cleaning robots according to claim 1, characterized in that: The feature fusion process involves dividing the two views into multiple corresponding depth bands along their radial dimension. In each depth band, average pooling and max pooling are performed to obtain column-direction statistical descriptors for the two views. The difference signal between the two views is calculated and combined with the descriptors to generate gating weights for the two views. The gating weights are then broadcast to the corresponding depth band regions, and element-wise modulation is performed on the original features to generate fusion increments. The enhanced fusion features are obtained through residual methods.

4. The cross-view fusion point cloud segmentation method for outdoor cleaning robots according to claim 3, characterized in that: The spherical distance map branch is divided into multiple elevation zones in terms of its height dimension, with each elevation zone covering its corresponding elevation angle range, corresponding to different distance layers from near to far in physical space; the polar coordinate bird's-eye view branch is divided into multiple radial zones in terms of its radial dimension, with each radial zone covering its corresponding radial distance range; the two branches have the same number of zones to form corresponding zones of the same depth. In the dual-view branch, the depth band extracts the corresponding local feature regions from the feature maps of the corresponding scales based on the grid row intervals of the image. Average pooling and max pooling operations are performed along the height dimension within the local feature regions to obtain two statistical descriptors representing the average response features and the salient response features. The difference between the average pooling results of the two views in the same azimuth column is calculated within each depth band to obtain the difference signal.

5. A cross-view fusion point cloud segmentation method for outdoor cleaning robots according to claim 3, characterized in that: The descriptor of the dual-view branch corresponding to the depth band is concatenated with the difference signal in the channel dimension to form a joint context. The joint context is then compressed and nonlinearly mapped through an aggregated convolutional layer, followed by channel-by-channel mapping to obtain the gating coefficients for the dual-view branch. The gating coefficients are broadcast along the height dimension to the corresponding grid row intervals in the feature map of the dual view, and multiplied element by element with the features in that interval to obtain the local fusion increment of the depth band on the dual view branch. For the local fusion increments of all depth bands, the corresponding radial dimensions are stitched together and restored to obtain the complete incremental features; The complete incremental features are residually fused with the original features to obtain the enhanced bi-branch features.

6. The cross-view fusion point cloud segmentation method for outdoor cleaning robots according to claim 1, characterized in that: The dual-branch decoder first performs bilinear upsampling on the fused features, then concatenates them with the corresponding encoded features, and fuses them through depthwise separable convolution to recover the decoded features; By using the normalized coordinates of each point on the dual views, bilinear interpolation sampling is performed on the corresponding decoded features to obtain point-by-point features; A point-by-point view confidence gating mechanism is employed. After stitching together the point-by-point features of the two views, a linear transformation is performed, and an activation function is used to obtain the channel-by-channel gating coefficients of the spherical distance map branch. These coefficients are then used to generate the channel-by-channel gating coefficients of the complementary polar coordinate bird's-eye view. The weighted features are obtained by multiplying the channel-by-channel gating coefficients with the corresponding point-by-point features, and then by adding the gated weighted features of the two views together to output a weighted fused feature.

7. The cross-view fusion point cloud segmentation method for outdoor cleaning robots according to claim 1, characterized in that: The existence of a real occupied grid in the receptive field at the corresponding scale of the two views is used as the true value of occupancy. A binary cross-entropy loss with positive sample weights is applied to each scale. The initial residual is predicted based on the decoded features. The real residual is used for supervision at the visible position and the initial residual is used to fill the masked position. The final displacement residual is predicted based on the obtained fused context features and the decoded features. For each supervised grid, a displacement regression loss is used to perform regression learning for the displacement residual prediction, and a weighting coefficient based on distance and positive samples is introduced. Based on neighborhood visibility, sparse regions without visible occupancy are divided from the masked grid, and the displacement regression supervision position is limited to the set of real occupied grids in the sparse region to apply the displacement regression loss. Based on the decoding features of the two views, a per-azimuth column descriptor is obtained. The visibility weight is obtained according to whether the azimuth column is completely masked in the two views. Based on the per-azimuth column descriptor and the visibility weight, a similarity-based co-azimuth consistency loss of the two views is constructed.

8. The cross-view fusion point cloud segmentation method for outdoor cleaning robots according to claim 1, characterized in that: The parameters of the jointly trained backbone network are loaded into the semantic segmentation model, and a classification head corresponding to the target classification is adopted. A small learning rate is used for the dual-branch encoder, a medium learning rate is used for the feature fusion and dual-molecule decoder, and a large learning rate is used for the classification head. A supervised loss function is constructed, and supervised fine-tuning training is performed using labeled frames. During the training process, a learning rate scheduling strategy combining linear learning rate warm-up and cosine annealing is adopted. After training is completed, the final semantic segmentation model is obtained.

9. A cross-view fusion point cloud segmentation method for outdoor cleaning robots according to claim 1, characterized in that: The semantic segmentation model is divided into a preprocessing stage, a backbone inference stage, and a postprocessing stage when deployed on the edge. The preprocessing stage and the postprocessing stage run on a general-purpose central processing unit (CPU). The backbone inference stage is exported as a static shape open neural network exchange format and, after being converted by a toolchain, runs on an embedded neural network processor. This allows the CPU and the embedded neural network processor to process different stages of different frames within the same time window, forming a three-stage pipeline. The preprocessing stage includes point cloud reading, dual-view projection, and coordinate normalization; The main inference stage includes a dual-branch encoder, feature fusion, and a dual-branch decoder; The post-processing stage includes bilinear sampling of point-by-point fusion features, gating computation, and forward inference of the classification head for point cloud segmentation.

10. A cross-view fusion point cloud segmentation method for outdoor cleaning robots according to claim 9, characterized in that: The three-stage pipeline at steady state Simultaneous execution: Central Processing Unit thread A on the... Frame pre-processing output Embedded neural network processors for the first Frame execution backbone inference output ; Central Processing Unit thread B on the first Frame post-processing output ; In a three-stage pipeline at time T+1: CPU thread A processes thread A... Frame preprocessing; Embedded neural network processors for the first Frame execution backbone inference; CPU thread B performs inference on the first... Frame post-processing and output of final result A first-in-first-out buffer queue is set up between the preprocessing stage and the main inference stage, and a buffer queue of the same length is set up between the main inference stage and the postprocessing stage. After the upstream stage completes the calculation, it writes the result into the corresponding queue. The downstream stage reads data from the queue. When the queue is full, the upstream stage is blocked and waits. When the queue is empty, the downstream stage is blocked and waits.

Images