A multi-modal multi-task spatial perception method for environmental sanitation operation

Abstract

This invention discloses a multimodal, multi-task spatial perception method for sanitation operations, comprising: acquiring LiDAR data and multi-view camera image sequences, constructing bird's-eye view representations for each, obtaining point cloud bird's-eye view features and image bird's-eye view features respectively; calculating fusion weights for the point cloud bird's-eye view features and image bird's-eye view features at the bird's-eye view grid positions based on the difference in semantic distribution between the two modalities and the confidence of each modality, and obtaining a shared bird's-eye view feature base through dynamic gating weighted fusion; performing target detection, semantic segmentation, and vector map detection on the shared bird's-eye view feature base respectively, obtaining the 3D bounding box parameters, semantic category probabilities, map element categories, and their point coordinate sequences of the target objects respectively; and correlating and cross-validating the 3D bounding box parameters, semantic category probabilities, map element categories, and their point coordinate sequences of the target objects.

Description

Technical Field

[0001] This invention relates to the field of spatial perception technology, and in particular to a multimodal, multi-task spatial perception method for sanitation operations. Background Technology

[0002] In the field of autonomous sanitation sweepers, the environmental perception system is a crucial component for achieving high-precision spatial positioning, obstacle avoidance, road debris identification, and sweeping operation planning. Sanitation sweepers typically operate at low speeds in complex, unstructured environments, including areas with dense, fine debris (such as fallen leaves, paper scraps, and plastic bottles), low obstacles (such as curbs and cones), and moving pedestrians and vehicles. Furthermore, they must maintain stable perception capabilities under adverse lighting and weather conditions, such as at night and in rain or fog. To meet the engineering requirements of sweeping operations, the perception system not only needs high-precision spatial positioning and rich semantic understanding capabilities, but also requires low computational burden and short processing latency on the onboard edge computing platform to support real-time closed-loop decision-making.

[0003] Currently, the most common multimodal bird's eye view (BEV) perception methods in publicly available technologies typically fuse camera images and LiDAR point clouds to map multi-source information onto the same spatial coordinate system, thereby achieving tasks such as 3D object detection, BEV semantic segmentation, and vector map detection. While these methods have achieved some success in general autonomous driving scenarios, they still have significant shortcomings when directly applied to sanitation vehicles: firstly, existing multimodal BEV methods usually only cover single tasks such as 3D object detection or grid semantic segmentation; even if they can output multi-task results, they often struggle to directly obtain structured vector map elements that can be used for planning and control, such as lane lines, curbs, pedestrian crossings, and parking lines. On the other hand, the requirements for the accuracy and geometric consistency of structured road information are higher when sanitation sweepers operate close to the edge, avoid boundaries, and delineate work areas. However, existing multi-task models mostly adopt the "shared backbone + parallel multi-head" approach, which lacks effective information interaction and spatial consistency constraints between task branches. This makes it difficult to improve detection, segmentation, and map output in sync and maintain geometric correspondence, resulting in problems such as misalignment between map elements and segmentation boundaries, and inconsistency between detection targets and drivable areas, which affects the reliability of downstream planning and control.

[0004] Furthermore, sanitation scenarios contain numerous low-lying and irregularly shaped targets (such as fallen leaves, curbs, and traffic cones). Existing BEV feature encoding and post-processing methods lack the ability to model multi-scale context and edge details, easily leading to a high false negative rate for low-lying targets and exhibiting jagged segmentation or boundary prediction results, as well as topological discontinuities. Consequently, even with multi-task output capabilities, it is difficult to achieve the boundary accuracy and map feature quality required for cleaning operations.

[0005] To address the problems of the existing technologies mentioned above, a multimodal, multi-task spatial perception method for sanitation operations is designed. Summary of the Invention

[0006] In view of this, the purpose of this invention is to propose a multimodal, multi-task spatial perception method for sanitation operations, which can solve the above-mentioned problems.

[0007] This invention provides a multimodal, multi-task spatial perception method for sanitation operations, comprising: S1 acquires LiDAR data and multi-view camera image sequences, and constructs bird's-eye view representations for them respectively, obtaining point cloud bird's-eye view features and image bird's-eye view features respectively; S2 calculates the fusion weights of point cloud bird's-eye view features and image bird's-eye view features at the bird's-eye view grid positions based on the difference in semantic distribution between the two modalities and the confidence of each modality. Through dynamic gating weighted fusion, a shared bird's-eye view feature base is obtained. S3 performs target detection, semantic segmentation, and vector map detection on the shared bird's-eye view feature base, respectively, and obtains the 3D bounding box parameters, semantic category probability, map element category and its point coordinate sequence of the target; S4 correlates and verifies the target object's 3D bounding box parameters, semantic category probability, map element category, and its point coordinate sequence, and outputs the verified target object's 3D bounding box parameters, semantic category probability, map element category, and its point coordinate sequence.

[0008] The beneficial effects of this invention are: First, a unified coordinate BEV representation is constructed using point clouds and multi-view cameras, and pose alignment is performed. Pillar columnar volumetric height encoding is used to preserve the vertical structure of low-lying targets. This solves the problems of difficult cross-modal geometric alignment in sanitation scenarios, easy loss of height information of low-lying and small obstacles, and single-modal instability under night, rain, and fog conditions. As a result, an alignable point cloud BEV and image BEV feature base are obtained, providing reliable input for subsequent fusion and multi-task output.

[0009] Secondly, by calculating the semantic distribution consistency (normalized KL) and modal confidence of point cloud / image at the BEV grid location, and combining class priors for dynamic gated weighted fusion; at the same time, residual compensation and encoder (SalsaNext variant) are used to extract multi-scale texture and global semantics to meet the real-time requirements of the edge; to solve the modal conflict and insufficient shared feature expression caused by rain, fog reflection and point cloud sparsity; thereby improving the stability of cross-modal fusion and obtaining a high-quality shared bird's-eye view base.

[0010] Third, by sharing a bird's-eye view base, it outputs 3D object detection, height-wise semantic segmentation, and end-to-end structure prediction of vector maps based on queries and multi-scale attention; it solves the problem that "object localization - drivable / obstacle semantics - road structured boundary" in sanitation operations is difficult to express in a unified way and difficult to directly serve planning; thus obtaining structured results that can be used for obstacle avoidance and cleaning boundary constraints, reducing the risk of topological breakage and boundary jaggedness caused by grid-to-vector post-processing.

[0011] Fourth, by projecting and associating 3D bounding boxes, segmented semantics, and vector maps, and performing multiple checks based on the consistency of allowed semantic sets, consistency of projection geometric hits, and consistency of map sampling semantics, only candidates that are consistent in all three aspects are retained; this solves cross-task semantic conflicts, geometric mismatches, and invalid boundaries caused by occlusion, reflection, and projection errors; thereby improving the credibility and consistency of the joint results of the three tasks and reducing the risk of obstacle avoidance misjudgment and operational boundary jitter caused by erroneous results entering the planning process. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings required in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of the method in this embodiment. Detailed Implementation

[0014] To facilitate understanding by those skilled in the art, the structure of the present invention will now be described in further detail with reference to the accompanying drawings. It should be understood that, unless otherwise specified, the order of the steps mentioned in this embodiment can be adjusted according to actual needs, and they can even be executed simultaneously or partially simultaneously.

[0015] like Figure 1 As shown, this embodiment of the invention provides a multimodal, multi-task spatial perception method for sanitation operations, including: S1 acquires LiDAR data and multi-view camera image sequences, and constructs bird's-eye view representations for them respectively, obtaining point cloud bird's-eye view features and image bird's-eye view features respectively; Sanitation work areas are low-speed, unstructured environments with diverse target types and large scales. Representing these targets using only a single modality can easily lead to insufficient geometric accuracy or semantic understanding. BEV (Balanced Elementary Vehicle) representation projects three-dimensional space onto a regular two-dimensional plane, facilitating subsequent fusion, encoding, and multi-task sharing. The specific steps are as follows: S101 projects LiDAR data and multi-view camera image sequences onto the global coordinate system according to pose transformation parameters; S102 divides the unified coordinate system LiDAR data into a fixed grid on the bird's-eye view plane, forms columnar voxels in a columnar manner, and truncates or fills zeros on the number of points within the columnar voxels. S103 encodes columnar voxels according to the sanitation scene by height interval, forming columnar voxel feature vectors that retain the height structure. S1031 divides the Z direction of the global coordinate system into four unequal height intervals according to the prior low target in the sanitation scene. The height intervals include: ground layer, low layer, medium layer, and high layer. Each height interval is defined by the corresponding Z coordinate threshold. S1032 For each column voxel, collect the set of points it contains. For each point within the column voxel, determine its assigned height interval based on the point's Z coordinate, and assign the point to the set of points in the corresponding height interval. S1033 If the number of points in a certain height interval corresponding to a certain column voxel is 0, then the interval-level partitioning feature of that height interval is set to the zero vector. For each height interval of the columnar voxel, S1034 calculates the geometric center of the point within the height interval based on the Z coordinate of the point within the height interval, and calculates the offset vector of each point within the height interval relative to the geometric center, and uses the offset vector as the point-level feature input of the point within the height interval. S1035 performs point-by-point mapping on the point-level features of each height interval of the column voxel, and aggregates them into a fixed-length interval-level partition feature vector by applying a symmetric aggregation function to the set of points in the height interval. S1036 concatenates the interval-level partitioning features corresponding to the four height intervals of the column voxel according to the channel dimension to obtain the column voxel feature vector that retains the height structure.

[0016] In this step, low-lying obstacles in sanitation operations are sensitive to their Z-direction structure. Traditional methods of crudely compressing height information tend to smooth out fine-grained differences. This invention explicitly preserves vertical structure information by dividing the Z-direction into multiple unequal height intervals and encoding four height layers within each pillar (column voxel). For example: ground layer: fallen leaves / paper scraps and other ground-level targets; low-lying layer: curbs, gravel, low traffic cones; medium-lying layer: stone blocks, standard traffic cones, etc.; higher layer: pedestrians, vehicles / garbage cans, etc.

[0017] An independent point set encoding channel is constructed for each height layer to avoid the loss of height information caused by mixing points from multiple layers. Within the same pillar, there may be both ground-level leaf points and slightly higher gravel points. This step ensures that they enter different height encoding channels, improving boundary recognition. In open road areas where only ground-level points exist, if the middle and upper levels are not set to zero, the network may generate "false middle and upper level structures," leading to false detections.

[0018] Road edges are typically not perfectly planar and often exhibit discontinuities in some areas. Using relative geometric center offset makes it easier to capture local shape differences and improves boundary continuity. Relative offset encoding is more robust than absolute coordinates and enhances the ability to represent local shapes and distributions, making it particularly suitable for irregular structures such as fallen leaf debris and road edge.

[0019] The point cloud has a variable and unordered number of points. Symmetrical aggregation ensures that features are independent of the point order and maps the variable-length point set to a fixed-length vector, facilitating end-to-end training. At the same horizontal location, both ground debris and low curbs may coexist. The stitched four-channel height structure features help the model distinguish between situations where debris lies below and obstacles rise above.

[0020] S104 scatters the columnar voxel feature vectors that retain the height structure back to the bird's-eye view to obtain the point cloud bird's-eye view features; The S105 extracts multi-scale two-dimensional features from the image from each camera's perspective. It then aligns and fuses the different scales through a feature pyramid network to obtain a multi-scale two-dimensional feature map. Finally, it performs viewpoint feature transformation to obtain the bird's-eye view features of the image.

[0021] S1051 acquires a multi-scale two-dimensional feature map, performs depth branch prediction on the multi-scale two-dimensional feature map, and outputs the discrete depth probability distribution corresponding to each pixel. S1052 elevates each pixel and each depth to a set of 3D sampling points in the camera coordinate system based on the camera imaging geometry. S1053 transforms 3D points in the camera coordinate system to a unified coordinate system and uses the bird's-eye view grid definition to map the planar coordinates of 3D points to the bird's-eye view grid index. S1054 performs weighted aggregation of the features of all samples whose projections fall into the same bird's-eye view grid cell using discrete depth probabilities to obtain the bird's-eye view grid features; S1055 rearranges the bird's-eye view grid features into tensors by channel to obtain the bird's-eye view features of the image.

[0022] In this step, rain, fog, or strong nighttime reflections make it difficult to stably estimate depth at distant locations. Utilizing depth probability distributions in conjunction with weighted aggregation can improve robustness and reduce misprojection noise. The same pixel may correspond to different candidates for curb height or ground debris height. Multi-sampling helps the network select features corresponding to the correct height structure. Parallax exists between multi-view cameras. By unifying the coordinate system and grid index mapping, 3D candidates resulting from parallax can be mapped to the same BEV grid cell, reducing fusion noise. Distant road textures are prone to distortion at night. Regions with more dispersed depth probabilities contribute less weight to features, thus reducing segmentation boundary jitter caused by misprojection. Rearrangement organization ensures that the tensor dimension and spatial size are consistent during the fusion stage, avoiding errors in feature correspondence caused by channel misalignment.

[0023] S2 calculates the fusion weights of point cloud bird's-eye view features and image bird's-eye view features at the bird's-eye view grid positions based on the difference in semantic distribution between the two modalities and the confidence of each modality. Through dynamic gating weighted fusion, a shared bird's-eye view feature base is obtained. S201 calculates the local semantic distribution of point cloud bird's-eye view features and image bird's-eye view features at the bird's-eye view grid positions, respectively, to obtain the semantic distribution probability of point cloud bird's-eye view and image bird's-eye view. S202 calculates the semantic distribution probability of the point cloud bird's-eye view and the normalized KL divergence of the semantic distribution probability of the image bird's-eye view to obtain the semantic consistency score. S203 estimates the confidence scores of two modalities for each bird's-eye view location. The semantic consistency score, the confidence score of the point cloud modality, and the category prior are input into a dynamic gating function to obtain the location-related point cloud fusion weights and image fusion weights. The calculation formula is as follows: , , in, Represents the point cloud modal fusion weights. express Sigmoid Activation function 、 、 This represents the learnable balance coefficient. This represents the prior biases guided by semantic categories. Indicates semantic consistency score, Represents the confidence level of the point cloud mode. Represents the image modality fusion weights; In this step, camera uncertainty is often the direct cause of semantic inconsistency. For example, in a region lacking texture (such as a white wall), camera semantic segmentation will output a uniform, high-entropy distribution, which will inevitably produce a large KL divergence with the geometry-based semantic prediction of the radar point cloud (such as "wall"), thus leading to a lower semantic consistency score. Therefore, the semantic consistency score already encodes the main information about camera unreliability, and there is no need to further estimate the image confidence. Therefore, the confidence of the point cloud modality is relied upon first, and the image confidence is considered only when the confidence of the cloud modality fails.

[0024] From an asymmetric cost perspective: In autonomous driving perception, the cost of missing or misdetecting geometric structures (such as curbs and vehicle edges) is far higher than that of texture errors. LiDAR's geometric accuracy is generally superior to that of cameras, so fusion strategies naturally tend to prioritize LiDAR when information conflicts occur, and then refer to the camera when LiDAR is uncertain. This asymmetry is reflected in: Only learn σ_lidar² (because the confidence level of LiDAR needs to be precisely quantized); Camera uncertainty is triggered by "semantic divergence," and the negative coefficient α transforms the conflict into an increase in LiDAR weights. S204 performs a weighted summation of the two modal features based on the location-related point cloud fusion weights to obtain the weighted fusion features; S205 stitches together point cloud bird's-eye view features and image bird's-eye view features to obtain stitched features. It then superimposes the weighted fusion features and stitched features through residual connection. Finally, it downsamples to compress the spatial size to half of the original to obtain fused bird's-eye view features. In this step, when the sweeper approaches the curb in rainy weather, the curb may be confused in color due to reflection / rain film in the camera's view. This causes the probability distribution of the image branch for the "curb / shadow / wetland" category to diverge, and the point cloud branch may become sparse due to local rain attenuation. The semantic distribution is also uncertain. Therefore, it is necessary to calculate the semantic distribution probability of each modality to provide a basis for subsequent fusion.

[0025] Fusion methods need to avoid forcibly mixing "modal conflict regions" to prevent mislearning. KL divergence can measure the degree of deviation between the semantic distributions of two modalities, providing a learnable / derivable conflict intensity signal for dynamic gating. Traditional fusion methods are difficult to operate stably under conditions such as drastic changes in illumination, rain and fog obstruction, and strong reflection interference. Dynamic gating achieves "using the modality that is more reliable at the current position" and arbitrates in the event of semantic conflicts.

[0026] In large drivable road areas, downsampling reduces blank calculations; in areas near curbs and cones, residual stacking preserves local details and avoids boundary blurring caused by downsampling.

[0027] S206 uses the SalsaNext variant encoding network to perform deep encoding on the fused bird's-eye view features to obtain a shared bird's-eye view feature base.

[0028] In this step, the SalsaNext variant encoder network refers to an improved version of the BEV feature encoder based on the classic semantic segmentation network SalsaNext. It is used to perform deep encoding on the fused BEV features, thereby outputting a shared BEV feature base. Key sanitation targets are often low-lying and irregularly shaped, requiring consideration of both local texture details and global road structure semantics. The encoder needs to improve feature representation capabilities without significantly increasing computational cost, providing a high-quality shared base for the subsequent three tasks (detection, segmentation, and vector mapping). The specific steps are as follows: S2061 extracts local texture information and global semantic information from the fused bird's-eye view features through residual context blocks of the SalsaNext variant encoding network; S2062 aggregates local texture information and global semantic information based on the dilated convolution branches of the residual context block to obtain aggregated texture semantic information. The aggregated texture semantic information of each dilated convolution branch is then weighted and fused by channel attention to output weighted and fused aggregated texture semantic information. S2063 first determines the effective region and background region based on the feature response intensity. It performs full convolution and downsampling only on the weighted fusion aggregated texture semantic information with a response higher than the adaptive threshold. It compresses the low-response background region with low-resolution pooling to obtain sparse texture semantic information. S2064 performs shallow feature sampling and deep feature sampling on the sparse texture semantic information respectively, and then fuses them to obtain the skipping sparse texture semantic information; S2065 adaptively selects transposed convolution or bilinear interpolation to perform upsampling on the skipping sparse texture semantic information content, and obtains the upsampled skipping sparse texture semantic information through gated fusion. S2066 performs channel progressive compression and lightweight convolution on the upsampled sparse texture semantic information to obtain lightweight sparse texture semantic information, aligns it with the required spatial size, and obtains a shared bird's-eye view feature base.

[0029] In this step, local texture information and global semantic information are extracted through residual context blocks to address the problem of easily lost edge details of low-lying obstacles in sanitation scenarios, and to provide global semantics for road topology understanding. Dilated convolution can expand the receptive field without significantly increasing the number of parameters, enabling the network to capture both distant road structures and nearby details. In BEV space, many grids may have no targets or are merely background, and standard convolution would be computationally wasteful; sparse downsampling is used to improve efficiency and maintain the quality of key region features. Downsampling can lead to the loss of edge details; shallow-deep skip fusion can preserve the edge localization capability of low-lying targets. The shared base needs to support three types of tasks simultaneously: detection, segmentation, and vector mapping, requiring feature dimensions and spatial size to be adapted. Channel compression is used to reduce the number of parameters and computational load to meet the real-time inference requirements of the vehicle edge.

[0030] S3 performs target detection, semantic segmentation, and vector map detection on the shared bird's-eye view feature base, respectively, and obtains the 3D bounding box parameters, semantic category probability, map element category and its point coordinate sequence of the target; In this step, sanitation operations need to simultaneously acquire "objects (obstacles / pedestrians / vehicles)," "semantics (road surface and drivable area, low obstacles)," and "road structure elements (lane lines, curbs, etc.)." Detection alone or only raster semantics cannot directly support edge-to-edge cleaning paths and boundary definition. All three tasks need to share the same spatial reference (BEV) to ensure geometric consistency and reduce the risk of error accumulation and topology breakage caused by post-processing vectorization. The specific steps are as follows: S301 inputs the shared bird's-eye view feature base into the 3D target detection module, uses the bird's-eye view grid position as a query, performs feature interaction on the depth information of the image and point cloud source, obtains the target representation corresponding to each bird's-eye view position, and then regresses to obtain the 3D bounding box parameters; S302 restores the spatial resolution of the shared bird's-eye view feature base to the target scale aligned with the bird's-eye view grid. Through convolutional mapping and channel reorganization, it outputs the point-by-point semantic category probability in the discrete height dimension for each bird's-eye view grid cell. In this step, traditional detection lacks precise coupling between "object center / occupied area" and cross-modal depth information on BEV, which can easily lead to bounding box position offsets or inaccurate depth on low-profile targets. By using mesh query and depth interaction, the detection bounding box can be more consistent with the real 3D geometry, improving the localization accuracy of small targets (cones, piles of gravel, and stacks of garbage) near the curb.

[0031] BEV semantics only outputs height difference information for low-lying targets in 2D categories, which is easily lost; in sanitation scenarios, differentiation in the Z direction is particularly critical (for example, curbside and ground-level garbage have different geometric height layers). By outputting probabilities for each height layer, usable fine-grained semantic basis can be provided for subsequent cross-task consistency constraints.

[0032] S303 inputs the shared bird's-eye view feature base into the vector map detection module to obtain the map element categories and their point coordinate sequences.

[0033] S3031 sets up a set of learnable map query vectors, uses them as query terms, and models the relationships between queries in the query dimension through multi-head attention; S3032 generates a multi-scale feature pyramid from shared bird's-eye view features. In the cross-attention of each decoding layer, it predicts a set of sampling points on the multi-scale feature map for each query term, which serves as the basis for reading information from the local geometric location of the bird's-eye view. For each sampling point, S3033 uses bilinear interpolation on the corresponding scale feature map to obtain sampling features. The sampling features are weighted and summed according to the weights predicted by cross attention to obtain the aggregated features used to update the query term. The aggregated query term is residually connected with the current query term and normalized. The feedforward network nonlinear transformation is applied to the updated query term to obtain the next layer of decoded term representation. S3034 inputs the finally decoded lexical representation into the category multilayer perceptron network of the vector map detection module, and outputs the predicted map element category corresponding to each query; The S3035 inputs the finally decoded lexical representations into the coordinate regression multilayer perceptron network of the vector map detection module, and outputs the structured point coordinate sequence corresponding to each query.

[0034] In this step, topological constraints exist between multiple elements in the vector map (e.g., continuity of lane lines, connectivity of curbs, and relative positional relationships between different elements). Modeling query associations can improve the global consistency of the prediction sequence and reduce mutual interference between similar elements. Vector maps need to accurately capture local geometry and boundary orientation. Sampling-interpolation-weighted aggregation can acquire stable local information while maintaining end-to-end trainability. Different map element categories (lane lines / curb / crossroads, etc.) have differences in appearance and geometric features. Categorical MLP provides separable semantic discrimination capabilities, providing supervision signals for subsequent geometric consistency constraints.

[0035] Furthermore, the vector map detection module is trained through the following steps: Construct training samples. In each training sample, the real map consists of several ground value elements. Each ground value element contains map element category labels and a sequence of ground value point coordinates, thus obtaining a set of ground values. The real map is input into the category multilayer perceptron network and the regression multilayer perceptron network of the vector map prediction module, respectively, and the predicted map element categories and predicted point coordinate sequences are output to obtain the prediction set. The Hungarian algorithm is used to perform an optimal one-to-one matching between the prediction set and the truth set to obtain the matching relationship; For the matched predicted map element categories, they are supervised as ground truth categories; for the unmatched predicted map element categories, the classification loss is calculated and included in the total classification loss. The difference between the predicted point coordinate sequence and the true point coordinate sequence is calculated as the coordinate loss. The classification loss and coordinate loss are weighted and summed to obtain the total loss. By minimizing the total loss, the vector map prediction module is optimized through backpropagation.

[0036] In this step, vector map detection falls under the category of ensemble prediction problems. The number of elements in a single BEV map is not fixed, and the query is "placed in a fixed number of ensemble slots." Hungarian matching can avoid one-to-many missupervision and ensure stable training convergence. Lane endpoints and curb inflection points may be sparse and disproportionate in real data. Hungarian matching and classification / regression separation supervision can reduce the training bias caused by "endpoint offset leading to path planning errors," making the predicted sequence more closely match the actual topology.

[0037] S4 correlates and verifies the target object's 3D bounding box parameters, semantic category probability, map element category, and its point coordinate sequence, and outputs the verified target object's 3D bounding box parameters, semantic category probability, map element category, and its point coordinate sequence.

[0038] S401 projects the 3D bounding box of each target object onto the bird's-eye view to obtain the 3D bounding box projection area. Based on the spatial proximity relationship between the 3D bounding box projection area and the coordinate sequence of map element points, the corresponding map element is established. S402 obtains the allowed semantic set of the box from the map element category of the corresponding map element, reads the semantic category probability vector of the 3D box, calculates the sum of probabilities of all semantic categories of the 3D box in the allowed semantic set, and if the sum of probabilities is ≥ 3D box - semantic threshold, then the 3D box - semantic consistency is satisfied. In this step, within the BEV space, bounding boxes belong to "object-level geometry," while vector maps belong to "road structure-level geometry." It's necessary to achieve spatial association between the two types of outputs through projection and proximity relationships; otherwise, consistency verification is difficult. Some map element categories have strong prior correspondences with typical semantic categories. If the model output causes the bounding box category to contradict the semantic prior of its neighboring map elements, it should be considered an unreliable candidate. If the map element category is "roadside," the allowed semantic set may primarily include "roadside / low obstacle / edge-related semantics." If the semantic probability of a bounding box is higher in disallowed categories such as "drivable ground / vegetation," then that bounding box is more likely to be eliminated, thus avoiding mistaking texture noise near the roadside as cleanable space.

[0039] S403 uses the coordinate sequence of map elements to calculate the hit rate of the 3D bounding box projection area and normalizes it to obtain a geometric distribution score. If the geometric distribution score is greater than or equal to the 3D bounding box-map threshold, then the 3D bounding box-map threshold is satisfied. Figure 1 To the point of being compatible; In this step, semantic probabilities may be biased by lighting / point cloud sparsity, but geometric hits are a stronger spatial constraint. Geometric assignment scores can filter out bounding boxes that are "near the projection but do not actually correspond to the map element." For example, a bounding box might be incorrectly placed near a road edge texture, but the sequence of road edge points in the vector map does not actually fall within the bounding box's BEV projection area. In this case, the geometric assignment score will be low, and the bounding box can be removed to avoid incorrect obstacle avoidance / false boundary clearing.

[0040] S404 selects several sampling points from the point coordinate sequence of map element categories, maps each sampling point to the index position of the segmented semantic probability, and reads the semantic category probability of the point in the allowed semantic set; S405 calculates the mean of the semantic category probabilities for all sampling points. If the mean is greater than or equal to the semantic-map threshold, then the semantic-map threshold is satisfied. Figure 1 To the point of being compatible; S406 will satisfy 3D frame semantic consistency, 3D frame grounding... Figure 1 Consistency, semantics Figure 1 Output the 3D bounding box parameters, semantic category probability, map element category and its point coordinate sequence of the target object.

[0041] In this step, vector map elements provide the geometric location of road structures, but whether these structures correspond in the segmentation semantic space needs to be verified. Sampling the map point sequence and reading the segmentation probabilities can effectively assess whether "the map structure has been segmented into similar semantic regions." For example, a sweeper may encounter segmentation errors caused by temporary markings / reflections on complex road surfaces. If the map predicts "lane lines" but the segmentation semantic probability is more biased towards "curb shadows / wetlands / noise" near those lines, the mean will decrease, thus failing the consistency check and avoiding outputting incorrect lane boundaries to the planning system.

[0042] The model training in this invention employs an end-to-end multi-task joint optimization strategy. The total loss function is composed of the independent loss of each task, the cross-task correlation loss, and the robust training loss.

[0043] Among them, each loss weight The settings can be preset or dynamically adjusted based on the importance of the task and the convergence speed.

[0044] To improve the model's robustness to single sensor failure, this invention introduces a progressive modal random dropout strategy during the training phase and combines it with consistency distillation loss for joint optimization. The specific training process is as follows: (1) Modal random drop strategy: In each training batch, with probability Randomly select one of the following three forward propagation modes to execute: Normal mode (probability) ): Simultaneously retains point cloud BEV features and image BEV features, and performs standard dual-modal fusion and multi-task output.

[0045] Point cloud drop mode (probability) ): Set all BEV features of the point cloud to zero, and rely solely on the BEV features of the image for fusion and subsequent task output.

[0046] Image dropout pattern (probability) ): Set all BEV features of the image to zero, and rely only on the BEV features of the point cloud for fusion and subsequent task output.

[0047] In modality dropout mode, the dynamic gating weights of the SUF fusion module automatically reset the fusion weights of failed modalities to zero, and the model relies entirely on a single effective modality to complete the perception task.

[0048] (2) Progressive dropout probability scheduling: A progressive scheduling strategy is employed during training. Early training phase (first 30% of Epochs): This stage primarily focuses on bimodal training, enabling the model to fully learn complementary modal information and establish a stable foundation for feature representation.

[0049] Mid-training phase (30%~70% of Epoch): The modality is increased linearly from 0.1 to 0.4. More modal dropout scenarios are gradually introduced to allow the model to gradually adapt to single-modal inputs.

[0050] Late training phase (last 30% of epochs): By maintaining a value of 0.4 and randomly introducing zero values ​​in certain regions (simulating limited LiDAR field of view or localized camera overexposure), the robustness of the model to complex failure modes is further improved.

[0051] (3) Consistent distillation training: For each training sample that performs modal discarding, an additional bimodal full forward propagation is performed (without calculating gradients) to obtain the "teacher" features and output. The consistency distillation loss is then calculated. The loss is then weighted and summed with the loss from the main task of that sample before backpropagation. This mechanism encourages the model to recover near-bimodal perceptual quality from a single-modal input.

[0052] (4) Gradient accumulation and parameter update: Since the samples in the modal dropout mode are more difficult to obtain, a gradient accumulation strategy can be adopted. After accumulating several batches of gradients, a parameter update is performed to stabilize the training process.

[0053] Through the robust training strategy described above, even if the model encounters sensor failures such as camera obstruction, direct sunlight, or LiDAR rain attenuation during the inference phase, it can gracefully reduce the perception accuracy without switching models or performing additional post-processing, ensuring the safe operation of sanitation sweepers in complex working environments around the clock.

[0054] Compared to baseline methods, the method of this invention achieves synergistic gains in detection, segmentation, and structured mapping in sanitation scenarios: In terms of 3D target detection, the average accuracy of vehicle and pedestrian distances within the range of 0–50 meters is improved (see Table 1), with a more significant improvement for low-profile and small targets such as traffic cones; at the same time, the introduction of point cloud semantic segmentation task results in an overall average intersection-union ratio improvement, while maintaining high segmentation accuracy in key categories such as curbs, stone blocks / others; in terms of vector map detection, the average accuracy is improved under a 0.5-meter tolerance condition, and the lane line endpoint positioning error is controlled at a low level to meet the planning implementation requirements.

[0055] Furthermore, under the condition of joint training of three tasks, additional gains were obtained in the average accuracy of detection, the average intersection-union ratio of segmentation, and the average accuracy of map compared with independent training, verifying the effective collaboration between multiple tasks. At the same time, the present invention takes into account both model efficiency and robustness and engineering deployability: the floating-point operation per second and the number of parameters of the model are reduced, the single-frame inference time is shortened, and the number of training convergence rounds is reduced (see Table 1). Moreover, the performance degradation under sensor failure conditions is significantly reduced (see Table 1), indicating that the proposed method has more stable performance under complex environments and engineering constraints.

[0056] Table 1

[0057] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0058] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0059] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0060] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0061] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The words first, second, and third, etc., do not indicate any order. These words can be interpreted as names.

[0062] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0063] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0064] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0065] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

Claims

1. A multimodal, multi-task spatial perception method for sanitation operations, characterized in that, include: S1 acquires LiDAR data and multi-view camera image sequences, and constructs bird's-eye view representations for them respectively, obtaining point cloud bird's-eye view features and image bird's-eye view features respectively; S2 calculates the fusion weights of point cloud bird's-eye view features and image bird's-eye view features at the bird's-eye view grid positions based on the difference in semantic distribution between the two modalities and the confidence of each modality. Through dynamic gating weighted fusion, a shared bird's-eye view feature base is obtained. S3 performs target detection, semantic segmentation, and vector map detection on the shared bird's-eye view feature base, respectively, and obtains the 3D bounding box parameters, semantic category probability, map element category and its point coordinate sequence of the target; S4 correlates and verifies the target object's 3D bounding box parameters, semantic category probability, map element category, and its point coordinate sequence, and outputs the verified target object's 3D bounding box parameters, semantic category probability, map element category, and its point coordinate sequence.

2. The multimodal, multi-task spatial perception method for sanitation operations according to claim 1, characterized in that, The acquisition of LiDAR data and multi-view camera image sequences, followed by the construction of bird's-eye view representations, yields point cloud bird's-eye view features and image bird's-eye view features, including: S101 projects LiDAR data and multi-view camera image sequences onto the global coordinate system according to pose transformation parameters; S102 divides the unified coordinate system LiDAR data into a fixed grid on the bird's-eye view plane, forms columnar voxels in a columnar manner, and truncates or fills zeros on the number of points within the columnar voxels. S103 encodes columnar voxels according to the sanitation scene by height interval, forming columnar voxel feature vectors that retain the height structure. S104 scatters the columnar voxel feature vectors that retain the height structure back to the bird's-eye view to obtain the point cloud bird's-eye view features; The S105 extracts multi-scale two-dimensional features from the image from each camera's perspective. It then aligns and fuses the different scales through a feature pyramid network to obtain a multi-scale two-dimensional feature map. Finally, it performs viewpoint feature transformation to obtain the bird's-eye view features of the image.

3. The multimodal, multi-task spatial perception method for sanitation operations according to claim 2, characterized in that, The process of encoding columnar voxels according to height intervals based on the sanitation scenario to form columnar voxel feature vectors that retain height structure includes: S1031 divides the Z direction of the global coordinate system into four unequal height intervals according to the prior low target in the sanitation scene. The height intervals include: ground layer, low layer, medium layer, and high layer. Each height interval is defined by the corresponding Z coordinate threshold. S1032 For each column voxel, collect the set of points it contains. For each point within the column voxel, determine its assigned height interval based on the point's Z coordinate, and assign the point to the set of points in the corresponding height interval. S1033 If the number of points in a certain height interval corresponding to a certain column voxel is 0, then the interval-level partitioning feature of that height interval is set to the zero vector. For each height interval of the columnar voxel, S1034 calculates the geometric center of the point within the height interval based on the Z coordinate of the point within the height interval, and calculates the offset vector of each point within the height interval relative to the geometric center, and uses the offset vector as the point-level feature input of the point within the height interval. S1035 performs point-by-point mapping on the point-level features of each height interval of the column voxel, and aggregates them into a fixed-length interval-level partition feature vector by applying a symmetric aggregation function to the set of points in the height interval. S1036 concatenates the interval-level partitioning features corresponding to the four height intervals of the column voxel according to the channel dimension to obtain the column voxel feature vector that retains the height structure.

4. A multimodal, multi-task spatial perception method for sanitation operations according to claim 2, characterized in that, The process of extracting multi-scale two-dimensional features from images at each camera's viewpoint, aligning and fusing different scales through a feature pyramid network to obtain a multi-scale two-dimensional feature map, and then performing viewpoint feature transformation to obtain the image's bird's-eye view features includes: S1051 acquires a multi-scale two-dimensional feature map, performs depth branch prediction on the multi-scale two-dimensional feature map, and outputs the discrete depth probability distribution corresponding to each pixel. S1052 elevates each pixel and each depth to a set of 3D sampling points in the camera coordinate system based on the camera imaging geometry. S1053 transforms 3D points in the camera coordinate system to a unified coordinate system and uses the bird's-eye view grid definition to map the planar coordinates of 3D points to the bird's-eye view grid index. S1054 performs weighted aggregation of the features of all samples whose projections fall into the same bird's-eye view grid cell using discrete depth probabilities to obtain the bird's-eye view grid features; S1055 rearranges the bird's-eye view grid features into tensors by channel to obtain the bird's-eye view features of the image.

5. A multimodal, multi-task spatial perception method for sanitation operations according to claim 1, characterized in that, The process of calculating fusion weights based on the difference in semantic distribution between the point cloud bird's-eye view features and the image bird's-eye view features at the bird's-eye view grid positions, and obtaining a shared bird's-eye view feature base through dynamic gating weighted fusion, includes: S201 calculates the local semantic distribution of point cloud bird's-eye view features and image bird's-eye view features at the bird's-eye view grid positions, respectively, to obtain the semantic distribution probability of point cloud bird's-eye view and image bird's-eye view. S202 calculates the semantic distribution probability of the point cloud bird's-eye view and the normalized KL divergence of the semantic distribution probability of the image bird's-eye view to obtain the semantic consistency score. S203 estimates the confidence scores of two modalities for each bird's-eye view location. The semantic consistency score, the confidence score of the point cloud modality, and the category prior are input into a dynamic gating function to obtain the location-related point cloud fusion weights and image fusion weights. The calculation formula is as follows: ， ， in, Represents the point cloud modal fusion weights. express Sigmoid Activation function 、 、 This represents the learnable balance coefficient. This represents the prior biases guided by semantic categories. Indicates semantic consistency score, Represents the confidence level of the point cloud mode. Represents the image modality fusion weights; S204 performs a weighted summation of the two modal features based on the location-related point cloud fusion weights to obtain the weighted fusion features; S205 stitches together point cloud bird's-eye view features and image bird's-eye view features to obtain stitched features. It then superimposes the weighted fusion features and stitched features through residual connection. Finally, it downsamples to compress the spatial size to half of the original size to obtain fused bird's-eye view features. S206 uses the SalsaNext variant encoding network to perform deep encoding on the fused bird's-eye view features to obtain a shared bird's-eye view feature base.

6. A multimodal, multi-task spatial perception method for sanitation operations according to claim 5, characterized in that, The process of deep encoding the fused bird's-eye view features using the SalsaNext variant encoding network to obtain the shared bird's-eye view feature base includes: S2061 extracts local texture information and global semantic information from the fused bird's-eye view features through residual context blocks of the SalsaNext variant encoding network; S2062 aggregates local texture information and global semantic information based on the dilated convolution branches of the residual context block to obtain aggregated texture semantic information. The aggregated texture semantic information of each dilated convolution branch is then weighted and fused by channel attention to output weighted and fused aggregated texture semantic information. S2063 first determines the effective region and background region based on the feature response intensity. It performs full convolution and downsampling only on the weighted fusion aggregated texture semantic information with a response higher than the adaptive threshold. It compresses the low-response background region with low-resolution pooling to obtain sparse texture semantic information. S2064 performs shallow feature sampling and deep feature sampling on the sparse texture semantic information respectively, and then fuses them to obtain the skipping sparse texture semantic information; S2065 adaptively selects transposed convolution or bilinear interpolation to perform upsampling on the skipping sparse texture semantic information content, and obtains the upsampled skipping sparse texture semantic information through gated fusion. S2066 performs channel progressive compression and lightweight convolution on the upsampled sparse texture semantic information to obtain lightweight sparse texture semantic information, aligns it with the required spatial size, and obtains a shared bird's-eye view feature base.

7. A multimodal, multi-task spatial perception method for sanitation operations according to claim 1, characterized in that, The steps of performing target detection, semantic segmentation, and vector map detection on the shared bird's-eye view feature base to obtain the target's 3D bounding box parameters, semantic category probability, map element category, and its point coordinate sequence include: S301 inputs the shared bird's-eye view feature base into the 3D target detection module, uses the bird's-eye view grid position as a query, performs feature interaction on the depth information of the image and point cloud source, obtains the target representation corresponding to each bird's-eye view position, and then regresses to obtain the 3D bounding box parameters; S302 restores the spatial resolution of the shared bird's-eye view feature base to the target scale aligned with the bird's-eye view grid. Through convolutional mapping and channel reorganization, it outputs the point-by-point semantic category probability in the discrete height dimension for each bird's-eye view grid cell. S303 inputs the shared bird's-eye view feature base into the vector map detection module to obtain the map element categories and their point coordinate sequences.

8. A multimodal, multi-task spatial perception method for sanitation operations according to claim 7, characterized in that, The step of inputting the shared bird's-eye view feature base into the vector map detection module to obtain map element categories and their point coordinate sequences includes: S3031 sets up a set of learnable map query vectors, uses them as query terms, and models the relationships between queries in the query dimension through multi-head attention; S3032 generates a multi-scale feature pyramid from shared bird's-eye view features. In the cross-attention of each decoding layer, it predicts a set of sampling points on the multi-scale feature map for each query term, which serves as the basis for reading information from the local geometric location of the bird's-eye view. For each sampling point, S3033 uses bilinear interpolation on the corresponding scale feature map to obtain sampling features. The sampling features are weighted and summed according to the weights predicted by cross attention to obtain the aggregated features used to update the query term. The aggregated query term is residually connected with the current query term and normalized. The feedforward network nonlinear transformation is applied to the updated query term to obtain the next layer of decoded term representation. S3034 inputs the finally decoded lexical representation into the category multilayer perceptron network of the vector map detection module, and outputs the predicted map element category corresponding to each query; The S3035 inputs the finally decoded lexical representations into the coordinate regression multilayer perceptron network of the vector map detection module, and outputs the structured point coordinate sequence corresponding to each query.

9. A multimodal, multi-task spatial perception method for sanitation operations according to claim 7, characterized in that, The vector map detection module is trained through the following steps: Construct training samples. In each training sample, the real map consists of several ground value elements. Each ground value element contains map element category labels and a sequence of ground value point coordinates, thus obtaining a set of ground values. The real map is input into the category multilayer perceptron network and the regression multilayer perceptron network of the vector map prediction module, respectively, and the predicted map element categories and predicted point coordinate sequences are output to obtain the prediction set. The Hungarian algorithm is used to perform an optimal one-to-one matching between the prediction set and the truth set to obtain the matching relationship; For the matched predicted map element categories, they are supervised as ground truth categories; for the unmatched predicted map element categories, the classification loss is calculated and included in the total classification loss. The difference between the predicted point coordinate sequence and the true point coordinate sequence is calculated as the coordinate loss. The classification loss and coordinate loss are weighted and summed to obtain the total loss. By minimizing the total loss, the vector map prediction module is optimized through backpropagation.

10. A multimodal, multi-task spatial perception method for sanitation operations according to claim 1, characterized in that, The step of correlating and cross-validating the 3D bounding box parameters, semantic category probability, map element category, and their point coordinate sequence of the target object includes: S401 projects the 3D bounding box of each target object onto the bird's-eye view to obtain the 3D bounding box projection area. Based on the spatial proximity relationship between the 3D bounding box projection area and the coordinate sequence of map element points, the corresponding map element is established. S402 obtains the allowed semantic set of the box from the map element category of the corresponding map element, reads the semantic category probability vector of the 3D box, calculates the sum of probabilities of all semantic categories of the 3D box in the allowed semantic set, and if the sum of probabilities is ≥ 3D box - semantic threshold, then the 3D box - semantic consistency is satisfied. S403 uses the coordinate point sequence of map elements to calculate the hit calculation of the 3D bounding box projection area and normalizes it to obtain the geometric allocation score. If the geometric allocation score is greater than or equal to the 3D bounding box-map threshold, then the 3D bounding box-map consistency is satisfied. S404 selects several sampling points from the point coordinate sequence of map element categories, maps each sampling point to the index position of the segmented semantic probability, and reads the semantic category probability of the point in the allowed semantic set; S405 calculates the mean of the semantic category probability of all sampling points. If the mean is greater than or equal to the semantic-map threshold, then the semantic-map consistency is satisfied. S406 will output the 3D bounding box parameters, semantic category probabilities, map element categories and their point coordinate sequences of the target object that satisfy 3D bounding box-semantic consistency, 3D bounding box-map consistency, and semantic-map consistency.

Images