A multi-modal environment perception and post-fusion method for a sweeping robot

Abstract

The application relates to a multi-modal environment perception and post-fusion method of a sweeping robot, and relates to mobile robot environment perception. Geometric processing is first performed on depth point cloud data: voxel down-sampling, ROI cropping and outlier removal are performed on the point cloud by a CUDA parallel preprocessing unit, a ground surface is segmented by using a RANSAC algorithm with strong constraint of a gravity direction normal vector, and obstacle clustering is completed in combination with an Euclidean clustering algorithm; in parallel, two-dimensional semantic features are extracted from RGB image data by using a deep learning target detection model; the geometric and semantic information is input into a multi-modal cascade fusion and classification module, cross-modal space mapping is established, visual semantics is injected into geometric point cloud, and garbage and obstacles are accurately distinguished by using a multi-constraint cascade reasoning strategy; AED time sequence tracking and output are executed, a correlation measurement based on an aggregated Euclidean distance is introduced, real-time perception output with low delay and high robustness is realized on an embedded platform, and the problems of low obstacle detection and garbage misjudgment are solved.

Description

Technical Field

[0001] This invention relates to the field of mobile robot environmental perception technology, and in particular to a multimodal environmental perception and post-fusion method for a sweeping robot based on vision and point cloud multimodal fusion. Background Technology

[0002] In complex ground environments, robotic vacuum cleaners face a dual perception challenge: on the one hand, they need to accurately perceive the three-dimensional position of obstacles to avoid collisions; on the other hand, they need to understand the semantics of the scene and distinguish between "trash that should be cleaned" and "obstacles that should be avoided".

[0003] Currently, most mainstream robotic vacuum cleaners on the market use binocular structured light cameras to acquire depth data and segment the ground. However, single sensors have significant limitations. When faced with slender obstacles such as electrical wires, which are most easily tangled in commercial robotic vacuum cleaners, existing geometric segmentation algorithms tend to fail. Because slender obstacles appear as extremely sparse and discontinuous outliers in depth point clouds, they are often mistakenly rejected as noise by density-based statistical filters. In addition, existing obstacle avoidance algorithms often adopt a conservative strategy, treating all detected objects as obstacles and avoiding them. This causes the robotic vacuum cleaner to frequently avoid targets that should be cleaned, such as crumpled paper and fruit peels, severely affecting the coverage of the cleaning operation.

[0004] Multi-sensor fusion is an inevitable path to overcome the bottleneck of single-modal perception and improve the environmental adaptability of robotic vacuum cleaners. While feature-level fusion algorithms, such as BEVFusion, have emerged in academia and offer high accuracy, their enormous computational cost makes them unsuitable for the cost- and power-sensitive embedded computing platforms used by robotic vacuum cleaners. On the other hand, current commercial products generally employ a hard fusion strategy based on "logic gates": when any of the laser, vision, or infrared sensors triggers a threshold, the robotic vacuum cleaner decelerates and avoids obstacles. This simple strategy fails to achieve deep complementarity of multi-modal information, easily leading to misjudgments and jitter in motion decisions.

[0005] Therefore, there is an urgent need for a real-time environmental perception method that is adapted to the computing power constraints of embedded platforms, can achieve deep post-fusion of vision and point cloud, can effectively solve the problem of missing detection of low and slender obstacles, and can accurately distinguish between cleanable garbage and obstacles that need to be avoided. Summary of the Invention

[0006] The purpose of this invention is to address the challenges of existing robotic vacuum cleaners in complex ground environments, such as difficulty in simultaneously avoiding low-lying obstacles and accurately identifying specific debris, as well as the technical bottlenecks of existing end-to-end feature fusion algorithms, which suffer from excessive computational consumption and difficulty in real-time deployment on embedded platforms. This invention provides a multimodal environmental perception and post-fusion method for robotic vacuum cleaners. Based on a Compute Unified Device Architecture (CUDA) parallel acceleration architecture, this method extracts three-dimensional geometric features from raw sensor data, extracts semantic features using a visual network, injects semantic information into the geometric space through a designed post-fusion strategy, and combines multi-constraint cascaded inference and AED metric temporal tracking to achieve accurate classification and robust tracking of ground debris and obstacles.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0008] A method for multimodal environmental perception and post-fusion of a robotic vacuum cleaner includes the following five parts.

[0009] Step 1: Obtain RGB image data and depth point cloud data of the environment in which the robot vacuum cleaner is located;

[0010] Step 2: Preprocess, segment, and cluster the depth point cloud data to obtain independent non-ground point cloud clusters;

[0011] Step 3: Perform two-dimensional target detection on the RGB image data to obtain a two-dimensional bounding box containing semantic category and confidence level;

[0012] Step 4: Perform cross-modal spatial mapping and semantic injection: Construct a projection mapping from 3D space to a 2D image plane, projecting the depth point cloud data onto the 2D plane; when the projected point falls within the 2D bounding box, inject the semantic category attribute of the 2D bounding box into the corresponding point cloud;

[0013] Step 5: Perform multi-constraint cascaded reasoning and classification: Combining the geometric and physical properties of the point cloud cluster with the injected semantic category properties, the target object is determined to be "cleanable garbage" or "obstacles that need to be avoided" through a cascaded filtering strategy.

[0014] Step 6: Perform time-series tracking on the classified target objects and output the state information of the target objects in the robot vacuum's coordinate system.

[0015] In step 2, the preprocessing of the depth point cloud data employs a pipeline based on a parallel architecture of a graphics processing unit (GPU). The raw depth point cloud data stream acquired by the depth camera is defined as follows: ,in Let be the Euclidean coordinate vector of the point cloud in the camera coordinate system. To address the high latency and heavy CPU load issues of traditional serial processing architectures, a fully parallel processing pipeline based on the ComputeUnified Device Architecture (CUDA) is designed. This pipeline leverages the massively multi-instruction multiple data (SIMD) capability of GPUs to directly unpack the sensor binary stream in parallel. The pipeline comprises four sub-steps: parallel data unpacking, voxel mesh downsampling, region of interest clipping, and outlier removal.

[0016] 1) Parallel data unpacking steps: Utilize the GPU's Single Instruction Multiple Data (SIMD) feature to allocate video memory space matching the size of the sensor's original binary stream; use parallel threads to directly perform bitwise operations on the binary data to decode the depth value into three-dimensional coordinates, avoiding the serialization and unpacking overhead on the CPU side;

[0017] 2) Voxel Mesh Downsampling Sub-step: To reduce data redundancy and preserve the environmental topology, a voxel mesh filter is used. The three-dimensional space is divided into cubic units of equal size. To address the inefficiency of using hash tables or octree indexes in traditional CPU algorithms, a sorting-reduction-based parallel strategy is adopted in the CUDA kernel. This strategy transforms the spatial clustering problem into efficient continuous sorting and key-value reduction operations in memory. Specifically, the voxel mesh index of each point is calculated in parallel according to the preset voxel size as the key, and the coordinate index of the point is used as the value. Parallel primitives are used to perform key-based fast sorting on all key-value pairs so that spatial points belonging to the same voxel are arranged continuously in memory. A reduction operation is performed on the set of points with the same key to calculate and retain the geometric center point of the voxel.

[0018] 3) Region of Interest (ROI) clipping sub-step: Based on the physical shape and kinematic constraints of the robot vacuum cleaner, set the clipping boundary in three-dimensional space; use memory pooling technology and GPU atomic counters to complete the removal of invalid region point clouds and compact rearrangement of valid data through a single kernel function call;

[0019] 4) Outlier Removal Sub-step: To address sensor noise caused by depth value jumps and ambient light interference from the depth camera, a two-stage parallel cascaded filtering strategy is designed:

[0020] First-level statistical filtering: Identify outliers based on the statistical characteristics of local point clouds, calculate the average distance of each point within a preset number of nearest neighbors in parallel, and remove outliers whose average distance exceeds a preset multiple of the global standard deviation;

[0021] Second-level radius filtering: Identify isolated noise points based on the density characteristics of local point clouds, check the number of neighbors of each point within a preset radius, and remove isolated points whose number of neighbors is less than the minimum cluster size threshold.

[0022] In step 2, the ground segmentation employs a geometrically constrained Random Sample Consensus (RANSAC) algorithm. Addressing the technical challenge of the standard RANSAC algorithm misclassifying large vertical planes such as walls and cabinet sides as "pseudo-ground" surfaces in complex indoor environments, a strong constraint based on gravity-direction alignment of the normal vector is introduced. The specific steps are as follows:

[0023] 1) When iteratively sampling candidate plane models, calculate the candidate plane normal vector. gravity axis of the global coordinate system The included angle;

[0024] 2) Construct a direction constraint filter to retain only those that satisfy the constraint. Candidate planar models, where Preset angle tolerance;

[0025] 3) Select the model with the largest number of interior points as the optimal ground parameters, and divide the point cloud set into ground point set and non-ground point set.

[0026] Furthermore, the ground segmentation also includes an interior point determination method based on algebraic distance:

[0027] For candidate planes that satisfy the constraints Calculate any point in the point cloud to candidate plane algebraic distance :

[0028]

[0029] in, Represents any three-dimensional point in a point cloud. Point X-axis coordinates Point Y-axis coordinate, Point Z-axis coordinate, Indicates the first One candidate ground plane model, Representing candidate planes The x-axis coefficients of the equation, Representing candidate planes The Y-axis coefficients of the equation, Representing candidate planes Z-axis coefficients of the equation Representing candidate planes The constant term of the equation, Representing candidate planes The magnitude of the normal vector, if Less than the preset distance threshold If so, then mark that point as an interior point.

[0030] In step 2, to achieve instance-level perception of ground objects, the non-ground point set needs to be segmented into object clusters with independent physical meaning. A density-based Euclidean clustering algorithm is used, combined with a K-dimensional tree data structure to optimize search efficiency. Specifically, this includes:

[0031] 1) Construct a K-Dimensional Tree (KD-Tree) data structure to spatially index non-ground point sets;

[0032] 2) A recursive region growth strategy is adopted. For unvisited points, the set of neighboring points within the search radius of the Euclidean clustering neighborhood is searched. If neighboring points exist, they are assigned to the same cluster and the cluster is expanded recursively.

[0033] 3) Introduce a minimum cluster size constraint, remove noise clusters with fewer points than the constraint, and output a point cloud cluster set.

[0034] In step 4, the execution of cross-modal space mapping and semantic injection includes a semantic priority enhancement strategy for predefined elongated linear obstacles:

[0035] 1) When the category of the 2D target detection is a preset thin, elongated linear obstacle category, cross-modal semantic injection is performed;

[0036] 2) Any 3D point that falls within the bounding box projection range of this elongated linear obstacle category, regardless of whether its geometric density meets the clustering requirements, is forcibly assigned an obstacle semantic label and is prioritized for recall as an obstacle candidate cluster.

[0037] The elongated linear obstacles include, but are not limited to: wires, data cables, ropes, ribbons, hair strands, and thin, sparse obstacles that are easily mistakenly removed by point cloud filtering.

[0038] In step 5, the multi-constraint cascaded reasoning uses a joint decision function, and the specific decision logic is as follows:

[0039] Level 1 determination: If the proportion of points with semantic labels in a point cloud cluster exceeds the preset semantic confidence threshold, and the physical size of the point cloud cluster is within the preset limit of the robot vacuum cleaner's suction port, then it is determined to be "cleanable garbage".

[0040] Second-level judgment: If the first-level judgment conditions are not met, but the maximum height and maximum length of the cloud cluster are both less than the set ground attachment threshold, then it is judged as "cleanable garbage".

[0041] Level 3 judgment: If none of the above conditions are met, the judgment is "obstacles need to be avoided".

[0042] In step 6, the temporal tracking employs an association metric strategy based on Aggregated Euclidean Distance (AED) to calculate the difference between the predicted tracking box and the actual observed box; and the temporal tracking method based on AED association metric measures... The calculation formula is:

[0043]

[0044] in, This represents the number of corner points of the bounding box. The coordinate vector of the corner point. The center point coordinate vector, superscript Indicates the current sensor detection value, superscript This represents the Kalman filter prediction value.

[0045] Furthermore, the time-series tracking employs a greedy nearest neighbor association strategy for association, prioritizing associations. The prediction-observation pair with the smallest value.

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0047] This invention overcomes the computational bottleneck of 3D point cloud processing through GPU-based parallel acceleration, achieving high-frame-rate real-time perception on embedded platforms. By introducing a gravity-constrained RANSAC algorithm and a cross-modal semantic injection strategy, it effectively solves the problem of traditional geometric methods struggling to perceive small obstacles (such as wires), avoiding missed detections. Employing a decision-level post-fusion architecture, it first extracts geometric and semantic features separately, then performs spatial mapping and semantic injection, avoiding the huge computational overhead of feature-level fusion and overcoming the insufficient information complementarity of simple logic gate fusion, achieving high-precision multimodal perception on low-computing-power embedded platforms. Through a multi-constraint cascaded inference mechanism, combining geometric dimensions and semantic information, it achieves accurate classification of cleanable debris and obstacles to be avoided. Through time-series tracking based on AED metrics, it comprehensively utilizes the position and shape features of objects to solve the target loss problem in cases of rapid turning or non-overlapping (IoU=0) by the robot vacuum, ensuring the robustness and stability of perception. Attached Figure Description

[0048] Figure 1 This is an execution flowchart of a multimodal environmental perception and post-fusion method for a sweeping robot provided in an embodiment of the present invention.

[0049] Figure 2This is a flowchart of point cloud preprocessing, segmentation, and clustering results. (a) is an RGB image; (b) is the original point cloud; (c) is the preprocessed point cloud; (d) is the segmented ground point cloud; (e) is the segmented non-ground point cloud; and (f) shows the clustering results.

[0050] Figure 3 This is a schematic diagram of the sensor's field of view distribution and coordinate system.

[0051] Figure 4 These are comparison images of the perception effects for specific obstacles (wires). Among them, (a) shows the wires detected by YOLO; (b) shows the point cloud results without fusion; and (c) shows the point cloud results of this invention.

[0052] Figure 5 This is a schematic diagram of multi-constraint cascaded reasoning and classification logic. Detailed Implementation

[0053] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] This invention achieves high-speed point cloud preprocessing based on CUDA parallel acceleration, improves ground segmentation accuracy by using gravity-constrained RANSAC, solves the problem of missing detection of slender obstacles through cross-modal semantic injection, achieves accurate classification of garbage / obstacles through multi-constraint cascaded inference, and achieves robust temporal tracking by aggregating Euclidean distance using AED. It realizes high real-time performance and high robustness of environmental perception output on a low-power embedded platform.

[0055] The multimodal environment perception and post-fusion method for a robotic vacuum cleaner, as described in this embodiment of the invention, comprises the following five parts:

[0056] (1) Parallel preprocessing and feature extraction of point clouds based on CUDA acceleration

[0057] Define the raw 3D point cloud data stream acquired by the depth camera as ,in This represents the Euclidean coordinate vector of the point cloud in the camera coordinate system. Addressing the high latency and heavy CPU load issues of traditional serial processing architectures, this invention designs a fully parallel processing pipeline based on the CUDA architecture, utilizing the large-scale SIMD (Single Instruction Multiple Data) characteristics of the GPU to directly unpack the sensor binary stream in parallel. This step specifically includes the following three parallel processing sub-modules:

[0058] 1) Parallel voxel downsampling based on sorting-reduction strategy

[0059] To reduce data redundancy and preserve the environmental topology, a voxel mesh filter is used; the three-dimensional space is divided into cubic units of equal size (set as follows in this embodiment). ).

[0060] To address the inefficiency of traditional CPU algorithms using hash tables or octree indexes, this invention innovatively employs a sorting-reduction parallel strategy based on the Thrust library within the CUDA kernel. This strategy transforms the spatial clustering problem into efficient memory-based contiguous sorting and key-value reduction operations: First, voxel indices are calculated based on the spatial coordinates of points, and all points are sorted in parallel to ensure that points within the same voxel are stored contiguously in memory; then, a reduction operation is performed, calculating the geometric center of all points within each voxel as the retained point. :

[0061]

[0062] in, This represents the representative points retained after voxelization. Represents any three-dimensional point in a point cloud. Point Any point in the neighborhood set of the voxel, Point The set of neighborhood points of the voxel. This represents the value of the independent variable that minimizes the objective function. This represents the L2 norm of a vector (i.e., the Euclidean distance).

[0063] 2) Parallel cropping of regions of interest based on a zero-copy pipeline

[0064] Based on the physical dimensions and kinematic constraints of the robotic vacuum cleaner, a CropBox filter was constructed to remove invalid background data; the cropping range was set to retain data along the Z-axis height direction. (Slightly higher than the fuselage height to allow for a safety margin), X and Y axes are reserved. (Approximately twice the width of the chassis); In terms of implementation, a zero-copy, high-throughput data processing pipeline is constructed by combining memory pooling technology and atomic operations; GPU threads determine in parallel whether the coordinates of each point are within the bounding box of the Region of Interest (ROI) set above, and invalid points can be marked and removed with only one memory read and write, directly outputting point cloud data within the effective workspace, reducing the data scale of subsequent processing.

[0065] 3) Outlier removal through both statistical and radius methods

[0066] To address sensor noise caused by depth value jumps and ambient light interference from depth cameras, a two-stage parallel outlier removal strategy was designed:

[0067] Level 1: Statistical outlier removal. Outliers are identified based on the statistical characteristics of the local point cloud; for each point... Parallel computing of its and The average distance between the nearest neighbors Calculate the global distance mean. with standard deviation ,like (in If the threshold is a multiple of the standard deviation, then it is identified as an outlier and removed.

[0068] Level 2: Radius Outlier Removal. Identify isolated noise points based on local density characteristics; remove outliers within a given radius. Internally count the number of neighbors for each point ,like If the isolated point is not found, it is considered an invalid isolated point and removed. Through the above dual filtering, non-physical noise points caused by object edge diffraction and highly reflective surfaces are effectively filtered out, providing a high-quality data foundation for subsequent ground segmentation.

[0069] (2) Ground segmentation and non-ground point cloud clustering based on geometrically constrained RANSAC

[0070] Although the preprocessed point cloud data removes most of the noise, it still contains a mixture of ground and non-ground point cloud information. In order to accurately extract potential object instances from three-dimensional space, this invention designs a serial segmentation and clustering process, which specifically includes the following two steps:

[0071] 1) RANSAC ground segmentation based on strong constraints of gravity alignment normal vector

[0072] To address the technical challenge of the standard RANSAC algorithm misclassifying large vertical planes such as walls and cabinet sides as "pseudo-ground surfaces" in complex indoor environments, this invention introduces a strong constraint mechanism based on the normal vector of physical gravity. First, it assumes that the local ground conforms to a planar mathematical model:

[0073]

[0074] in, Let be the three-dimensional spatial coordinates of any point on the plane. Let be the unit normal vector of the plane. Let be the distance from the origin to the plane; during the RANSAC iterative sampling phase, a direction constraint filter is constructed: considering the physical characteristics of the mobile robot traveling on a horizontal surface, the normal vector of the effective road surface should be strictly parallel to the gravity direction of the global coordinate system (i.e., axis Therefore, for any candidate plane model generated by random sampling, its normal vector is first calculated. and The included angle, and then the following filtering is performed:

[0075]

[0076] in, A preset angle tolerance is used; any candidate plane that does not meet this angle constraint (such as a perpendicular wall) is directly eliminated during the model generation stage, and no further interior point calculations are performed, thereby greatly reducing the parameter search space and eliminating geometric ambiguity; for candidate planes that meet the constraints... Calculate all points in the point cloud algebraic distance to the plane :

[0077]

[0078] If the distance is less than the preset distance threshold If a point has a large number of inliers, it is marked as an interior point. Finally, the model with the largest number of interior points is selected as the optimal ground parameter, and the original point cloud set is... Precisely divided into ground point sets Non-ground point set In this embodiment, the preset distance threshold The range is 0.01m to 0.05m.

[0079] 2) KD-Tree-accelerated Euclidean distance point cloud clustering

[0080] To achieve instance-level perception of ground objects, it is necessary to combine non-ground point sets. The data is segmented into clusters of objects with independent physical meaning. This invention employs a density-based Euclidean clustering algorithm, combined with a K-dimensional tree data structure to optimize search efficiency. First, for non-ground point sets... Constructing a KD-Tree significantly reduces the time complexity of spatial neighborhood search for large-scale point clouds; the clustering process employs a recursive region growth strategy: for any unvisited point... The KD-Tree method can be used to quickly retrieve the search radius of a node within a Euclidean cluster neighborhood. All neighborhood points within (cluster tolerance) :

[0081]

[0082] If neighboring points If it exists, then it will be assigned to the current cluster. and with The recursive search continues for new seed points until all points within the connected region have been visited; finally, to suppress residual outliers (such as suspended dust or sensor edge noise), a minimum cluster size constraint is introduced. The system iterates through all generated clusters and retains only those clusters that meet the point density requirements.

[0083]

[0084] The final output is a series of independent non-terrestrial point cloud clusters. This serves as the geometric input for subsequent multimodal fusion and classification.

[0085] (3) Visual inspection module

[0086] To endow robotic vacuum cleaners with high-level semantic understanding of complex ground environments and address the semantic ambiguity problem caused by the inability of 3D geometric sensors to distinguish between "garbage that should be cleaned" and "obstacles that should be avoided," this invention constructs a 2D visual perception module based on a deep convolutional neural network. This module operates independently of the point cloud processing pipeline and is responsible for extracting target category and 2D spatial positioning information from RGB image streams in real time. The specific implementation steps are as follows:

[0087] 1) Construction and deployment of a lightweight single-stage detection model

[0088] Considering the limitations of computing power and power consumption of the embedded computing platform of the robotic vacuum cleaner, this invention abandons the two-stage detector with huge computational overhead and selects the single-stage detection model YOLOv11 (You Only Look Once v11) as the visual perception network. YOLOv11 adopts an end-to-end regression detection paradigm. By optimizing the backbone network structure and feature pyramid network, it significantly reduces the number of parameters and floating-point operations while maintaining high accuracy.

[0089] 2) Dataset Construction

[0090] To address the lack of specific cleaning scenario semantics in general datasets (such as distinguishing between "paper balls" and "stones"), this invention constructs a hybrid heterogeneous dataset containing multi-source data. This dataset consists of two parts: real-world scene images collected by a depth camera mounted on a robotic vacuum cleaner under various typical indoor and outdoor lighting conditions, different viewing angles, and occlusion conditions; and an expansion dataset made from an open-source road garbage dataset. Finally, a dedicated dataset containing approximately 4,000 finely annotated images is constructed, covering complex ground backgrounds and diverse garbage / obstacle morphologies, enhancing the model's generalization ability in real-world working environments.

[0091] 3) Multi-target robust tracking ensemble based on BoT-SORT

[0092] To overcome the problems of flickering, missed detections, and frequent target ID switching in single-frame detection results, this invention deeply integrates the YOLOv11 detector with the BoT-SORT (Bag of Tricks for Simple Online and Realtime Tracking) multi-target tracking algorithm. This tracking algorithm uses the high-precision positioning information of the detection box to initialize the target state and introduces Kalman filtering to construct a motion model to predict the target's position in the next frame. At the same time, for the perspective changes during the robot's movement, camera motion compensation technology is used to correct the background flow interference caused by the camera's own motion. By associating and matching the detection results of the current frame with the predicted trajectory, robust tracking in dynamic and complex environments is achieved, ensuring the uniqueness and continuity of the target's identity in the time dimension.

[0093] (4) Cross-modal space mapping and multi-constraint cascaded reasoning classification

[0094] To address the shortcomings of geometric perception lacking semantics and visual perception lacking spatial measurement, this invention constructs a post-fusion framework that tightly couples vision and geometry; it utilizes the YOLOv11 network to detect 2D images and outputs a set of bounding boxes. Establish a projection mapping function from a three-dimensional manifold to a two-dimensional image plane. :

[0095]

[0096] in The camera intrinsic parameter matrix is ​​used to project 3D points onto the pixel plane; if a point falls within the visual detection bounding box... Within this, visual semantic labels are "injected" into the 3D point; based on this, a multi-constraint cascaded inference function is designed. Used to accurately distinguish between "cleanable trash" and "obstacles that need to be avoided":

[0097]

[0098] This formula contains two layers of decision logic:

[0099] Semantic and physical dual confirmation: the percentage of points with semantic labels within a point cloud cluster. Exceeding the semantic confidence threshold And the physical size of the target The enclosed box allowed by the robot vacuum cleaner's suction inlet. Within the designated area, the area is identified as cleanable garbage.

[0100] Geometric Recall Mechanism: If there are no semantic labels (e.g., missed detections), but the maximum height of the point cloud cluster is... and length All are less than the ground adhesion threshold. This mainly targets small objects, which are also identified as cleanable trash; except for the above cases, all other clusters are identified as obstacles, thus correcting visual false detection and geometric ambiguity problems.

[0101] (5) Robust timing tracing based on AED metric

[0102] To address the problem that rapid turning of a robotic vacuum cleaner results in extremely low or even zero Intersection over Union (IoU) ratios between consecutive frames, this invention proposes an association metric strategy based on Aggregated Euclidean Distance (AED).

[0103] A motion model is constructed using Kalman filtering to calculate the AED distance between the detection box and the prediction box at the current time. It replaces traditional center distance or IoU metrics:

[0104]

[0105] in and These represent the detection box and the tracking prediction box, respectively. The coordinates of the corner points (projected onto the XY plane) ), and Represents the center point coordinates; this metric integrates the position information of the four corners and the center of the object, and uses a greedy nearest neighbor strategy for association. It can still maintain an effective gradient in non-overlapping areas of the target, which significantly reduces the ID switching rate.

[0106] The following are specific examples.

[0107] See Figure 1 This invention provides a multimodal environmental perception and post-fusion method for a robotic vacuum cleaner, which runs on an embedded computing platform (such as NVIDIA Jetson Orin Nano) and acquires environmental data through a depth camera (such as Orbbec Gemini 336L). The method includes the following steps:

[0108] Step 1: Acquire the raw point cloud data and RGB image data collected by the depth camera, and perform parallel preprocessing on the raw point cloud data based on the GPU parallel architecture to obtain the preprocessed point cloud.

[0109] In this embodiment, the overall point cloud preprocessing and segmentation clustering process is as follows: Figure 2 As shown. Considering the limited computing power of embedded platform CPUs, step 1 adopts a CUDA architecture to design a fully parallel acceleration process, specifically including:

[0110] Step 11, Parallel Unpacking: Utilizing the GPU's large-scale Single Instruction Multiple Data (SIMD) feature, the raw binary data stream output by the depth camera is directly unpacked in parallel, eliminating the serialization overhead of serial unpacking on the CPU side.

[0111] Step 12, Parallel Voxel Downsampling: Abandoning the inefficient hash tables or octree indexes in traditional CPU algorithms, we innovatively adopt a sorting-reduction parallel strategy based on the Thrust library.

[0112] First, the three-dimensional space is divided into areas of size . voxel grid;

[0113] Secondly, the three-dimensional coordinates of the point cloud are mapped to one-dimensional linear index keys, and the keys are continuously sorted in the video memory so that points belonging to the same voxel are physically adjacent in memory.

[0114] Finally, a reduction operation is performed to calculate the geometric center of all points within each voxel, which serves as the representative point of that voxel. This method significantly improves memory access locality and reduces downsampling time.

[0115] Step 13, Region Cropping and Filtering: Implement the CropBox filter using memory pooling technology and atomic operations to crop the region of interest (ROI); based on the physical shape and kinematic constraints of the robot vacuum cleaner, the ROI range is set as follows: Z-axis height less than 0.5m (slightly higher than the body), X-axis and Y-axis range of 1.2m (approximately twice the width of the body).

[0116] Step 14: Perform statistical outlier removal filtering to remove isolated outliers caused by sensor noise and output a high-quality preprocessed point cloud.

[0117] Step 2: Perform ground segmentation and clustering based on geometric constraints on the preprocessed point cloud to obtain ground point cloud and independent non-ground point cloud clusters; in this embodiment, step 2 specifically includes:

[0118] Step 21, Construct a gravity direction constraint filter: such as Figure 3 As shown, the body coordinate system and global coordinate system of the sweeping robot are established; to address the problem that the traditional RANSAC algorithm easily misclassifies the wall as the ground, a physical prior is introduced; it is assumed that the normal vector of the effective road surface should be parallel to the direction of gravity (i.e., the Z-axis of the global coordinate system). Construct the following directional constraints:

[0119]

[0120] in, Let be the normal vector of the candidate plane. In this embodiment, the preset angle tolerance is... The range is 5° to 15°.

[0121] Step 22: Perform restricted RANSAC ground segmentation: During the RANSAC iteration process, only candidate planar models that satisfy the above directional constraints are retained for interior point statistics; the model with the most interior points is selected as the ground model, and the point cloud set is divided into ground point sets. Non-ground point set .

[0122] Step 23, Euclidean distance clustering: for non-ground point sets Construct a KD-tree to accelerate the index and execute a density-based Euclidean clustering algorithm.

[0123] For any unvisited point Search its Euclidean clustering neighborhood radius (Clustering tolerance, e.g., 0.03m) Neighborhood point set; recursively expand the neighborhood until all connected points are absorbed into the current cluster; apply minimum cluster size constraint. After filtering out noise clusters, the final output is a series of independent non-terrestrial point cloud cluster sets. .

[0124] Step 3: Perform object detection inference on the RGB image data to obtain visual detection results including semantic labels and two-dimensional bounding boxes.

[0125] In this embodiment, step 3 specifically involves: inputting the synchronously acquired RGB image into a pre-trained YOLOv11 lightweight object detection model; the model outputs a set of region of interest detection results. Each result It includes two-dimensional bounding box coordinates, semantic categories (such as "bottle", "cardboard box", "wire", etc.) and confidence level.

[0126] Step 4: Construct cross-modal spatial mapping relationships, inject semantic information from visual detection results into the corresponding non-ground point cloud clusters, and perform forced semantic injection for preset special obstacle categories.

[0127] In this embodiment, step 4 aims to address the semantic gaps in geometry perception, specifically including:

[0128] Step 41, Spatial Projection Mapping: Based on Pinhole Camera Model and Camera Intrinsic Matrix Establish a projection function from a 3D point cloud to a 2D image plane. :

[0129]

[0130] in, Indicates the projection scaling factor. This represents the pixel coordinates of a 3D point projected onto a 2D image plane. Represents the camera intrinsic parameter matrix. This represents the three-dimensional spatial coordinates of a point in a point cloud cluster in the camera coordinate system. Representing three-dimensional real space, It represents a two-dimensional real number space.

[0131] Step 42, Semantic Injection and Association: Traverse all non-ground point cloud clusters and project the points they contain onto the image plane; if the projected point falls within a visual detection box... Within a pixel range, the semantic category of the detection box is determined. and confidence level "Injected" into the point cloud cluster to establish geometric-semantic association.

[0132] Step 43, Forced Recall of Special Obstacles (Wires): Traditional geometric algorithms often filter out small and low-lying special obstacles such as wires as noise due to the sparse point cloud; this invention implements a forced injection strategy:

[0133] When the visual inspection results include the "wire" category, the system ignores the minimum cluster density constraint of the point cloud; it directly backtracks to the preprocessed original point cloud and forcibly recalls all discrete 3D points whose projections fall within the "wire" detection box; it aggregates these discrete points into a new point cloud cluster and assigns a high-priority "obstacle" label to ensure that small obstacles are not missed.

[0134] Step 5: Based on the injected semantic information and the geometric and physical properties of the point cloud clusters, perform multi-constraint cascade reasoning and classification to classify the non-ground point cloud clusters as either cleanable debris or obstacles that need to be avoided.

[0135] In this embodiment, step 5 employs a cascaded criterion. For each non-ground point cloud cluster to be classified The classification process, specifically the cascading reasoning logic flow, is as follows: Figure 5 As shown, the detailed steps are as follows:

[0136] Step 51, First-level judgment (special prior): Check whether the point cloud cluster contains the special obstacle (such as wire) semantic tag injected in step 43; if so, regardless of its size, directly determine it as "obstacle to be avoided" to ensure the safety of the robot vacuum cleaner.

[0137] Step 52, Second-level judgment (semantic + geometric constraints): If the point cloud cluster contains ordinary semantic labels (such as bottle, paper ball), and the semantic confidence is greater than the semantic confidence threshold. Then, its three-dimensional physical dimensions will be further examined.

[0138] Set the cleaning suction inlet size limit of the robot vacuum cleaner to ;

[0139] If the 3D bounding box size of the point cloud cluster is smaller than the limit, it is determined to be "cleanable debris"; if the size exceeds the limit and cannot be cleaned and sucked in by the robot vacuum, it is determined to be "obstacles need to be avoided".

[0140] Step 53, Level 3 Decision (Pure Geometric Recall): If the point cloud cluster does not match any visual semantics (such as visual missed detection or unknown objects), recall is performed based on geometric and physical attributes.

[0141] Maximum height of checkpoint cloud clusters and maximum length ;

[0142] If satisfied and ( If a ground attachment threshold is set (e.g., height less than 5cm), the object is considered a small piece of debris on the ground and is classified as "cleanable trash".

[0143] Step 54, Default Judgment: If none of the above conditions are met, the point cloud cluster is judged as "needing to avoid obstacles" by default.

[0144] Step 6: Based on the classification results, use a time-series tracking strategy based on the aggregated Euclidean distance (AED) metric to obtain the continuous state estimation results of the target.

[0145] In this embodiment, step 6 addresses the issue of non-overlapping frames (IoU=0) caused by the robot vacuum's rapid turning, by designing a robust tracking method:

[0146] Step 61, State Prediction: Construct a constant velocity motion model using Kalman filtering; based on the target's state (position, velocity) at the previous moment, calculate the theoretical position of the target at the current moment and generate a "predictive tracking box".

[0147] Step 62, AED Association Metric Calculation: Calculate the Aggregated Euclidean Distance (AED) between the "predicted tracking box" and the "actual observation box" detected by vision in the current frame. The formula is as follows:

[0148]

[0149] in, This represents the aggregated Euclidean distance metric. In this embodiment, the number of bounding box corner points is indicated. This represents the four corner points of the rectangle, where i represents the index of the corner point. This represents the coordinate vector of the i-th corner point of the bounding box detected by the current sensor. This represents the coordinate vector of the i-th corner point of the bounding box predicted by the Kalman filter. The L2 norm of a vector (i.e., Euclidean distance) is used to represent the vector. This represents the coordinate vector of the center point of the bounding box detected by the current sensor. The superscript represents the coordinate vector of the center point of the bounding box predicted by the Kalman filter. Indicates the current sensor detection value, superscript This represents the Kalman filter prediction; this metric integrates information about the object's position and shape, making it more discriminative than simply the distance to the center point.

[0150] Step 63, Data Association and Update: Employ a greedy nearest neighbor strategy, prioritizing data association. Find the prediction-observation pair with the smallest value; after successful association, update the state of the Kalman filter using the observation value, and output the smoothed target location and class ID.

[0151] To verify the effectiveness of the method of the present invention, the following experimental platform was built for testing:

[0152] Hardware platform: Robot vacuum cleaner chassis + NVIDIA Jetson Orin Nano (8GB) + Orbbec Gemini336L depth camera.

[0153] Software environment: Ubuntu 22.04 LTS + ROS 2 Humble + CUDA 12.2.

[0154] 1. Real-time performance verification: The performance of pure CPU processing and the GPU acceleration solution of this invention are compared on the above platform.

[0155] CPU benchmark: The average end-to-end latency is 102.3ms, and the frame rate is only about 9.8 FPS, which cannot meet the requirements of high-speed obstacle avoidance.

[0156] The solution of this invention: thanks to the parallel optimization in step 1, the end-to-end latency is reduced to 32.7ms, the overall system frame rate is stabilized at 30 FPS, and the energy efficiency ratio (FPS / W) is improved by more than 3 times, meeting the real-time requirements.

[0157] 2. Accuracy and robustness verification: Testing was conducted in a complex indoor environment containing tables, chairs, cardboard boxes, aluminum cans, and thin wires.

[0158] Special obstacle detection: such as Figure 4 As shown, traditional geometric methods are ineffective in processing thin wires (Figure 4 In case (a) of the point cloud, the sparse point cloud leads to a complete missed detection. Figure 4 (b)); The present invention successfully recovered the three-dimensional point cloud of the wire and marked it as an obstacle through the "forced semantic injection" strategy in step 4. Figure 4 (c) in the text effectively avoids the risk of the robot vacuum cleaner getting caught in electrical wires.

[0159] Waste sorting accuracy: Through the cascaded sorting logic in step 5, the system can accurately distinguish between "cleanable waste" and "obstacles that need to be avoided"; experimental data shows that compared with methods that rely solely on geometric information, this invention improves the F1 score of waste identification from 0.62 to 0.88.

[0160] Tracking stability: In scenarios where the robot vacuum cleaner rotates rapidly, traditional IoU-based tracking algorithms frequently lose the target (ID switching rate of 6.2 times / min); after adopting AED measurement, this invention can maintain the continuity of the target ID even when the detection boxes of consecutive frames do not overlap, reducing the ID switching rate to 3.5 times / min and making the trajectory smoother.

[0161] The main innovation of this invention is as follows:

[0162] This invention employs an architecture of separate processing followed by decision-level fusion, avoiding the excessive computational power required for feature-level fusion that precludes embedded deployment, and overcoming the insufficient information complementarity of traditional single-modal or simple logic gate fusion. This enables high-precision environmental perception on low-computing-power embedded platforms. Through parallel data unpacking, sorting-reduction voxel downsampling, parallel region of interest pruning, and two-level parallel filtering (statistical and radius), this invention achieves high-speed point cloud preprocessing, significantly reducing end-to-end latency and meeting the real-time obstacle avoidance requirements of robotic vacuum cleaners. By adding an angle constraint between the normal vector and the gravity axis during plane fitting, the invention eliminates the problem of vertical planes such as walls and cabinets being misidentified as the ground, greatly improving the robustness and accuracy of ground segmentation in complex indoor environments. Addressing the issue of sparse point clouds such as wires and ropes being easily filtered out, this invention forcibly recalls 3D points within the projection area through visual semantic detection results, preserving obstacle information without relying on geometric density, fundamentally solving the problem of missing detection of slender obstacles. By combining semantic confidence, physical size, and geometric attachment characteristics for hierarchical judgment, this invention achieves accurate differentiation between cleanable debris and obstacles to be avoided, solving the problem of low cleaning coverage caused by traditional obstacle avoidance algorithms. This invention comprehensively utilizes the joint distance metric of the corner points and center point of the detection box, replacing traditional IoU or center distance, enabling stable tracking even when the robot is turning rapidly and the target does not overlap between consecutive frames, reducing ID switching rate and improving perception continuity.

[0163] In summary, the multimodal environmental perception and post-fusion method for sweeping robots proposed in this invention solves the computing power bottleneck of embedded platforms through hardware-software co-design (CUDA acceleration), and solves the problems of missed detection of low obstacles and misjudgment of garbage through tightly coupled vision-geometry logic (semantic injection, cascade classification), which has extremely high engineering application value.

[0164] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for multimodal environmental perception and post-fusion of a robotic vacuum cleaner, characterized in that, Includes the following steps: Step 1: Obtain RGB image data and depth point cloud data of the environment in which the robot vacuum cleaner is located; Step 2: Preprocess, segment, and cluster the depth point cloud data to obtain independent non-ground point cloud clusters; Step 3: Perform two-dimensional target detection on the RGB image data to obtain a two-dimensional bounding box containing semantic category and confidence level; Step 4: Perform cross-modal spatial mapping and semantic injection: Construct a projection mapping from 3D space to a 2D image plane, projecting the depth point cloud data onto the 2D plane; when the projected point falls within the 2D bounding box, inject the semantic category attribute of the 2D bounding box into the corresponding point cloud; Step 5: Perform multi-constraint cascaded reasoning and classification: Combining the geometric and physical properties of the point cloud cluster with the injected semantic category properties, the target object is determined to be cleanable garbage or an obstacle that needs to be avoided through a cascaded filtering strategy. Step 6: Perform time-series tracking on the classified target objects and output the state information of the target objects in the robot vacuum's coordinate system.

2. The method for multimodal environmental perception and post-fusion of a sweeping robot according to claim 1, characterized in that, In step 2, the preprocessing of the deep point cloud data adopts a pipeline based on GPU parallel architecture. This pipeline includes four sub-steps: parallel data unpacking, voxel mesh downsampling, region of interest pruning, and outlier removal.

3. The method for multimodal environmental perception and post-fusion of a sweeping robot according to claim 2, characterized in that, The GPU-based parallel architecture pipeline specifically includes: 1) Parallel data unpacking: Utilize the single instruction multiple data feature of the GPU to allocate video memory space that matches the size of the sensor's original binary stream; use parallel threads to directly perform bitwise operations on the binary data to decode the depth value into three-dimensional coordinates, avoiding the serialization and unpacking overhead on the CPU side. 2) Voxel mesh downsampling: A parallel strategy based on sorting-reduction is adopted. Specifically, the voxel mesh index to which each point belongs is calculated in parallel according to the preset voxel size as the key and the coordinate index of the point is used as the value. Parallel primitives are used to perform key-based fast sorting on all key-value pairs so that spatial points belonging to the same voxel are arranged continuously in the video memory. Then, a reduction operation is performed on the set of points with the same key to calculate and retain the geometric center point of the voxel. 3) Region of Interest (ROI) Clipping: Combining the physical shape and kinematic constraints of the robot vacuum cleaner, a three-dimensional clipping boundary is set; using memory pooling technology and GPU atomic counters, invalid region point clouds are removed and valid data is compactly rearranged through a single kernel function call; 4) Outlier removal: A two-stage cascaded parallel filtering strategy is employed. First-level statistical filtering: Based on the statistical characteristics of local point clouds, the average distance of each point within a preset number of nearest neighbors is calculated in parallel, and outliers with average distances exceeding a preset multiple of the global standard deviation are removed. Second-level radius filtering: Based on the density characteristics of the local point cloud, check the number of neighbors of each point within a preset radius, and remove isolated points whose number of neighbors is less than the minimum cluster size threshold.

4. The method for multimodal environmental perception and post-fusion of a sweeping robot according to claim 1, characterized in that, The ground segmentation in step 2 uses the RANSAC algorithm based on geometric constraints. This algorithm introduces a strong constraint on the normal vector aligned with the gravity direction. The specific steps are as follows: 1) When iteratively sampling candidate plane models, calculate the candidate plane normal vector. gravity axis of the global coordinate system The included angle; 2) Construct a direction constraint filter to retain only those that satisfy the constraint. Candidate planar models, where Preset angle tolerance; Represents the inverse cosine function. This represents the magnitude of the candidate plane normal vector. The magnitude of the axial quantity of gravity in the global coordinate system; 3) Select the model with the largest number of interior points as the optimal ground parameters, and divide the point cloud set into ground point set and non-ground point set.

5. The method for multimodal environmental perception and post-fusion of a sweeping robot according to claim 4, characterized in that, The ground segmentation also includes an interior point determination method based on algebraic distance: Calculate any point in the point cloud to candidate plane algebraic distance : in, Represents any three-dimensional point in a point cloud. Point X-axis coordinates Point Y-axis coordinate, Point Z-axis coordinate, Indicates the first One candidate ground plane model, Representing candidate planes The x-axis coefficients of the equation, Representing candidate planes The Y-axis coefficients of the equation, Representing candidate planes Z-axis coefficients of the equation Representing candidate planes The constant term of the equation, Representing candidate planes The magnitude of the normal vector, if Less than the preset distance threshold If so, then mark that point as an interior point.

6. The method for multimodal environmental perception and post-fusion of a sweeping robot according to claim 1, characterized in that, The clustering in step 2 employs a density-based Euclidean clustering algorithm, specifically including: 1) Construct a K-dimensional tree data structure to spatially index the non-ground point set; 2) For unvisited points, search for their neighborhood point set within the search radius of the Euclidean cluster. If neighborhood points exist, they are assigned to the same cluster and the cluster is expanded recursively. 3) Introduce a minimum cluster size constraint, remove noise clusters with fewer points than the constraint, and output a point cloud cluster set.

7. The method for multimodal environmental perception and post-fusion of a sweeping robot according to claim 1, characterized in that, Step 4, which involves performing cross-modal spatial mapping and semantic injection, includes a semantic priority enhancement strategy for pre-defined elongated linear obstacles: 1) When the category of the 2D target detection is a preset thin, elongated linear obstacle category, cross-modal semantic injection is performed; 2) Any 3D point that falls within the bounding box projection range of this elongated linear obstacle category, regardless of whether its geometric density meets the clustering requirements, is assigned an obstacle semantic label and is given priority recall as an obstacle candidate cluster.

8. The method for multimodal environmental perception and post-fusion of a sweeping robot according to claim 1, characterized in that, The multi-constraint cascaded reasoning in step 5 employs a joint decision function, with the specific decision logic as follows: Level 1 determination: If the proportion of points with semantic labels in a point cloud cluster exceeds the preset semantic confidence threshold, and the physical size of the point cloud cluster is within the preset limit of the robot vacuum cleaner's suction port, then it is determined that the garbage can be cleaned. Second-level judgment: If the first-level judgment conditions are not met, but the maximum height and maximum length of the cloud cluster are both less than the set ground attachment threshold, then it is judged as cleanable garbage. Level 3 judgment: If none of the above conditions are met, it is determined that an obstacle needs to be avoided.

9. The method for multimodal environmental perception and post-fusion of a sweeping robot according to claim 1, characterized in that, The temporal tracking in step 6 employs an association metric based on aggregated Euclidean distance (AED) to calculate the difference between the predicted tracking box and the actual observed box; and the temporal tracking method using the AED association metric measures... The calculation formula is: in, This represents the number of corner points of the bounding box. The coordinate vector of the corner point. The center point coordinate vector, superscript Indicates the current sensor detection value, superscript This represents the Kalman filter prediction value. The L2 norm of a vector is used to represent the vector's L2 norm.

10. A method for multimodal environmental perception and post-fusion of a sweeping robot according to claim 9, characterized in that, The time-series tracking employs a greedy nearest neighbor strategy, prioritizing associations. The prediction-observation pair with the smallest value.

Images