A method for obstacle avoidance in cleaning robots that integrates vision, laser, and ultrasonic sensing.

Abstract

This invention relates to the field of autonomous obstacle avoidance control technology for intelligent robots, and discloses an obstacle avoidance method for cleaning robots that integrates visual, laser, and ultrasonic perception. The method first establishes a body coordinate system, sensor extrinsic parameter calibration parameters, a unified clock reference, and a material reflection feature template library. During its movement, the cleaning robot simultaneously collects visual data, laser data, ultrasonic data, and body motion data, aligning the multimodal perception data to the same spatiotemporal reference. Subsequently, it extracts the material reflection features of candidate obstacles and performs template matching. When there are inconsistencies in the position or material type of the perception results, it identifies low-confidence sensing modalities, dynamically adjusts the fusion confidence weight, and determines the safe distance and obstacle avoidance motion control commands accordingly. This improves the reliability of obstacle avoidance, cleaning coverage, and operational stability in complex material obstacle scenarios.

Description

Technical Field

[0001] This invention relates to the field of autonomous obstacle avoidance control technology for intelligent robots, specifically to an obstacle avoidance method for cleaning robots that integrates vision, laser, and ultrasonic perception. Background Technology

[0002] As smart home devices, cleaning robots typically use visual sensors, lidar sensors, and ultrasonic sensors to detect indoor obstacles and perform path planning and obstacle avoidance control based on the obstacle detection results. In the prior art, patent CN111158369B discloses a sweeping robot and its detection and control method for cleaning, which expands the detection range and acquires information about the robot's surrounding environment by setting a retractable detection module. Similarly, patent CN106527446B discloses a control method and device for a sweeping robot, which acquires cleaning task information and generates corresponding control information to control the sweeping robot to perform cleaning tasks. While these technologies can improve environmental detection, cleaning control, and obstacle avoidance control to some extent, their focus remains on expanding the detection range, controlling cleaning tasks, or controlling conventional paths. They do not address the differential perception failures caused by different types of sensors due to obstacles of different materials.

[0003] In real-world home environments, obstacles such as transparent glass, mirrored cabinet doors, black furniture bases, low-reflectivity soft objects, and thin cables exhibit different reflection, transmission, or absorption characteristics. Transparent obstacles can easily cause unclear visual features and missing lidar echoes; mirrored objects can easily cause laser echoes to deviate in direction; black light-absorbing objects can easily reduce laser echo intensity; and low-reflectivity soft objects and small, thin objects can easily lead to unstable ultrasonic echoes or insufficient spatial resolution. Existing multi-sensor fusion obstacle avoidance methods typically use fixed priorities or static weights to fuse the detection results from different sensors. When visual, lidar, and ultrasonic detection results contradict each other, the system struggles to make a judgment. Specifically, identifying which type of sensor is in a low-reliability or malfunctioning state can easily lead to the use of abnormal detection results as the basis for obstacle avoidance. This can result in missed detection or misjudged collisions with critical obstacles such as transparent objects, mirror objects, and black light-absorbing objects. It may also lead to excessive avoidance and repeated detours near ordinary obstacles, making it difficult to achieve both obstacle avoidance success rate and cleaning coverage. Therefore, there is an urgent need to provide an obstacle avoidance method for cleaning robots that can dynamically identify the reliability of sensors based on the reflective characteristics of obstacle materials, and perform confidence allocation and conflict arbitration on the detection results of multiple sensors. This would improve the reliability of obstacle avoidance in complex material obstacle scenarios, reduce unnecessary detours, and improve cleaning coverage and overall operational stability. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method for obstacle avoidance in cleaning robots that integrates visual, laser, and ultrasonic sensing, thus solving the problems mentioned in the background section.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A method for obstacle avoidance in cleaning robots that integrates vision, laser, and ultrasonic sensing includes:

[0007] S1. Establish the body coordinate system, sensor extrinsic calibration parameters, and unified clock reference for the cleaning robot, and load the material reflection feature template library;

[0008] S2. During the movement of the cleaning robot, visual data, laser data, ultrasonic data and body motion data are collected simultaneously, and the visual data, laser data and ultrasonic data are aligned to the same spatiotemporal reference based on the body motion data;

[0009] S3. Extract the material reflection features corresponding to the candidate obstacles from the visual data, laser data and ultrasonic data respectively, and match the material reflection features with the material reflection feature template library;

[0010] S4. When visual data, laser data, and ultrasonic data are inconsistent in terms of obstacle location or material type, identify low-confidence sensing modalities based on material reflection characteristics and adjust the fusion confidence weights of visual data, laser data, and ultrasonic data.

[0011] S5. Based on the adjusted fusion confidence weight, the obstacle position and material type are fused and determined, and a safe distance is determined based on the fusion determination result. The obstacle avoidance motion control command is then sent to the drive unit of the cleaning robot.

[0012] Preferably, the body coordinate system, sensor extrinsic calibration parameters, and unified clock reference of the cleaning robot are established, and a material reflection feature template library is loaded, including:

[0013] Basic parameter initialization is performed in the cleaning preparation state. The body coordinate reference is established with the geometric center of the machine body and the direction of travel, and the external parameter calibration parameters of the vision, laser and ultrasonic sensors are read and verified.

[0014] Add timestamps to common time sources and compensate for sampling delays;

[0015] When an external parameter verification error occurs, a valid calibration parameter is invoked.

[0016] When time deviation is abnormal, a time quality label is attached, and a material reflection feature template library with multiple types of material visual, laser and ultrasonic reference features is loaded and stored;

[0017] Call the general template or overlay localized correction parameters based on the template validation results.

[0018] Preferably, visual data, laser data, ultrasonic data, and body motion data are collected simultaneously during the movement of the cleaning robot, including:

[0019] During the data acquisition process, the data acquisition and buffering of the vision sensor, laser ranging unit, ultrasonic transducer array, inertial measurement unit and wheeled odometer are organized according to the sliding processing cycle.

[0020] For sensor data that has not been updated in the current period, retrieve the most recent sensor data with a valid timestamp from the data buffer;

[0021] For low-frequency scan data, retrieve the most recent complete scan data;

[0022] Based on the time difference between the sampling time and the reference time, data integrity, and sensor operating status, data timeliness identifier, data quality identifier, and sensor health status identifier are generated.

[0023] Preferably, aligning visual data, laser data, and ultrasonic data to the same spatiotemporal reference based on fuselage motion data includes:

[0024] During spatiotemporal alignment, the start time of the sliding processing cycle is used as the reference time, and the changes in fuselage attitude are cross-verified based on the inertial measurement unit and wheeled odometer.

[0025] By combining the sensor extrinsic calibration parameters, the visual candidate region, laser scanning point, and ultrasonic echo direction are mapped to the body coordinate system;

[0026] The data reliability level is marked based on the data timeliness, static feature verification results, and abnormal fuselage movement status.

[0027] Preferably, the material reflection features corresponding to candidate obstacles are extracted from visual data, laser data, and ultrasonic data, respectively, including:

[0028] During material feature extraction, the main control unit establishes a record of candidate obstacles within the forward region of interest in the body coordinate system;

[0029] Candidate regions are generated based on the stability of visual regions;

[0030] Candidate directions are generated based on the laser echo segments;

[0031] Candidate beam data are generated based on the ultrasonic beam response;

[0032] Based on the multimodal candidate data of the same obstacle associated with angle difference and distance difference, visual edge, texture and specular features, laser echo intensity and continuity features, and ultrasonic echo amplitude and frequency and beam response features are extracted respectively.

[0033] Preferably, the material reflection features are matched with a material reflection feature template library, including:

[0034] During template matching, the main control unit normalizes the material reflection characteristics of vision, laser, and ultrasound.

[0035] The weighted distance of each material category is calculated based on the discrimination weights in the template library. The weighted distance is converted into a matching score, generating visual evidence groups, laser evidence groups, and ultrasonic evidence groups corresponding to the scores of multiple material categories. When any modal data is missing, the corresponding modal state is marked, and the valid modal scores are normalized.

[0036] Preferably, when visual data, laser data, and ultrasonic data are inconsistent in terms of obstacle location or material type, low-confidence sensing modalities are identified based on material reflection characteristics, including:

[0037] When recognizing low-confidence sensing modalities, the determination is triggered based on the positional deviation and material score difference of the candidate obstacle.

[0038] Based on the laser echo loss, echo intensity fluctuation, visual texture change, ultrasonic amplitude-frequency response, and beam response characteristics corresponding to transparent bodies, specular bodies, black light-absorbing bodies, low-reflection soft bodies, and small-sized slender bodies, the low confidence type and regular confidence level of the corresponding sensing modes are marked according to the satisfaction of core conditions and auxiliary conditions.

[0039] Preferably, the confidence weights for the fusion of visual data, laser data, and ultrasound data are adjusted, including:

[0040] When adjusting the fusion confidence weight, the sum of the visual fusion weight, laser fusion weight, and ultrasonic fusion weight is used as a constraint;

[0041] Low-confidence sensing modes are downweighted, while sensing modes with stable material responses are upweighted.

[0042] Select the target weight, transition weight, or recalculate the branch according to the confidence level of the rule, and after resetting the abnormal mode weight to zero when the sensor health status is abnormal, renormalize the available mode weights.

[0043] Preferably, the obstacle location and material category are fused and determined based on the adjusted fusion confidence weights, including:

[0044] During the fusion determination, the main control unit takes the candidate obstacle as the object and performs obstacle existence determination, fusion distance calculation and multi-material scoring fusion in sequence according to visual fusion weight, laser fusion weight and ultrasonic fusion weight;

[0045] When only single-modal or dual-modal data is valid, the fusion weights corresponding to the valid modes are renormalized, and active exploration or conservative judgment is triggered based on the existence of support interval, material score interval or distance uncertainty.

[0046] Preferably, based on the fusion judgment result, a safe distance is determined, and obstacle avoidance motion control commands are issued to the drive unit of the cleaning robot, including:

[0047] During motion control, the main control unit determines the basic safety distance based on the material classification and determines the target safety distance by combining the distance uncertainty and the fuselage motion state;

[0048] Based on the cleaning operation zone, obstacle location, and material type, control commands such as deceleration, detour, edge-keeping, reversal, or stop are generated. The obstacle avoidance process data is written to the task log, and localized correction parameters are generated based on positive and negative samples.

[0049] Compared with existing technologies, this invention provides a method for obstacle avoidance in cleaning robots that integrates vision, laser, and ultrasonic sensing, and has the following beneficial effects:

[0050] 1. This invention establishes a unified spatiotemporal benchmark and material reflection feature template library, synchronously collects, spatiotemporally aligns, and matches material features of visual data, laser data, and ultrasonic data. When the obstacle position or material category is inconsistent among the three types of perception results, it identifies the low-confidence sensing mode corresponding to the current candidate obstacle based on the reflection, transmission, absorption, and echo response characteristics of transparent bodies, specular bodies, black light-absorbing bodies, low-reflection soft bodies, and small slender objects. It dynamically adjusts the fusion confidence weights of visual data, laser data, and ultrasonic data, and then determines the safe distance and generates obstacle avoidance motion control commands based on the fusion judgment results. This reduces the interference of abnormal sensing data on the fusion judgment, thereby improving the reliability of obstacle avoidance in complex material obstacle scenarios, reducing missed detections of key obstacles, misjudged collisions, and unnecessary detours, and improving cleaning coverage and overall operational stability.

[0051] 2. This invention generates data timeliness identifiers, data quality identifiers, sensor health status identifiers, and low-confidence sensor mode identifiers during data acquisition, spatiotemporal alignment, material matching, and fusion control processes. After obstacle avoidance is completed, it records the obstacle material category, three-mode feature vector, fusion confidence weight, safe distance, motion trajectory, and abnormal events. Based on positive and negative samples, it generates localized correction parameters, enabling the cleaning robot to continuously correct the characteristics of obstacles such as glass doors, mirrored cabinet doors, black furniture bases, soft fabrics, and thin cables that have long existed in the same home environment. This reduces repeated identification biases caused by differences in home environment, sensor batch differences, insufficient historical samples, or failure to utilize abnormal samples, thereby improving the consistency of material judgment in subsequent cleaning tasks, the stability of fusion parameter calls, and the traceability of control in abnormal scenarios. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the obstacle avoidance method for a cleaning robot that integrates vision, laser and ultrasonic perception according to the present invention.

[0053] Figure 2 This is a schematic diagram illustrating the multi-sensor unified spatiotemporal reference and data alignment of the present invention;

[0054] Figure 3 This is a schematic diagram illustrating the association between the forward region of interest and candidate obstacles in this invention;

[0055] Figure 4 This is a logic diagram for material reflection feature extraction and template matching in this invention;

[0056] Figure 5 This is a schematic diagram of the low-reliability sensing modality recognition and fusion weight adjustment of the present invention;

[0057] Figure 6 This is a schematic diagram illustrating the safety distance determination and obstacle avoidance motion control of the present invention;

[0058] Figure 7 This is a schematic diagram illustrating the update of the task log and localized correction parameters of this invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] Example 1: Figure 1 - Figure 6A method for obstacle avoidance in cleaning robots that integrates vision, laser, and ultrasonic sensing is presented, including:

[0061] S1. Establish the body coordinate system, sensor extrinsic calibration parameters, and unified clock reference for the cleaning robot, and load the material reflection feature template library;

[0062] S2. During the movement of the cleaning robot, visual data, laser data, ultrasonic data and body motion data are collected simultaneously, and the visual data, laser data and ultrasonic data are aligned to the same spatiotemporal reference based on the body motion data;

[0063] S3. Extract the material reflection features corresponding to the candidate obstacles from the visual data, laser data and ultrasonic data respectively, and match the material reflection features with the material reflection feature template library;

[0064] S4. When visual data, laser data, and ultrasonic data are inconsistent in terms of obstacle location or material type, identify low-confidence sensing modalities based on material reflection characteristics and adjust the fusion confidence weights of visual data, laser data, and ultrasonic data.

[0065] S5. Based on the adjusted fusion confidence weight, the obstacle position and material type are fused and determined, and a safe distance is determined based on the fusion determination result. The obstacle avoidance motion control command is then sent to the drive unit of the cleaning robot.

[0066] This method is applicable to the autonomous cleaning and obstacle avoidance control of indoor floor cleaning robots in environments such as residential living rooms, bedrooms, hallways, balconies, and office areas. The cleaning robot implementing this method can be a wheeled cleaning robot, equipped with at least a visual sensor, a laser rangefinder unit, an ultrasonic transducer array, an inertial measurement unit, a wheeled odometer, and a drive unit. The input information collected during operation includes at least visual image data, laser distance data, laser echo intensity data, ultrasonic echo delay data, ultrasonic echo amplitude data, body angular velocity, body acceleration, and left and right wheel rotation speeds. It is suitable for obstacles such as transparent glass, mirrored cabinet doors, black furniture bases, low-reflectivity soft objects, and small, slender objects. This can easily lead to inconsistencies in position or material determination among visual, laser, and ultrasonic data. The main control unit identifies the low-confidence sensing modality corresponding to the current candidate obstacle based on the reflection, transmission, absorption, and echo response characteristics of obstacles of different materials. It then adjusts the fusion confidence weights of the visual, laser, and ultrasonic data to form an obstacle avoidance motion control process that matches the material characteristics of the current obstacle. After the above processing, data such as obstacle position, material type, safety distance, obstacle avoidance trajectory control commands, data quality identifiers, and sensing modality confidence status identifiers are generated. These data are then used by the cleaning robot's drive unit to execute corresponding deceleration, detour, edge-hugging, retraction, or stop controls.

[0067] Specifically, such as Figure 2 As shown: When the cleaning robot completes its power-on self-test, receives a cleaning task, or detaches from its charging dock and enters the cleaning preparation state, it initiates a basic parameter initialization process. During initialization, the position of the cleaning robot's geometric center on the horizontal projection plane is used as the origin of the body coordinate system, the forward direction of the body is used as the positive longitudinal direction of the body coordinate system, the left side of the body is used as the positive lateral direction of the body coordinate system, and the vertical direction above the body is used as the positive height direction of the body coordinate system, thus establishing a three-dimensional right-handed coordinate reference. This body coordinate system serves as a unified reference for spatial mapping and spatiotemporal alignment of visual data, laser data, ultrasonic data, and body motion data.

[0068] The sensor extrinsic calibration parameters are derived from the factory calibration or maintenance calibration process. The calibration objects include the optical center position of the vision sensor, the optical axis direction of the vision sensor, the installation position of the laser transmitter / receiver unit, the laser scanning plane direction, the center position of each ultrasonic transducer, and the main beam direction. After calibration, the transformation matrices from the vision coordinate system to the body coordinate system, the laser coordinate system to the body coordinate system, and the transformation matrices from the measurement coordinate system of each ultrasonic transducer to the body coordinate system are written to a non-volatile storage area and read before each cleaning task starts. During reading, the completeness, matrix dimensions, and parameter check codes of the extrinsic calibration parameters are verified. If any extrinsic calibration parameters are missing... When there are errors in the item or matrix dimensions or inconsistent parameter check codes, the most recently valid calibration parameters are called. When two consecutive reads are abnormal, the cleaning robot enters a low-speed safety cleaning mode, with the upper limit of the robot speed not exceeding 0.2 m / s. Two consecutive reads being abnormal serves as the trigger condition for the low-speed safety cleaning mode, used to exclude single, occasional read failures caused by power-on transients, power fluctuations, or storage access delays. The speed limit of 0.2 m / s corresponds to a robot displacement of approximately 1 cm within a 50-millisecond processing cycle. This displacement is lower than the commonly used forward safety distance adjustment amount for cleaning robots, which can reserve motion margin for braking, resampling, and backtracking when external parameters are abnormal.

[0069] The unified clock reference uses the local clock of the main control unit as the common time source; visual data, laser data, ultrasonic data, inertial measurement data, and wheel odometry data are all appended with a unified timestamp when written to the data buffer, and the timestamp resolution can be selected as 1 millisecond; this value is determined according to the common travel speed of the cleaning robot and the resolution of the local obstacle avoidance grid; when the common travel speed of the indoor cleaning robot is 0.25 m / s to 0.5 m / s, the maximum travel speed of 0.5 m / s corresponds to a body displacement of 0.5 mm within 1 millisecond, which is significantly smaller than the common resolution of 1 cm to 3 cm for the local obstacle avoidance grid, and can meet the time alignment requirements of visual, laser, and ultrasonic data;

[0070] For cases where there is a fixed delay in the sampling links of different sensors, the sampling delay compensation is corrected based on the amount obtained from the factory synchronous trigger test. Specifically, the visual frame delay compensation can be selected from 10 to 30 milliseconds, the laser scanning delay compensation from 5 to 20 milliseconds, and the ultrasonic echo processing delay compensation from 2 to 10 milliseconds. These compensation amounts are obtained by recording the time difference between the sensor sampling time, the signal processing completion time, and the main control unit receiving time under the same trigger signal, and the average of multiple test results or the larger value covering 95% of the test samples is taken as the compensation parameter. The visual frame delay compensation is greater than the laser scanning delay compensation and the ultrasonic echo processing delay compensation because the visual sensor involves exposure, image readout, and frame buffering processes. The laser scanning delay mainly comes from the angle unfolding and echo encoding processes within the scanning cycle. The ultrasonic echo processing delay mainly comes from the transmission-reception interval, envelope extraction, and echo peak confirmation processes.

[0071] If, during operation, a deviation of more than 3 milliseconds is detected between a sensor's timestamp and the common time source, and this deviation persists for three consecutive sampling cycles, a time quality flag is added to the sensor data. 3 milliseconds corresponds to approximately 1.5 millimeters of fuselage displacement at a maximum travel speed of 0.5 m / s, which is less than 15% of the local obstacle avoidance grid resolution and can be used as a threshold for determining time synchronization anomalies. Three consecutive sampling cycles serve as a condition for continued anomaly, used to filter out instantaneous deviations caused by single-frame transmission jitter or buffer queuing, and to identify persistent synchronization anomalies. Sensor data with the added time quality flag still enters the subsequent processing flow and participates in weight adjustment as low-time-reliability data in the fusion determination.

[0072] The material reflection feature template library is loaded into the main control unit's cache area during the basic parameter initialization process. This library can be formed at the factory stage through standard sample collection. The standard samples must cover at least transparent glass, plane mirrors, black plastic, ferrous metal, fabrics, plush materials, cables, and the surfaces of regular wooden or plastic furniture. The collection distance can be selected as 10 cm, 30 cm, 80 cm, and 150 cm, corresponding to close-range edge-hugging and low-speed obstacle avoidance scenarios, regular close-range detour scenarios, forward obstacle avoidance recognition scenarios, and early deceleration and long-range prediction scenarios at regular speeds, respectively. The collection lighting can be selected as bright, dim, backlight, and locally strong reflection environments to correspond to common situations in home environments such as daytime lighting, nighttime lighting, window backlighting, and specular reflection. The feature values ​​obtained from each type of standard sample under different distances and lighting conditions are statistically analyzed to generate reference feature data for various materials.

[0073] The material reflection feature template library stores reference features for at least six types of obstacles: transparent bodies, specular bodies, black light-absorbing bodies, low-reflection soft bodies, regular opaque bodies, and small-sized slender bodies. Each type of obstacle corresponds to a visual feature reference vector, a laser feature reference vector, and an ultrasonic feature reference vector. The visual feature reference vector includes at least the following fields: edge gradient density, texture entropy fluctuation, specular highlight ratio, chromaticity saturation variance, and stereo matching confidence. When the cleaning robot is equipped with near-infrared or polarization acquisition hardware, the visual feature reference vector also includes fields such as near-infrared brightness and polarization response difference. The laser feature reference vector includes at least the following fields: echo missing rate, mean echo intensity, standard deviation of echo intensity, echo time stability, and continuity of echoes from adjacent angles. The ultrasonic feature reference vector includes at least the following fields: echo amplitude-frequency ratio, second echo presence indicator, first echo time delay stability, echo envelope main peak steepness, and echo amplitude difference between adjacent transducers.

[0074] Each reference feature stores the feature mean, feature variance, discrimination weight, and applicable distance range. The discrimination weight is determined based on the distinguishability of the corresponding feature among different material categories, and the distinguishability is represented by the ratio of the difference between inter-class means to the standard deviation within a class. When the difference between the means of a feature among different material categories is large and the fluctuation within the same type of sample is small, the discrimination weight of that feature is increased. When a feature is greatly affected by lighting, distance, or posture, the discrimination weight of that feature is decreased. The applicable distance range can be selected from 10 cm to 150 cm, where 10 cm is the lower limit for stable collection of effective features during close-range edge cleaning and low-speed obstacle avoidance, and 150 cm is the upper limit for the indoor cleaning robot to perform early deceleration, detour planning, and material prediction at normal speeds. At the maximum speed of the robot body of 0.5 m / s, 150 cm corresponds to approximately 3 seconds of forward prediction time, which can reserve processing time for obstacle confirmation, speed adjustment, and detour trajectory generation.

[0075] After the template library is loaded, the main control unit verifies the feature dimensions, normalization range, version number, and parameter check codes of various templates. When a certain type of material template is missing or the verification is abnormal, the factory-issued universal template is used to supplement it. When there are already localized correction parameters generated by historical cleaning tasks, the localized correction parameters are superimposed after loading the factory-issued universal template to correct the feature differences of obstacles such as glass doors, mirrored cabinet doors, black furniture bases, soft fabrics, and thin cables that have been present in the same home environment for a long time. The localized correction parameters only make limited corrections to the feature mean, feature variance, or discrimination weight in the factory-issued universal template. The single correction amplitude does not exceed 10% of the corresponding factory parameters to avoid a small number of abnormal samples causing the template to continuously deviate from the actual material characteristics. Through the above processing, the body coordinate reference, external parameter calibration parameters, unified clock reference, and material reflection feature template library required for subsequent data acquisition alignment and material feature matching are obtained.

[0076] Specifically, such as Figure 2 As shown: After the cleaning robot enters the moving cleaning state, it organizes the acquisition, buffering, and alignment of visual data, laser data, ultrasonic data, and body motion data according to the sliding processing cycle. The sliding processing cycle serves as the basic cycle for multimodal fusion processing and motion control command refresh, and does not require each type of sensor to generate new complete data in every processing cycle. When a sensor does not generate new complete data in the current processing cycle, it reads the most recent data with a valid timestamp in the data buffer and adds a data timeliness identifier according to the time difference between the data sampling time and the current reference time. In the case where the laser ranging unit cannot complete a new frame scan in the current sliding processing cycle at a working frequency of 10 Hz, it uses the most recent complete scan data in the data buffer as the basis and performs motion compensation according to the time difference between the scan data sampling time and the reference time.

[0077] The sliding processing cycle can be selected as 50 milliseconds. At a body speed of 0.5 m / s, 50 milliseconds corresponds to a body displacement of approximately 2.5 cm, which falls within the commonly used resolution range of 1 cm to 3 cm for local obstacle avoidance grids. This reduces positional offset between different modal data while maintaining the control refresh rate. When the body speed is below 0.3 m / s, the sliding processing cycle can be adjusted to 70 milliseconds, at which point the body displacement is approximately 2.1 cm, still within the commonly used resolution range for local obstacle avoidance grids. When the body is in a low-speed edge-following cleaning state and the speed is below 0.15 m / s, the sliding processing cycle can be adjusted to 100 milliseconds, at which point the body displacement is approximately 1.5 cm, balancing edge-following sampling stability and the computational load of the main control unit. When the body is in a straight-line rapid coverage state... In this state, the sliding processing cycle can be shortened to 30 milliseconds, corresponding to a displacement of about 1.5 centimeters at a maximum speed of 0.5 m / s. This can improve the refresh frequency of motion control commands without exceeding the local obstacle avoidance grid resolution. The minimum value of the sliding processing cycle can be selected as no less than 25 milliseconds, and the maximum value can be selected as no more than 120 milliseconds. 25 milliseconds is used as the lower limit to avoid the main control unit from frequently processing the same cached data repeatedly when the visual, laser, or ultrasonic data has not been updated. 120 milliseconds is used as the upper limit because at a maximum speed of 0.5 m / s, it corresponds to a displacement of about 6 centimeters. This displacement is close to the lateral scale of small obstacles such as common cable tangles, furniture legs, and pet toy edges on the cleaning path. Further increasing this displacement will reduce the accuracy of cross-modal candidate target association.

[0078] Within each sliding processing cycle, the vision sensor acquires forward or upward color images, grayscale images, or depth-assisted images; the laser ranging unit acquires distance sequences and echo intensity sequences arranged according to scanning angles; the ultrasonic transducer array acquires echo delay, echo amplitude, and echo envelope in each emission direction; the inertial measurement unit acquires triaxial angular velocity and triaxial acceleration; and the wheeled odometer acquires the rotational speeds of the left and right drive wheels. The working frame rate of the vision sensor can be selected from 25 frames / second to 30 frames / second, the working frequency of the laser ranging unit can be selected from 10 Hz to 30 Hz, and the working frequency of the ultrasonic transducer array can be selected from 30 Hz to 40 Hz. The above frequency range corresponds to the low-power sensor configuration commonly used in household cleaning robots, and can support real-time obstacle avoidance control under the premise that the main control unit's computational load is acceptable.

[0079] After various types of data enter the data buffer, they are arranged according to a unified timestamp to form periodic data packets. Each periodic data packet includes at least the following fields: sampling time, sensor number, raw data, data timeliness identifier, data quality identifier, and sensor health status identifier. The data timeliness identifier indicates the time difference between the data sampling time and the reference time, while the data quality identifier indicates changes in data reliability caused by sensor anomalies, motion anomalies, or signal noise. The data timeliness identifier can be divided into three levels based on the time difference: normal, delayed, and expired. A time difference of no more than one sliding processing cycle is marked as normal; a difference greater than one but no more than two sliding processing cycles is marked as delayed; and a difference greater than two sliding processing cycles is marked as expired.

[0080] The spatiotemporal alignment process uses the start time of each sliding processing cycle as the reference time. The actual sampling times are read from the visual image, laser scanning point, and ultrasonic echo direction. Based on the time difference between the actual sampling time and the reference time, combined with the angular velocity output by the inertial measurement unit (IMU) and the linear velocity output by the wheel odometer, the fuselage's pose change within this time difference is estimated. The linear velocity can be determined by the rotational speed and radius of the left and right drive wheels, while the angular velocity can be determined by the vertical angular velocity of the IMU, and cross-validated using the differential speed results of the left and right wheels from the wheel odometer. The angular velocity difference can be calculated by dividing the absolute value of the difference between the angular velocity estimated by the IMU and the angular velocity estimated by the wheel odometer by the larger of the two absolute values. Value calculation; if the angular velocity difference is less than 15%, the weighted average of the two is used as the motion compensation parameter; when weighting, the weights can be assigned according to the sensor health status and short-term stability. When both are in normal health status, the weights of the inertial measurement unit and the wheel odometer can be 0.5 and 0.5 respectively; if the angular velocity difference is greater than 15% and lasts for two consecutive sliding processing cycles, it is determined that the body may slip, the ground may be uneven or vibrating, and the motion compensation result of this cycle is marked as low confidence in motion compensation; the 15% difference threshold is used to distinguish between normal sensing errors and abnormal motion differences; two consecutive sliding processing cycles are used as a continuous condition to filter out single-cycle impacts or instantaneous encoder jitter;

[0081] For any sensor data point, the translation and rotation of the data point relative to the reference time are inferred from the time difference between the sampling time and the reference time. Combined with the sensor extrinsic calibration parameters, the data point is mapped to the body coordinate system at the reference time. Visual data is mapped to the body orientation domain by combining image plane coordinates with depth estimation or scale prior. Laser data is mapped to body coordinate points by scanning angle and distance value. Ultrasonic data is mapped to body orientation sectors by transducer installation orientation, main beam angle, and echo delay. For data with data timeliness indicators, a corresponding confidence level is added according to its time difference level: normal data corresponds to a high confidence level, delayed data corresponds to a medium confidence level, and expired data corresponds to a low confidence level, so as to adjust the weights during subsequent fusion judgment.

[0082] To reduce the impact of cumulative errors in the inertial measurement unit (IMU) on spatiotemporal alignment, the cleaning robot selects static feature points such as wall corners, wall edges, and fixed furniture edges from the laser scan data every 5 to 10 seconds to verify the robot's pose increment. The 5 to 10-second verification interval is determined based on the common speed of the indoor cleaning robot, which is 0.25 m / s to 0.5 m / s, corresponding to a travel distance of approximately 1.25 m to 5 m. Within this distance range, static features such as wall edges, wall corners, or fixed furniture edges can usually be observed again. If the difference between the pose increment estimated by the laser static feature points and the cumulative pose increment of the IMU exceeds 3 cm, or the angular difference exceeds 3 degrees, a pose error correction is generated and the motion compensation parameters are updated. 3 cm is consistent with the common resolution upper limit of 1 cm to 3 cm for local obstacle avoidance grids. A 3-degree angular error corresponds to a lateral deviation of approximately 5.2 cm at a distance of 1 meter, which may already affect the cross-modal association of candidate obstacles, and therefore can be used as a threshold to trigger pose error correction.

[0083] If the peak acceleration of the machine body exceeds 5 m / s² or the peak angular velocity exceeds 1 radian / s, the machine body is considered to be in a collision, obstacle crossing, or severe vibration state. The posture error correction is suspended, and only the original sensor data is retained with an additional motion abnormality quality label to prevent erroneous corrections under abnormal motion conditions from entering the subsequent process. 5 m / s² is approximately 0.5 times the acceleration due to gravity, which is usually higher than the acceleration fluctuation during normal cleaning on a flat surface. 1 radian / s is approximately equal to 57 degrees / s. This conversion is derived from π radians equaling 180 degrees, which is usually higher than the smooth turning angular velocity during regular edge cleaning or overlay cleaning. Therefore, it can be used to identify collision, obstacle crossing, or severe vibration states.

[0084] After completing the spatiotemporal alignment, a three-mode sensing data packet is generated at the same reference time. This three-mode sensing data packet retains the data timeliness identifier, data quality identifier, and sensor health status identifier from the periodic data packet, and associates them with visual candidate image regions, laser candidate direction data, ultrasonic candidate beam data, and fuselage pose data as inputs for subsequent material reflection feature extraction and multimodal fusion determination.

[0085] Specifically, such as Figure 3 and Figure 4 As shown: After receiving the three-mode perception data packets at the same reference time, the main control unit uses the forward region of interest in the body coordinate system as the processing range to establish a candidate obstacle record. The forward region of interest can be selected as a 120-degree angular domain in front of the robot body and a distance range of 0.1 meters to 1.5 meters. The 120-degree angular domain covers the conventional detour areas in the forward direction, left front, and right front of the cleaning robot, and is suitable for candidate obstacle extraction during forward obstacle avoidance, edge turning, and local detour. The distance range of 0.1 meters to 1.5 meters corresponds to the applicable distance of the material reflection feature template library. Among them, 0.1 meters is used to cover the close-range edge and low-speed obstacle avoidance areas, and 1.5 meters corresponds to about 3 seconds of forward prediction time at the maximum speed of the robot body of 0.5 meters / second, which can reserve processing margin for obstacle confirmation, deceleration processing, and detour trajectory generation.

[0086] The initial extraction of candidate obstacles is performed using three types of data: visual, laser, and ultrasonic. Visual candidate regions are formed through edge gradient detection, connected region merging, and region stability screening. Region stability screening is based on the center position shift, area change rate, and edge overlap of candidate regions over two to four consecutive frames. Candidate regions are retained when the center position shift does not exceed 20% of the region width, the area change rate does not exceed 30%, and the edge overlap is higher than 60%. This number of consecutive frames filters out single-frame light spots and short-term occlusions. A 20% center position shift, a 30% area change rate, and a 60% edge overlap are used to distinguish between real obstacle regions and noisy edge regions. Laser candidate directions are determined by continuous invalid echoes. The ultrasound candidate direction is formed by the presence of abrupt distance changes, narrow-angle strong echo points, and low-intensity continuous echo points. Abrupt distance changes can be determined based on whether the distance difference between adjacent scanning angles exceeds 10 cm. Narrow-angle strong echo points can be determined based on echo intensity exceeding twice the neighborhood mean and an angle width less than 3 degrees. Low-intensity continuous echo points can be determined based on normalized echo intensity at multiple consecutive angles being below 0.1. 10 cm is close to the scale of a typical small obstacle; twice the neighborhood mean is used to identify local strong reflection peaks; a 3-degree angle width is used to distinguish narrow-angle strong echoes from large-area reflective surfaces; and 0.1 is the low-reflection response boundary within the normalized echo intensity range. The ultrasound candidate direction is formed by the presence of the first echo, abrupt changes in echo amplitude, amplitude differences between adjacent transducers, and differences in multi-band echo responses.

[0087] To avoid the three modalities forming isolated candidate targets that are not related to each other, the main control unit uses the body coordinate direction as an index to group candidate data with adjacent angle differences of less than 5 degrees and distance differences of less than 10 centimeters into the same candidate obstacle record; 5 degrees corresponds to about 8.7 centimeters of lateral deviation at a distance of 1 meter, which matches the lateral tolerance of the home cleaning robot for cross-modal association of small targets such as furniture legs and cables; 10 centimeters is close to the conventional obstacle avoidance safety distance and the scale of small obstacles, which can reduce the probability of adjacent but physically separate targets being mistakenly merged;

[0088] The extraction target for visual material reflection features is the region of interest (ROI) of the candidate obstacle in the image plane. The visual features extracted from this ROI include at least the following fields: edge gradient density, texture entropy inter-frame fluctuation amplitude, specular highlight pixel ratio, chroma saturation variance, binocular stereo matching confidence, and inter-frame grayscale consistency. When the cleaning robot is equipped with near-infrared or polarization acquisition hardware, fields such as near-infrared brightness mean and polarization response difference can also be extracted. If the corresponding hardware is not configured, no corresponding fields are generated, and the discrimination weights of the remaining visual features are normalized and adjusted. Edge gradient density is the ratio of the number of effective edge pixels to the total number of pixels in the ROI. Effective edge pixels can be defined as those whose image gradient amplitude is higher than that of the ROI. Pixels with a domain average gradient magnitude of 1.5 are used to characterize the visual edge response of transparent objects or low-texture targets; the inter-frame fluctuation amplitude of texture entropy is determined by the difference in texture entropy of the same region in multiple consecutive frames, used to characterize changes in specular reflection or environmental reflection; the proportion of specular highlight pixels is the ratio of the number of overexposed bright pixels to the total number of pixels in the region of interest. Overexposed bright pixels can be determined by pixels with a gray value higher than 230 and a gray value difference higher than 80 from the average gray value of their neighbors, used to characterize specular objects or highly reflective surfaces; the chromaticity saturation variance is used to distinguish between black light-absorbing objects and ordinary colored targets; the stereo matching confidence is the ratio of the number of effective disparity pixels to the total number of pixels in the region of interest, used to characterize the stereo matching reliability of transparent objects or highly reflective objects;

[0089] In the factory-standard template, the initial determination boundary for the edge gradient density of transparent objects can be selected to be below 0.08, the initial determination boundary for the proportion of specular highlight pixels of mirror objects can be selected to be above 0.2, the initial determination boundary for the chromaticity saturation variance of black light-absorbing objects can be selected to be below 0.05, and the initial determination boundary for the binocular stereo matching confidence of transparent and highly reflective objects can be selected to be below 0.4. The above values ​​are determined by the statistical results of standard samples. For example, visual features of samples of the same material are collected at different distances, under different lighting conditions and at different incident angles, and the initial determination boundary is formed by the mean and standard deviation of the same samples. For the edge gradient density of transparent objects, the mean of transparent object samples plus twice the standard deviation can be taken as the upper limit, and for the proportion of specular highlight pixels, the mean of mirror object samples minus twice the standard deviation can be taken as the lower limit. For different hardware batches or different home scenarios, the above initial determination boundaries can be corrected according to the mean, variance and localized correction parameters stored in the material reflection feature template library.

[0090] The laser material reflection features are extracted from distance data and echo intensity data within the candidate direction window. Using the candidate center direction as a reference, the laser feature statistical window can be selected to be within a 5-degree range to the left and right. This statistical window is used to extract the continuity, distance variation, and intensity distribution of laser echoes near the candidate center direction, and is not used as a merging condition between different candidate obstacles. At a distance of 1 meter, a 5-degree range to the left and right corresponds to a lateral coverage width of approximately 17.5 centimeters, suitable for statistically analyzing local echo trends near furniture legs, cable tangles, glass edges, and mirror edges. Laser features include at least the echo missing rate and echo intensity average. Fields include echo intensity mean, echo intensity standard deviation, echo time stability, echo continuity of adjacent angles, and transmit power response consistency. The echo missing rate is the ratio of the number of angles with no valid returns within the candidate direction window to the total number of angles. The echo intensity mean and echo intensity standard deviation are calculated based on echo intensity normalized to the range of 0 to 1. Echo time stability is determined by the distance change of the same candidate direction over multiple consecutive scan cycles. Echo continuity of adjacent angles is used to determine whether there is continuous effective reflection around the candidate direction. Transmit power response consistency is used to determine whether the echo intensity changes approximately linearly with the transmit power.

[0091] In the factory-standard template, an echo missing rate higher than 0.6 can be used as the initial judgment boundary for deviations in transparent body transmission or specular reflection. This value indicates that more than half of the angles within the candidate window have not formed effective returns, which can distinguish between a small amount of noise missing and continuous detection missing. When the average echo intensity is lower than 0.1 and there are still weak echoes in the adjacent angles, it can be used as the initial judgment boundary for black light-absorbing bodies, where 0.1 is the low reflection response boundary within the normalized echo intensity range. When the standard deviation of the echo intensity exceeds 3 times the historical fluctuation benchmark in the same area and the echo time stability exceeds 5 cm, it can be used as the initial judgment boundary for angle-sensitive reflection of specular bodies. The historical fluctuation benchmark can be the average of the standard deviation of the echo intensity in the most recent 5 to 10 effective scanning cycles in the same candidate direction. 3 times the historical fluctuation benchmark is used to distinguish between normal intensity changes and abnormal peak responses. 5 cm is close to the local obstacle avoidance safety distance adjustment of the cleaning robot. Exceeding this distance can easily affect the obstacle position association.

[0092] The consistency of laser emission power response can be determined by the echo intensity variation relationship under three emission power levels: low, medium, and high. The ratio of the three emission power levels can be selected as 0.4, 0.7, and 1.0. This ratio can differentiate the echo response without exceeding the rated safe emission power. The linear correlation coefficient is calculated based on the correlation between the three emission power ratios and the corresponding echo intensities. The linear correlation coefficient for normal diffuse reflection targets is usually higher than 0.9, while for transparent or specular objects, due to transmission, specular reflection, or reception deviation, the linear correlation coefficient is usually lower than 0.6. The above laser judgment boundary can be corrected based on factory standard samples and historical valid data to compensate for the differences in emission power, receiving sensitivity, and echo encoding between different batches of laser ranging units.

[0093] The extraction of ultrasonic material reflection features includes echo delay, echo amplitude, echo envelope, and adjacent transducer response along the candidate beam direction. The ultrasonic detection frequency band can be selected as the 40 kHz standard band, the 25 kHz low-frequency auxiliary band, and the 65 kHz high-frequency auxiliary band. When the cleaning robot is equipped with only a single-frequency ultrasonic transducer, the standard band echo features are retained, and multi-band related fields are marked as unavailable. 40 kHz is a commonly used frequency band for indoor short-range ultrasonic obstacle avoidance; 25 kHz has relatively low attenuation for rough, soft materials; and 65 kHz is more sensitive to surface details and small targets. For multi-band detection, the interval between three transmissions can be selected as 4 to 6 milliseconds. This interval is greater than the main echo reception time of nearby indoor obstacles, which can reduce the superposition of echoes from different frequency bands in the time domain without significantly increasing the detection delay within a single processing cycle.

[0094] Ultrasonic features include at least the following fields: echo amplitude-frequency ratio, second echo presence indicator, first echo delay stability, echo envelope peak steepness, echo amplitude difference between adjacent transducers, and echo phase consistency. The echo amplitude-frequency ratio, the ratio of high-frequency echo amplitude to low-frequency echo amplitude, characterizes the reflection differences of different materials at different acoustic frequencies. The second echo presence indicator characterizes the multiple reflections that may occur in transparent, rigid materials. The first echo delay stability characterizes the stability of candidate direction distance estimation. The echo envelope peak steepness characterizes the surface hardness and roughness of the target. The echo amplitude difference between adjacent transducers characterizes the response differences of small, slender objects to different beam directions. Echo phase consistency characterizes large, smooth surfaces. The acoustic reflection consistency of a mirror-like hard surface; in the factory-standard template, the echo amplitude-frequency ratio of a smooth hard surface can be selected from 0.85 to 1.1, and the echo amplitude-frequency ratio of a rough soft surface can be selected from less than 0.5; when the second echo delay is 1.1 to 1.3 times the first echo delay, and the second echo amplitude is less than 70% of the first echo amplitude but higher than the background noise baseline, it can be used as auxiliary evidence of a transparent hard body; when the difference in echo amplitude between adjacent transducers is greater than 0.4, it can be used as an auxiliary judgment boundary for insufficient ultrasonic resolution caused by small-sized slender objects; the above ultrasonic judgment boundary is determined by the statistical results of the ultrasonic response of standard samples at different distances, angles and surface roughnesses, and can be corrected as the template library is updated;

[0095] After extracting the three types of features, the main control unit assembles visual, laser, and ultrasonic features into feature vectors and matches them with the loaded material reflection feature template library. During matching, each feature dimension is first normalized. Normalization can be performed linearly using the applicable range of the corresponding feature in the template library, or it can be standardized using the mean and standard deviation of the corresponding feature. Subsequently, the weighted distance between the real-time feature vector and the reference vectors for each material category is calculated according to the discrimination weights in the template library. The discrimination weights for each dimension can be normalized so that the sum of the weights for each dimension within the same modality is 1. The weighted distance D can be expressed as the cumulative value of the product of the absolute value of the difference between each feature dimension and the corresponding discrimination weight, i.e.:

[0096]

[0097] Where wi is the discrimination weight of the i-th feature, xi is the real-time feature value of the i-th dimension, and ri is the reference feature value of the i-th dimension; the smaller the weighted distance, the closer the real-time feature vector is to the corresponding material category;

[0098] The main control unit converts the weighted distance into a matching score within the range of 0 to 1. The matching score P can be expressed as:

[0099]

[0100] Where D is the weighted distance; the higher the score, the stronger the support of the current candidate obstacle for the corresponding material category; each candidate obstacle generates a visual evidence group, a laser evidence group, and an ultrasonic evidence group. Each evidence group includes at least six material score fields, such as transparent body, specular body, black light-absorbing body, low-reflection soft body, regular opaque body, and small slender body. It also receives the data timeliness identifier, data quality identifier, and sensor health status identifier from the three-mode perception data package and associates them with candidate position and distance estimation; if data for a certain modality is missing, no material score is generated for that modality, and the modality status is marked as missing data; the remaining valid modalities still generate corresponding evidence groups according to the above process, and the weights of the valid modalities are renormalized during subsequent fusion; the extracted evidence groups, candidate obstacle spatial association records, and modal quality identifiers are used as inputs for subsequent modal consistency judgment and confidence weight adjustment.

[0101] Specifically, such as Figure 5 As shown: The main control unit judges the consistency of spatial position and material type of the same candidate obstacle based on the visual evidence group, laser evidence group, ultrasonic evidence group, and spatial association records of candidate obstacles. Spatial position consistency is based on the position of the candidate obstacle in the body coordinate system. When the angle difference between the back projection direction of the visual candidate area, the laser candidate direction, and the ultrasonic candidate beam direction exceeds 8 degrees, or the maximum difference between the distance estimates of the three exceeds 15 centimeters, it is determined that there is inconsistency in the obstacle position of the three types of perception data. An 8-degree angle difference corresponds to a lateral deviation of about 14 centimeters at a distance of 1 meter in front of the cleaning robot, and is greater than the 5-degree angle difference used when spatially associating candidate obstacles, indicating that the candidate target has exceeded the stable association range and may affect the obstacle avoidance trajectory of targets such as small furniture legs, cables, and glass door edges. A 15-centimeter distance difference is greater than the conventional edge safety distance and the size of small obstacles. When it exceeds this range, it is easy to cause deviation in the calculation of the detour radius. Therefore, it can be used as the judgment boundary for distance inconsistency.

[0102] Material category consistency is based on the category corresponding to the highest material score among the visual evidence group, laser evidence group, and ultrasonic evidence group. When the categories corresponding to the highest material scores of the three evidence groups are not completely consistent, or the highest material score of any evidence group is lower than 0.7, or the difference between the highest and second-highest material scores in the same evidence group is less than 0.1, it is considered that there is inconsistency in the material category among the three types of perceptual data. 0.7 is used as the high confidence score threshold, which is determined by the matching score distribution of standard samples in the material reflectance feature template library, and is used to indicate that the real-time feature has a high degree of proximity to a certain material template. 0.1 is used as the support interval threshold to distinguish between clear classification and boundary classification. When the highest material score is lower than 0.7 or the difference between the highest and second-highest material scores is less than 0.1, it indicates that the current material evidence is insufficient to support a stable classification on its own.

[0103] When spatial inconsistency or material inconsistency is triggered, the main control unit invokes the low-confidence judgment rule of the sensing mode. The transparent body judgment rule is as follows: the echo missing rate in the laser evidence group is higher than 0.6, the edge gradient density in the visual evidence group is lower than 0.08, and the polarization response difference is higher than 0.15 when the polarization acquisition hardware is configured. The ultrasonic evidence group has a second echo, and the echo amplitude-frequency ratio is in the range of 0.85 to 1.1. When all the above conditions are met, a transparent body rule triggering identifier is generated, and the laser data is marked as low confidence. An echo missing rate higher than 0.6 indicates that more than half of the angles in the candidate window have not formed a valid return. An edge gradient density lower than 0.08 indicates a weak visual edge response. A polarization response difference higher than 0.15 is used to distinguish between transparent or specular bodies and ordinary diffuse reflection targets. This value is determined based on the polarization response distribution interval between the standard sample of transparent or specular bodies and the diffuse reflection sample. An echo amplitude-frequency ratio in the range of 0.85 to 1.1 indicates that the ultrasound has a stable response to smooth and hard surfaces. The above conditions are used together to distinguish between the low confidence state of laser caused by transparent bodies and ordinary occlusion noise.

[0104] The criteria for determining a specular body are as follows: In the laser evidence group, the standard deviation of echo intensity exceeds three times the historical fluctuation benchmark for the same candidate direction, and the echo time stability exceeds 5 cm; in the visual evidence group, the proportion of specular highlight pixels is higher than 0.2, and the inter-frame fluctuation amplitude of texture entropy is higher than 0.15; in the ultrasonic evidence group, the echo phase consistency score is higher than 0.7, and the echo amplitude-frequency ratio is within the range of 0.85 to 1.1. When all the above conditions are met, a specular body rule trigger flag is generated, and the laser data is marked as angle-sensitive low confidence. The historical fluctuation benchmark can be the 5 to 10 most recent valid data from the same candidate direction. The average standard deviation of echo intensity within the scanning period, and three times the historical fluctuation benchmark are used to distinguish between normal intensity changes and abnormal peak responses; 5 cm is the local obstacle avoidance safety distance adjustment for the cleaning robot, exceeding this distance can easily affect the position association of candidate obstacles; the inter-frame fluctuation amplitude of texture entropy is higher than 0.15, indicating that the reflected texture in the same area changes significantly with the body pose, and this value is determined based on the difference in inter-frame texture entropy fluctuation between mirror body samples and regular opaque body samples; the echo phase consistency score is higher than 0.7, indicating that the phase difference between multiple ultrasonic receiving channels is small, which can serve as auxiliary evidence for smooth hard surfaces;

[0105] The criteria for determining a black light absorber are as follows: the mean echo intensity in the laser evidence group is less than 0.1, the variance of chromatic saturation in the visual evidence group is less than 0.05, and the ultrasonic evidence group has effective first echoes in both the low-frequency and high-frequency bands, with the standard deviation of the first echo delay for three consecutive processing cycles being less than 0.5 milliseconds. When all the above conditions are met, a black light absorber rule triggering flag is generated, and the laser data is marked as low confidence due to insufficient echo intensity. A mean echo intensity less than 0.1 indicates that the laser normalized echo intensity is in the low reflection response range, and a variance of chromatic saturation less than 0.05 indicates that the target area has low saturation and low color fluctuation characteristics. A standard deviation of the first echo delay for three consecutive processing cycles less than 0.5 milliseconds indicates that the ultrasonic distance response is stable and can eliminate misjudgments caused by single echo noise. Among them, 0.5 milliseconds corresponds to a round-trip sound path change of about 8.6 cm, or a distance change of about 4.3 cm, which is within the acceptable distance fluctuation range for near-range obstacle avoidance.

[0106] The criteria for determining low-reflection soft bodies are as follows: the width of the main peak of the multi-pulse echo in the laser evidence group exceeds 1.5 times the width of the main peak of the standard diffuse reflection; the echo amplitude-frequency ratio in the ultrasonic evidence group is less than 0.5; and the steepness of the main peak of the echo envelope is less than 0.2. When all the above conditions are met, a low-reflection soft body rule triggering identifier is generated, and the laser data is marked as low confidence due to decreased distance accuracy. A main peak width exceeding 1.5 times is used to characterize the scattering broadening caused by the laser echo by soft or porous surfaces, and an echo amplitude-frequency ratio less than 0.5 is used to characterize the enhanced absorption of high-frequency ultrasound by rough soft materials. The steepness of the main peak of the echo envelope is the ratio of the rise slope of the echo to the normalized peak amplitude. When it is less than 0.2, it indicates that the rise slope of the echo is relatively gentle, which can characterize the diffuse scattering characteristics of the soft surface.

[0107] The criteria for identifying small, slender objects are as follows: the difference in echo amplitude between adjacent transducers in the ultrasonic evidence set is greater than 0.4; the area of ​​the visual candidate region is less than 5% of the area of ​​the forward region of interest; and the edge gradient density is greater than 0.2. In the laser evidence set, there is a narrow-angle strong echo point, and the echo intensity of this narrow-angle strong echo point is more than twice the neighborhood average, and the angle width is less than 3 degrees. When all the above conditions are met, a rule triggering identifier for small, slender objects is generated, and the ultrasonic data is marked as low confidence due to insufficient resolution. A difference in echo amplitude between adjacent transducers greater than 0.4 indicates that the target has a significant difference in response to different ultrasonic beam directions. A visual candidate region area of ​​less than 5% is used to distinguish targets such as cables and thin furniture legs from large areas of walls or cabinet doors. An edge gradient density greater than 0.2 indicates that the target edge is relatively clear. A narrow-angle strong echo point is used to characterize the local strong reflection characteristics of targets such as cables and thin furniture legs in laser scanning.

[0108] The above-mentioned low-confidence determination rule uses the satisfaction of all conditions as the high-confidence trigger condition; when the core condition is satisfied but some auxiliary conditions are missing, it is not directly used as a high-confidence trigger, but instead a medium-confidence rule identifier is generated and it enters the recalculation branch or conservative weight adjustment; for transparent bodies, specular bodies and black light-absorbing bodies, the laser features that can directly characterize the low-confidence modal anomaly are used as the core condition, and visual features and ultrasonic features are used as auxiliary conditions; for low-reflection soft bodies, the width of the laser main peak and the ultrasonic amplitude-frequency ratio are used as the core condition, and the steepness of the echo envelope main peak is used as an auxiliary condition; for small-sized slender objects, the difference in echo amplitude between adjacent ultrasonic transducers and the strong laser echo at a narrow angle are used as the core condition, and the visual area ratio and edge gradient density are used as auxiliary conditions;

[0109] The fusion confidence weights are constrained by the sum of the visual fusion weight, laser fusion weight, and ultrasonic fusion weight being 1, and are determined according to the principle of downweighting low-confidence sensing modes and upweighting dominant sensing modes. The initial weights can be selected as laser fusion weight 0.4, visual fusion weight 0.35, and ultrasonic fusion weight 0.25. This initial configuration is determined based on the stability of the three types of sensor data in a typical opaque obstacle scenario. Among them, laser ranging accuracy is usually higher than that of vision and ultrasound. Vision has a strong ability to recognize material texture and edge structure, while ultrasound has a supplementary perception ability for transparent hard objects and near-field obstacles. Sensing modes judged as low-confidence are not directly eliminated, but are retained with a low participation weight of 0.05 to 0.12 to avoid the judgment jump caused by completely excluding the mode in the boundary scene. The weight of sensing modes with stable response to the current material can be adjusted to 0.4 to 0.48, so that they play a dominant role in the fusion judgment.

[0110] When the transparent object rule is triggered, the laser fusion weight is adjusted to 0.05, the visual fusion weight to 0.47, and the ultrasonic fusion weight to 0.48; when the mirror object rule is triggered, the laser fusion weight is adjusted to 0.08, and the visual and ultrasonic fusion weights are each adjusted to 0.46; when the black light-absorbing object rule is triggered, the laser fusion weight is adjusted to 0.12, the visual fusion weight to 0.42, and the ultrasonic fusion weight to 0.46; when the low-reflectivity soft object rule is triggered, the laser fusion weight is adjusted to 0.3, the visual fusion weight to 0.4, and the ultrasonic fusion weight to 0.3; when the small, slender object rule is triggered, the ultrasonic fusion weight is adjusted to 0.1, and the visual and laser fusion weights are each adjusted to 0.45. All the above weight values ​​satisfy the constraint that the total weight is 1, and correspond to the physical response characteristics of the three types of sensor data under different material scenarios. The weight values ​​can be fine-tuned based on factory verification data or localized feedback samples, and the total weight remains 1 after fine-tuning.

[0111] If two or more low-confidence judgment rules are triggered simultaneously, the rule corresponding to the highest material score is used first for weight adjustment. If the difference between the material scores corresponding to multiple rules that are triggered simultaneously is less than 0.1, the recalculation branch is entered. The recalculation branch performs time averaging on the visual evidence score, laser evidence score, and ultrasonic evidence score of the same candidate obstacle in the last three processing cycles, and then re-executes the low-confidence judgment rule. The last three processing cycles are used to balance response speed and anti-transient interference capability. Under the commonly used 50-millisecond processing cycle, the three processing cycles correspond to about 150 milliseconds, which can filter out single-cycle noise without significantly delaying obstacle avoidance decision.

[0112] The main control unit generates rule confidence levels based on the triggering of low-confidence judgment rules. Rule confidence levels can be divided into high confidence, medium confidence, and low confidence. When all conditions in a rule are met and the highest material score of the corresponding material category is higher than 0.7, it is marked as high confidence and the target weight of the corresponding rule is adopted. When the core conditions in a rule are met but some auxiliary conditions are missing, or the highest material score is in the range of 0.6 to 0.7, it is marked as medium confidence and the average of the default weight and the target weight of the corresponding rule is used as the transition weight. When only a few conditions are met or the highest material score is lower than 0.6, it is marked as low confidence, and it is not directly adjusted to the target weight, but enters the recalculation branch.

[0113] For inconsistencies caused by sensor malfunctions, downgrading is performed according to the sensor health status identifier. When any sensing mode is determined to be unusable due to a health status malfunction, the fusion weight of that mode is reset to zero, and the remaining effective modes are normalized again according to the weight ratio before the malfunction. The criteria for visual malfunctions are: the average pixel value of the visual image is lower than 20 or higher than 230 for 5 consecutive processing cycles, or the pixel variance is lower than 5. An average pixel value lower than 20 indicates that the image is in a severely dark state, an average pixel value higher than 230 indicates that the image is in a severely overexposed state, and a pixel variance lower than 5 indicates that the image lacks effective texture or may have lens occlusion. 5 consecutive processing cycles are used to filter out single-frame illumination changes, which corresponds to about 250 milliseconds under the commonly used 50 millisecond processing cycle, and can characterize continuous visual malfunctions.

[0114] The criteria for determining laser anomalies are as follows: the effective echo rate of laser data is less than 30% for three consecutive processing cycles, or the laser rotation scanning frequency or scanning angular velocity deviates from the rated value by more than 10%. An effective echo rate of less than 30% indicates that the laser ranging unit lacks sufficient effective return in the forward or full circumference range, and short-term reflection loss can be filtered out for three consecutive processing cycles. When the scanning frequency or scanning angular velocity deviates from the rated value by more than 10%, it indicates that the laser scanning state has affected the accuracy of angle positioning. The criteria for determining ultrasonic anomalies are: the full-band echo amplitude of ultrasonic data is lower than the background echo amplitude for three consecutive processing cycles. Noise baseline plus safety margin; the background noise baseline can be determined by the average received signal when there are no nearby obstacles, and the safety margin can be twice the standard deviation of the background noise to avoid misjudgment caused by environmental noise; reduce the upper limit of the machine speed when the laser is abnormal, and disable the material judgment rule that relies on multi-band sweep frequency response when the ultrasound is abnormal; the abnormal recovery condition can be set to return to normal after 2 consecutive processing cycles; after recovery, the average of the weight before recovery and the weight after redistribution is used as the transition weight for 1 to 2 processing cycles, and then the weight after redistribution is switched to the fusion confidence weight to avoid sudden weight changes;

[0115] After completing the above processing, output the low-confidence sensing modality identifier, the adjusted fusion confidence weight, and the recalculation identifier for each candidate obstacle; the low-confidence sensing modality identifier includes at least the modality type, low-confidence reason, trigger rule identifier, and rule confidence level, etc. The above output results are used for subsequent fusion judgment and motion control.

[0116] Specifically, such as Figure 6 As shown: After receiving the adjusted visual fusion weight, laser fusion weight, ultrasonic fusion weight, candidate obstacle evidence group and modal quality label, the main control unit performs obstacle existence determination, obstacle position fusion, material category fusion, safety distance determination and motion control command generation for each candidate obstacle.

[0117] When determining the presence of an obstacle, the visual presence support is obtained by weighting the candidate region stability score, region saliency score, and edge continuity score; the laser presence support is obtained by weighting the effective distance echo ratio score, echo continuity score, and distance stability score; and the ultrasonic presence support is obtained by weighting the first echo presence score, echo repeatability score, and beam direction consistency score. Each presence support is normalized to the range of 0 to 1, with a larger value indicating that the corresponding mode supports the presence of an obstacle in that candidate direction. The internal weights of each item are determined based on the discrimination of the corresponding features against the obstacle presence state in the factory verification samples.

[0118] The total support with the presence of obstacles can be expressed as:

[0119]

[0120] Where wv, wl, and wu are the visual fusion weight, laser fusion weight, and ultrasonic fusion weight, respectively; Ev, El, and Eu are the visual presence support, laser presence support, and ultrasonic presence support, respectively; the total absence support Sn is obtained by weighting the absence support of each modality in the same way. The absence support of each modality can be taken as 1 minus the presence support of the corresponding modality, or it can be generated independently based on features such as no effective echo, unstable candidate region, and non-repeating echo; if Se is greater than Sn, and the difference between Se and Sn is not less than 0.1, then the candidate direction is determined to have an obstacle; if the difference between Se and Sn is less than 0.1, then the candidate direction is marked as a boundary determination; 0.1 is used as the presence support interval, which is consistent with the boundary interval between the highest and second highest scores in the material score, and is used to unify the confidence scale of presence determination and material determination;

[0121] During obstacle location fusion, visual distance, laser distance, and ultrasonic distance are weighted according to their respective fusion weights. Visual distance can be obtained from binocular parallax, monocular scale prior, or structured light-assisted data; laser distance is obtained from scanning distance data or echo delay data; ultrasonic distance is obtained from sound velocity and first echo delay, with sound velocity corrected for indoor temperature. If all three types of distances are valid, the fused distance df can be expressed as:

[0122]

[0123] Where dv, dl, and du represent visual distance, laser distance, and ultrasonic distance, respectively; if only two types of distance are valid, the fusion weights of the two valid modalities are renormalized before calculating the fusion distance; if only one type of distance is valid, the valid distance is used as the fusion distance, and the distance uncertainty level is increased.

[0124] Distance uncertainty can be determined based on the deviation between the distance of each effective mode and the fused distance. Preferably, the weighted average of the absolute values ​​of the differences between the distances of each mode and the fused distance is taken, i.e.:

[0125]

[0126] Where wk is the normalized fusion weight of the kth effective mode, dk is the distance of the kth effective mode, and df is the fusion distance; the distance uncertainty can be divided into three levels: low, medium, and high. The low uncertainty level corresponds to less than 5 cm, the medium uncertainty level corresponds to 5 cm to 10 cm, and the high uncertainty level corresponds to no less than 10 cm; if only one type of distance is valid, the distance uncertainty value is no less than 10 cm, or it is set to the high uncertainty level; 10 cm corresponds to the candidate obstacle merging distance threshold and the scale of conventional small obstacles, and is used to retain the necessary safety margin when determining the single-mode distance;

[0127] When fusing material categories, six material categories—transparent, specular, black light-absorbing, low-reflection soft, conventional opaque, and small-sized slender—are scored and fused separately. For any given material category, the fused material score can be expressed as:

[0128]

[0129] Pvm, Plm, and Pum represent the scores for the material category from the visual evidence group, laser evidence group, and ultrasonic evidence group, respectively. The main control unit uses the category corresponding to the highest score among the six fused material scores as the material determination category. If the highest fused material score is lower than 0.6, or the difference between the highest and second-highest fused material scores is less than 0.1, the material category is determined to be in a boundary state, and active exploration is triggered. 0.6 is the minimum acceptable score for material fusion determination, which is lower than the high confidence score threshold of 0.7, and is used to retain a certain margin of error during the fusion stage. 0.1 is used to distinguish between explicit classifications and boundary classifications, and is consistent with the existence support interval.

[0130] During active exploration, the cleaning robot moves 3 to 5 centimeters towards the candidate direction at a low speed of 0.05 to 0.1 meters per second, and re-acquires visual, laser, and ultrasonic data. This speed of 0.05 to 0.1 meters per second is lower than the conventional cleaning speed, allowing for braking margin when approaching the candidate target. The 3 to 5 centimeters distance is sufficient to change the visual observation angle, laser incident angle, and ultrasonic echo direction, while remaining less than the basic safety distance for most material scenarios, without significantly increasing the risk of collision. During active exploration, the exposure time of the visual sensor can be increased to 1.5 times its original value to enhance the image response of dark or low-texture targets and avoid overexposure. The laser ranging unit can continuously emit 5 to 10 ranging pulses in the candidate direction. Short-time averaging of echo results reduces random echo noise. When the cleaning robot is equipped with a multi-frequency ultrasonic transducer, the transducer switches to high-frequency sweep mode; when not equipped with a multi-frequency ultrasonic transducer, the number of repeated samplings in the standard frequency band is increased. After active exploration, the main control unit regenerates the evidence group based on the re-acquired data and re-performs existence determination, position fusion, and material category fusion. The number of active explorations can be set to no more than 3 times. If the scoring interval requirement is not met after 3 times, a conservative judgment is adopted, and the candidate direction is regarded as having an obstacle. The safe distance is determined according to the category corresponding to the highest fused material score. The number of active explorations is set to no more than 3 times to control the exploration time in boundary scenarios and avoid the robot repeatedly exploring the same candidate target for a long time.

[0131] The safety distance is determined based on the material classification, fusion distance uncertainty, and fuselage motion state. For transparent objects, the basic safety distance can be selected from 15 cm to 20 cm to address potential laser misses or weak visual edge responses caused by transparent glass, transparent plastic, and other targets. For mirrored objects, the basic safety distance can be selected from 15 cm to 20 cm, with an additional 5 cm to 8 cm lateral margin to compensate for directional deviations caused by specular reflection. For black light-absorbing objects, the basic safety distance can be selected from 8 cm to 12 cm to compensate for insufficient laser echo intensity caused by targets such as black furniture bases and black plastic. For low-reflectivity soft objects such as fabrics, plush materials, or the edges of soft pads—targets with low collision risk—the basic safety distance can be selected from [missing information]. For objects located between 3 and 5 centimeters, when they are in areas that the main brush, side brush, or drive wheel may contact, or when the boundaries are unstable and the contact risk cannot be confirmed, the safety distance should be determined according to the size of the object being small and slender or the size of the object being opaque. For small and slender objects, the basic safety distance can be 8 to 12 centimeters, with an additional longitudinal avoidance constraint of no less than 5 centimeters to reduce the risk of entanglement, jamming, or scratching caused by targets such as cables or furniture legs. For regular opaque objects, the basic safety distance can be 5 to 8 centimeters to take into account both regular detour and edge cleaning needs. The above distance range is matched with the radius of the robot body, braking distance, side brush coverage, edge cleaning needs, and collision risk of obstacles of different materials.

[0132] The final safe distance can be taken as the sum of the basic safe distance and twice the distance uncertainty. Using twice the distance uncertainty is because the multimodal distance difference can reflect the current ranging error expansion range, and the double expansion can retain additional avoidance margin in low-confidence scenarios. For dynamic obstacles, if the visual candidate region or laser point cloud cluster changes position for two consecutive processing cycles, and the relative velocity calculated based on the motion-compensated candidate obstacle position change is higher than 0.2 m / s, a short-term observation window of 5 processing cycles is activated to predict the position within the next 500 milliseconds to 1 second. The system is designed to provide a dynamic margin of 8 to 12 centimeters beyond the safe distance; its 0.2 m / s speed is higher than the conventional low-speed edge-following cleaning speed, which can be used to identify moving targets that affect the cleaning path; two consecutive processing cycles are used to filter out single-frame detection errors; five processing cycles correspond to approximately 250 milliseconds under the commonly used 50 millisecond processing cycle, which can form short-term motion trends; 500 milliseconds to 1 second matches the response time of the cleaning robot's deceleration, turning, and replanning; and the 8 to 12 centimeter dynamic margin is used to cover the position prediction error of short-term moving targets.

[0133] Motion control commands are generated based on the current cleaning path, obstacle position, material classification, and safety distance. The predetermined cleaning zone can be determined by the machine width, side brush coverage width, and the current path centerline. If the obstacle is outside the predetermined cleaning zone, the drive unit continues to execute the original cleaning path. If the obstacle is within the predetermined cleaning zone and the material is a low-reflection soft material, and the safety distance allows for close-range passage, the machine speed can be reduced to 0.1 m / s to 0.2 m / s and close-range edge cleaning can be performed. If the material is transparent, mirrored, black light-absorbing, small-sized slender, or conventional opaque, a detour trajectory is generated, ensuring that the detour trajectory maintains an interval of at least a safety distance from the obstacle throughout. 0.1 m / s to 0.2 m / s is the low-speed edge-tracing control speed range, which can reduce contact impact and improve close-range sampling stability.

[0134] The obstacle avoidance trajectory prioritizes paths with shorter arc lengths and gentler curvature changes. The turning radius can be selected to be no less than 1.5 times the fuselage radius to reduce slippage, sudden changes in angular velocity, and attitude errors caused by sharp turns. The obstacle avoidance motion control commands received by the drive unit include at least the target linear velocity, target angular velocity, detour direction, safety distance, material type identifier, and retreat conditions. If the fusion distance is detected to be less than 80% of the safety distance again during the detour, a warning state is entered and the fuselage speed is reduced. 80% is used as the warning threshold to prevent approaching the safety boundary. Forward deceleration; if the fusion distance is detected to be less than 60% of the safe distance for two consecutive processing cycles, an emergency avoidance state is entered, the aircraft stops moving forward and retreats 3 to 5 centimeters before re-investigating; 60% is used as an emergency threshold to identify a state where the safe distance has been clearly intruded, two consecutive processing cycles are used to eliminate single-cycle ranging noise, and retreating 3 to 5 centimeters can allow the aircraft to leave the near-field misjudgment area and restore the sensor observation distance; if visual data, laser data and ultrasonic data are abnormal at the same time, the drive unit controls the aircraft to stop in place and issues a manual takeover prompt;

[0135] After completing an obstacle avoidance maneuver, the main control unit writes the material type, three-mode feature vector, fusion confidence weight, safety distance, trajectory, whether a collision occurred, whether a stuck situation occurred, and whether an emergency stop was triggered to the task log. If the cleaning robot does not experience a collision, stuck situation, or abnormal stop within 1 to 2 meters after the obstacle avoidance, the record is considered a positive sample. The 1 to 2 meter range typically covers the short observation area where the robot re-enters the original cleaning path after completing the obstacle avoidance. If a collision, stuck situation, or emergency stop occurs, the record is considered a negative sample. Positive and negative samples are used to generate or update localized correction parameters during the charging standby phase. The conditions can be set as follows: at least 50 positive samples or at least 3 negative samples. 50 positive samples can cover multiple distances, angles, and lighting conditions, reducing the randomness of a single successful detour sample. 3 negative samples are fewer in number but have higher risk, which can trigger correction earlier. The change in template parameters in a single instance should not exceed 10% of the corresponding factory parameters or current parameters. Template parameters should include at least the feature mean, feature variance, and discrimination weight. The weight of negative samples can be 5 to 10 times that of positive samples to improve the impact of high-risk abnormal events on template parameter correction. The above task logs are used to form localized correction parameters and are loaded and called during the subsequent initialization of basic parameters.

[0136] Example 2: Based on Example 1, the specific application process of a cleaning robot obstacle avoidance method integrating vision, laser, and ultrasonic perception is further explained:

[0137] Specifically, such as Figure 7 As shown: The following example illustrates a cleaning robot performing a cleaning task in the area connecting the living room and balcony of a home. This area contains obstacles of various materials, such as transparent glass sliding doors, mirrored cabinet doors, black TV cabinet bases, thin charging cables on the floor, and the edges of fabric cushions. The cleaning robot is equipped with a forward-facing vision sensor, a laser ranging unit, an ultrasonic transducer array, an inertial measurement unit, a wheeled odometer, and a drive unit. Before starting the cleaning task, the main control unit loads the body coordinate system, a unified timestamp, sensor extrinsic calibration parameters, and a material reflection feature template library. The material reflection feature template library pre-stores visual, laser, and ultrasonic reference features of transparent bodies, mirrored bodies, black light-absorbing bodies, low-reflection soft bodies, regular opaque bodies, and small slender bodies, and allows the overlay of localized correction parameters formed by historical cleaning tasks.

[0138] After the cleaning robot leaves its charging dock, it performs a straight-line sweep along the living room floor at a speed of 0.35 m / s. The main control unit organizes multimodal data acquisition with a sliding processing cycle of 50 milliseconds. In each processing cycle, it receives forward images, laser distance sequences, laser echo intensity sequences, ultrasonic echo delays, ultrasonic echo amplitudes, body angular velocity, body acceleration, and left and right wheel speeds. Since the sampling times of visual frames, laser scans, and ultrasonic echoes are not completely consistent, the main control unit uses the start time of the current processing cycle as a reference time. Combining the pose changes obtained from the wheel odometer and inertial measurement unit, it maps the visual candidate region, laser candidate direction, and ultrasonic candidate beam to the body coordinate system under the same reference time. If a sensor does not generate new complete data in the current cycle, it calls the most recent valid data in the data buffer and adds a data timeliness identifier based on the time difference between the sampling time and the reference time.

[0139] When the cleaning robot moves to a position approximately 1.2 meters from the sliding glass door of the balcony, the forward-facing vision sensor detects a weakly textured area on the right front of the robot. The edge gradient density of this area is 0.06, and the stereo matching confidence level is 0.32. The laser ranging unit detects continuous invalid echoes in the same direction, with an echo missing rate of 0.72, and the distribution of valid echo points is discontinuous. The ultrasonic transducer array detects stable first and second echoes in adjacent beam directions, with an echo amplitude-to-frequency ratio of 0.94. The main control unit takes this direction as the candidate obstacle direction and classifies the visual candidate area, the laser invalid echo section, and the ultrasonic valid beam into the same candidate obstacle record. Based on the material reflection feature matching results, the visual evidence group scores 0.78 for transparent objects, the laser evidence group scores 0.81 for transparent objects, and the ultrasonic evidence group has a support level higher than 0.7 for transparent hard surfaces. At the same time, the distance output of the laser data is unstable. Therefore, the candidate target is determined to be a transparent object, and the laser data is marked as a low-confidence sensing mode.

[0140] During the transparent target processing, the main control unit does not use laser distance as the primary obstacle avoidance criterion. Instead, it reduces the laser fusion weight from the initial weight of 0.4 to 0.05, adjusts the visual fusion weight to 0.47, and adjusts the ultrasonic fusion weight to 0.48. Subsequently, the main control unit performs fusion judgment based on the direction of the visual candidate area, the ultrasonic echo distance, and a small number of effective laser boundary points. It finds that the boundary of the glass sliding door is located about 1.15 meters to the right front of the fuselage, with a fusion distance uncertainty of 6 centimeters. Since the material type is determined to be transparent, the basic safety distance is taken as 18 centimeters, and the final safety distance is taken as the basic safety distance plus twice the distance uncertainty, which is about 30 centimeters. Based on this, the main control unit generates a right-side deceleration and detour trajectory, so that the fuselage passes along the edge of the glass door at an interval of not less than 30 centimeters, and reduces the target linear velocity from 0.35 m / s to 0.18 m / s. The target angular velocity is output to the drive unit in real time according to the curvature of the detour trajectory.

[0141] After bypassing the glass sliding door, the cleaning robot continued moving towards the living room TV cabinet. At this point, the laser ranging unit detected a low-intensity echo 0.65 meters ahead, with a normalized echo intensity mean of 0.08. The visual sensor detected a black, elongated furniture base with a chromatic saturation variance of 0.04. The ultrasonic transducer array obtained stable first echoes in both low and high frequency bands, with the standard deviation of the first echo delay less than 0.5 milliseconds for three consecutive processing cycles. Based on these characteristics, the main control unit identified the candidate target as a black light-absorbing body and marked the laser data as an echo. Low reliability due to insufficient intensity; since the black light absorber can still provide relatively stable positional support from visual contours and ultrasonic echoes, the main control unit adjusts the laser fusion weight to 0.12, the visual fusion weight to 0.42, and the ultrasonic fusion weight to 0.46, and determines the safe distance to be 10 cm plus twice the distance uncertainty; if the fusion distance uncertainty is 3 cm, then the final safe distance is 16 cm; the drive unit performs low-speed detour based on this safe distance to avoid misjudging obstacles ahead as passable areas due to excessively low laser echo intensity;

[0142] When the cleaning robot approaches the coffee table area, a thin, elongated linear region appears in the visual image. This region accounts for 3% of the forward region of interest, with an edge gradient density of 0.24. The laser ranging unit detects a narrow-angle strong echo point in this direction, with an echo intensity more than twice the neighborhood average and an angle width of less than 3 degrees. The echo amplitude difference between adjacent transducers in the ultrasonic transducer array is 0.46. The main control unit identifies this target as a small, slender object and marks the ultrasonic data as low-confidence due to insufficient resolution. The main control unit adjusts the ultrasonic fusion weight to 0.1, and the visual fusion weight and laser fusion weight to 0.45 respectively. Based on the visual linear region and the narrow-angle laser echo point, the lateral position of the cable is determined. For this cable target, a basic safety distance is set to 10 cm, with an additional longitudinal avoidance constraint of no less than 5 cm to reduce the risk of the side brush or drive wheel getting tangled in the cable. If the current cleaning path would cause the side brush to pass through the cable location, the main control unit generates a detour trajectory instead of performing edge cleaning.

[0143] As the cleaning robot continues to enter the edge area of ​​the fabric sofa, the visual image detects a low-texture soft boundary. The width of the laser multi-pulse echo peak exceeds 1.5 times the width of the standard diffuse reflection peak, the ultrasonic echo amplitude-frequency ratio is less than 0.5, and the echo envelope peak steepness is less than 0.2. Based on this, the main control unit classifies the target as a low-reflection soft object. If the soft object is located on the outer side of the bottom of the sofa and is not in an area where the main brush, side brush, or drive wheel may get caught, the main control unit sets the basic safety distance to 4 cm and reduces the robot speed to 0.15 m / s to perform close-range edge cleaning to reduce missed areas around obstacles. If the soft object boundary is detected to be unstable in subsequent processing cycles, or if the target extends into an area where the main brush, side brush, or drive wheel may contact, the main control unit does not adopt the close-range edge strategy but redetermines the safety distance according to the conventional opaque object or small slender object.

[0144] During the obstacle handling process, if the difference in the highest material scores among the visual evidence group, laser evidence group, and ultrasonic evidence group is less than 0.1, or the highest fused material score is less than 0.6, the main control unit initiates active exploration. During active exploration, the cleaning robot moves 3 to 5 centimeters in the candidate direction at a speed of 0.05 m / s to 0.1 m / s. The visual sensor increases the exposure time to 1.5 times the original value, the laser ranging unit continuously emits 5 to 10 ranging pulses in the candidate direction, and the ultrasonic transducer switches to high-frequency sweep mode when configured with multi-frequency hardware. After active exploration, the main control unit regenerates the visual evidence group, laser evidence group, and ultrasonic evidence group, and re-executes the existence determination, position fusion, and material category fusion. If a stable category judgment cannot be formed after no more than 3 active explorations, the main control unit adopts a conservative judgment, considers the candidate direction as having an obstacle, and determines the safe distance according to the category corresponding to the current highest fused material score.

[0145] During obstacle avoidance trajectory execution, the main control unit continuously monitors the relationship between the fusion distance and the safe distance. If the fusion distance is detected to be less than 80% of the safe distance again, the drive unit enters a warning state and reduces the robot's speed. If the fusion distance is detected to be less than 60% of the safe distance for two consecutive processing cycles, the drive unit stops moving forward and controls the robot to retreat 3 to 5 centimeters before re-examining. If visual data, laser data, and ultrasonic data are all abnormal at the same time, the drive unit controls the robot to stop in place and issues a manual intervention prompt. Through the above processing, the cleaning robot can dynamically determine the obstacle avoidance strategy based on the current material reflection characteristics and the reliability status of each sensor modality in complex material scenarios such as transparent bodies, mirror bodies, black light-absorbing bodies, small slender bodies, and low-reflection soft bodies.

[0146] After completing the cleaning of the living room to balcony area, the main control unit writes the material type, three-mode feature vector, fusion confidence weight, safe distance, motion trajectory, whether a collision occurred, whether a jam occurred, and whether an emergency stop was triggered for each candidate obstacle into the task log. The task log includes at least the following fields: candidate obstacle identifier, material type, candidate location, three-mode feature vector, fusion confidence weight, safe distance, motion trajectory, collision identifier, jam identifier, and emergency stop identifier. If the robot does not experience a collision, jam, or abnormal stop within 1 to 2 meters after navigating around the obstacle, the corresponding record is taken as a positive sample; if a collision, jam, or emergency stop occurs, the corresponding record is taken as a negative sample. During the charging standby phase, the main control unit generates localized correction parameters based on positive and negative samples. When there are at least 50 positive samples or at least 3 negative samples, the mean, variance, and discrimination weight of the features in the material reflection feature template library are updated with a limited amplitude. The change in template parameters at one time does not exceed 10% of the corresponding factory parameters or current parameters, and the weight of negative samples is 5 to 10 times that of positive samples. As a result, when cleaning the same home environment in the future, the cleaning robot can more accurately identify the reflection features of specific glass doors, mirror cabinet doors, black furniture bases, thin cables, and the edges of fabric cushions in the home, and execute corresponding fusion obstacle avoidance control according to the recognition results.

[0147] It should be noted that this invention can be deployed on the device itself to realize embedded applications, or it can run on a PC or other terminal with a user interface, thereby meeting various hardware environments and usage requirements.

[0148] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented in software, the above embodiments can be implemented in whole or in part by a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions of the embodiments of this application are implemented in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted wirelessly or wiredly from one website, computer, server, or data center to another website, computer, server, or data center. Wired methods include optical fiber, twisted pair, coaxial cable, etc. Wireless methods include infrared, microwave, etc. Available media include any available media that can be accessed by a computer or data storage devices such as servers and data centers that contain one or more sets of available media. Available media can be magnetic media (floppy disks, hard disks, magnetic tapes), optical media (DVDs), or semiconductor media. Semiconductor media can be solid-state drives.

[0149] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0150] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for obstacle avoidance in a cleaning robot that integrates vision, laser, and ultrasonic sensing, characterized in that, include: S1. Establish the body coordinate system, sensor extrinsic calibration parameters, and unified clock reference for the cleaning robot, and load the material reflection feature template library; S2. Simultaneously collect visual data, laser data, ultrasonic data, and body motion data during the movement of the cleaning robot, and align the visual data, laser data, and ultrasonic data to the same spatiotemporal reference based on the body motion data; S3. Extract the material reflection features corresponding to the candidate obstacles from the visual data, laser data and ultrasonic data respectively, and match the material reflection features with the material reflection feature template library; S4. When visual data, laser data, and ultrasonic data are inconsistent in terms of obstacle location or material type, identify low-confidence sensing modalities based on material reflection characteristics and adjust the fusion confidence weights of visual data, laser data, and ultrasonic data. S5. Based on the adjusted fusion confidence weight, the obstacle position and material type are fused and determined, and a safe distance is determined according to the fusion determination result. The obstacle avoidance motion control command is then sent to the drive unit of the cleaning robot.

2. The obstacle avoidance method for a cleaning robot integrating vision, laser, and ultrasonic perception according to claim 1, characterized in that, Establish the body coordinate system, sensor extrinsic calibration parameters, and unified clock reference for the cleaning robot, and load the material reflection feature template library, including: Basic parameter initialization is performed in the cleaning preparation state. The body coordinate reference is established with the geometric center of the machine body and the direction of travel, and the external parameter calibration parameters of the vision, laser and ultrasonic sensors are read and verified. Add timestamps to common time sources and compensate for sampling delays; When an external parameter verification error occurs, a valid calibration parameter is invoked. When time deviation is abnormal, a time quality label is attached, and a material reflection feature template library with multiple types of material visual, laser and ultrasonic reference features is loaded and stored; Call the general template or overlay localized correction parameters based on the template validation results.

3. The obstacle avoidance method for a cleaning robot integrating vision, laser, and ultrasonic perception according to claim 1, characterized in that, During the cleaning robot's movement, visual data, laser data, ultrasonic data, and body motion data are collected simultaneously, including: During the data acquisition process, the data acquisition and buffering of the vision sensor, laser ranging unit, ultrasonic transducer array, inertial measurement unit and wheeled odometer are organized according to the sliding processing cycle. For sensor data that has not been updated in the current period, retrieve the most recent sensor data with a valid timestamp from the data buffer; For low-frequency scan data, retrieve the most recent complete scan data; Based on the time difference between the sampling time and the reference time, data integrity, and sensor operating status, data timeliness identifier, data quality identifier, and sensor health status identifier are generated.

4. The obstacle avoidance method for a cleaning robot integrating vision, laser, and ultrasonic perception according to claim 1, characterized in that, Based on airframe motion data, visual data, laser data, and ultrasonic data are aligned to the same spatiotemporal reference, including: During spatiotemporal alignment, the start time of the sliding processing cycle is used as the reference time, and the changes in fuselage attitude are cross-verified based on the inertial measurement unit and wheeled odometer. By combining the sensor extrinsic calibration parameters, the visual candidate region, laser scanning point, and ultrasonic echo direction are mapped to the body coordinate system; The data reliability level is marked based on the data timeliness, static feature verification results, and abnormal fuselage movement status.

5. The obstacle avoidance method for a cleaning robot integrating vision, laser, and ultrasonic perception according to claim 1, characterized in that, Material reflection features corresponding to candidate obstacles were extracted from visual data, laser data, and ultrasonic data, respectively, including: During material feature extraction, the main control unit establishes a record of candidate obstacles within the forward region of interest in the body coordinate system; Candidate regions are generated based on the stability of visual regions; Candidate directions are generated based on the laser echo segments; Candidate beam data are generated based on the ultrasonic beam response; Based on the multimodal candidate data of the same obstacle associated with angle difference and distance difference, visual edge, texture and specular features, laser echo intensity and continuity features, and ultrasonic echo amplitude and frequency and beam response features are extracted respectively.

6. The obstacle avoidance method for a cleaning robot integrating vision, laser, and ultrasonic perception according to claim 1, characterized in that, Matching material reflection features with a material reflection feature template library, including: During template matching, the main control unit normalizes the material reflection characteristics of vision, laser, and ultrasound. The weighted distance of each material category is calculated based on the discrimination weights in the template library. The weighted distance is converted into a matching score, generating visual evidence groups, laser evidence groups, and ultrasonic evidence groups corresponding to the scores of multiple material categories. When any modal data is missing, the corresponding modal state is marked, and the valid modal scores are normalized.

7. The obstacle avoidance method for a cleaning robot integrating vision, laser, and ultrasonic perception according to claim 1, characterized in that, When visual data, laser data, and ultrasonic data are inconsistent regarding obstacle location or material type, low-confidence sensing modalities are identified based on material reflection characteristics, including: When recognizing low-confidence sensing modalities, the determination is triggered based on the positional deviation and material score difference of the candidate obstacle. Based on the laser echo loss, echo intensity fluctuation, visual texture change, ultrasonic amplitude-frequency response, and beam response characteristics corresponding to transparent bodies, specular bodies, black light-absorbing bodies, low-reflection soft bodies, and small-sized slender bodies, the low confidence type and regular confidence level of the corresponding sensing modes are marked according to the satisfaction of core conditions and auxiliary conditions.

8. The obstacle avoidance method for a cleaning robot integrating vision, laser, and ultrasonic perception according to claim 1, characterized in that, Adjust the confidence weights for the fusion of visual, laser, and ultrasonic data, including: When adjusting the fusion confidence weight, the sum of the visual fusion weight, laser fusion weight, and ultrasonic fusion weight is used as a constraint; Low-confidence sensing modes are downweighted, while sensing modes with stable material responses are upweighted. Select the target weight, transition weight, or recalculate the branch according to the confidence level of the rule, and after resetting the abnormal mode weight to zero when the sensor health status is abnormal, renormalize the available mode weights.

9. The obstacle avoidance method for a cleaning robot integrating vision, laser, and ultrasonic perception according to claim 1, characterized in that, The obstacle location and material category are fused and determined based on the adjusted fusion confidence weights, including: During the fusion determination, the main control unit takes the candidate obstacle as the object and performs obstacle existence determination, fusion distance calculation and multi-material scoring fusion in sequence according to visual fusion weight, laser fusion weight and ultrasonic fusion weight; When only single-modal or dual-modal data is valid, the fusion weights corresponding to the valid modes are renormalized, and active exploration or conservative judgment is triggered based on the existence of support interval, material score interval or distance uncertainty.

10. The obstacle avoidance method for a cleaning robot integrating vision, laser, and ultrasonic perception according to claim 1, characterized in that, Based on the fusion judgment result, a safe distance is determined, and obstacle avoidance motion control commands are issued to the drive unit of the cleaning robot, including: During motion control, the main control unit determines the basic safety distance based on the material classification and determines the target safety distance by combining the distance uncertainty and the fuselage motion state; Based on the cleaning operation zone, obstacle location, and material type, control commands such as deceleration, detour, edge-keeping, reversal, or stop are generated. The obstacle avoidance process data is written to the task log, and localized correction parameters are generated based on positive and negative samples.

Images