Cleaning robot autonomous operation method and system based on multi-sensor fusion and reinforcement learning

Abstract

The invention discloses a cleaning robot autonomous operation method and system based on multi-sensor fusion and reinforcement learning. The method comprises the following steps: collecting visual, physical and fuzzy modal information through a multi-sensor system; performing fusion processing on the multi-modal information by using a deep learning model and a hesitant fuzzy set, and constructing a unified environment state representation; a hierarchical reinforcement learning framework is established based on the representation, a cleaning area is divided into a plurality of task units, a high-level strategy selects the task unit, and a bottom-level strategy executes a specific cleaning action; designing a multi-target reward function, comprehensively considering indexes such as cleaning coverage rate and energy consumption efficiency, and introducing a dynamic weight adjustment mechanism; adopting an option-evaluator framework to jointly train a high-level strategy and a bottom-level strategy; and dynamically adjusting the decision model through online learning and a local optimization algorithm in the operation process. According to the invention, closed-loop control from environment perception to task execution is realized, and the autonomy, adaptability and operation efficiency of the cleaning robot in a complex environment are improved.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robot technology, specifically relating to an autonomous operation method and system for a cleaning robot based on multi-sensor fusion and reinforcement learning. Background Technology

[0002] With the continuous development of robotics technology, intelligent cleaning robots have been widely used in various scenarios such as homes, offices, and industries. However, most existing cleaning robots still face significant challenges in terms of autonomy, adaptability, and efficiency when operating in complex, dynamic, and unstructured environments.

[0003] Traditional robotic vacuum cleaners often rely on pre-programmed paths or simple reactive control strategies, such as random collisions or edge cleaning. These methods have limited environmental perception capabilities and lack a deep understanding of the environment, leading to problems such as unreasonable path planning, low cleaning coverage, excessive repetitive cleaning, and high energy consumption. Although some advanced models have introduced single sensors (such as cameras or LiDAR) for navigation and obstacle avoidance, their perception reliability drops sharply under uncertainties such as changes in lighting, diverse ground materials, and the presence of dynamic obstacles, making it difficult to achieve accurate environmental modeling and state assessment.

[0004] In recent years, reinforcement learning technology has provided new insights into the autonomous decision-making of robots, enabling them to learn optimal strategies through interaction with their environment. However, directly applying reinforcement learning to cleaning robots presents challenges such as a huge state space, complex action space, sparse rewards, and low training efficiency. A single, flat reinforcement learning framework struggles to effectively handle the inherent hierarchical nature of cleaning tasks—namely, the long-term global task planning and short-term local action execution. Furthermore, designing a reward function that balances multiple objectives such as cleaning coverage, energy consumption, and repetitive cleaning rate remains a key challenge.

[0005] In addition, existing systems typically lack online learning capabilities. Once deployed, their strategies remain fixed and cannot adjust and optimize themselves according to environmental changes during operation, leading to gradual performance degradation over long-term operation.

[0006] Therefore, there is an urgent need in this field for an autonomous operation method for cleaning robots that can comprehensively utilize information from multiple sensor sources, possess hierarchical decision-making capabilities, and can adaptively optimize online, so as to comprehensively improve their intelligence level and operational efficiency in complex real-world environments. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for autonomous operation of a cleaning robot that can comprehensively improve its perception capabilities, decision-making intelligence and operational efficiency in complex and dynamic environments.

[0008] In a first aspect, embodiments of this application provide a method for autonomous operation of a cleaning robot based on multi-sensor fusion and reinforcement learning, the method comprising: Multimodal information about the cleaning environment is collected through a multi-sensor system, including visual modal information, physical modal information, and fuzzy modal information; The multimodal information is fused to construct a unified environmental state representation, including: using a deep learning model to extract features from the visual modal information to identify the workable area, obstacle boundaries, and pollution level of the cleaning area; modeling the fuzzy modal information based on hesitant fuzzy sets to handle environmental uncertainty factors; and fusing the extracted visual features, physical sensor data, and fuzzy modal information to generate a multidimensional environmental state representation. Based on the environmental state representation, a hierarchical reinforcement learning framework is established, including: dividing the cleaning area into multiple task units, with each task unit serving as an option in the high-level task planning layer; and in the low-level action execution layer, executing specific cleaning actions based on the environmental state representation. Design a multi-objective reward function that comprehensively considers cleaning coverage, energy efficiency, re-cleaning rate and cleanliness of key areas, and introduces a dynamic weight adjustment mechanism; An option-evaluator framework is used to jointly train high-level task planning strategies and low-level action execution strategies. The high-level strategy selects the optimal task unit based on the environmental state representation. The low-level strategy outputs cleaning path planning, task intensity adjustment and obstacle avoidance strategies within the selected unit. During the operation, the decision model is dynamically adjusted through online learning and local optimization algorithms to adapt to environmental changes and improve operational efficiency and robustness. The method of jointly training high-level option policies and low-level action policies using an option-judge architecture includes: Construct an end-to-end option-judge training framework, where the judge module is responsible for evaluating the high-level option value function. and underlying action value function ; The high-level actor module uses an inter-option strategy. Generate task unit selection decisions; The underlying actor module uses internal strategies Generate specific cleaning actions; Design a hierarchical gradient update mechanism, including: gradient updates of higher-level option policies based on the option dominance function. Through the formula: , in, High-level option strategy The parameters, The objective function of the high-level strategy. :gradient; The gradient update of the underlying action policy is based on the action advantage function. Through the formula: , in, Underlying action strategy The parameters, The objective function of the underlying strategy; Adaptive optimization of the option termination function is achieved through the gradient formula: , in, : Option to terminate function The parameters, The objective function of the terminating function. : Current state and next state Options : Low-level actions.

[0009] Secondly, embodiments of this application provide an autonomous operation system for a cleaning robot based on multi-sensor fusion and reinforcement learning, applied to the autonomous operation method for a cleaning robot based on multi-sensor fusion and reinforcement learning as described in the first aspect, the system comprising: The multimodal information acquisition module is used to acquire visual modal information, physical modal information and fuzzy modal information of the cleaning environment through a multi-sensor system; An environment state representation construction module is used to fuse the multimodal information and construct a unified environment state representation, including: The visual feature extraction unit uses a deep learning model to extract features from visual modal information and identify the workable area, obstacle boundaries, and contaminated area distribution. The fuzzy information modeling unit models fuzzy modal information based on hesitant fuzzy sets, handling environmental uncertainty factors; The multi-source data fusion unit fuses extracted visual features, physical sensor data, and fuzzy modal information to generate a multi-dimensional environmental state representation; A hierarchical reinforcement learning decision module, used to establish a hierarchical reinforcement learning framework based on the environmental state representation, includes: The task unit division divides the cleaning area into multiple task units, with each task unit serving as an option in the higher-level task planning layer. The action execution unit performs specific cleaning actions based on the environmental state representation at the bottom action execution layer; The multi-objective reward optimization module is used to design multi-objective reward functions, which comprehensively consider cleaning coverage, energy efficiency, re-cleaning rate and cleanliness of key areas, and introduces a dynamic weight adjustment mechanism. The policy training module is used to jointly train high-level task planning policies and low-level action execution policies using an option-judge framework, including: The high-level strategy unit selects the optimal task unit based on the environmental state representation. The underlying strategy unit outputs cleaning path planning, work intensity adjustment, and obstacle avoidance strategies within the selected unit; The online learning optimization module is used to dynamically adjust the decision model during the operation through online learning and local optimization algorithms to adapt to environmental changes and improve operation efficiency and robustness. The multimodal information acquisition module, environmental state representation construction module, hierarchical reinforcement learning decision-making module, multi-objective reward optimization module, strategy training module, and online learning optimization module are connected in sequence to form a complete autonomous operation control loop.

[0010] Thirdly, embodiments of this application provide an electronic device, including: processor; Memory used to store processor-executable instructions; The processor is configured to implement the autonomous operation method for a cleaning robot based on multi-sensor fusion and reinforcement learning as described in the first aspect when executing the instructions.

[0011] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program that instructs a device to execute the autonomous operation method for a cleaning robot based on multi-sensor fusion and reinforcement learning as described in the first aspect.

[0012] The beneficial effects of this invention are as follows:

[0013] 1. More comprehensive and accurate perception: By using multi-source sensor fusion and hesitant fuzzy set theory, environmental uncertainty is effectively handled, and a more accurate and robust representation of the environmental state is constructed.

[0014] 2. Smarter and more efficient decision-making: By adopting a hierarchical reinforcement learning framework, complex tasks are decomposed, and the option-evaluator architecture enables efficient collaboration between global planning and local execution, thereby improving decision-making efficiency.

[0015] 3. More balanced optimization of objectives: Through a multi-objective reward function with dynamic weight adjustment, the robot can autonomously balance multiple performance indicators such as coverage and energy consumption to achieve optimal overall efficiency.

[0016] 4. The system is more adaptable: It has the ability to learn online and optimize locally, enabling the robot to adapt to environmental changes in real time and continuously improve itself, ensuring the robustness and efficiency of long-term operation.

[0017] This invention forms a complete autonomous operation closed loop from environmental perception to task execution, which significantly improves the intelligence level and operation performance of cleaning robots in complex scenarios. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of a cleaning robot autonomous operation method based on multi-sensor fusion and reinforcement learning, provided as an embodiment of this application.

[0019] Figure 2 This is a diagram illustrating the architecture of an autonomous cleaning robot system based on multi-sensor fusion and reinforcement learning, provided for this application.

[0020] Figure 3 A schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0022] It should be noted that in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0023] Based on the embodiments described in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] Example 1

[0025] Figure 1 This is a schematic diagram illustrating a method for autonomous operation of a cleaning robot based on multi-sensor fusion and reinforcement learning, provided in an embodiment of this application. Figure 1 As shown, a method for autonomous operation of a cleaning robot based on multi-sensor fusion and reinforcement learning includes: S110. Collect multimodal information about the cleaning environment through a multi-sensor system. This multimodal information includes visual modal information, physical modal information, and fuzzy modal information. The multi-sensor system refers to a collection of various sensors mounted on the robot. Multimodal information refers to environmental data from different sources and of different types. This allows for comprehensive environmental perception, providing raw data for subsequent decision-making. Different types of sensors are responsible for perceiving different aspects of the environment. Visual modal information is collected by a camera. For example, it identifies a carpet (workable area) and a chair (obstacle) on the ground. Physical modal information is collected by LiDAR, IMU, and dust sensors. For example, LiDAR measures the precise distance to the chair as 0.5 meters; the dust sensor detects high dust concentration in a certain area. Fuzzy modal information is collected by a ground material sensor and a light sensor. For example, the light sensor detects that the area in front is dark, resulting in an unclear camera image, which is an uncertainty factor.

[0026] S120. The multimodal information is fused to construct a unified environmental state representation, including: using a deep learning model to extract features from the visual modal information, identifying the workable area, obstacle boundaries, and pollution level of the cleaning area; modeling the fuzzy modal information based on hesitant fuzzy sets to handle environmental uncertainties; and fusing the extracted visual features, physical sensor data, and fuzzy modal information to generate a multi-dimensional environmental state representation. The environmental state representation is a comprehensive data structure that a computer can understand and process, used to describe the overall environmental condition of the robot. This transforms the chaotic and heterogeneous raw sensor data into a unified, structured, and semantically rich environmental description. This is a crucial step from perception to cognition. Deep learning feature extraction is used to analyze camera images using convolutional neural networks, outputting feature vectors showing the floor on the left, the carpet on the right, and the outline of an obstacle in the center. Hesitant fuzzy set modeling is used to handle the uncertainty of very low light. Instead of simply judging whether it is bright or dark, a membership set is given, such as {(dark, 0.8), (medium, 0.15), (bright, 0.05)}, indicating that it is likely dark. By fusing visual features (carpet area), physical data (high dust concentration), and fuzzy assessment (high ground flatness), an environmental state vector is finally generated, for example: [coordinate X, coordinate Y, terrain = carpet, pollution level = 0.8, illumination confidence level = 0.7, obstacle distance = 0.5].

[0027] S130. Based on the environmental state representation, a hierarchical reinforcement learning framework is established, including: dividing the cleaning area into multiple task units, each task unit serving as an option in the high-level task planning layer; and executing specific cleaning actions in the low-level action execution layer based on the environmental state representation. The hierarchical reinforcement learning framework is a learning framework that decomposes complex tasks into different levels. The high-level task planning layer is responsible for macroscopic, long-term decisions on what to do. The low-level action execution layer is responsible for microscopic, short-term decisions on how to do it. Options are the basic units of high-level decision-making, representing an executable sub-task. The complex task of cleaning the entire room is decomposed into sub-tasks such as cleaning area A first, then area B, and these are handled by agents at different levels, reducing learning difficulty and improving efficiency. For example, the high-level layer divides the room into task units (options) such as the living room area, the area under the dining table, and the corridor area based on the global map. If the current environmental state shows a high level of contamination in the living room area, the high-level strategy selects the option to clean the living room area. In the "Living Room Area" option, the bottom layer strategy outputs specific cleaning actions based on the local environment (such as bypassing a coffee table), such as moving forward 0.3 meters and adjusting the suction power to the strong setting.

[0028] S140. Design a multi-objective reward function that comprehensively considers cleaning coverage, energy efficiency, repetitive cleaning rate, and cleanliness of key areas, and introduces a dynamic weight adjustment mechanism. The multi-objective reward function is a comprehensive scoring standard used to evaluate the quality of a robot's action or strategy. The dynamic weight adjustment mechanism is a method to automatically adjust the importance of different objectives based on the situation. This guides the robot's learning direction, ensuring its behavior meets multiple requirements (clean sweeping, energy saving, no repetition), and allows for flexible adjustment of the focus as needed.

[0029] S150. An option-judge framework is used to jointly train a high-level task planning strategy and a low-level action execution strategy. The high-level strategy selects the optimal task unit based on the environmental state representation; the low-level strategy outputs cleaning path planning, task intensity adjustment, and obstacle avoidance strategies within the selected unit. The option-judge framework is a specific algorithmic architecture used to train hierarchical reinforcement learning models. The judge is responsible for evaluating the value of a state or action, i.e., predicting the future score. The actor is responsible for executing the strategy, i.e., making decisions based on the current state. Simultaneously and collaboratively, the decision-making capabilities of the high-level and low-level frameworks are optimized. The high-level framework learns to select more valuable sub-tasks, and the low-level framework learns to execute actions more efficiently within sub-tasks.

[0030] S160. During operation, the decision-making model is dynamically adjusted through online learning and local optimization algorithms to adapt to environmental changes and improve operational efficiency and robustness. Online learning refers to the ability to continue learning and improving during actual operation in a real environment. Local optimization algorithms are used to fine-tune the current strategy to adapt to changes in the local environment. This enables the robot to cope with unseen new environments or unexpected situations, avoiding performance stagnation and becoming increasingly intelligent with use.

[0031] This invention forms a complete closed loop from environmental perception to task execution through the close coordination of four stages: multimodal perception, hierarchical task decomposition, multi-objective optimization, and online learning, thereby improving the autonomous operation capability of cleaning robots in complex environments.

[0032] Specifically, in step S120, the step of fusing multimodal information to construct a unified environmental state representation includes: Visual sensors are used to collect environmental image information as visual modal information. Visual sensors mainly refer to cameras (such as RGB cameras and depth cameras) to acquire rich visual information about the environment, such as color, texture, depth, and spatial layout. For example, a robot may see a sofa (obstacle) in front of it, a piece of paper (contaminant) on the floor next to the sofa, and a large area of ​​wooden floor (workable area) through a camera.

[0033] The robot acquires environmental structural features and physical states as physical modal information using lidar, an inertial measurement unit (IMU), and a dust detection sensor. LiDAR is used to perceive the precise geometry of the environment through laser ranging. The IMU measures the robot's own motion (acceleration, angular velocity). The dust detection sensor detects the concentration of particulate matter in the air or on the ground. This process obtains precise physical measurements of the environment and the robot's own physical state.

[0034] Environmental uncertainty parameters are collected as fuzzy modal information through ground material sensors, light sensors, and dynamic obstacle detection sensors. The ground material sensor is a near-infrared reflective sensor used to identify different materials such as flooring, carpet, and tiles. The light sensor is used to sense ambient light intensity. The dynamic obstacle detection sensor, whether an ultrasonic sensor or an algorithm combining vision / LiDAR, is used to detect suddenly appearing people or pets. This process senses environmental parameters with uncertainty and fuzziness, parameters that are difficult to answer with simple yes or no answers.

[0035] A convolutional neural network (CNN) is used to extract features from the visual modal information, identifying the spatial boundaries of the workable area, obstacle outlines, and the distribution of contaminated areas. CNNs are deep learning models particularly well-suited for processing image information. They automatically learn and extract high-level, meaningful features from raw pixels, replacing manually designed rules. For example, if raw images captured by a camera are input into a CNN, the network will output structured features, such as: workable area boundaries: a sequence of polygonal coordinates outlining all areas that can be cleaned; obstacle outlines: precisely identifying the outlines of sofas, tables, and chairs; and contaminated area distribution: a heatmap indicating the probability of where paper scraps are located.

[0036] Based on hesitant fuzzy set theory, the fuzzy modal information is modeled and processed to establish a ground material recognition model, an illumination adaptability assessment model, and a dynamic obstacle threat assessment model. Hesitant fuzzy set theory is a mathematical tool for handling uncertainty, allowing an element's membership degree to a set to have multiple possible values ​​(while traditional fuzzy sets only have one value). Fuzzy and uncertain sensor information is quantified into a mathematical representation containing multiple possibilities, thus preserving the hesitation space needed for decision-making. For example, for areas with excessively strong light, traditional fuzzy sets might directly assign a visual confidence level of 0.3. However, a hesitant fuzzy set representation might be {0.2, 0.3, 0.4}, indicating that due to incomplete information, the confidence level could be 0.2, 0.3, or 0.4, providing richer information for subsequent fusion.

[0037] The processed visual features, physical sensor data, and fuzzy modal evaluation results are fused at the feature level to construct a multi-dimensional environmental state representation that includes spatial structure features, operational feasibility domain distribution, and environmental uncertainty assessment. Feature-level fusion involves extracting the features of each component and then merging all these features into a unified representation vector. This creates a comprehensive, multi-dimensional environmental snapshot that simultaneously includes geometric structure, physical properties, and uncertainty assessment. For example, the final generated environmental state vector might contain information in the following dimensions: [coordinates (1,1), terrain = floor, obstacles = none, contamination = 0.1, visual confidence = 0.9] (a clean, open area), [coordinates (2,2), terrain = carpet, obstacles = none, contamination = 0.8, visual confidence = 0.6] (a dirty carpet area, and visually unreliable due to material issues), [coordinates (3,3), terrain = unknown, obstacles = dynamic, threat = 0.7, contamination = 0.0] (an area with dynamic obstacles, making cleaning infeasible).

[0038] This invention achieves comprehensive and accurate perception of environmental conditions through collaborative processing and feature-level fusion of multi-source sensor data, providing a reliable perception basis for subsequent decision-making and planning.

[0039] Furthermore, the modeling process of the fuzzy modal information based on hesitant fuzzy set theory includes: A hesitant fuzzy decision matrix is ​​constructed, which includes the following three evaluation dimensions: sensor reliability evaluation dimension, used to quantify the reliability level of different sensors in dynamic environments; environmental feature membership dimension, which generates a set of membership values ​​of environmental features through a multi-expert evaluation mechanism; and time series consistency dimension, used to evaluate the stability of environmental features over time.

[0040] The hesitant fuzzy decision matrix is ​​a structured table or matrix used to systematically evaluate the performance of different environmental features across various uncertainty dimensions. It provides a multi-dimensional, quantitative evaluation framework for environmental uncertainty. For example, consider evaluating the environmental feature of a carpeted area: Sensor reliability evaluation dimension: Due to the carpet texture, visual sensors may become less reliable, and LiDAR may become inaccurate due to light absorption. This dimension quantifies the reliability level of these sensors. For example: {Vision: 0.6, LiDAR: 0.7}. Environmental feature membership dimension: This assesses whether the area is easy to clean. Multiple experts (using pre-defined rules or algorithms) provide opinions: {Yes: 0.5, No: 0.3}, indicating a 50% confidence level that the area is easy to clean, a 30% confidence level that it is not easy, and a 20% uncertainty. Time series consistency dimension: This assesses whether the area's features are stable. If the area is sometimes exposed to sunlight and sometimes not, the consistency is low, for example, an evaluation value of 0.4 (1 indicates complete stability).

[0041] A mapping mechanism between hesitant fuzzy features and the state space is established. The confidence weights of each environmental feature are calculated using a hesitancy degree operator, forming a probabilistic state representation with confidence assessment. The hesitancy degree operator is a mathematical function used to calculate the degree of hesitation (i.e., the magnitude of uncertainty) of a hesitant fuzzy set. The probabilistic state representation transforms features with uncertainty into probability distributions with confidence weights. The fuzzy assessment results are then transformed into state inputs with confidence indices that can be understood and used by decision models such as reinforcement learning.

[0042] For example, when calculating hesitation, a membership set {0.5, 0.3} might have a high hesitation level, indicating significant uncertainty. Features with high hesitation are assigned lower weights in the final state representation. Conversely, features with high certainty have higher weights. Ultimately, for a carpeted area, the state representation might not be a simple "cleanable" or "uncleanable," but rather P(cleanable) = 0.6, P(uncleanable) = 0.4, with an overall confidence weight of 0.7. This probabilistic state representation contains more information than an arbitrary binary judgment.

[0043] This design employs a dynamic weighting-based information aggregation strategy to automatically adjust the fusion weights of the visual, physical, and fuzzy modalities based on environmental changes. The dynamic weighting strategy is a fusion algorithm that automatically adjusts the importance of different sensor information based on real-time conditions, allowing the system to intelligently trust more reliable information sources in specific environments and ignore unreliable ones. Under normal circumstances, the fusion weights might be visual: 0.4, physical: 0.4, and fuzzy: 0.2. In strong light, if a visual sensor malfunction is detected, the system automatically lowers the weight of the visual modality (e.g., to 0.1) and correspondingly increases the weights of the physical modality (LiDAR) and the fuzzy modality (ground material sensor) (e.g., to 0.5 and 0.4, respectively). On carpets, because the physical modality LiDAR may be inaccurate, the system increases the weight of the fuzzy modality (ground material sensor) because it better predicts motion resistance.

[0044] Hesitant fuzzy similarity calculation is used to intelligently cluster environmental states, grouping states with similar uncertainty characteristics into the same decision category. Hesitant fuzzy similarity calculation is a mathematical method for comparing the similarity between two hesitant fuzzy sets. Intelligent clustering groups different environmental states based on similarity. By summarizing thousands of subtly different environmental states into a few decision categories with similar uncertainty characteristics, the complexity of decision learning is greatly simplified. For example, state A: {dim lighting, flat ground, moderate sensor reliability}, state B: {thick carpet ground, high LiDAR noise, moderate sensor reliability}. Although the specific causes of A and B are different (one is a lighting issue, the other a material issue), their core uncertainty characteristics (both have moderate sensor reliability) are similar. Through hesitant fuzzy similarity calculation, the system finds that A and B are highly similar and thus groups them into the same decision category, such as a perception-restricted area requiring cautious and slow passage. In this way, when the robot learns, the strategies learned for this category (such as reducing speed or increasing suction) can be applied to both states A and B simultaneously, without needing to learn them separately, thus improving learning efficiency.

[0045] The aforementioned uncertainty management pipeline first quantifies uncertainty using a matrix, then maps it to states usable by the decision-making model. During fusion, it dynamically adjusts the trust level for different pieces of information, and finally simplifies complex problems through intelligent clustering. This ensures that the robot can still make robust and reasonable decisions when faced with uncertain or unpredictable situations. This invention, through the application of hesitant fuzzy set theory, effectively handles uncertainties in environmental perception, improving the accuracy of state representation and the reliability of decision-making.

[0046] Furthermore, the step of performing feature-level fusion of the processed visual features, physical sensor data, and fuzzy modality evaluation results includes: Timestamp synchronization and spatial coordinate unification are performed on visual, physical, and fuzzy modal information to establish cross-modal data association. Timestamp synchronization ensures that all sensor data is marked with precise time, or aligned to the same moment through interpolation or other methods. Spatial coordinate unification transforms all sensor-perceived information (such as image pixel coordinates and LiDAR point cloud coordinates) to the same coordinate system (such as a coordinate system with the robot's center as the origin). Solving the problem of data misalignment in time and space is a prerequisite for effective data fusion.

[0047] A three-dimensional hesitant fuzzy decision matrix is ​​used to handle environmental uncertainty. The first dimension handles sensor reliability by calculating the reliability weight of each sensor based on real-time signal-to-noise ratio and historical accuracy. The second dimension handles the membership degree of environmental features by generating a set of membership functions for ground material, obstacle type, and pollution level using a multi-expert evaluation mechanism. The third dimension handles time series consistency by analyzing the temporal stability of environmental features through a sliding window.

[0048] The three-dimensional hesitant fuzzy decision matrix extends the two-dimensional matrix in the previous steps by adding a time series dimension, forming a comprehensive evaluation structure. It comprehensively and three-dimensionally evaluates uncertainty from three orthogonal dimensions: sensor reliability, environmental features themselves, and temporal stability. For example, the first dimension, sensor reliability, calculates the camera's real-time signal-to-noise ratio under current lighting conditions and combines it with its historical accuracy in identifying obstacles over the past minute to give its current reliability weight; for example, reliability = 0.8. The second dimension, environmental feature membership, might be evaluated as follows: for ground material, multi-expert (algorithm) assessment might be {hard floor: 0.7, short-pile carpet: 0.3}. For obstacle type, the assessment might be {static chair: 0.9, others: 0.1}. The third dimension, temporal series consistency, uses a sliding window (e.g., the last 5 scans). If the feature of an obstacle appears 5 times in 5 scans, the temporal stability is high (=1.0); if it appears intermittently, the stability is low (=0.3).

[0049] A multi-scale convolutional neural network is used to extract visual features, and an attention mechanism is used to focus on key environmental regions, thereby improving the quality and efficiency of visual feature extraction.

[0050] A weighted adaptive fusion algorithm is employed, specifically including: constructing a feature confidence evaluation network to dynamically calculate the reliability score of each modality feature; designing a feature complementarity evaluation mechanism to identify complementary features between different modalities; and implementing weighted fusion of multimodal features based on confidence and complementarity. The feature confidence evaluation network, a small neural network, is responsible for calculating a reliability score for each feature of each modality. The feature complementarity evaluation mechanism is used to analyze whether there is information redundancy or complementarity between features of different modalities. This achieves intelligent and dynamic feature fusion, allowing reliable features to play a greater role and leveraging complementarity to compensate for the shortcomings of a single modality.

[0051] The fused features are encoded into a unified environmental state vector, which includes: spatial structure features (describing the environmental layout and obstacle distribution); workable domain (identifying cleanable and restricted areas); and environmental uncertainty (assessing the environmental dynamics and perceived reliability of each area). The environmental state vector is a fixed-dimensional numerical vector, the final output of the fusion process, and the direct input to the reinforcement learning decision model. The fused multidimensional information is then packaged into a standardized, machine-readable environmental report.

[0052] Incremental learning continuously updates the environmental state representation, ensuring it remains synchronized with environmental changes. Incremental learning is a learning method that allows the model to continuously update itself using new data without forgetting old knowledge. This enables the environmental perception model to adapt to long-term environmental changes and avoids model aging.

[0053] The steps described above depict an extremely sophisticated and adaptive fusion engine. Through spatiotemporal alignment, 3D uncertainty assessment, intelligent feature extraction and focusing, and dynamic fusion of self-confidence and complementarity, it ultimately generates a high-quality, evolvable environmental state vector, providing near-perfect perceptual input for subsequent decision-making.

[0054] Specifically, in step S130, the step of establishing the hierarchical reinforcement learning framework includes: Based on the spatial structure characteristics and feasible domain distribution in the environmental state representation, an adaptive grid partitioning algorithm is used to dynamically divide the cleaning area into multiple task units. The adaptive grid partitioning algorithm is one that dynamically partitions areas based on environmental characteristics, rather than using a fixed size. Each task unit, a sub-region obtained after partitioning, is the basic unit for high-level decision-making. The algorithm intelligently decomposes the vast global cleaning space into appropriately sized, easily manageable local areas. It doesn't simply divide a room into four equal-sized squares. Instead, it divides a spacious living room into a larger task unit, a densely furnished dining room into several smaller task units for precise obstacle avoidance, and a long, narrow corridor into a separate strip-shaped task unit.

[0055] Each task unit is formally described as an option, defining its execution characteristics through an initial state set, internal policy, and termination condition. An option is a core concept in hierarchical reinforcement learning, representing a subtask with start, execution, and termination conditions. Each task unit is given explicit semantics and execution logic, making it a complete schedulable subtask. For example, defining the option of cleaning the dining table area: Initial state set: The robot is located at the edge of the dining table area. Internal policy: The local cleaning strategy executed within the dining table area (e.g., a zigzag path). Termination condition: Termination occurs when the cleanliness level within the unit reaches 95%, or when the execution time exceeds 3 minutes.

[0056] A unit selection strategy based on option value functions is constructed at the high-level task planning layer to select the optimal task unit according to the environmental state. This is achieved through option value functions. Evaluate the long-term value of choosing to execute a specific task unit (option w) under environmental state s. Teach the robot to make optimal decisions about its next move from a global perspective. For example, given current state s: Unit A has high contamination but is far away, while Unit B has low contamination but is close by. High-level policies are implemented through queries. and Based on the value, it was found that although B is closer, cleaning A has a higher long-term return (because it is dirtier), so option A was chosen.

[0057] At the underlying motion execution layer, a local motion strategy is constructed for each task unit, generating a specific cleaning action sequence based on the environmental state within the unit. A local motion strategy is a strategy that takes effect within a specific task unit (option). It is responsible for determining exactly how to move within that small area. For example, when the option in the dining table area is activated, its internal strategy begins to work, outputting a specific action sequence based on the real-time perceived position of the table and chair legs, such as moving forward 0.5 meters -> turning 30 degrees to the left -> activating the side brush.

[0058] The design option terminates the function, dynamically deciding whether to terminate the current cell operation based on task completion and environmental changes. This allows the robot to flexibly interrupt the current subtask, avoiding wasting too much time in a cell or getting stuck due to environmental changes.

[0059] Intelligent switching between task units is achieved through inter-option strategies, supporting cross-unit job path planning. Inter-option strategies are another expression of high-level strategies, specifically responsible for switching and scheduling between different options (task units). This enables coherent cross-unit operations, forming efficient global paths. An efficient inter-option strategy will not randomly select unit 2 after cleaning unit 1, but will instead select unit 3, which is physically adjacent to unit 1, thereby reducing empty runs during movement.

[0060] An option-evaluator architecture is employed to jointly train high-level option strategies and low-level action strategies. High-level strategy training focuses on long-term task benefits, while low-level strategy training optimizes localized task efficiency. Joint training allows both high-level and low-level strategies to learn and optimize simultaneously, ensuring their learning objectives are aligned and that they collaborate to maximize long-term overall benefits. For example, high-level training focuses on learning the macro-level order of cleaning (where to clean first, where to clean later) to achieve the highest overall cleanliness score. Low-level training focuses on learning how to navigate within a specific shaped area to complete cleaning in the fastest and most energy-efficient way. The combined effect is that the high-level strategy learns to assign appropriate tasks to the low-level strategy, and the low-level strategy learns to execute tasks efficiently; the two complement each other.

[0061] Furthermore, the process of jointly training the high-level option policy and the low-level action policy using the option-judge architecture includes the following steps: Construct an end-to-end option-judge training framework, where the judge module is responsible for evaluating the high-level option value function. and underlying action value function The high-level actor module uses inter-option strategies. The task unit selection decision is generated; the underlying actor module uses internal strategies. Generate specific cleaning actions; design a hierarchical gradient update mechanism, including: gradient updates of high-level option policies based on the option dominance function. Through the formula: , in, High-level option strategy The parameters are typically the weights of a neural network. : The objective function of the high-level strategy, representing the expected cumulative reward. ω: The probability of selecting option (task unit) ω under environmental state s. Option advantage function: Measures the advantage of choosing option ω relative to the average level in state s. Calculation method: ,in, It is the option value function, which represents the long-term expected return of choosing option ω in state s. It is the state value function, which represents the long-term expected return of following the current policy in state s. : The direction of the policy gradient, indicating how to adjust the parameter θ to increase the probability of choosing the current option ω.

[0062] The gradient update of the underlying action policy is based on the action advantage function. Through the formula: , in, Underlying action strategy The parameters are typically the weights of a neural network. : The objective function of the underlying strategy. : The probability of choosing action a given state s and option ω. Action advantage function: Measures the relative advantage of performing action a given state s and option ω. Calculation method: , among which, among which It is the action value function, which represents the expected reward of performing action a in state s and option ω. It is the in-option state value function, representing the value of state s in option ω. : The direction of the underlying policy gradient.

[0063] Adaptive optimization of the option termination function is achieved through the gradient formula: , in, : Option to terminate function The parameters. : The objective function of the terminating function. In the next state The probability of terminating the current option ω. The direction of the gradient of the termination function indicates how to adjust the parameters. To change the termination probability. The option advantage function is used to evaluate the merits of terminating the current option. The current state and the next state are environmental state representations derived from multi-sensor fusion. : Option, which corresponds to a task unit (such as a room or area). Basic actions (such as moving forward, turning, sweeping, etc.).

[0064] These formulas together form a hierarchical reinforcement learning framework: the high-level policy (parameter θ) learns how to select task units (options). The low-level policy (parameter φ) learns to perform specific actions within the selected units. The termination function (parameter β) learns when to switch task units. The dominance function (… and This guides the gradient update direction of each part, ensuring policy optimization to improve long-term cumulative rewards.

[0065] This invention achieves intelligent decomposition and coordinated execution of cleaning tasks through a hierarchical reinforcement learning framework, thereby improving work efficiency and environmental adaptability.

[0066] Specifically, in step S140, the process of designing the multi-objective reward function includes the following steps: Establish a multi-objective reward system: cleaning coverage rate reward Calculated based on the ratio of the area cleaned to the total area cleaned; Energy efficiency bonus. Energy consumption rate calculation based on unit cleaning area; penalty for repeated cleaning rate. Calculation based on the frequency of repeated cleaning in the same area; Cleanliness bonus for key areas. Calculation based on the cleaning completion rate of preset key areas.

[0067] Design a dynamic weight adjustment mechanism: Construct an adaptive weight function The weighted components are dynamically adjusted over time t and environmental conditions s; when the battery level is detected to be below a threshold, the energy efficiency weight is automatically increased. When entering a preset key area, temporarily increase the cleanliness weight of the key area. .

[0068] Constructing the comprehensive reward function: The total reward function is defined as follows: , The course incentive mechanism is designed as follows: in the early stage of training, the focus is on rewarding the cleaning coverage rate to ensure basic cleaning ability; in the middle stage of training, energy efficiency optimization is introduced to balance cleaning effect and energy consumption; and in the later stage of training, the cleanliness of key areas is strengthened to improve the level of refined operation.

[0069] Establish reward normalization processing: normalize each reward component to ensure that reward values ​​of different dimensions are within a comparable range; adopt the sliding window standardization method to adapt to the reward distribution changes in different work scenarios.

[0070] This invention enables the cleaning robot to autonomously adjust and optimize its objectives in different work stages and environments through a dynamic weight adjustment mechanism and course reward design, thereby achieving intelligent balance and collaborative optimization among multiple objectives.

[0071] Specifically, in step S150, the process of jointly training the high-level task planning strategy and the low-level action execution strategy using an option-judge framework includes the following steps: Construct a multi-granularity strategy network architecture: The high-level task planning strategy network adopts a graph attention network, which generates task unit selection decisions by analyzing the environmental topology and regional connectivity; the low-level action execution strategy network adopts a multi-objective decision network, which outputs cleaning path control commands, suction intensity adjustment parameters and obstacle avoidance maneuver strategies in parallel.

[0072] Design a hierarchical value estimation network: Construct a two-branch value network structure, where: the option value network evaluates the long-term expected return of each task unit, taking into account the overall efficiency of cross-unit operations; the action value network evaluates the effect of specific operation parameters, combining local environmental characteristics and global task progress.

[0073] Implementing a policy collaborative optimization mechanism: A hierarchical proximal policy optimization algorithm is adopted, wherein: high-level policy updates are guided by an option advantage function. ; The underlying strategy update combines internal and external rewards: , in For the internal advantage function of the option, It is the global advantage function; , These are respectively about the parameters The gradient operator.

[0074] Establish a cross-level experience transfer mechanism: Design a hierarchical experience storage structure, where high-level experience focuses on state transitions and reward accumulation between options, and low-level experience records the immediate effects of action execution; realize experience sharing between levels, and low-level strategies can use the value estimation of high-level options to accelerate convergence.

[0075] Design a progressive training course: Implement training in three phases: Intra-unit skill mastery phase: Fixed task units, training basic skills such as bottom-level cleaning and obstacle avoidance; Inter-option coordination phase: Allowing option switching, training the ability to coordinate cross-unit operations; Global optimization phase: Fully opening up option creation and termination, training the ability to decompose global tasks.

[0076] Achieve dynamic option management: Automatically create new task unit options or merge inefficient options through an option utility evaluation mechanism; dynamically adjust the initial set of options and termination conditions based on environmental change detection.

[0077] Construct a multi-timescale update framework: The high-level strategy adopts an event-triggered update mechanism, updating when options are switched; the low-level strategy adopts a fixed time step update to ensure the real-time execution of actions.

[0078] This invention achieves end-to-end intelligent decision-making from macro-task planning to micro-action execution through a multi-granularity strategy architecture and hierarchical optimization mechanism, significantly improving the adaptability and operational efficiency of cleaning robots in dynamic environments.

[0079] Specifically, in step S160, the process of continuously adjusting the decision model through online learning and local optimization algorithms includes the following steps: Construct an online learning framework: Design an incremental policy update mechanism to collect environmental interaction data and update policy parameters in real time during the assignment process; establish a sliding window experience buffer to save the interaction experience of the most recent N time steps for online learning.

[0080] Implement a local environment adaptation mechanism: Construct an environment change detection module to monitor the rate of change of environmental features in real time, and trigger a rapid policy adaptation when a significant change in the environment is detected; design a local policy adjustment algorithm to fine-tune the underlying action policy based on the environmental features of the current task unit.

[0081] Establish a performance monitoring and optimization loop: calculate multi-objective performance indicators in real time, including instantaneous cleaning efficiency, energy consumption rate, and area coverage progress; build a performance trend analysis model to predict strategy degradation risks and trigger preventive optimization based on time-series performance data.

[0082] The incremental model predictive control mechanism is designed by adopting a rolling time-domain optimization strategy, which performs local trajectory optimization based on the current environmental state in each decision cycle; and dynamically adjusts the cleaning path planning and operation intensity parameters by combining real-time sensor data.

[0083] Online policy parameter calibration is achieved: High-level policy parameters are fine-tuned using an online policy gradient algorithm. , in, High-level policy network parameters The update amount (increment). : The learning rate of the high-level policy, a hyperparameter greater than 0, used to control the step size of parameter updates. R: The cumulative reward actually obtained after executing option ω starting from state s.

[0084] The underlying strategy parameters are updated using an online actor-critic algorithm: , in, φ: The update increment of the underlying policy network parameters. β: The learning rate of the underlying policy. r: The immediate reward obtained after performing action a. γ: The discount factor, ranging from [0,1], used to measure the value of future rewards relative to immediate rewards. , Each option ω represents the new state. , The estimated value of the state value function. This is called the temporal difference error. It is the current reward r plus the discounted value of the next state s'. Subtract the estimated value of the current state s. This error reflects the accuracy of the current value estimate and is used to guide policy updates. If the error is positive, it means that the effect of performing action 'a' is better than expected, and the probability of selecting that action should be increased.

[0085] These two formulas together enable the robot to learn online during the operation. They utilize real-time interactive data (R, r, s') and the predictions of the current model ( , The differences between the two are used as signals, and the parameters (θ, φ) of the high-level and low-level policy networks are fine-tuned through the stochastic gradient ascent method, so that the robot's decisions can continuously adapt to the environment and optimize performance.

[0086] An adaptive learning rate adjustment mechanism is constructed: the learning rate parameter is dynamically adjusted according to the dynamics of the environment and the rate of change of policy performance; when a performance decline is detected, the exploratory nature is automatically increased, and when the performance is stable, the focus is on utilizing existing knowledge.

[0087] To achieve cross-task knowledge transfer: Establish a strategy parameter sharing mechanism to quickly transfer effective strategies learned in a specific environment to similar environments; design a meta-learning framework to extract transferable strategy patterns from historical task experience.

[0088] This invention establishes a complete online learning and optimization cycle, enabling the cleaning robot to continuously adapt to environmental changes and optimize its performance during actual operations, significantly improving the system's practicality and robustness.

[0089] Example 2

[0090] like Figure 2 As shown, this application provides an architecture diagram of an autonomous operation system for a cleaning robot based on multi-sensor fusion and reinforcement learning. It is applied to the autonomous operation system for a cleaning robot based on multi-sensor fusion and reinforcement learning as described in Embodiment 1. It includes a multimodal information acquisition module 210, an environmental state representation construction module 220, a hierarchical reinforcement learning decision module 230, a multi-objective reward optimization module 240, a strategy training module 250, and an online learning optimization module 260 connected in sequence, forming a complete autonomous operation control loop.

[0091] The multimodal information acquisition module 210 is used to acquire visual modal information, physical modal information and fuzzy modal information of the cleaning environment through a multi-sensor system.

[0092] The environmental state representation construction module 220 is used to fuse the multimodal information and construct a unified environmental state representation. It includes: a visual feature extraction unit, which uses a deep learning model to extract features from visual modal information and identify the distribution of workable areas, obstacle boundaries, and polluted areas; a fuzzy information modeling unit, which models fuzzy modal information based on hesitant fuzzy sets to handle environmental uncertainty factors; and a multi-source data fusion unit, which fuses the extracted visual features, physical sensor data, and fuzzy modal information to generate a multi-dimensional environmental state representation.

[0093] The hierarchical reinforcement learning decision module 230 is used to establish a hierarchical reinforcement learning framework based on the environmental state representation, including: a task unit division unit, which divides the cleaning area into multiple task units, each task unit serving as an option in the high-level task planning layer; and an action execution unit, which executes specific cleaning actions in the low-level action execution layer based on the environmental state representation.

[0094] The multi-objective reward optimization module 240 is used to design a multi-objective reward function, which comprehensively considers cleaning coverage, energy efficiency, repetitive cleaning rate and cleanliness of key areas, and introduces a dynamic weight adjustment mechanism. The strategy training module 250 is used to jointly train high-level task planning strategies and low-level action execution strategies using an option-judge framework. It includes: a high-level strategy unit that selects the optimal task unit based on the environmental state representation; and a low-level strategy unit that outputs cleaning path planning, task intensity adjustment, and obstacle avoidance strategies within the selected unit.

[0095] The online learning optimization module 260 is used to dynamically adjust the decision model during the operation through online learning and local optimization algorithms to adapt to environmental changes and improve operation efficiency and robustness.

[0096] Figure 3 This is an electronic device provided in one embodiment of this application. For example... Figure 3 As shown, the electronic device includes at least the following components: processor 301 and memory 300, communication interface 303, and bus 302.

[0097] In this embodiment of the application, memory 300 is used to store executable instructions of processor 301, which, when configured to execute instructions, implements the method as described in the first aspect.

[0098] In embodiments of this application, a computer-readable storage medium includes instructions that instruct a device to perform the method as described in the first aspect. For example, the instructions instruct the device to perform... Figure 1 The method is shown in the process steps.

[0099] In one embodiment of this application, the program operating in the electronic device may be a program that controls a central processing unit (CPU) or similar device to achieve the functions of the above-described embodiments of the present invention (a program that enables the computer to function). Information processed by these systems is then temporarily stored in random access memory (RAM) during processing, and subsequently stored in various ROMs such as read-only memory (FlashROM) and hard disk drives (HDDs), and read, corrected, and written by the CPU as needed.

[0100] It should be noted that a portion of the electronic device described above can also be implemented using a computer. In this case, the program for implementing the control function can be recorded on a computer-readable recording medium, and the program recorded on the recording medium can be read into the computer and executed.

[0101] It should be noted that the computer mentioned here refers to a computer built into an electronic device, employing hardware including an operating system and peripheral devices. Furthermore, computer-readable recording media refers to removable media such as floppy disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage systems such as hard drives built into the computer.

[0102] Furthermore, computer-readable recording media can include: media that dynamically stores programs for short periods of time, such as communication lines used when transmitting programs via networks like the Internet or communication lines like telephone lines; and media that store programs for fixed periods of time, such as volatile memory inside a computer that serves as a server or client in this case. In addition, the aforementioned program can be a program used to implement the above-mentioned functions, or it can be a program that can implement the above-mentioned functions by combining them with programs already recorded in the computer.

[0103] Furthermore, the electronic device in the above embodiments can also be implemented as an assembly (system group) composed of multiple systems. Each system constituting the system group can possess some or all of the functions or functional blocks of the electronic device in the above embodiments. As a system group, it is sufficient to have all the functions or functional blocks of the electronic device.

[0104] Those skilled in the art should recognize that the above embodiments are only used to illustrate this application and are not intended to limit this application. Any appropriate changes and variations made to the above embodiments within the essential spirit and scope of this application fall within the scope of protection claimed in this application.

Claims

1. A method for autonomous operation of a cleaning robot based on multi-sensor fusion and reinforcement learning, characterized in that, The method comprises the following steps: Collecting multi-modal information of the cleaning environment through a multi-sensor system, the multi-modal information comprising visual modal information, physical modal information and fuzzy modal information; Fusing the multi-modal information to construct a unified environment state representation, comprising: using a deep learning model to extract features from the visual modal information, identify the workable area, obstacle boundary and pollution level of the cleaning area; modeling the fuzzy modal information based on hesitant fuzzy sets to handle environmental uncertainty factors; fusing the extracted visual features, physical sensor data and fuzzy modal information to generate a multi-dimensional environment state representation; Based on the environment state representation, a hierarchical reinforcement learning framework is established, including: dividing the cleaning area into multiple task units, each task unit serving as an option for the high-level task planning layer; based on the environment state representation, executing specific cleaning actions at the bottom layer action execution layer; Designing a multi-objective reward function that takes into account cleaning coverage, energy efficiency, repeated cleaning rate and key area cleanliness, and introducing a dynamic weight adjustment mechanism; Using an option-judge framework to jointly train the high-level task planning strategy and the bottom layer action execution strategy, wherein: the high-level strategy selects the optimal task unit based on the environment state representation; the bottom layer strategy outputs cleaning path planning, work intensity adjustment and obstacle avoidance strategy within the selected unit; During the operation, the decision model is dynamically adjusted through online learning and local optimization algorithm to adapt to environmental changes and improve operation efficiency and robustness; Wherein, the joint training of high-level option strategy and bottom layer action strategy using option-judge architecture includes: Building an end-to-end options-evaluator training framework, where: the evaluator module is responsible for evaluating high-level option value functions and low-level action value functions ; The high-level actor module selects an action by inter-option policy generates a task unit selection decision; The bottom actor module generates a specific cleaning action by an internal strategy a specific cleaning action; The layered gradient update mechanism is designed, including: the gradient update of the high-level option policy is based on the option advantage function , through the formula: ， wherein : high-level option policy : parameter of the high-level policy, : objective function of the high-level policy, : gradient; Gradient updates to the underlying policy are based on an action advantage function by the equation: ， wherein, : a bottom action policy : parameters of the bottom action policy, : an objective function of the bottom policy. Adaptive optimization of the option termination function is achieved through the gradient formula: ， wherein, : option termination function : parameter of the option termination function, : target function of the termination function, : current state and next state, : option, : underlying action. 2.The autonomous operation method of a cleaning robot based on multi-sensor fusion and reinforcement learning according to claim 1, characterized in that, The step of fusing the multi-modal information to construct a unified environment state representation comprises: Collecting environmental image information as visual modal information through a visual sensor; Collecting environmental structure features and physical state as physical modal information through a laser radar, an inertial measurement unit and a dust detection sensor; Collecting environmental uncertainty parameters as fuzzy modal information through a ground material sensor, a light sensor and a dynamic obstacle detection sensor; Using a convolutional neural network to extract features from the visual modal information to identify the spatial boundary of the workable area, the obstacle contour and the pollution area distribution; Modeling and processing the fuzzy modal information based on the theory of hesitant fuzzy sets to establish a ground material recognition model, a light adaptability evaluation model and a dynamic obstacle threat degree evaluation model; Fusing the processed visual features, physical sensor data and fuzzy modal information evaluation results at the feature level to construct a multi-dimensional environment state representation containing spatial structure features, workable area distribution and environmental uncertainty evaluation. 3.The autonomous operation method of a cleaning robot based on multi-sensor fusion and reinforcement learning according to claim 2, characterized in that, The step of modeling and processing the fuzzy modal information based on the theory of hesitant fuzzy sets comprises: A hesitant fuzzy decision matrix is constructed, which includes three evaluation dimensions: a sensor reliability evaluation dimension for quantifying the reliability level of different sensors in a dynamic environment; an environment feature membership degree dimension for generating a set of membership values of environment features through a multi-expert evaluation mechanism; and a time series consistency dimension for evaluating the stability of environment features over time; A mapping mechanism of hesitant fuzzy features and state space is established, and the reliability weight of each environment feature is calculated through a hesitancy operator to form a probabilistic state representation with confidence evaluation; An information aggregation strategy based on dynamic weights is designed to automatically adjust the fusion weights of visual, physical and fuzzy modalities according to the characteristics of environmental changes; A hesitant fuzzy similarity calculation method is used to intelligently cluster the environment state, and states with similar uncertain characteristics are merged into the same decision category. 4.The autonomous operation method of a cleaning robot based on multi-sensor fusion and reinforcement learning according to claim 2, characterized in that, The step of fusing the processed visual features, physical sensor data and fuzzy modality evaluation results at the feature level includes: Timestamp synchronization and spatial coordinate unification are performed on the visual, physical and fuzzy three types of modal information to establish cross-modal data association; A three-dimensional hesitant fuzzy decision matrix is used to process environmental uncertainty, wherein: the first dimension processes sensor reliability, and the reliability weight of each sensor is calculated through real-time signal-to-noise ratio and historical accuracy; the second dimension processes environment feature membership degree, and a set of membership function of ground material, obstacle type and pollution degree is generated through a multi-expert evaluation mechanism; the third dimension processes time series consistency, and the time series stability of environment features is analyzed through a sliding window; A multi-scale convolutional neural network is used to extract visual features, and an attention mechanism is used to focus on key environmental areas; An adaptive fusion algorithm based on weights is used, which specifically includes: a feature confidence evaluation network is constructed to dynamically calculate the reliability score of each modal feature; a feature complementarity evaluation mechanism is designed to identify complementary features between different modalities; and weighted fusion of multi-modal features is realized based on confidence and complementarity; The fused features are encoded into a unified environment state vector, which includes: spatial structure features: describing the environment layout and obstacle distribution; workable region: identifying the cleanable area and restricted area; and environmental uncertainty: evaluating the environmental dynamics and perception reliability of each area; An incremental learning mechanism is used to continuously update the environment state representation to ensure that it is synchronized with environmental changes. 5.The autonomous operation method of a cleaning robot based on multi-sensor fusion and reinforcement learning according to claim 1, characterized in that, The step of establishing a hierarchical reinforcement learning framework includes: Based on the spatial structure features and workable region distribution in the environment state representation, an adaptive grid division algorithm is used to dynamically divide the cleaning area into multiple task units; Each task unit is described as an option, and the execution characteristics of the option are defined through an initial state set, an internal policy and a termination condition; An option value function-based unit selection strategy is constructed at the high-level task planning layer to select the optimal task unit according to the environment state; A local action strategy is constructed for each task unit at the bottom action execution layer to generate a specific cleaning action sequence based on the environment state within the unit; An option termination function is designed to dynamically determine whether to terminate the current unit job according to the task completion degree and environmental changes. Intelligent switching between task units is achieved through inter-option strategies, supporting cross-unit job path planning; An option-judge architecture is used to jointly train high-level option policies and low-level action policies. High-level policy training focuses on long-term task benefits, while low-level policy training optimizes local task efficiency. 6.The autonomous operation method of a cleaning robot based on multi-sensor fusion and reinforcement learning according to claim 5, wherein, The method of jointly training high-level option strategies and low-level action strategies using an option-judge architecture also includes: Dynamically adjust the termination conditions of the option; Establish a dual experience playback mechanism to store high-level option conversion experience and low-level action execution experience respectively; The course design employs a progressive training strategy, moving from simple environments to complex scenarios. In the initial stage, high-level options are fixed to focus on learning underlying actions, while subsequent stages involve jointly optimizing strategies at each level. 7.The autonomous operation method of a cleaning robot based on multi-sensor fusion and reinforcement learning according to claim 1, characterized in that, The steps for designing the multi-objective reward function include: Establish a multi-objective reward system, including rewards for cleaning coverage, rewards for energy efficiency, penalties for repeated cleaning, and rewards for the cleanliness of key areas; Design a dynamic weight adjustment mechanism and construct a weight adaptive function so that each weight component can be dynamically adjusted with time and environmental conditions. Construct a comprehensive reward function by multiplying each reward component by its corresponding dynamic weight and then summing the results; Design a course reward mechanism that emphasizes different optimization goals at different stages of training; A reward normalization process was established, and a sliding window standardization method was used to normalize each reward component to ensure that reward values ​​of different dimensions are within a comparable range. 8.The autonomous operation method of a cleaning robot based on multi-sensor fusion and reinforcement learning according to claim 6, characterized in that, The steps for jointly training high-level task planning strategies and low-level action execution strategies using an option-judge framework include: A multi-granularity strategy network architecture is constructed, in which the high-level task planning strategy network uses graph attention network to analyze the environmental topology to generate task unit selection decisions, and the low-level action execution strategy network uses multi-objective decision network to output cleaning path control commands, suction intensity adjustment parameters and obstacle avoidance maneuver strategies in parallel. Design a hierarchical value estimation network to evaluate the long-term expected return of task units and the performance of specific operational parameters through a two-branch structure. Establish a cross-level experience transfer mechanism to achieve interactive transfer of high-level option experience and low-level action experience through a hierarchical storage structure and experience sharing; Implement a progressive training curriculum, which is divided into three stages to train basic skills within a unit, coordination ability between options, and global task decomposition ability in turn. To achieve dynamic option management, based on a utility evaluation mechanism and environmental change detection, task unit options are dynamically created, merged, and their execution conditions are adjusted. A multi-timescale update framework is constructed, employing an event-triggered mechanism to update high-level strategies and a fixed-time-step mechanism to update low-level strategies. 9.The autonomous operation method of a cleaning robot based on multi-sensor fusion and reinforcement learning according to claim 8, characterized in that, The steps of continuously adjusting the decision model through online learning and local optimization algorithms include: An online learning framework is constructed, which enables real-time learning and optimization during the assignment process through an incremental strategy update mechanism and a sliding window experience buffer. Establish a local environmental adaptation mechanism, which uses an environmental change detection module to monitor changes in environmental characteristics in real time and triggers rapid strategy adaptation and local adjustment. Design a performance monitoring and optimization loop, calculate multi-objective performance indicators in real time, and predict the risk of strategy degradation through trend analysis to achieve preventive optimization; Implement incremental model predictive control and use a rolling time-domain optimization strategy to dynamically adjust cleaning path planning and work intensity parameters; Construct an adaptive learning rate adjustment mechanism to dynamically adjust learning parameters based on environmental dynamics and the rate of change in policy performance. To achieve cross-task knowledge transfer, historical task experience can be quickly transferred to similar environments through a strategy parameter sharing mechanism and a meta-learning framework.

10. A multi-sensor fusion and reinforcement learning based autonomous cleaning robot working system applied to the multi-sensor fusion and reinforcement learning based autonomous cleaning robot working method according to any one of claims 1 to 9, characterized in that, The system includes: The multimodal information acquisition module is used to acquire visual modal information, physical modal information and fuzzy modal information of the cleaning environment through a multi-sensor system; An environment state representation construction module is used to fuse the multimodal information and construct a unified environment state representation, including: The visual feature extraction unit uses a deep learning model to extract features from visual modal information and identify the workable area, obstacle boundaries, and contaminated area distribution. The fuzzy information modeling unit models fuzzy modal information based on hesitant fuzzy sets, handling environmental uncertainty factors; The multi-source data fusion unit fuses extracted visual features, physical sensor data, and fuzzy modal information to generate a multi-dimensional environmental state representation; A hierarchical reinforcement learning decision module, used to establish a hierarchical reinforcement learning framework based on the environmental state representation, includes: The task unit division divides the cleaning area into multiple task units, with each task unit serving as an option in the higher-level task planning layer. The action execution unit performs specific cleaning actions based on the environmental state representation at the bottom action execution layer; The multi-objective reward optimization module is used to design multi-objective reward functions, which comprehensively consider cleaning coverage, energy efficiency, re-cleaning rate and cleanliness of key areas, and introduces a dynamic weight adjustment mechanism. The policy training module is used to jointly train high-level task planning policies and low-level action execution policies using an option-judge framework, including: The high-level strategy unit selects the optimal task unit based on the environmental state representation. The underlying strategy unit outputs cleaning path planning, work intensity adjustment, and obstacle avoidance strategies within the selected unit; The online learning optimization module is used to dynamically adjust the decision model during the operation through online learning and local optimization algorithms to adapt to environmental changes and improve operation efficiency and robustness. The multimodal information acquisition module, environmental state representation construction module, hierarchical reinforcement learning decision-making module, multi-objective reward optimization module, strategy training module, and online learning optimization module are connected in sequence to form a complete autonomous operation control loop.

Images