Reinforcement learning method, device and equipment applicable to cleaning robot and storage medium
Patent Information
- Application Number
- CN202611036421.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-13
- Publication Date
- 2026-10-13
Smart Images
Figure CN122904993A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of reinforcement learning technology for cleaning robots, and more specifically, to a reinforcement learning method, apparatus, device, and storage medium suitable for cleaning robots. Background Technology
[0002] Cleaning robots need to perform tasks such as perception, decision-making, navigation, interaction, and operation in complex physical environments.
[0003] As visual language action models are increasingly used in robot strategies, how to explicitly incorporate safety constraints while ensuring task performance has become a key issue affecting their deployment. This is especially true for cleaning robots, whose working environment inevitably intersects with human activities, so safety risks should be given more attention in cleaning robot strategies.
[0004] In related technologies, the SafeVLA model integrates safety constraint modeling, risk arousal, safety reinforcement learning constraint optimization, and safety evaluation into a unified framework through an integrated safety scheme. It also achieves safety-oriented policy training based on constrained Markov decision processes and Lagrange constraint optimization. This allows robots to maximize task performance while ensuring the safety of people, the environment, and themselves in the physical world when performing complex tasks.
[0005] However, existing reinforcement learning methods based on the SafeVLA model still use methods such as list-order sampling, uniform random sampling, or approximately static sampling to schedule training tasks. This can easily lead to a lack of learning resources for the long-tailed hazardous scenarios that are prevalent in reinforcement learning of cleaning robots, ultimately resulting in the cleaning robot still being unable to effectively cope with long-tailed hazardous scenarios after reinforcement learning. Summary of the Invention
[0006] The summary section of this application is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.
[0007] Some embodiments of this application propose reinforcement learning methods, apparatuses, electronic devices, and computer-readable media suitable for cleaning robots to solve the technical problems mentioned in the background section above.
[0008] As a first aspect of this application, some embodiments of this application provide a reinforcement learning method suitable for cleaning robots, comprising: dividing multiple training tasks for reinforcement learning of the cleaning robot into multiple training task subsets according to task data; performing reinforcement learning on all the training tasks in one training task subset in an iterative manner; generating or updating a dynamic scheduling score of the training task subset based on the reinforcement learning results of all the training tasks in the training task subset; and calling the training task subset to be used in the next round based on the dynamic scheduling scores of the multiple training task subsets; wherein the task data includes: scene information, task information, and / or result information; and the dynamic scheduling score of the training task subsets. The calculation formula is: ;in, For the first A set of risk scores for a subset of training tasks; To explore coefficients; This represents the cumulative number of scheduling attempts for all training task subsets. Indicates the first The historical call count of a subset of training tasks; the set risk score The risk value is obtained by accumulating or updating the single-task risk values of multiple training tasks in the subset of training tasks; the single-task risk value The calculation formula is: ;in, The corresponding value of the collision index for the associated training task; The corresponding value of the cumulative cost metric for the associated training task; The corresponding value for the task failure metric of the associated training task; , , These are the weighting coefficients.
[0009] Optionally, in some embodiments of this application, the corresponding value of the cumulative cost metric associated with the training task is... The calculation formula is: ;in, This represents the actual value of the cumulative cost metric for the training task. This is a limit value for the cumulative cost metric of the training task.
[0010] Optionally, in some embodiments of this application, when reinforcement learning is performed using the training task, if a physical collision occurs, the corresponding value of the collision metric associated with the training task is... The value is 1; when reinforcement learning is performed using the training task, if there is no physical collision, the corresponding value of the collision metric associated with the training task. The value is 0.
[0011] Optionally, in some embodiments of this application, when reinforcement learning is performed using the training task, if the task fails, the corresponding value of the task failure index associated with the training task is... The value is 1; when reinforcement learning is performed using the training task, if the task is successful, the corresponding value of the task failure metric associated with the training task is 1. It is 0.
[0012] Optionally, in some embodiments of this application, generating or updating the dynamic scheduling score of the training task subset based on the reinforcement learning results of all the training tasks in the training task subset includes: calculating the single-task danger value based on the reinforcement learning results of all the training tasks in the training task subset. cumulative value Based on the risk score of the training task subset updated in the previous round. Obtain the updated set danger rating. Among them, the updated set hazard score in this round The calculation formula is: ;in, This is the smoothing coefficient.
[0013] Optionally, in some embodiments of this application, the step of calling the training task subset to be used in the next round based on the dynamic scheduling score of the plurality of training task subsets includes: allocating more reinforcement learning times to the training task subset with the higher dynamic scheduling score.
[0014] Optionally, in some embodiments of this application, the scene information includes: house identifier, scene identifier, and physical environment identifier; the task information includes: task identifier, action identifier, and language instruction identifier; and the result information includes: task success rate identifier, safety cost identifier, and balance parameter identifier.
[0015] Optionally, in some embodiments of this application, the reinforcement learning method is based on the SafeVLA model; the training task is constructed as a data combination of three-dimensional asset data and building layout data suitable for the SafeVLA model.
[0016] As a second aspect of this application, some embodiments of this application provide a reinforcement learning apparatus suitable for cleaning robots, comprising: a classification module for dividing multiple training tasks for reinforcement learning of the cleaning robot into multiple training task subsets according to task data; a learning module for performing reinforcement learning using all the training tasks in one of the training task subsets in an iterative manner; a scoring module for generating or updating a dynamic scheduling score for the training task subset based on the reinforcement learning results of all the training tasks in the training task subset; and a calling module for calling the training task subset to be used in the next round based on the dynamic scheduling scores of the multiple training task subsets; wherein the task data includes: scene information, task information, and / or result information; and the dynamic scheduling score of the training task subsets. The calculation formula is: ;in, For the first A set of risk scores for a subset of training tasks; To explore coefficients; This represents the cumulative number of scheduling attempts for all training task subsets. Indicates the first The historical call count of a subset of training tasks; the set risk score The risk value is obtained by accumulating or updating the single-task risk values of multiple training tasks in the subset of training tasks; the single-task risk value The calculation formula is: ;in, The corresponding value of the collision index for the associated training task; The corresponding value of the cumulative cost metric for the associated training task; The corresponding value for the task failure metric of the associated training task; , , These are the weighting coefficients.
[0017] As a third aspect of this application, some embodiments of this application provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.
[0018] As a fourth aspect of this application, some embodiments of this application provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any implementation of the first aspect above.
[0019] The beneficial effects of this application are: it provides a reinforcement learning method, device, equipment, and storage medium suitable for cleaning robots that dynamically concentrates training resources on training tasks that require emphasis through a training scheduling mechanism. Attached Figure Description
[0020] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application.
[0021] Furthermore, throughout the accompanying drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the elements are not necessarily drawn to scale.
[0022] In the attached diagram: Figure 1 This is a schematic diagram of the main steps of a reinforcement learning method according to one embodiment of this application; Figure 2 This is a schematic flowchart illustrating a reinforcement learning method according to one embodiment of this application; Figure 3 This is a schematic diagram of the architecture of training data in a reinforcement learning method according to one embodiment of this application; Figure 4 This is a schematic diagram of a training scenario in a reinforcement learning method according to one embodiment of this application; Figure 5 This is a schematic diagram illustrating the principle of the SafeVLA model in a reinforcement learning method according to one embodiment of this application; Figure 6 This is a schematic diagram of the architecture of a reinforcement learning device according to one embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device according to one embodiment of this application. Detailed Implementation
[0023] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0024] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described herein can be combined with each other.
[0025] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.
[0026] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0027] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
[0028] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.
[0029] As a first aspect of this application, reference is made to Figure 1 As shown, the reinforcement learning method for cleaning robots proposed in this application mainly includes the following steps: S1: Divide the multiple training tasks used for reinforcement learning of the cleaning robot into multiple training task subsets according to the task data.
[0030] S2: Reinforcement learning is performed using all training tasks in a subset of training tasks in an iterative manner.
[0031] S3: Generate or update the dynamic scheduling score of a training task subset based on the reinforcement learning results of all training tasks in that subset.
[0032] S4: Call the next training task subset to be used based on the dynamic scheduling score of multiple training task subsets.
[0033] Specifically, task data includes: scenario information, task information, and / or result information.
[0034] Dynamic scheduling score for training task subsets The calculation formula is: ;in, For the first A set of risk scores for a subset of training tasks; To explore coefficients; This represents the cumulative number of scheduling attempts for all training task subsets. Indicates the first The number of historical calls for a subset of training tasks.
[0035] Collective risk rating It is obtained by accumulating or updating the single-task hazard values of multiple training tasks in this subset of training tasks; whereby, the single-task hazard value... The calculation formula is: ;in, The corresponding values of the collision index for the associated training tasks; The corresponding value of the cumulative cost metric for the associated training tasks; This corresponds to the task failure metric for the associated training task; , , These are the weighting coefficients.
[0036] Existing technologies for scheduling training tasks, such as list-order sampling, uniform random sampling, or approximately static sampling, have the following drawbacks: high-risk scenarios account for a relatively small proportion of the training distribution; uniform sampling tends to consume too many training resources on low-risk, easily completed scenarios, while high-risk scenarios are under-trained; and when a high level of completion capability is already achieved in some scenarios, continuing to repeatedly invest training resources will reduce the overall sample utilization efficiency.
[0037] Existing technologies typically revolve around loss functions, constraint functions, and policy updates, but lack a scenario-level scheduling loop based on historical risk feedback, making it difficult to continuously focus the training process on scenario units that truly expose security vulnerabilities.
[0038] The above-mentioned technical solution can provide a dynamic scheduling mechanism based on historical hazard feedback, under the premise of reinforcing the learning model, so that training resources are prioritized to training tasks that need to be tilted, reducing the probability of such training tasks being randomly diluted, and improving the safety risk control capability and training sample utilization efficiency of the cleaning robot.
[0039] More specifically, by dividing the training tasks into subsets, the system can prioritize resources for training tasks that need to be allocated by calling the entire subset. Then, by iterating through all training tasks in a subset, the probability of random dilution of training resources caused by random switching is avoided. Finally, the system selects the set of training tasks that are more likely to have long-tail risks through dynamic scheduling and scoring. This not only provides feedback on dangerous situations in the learning history, but also effectively allocates learning resources.
[0040] In some embodiments of this application, the corresponding value of the cumulative cost metric of the associated training task is... The calculation formula is: ;in, This represents the actual value of the cumulative cost metric for the training task. This is a limit value for the cumulative cost metric of the training task.
[0041] The advantage of this is that it can ensure the comparability between different training tasks by eliminating the units, while preventing outliers from corrupting the accumulated data through truncation and normalization.
[0042] In some embodiments of this application, when reinforcement learning is performed using a training task, if a physical collision occurs, the corresponding value of the collision metric associated with the training task is... The value is 1; when using the training task for reinforcement learning, if there is no physical collision, the corresponding value of the collision metric of the associated training task. The value is 0.
[0043] In some embodiments of this application, when reinforcement learning is performed using a training task, if the task fails, the corresponding value of the task failure metric for the associated training task is... The value is 1; when using a training task for reinforcement learning, if the task succeeds, the corresponding value of the task failure metric for the associated training task is 1. It is 0.
[0044] In some embodiments of this application, step S3 includes: calculating a single-task hazard value based on the reinforcement learning results of all training tasks in a subset of training tasks. cumulative value Based on the ensemble hazard score updated in the previous round for this subset of training tasks. Obtain the updated set danger rating. Among them, the updated ensemble hazard score The calculation formula is: ;in, This is the smoothing coefficient.
[0045] The advantage of adopting this approach is that, through This represents the "long-term baseline risk level" accumulated over all past training rounds for this subset of training tasks. Its purpose is to provide statistical stability, preventing drastic jumps in scores due to a single, accidental poor performance. This represents the "real-time performance" of the training task subset in the current iteration. Its purpose is to capture the latest changes in the algorithm's capabilities. For example, if the algorithm has just fixed a critical vulnerability in the training task subset, this will immediately reflect a significant drop in the set's danger score.
[0046] Smoothing coefficient The value range is typically between 0 and 1, and it determines the system's level of trust in "new evidence." (Smoothing coefficient) The larger the value, the more aggressive the system updates; the ensemble risk score will quickly follow recent performance fluctuations, making it suitable for rapid algorithm iteration phases; smoothness coefficient The smaller the value, the smoother the system updates and the more stable the scores, making it suitable for fine-tuning stages where the algorithm is mature and requires the removal of random noise. Compared to storing all... The advantage of using this update method, based on historical values, is that it saves storage overhead and has high sensitivity; furthermore, because the historical set of risk scores is exponentially compressed round by round, when... When the value is fixed, the contribution of recent data rounds to the current score is absolutely dominant. This aligns with the understanding in robot reinforcement learning that "recent model capabilities are more representative of the current state," and is more reasonable than applying an equal-weighted average (or hard truncation) to all historical data. Smoothing coefficient The learning stage, needs, or strategies can be selected to reinforce learning. As an option, a dynamic smoothing coefficient can be used. It can be based on the corresponding value of the cumulative cost metric of the training task associated with the current round. You can configure it, or you can adjust it according to the current round. Make dynamic adjustments.
[0047] In some embodiments of this application, step S4 includes allocating more reinforcement learning iterations to a subset of training tasks with a higher dynamic scheduling score. A higher dynamic scheduling score indicates that, according to the criteria for dividing the training task subset, that subset of training tasks requires more training resource allocation.
[0048] In some embodiments of this application, scene information includes: house identifier, scene identifier, physical environment identifier; task information includes: task identifier, action identifier, language instruction identifier; result information includes: task success rate identifier, safety cost identifier, balance parameter identifier.
[0049] Reference Figure 3 As shown, in one implementation, multiple training tasks for reinforcement learning of cleaning robots can be divided into multiple training task subsets according to scene information. This is equivalent to filtering out scenes that are more likely to generate long-tail hazards and dynamically allocating training resources to training tasks where these hazards occur based on training results.
[0050] Understandably, resources can be allocated to training tasks of a specific type or outcome based on task or result information. As an option, the criteria for dividing the training personnel set can be either single-dimensional task data or a combination of multi-dimensional task data.
[0051] In practice, labels can be added to the task data of the training task, or classification criteria can be obtained from key fields or key data items in the content data or result data of the training task. If the "clean" field in the task data or the data item in the result data represents the failure of the task.
[0052] More specifically, scene information can be expressed through house icons, scene icons, and physical environment icons, while task information can be marked with task icons to indicate a certain type of task, such as "cleaning the toilet", or with action icons to indicate a certain type of action feature, such as the action control command or trajectory representing "opening the door", and with language command icons to indicate similar language commands, such as "moving quickly".
[0053] The results information can include labels related to task success rate. For example, a task success rate of 0 is assigned to a training task with a success rate below 50%, while a task success rate of 1 is assigned to a training task with a success rate of 50% or higher. Similarly, different labels can be set for different safety cost values. The balance parameter label is data representing the classification obtained by assigning values to parameters such as Success-But-Safe Rate (SBU) or Violation Severity Index (VSI), which represent the balance between success and safety.
[0054] As an alternative, this application can obtain a set of multi-dimensional training task subsets through different classification criteria. For example, multiple training task subsets in the scene dimension can be obtained first using scene information, and then the reinforcement learning method of this application can be implemented in that dimension. Then, multiple training task subsets in the task dimension can be obtained using task information, and then the reinforcement learning method of this application can be implemented again in that dimension. In this way, training tasks that may have long-tail risks can be avoided from not effectively obtaining training resources from multiple attention dimensions.
[0055] Reference Figure 2 As shown, the reinforcement learning method of this application mainly includes the following process: S10: Identify the multiple training tasks used for reinforcement learning of cleaning robots according to the dimensions of interest.
[0056] S20: Based on the identification results, form multiple training task subsets of corresponding dimensions.
[0057] S30: Use multiple training task subsets of the corresponding dimension to perform reinforcement learning for that dimension. Reinforcement learning requires iterative training of the training tasks in the training task subsets.
[0058] S40: Allocate training resources based on the results of a subset of training tasks.
[0059] Reference Figure 4 and Figure 5 As shown, in some embodiments of this application, the reinforcement learning method is based on the SafeVLA model; the training task is constructed as a data combination of three-dimensional asset data and building layout data suitable for the SafeVLA model.
[0060] The SafeVLA model is a technical solution well-known to those skilled in the art, and its model principles and reinforcement learning implementation will not be elaborated here.
[0061] Reference Figure 4 and Figure 5 As shown, the input data for the SafeVLA model is multimodal, designed to enable the model to fully understand the environment and instructions. The input data for the SafeVLA model mainly includes visual input and language input. The visual input comes from the raw RGB image pixels of the robot's camera; this provides the model with visual information about the environment. The language input consists of task instructions described in natural language (e.g., "perform the task of cleaning the toilet"), meaning that corresponding visual input information exists in the training task.
[0062] The output data of the SafeVLA model are the robot's low-level control commands, which are used to drive the robot to perform specific physical actions.
[0063] Reference Figure 4 As shown, the visual input information can be generated from 3D simulation data, which can be generated from 3D asset data and building layout data of 3D simulation.
[0064] More specifically, the training task is the core link between the input and output, defining a specific learning objective. In the SafeVLA model, a training task is a complete data packet containing the following information.
[0065] Environment and Scene Data: Provides a simulation environment of the physical world in which the task takes place.
[0066] 3D Assets and Annotations (Optimized Objaverse assets): These are the 3D object models in the scene and their textual descriptions.
[0067] ProcTHOR-Objaverse houses: Defines the procedurally generated room or scene layout, specifying the initial placement of objects.
[0068] Task Specification: Precisely defines the target task itself.
[0069] The TaskSpec class in the code (located in tasks / task_specs.py) is the core data structure for defining tasks.
[0070] It typically contains key fields such as task type and house index, as well as a core natural language specification to describe the task objective.
[0071] Safety-related data: This is the key difference between SafeVLA and standard VLA training, and it is the core of its ability to achieve safe reinforcement learning.
[0072] Safety constraints are explicitly and formally defined by constraining the Markov Decision Process (CMDP) paradigm. These constraints specify which behaviors are "unsafe".
[0073] Safety-positive / negative samples: Contrastive safe positive and safe negative samples are used during training to help the model learn to distinguish between safe and unsafe trajectories.
[0074] Safety Rewards: In reinforcement learning frameworks, in addition to task completion rewards, reward or cost signals related to safety constraints are introduced to guide the model to learn safe behaviors.
[0075] Reference Figure 6 As shown, some embodiments of this application provide a reinforcement learning device suitable for cleaning robots, including: a classification module, a learning module, a scoring module, and a calling module.
[0076] Specifically, the classification module is used to divide multiple training tasks for reinforcement learning of the cleaning robot into multiple training task subsets according to the task data; the learning module is used to perform reinforcement learning using all the training tasks in one of the training task subsets in an iterative manner; the scoring module is used to generate or update the dynamic scheduling score of the training task subset based on the reinforcement learning results of all the training tasks in one of the training task subsets; and the calling module is used to call the training task subset to be used in the next round based on the dynamic scheduling scores of the multiple training task subsets.
[0077] like Figure 7As shown, the electronic device 800 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0078] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although... Figure 7 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 7 Each box shown can represent a device or multiple devices as needed.
[0079] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, it performs the functions defined in the methods of some embodiments of this disclosure.
[0080] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.
[0081] In some embodiments of this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0082] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.
[0083] The aforementioned computer-readable medium may be included in the aforementioned electronic device, or it may exist independently without being assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: divide multiple training tasks for reinforcement learning of the cleaning robot into multiple training task subsets according to task data; perform reinforcement learning using all the training tasks in one of the training task subsets in a traversal manner; generate or update a dynamic scheduling score for the training task subset based on the reinforcement learning results of all the training tasks in the training task subset; and invoke the training task subset to be used in the next round based on the dynamic scheduling scores of the multiple training task subsets.
[0084] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages or a combination thereof. These programming languages include object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0085] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function.
[0086] It should also be noted that in some alternative implementations, the functions marked in the box may occur in a different order than those marked in the attached figures.
[0087] For example, two consecutively represented blocks can actually be executed in substantially parallel order, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, as well as combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.
[0088] The units described in some embodiments of this disclosure can be implemented in software or in hardware. The functions described above herein can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.
[0089] More specifically, some implementation methods of this application may produce the following specific beneficial effects: It does not require replacing the existing security reinforcement learning framework and has good compatibility.
[0090] It can dynamically reallocate training resources based on historical risk feedback, concentrating training resources on high-risk, long-tail scenarios.
[0091] The full task traversal mechanism reduces the problem of high-risk samples being diluted by random switching.
[0092] This forms a closed-loop scheduling mechanism of "danger perception - task selection - execution feedback - danger update," thereby improving the efficiency of training sample utilization.
[0093] Reinforcement learning in cleaning robots can help reduce cumulative safety costs and improve the ability to control safety risks.
[0094] The above description is merely an explanation of some preferred embodiments of this disclosure and the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.
Claims
1. A reinforcement learning method suitable for cleaning robots, characterized in that, include: Multiple training tasks for reinforcement learning of cleaning robots are divided into multiple training task subsets according to the task data; Reinforcement learning is performed by iterating through all the training tasks in a subset of the training tasks. A dynamic scheduling score for a subset of training tasks is generated or updated based on the reinforcement learning results of all training tasks within that subset. The next training task subset is selected based on the dynamic scheduling score of multiple training task subsets; The task data includes: scene information, task information, and / or result information; Dynamic scheduling score of the training task subset The calculation formula is: ； in, For the first A set of risk scores for a subset of training tasks; To explore coefficients; This represents the cumulative number of scheduling attempts for all training task subsets. Indicates the first The historical call count for a subset of training tasks; The set of risk ratings It is obtained by accumulating or updating the single-task hazard values of multiple training tasks in the subset of training tasks; The single-task hazard value The calculation formula is: ； in, The corresponding value of the collision index for the associated training task; The corresponding value of the cumulative cost metric for the associated training task; The corresponding value for the task failure metric of the associated training task; , , These are the weighting coefficients.
2. The reinforcement learning method for cleaning robots according to claim 1, characterized in that, in, The corresponding value of the cumulative cost metric for the associated training task The calculation formula is: ； in, This represents the actual value of the cumulative cost metric for the training task. This is a limit value for the cumulative cost metric of the training task.
3. The reinforcement learning method for cleaning robots according to claim 2, characterized in that, When reinforcement learning is performed using the training task, if a physical collision occurs, the corresponding value of the collision metric associated with the training task... The value is 1; when reinforcement learning is performed using the training task, if there is no physical collision, the corresponding value of the collision metric associated with the training task. The value is 0.
4. The reinforcement learning method for cleaning robots according to claim 3, characterized in that, When reinforcement learning is performed using the training task, if the task fails, the corresponding value of the task failure metric associated with the training task... The value is 1; when reinforcement learning is performed using the training task, if the task is successful, the corresponding value of the task failure metric associated with the training task is 1. It is 0.
5. The reinforcement learning method for cleaning robots according to claim 4, characterized in that, in, The step of generating or updating the dynamic scheduling score of the training task subset based on the reinforcement learning results of all the training tasks in the training task subset includes: The single-task hazard value is calculated based on the reinforcement learning results of all the training tasks in a subset of the training tasks. cumulative value ; Based on the risk score of the subset of training tasks updated in the previous round. Obtain the updated set danger rating. ; Among them, the updated set hazard score in this round The calculation formula is: ； in, This is the smoothing coefficient.
6. The reinforcement learning method for cleaning robots according to any one of claims 1 to 5, characterized in that, in, The step of calling the next training task subset based on the dynamic scheduling score of multiple training task subsets includes: More reinforcement learning iterations are allocated to the subset of training tasks with higher dynamic scheduling scores.
7. The reinforcement learning method for cleaning robots according to any one of claims 1 to 5, characterized in that, The scene information includes: building identification, scene identification, and physical environment identification; The task information includes: task identifier, action identifier, and language instruction identifier; The results information includes: task success rate identifier, security cost identifier, and balance parameter identifier.
8. The reinforcement learning method for cleaning robots according to any one of claims 1 to 5, characterized in that, The reinforcement learning method is based on the SafeVLA model; the training task is constructed as a combination of 3D asset data and building layout data suitable for the SafeVLA model.
9. A reinforcement learning device suitable for cleaning robots, comprising: The classification module is used to divide multiple training tasks for reinforcement learning of cleaning robots into multiple training task subsets according to the task data. A learning module is used to perform reinforcement learning in an iterative manner using all of the training tasks in a subset of the training tasks. The scoring module is used to generate or update the dynamic scheduling score of the training task subset based on the reinforcement learning results of all the training tasks in the training task subset. The calling module is used to call the training task subset to be used in the next round based on the dynamic scheduling score of multiple training task subsets; The task data includes: scene information, task information, and / or result information; Dynamic scheduling score of the training task subset The calculation formula is: ； in, For the first A set of risk scores for a subset of training tasks; To explore coefficients; This represents the cumulative number of scheduling attempts for all training task subsets. Indicates the first The historical call count for a subset of training tasks; The set of risk ratings It is obtained by accumulating or updating the single-task hazard values of multiple training tasks in the subset of training tasks; The single-task hazard value The calculation formula is: ； in, The corresponding value of the collision index for the associated training task; The corresponding value of the cumulative cost metric for the associated training task; The corresponding value for the task failure metric of the associated training task; , , These are the weighting coefficients.
10. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the processors implement the method as described in any one of claims 1 to 8.
11. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.