Online reinforcement learning method and system, storage medium and degree product

Abstract

The embodiment of the invention provides an online reinforcement learning method and system, a storage medium and a degree product, and the method comprises the steps: controlling a robot to execute a target operation task through a strategy model, so as to obtain corresponding first training data; receiving first interaction information from a human expert to construct corresponding second training data; performing time difference loss calculation based on the first training data and the second training data to train a value evaluation model, and obtaining value evaluation information output by the value evaluation model; performing advantage calculation based on the second training data and the corresponding value evaluation information to obtain advantage information; performing fusion loss calculation based on the advantage information to train a strategy model; the fusion loss comprises advantage weighting strategy loss and near-end strategy optimization loss. According to the embodiment of the invention, the success rate of the robot on the target operation task can be improved under the conditions of lower human teaching cost and fewer environment interaction samples.

Description

Technical Field

[0001] This application relates to the fields of embodied intelligence and reinforcement learning technology, and particularly to online reinforcement learning methods and systems, storage media, and degree products. Background Technology

[0002] With the continuous development of robotics and artificial intelligence technologies, the demand for robots in fields such as industrial manufacturing, warehousing and logistics, home services, and medical assistance is increasing. To enable robots to autonomously complete tasks in complex and changing environments, researchers are gradually introducing machine learning-based methods to model and optimize robot control strategies.

[0003] In reinforcement learning methods, a large amount of environmental interaction data is typically used to train the policy model. However, in real-world robotic applications, the environment is complex and the cost of trial and error is high. Simply relying on automatic exploration often requires a large number of interactions, and unsafe or inefficient behaviors can easily occur during the exploration process, resulting in a low task success rate.

[0004] Based on this, embodiments of this application provide online reinforcement learning methods and systems, storage media, and degree products to improve related technologies. Summary of the Invention

[0005] The purpose of this application is to provide online reinforcement learning methods and systems, storage media, and degree products to improve the success rate of robots in performing target operation tasks under conditions of lower human teaching costs and fewer environmental interaction samples.

[0006] In a first aspect, embodiments of this application provide an online reinforcement learning method, the method comprising: controlling a robot to perform a target operation task through a policy model to obtain corresponding first training data; during the period when the policy model controls the robot to perform the target operation task, and / or, based on the execution result of the target operation task, receiving first interaction information from a human expert to construct corresponding second training data; the first interaction information being used to characterize at least one of teaching, intervention, and evaluation of the robot's execution process; calculating a temporal difference loss based on the first training data and the second training data to train a value assessment model, and obtaining value assessment information output by the value assessment model; calculating an advantage based on the second training data and its corresponding value assessment information to obtain advantage information; and calculating a fusion loss based on the advantage information to train the policy model; the fusion loss including an advantage-weighted policy loss and a proximal policy optimization loss.

[0007] In some embodiments, prior to the step of controlling the robot to perform a target operation task through a policy model to obtain corresponding first training data, the method further includes: constructing third training data for the target operation task based on second interaction information from a human expert; the second interaction information being used to characterize the teaching for the robot's execution process; and calculating behavioral cloning loss based on the third training data to train a preset initial policy model to obtain the policy model.

[0008] In some embodiments, prior to the step of controlling the robot to perform a target operation task through a strategy model to obtain corresponding first training data, the method further includes: controlling the robot to perform the target operation task through the strategy model to obtain corresponding fourth training data; during the period when the strategy model controls the robot to perform the target operation task, and / or, based on the execution result of the target operation task, receiving third interactive information from a human expert to construct corresponding fifth training data; the third interactive information is used to characterize at least one of teaching, intervention, and evaluation for the robot's execution process; and performing time difference loss calculation based on the fourth training data and the fifth training data respectively to train a preset initial value assessment model to obtain the value assessment model.

[0009] In some embodiments, the first training data, the second training data, the third training data, the fourth training data, and the fifth training data include state information and action information, and the first interaction information, the second interaction information, and the third interaction information are provided by human experts through remote operation devices.

[0010] In some embodiments, the step of calculating the advantage based on the second training data and its corresponding value assessment information to obtain advantage information includes: based on the value assessment information, using a preset advantage function to evaluate the relative merits of the state-action pairs in the corresponding second training data to obtain the corresponding advantage information.

[0011] In some embodiments, the step of calculating the fusion loss based on the advantage information to train the policy model includes: calculating the advantage-weighted policy loss based on the advantage information to obtain an advantage-weighted policy loss value; calculating the proximal policy optimization loss based on the advantage information to obtain a proximal policy optimization loss value; calculating the fusion loss based on the advantage-weighted policy loss value and the proximal policy optimization loss value to obtain a fusion loss value; and updating at least one model parameter of the policy model based on the fusion loss value.

[0012] In some embodiments, the step of calculating the advantage-weighted policy loss based on the advantage information to obtain an advantage-weighted policy loss value includes: for at least a portion of the state-action pairs in the second training data, inputting the corresponding state information into the policy model to obtain policy output information output by the policy model; assigning corresponding sample weights to the state-action pairs based on the advantage information; performing a weighted calculation on the matching result of the policy output information and the corresponding action information based on the sample weights, and determining the advantage-weighted policy loss value based on the weighted calculation result.

[0013] In some embodiments, the calculation of proximal policy optimization loss based on the advantage information to obtain a proximal policy optimization loss value includes: for the policy model, determining the policy output change ratio of the second policy version relative to the first policy version based on the first policy output information of the first policy version used to collect the second training data and the second policy output information of the second policy version used for training; constraining the update magnitude of the second policy version relative to the first policy version based on the policy output change ratio and the advantage information to obtain the proximal policy optimization loss value.

[0014] In some embodiments, the step of calculating the fusion loss based on the advantage-weighted strategy loss value and the proximal strategy optimization loss value to obtain the fusion loss value includes: performing a combined calculation on the advantage-weighted strategy loss value and the proximal strategy optimization loss value based on a preset fusion rule to obtain the fusion loss value; wherein the fusion rule includes at least one of weighted summation, linear combination, and phased joint optimization, so that the proximal strategy optimization loss value is used to limit the update magnitude corresponding to the advantage-weighted strategy loss value.

[0015] Secondly, embodiments of this application provide an online reinforcement learning system, the system comprising: a control module, configured to control a robot to perform a target operation task through a policy model to obtain corresponding first training data; an interaction module, configured to receive first interaction information from a human expert during the period when the policy model controls the robot to perform the target operation task, and / or, based on the execution result of the target operation task, to construct corresponding second training data; the first interaction information is used to characterize at least one of teaching, intervention, and evaluation for the robot's execution process; a training module, configured to perform temporal difference loss calculation based on the first training data and the second training data to train a value assessment model, and obtain value assessment information output by the value assessment model; and an advantage calculation module, configured to perform advantage calculation based on the second training data and its corresponding value assessment information to obtain advantage information; wherein, the training module is further configured to perform fusion loss calculation based on the advantage information to train the policy model; the fusion loss includes advantage-weighted policy loss and proximal policy optimization loss.

[0016] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0017] Fourthly, embodiments of this application provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the steps of any of the above methods.

[0018] This application provides an online reinforcement learning method and system, storage medium, and degree product. The method includes: controlling a robot to perform a target operation task through a policy model to obtain corresponding first training data; during the execution of the target operation task by the robot controlled by the policy model, and / or, based on the execution result of the target operation task, receiving first interaction information from a human expert to construct corresponding second training data; the first interaction information is used to characterize at least one of teaching, intervention, and evaluation for the robot's execution process; calculating temporal difference loss based on the first and second training data to train a value assessment model and obtain value assessment information output by the value assessment model; calculating advantage based on the second training data and its corresponding value assessment information to obtain advantage information; and calculating fusion loss based on the advantage information to train a policy model; the fusion loss includes advantage-weighted policy loss and proximal policy optimization loss.

[0019] This application's embodiments introduce teaching, intervention, and evaluation information from human experts during the robot's execution of a target operation task to construct second training data. This second training data, along with the first training data generated autonomously, is used for time-difference training of the value assessment model. This allows the value assessment results to more accurately reflect human task intent, thereby improving the reliability of advantage calculation. By constructing advantage-weighted policy loss and proximal policy optimization loss based on the aforementioned advantage information, and fusing them to train the policy model, the model can be guided to focus on learning high-quality human interaction data while limiting the magnitude of single policy updates, reducing policy divergence or performance degradation. By continuously executing the above policy update process within an online reinforcement learning framework, real-time correction and gradual improvement of the robot's policy in actual interactions can be achieved. This improves the robot's learning efficiency, convergence stability, and final task success rate for the target operation task under conditions of lower human teaching costs and fewer environmental interaction samples. Attached Figure Description

[0020] The embodiments of this application are further described below with reference to the accompanying drawings and specific implementation details.

[0021] Figure 1 This is a flowchart illustrating an online reinforcement learning method provided in an embodiment of this application.

[0022] Figure 2 This is a flowchart of an online reinforcement learning process provided in an embodiment of this application.

[0023] Figure 3 This is a structural block diagram of an online reinforcement learning system provided in an embodiment of this application.

[0024] Figure 4 This is a structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the embodiments of this application.

[0026] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0027] The following is a brief explanation of the technical terms that may be involved in the embodiments of this application.

[0028] Robot Manipulation Model: This refers to a model that controls a robot to complete operational tasks.

[0029] Online learning refers to a learning setup where a robot continuously collects data and updates its strategy while interacting with its environment. Data arrives in chronological order and is immediately used for learning, without relying on a pre-set, complete dataset. In contrast, offline learning utilizes only pre-collected data for training.

[0030] Reinforcement learning refers to a learning paradigm in which robots interact with their environment, select actions in a given state, obtain immediate rewards, and influence environmental changes in order to maximize long-term cumulative rewards.

[0031] Interactive learning refers to human-machine interaction, where humans provide instructional demonstrations and rate the machine's autonomous operations. Furthermore, the human-in-the-loop approach can influence data collection and strategy updates through demonstrations, interventions / corrections, ratings, or preference feedback, making the learning process more efficient, safer, and more aligned with task intent.

[0032] Advantage Weighted Actor Critic (AWAC) is a policy learning method based on an actor-critic framework, treating policy updates as weighted behavioral cloning. It uses an advantage function A(s,a) to weight actions in the data; a higher advantage results in a higher weight, thus favoring the imitation of "good actions" and suppressing "bad actions." AWAC can utilize demonstration data, offline replay data, and online interactive data simultaneously. In scenarios such as robot operation, it incorporates human teaching and ratings / preferences for autonomous execution into advantage estimation, mitigating distribution bias and accelerating convergence. In the advantage function A(s,a), s represents the state, and a represents the action.

[0033] Proximal Policy Optimization (PPO) restricts policy updates to the "proximal" region. It constructs a pruned proxy target using the probability ratio of importance sampling, thus suppressing performance crashes caused by excessively large single updates. It leverages the computational advantages of Q-networks and V-networks, employs mini-batch multi-round optimization, and trains the algorithm together with a value function loss and entropy regularization (to encourage exploration).

[0034] Most current robot operation learning algorithms are imitation learning algorithms, aiming to directly fit a policy using a limited set of taught trajectories to reproduce expert behavior. However, imitation learning is essentially "rote memorization," with poor generalization to out-of-distribution states, difficulty in self-correction, and a tendency to fail in cascaded failures over long-term tasks, resulting in limited actual success rates. The AWAC+PPO fusion algorithm proposed in this application, within an online reinforcement learning framework, incorporates human demonstrations and ratings into advantage estimation. It uses advantage-weighted approach for biased policy updates, enabling continuous correction during interactions. This allows the final policy level to narrow the gap with the taught approach, and even match or surpass it. Compared to related reinforcement learning methods, this application's embodiment achieves higher task success rates with the same or lower human and environmental interaction levels, significantly improving model training efficiency.

[0035] See Figure 1 , Figure 1 This is a flowchart illustrating an online reinforcement learning method provided in an embodiment of this application.

[0036] In order to improve the success rate of robots in performing target operation tasks with lower human teaching costs and fewer environmental interaction samples, this application provides an online reinforcement learning method, which includes steps S101 to S105.

[0037] Step S101: Control the robot to perform the target operation task through the strategy model in order to obtain the corresponding first training data.

[0038] Step S102: During the period when the strategy model controls the robot to perform the target operation task, and / or, based on the execution result of the target operation task, receive first interaction information from a human expert to construct corresponding second training data; the first interaction information is used to characterize at least one of teaching, intervention and evaluation for the robot's execution process.

[0039] Step S103: Calculate the time difference loss based on the first training data and the second training data to train the value assessment model and obtain the value assessment information output by the value assessment model.

[0040] Step S104: Calculate the advantage based on the second training data and its corresponding value assessment information to obtain advantage information.

[0041] Step S105: Calculate the fusion loss based on the advantage information to train the policy model; the fusion loss includes advantage-weighted policy loss and proximal policy optimization loss.

[0042] A robot can refer to an automated entity capable of performing target tasks and interacting with the environment. It may include a body structure, actuators, sensors, and a computational control unit. In some embodiments, a robot can be a robotic arm robot (such as a six-DOF industrial robotic arm, a collaborative robotic arm, or a dual-arm collaborative robotic arm), a mobile manipulation robot (such as an integrated robot with a mobile chassis and robotic arm), a humanoid robot, or a quadrupedal / wheeled robot. In some embodiments, a robot can also be a simulation robot or a digital twin robot, used for online interactive reinforcement learning training in a simulation environment. Configurable sensors for the robot include, for example, cameras, depth cameras, force / torque sensors, tactile sensors, encoders, inertial measurement units, and lidar; configurable actuators for the robot include, for example, articulated motors, grippers, suction cups, dexterous hands, and end-effectors (electric screwdrivers, nozzles, welding torches, etc.). Where feasible, this embodiment can be applied to online learning and strategy updates for robots of different forms in different operating scenarios.

[0043] A target operation task can refer to the operational objective that a robot needs to complete in a specific application scenario, or its corresponding execution process. It can be a single action task or a long-sequence task consisting of multiple sub-steps. In some embodiments, a target operation task may include grasping and placing operation tasks (such as grasping parts from a bin and placing them in a designated position, inserting objects into tray positioning holes, etc.), assembly operation tasks (such as plugging and unplugging connectors, assembling pins, tightening screws, pressing buckles, threading wires and drilling holes, etc.), tool operation tasks (such as using an electric screwdriver to fasten, applying glue, spraying, welding, grinding and polishing, etc.), opening and closing operation tasks (such as opening doors and drawers, pressing buttons, turning switches, adjusting knobs, etc.), cable management and organization operation tasks (such as organizing wire harnesses, wiring, stacking, arranging on trays, etc.), and flexible object operation tasks (such as folding fabric, sealing bags, connecting hoses, etc.). In some embodiments, the target operation task can be a multi-stage, long-sequential task, such as a chained process of "moving to the target area - locating the target - grasping - aligning - inserting / assembling - resetting," and can be combined with teaching, intervention, and evaluation information from human experts to achieve continuous correction and performance improvement in online reinforcement learning. The target operation task is usually related to a specific scenario, workpiece type, or human usage needs, and can be used as the learning objective and optimization direction for constraint policy models and value assessment models.

[0044] Online reinforcement learning refers to a learning method in which a robot continuously interacts with its environment and synchronously learns and updates its policies while performing actual target tasks. Unlike offline reinforcement learning, which relies solely on pre-collected data for training, online reinforcement learning allows the policy model to receive new states, actions, and feedback information in real time during operation, and iteratively updates the model parameters accordingly, thereby achieving a closed-loop control process of execution, learning, and optimization. In this embodiment, online reinforcement learning is used to enable the robot to continuously improve its operational capabilities in a real or high-fidelity simulation environment, and can introduce interactive guidance from human experts during the learning process to improve learning efficiency and safety.

[0045] A policy model can be used to characterize the decision-making logic of a robot in selecting actions to perform given environmental conditions. It can be, for example, a function model that maps a state space to an action space or action distribution. Policy models can be implemented in various ways, such as neural network-based policy models, probabilistic graphical model-based policy models, or parameterized function-based policy models. In some embodiments, the policy model can output the selection probability of discrete actions; in other embodiments, it can output parameter information (such as mean, variance, etc.) of continuous actions. In this embodiment, the policy model can be continuously trained and updated to control the robot to perform target operation tasks and generate corresponding training data.

[0046] The first training data can refer to the training data generated during the process of the robot executing a target operation task controlled by the policy model. This data includes, for example, the robot's state information, action information, and environmental feedback information during execution. The first training data reflects the robot's behavioral results in autonomously executing the task based on the current policy model, and is used to characterize the actual performance of the policy model under the current parameter state. In some embodiments, the first training data can be stored in the form of "state-action-reward-next state," while in other embodiments it may only contain state information (such as state trajectory) and action information (such as action trajectory). The reward can be used to characterize the immediate feedback obtained by the robot after executing the corresponding action information under the current state information. This immediate feedback can originate from environmental feedback and / or evaluation information provided by human experts.

[0047] The first interactive information can refer to the interactive feedback information provided by human experts during or after the robot performs a target operation task, used to guide or evaluate the robot's behavior. The first interactive information may include, but is not limited to, at least one of teaching information, intervention information, and evaluation information. Teaching information, for example, is direct control of the robot's actions by a human expert through a teleoperation device; intervention information, for example, is correction of erroneous actions that the robot is about to perform or has already performed during human-robot collaboration; evaluation information, for example, is scoring or providing preference feedback on the robot's task execution results. By introducing the first interactive information, human experience and task intent can be effectively integrated into the reinforcement learning process. The execution result of the target operation task can be used to characterize the overall effect of the robot in completing the target operation task, which may include at least one of whether the task was completed, the quality of task completion, and whether the execution process met preset constraints.

[0048] The second training data can refer to training data constructed based on the first interaction information, reflecting the teaching, correction, or evaluation results of human experts on the robot's operation process. The second training data may include state-action pairs generated by human teaching, action trajectories replaced or corrected by human intervention, or execution trajectories with evaluation labels. In this embodiment, the second training data is used to guide the policy model to focus on learning high-quality human operational experience, thereby improving the stability of policy learning and sample utilization efficiency.

[0049] Temporal difference loss calculation can be a loss calculation method used to train a value assessment model, utilizing the difference in value estimates between adjacent time steps to update model parameters. In specific implementations, the temporal difference loss can be constructed based on the value estimate of the current state, immediate feedback, and the value estimate of the next state, thereby enabling the value assessment model to gradually approximate the true long-term return. In this embodiment, the first training data can be used to calculate the temporal difference loss to train the value assessment model, and the second training data can also be used to calculate the temporal difference loss to train the value assessment model, allowing the value assessment model to simultaneously learn the value information corresponding to autonomously executed behaviors and human-guided behaviors.

[0050] Value assessment models can be used to estimate the long-term returns of a robot under different states or state-action pairs, and their output reflects the expected benefits of the current strategy in a given context. Value assessment models can be implemented in various forms, such as state-value models, action-value models, or combinations thereof. In this embodiment, the value assessment model can be trained using temporal difference loss and outputs value assessment information, providing a foundation for subsequent advantage calculation and policy updates.

[0051] Valuation information refers to the estimates of the long-term returns of a state or state-action relationship, output by a valuation model. Valuation information can include state value, action value, or their related derivatives, reflecting the merits of the current decision.

[0052] Advantage calculation refers to the process of calculating the superiority or inferiority of a given action relative to a reference action based on second training data and its corresponding value assessment information. Advantage calculation yields advantage information, which characterizes the incremental benefit of a particular action relative to a reference action in a specific state.

[0053] Fusion loss calculation refers to the joint calculation of multiple losses to form a comprehensive loss function used to train the policy model. In this embodiment, the fusion loss includes advantage-weighted policy loss and proximal policy optimization loss. Advantage-weighted policy loss can be used to weight the learning contributions of different training samples based on advantage information, thereby highlighting high-advantage behaviors; proximal policy optimization loss can be used to constrain the policy update magnitude, reducing excessive changes in the policy during a single update. By calculating the fusion loss, the relevance and effectiveness of policy updates can be improved while ensuring learning stability.

[0054] Advantage-weighted policy loss can be a form of loss that weights and guides the policy learning process based on advantage information. By assigning different sample weights to different state-action pairs, it makes the policy model focus more on information with higher advantages during training. In this embodiment, advantage-weighted policy loss can effectively integrate teaching and intervention data provided by human experts, enabling the policy model to gradually move towards high-quality operational behaviors during online learning.

[0055] Proximal policy optimization loss can be used to limit the magnitude of change in the policy model during the update process relative to the policy model at the time of data acquisition. For example, by constraining the proportion of policy output change, it can reduce performance degradation caused by drastic policy fluctuations. In this embodiment, the proximal policy optimization loss and the advantage-weighted policy loss work synergistically to improve the stability and reliability of the online reinforcement learning process while ensuring that the policy update direction is reasonable.

[0056] In some embodiments, the online reinforcement learning method can be implemented using an actor-critic architecture. The policy model can be implemented by an actor model, which outputs action decisions for the robot to perform the target operation task based on the environmental state; the value assessment model can be implemented by a critic model, which estimates the long-term reward corresponding to a state or state-action pair and outputs value assessment information for policy updates.

[0057] In some embodiments, the actor model and the critic model can be implemented using independent neural network structures, or they can be implemented by sharing some network parameters. For example, the actor model and the critic model can share a front-end network for state feature extraction, while setting up separate network branches at the back-end for policy output and value evaluation.

[0058] In this embodiment, during the continuous interaction between the robot and the real or simulated environment, a strategy model is used to control the robot to perform target operation tasks to form first training data. Simultaneously or after the task execution, teaching, intervention, or evaluation information from human experts is introduced to construct second training data, thereby achieving joint data acquisition of "autonomous exploration-human guidance" within an online learning framework. Subsequently, based on the first and second training data, a value assessment model is trained using temporal difference loss, enabling the model to estimate the long-term rewards of different state-action pairs and output corresponding value assessment information. Building upon this, for the second training data containing human interaction information, advantage calculation is performed in conjunction with the value assessment information to obtain advantage information characterizing the superiority or inferiority of the current action relative to the reference strategy. Finally, advantage-weighted policy loss and proximal policy optimization loss are constructed using the advantage information, and the strategy model is trained and updated using fusion loss. The advantage-weighted policy loss guides the strategy model to prioritize learning human teaching or intervention actions with high advantage, while the proximal policy optimization loss constrains the policy update magnitude, thereby achieving continuous policy optimization while ensuring learning stability.

[0059] This embodiment introduces teaching, intervention, and evaluation information from human experts during the robot's execution of the target operation task to construct a second training data set. This second training data is then used together with the first training data generated during autonomous execution for time-difference training of the value assessment model. This allows the value assessment results to more accurately reflect human task intent, thereby improving the reliability of advantage calculation. By constructing advantage-weighted policy loss and proximal policy optimization loss based on the aforementioned advantage information, and fusing them to train the policy model, the model can be guided to focus on learning high-quality human interaction data while limiting the magnitude of single policy updates, reducing policy divergence or performance degradation. By continuously executing the above policy update process within an online reinforcement learning framework, the robot's policy can be corrected in real-time and gradually improved in actual interactions. This improves the robot's learning efficiency, convergence stability, and final task success rate for the target operation task under conditions of lower human teaching costs and fewer environmental interaction samples.

[0060] In some embodiments, prior to the step of controlling the robot to perform a target operation task through a policy model to obtain corresponding first training data, the method may further include: constructing third training data for the target operation task based on second interaction information from a human expert; the second interaction information being used to characterize the teaching for the robot's execution process; and calculating behavioral cloning loss based on the third training data to train a preset initial policy model to obtain the policy model.

[0061] The second interactive information can refer to information actively provided by human experts during the robot's execution of a target task, used to teach the robot correct operational behavior and to build high-quality initial training samples before online reinforcement learning begins. The second interactive information can include teaching trajectory information formed by human experts directly controlling the robot to perform actions through teleoperation devices, or it can include operational sequence information where human experts gradually guide the robot to complete key steps of the task. In this embodiment, the second interactive information can provide a clear and stable behavioral reference for the policy model, thereby reducing the low learning efficiency or execution risk caused by the policy model relying on random exploration in the initial stage.

[0062] The third training data can refer to training data constructed based on the second interaction information for pre-training the initial policy model, reflecting the teaching behavior of human experts in a specific target operation task. The third training data may include data samples in the form of state-action pairs, or complete combinations of state trajectories and action trajectories, or teaching sequence data with time-series labels. In some embodiments, the third training data only contains action information generated by human teaching; in other embodiments, it may be further constructed by combining state information. By using the third training data, the policy model can acquire a preliminary understanding of the basic operation patterns of the task before entering the online reinforcement learning stage.

[0063] Teaching refers to the process by which human experts explicitly demonstrate to a robot how to complete a target task, aiming to directly inject human experience and prior knowledge of the task into an online reinforcement learning system. In this embodiment, teaching can be implemented in various forms, such as real-time control of the robot's actions via teleoperation devices, gradual recording of key operation points via a teach pendant, or recording standard operating procedures in a virtual simulation environment. Teaching behaviors typically have a high success rate and stability, and therefore can serve as an important basis for the initial behavior distribution learned by the policy model.

[0064] Behavioral cloning loss calculation can be a loss calculation method based on supervised learning. For example, it optimizes the policy model parameters by measuring the difference between the actions output by the policy model and the actions taught by human experts in the third training data. In specific implementations, behavioral cloning loss can employ mean squared error loss, cross-entropy loss, or other loss functions suitable for the action space, guiding the policy model to output decision results consistent with human-taught actions under the same state conditions. Through behavioral cloning loss calculation, the policy model can quickly approximate the distribution of human expert actions in the parameter space.

[0065] The initial policy model can refer to a policy model that has not undergone task-related learning or has only undergone general initialization before behavior cloning pre-training. Its parameters may not contain effective decision information specific to the target operation task. The initial policy model can be obtained based on random parameter initialization or based on other general task or pre-trained models. In this embodiment, by training the initial policy model through behavior cloning loss calculation, it can be transformed into a policy model with basic task execution capabilities, providing a stable starting point for the subsequent online reinforcement learning stage.

[0066] A policy model can refer to a model obtained by training an initial policy model using behavioral cloning loss, which can output corresponding action decisions based on the environmental state. In this embodiment, after completing behavioral cloning pre-training, the policy model already possesses the basic execution capability for the target operation task and can participate in data collection and policy optimization as a behavioral policy model in the subsequent online reinforcement learning stage.

[0067] In this embodiment, before formally entering online reinforcement learning, second interaction information provided by human experts for the target operation task is first introduced, and third training data is constructed based on the second interaction information, enabling the initial policy model to acquire basic task execution capabilities without relying on random exploration. Subsequently, the behavior cloning loss is calculated on the preset initial policy model using the third training data, causing the policy model to converge to the action distribution taught by the human expert in the parameter space, thereby establishing an initial mapping relationship between "state-action". Through this pre-training process, the policy model has a basic understanding of the key action patterns and operation sequences of the task before entering the subsequent online interaction and reinforcement learning stages, providing a stable initial policy foundation for subsequent policy optimization that combines human interaction information and advantage-driven approaches.

[0068] This embodiment introduces a behavior cloning pre-training step based on human expert instruction before online reinforcement learning, enabling the policy model to possess basic executable operational capabilities from the initial stage. This mitigates the safety risks and efficiency issues associated with randomly initialized policies exploring real-world environments. By calculating the behavior cloning loss based on the third training data, the policy model rapidly approximates the human expert's instruction behavior in the parameter space, significantly reducing the dependence on the number of environmental interaction samples and the frequency of human intervention in subsequent online reinforcement learning stages. By combining behavior cloning pre-training with subsequent online reinforcement learning based on advantage information, the policy performance can be continuously improved while maintaining initial policy stability, thereby enhancing the robot's learning efficiency, convergence speed, and overall task success rate for the target operation task.

[0069] In some embodiments, prior to the step of controlling the robot to perform a target operation task through a strategy model to obtain corresponding first training data, the method may further include: controlling the robot to perform the target operation task through the strategy model to obtain corresponding fourth training data; during the period when the strategy model controls the robot to perform the target operation task, and / or, based on the execution result of the target operation task, receiving third interactive information from a human expert to construct corresponding fifth training data; the third interactive information being used to characterize at least one of teaching, intervention, and evaluation for the robot's execution process; and performing time difference loss calculations based on the fourth and fifth training data respectively to train a preset initial value assessment model to obtain the value assessment model.

[0070] The fourth training data refers to the training data generated during the warm-up phase of the value assessment model, when the strategy model controls the robot to perform the target operation task. This data reflects the autonomous execution behavior of the strategy model under the current model parameter state. The fourth training data may include the robot's state information, action information, environmental feedback information, and temporal sequence relationships during execution, used to characterize the actual behavioral distribution of the strategy model in the target operation task. In this embodiment, the fourth training data can be used to provide the value assessment model with samples related to the current strategy behavior, enabling the value assessment model to learn the value estimation of the autonomous execution results of the strategy model.

[0071] Third-party interactive information refers to the interactive feedback provided by human experts during or after the robot performs a target task, used to guide, correct, or evaluate the robot's behavior. Third-party interactive information can include at least one of teaching information, intervention information, and evaluation information. Examples include directly controlling the robot to complete some or all of its actions via teleoperation devices, manually correcting the robot when it is about to perform an erroneous action, or scoring or labeling the robot's task performance.

[0072] The fifth training data can refer to training data constructed based on the third interaction information, reflecting the teaching, intervention, or evaluation results of human experts on the robot's execution of the target operation task. The fifth training data may include state-action pairs generated by human teaching, manually corrected action trajectories, execution sequences with accompanying evaluation labels, etc. In this embodiment, the fifth training data can be used to enable the value assessment model to not only consider the autonomous execution results of the policy model during training, but also to learn the behavioral patterns and corresponding value levels expected by human experts.

[0073] The initial value assessment model can refer to a model that has not undergone learning related to the target operation task before the value assessment model warm-up training, or a model that only has general initialization parameters. The initial value assessment model can be constructed using a random parameter initialization method, or it can be pre-trained based on other task or general data. In this embodiment, by training the initial value assessment model with temporal difference loss based on the fourth and fifth training data, it can gradually acquire the ability to assess the long-term rewards of different states or state actions in the target operation task.

[0074] A value assessment model can refer to an initial value assessment model trained using temporal difference loss, used to output value assessment information. It can estimate the long-term rewards of the robot in different states or state-action pairs during a target task. In this embodiment, after completing pre-training, the value assessment model can serve as the basis for subsequent advantage calculations and strategy model optimization.

[0075] In this embodiment, before the policy model formally enters online reinforcement learning training based on fusion loss, the current policy model is used to control the robot to execute the target operation task to obtain fourth training data reflecting the autonomous execution behavior of the policy model. Simultaneously, teaching, intervention, or evaluation information provided by human experts is introduced during the robot's execution to construct corresponding fifth training data. Subsequently, temporal difference loss is calculated based on the fourth and fifth training data respectively, and a preset initial value assessment model is trained. This allows the value assessment model to simultaneously consider the current behavior distribution of the policy model and the expected behavioral results of the human experts during the learning process. Through this pre-training process, the value assessment model possesses a basic ability to assess the long-term returns of different states or state actions in the target operation task before entering the subsequent advantage calculation and policy optimization stages, thus providing a reliable value reference for the stable updating of the subsequent policy model.

[0076] This embodiment trains the initial value assessment model using temporal difference loss based on fourth training data generated autonomously by the policy model and fifth training data guided by human experts before online reinforcement learning. This enables the value assessment model to acquire value estimation capabilities highly correlated with the target operation task at an early stage, thereby reducing policy update instability caused by inaccurate value estimation during subsequent policy model training. By introducing teaching, intervention, and evaluation information from human experts during the value assessment model warm-up process, the value assessment results can be more closely aligned with human task intent, providing a more reliable foundation for advantage calculation. By conducting policy optimization based on advantage information after the value assessment model has basic evaluation capabilities, the convergence speed and stability of online reinforcement learning can be improved, and ineffective exploration and repeated trial and error can be reduced, thereby improving the overall learning efficiency and execution success rate of the robot's target operation task.

[0077] In some embodiments, the first training data, the second training data, the third training data, the fourth training data, and the fifth training data may include state information and action information.

[0078] The training data, including the first training data, second training data, third training data, fourth training data, and fifth training data, refers to the training sample sets constructed under different learning stages and data source conditions in this embodiment. These sets are used to support the training and updating of the policy model and value evaluation model at each stage. Although the various types of training data differ in their generation stage and purpose, they can adopt a unified or compatible representation at the data structure level to facilitate model training and data reuse. In some embodiments, the aforementioned training data may include state information and action information, thereby forming state-action pairs or state-action sequences. In other embodiments, the aforementioned training data may further include additional fields such as reward information, timestamp information, or next state information. By uniformly defining the training data at different stages at the data structure level, the training process can be simplified and the versatility of the algorithm implementation can be improved.

[0079] State information can be used to characterize the environmental and internal states of a robot during the execution of a task, reflecting task-related information currently perceptible to the robot. In some embodiments, state information may include the robot's pose, joint angles, joint velocities, end effector state, force or torque sensing information, etc. In other embodiments, state information may also include environmental perception information, such as visual features, point cloud features, or distance information from cameras, depth sensors, or LiDAR.

[0080] Motion information can be used to characterize the control commands or operational behaviors executed by a robot in a given state. For example, it reflects the specific control results of a strategy model or human expert on the robot's motion. In some embodiments, motion information may include control commands in joint space (such as joint angle, angular velocity, or torque commands), and in other embodiments, it may also include end-effector pose change commands, velocity commands, or force control commands in Cartesian space.

[0081] In some embodiments, the first training data, the second training data, the third training data, the fourth training data, and the fifth training data can form a playback dataset. The second training data, the third training data, and the fifth training data can form a demonstration dataset.

[0082] A replay dataset refers to a collection of training data generated by a robot at different learning stages, which can be used to support repeated training and updates of policy models and value evaluation models. In this embodiment, the first, second, third, fourth, and fifth training data can all be uniformly stored in the replay dataset, forming a comprehensive data pool covering the robot's autonomous execution behavior, human teaching behavior, and human-robot interaction behavior. By constructing a replay dataset, historical data can be repeatedly sampled and utilized during training, reducing reliance on only the latest interaction data for learning, thereby improving sample utilization and enhancing the stability of the online reinforcement learning process.

[0083] The demonstration dataset can refer to a subset of training data obtained by further filtering from the replay dataset, containing explicit human expert intent and guidance information. In this embodiment, the second, third, and fifth training data, since they directly originate from the teaching, intervention, or evaluation information of human experts, can collectively constitute the demonstration dataset. The demonstration dataset can be used to guide the policy model to learn high-quality human operational behaviors and plays a crucial role in stages such as advantage calculation and advantage-weighted policy loss calculation. By distinguishing the demonstration dataset from other autonomously executed data, the key demonstration information provided by human experts can be highlighted without losing exploratory capabilities.

[0084] The replay dataset and the demonstration dataset have a containment relationship. The replay dataset, as the higher-level dataset, covers all training data generated by the robot at each learning stage, while the demonstration dataset, as a subset, focuses on data samples containing human expert interaction information. In this embodiment, by simultaneously utilizing the replay dataset and the demonstration dataset, the current behavior distribution of the policy model and the expected behavior distribution of humans can be considered during the value assessment model training stage. Furthermore, during the policy model training stage, the focus is on learning high-quality human teaching behaviors by incorporating advantage information, thus achieving a balance between exploration and stability.

[0085] In practical implementations, the replay dataset can be used for temporal difference training of the value assessment model and online updates of the policy model, while the demonstration dataset can serve as an important input source for advantage calculation and advantage-weighted policy loss calculation. In some embodiments, samples in the demonstration dataset can be assigned higher training weights to enhance the guiding role of human instruction in policy updates. In other embodiments, the demonstration dataset and the replay dataset can also be jointly trained using different sampling strategies. Through the above-mentioned dataset partitioning and collaborative use, fine-grained control over the online reinforcement learning process can be achieved. In summary, the replay dataset addresses the question of "can the data be reused repeatedly?", while the demonstration dataset addresses the question of "which data is more worthwhile to learn from?"

[0086] In some embodiments, the first interaction information, the second interaction information, and the third interaction information may be provided by human experts through a remotely operated device.

[0087] The first, second, and third interactive information refer to the guidance information provided by human experts to the robot at different stages of the robot's operation task through interaction. Although the above interactive information differs in its stage of action and specific purpose, its essence is to introduce human experience, task intent, or operational preferences into the learning process.

[0088] Teleoperation devices can refer to interactive devices used to support human experts in remotely or locally controlling robots to perform actions. For example, they can translate human operational intentions into control commands that the robot can execute. In some embodiments, teleoperation devices may include force feedback handles, spatial mice, data gloves, joysticks, VR / AR (virtual reality / augmented reality) controllers, touch interfaces, or keyboard and mouse input devices. In other embodiments, teleoperation devices may include input devices based on vision or motion capture systems for acquiring the movement trajectories of human experts. The interactive information acquired through teleoperation devices can be used to construct training data, thereby enabling the teaching, correction, or evaluation of robot operation strategies.

[0089] This embodiment enables information interaction between human experts and robots through teleoperation devices, allowing human experts to participate in control decisions in real-time or offline during robot task execution. In this embodiment, the interaction mechanism can be used to generate first, second, and third interaction information, enabling human experts to teach, intervene in, or evaluate robot behavior at different learning stages. This mechanism allows for the introduction of high-quality human guidance information without entirely relying on environmental reward design, improving the safety, stability, and sample efficiency of the online reinforcement learning process.

[0090] In some embodiments, the step of calculating the advantage based on the second training data and its corresponding value assessment information to obtain advantage information may include: using a preset advantage function to evaluate the relative merits of the state-action pairs in the corresponding second training data based on the value assessment information to obtain the corresponding advantage information.

[0091] In this embodiment, the value assessment information provides a quantitative basis for advantage calculation, enabling the superiority or inferiority of different state-action pairs to be compared on a unified scale.

[0092] Advantage calculation refers to the process of calculating the relative gains of a robot performing a specific action in a given state, based on second training data and its corresponding value assessment information. Unlike directly using value assessment information, advantage calculation focuses on the degree of improvement of an action relative to a reference action (such as an average action or a baseline action). In this embodiment, advantage calculation is used to extract key behavioral features from the second training data that can significantly improve task performance, providing a core basis for subsequent advantage-driven strategy updates.

[0093] Advantage information can refer to the quantitative result obtained through advantage calculation, which characterizes the superiority or inferiority of a certain state-action pair relative to a reference action. Advantage information is represented, for example, in numerical form, where its sign and magnitude reflect the direction and magnitude of improvement of the action relative to the reference action, respectively. In this embodiment, advantage information can be used to distinguish the contributions of different training samples in the policy update process, thereby guiding the policy model to prioritize learning actions with higher advantages.

[0094] The advantage function can be a functional form used to convert value assessment information into advantage information. It quantifies the relative merits of actions by combining or comparing different value quantities. The advantage function can be implemented in various ways, such as a function based on the difference between the state-action value and the state value, a function based on the difference between the expected return and the reference return, or a function formed by normalizing and smoothing the value assessment information. In this embodiment, the advantage function can be pre-defined to flexibly adapt to different operational tasks and the output format of the value assessment model without changing the overall learning framework.

[0095] The relative merits evaluation of state-action pairs refers to comparing and analyzing the reward levels corresponding to different actions under the same state conditions to determine the superiority or inferiority of each action relative to a reference action. This relative evaluation process can be based on the value assessment information output by a value assessment model and achieve a unified measurement through a dominance function. In this embodiment, by performing a relative merits evaluation on the state-action pairs in the second training data, the high-quality actions corresponding to human expert teaching or intervention can be effectively highlighted, while action samples that do not significantly contribute to the task or have a negative impact can be suppressed.

[0096] In this embodiment, after the value assessment model has been trained based on training data and outputs value assessment information, a pre-defined advantage function is introduced to further analyze and process the second training data containing human interaction information. For example, for each state-action pair in the second training data, combined with the state value or state-action value information output by the value assessment model, the advantage function is used to calculate the relative benefit difference between the current action and the reference action, thereby achieving a quantitative assessment of the superiority or inferiority of different actions in the same state. Through this relative assessment process, behavioral results that were originally difficult to compare directly can be transformed into advantage information on a unified scale, providing a clear and calculable decision-making basis for subsequent advantage-driven strategy optimization.

[0097] This embodiment, based on the second training data and its corresponding value assessment information, uses a preset advantage function to evaluate the relative merits of state-action pairs. This further transforms the long-term reward information output by the value assessment model into advantage information reflecting the relative improvement of actions, thereby mitigating the instability caused by relying solely on absolute value for policy updates. By using advantage information to distinguish different behaviors, high-quality actions corresponding to human expert teaching or intervention can be highlighted, while low-quality or ineffective actions can be suppressed from interfering with policy updates. By introducing the aforementioned advantage information into the policy optimization process, the update direction of the policy model becomes clearer, and the gradient information becomes more concentrated, thereby improving the convergence speed, stability, and final performance of the robot's task in online reinforcement learning.

[0098] In some embodiments, the step of calculating the fusion loss based on the advantage information to train the policy model may include: calculating the advantage-weighted policy loss based on the advantage information to obtain an advantage-weighted policy loss value; calculating the proximal policy optimization loss based on the advantage information to obtain a proximal policy optimization loss value; calculating the fusion loss based on the advantage-weighted policy loss value and the proximal policy optimization loss value to obtain a fusion loss value; and updating at least one model parameter of the policy model based on the fusion loss value.

[0099] In some embodiments, the step of calculating the advantage-weighted strategy loss based on the advantage information to obtain the advantage-weighted strategy loss value may include: calculating the advantage-weighted strategy loss based on the advantage information using a preset advantage-weighted strategy loss function to obtain the advantage-weighted strategy loss value.

[0100] In some embodiments, the step of calculating the near-end policy optimization loss based on the advantage information to obtain the near-end policy optimization loss value may include: calculating the near-end policy optimization loss based on the advantage information using a preset near-end policy optimization loss function to obtain the near-end policy optimization loss value.

[0101] In some embodiments, the step of calculating the fusion loss based on the advantage-weighted policy loss value and the proximal policy optimization loss value to obtain the fusion loss value may include: calculating the fusion loss using a preset fusion loss function based on the advantage-weighted policy loss value and the proximal policy optimization loss value to obtain the fusion loss value.

[0102] Policy model parameter update can refer to the process of adjusting at least one trainable parameter (also known as a learnable parameter) in the policy model based on the fusion loss, which can be achieved through gradient calculation and backpropagation mechanism.

[0103] This embodiment uses advantage information as a common input to calculate the advantage-weighted policy loss and the proximal policy optimization loss separately, and then performs unified scheduling of the two at the fusion loss level, achieving fine-grained control over the policy model training process. On the one hand, advantage information determines the "learning focus" of policy updates; on the other hand, the proximal policy optimization mechanism constrains the "range of change" in policy updates. Their synergistic effect enables the policy model to achieve stable, efficient, and sustainable online learning in complex operational tasks and human-computer interaction environments.

[0104] In this embodiment, a fusion loss calculation mechanism based on advantage information is introduced to provide stable and guided updates to the policy model during online reinforcement learning. Specifically, firstly, based on the value assessment information output by the value assessment model, advantage information reflecting the relative superiority or inferiority of state-action pairs is obtained through an advantage function. Subsequently, using this advantage information as the core basis, on the one hand, an advantage-weighted policy loss value is calculated, enabling the policy model to assign greater update weights to action samples with higher advantages during the update process, thereby enhancing the learning of high-quality behaviors. On the other hand, a proximal policy optimization loss value is calculated based on the advantage information. By constraining the output change amplitude between adjacent update versions of the policy model, the step size range of policy updates is limited, reducing drastic oscillations in the policy model. Based on this, the advantage-weighted policy loss and the proximal policy optimization loss are fused to form a unified fusion loss value, which is then used to update the model parameters of the policy model, thereby ensuring the stability of policy updates while achieving continuous reinforcement learning of high-quality behaviors.

[0105] This embodiment calculates the advantage-weighted policy loss and the proximal policy optimization loss separately based on advantage information, and then fuses them for calculation. This allows the policy model to simultaneously consider the "effectiveness of the learning direction" and the "stability of the update process" during the update process, thereby improving the policy oscillation or slow convergence problems caused by relying on only a single loss function. By using advantage information to distinguish the training contributions of different state-action pairs, the policy model is guided to prioritize learning actions with higher long-term reward potential, improving the efficiency of policy learning and the task success rate. By introducing the proximal policy optimization loss to constrain the policy update magnitude, the risk of performance degradation caused by changes in data distribution or human-computer interaction intervention during online reinforcement learning is reduced, enhancing the safety and robustness of the training process. Therefore, this embodiment can achieve stable convergence and continuous performance improvement of the policy model with less human interaction cost and fewer environmental interactions, making it suitable for application scenarios with high requirements for safety and learning efficiency, such as robot operation.

[0106] In some embodiments, the step of calculating the advantage-weighted policy loss based on the advantage information to obtain an advantage-weighted policy loss value may include: for at least a portion of the state-action pairs in the second training data, inputting the corresponding state information into the policy model to obtain policy output information output by the policy model; assigning corresponding sample weights to the state-action pairs based on the advantage information; performing a weighted calculation on the matching result of the policy output information and the corresponding action information based on the sample weights, and determining the advantage-weighted policy loss value based on the weighted calculation result.

[0107] A state-action pair (i.e., a state information-action information pair) refers to the associated sample formed between the state of a robot at a certain moment and the action actually performed in that state, used to characterize the decision-making behavior of the policy model in a specific context. In this embodiment, the state-action pair can be used as the basic calculation unit of the advantage-weighted policy loss, used to evaluate the relative advantages and disadvantages of different operations in a given state.

[0108] The policy model can be implemented using neural networks, decision trees, parameterized functions, or combinations thereof. In this embodiment, during the training phase, the policy model can receive state information as input and output policy output information, which is used to match and calculate loss with the actual action information in the second training data.

[0109] Policy output information refers to the information generated by the policy model after receiving state information, used to characterize the degree of preference for each candidate action or the selection result. Policy output information can take the form of action probability distribution, action parameter distribution, action scoring results, direct control instructions, or other quantitative forms that can reflect the policy decision result. In the calculation of the advantage-weighted policy loss, policy output information can be compared with the action information in the second training data to evaluate the degree of support the current policy model provides for the target action in that state.

[0110] Advantage information can be positive, negative, or zero, and is used to distinguish the degree to which different state actions contribute to the improvement of policy performance.

[0111] Sample weights refer to the weight parameters assigned to different state-action pairs, used to adjust the influence of each training sample in the policy loss calculation process. In this embodiment, sample weights can be determined based on the advantage information of the corresponding state-action pair, so that samples with higher advantages have a higher proportion in loss calculation and parameter updates. Sample weights can be generated using methods such as linear mapping, nonlinear function mapping, normalization processing, or piecewise mapping, thereby adapting to different task scenarios and training requirements.

[0112] The matching result between the policy output information and the corresponding action information can refer to the consistency or conformity assessment result between the policy output information output by the policy model and the corresponding action information in the second training data. This matching result can be obtained by calculating action probability, action parameter deviation, confidence score, or other similarity indicators. In this embodiment, the matching result is used to reflect the degree of support the current policy model provides for the corresponding action in the second training data under a given state.

[0113] The advantage-weighted strategy loss value refers to the strategy loss determined by weighting the matching results based on sample weights. It is used to comprehensively reflect the overall performance of the strategy model on different state-action pairs. In this embodiment, the advantage-weighted strategy loss value can simultaneously reflect the difference in action matching degree and behavior quality, thereby providing a more refined and effective optimization signal for updating the strategy model parameters.

[0114] In this embodiment, at least a portion of state-action pairs are first selected from the second training data, and the corresponding state information is input into the policy model to obtain the policy output information of the policy model in that state. This policy output information is used to characterize the policy model's preference for each selectable action. Subsequently, based on the advantage information, corresponding sample weights are assigned to each state-action pair so that the contribution of different training samples in the loss calculation matches their relative merits. On this basis, according to the sample weights, the matching results between the policy output information and the corresponding action information in the second training data are weighted and calculated to obtain the advantage-weighted policy loss value that reflects the current behavioral quality of the policy model.

[0115] This embodiment assigns sample weights to state-action pairs based on advantage information and performs weighted calculations on the matching results between policy output information and actual action information. This enables the policy model to distinguish the importance of different training samples during training, thereby prioritizing the reinforcement of actions with higher relative returns and improving the targeting and efficiency of policy learning. By combining secondary training data formed through human teaching, intervention, or high-quality interaction with advantage information, the policy model can more accurately absorb effective operational experience provided by human experts during online reinforcement learning, reducing performance fluctuations caused by ineffective exploration. By introducing an advantage-weighted policy loss value, policy model updates no longer simply depend on the number of samples but are guided by the quality of behavior, thereby improving the convergence speed and task success rate of the policy model in complex operational tasks and enhancing the stability and reliability of the overall training process.

[0116] In some embodiments, the calculation of proximal policy optimization loss based on the advantage information to obtain a proximal policy optimization loss value may include: for the policy model, determining the policy output change ratio of the second policy version relative to the first policy version based on the first policy output information of the first policy version used to collect the second training data and the second policy output information of the second policy version used for training; constraining the update magnitude of the second policy version relative to the first policy version based on the policy output change ratio and the advantage information to obtain the proximal policy optimization loss value.

[0117] In this embodiment, the policy model exists in different parameter states at different training stages, thereby forming multiple policy versions to support the near-end policy optimization process.

[0118] The first strategy version can refer to the state of the strategy model parameters used when collecting the second training data. It is used to actually control the robot to perform operational tasks and generate corresponding training samples (i.e., the second training data). The parameters of the first strategy version are kept fixed during the sampling phase, and its output strategy information truly reflects the distribution of strategy behavior at the time of data collection. In this embodiment, the first strategy version can be used as a reference strategy in near-end strategy optimization to measure the magnitude of change brought about by subsequent strategy updates.

[0119] The second policy version can refer to the current parameter state used to update the policy model during training. It can be gradually adjusted by fusing loss values ​​based on the first policy version. The second policy version does not directly participate in the collection of the second training data, but is used as the policy to be updated to calculate the loss and perform parameter updates. In this embodiment, by comparing the output differences between the second policy version and the first policy version, the policy update process can be effectively constrained.

[0120] The first and second policy output information can refer to the decision result representations generated by different policy versions of the policy model under the same state input conditions, reflecting the degree of preference of the corresponding policy version for each candidate action. In this embodiment, the first and second policy output information correspond to the output results of the first and second policy versions under the same state conditions, respectively, and are the basic data for calculating the proportion of policy output change.

[0121] The policy output change ratio can be an indicator used to quantify the degree of change in policy output between the second policy version and the first policy version, reflecting the magnitude of the difference in action selection preferences before and after the policy update. The policy output change ratio can be calculated using probability ratios, difference normalization results, similarity measures, or other functional forms. In this embodiment, the policy output change ratio can be used to determine whether the policy update is within the "near-end" range, and is an important basis for constructing the near-end policy optimization loss.

[0122] In this embodiment, the advantage information does not directly determine the magnitude of the update, but rather works in conjunction with the proportion of change in the policy output to constrain the policy update process.

[0123] Update magnitude constraint refers to a control mechanism that limits the degree of parameter or output changes between adjacent policy versions during policy model training. Update magnitude constraint can be implemented by combining the proportion of policy output changes with advantage information, ensuring that policy updates do not deviate too far from the original policy while optimizing in the direction of improving policy performance. In this embodiment, update magnitude constraint is used to improve the performance degradation or training instability of policy models under conditions of frequent online reinforcement learning and human-computer interaction.

[0124] The proximal policy optimization loss value refers to a numerical result calculated using the proximal policy optimization loss function based on the proportion of policy output change and advantage information. It is used to quantify whether the current update of the policy model satisfies the proximal constraints. During policy model training, the proximal policy optimization loss value serves as one of the optimization objectives, participating in the fusion loss calculation along with the advantage-weighted policy loss value. In this embodiment, the proximal policy optimization loss value is used to guide the policy model to continuously improve its execution performance while ensuring stability.

[0125] In terms of overall control logic, this embodiment distinguishes between a first policy version used for data acquisition and a second policy version used for training and updating, and constructs loss constraints based on the differences in policy outputs of the two policies under the same input state, thereby achieving fine-grained control over the policy update process. This proximal control logic effectively balances the speed of policy performance improvement with the stability of the training process within an online reinforcement learning framework, making it suitable for applications with high safety and reliability requirements, such as robot operations.

[0126] In this embodiment, the calculation of the proximal policy optimization loss is used to constrain the update magnitude between adjacent policy versions during policy model training, ensuring the stability of policy evolution. Specifically, for the policy model, the first policy version used to collect the second training data and the second policy version currently used for training and updating are first distinguished, and the policy output information of both under the same input state is obtained. Then, by comparing the output information of the first and second policies, the proportion of policy output change of the second policy version relative to the first policy version is determined to quantify the difference in action selection before and after the policy update. Based on this, advantage information is introduced as a reference for the update direction. The proportion of policy output change is combined with the advantage information to constrain the update magnitude of the second policy version relative to the first policy version, thereby constructing the proximal policy optimization loss value. This proximal policy optimization loss value can be used to limit the update step size of the policy model during subsequent training, reducing excessive policy shift in a single iteration.

[0127] This embodiment quantifies and constrains the adjacent update process of the policy model by using the ratio of policy output change between the first and second policy versions. This effectively suppresses drastic fluctuations in the policy during online reinforcement learning and human-computer interaction scenarios, improving the stability of the training process. By combining the ratio of policy output change with dominance information, the policy model can optimize in the direction of improving long-term returns during updates while reducing policy imbalance caused by over-reliance on locally dominant samples, thus achieving a balance between exploration and safety. By introducing proximal policy optimization loss, the policy model maintains smooth evolution during continuous online updates, reducing the risk of performance degradation and improving the success rate and robustness in robot operation tasks.

[0128] In some embodiments, the step of calculating the fusion loss based on the advantage-weighted policy loss value and the proximal policy optimization loss value to obtain the fusion loss value may include: performing a combined calculation on the advantage-weighted policy loss value and the proximal policy optimization loss value based on a preset fusion rule to obtain the fusion loss value. The fusion rule includes at least one of weighted summation, linear combination, and staged joint optimization, such that the proximal policy optimization loss value is used to limit the update magnitude corresponding to the advantage-weighted policy loss value.

[0129] The advantage-weighted policy loss value refers to the loss obtained by weighting the training error of the policy model based on advantage information. It reflects the difference in learning quality of the policy model on different state-action pairs. Different training samples have different contribution weights in the loss calculation, with samples having higher advantage information corresponding to larger loss weights, thus generating a stronger guiding effect in parameter updates. In this embodiment, the advantage-weighted policy loss value is used to drive the policy model to optimize towards high-value behaviors, and is an important optimization objective reflecting behavior quality guidance.

[0130] The proximal policy optimization loss value refers to the loss amount used to constrain the magnitude of output changes during adjacent policy model updates. It is constructed by comparing the differences in policy outputs of different policy versions under the same input state. The proximal policy optimization loss value does not directly pursue rapid improvement in policy performance, but rather focuses on limiting the range of policy updates to reduce excessive shifts in policy distribution within a single update. In this embodiment, the proximal policy optimization loss value serves as a stability constraint term, working in conjunction with the advantage-weighted policy loss value to ensure the stability and safety of the online reinforcement learning process.

[0131] Preset fusion rules refer to a set of rules used to determine the participation and weighting relationships of different policy losses in the fusion loss. These rules regulate the combination of the advantage-weighted policy loss value and the proximal policy optimization loss value. The fusion rules can take various forms, such as weighted summation rules, linear combination rules, phased joint optimization rules, or master-slave loss structures. In this embodiment, the introduction of preset fusion rules ensures a clear division of labor among different loss terms during the policy update process, thereby achieving controllable and stable joint optimization.

[0132] Weighted summation involves assigning corresponding weight coefficients to different loss terms and then summing them up to obtain a fused loss value. The advantages of weighted summation are its simplicity, computational stability, and the ability to flexibly control the influence of different loss terms during training by adjusting the weight coefficients. In this embodiment, weighted summation can be used to balance the relative strength of the influence between the advantage-weighted policy loss value and the proximal policy optimization loss value.

[0133] Linear combination combines multiple loss terms according to a linear relationship, and can be viewed as a generalized form of weighted summation. In specific implementations, linear combination can introduce constant terms, scaling coefficients, or regularization terms to further adjust the coupling relationship between different loss terms. In this embodiment, linear combination can be used to perform fine-grained control over policy update behavior at different training stages.

[0134] Phased joint optimization can achieve gradual optimization of the policy model by emphasizing different loss terms at different training stages. For example, in the early stages of training, the advantage-weighted policy loss value can be emphasized to quickly guide policy convergence, while in the later stages of training, the constraint effect of the proximal policy optimization loss value can be gradually strengthened to improve policy stability. In this embodiment, phased joint optimization helps to dynamically balance exploration efficiency and policy reliability in online reinforcement learning.

[0135] This embodiment uses the advantage-weighted policy loss as the performance-driving term and the near-end policy optimization loss as the update constraint term, and unifies the two into the fusion loss through a preset fusion rule, thereby achieving coordinated control over the direction and magnitude of policy model updates. This joint control logic can significantly reduce the risk of training instability or performance degradation during online reinforcement learning while ensuring continuous improvement in policy performance.

[0136] In this embodiment, by introducing the advantage-weighted policy loss value and the proximal policy optimization loss value into a unified fusion loss calculation framework, coordinated control of the policy model update direction and magnitude is achieved. Specifically, the advantage-weighted policy loss value and the proximal policy optimization loss value are first calculated based on advantage information. The advantage-weighted policy loss value characterizes the policy model's learning degree on high-advantage state-action pairs, while the proximal policy optimization loss value characterizes the change constraints between adjacent update versions of the policy model. Subsequently, according to a preset fusion rule, the two types of loss values ​​are combined to form a fusion loss value. The fusion rule can employ weighted summation, linear combination, or phased joint optimization to limit the policy update magnitude driven by the advantage-weighted policy loss value during the fusion process. Finally, the model parameters of the policy model are updated based on the fusion loss value, thereby simultaneously achieving performance guidance and stability constraints under a single optimization objective.

[0137] This embodiment combines the advantage-weighted policy loss value with the proximal policy optimization loss value based on a preset fusion rule. This allows the policy model to fully utilize advantage information to reinforce high-value behaviors during the update process, while also reducing excessive policy distribution shifts under the constraints of proximal policy optimization, thereby improving the stability of the online reinforcement learning process. By limiting the update magnitude of the advantage-weighted policy loss value corresponding to the proximal policy optimization loss value, it effectively reduces drastic policy oscillations caused by local high-advantage samples or human intervention, lowering the risk of training divergence or performance degradation. By constructing a unified fusion loss calculation mechanism, the policy convergence speed and task success rate are improved while ensuring the safety of policy updates. This embodiment enables stable and efficient policy learning in complex robot operation tasks and scenarios with frequent human-computer interaction.

[0138] See Figure 2 , Figure 2 This is a flowchart of an online reinforcement learning process provided in an embodiment of this application.

[0139] like Figure 2 As shown, the online reinforcement learning process can include the following three stages.

[0140] The first stage is the strategy model pre-training stage. A human expert controls the robot to perform a target operation task via a teleoperation device. The teleoperation device receives second interaction information from the human expert and constructs third training data for the target operation task based on this second interaction information. This third training data is stored in a demonstration dataset, and behavioral cloning loss is calculated based on the demonstration dataset to train a pre-set initial strategy model, resulting in a pre-trained strategy model. The strategy model can be implemented using an actor model as an example.

[0141] The second stage is the value assessment model warm-up stage. Based on the pre-trained policy model, the robot is controlled to perform the target operation task, acquiring the corresponding fourth training data, which is then stored in the playback dataset. During the robot's execution of the target operation task, human experts monitor the robot's behavior and, when necessary, provide teaching or intervention via teleoperation, storing the acquired fifth training data in both the playback dataset and the demonstration dataset. Based on the playback dataset and the demonstration dataset, a pre-set initial value assessment model is trained using temporal difference loss to obtain the warmed-up value assessment model. The value assessment model can be implemented using a critic model as an example.

[0142] The third stage is the online reinforcement learning stage. In this stage, the policy model continuously interacts with the environment under the current policy parameters, controlling the robot to execute the target operation task, generating first training data and storing it in the playback dataset. Simultaneously, human experts monitor the robot's execution in real time and, when the robot enters an error or inefficient state, teach or intervene remotely, storing the acquired second training data in the demonstration dataset. Based on the playback dataset and the demonstration dataset, the value assessment model is continuously trained using temporal difference loss to obtain value assessment information for the long-term rewards of each state action. Based on this value assessment information and combined with the demonstration dataset, the advantage information corresponding to each state action pair in the demonstration dataset is calculated using an advantage function. A fusion loss is calculated based on this advantage information, including advantage-weighted policy loss and proximal policy optimization loss. The model parameters (e.g., weight updates) are then updated based on the fusion loss calculation results (e.g., fusion loss value), thereby achieving continuous optimization of the robot's operation strategy during online interaction.

[0143] In some embodiments, human experts may also conduct overall or phased evaluations of the training data (such as the first training data and the fourth training data) generated by the autonomous execution of the policy model. The resulting evaluation information, as an example of interactive information, is associated with the corresponding training data and stored in the demonstration dataset or replay dataset for subsequent value assessment model training and advantage calculation.

[0144] See Figure 3 , Figure 3 This is a structural block diagram of an online reinforcement learning system provided in an embodiment of this application.

[0145] This application also provides an online reinforcement learning system, which includes a control module, an interaction module, a training module, and an advantage calculation module.

[0146] The control module is used to control the robot to perform target operation tasks through a strategy model in order to obtain the corresponding first training data.

[0147] The interaction module is used to receive first interaction information from a human expert during the period when the strategy model controls the robot to perform the target operation task, and / or, based on the execution result of the target operation task, to construct corresponding second training data; the first interaction information is used to characterize at least one of teaching, intervention and evaluation for the robot's execution process.

[0148] The training module is used to calculate the time difference loss based on the first training data and the second training data to train the value assessment model and obtain the value assessment information output by the value assessment model.

[0149] The advantage calculation module is used to calculate the advantage based on the second training data and its corresponding value assessment information to obtain advantage information.

[0150] The training module is further used to calculate the fusion loss based on the advantage information in order to train the policy model; the fusion loss includes advantage-weighted policy loss and proximal policy optimization loss.

[0151] In some embodiments, the interaction module can also be used to construct third training data for the target operation task based on second interaction information from a human expert, prior to the step of controlling the robot to perform the target operation task through the policy model to obtain the corresponding first training data; the second interaction information is used to characterize the teaching for the robot's execution process. The training module can also be used to calculate the behavior cloning loss based on the third training data to train a preset initial policy model to obtain the policy model.

[0152] In some embodiments, the control module can also be used to control the robot to perform the target operation task through the strategy model to obtain corresponding fourth training data before the step of controlling the robot to perform the target operation task through the strategy model to obtain corresponding first training data. The interaction module can also be used to receive third interaction information from a human expert during the execution of the target operation task by the strategy model, and / or, based on the execution result of the target operation task, to construct corresponding fifth training data; the third interaction information is used to characterize at least one of teaching, intervention, and evaluation of the robot's execution process. The training module can also be used to calculate time difference loss based on the fourth and fifth training data respectively to train a preset initial value assessment model to obtain the value assessment model.

[0153] In some embodiments, the advantage calculation module may perform advantage calculation in the following manner to obtain advantage information: based on the value assessment information, a preset advantage function is used to evaluate the relative merits of the state-action pairs in the corresponding second training data to obtain the corresponding advantage information.

[0154] In some embodiments, the training module may perform fusion loss calculation to train the policy model in the following manner: calculate the advantage-weighted policy loss based on the advantage information to obtain an advantage-weighted policy loss value; calculate the proximal policy optimization loss based on the advantage information to obtain a proximal policy optimization loss value; calculate the fusion loss based on the advantage-weighted policy loss value and the proximal policy optimization loss value to obtain a fusion loss value; and update at least one model parameter of the policy model based on the fusion loss value.

[0155] In some embodiments, the training module may perform advantage-weighted policy loss calculation in the following manner to obtain the advantage-weighted policy loss value: for at least a portion of the state-action pairs in the second training data, the corresponding state information is input into the policy model to obtain the policy output information output by the policy model; based on the advantage information, corresponding sample weights are assigned to the state-action pairs; based on the sample weights, the matching result of the policy output information and the corresponding action information is weighted and calculated, and the advantage-weighted policy loss value is determined based on the weighted calculation result.

[0156] In some embodiments, the training module may perform proximal policy optimization loss calculation in the following manner to obtain the proximal policy optimization loss value: for the policy model, based on the first policy output information of the first policy version used to collect the second training data and the second policy output information of the second policy version used for training, determine the policy output change ratio of the second policy version relative to the first policy version; based on the policy output change ratio and the advantage information, constrain the update magnitude of the second policy version relative to the first policy version to obtain the proximal policy optimization loss value.

[0157] In some embodiments, the training module may perform fusion loss calculation in the following manner to obtain the fusion loss value: based on a preset fusion rule, the advantage-weighted policy loss value and the proximal policy optimization loss value are combined and calculated to obtain the fusion loss value; wherein, the fusion rule includes at least one of weighted summation, linear combination, and phased joint optimization, so that the proximal policy optimization loss value is used to limit the update magnitude corresponding to the advantage-weighted policy loss value.

[0158] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0159] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0160] The computer program product may be a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the computer program product of the embodiments of this application is not limited thereto, and the computer program product may be any combination of one or more computer-readable media.

[0161] See Figure 4 , Figure 4 This is a structural block diagram of a computer device provided in an embodiment of this application.

[0162] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.

[0163] The embodiments of this application do not limit the computer device, which may be, for example, a local computer device, a cloud computer device, a distributed computer device, etc.

[0164] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected through internal connection paths.

[0165] The memory 110 is used to store computer programs, which in some implementations may include code for implementing the methods of the embodiments of this application.

[0166] The processor 120 executes the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information, and output operation results and other data. In some implementations, when the solutions of the embodiments of this application are implemented by software or firmware, the computer program used to implement the solutions of the embodiments of this application can be stored in the processor 120 and executed by the processor 120.

[0167] The memory 110 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any memory of these and other suitable types. As an example, the memory 110 includes random access memory (RAM), cache memory, and read-only memory (ROM). The memory 110 stores a computer program that can be executed by processor 120, causing processor 120 to implement the steps of any of the methods described above.

[0168] The processor 120 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 120 can be any conventional processor.

[0169] In implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware in the processor 120 or by instructions in software form. The method disclosed in the embodiments of this application can be directly implemented by the hardware processor, or by a combination of hardware and software modules in the processor 120. The software modules can be located in mature storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in the memory 110, and the processor 120 reads the information in the memory 110 and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are not provided here.

[0170] In some implementations, in addition to the hardware units described above, computer devices may also include software modules, such as operating systems, basic input / output systems (BIOS), and application software.

[0171] An operating system is used to manage the hardware and / or software resources of a computer device; it is the kernel and foundation of the computer. The operating system handles fundamental tasks such as managing and configuring memory, determining the priority of system resource allocation, controlling input and output devices, operating the network, and managing the file system. To facilitate user operation, most operating systems provide a user interface for interaction with the system.

[0172] The BIOS is used to perform hardware initialization during the power-on boot phase and to provide runtime services for the operating system and applications. In some implementations, the BIOS can also monitor and display processor temperature and execute temperature protection strategies.

[0173] Application software, also known as an application program, can be understood as software written for a specific user application purpose, and is one of the main categories of computer software. For example, application software can be a program used to achieve purposes such as power control and temperature management.

[0174] It is understood that the specific examples in this application are only intended to help those skilled in the art better understand the implementation of this application, and are not intended to limit the scope of protection of this application.

[0175] It is understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0176] It is understood that the various implementation methods described in this application can be implemented individually or in combination, and this application does not limit them.

[0177] Unless otherwise stated, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0178] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0179] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the embodiments described above can be referred to the corresponding processes in other embodiments, and will not be repeated here.

[0180] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0181] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the technical solution in this application, depending on actual needs.

[0182] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0183] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to related technologies, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0184] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An online reinforcement learning method, characterized in that, The method includes: The robot is controlled to perform target operation tasks through a strategy model in order to obtain the corresponding first training data. During the period when the strategy model controls the robot to perform the target operation task, and / or, based on the execution result of the target operation task, first interactive information from a human expert is received to construct corresponding second training data; the first interactive information is used to characterize at least one of teaching, intervention and evaluation for the robot's execution process; Based on the first training data and the second training data, time difference loss is calculated to train the value assessment model and obtain the value assessment information output by the value assessment model. Advantage calculation is performed based on the second training data and its corresponding value assessment information to obtain advantage information; The fusion loss is calculated based on the aforementioned advantage information to train the policy model; the fusion loss includes advantage-weighted policy loss and proximal policy optimization loss.

2. The online reinforcement learning method according to claim 1, characterized in that, Before the step of controlling the robot to perform the target operation task through the policy model to obtain the corresponding first training data, the method further includes: Based on second interaction information from human experts, third training data is constructed for the target operation task; the second interaction information is used to characterize the teaching for the robot's execution process. The behavior cloning loss is calculated based on the third training data to train the preset initial policy model and obtain the policy model.

3. The online reinforcement learning method according to claim 2, characterized in that, Before the step of controlling the robot to perform the target operation task through the policy model to obtain the corresponding first training data, the method further includes: The strategy model is used to control the robot to perform the target operation task in order to obtain the corresponding fourth training data. During the period when the strategy model controls the robot to perform the target operation task, and / or, based on the execution result of the target operation task, a third interactive information from a human expert is received to construct corresponding fifth training data; the third interactive information is used to characterize at least one of teaching, intervention and evaluation for the robot's execution process; Based on the fourth training data and the fifth training data, time difference loss is calculated to train a preset initial value assessment model, thereby obtaining the value assessment model.

4. The online reinforcement learning method according to claim 3, characterized in that, The first training data, the second training data, the third training data, the fourth training data, and the fifth training data include state information and action information. The first interaction information, the second interaction information, and the third interaction information are provided by human experts through remote operation devices.

5. The online reinforcement learning method according to claim 1, characterized in that, The step of calculating the advantage based on the second training data and its corresponding value assessment information to obtain advantage information includes: Based on the value assessment information, a preset advantage function is used to evaluate the relative merits of the state-action pairs in the corresponding second training data in order to obtain the corresponding advantage information.

6. The online reinforcement learning method according to claim 1, characterized in that, The step of calculating the fusion loss based on the advantage information to train the policy model includes: Based on the aforementioned advantage information, the advantage-weighted strategy loss is calculated to obtain the advantage-weighted strategy loss value. Based on the aforementioned advantage information, the near-end strategy optimization loss is calculated to obtain the near-end strategy optimization loss value; The fusion loss is calculated based on the advantage-weighted strategy loss value and the near-end strategy optimization loss value to obtain the fusion loss value. Based on the fusion loss value, at least one model parameter of the policy model is updated.

7. The online reinforcement learning method according to claim 6, characterized in that, The step of calculating the advantage-weighted strategy loss based on the advantage information to obtain the advantage-weighted strategy loss value includes: For at least a portion of the state-action pairs in the second training data, the corresponding state information is input into the policy model to obtain the policy output information output by the policy model; Based on the aforementioned advantage information, corresponding sample weights are assigned to the state-action pairs; Based on the sample weights, the matching results of the strategy output information and the corresponding action information are weighted and calculated, and the advantage weighted strategy loss value is determined based on the weighted calculation results.

8. The online reinforcement learning method according to claim 6, characterized in that, The step of calculating the near-end policy optimization loss based on the aforementioned advantage information to obtain the near-end policy optimization loss value includes: For the policy model, based on the first policy output information of the first policy version used to collect the second training data and the second policy output information of the second policy version used for training, the policy output change ratio of the second policy version relative to the first policy version is determined. Based on the change ratio of the strategy output and the advantage information, the update range of the second strategy version relative to the first strategy version is constrained to obtain the near-end strategy optimization loss value.

9. The online reinforcement learning method according to claim 6, characterized in that, The step of calculating the fusion loss based on the advantage-weighted policy loss value and the near-end policy optimization loss value to obtain the fusion loss value includes: Based on the preset fusion rules, the advantage-weighted strategy loss value and the near-end strategy optimization loss value are combined and calculated to obtain the fusion loss value; The fusion rule includes at least one of weighted summation, linear combination, and phased joint optimization, so that the proximal policy optimization loss value is used to limit the update magnitude corresponding to the advantage weighted policy loss value.

10. An online reinforcement learning system, characterized in that, The system includes: The control module is used to control the robot to perform target operation tasks through the strategy model in order to obtain the corresponding first training data; An interaction module is configured to receive first interaction information from a human expert during the period when the strategy model controls the robot to perform the target operation task, and / or, based on the execution result of the target operation task, to construct corresponding second training data; the first interaction information is used to characterize at least one of teaching, intervention, and evaluation for the robot's execution process; The training module is used to calculate the time difference loss based on the first training data and the second training data to train the value assessment model and obtain the value assessment information output by the value assessment model. The advantage calculation module is used to perform advantage calculation based on the second training data and its corresponding value assessment information to obtain advantage information; The training module is further used to calculate the fusion loss based on the advantage information in order to train the policy model; the fusion loss includes advantage-weighted policy loss and proximal policy optimization loss.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

Images