Offline-to-online generalizable reinforcement learning method and device based on continuous policy reinvigoration

Abstract

The application discloses an offline-to-online generalizable reinforcement learning method and device based on continuous policy rejuvenation, which first restores the learning ability of the policy network through periodic policy rejuvenation, breaks the influence of the initial bias on the robot in the online learning process, and thus improves the adaptability to the online distribution. Then, through adaptive policy constraints, the target policy of the policy constraints is updated according to the competitive behavior expression of the policy in the current distribution, and this mechanism effectively improves the stability and generalization of the robot policy constraints based on the real-time policy access distribution and behavior policy competitive expression of the robot. Finally, a policy mixing method based on a historical policy pool is adopted, which further improves the ability of the robot to balance memory and exploration, and ensures that the robot can avoid the policy collapse caused by catastrophic forgetting in the online policy updating process.

Description

Technical Field

[0001] This invention relates to the field of intelligent agent control technology, and in particular to a control method and system for multi-joint robots. Specifically, this invention provides a reinforcement learning control strategy that combines offline training and online adjustment, aiming to solve the dynamic environment adaptability and policy transfer problems encountered by multi-joint robots in practical applications. Background Technology

[0002] When performing complex tasks, multi-jointed robots require precise control strategies to ensure the accuracy and safety of their movements. Traditional control methods, such as PID control and model predictive control (MPC), rely on accurate models and environmental parameters. However, these methods often struggle to adapt to dynamically changing environments. Therefore, reinforcement learning has emerged as a promising solution, allowing multi-jointed robots to learn optimal strategies through interaction with their environment. Despite its theoretical advantages, reinforcement learning often suffers from low sample efficiency and poor adaptability in practical applications. Therefore, offline-to-online reinforcement learning systems play a crucial role. These systems aim to improve the adaptability and decision-making efficiency of agents in real-world environments by combining offline training with online tuning.

[0003] One of the core challenges of Offline-to-Online (O2ORL) reinforcement learning is balancing the utilization of offline-trained models with online tuning of the initial models. Existing O2ORL methods primarily address specific O2ORL problems by heuristically controlling policy conservatism during online reinforcement learning updates. When the offline dataset is of high quality, these methods successfully balance policy conservatism and exploration through progressive policy constraint methods such as AWAC, TD3BC-FT, and PROTO. However, these methods still suffer from insufficient generalization ability to low-quality offline datasets with low data coverage, converging to suboptimal policies or experiencing significant performance degradation during online learning. Furthermore, previous O2ORL methods, such as Off2On, estimated the online policy access distribution and recalled suitable offline samples from the offline dataset to expand the online data replay pool, aiming to enable the agent to adapt more smoothly to the online data distribution and reduce performance loss caused by data distribution drift.

[0004] Another core challenge in transitioning from offline to online reinforcement learning is addressing primacy bias. Primacy bias manifests in deep reinforcement learning systems when the policy model is overtrained on a specific data distribution during the early stages of training. This overfitting leads to a decrease in the policy's adaptability to new data distributions as the policy accesses them. Methods such as JSRL and OOO address this by jointly learning multiple policies to overcome the decline in policy adaptability caused by overfitting in the offline component. PEX introduces policy fusion to dynamically combine offline and online learning policies, automatically achieving the mixing of dual policies.

[0005] The aforementioned methods typically focus solely on achieving stability or efficiency in policy learning during the transition from offline to online reinforcement learning. While they have made some progress in offline-to-online reinforcement learning systems, they still possess certain limitations. For instance, methods like Cal-QL and SO2 have achieved some progress in transferring policies from offline to online when offline data quality is high, but they primarily concentrate on leveraging the advantages of offline policies without comprehensively considering the negative impacts of offline pre-trained policy models. In offline-to-online reinforcement learning systems, balancing policy conservatism and exploratory nature is a core issue. This requires not only utilizing the advantages of offline-trained policies but also minimizing the inefficient policy updates resulting from excessive offline training in online learning. Furthermore, existing methods do not consider the sudden forgetting of decision knowledge during model updates, which may lead to overly aggressive policy updates and catastrophic forgetting in practical applications.

[0006] In summary, existing technologies for offline-to-online reinforcement learning still suffer from problems such as insufficient policy generalization ability, poor adaptability to initial biases, and loss of decision knowledge during policy updates. Applying this offline-to-online reinforcement learning approach to the control of multi-joint robots leads to poor learning and adaptation capabilities of the control policy to the online robot control scenario when there are differences between the online training and testing environment and the offline training environment of the multi-joint robot in terms of sensor inputs and dynamic coefficients in the environment. This results in weak scalability and catastrophic forgetting of offline-learned skills during policy updates. These problems limit the performance and reliability of reinforcement learning in practical applications. Summary of the Invention

[0007] Purpose of the Invention: This invention aims to address the data distribution generalization problem encountered in the practical application of multi-joint robot control problems, specifically the transition from offline to online reinforcement learning. In existing technologies, control algorithms often fail to learn and adapt effectively when facing new online environments due to over-reliance on low-quality offline datasets, resulting in low online learning efficiency. Furthermore, existing methods are overly conservative in policy initialization and updating, limiting the learning and adaptability of the intelligent robot.

[0008] To overcome these limitations, this invention proposes an innovative continuous policy regeneration framework. First, this framework stimulates the learning ability of the intelligent robot through a continuous policy regeneration mechanism, ensuring that its online learning phase can make decisions based on a better policy network, breaking the limitations of local optima. Second, through adaptive policy constraints, the robot can adjust its policies more flexibly, improving the stability and efficiency of the learning process. Furthermore, this invention employs policy pooling and policy hybridization techniques to save and optimize historical behavioral policies, achieving progressive optimization of behavioral policies through competitive policy expressions, thereby enhancing the agent's generalization ability.

[0009] The core advantage of this invention lies in its significantly improved adaptability and learning efficiency of jointed robots when faced with changes in offline data distribution and online environments, especially when offline data quality is poor. This not only addresses the shortcomings of existing technologies but also provides a new solution for the widespread application of jointed robots in fields such as lifelong learning agents and high-precision industrial control with small sample sizes. It is expected to achieve more efficient learning and superior performance in practical offline-to-online reinforcement learning tasks.

[0010] Technical Solution: This paper presents an offline-to-online generalizable reinforcement learning method based on continuous policy re-enhancement for multi-joint robot control, aiming to improve the robot's generalization ability when transitioning from offline datasets to online environments. The solution employs an online "hetero-policy" reinforcement learning framework and introduces innovative mechanisms to address the problems of excessive reliance on offline datasets and insufficient generalization ability in existing technologies. The core idea is to restore the learning capability of the policy network through fixed-period continuous policy re-enhancement, maintain a historical policy pool for efficient policy mixing, and implement an adaptive policy regularization term based on competitive behavior representation. This solution not only enhances the robot's learning efficiency in the online learning phase but also improves the stability of policy learning while avoiding access to offline datasets. Furthermore, it enhances the robot's adaptability and efficiency to different data distributions, enabling it to better cope with various challenges in multi-joint robot control.

[0011] This technical solution first restores the learning ability of the policy network through periodic policy revitalization, breaking the influence of initial biases during online learning and thus improving the robot's adaptability to online distributions. Next, through adaptive policy constraints, the target policy of the policy constraints is updated based on the competitive behavioral expressions of the policies in the current distribution. This mechanism, based on the robot's real-time policy access distribution and competitive behavioral policy expressions, effectively improves the stability and generalization of the robot's policy constraints. Finally, a policy fusion method based on a historical policy pool is employed to further enhance the robot's ability to balance memory and exploration, ensuring that catastrophic forgetting that could lead to policy collapse during online policy updates is avoided.

[0012] The method includes:

[0013] Periodic policy re-enhancing: Based on the multi-joint robot's execution in online scenarios during task rounds, a policy re-enhancing is performed when this condition is met during online learning. Policy re-enhancing resets the parameters of the last N layers of the multi-joint robot's control policy network. The parameter N is determined by the network size and task characteristics; in our test environment, it is typically set to 3. For the initialized control policy network, a new control policy is initialized and adapted to online learning based on the historical logs of robot actions in the historical experience replay pool.

[0014] Adaptive Policy Constraints: To better constrain policies in online learning and improve policy stability, adaptive policy constraints are proposed. During actual training, historical behavioral policy data stored in the experience replay pool is analyzed. By modeling these action sequences, state transitions, and reward acquisition patterns, a constraint objective for the behavioral policy is constructed. By introducing a constraint loss term that minimizes both the current learning policy and the behavioral policy, out-of-distribution errors that cause significant deviations from the learning policy are avoided.

[0015] Policy Fusion Based on Policy Pool: This invention employs a policy fusion method based on a policy pool. This method accumulates the old policy functions reset during each policy re-enactment process into a policy pool, forming a set of historical control policies containing historical control functions defined by a neural network. During robot-environment interaction, a set of candidate actions is obtained by passing task input information from the robot's sensors to each function in the policy pool. By evaluating state-action pairs using a value-based Q-function, a Boltzmann distribution of the execution action value is proposed, enabling policy fusion and sampling the control policy best suited for the current environment.

[0016] To verify the effectiveness of the method of this invention, we conducted extensive experiments in multiple reinforcement learning scenarios, ranging from offline to online. These experiments covered various operational tasks of multi-joint robots, such as motion control and navigation. In these experiments, we evaluated the learning efficiency of the control policy, the speed of adaptation to online environments, and the generalization ability after adopting the method of this invention. Experimental results show that compared with traditional methods, the present invention achieves significant performance improvements from offline to online, especially when dealing with low-quality offline datasets.

[0017] This invention, through innovative technical solutions, significantly improves the control stability and operational efficiency of multi-joint robots in complex and variable environments, providing strong technical support for the application of reinforcement learning in the field of robot control, from offline to online.

[0018] The multi-joint robot control device based on continuous strategy re-vibration consists of the following core components:

[0019] Periodic Policy Refresh Component: This component is responsible for periodically refreshing the control policy network of the multi-joint robot to keep its learning capabilities active. Through this periodic policy refresh, the robot can quickly adapt to new data distributions appearing in the online environment, thereby effectively responding to real-time changing task requirements. The goal of this component is to overcome the initial biases that may form in the robot during the offline training phase, thereby improving its adaptability and generalization ability in practical applications.

[0020] Periodic policy re-enhancing component: Based on the multi-joint robot's execution in online scenarios during task rounds, using relative time as the trigger condition, policy re-enhancing is performed when this condition is met during online learning. Policy re-enhancing resets the parameters of the last N layers of the multi-joint robot's control policy network. The parameter N is determined by the network size and task characteristics. For the initialized control policy network, a new control policy is initialized and adapted to online learning based on the historical logs of robot action execution in the historical experience replay pool.

[0021] Adaptive Policy Adjustment Component: This component is responsible for dynamically adjusting the policy constraints in the control strategy of the multi-joint robot to balance conservatism and exploratory nature during the learning process. Utilizing real-time behavior analysis and historical interaction data, this component can adaptively adjust the policy objectives, ensuring that the robot can make appropriate action adjustments when faced with new data distributions, thereby improving its operational efficiency and accuracy in dynamic environments.

[0022] Adaptive policy adjustment component: During the actual training of the neural network representing the policy control function, historical behavioral policy data stored in the experience replay pool is analyzed. By modeling the action sequence, state transition, and reward acquisition pattern of the policy data, the constraint objective of the behavioral policy is constructed. By introducing a constraint loss term that minimizes the current learning policy and the behavioral policy, the out-of-distribution error that causes serious deviations in the learning policy is avoided.

[0023] Policy Pool and Hybrid Policy Execution Component: This component maintains a policy pool containing multiple historical control policies, analyzing and learning the behavioral patterns of multi-joint robots under different data distributions. During online control, it competitively expresses and evaluates these historical policies to select the optimal action execution plan. This hybrid policy approach helps the robot avoid catastrophic forgetting during the learning process and enables policy adjustments based on online control requirements, thereby improving the robot's adaptability and generalization performance to new tasks.

[0024] Policy Pool and Hybrid Policy Execution Component: During robot-environment interaction, a set of candidate actions is obtained by passing task input information from the robot's sensors to each function in the policy pool. A Boltzmann distribution of the execution action value is proposed by evaluating the "state-action" relationship using a value-based Q-function, thus achieving policy hybridization and sampling the control policy best suited for the current environment.

[0025] The implementation process and methods for each component of the device are the same.

[0026] Computer Hardware Architecture: The physical implementation of this device is based on a high-performance computer hardware architecture, including a powerful processor, large-capacity memory, and multifunctional input / output devices. The processor is responsible for executing the algorithms and logic of the aforementioned components, while the memory stores control program code, offline datasets, and data processed in real time. Input / output devices may include sensors for monitoring robot status and environmental feedback, as well as communication interfaces for sending control commands. All these hardware components work closely together via a high-speed data transmission bus or network to achieve precise control and rapid learning of the multi-jointed robot.

[0027] By combining these innovative components and hardware architecture, this device can significantly improve the control stability and execution efficiency of multi-joint robots in complex tasks, providing a practical solution to the data distribution generalization problem in robot control. Attached Figure Description

[0028] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram of the continuous strategy revitalization mechanism in an embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of the device principle according to an embodiment of the present invention. Detailed Implementation

[0031] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0032] As mentioned earlier, many real-world scenarios involving multi-joint robot control involve generalizable reinforcement learning techniques, ranging from offline to online, particularly in fields such as intelligent domestic robots, industrial automation, and intelligent medical assistance. In these applications, multi-joint robots need to make precise decisions in dynamic real-world environments and be able to fine-tune based on real-time feedback to ensure safe and efficient service delivery.

[0033] In the field of intelligent domestic robots, these robots often face the challenge of operating in real, dynamic, open environments beyond the distribution of offline datasets. For example, they may need to autonomously navigate, clean, and move items within a home environment. These tasks require the robots to handle unexpected tasks and unseen targets, situations that may not have occurred in the offline dataset. Therefore, the robots must be able to learn and adapt quickly in real-world environments to ensure operational safety and efficiency. Similarly, multi-joint robots in industrial automation need to handle small-sample decision-making problems and must be able to generalize narrowly distributed offline reinforcement learning data to actual production to improve efficiency and reduce errors. These robots perform precise assembly, handling, and welding tasks on production lines, requiring them to quickly adapt to changes on the production line, such as new part designs or adjustments to processes. In the field of intelligent medical assistance, multi-joint robots are used to assist in high-precision, complex, and dynamically changing stages of surgical procedures. These surgical robots need to exhibit stable performance with the help of empirical data and be able to respond quickly to various situations that may arise during surgery to ensure the safety and success of the procedure.

[0034] Existing offline-to-online reinforcement learning methods have significant limitations when facing these real-world application scenarios. Many methods are based on assumptions of fixed policy constraints and static environments, which are unrealistic in the ever-changing real world. Furthermore, existing offline datasets are often of low quality and have limited coverage, making it difficult for multi-joint robots to make accurate decisions when encountering new situations. For example, domestic robots in real-world home environments may encounter layouts and unexpected situations not seen in offline data; in such cases, policies relying on offline datasets may not provide effective guidance. Moreover, existing methods often assume access to offline datasets when handling offline-to-online reinforcement learning, which poses risks to privacy and data security. For instance, in medical assistive robot systems, due to privacy requirements, robots deployed to medical institutions after offline training may no longer have extensive access to original medical data. Furthermore, existing methods still face challenges in balancing stable fine-tuning of multi-joint robots with online learning efficiency. In industrial robot applications, pursuing stable performance may lead to online policies over-reliance on past experience, while excessive pursuit of learning efficiency may result in policy instability and catastrophic forgetting.

[0035] While existing offline-to-online reinforcement learning methods have made some progress, their generalization ability and adaptability in dynamic real-world environments still need further improvement. To better address real-world challenges, there is an urgent need to develop more flexible offline-to-online reinforcement learning strategies that can quickly adapt to changes in environment and task requirements. These strategies should be able to rapidly adapt to new operating conditions and task requirements through online learning without relying on large amounts of offline data, thereby improving the performance and reliability of articulated robots in practical applications.

[0036] To address the aforementioned issues, this invention proposes a generalizable reinforcement learning method for offline-to-online systems based on continuous policy re-enhancement, specifically targeting the control problems of multi-joint robots in dynamic online environments. This method aims to improve the stability and generalization ability of the robot control system. It activates the learning capability of the policy network through policy re-enhancement, constructs a historical policy pool to store past knowledge, and introduces a policy competition mechanism to achieve adaptive policy constraints. This enables the application of past experience and autonomous adjustment when generalizing to new data distributions. This method is particularly suitable for environments where the distribution of behavioral policies accessed offline differs significantly from the distribution of online samples, such as smart home robots that need to learn policies from online samples that differ from past service experiences. This method not only emphasizes effective solutions to initial biases but also improves the quality of decisions made in complex environments using historical experience. Furthermore, this invention eliminates the need for offline data access by separating offline data and offline pre-trained policies, thereby improving the overall privacy and security of the offline-to-online reinforcement learning system.

[0037] Specifically, this method can be applied to many real-world environments, such as industrial automation, smart home management, and emergency rescue operations. In industrial automation, this method can be used to train robots to adapt to subtle parameter changes in different production lines, pursuing production line stability and improving production efficiency and quality. In smart home management, this method helps optimize the adaptation of home appliances, adjusting their decisions based on the actual living environment to meet the different needs of residents. In emergency rescue operations, this method can train intelligent agents to adapt to short-term real-world environments, fully utilizing background knowledge to more quickly engage in disaster relief responses. For example, in disaster relief applications, this method can train robots or intelligent systems to adapt to the changing environment based on the actual disaster relief site environment and trained background knowledge, updating with a small number of samples. Robots can learn and understand the execution requirements and actual execution effects of decisions by analyzing specific data distributions in different disaster scenarios, thereby providing more efficient and stable services. For example, in earthquake search and rescue scenarios, search and rescue robots can use this method to learn how to better and more stably and efficiently implement terminal strategies derived from pre-trained models. The following steps use robotic disaster relief response as an example, but it should be noted that this method is not limited to the task scenario of robotic disaster relief response, and can also be used for any other generalizable reinforcement learning problem from offline to online under conditions without offline data access.

[0038] like Figure 1 As shown in the figure, this is a flowchart of the offline-to-online generalizable reinforcement learning method described in this application embodiment. As illustrated, the method includes steps S101 to S105, and its core consists of three main parts: First, continuous policy revitalization, which aims to periodically reactivate the learning ability of the current policy network, ensuring that the agent can quickly adapt to the new data distribution during online training. Second, adaptive policy constraints, which constrain the policy to remain within the actual data distribution to prevent a significant mismatch between the target policy and the current data distribution. Finally, policy mixing, which provides a mixing method for historical policies, utilizing past knowledge to better assist policy selection on the current sample distribution. The entire process design ranges from continuous policy revitalization to restore the learning ability of the policy network, to using an experience replay pool to constrain online policy optimization, and then to a policy mixing method for the historical policy pool. During deployment, appropriate historical policies can be selected to achieve adaptability to the corresponding data distribution.

[0039] S101 pre-builds an offline policy, using an existing offline sequential decision dataset and an "actor-critic" based offline reinforcement learning algorithm as a combination to train an offline reinforcement learning policy. Specifically, it models the multi-joint robot control problem as a Markov sequential decision model. Our constructed offline dataset includes "state-action-reward" (s, a, r). In multi-joint robot control, the state space can be composed of all possible robot positions and joint angles. Each state represents the robot's configuration at a given moment. Due to the varying number of joints and the different ranges of motion of each joint, the state space dimensions of a specific robot often differ. The state transition probability describes the probability that the robot will transition to a new state given the current state and the action taken. In multi-joint robot control, these probabilities are typically modeled based on physical dynamics and environmental interactions. Because the robot's motion is subject to physical constraints and possible external perturbations, state transitions may not be completely deterministic, thus requiring the use of probabilistic models. The action space contains all possible control inputs, such as velocity or torque commands for each joint. In multi-joint robot control, the action space is usually continuous because each joint of the robot can have an infinite number of possible velocity or torque values. The reward function assigns a value to each state or state-action pair, representing the merit of taking a certain action in that state. In multi-joint robot control, rewards are usually related to factors such as task completion, energy consumption, and time efficiency. By collecting behavioral policies or expert policies and executing the collected state-action-reward sequences in the environment, we obtain an offline reinforcement learning dataset. In the offline reinforcement learning process, we alternately optimize the value network and policy network to pre-train the offline policy, so as to make full use of the sample information in the offline dataset to generate a better initial policy for subsequent online learning. The Q-optimization equation of the value network in offline reinforcement learning is as follows:

[0040] .

[0041] in, Represents the mathematical expectation. The network parameters representing the value network Q are... These are samples (state-action-reward-next state) from the offline dataset D. It is the regression target of TD learning. It is the value network's estimation of the value of the current state-action sequence. (Policy network) The optimization equation is as follows:

[0042] .

[0043] in, Represents the mathematical expectation. Representative Policy Network The parameters, It comes from the states and actions of samples in the offline dataset D. These are hyperparameters used to balance the weights of policy boosting and action cloning. It is the policy network's prediction of actions in the current state.

[0044] S102 Continuous Policy Revitalization Mechanism: Reactivating the Policy Network's Learning Capability. In this step, we construct a policy revitalization module to effectively activate the policy network's learning capability. The continuous policy revitalization module includes two key steps: resetting the policy and initializing a new optimized policy. In the policy reset step, the original input policy function parameters are stored, and the parameters of the last N layers of the network corresponding to the policy function are randomly discarded. This allows the policy to avoid harmful overfitting to the sample distribution and behavior policy corresponding to the offline behavior policy set. The value of N is a hyperparameter, influenced by the network structure and task requirements; in our experiments, it is typically set to 3 based on empirical experience. To ensure that the policy function obtains relatively stable initial performance, we initialize the policy function using an online experience replay pool, optimizing the following formula:

[0045] .

[0046] in, Represents the mathematical expectation. It comes from the states and actions of samples in the online experience replay pool B, and the value function parameters are kept fixed in the policy re-enactment module.

[0047] S103 utilizes an adaptive policy constraint module to achieve stable online policy updates. This step involves online policy learning, leveraging the learning capacity recovery brought by the policy re-emergence mechanism to achieve efficient policy learning. During online policy learning, in addition to requiring the policy to maximize reward, adaptive policy constraints are also applied to minimize the difference between the policy's access data distribution and the access data distribution of the behavioral policy in the online experience replay pool, under the distance metric of the divergence between distributions, thus achieving stable policy improvement. During online policy learning, we alternately optimize the value function and the policy function. The optimization objective for the value function is as follows:

[0048] .

[0049] in, Represents the mathematical expectation. The network parameters representing the value network Q are... These samples are from the online experience replay pool; their meanings remain consistent with the offline optimization objective. The optimization objective for the policy function is as follows:

[0050] .

[0051] The meanings of each item are consistent with the policy initialization objectives in the policy revitalization module.

[0052] S104 constructs a historical policy set and performs policy blending on these historical policies. Based on the continuous policy regeneration mechanism, we restore the learning capability of online learning policies by resetting historical policies. During the actual policy execution phase, to avoid catastrophic forgetting during the resetting process and the loss of decision knowledge from historical data distribution, a persistent policy pool is used to store experiential knowledge. For the collected historical policies, the action that best fits the current data distribution is selected for execution through policy blending. The probability of each policy's proposed action being selected is calculated as follows:

[0053] .

[0054] in Is the first choice The probability of a strategy proposing an action, where k is the total number of historical strategies. It is the value function that predicts the value of the action proposed by the j-th strategy. It is the temperature coefficient.

[0055] S105 combines continuous policy re-enhancing and adaptive policy regularization constraints to deploy the policy in practical applications. In this stage, based on the policy set mixing from the previous step of the multi-joint robot control strategy, after the state perception obtained from the sensors is transmitted to the policy set, all control strategies predict possible robot joint angles and torque output actions, forming a set of candidate actions. By learning the Boltzmann distribution of these action predictions based on the value function, we can select actions that adapt to the current distribution to execute in the actual environment. After the robot executes an action, the environmental samples are added to the experience replay pool, further changing the distribution of behavioral policy samples in the experience replay pool and improving the target policy in the adaptive policy constraint module. The optimized policy is continuously added to the historical policy set through policy re-enhancing, forming a mutually reinforcing cycle as described in steps S102-S105 above.

[0056] By executing this series of steps, this method enables rapid and effective adaptation to new data sample distributions, optimizing overall learning efficiency and policy performance stability in environments with constantly changing sample distributions. The execution of this process involves complex reinforcement learning policy interactions and learning processes, requiring meticulous policy design and adaptive considerations to ensure optimal learning performance in dynamic environments. The above step description provides a detailed implementation framework, laying a solid foundation for a deep understanding and application of this generalizable reinforcement learning method, applicable from offline to online.

[0057] See Figure 2 The diagram illustrates the continuous strategy re-execution process in this embodiment. We consider a robot-to-online interaction scenario in a robotic disaster relief response system. The system includes a robot trained on an offline log dataset for search and rescue missions. This robot needs to be adapted to interact with the environment in a specific scenario to perform search and rescue operations. The following is a description of the entire application process of this invention in a robotic disaster relief response system.

[0058] Before an actual disaster occurs, the robotic disaster response system pre-collects simulated datasets for disaster relief missions, such as simulated search and rescue trajectories in laboratory environments or computer-generated virtual scenarios. This helps the robot cover various terrains and obstacles it might encounter. The system analyzes these simulated datasets, along with a small amount of real search and rescue mission data, to establish a basic offline dataset, laying the foundation for subsequent training and optimization. During the offline reinforcement learning phase, the robot optimizes its search and rescue strategy on the offline dataset, improving the quality of the strategy without straying too far from the instance strategies in the offline strategy samples.

[0059] With offline training completed, the system enters the "continuous policy re-enhancement" phase. In this phase, the robot is deployed in a real-world environment for limited sample collection. By resetting the weights of some model samples overfitted to the offline dataset, the learning ability of the policy model is restored. This process is repeated periodically to confirm the robot's optimal learning capability. For example, after deploying the robot in actual outdoor conditions for limited exploration and searching, data analysis reveals that the real environment differs from the simulated laboratory conditions due to friction caused by precipitation. In this case, the knowledge learned by the robot's policy network in the laboratory environment may hinder execution in the current working environment. Therefore, the robot's policy is partially rolled back, eliminating some of its memory of the offline environment distribution. After each period of data collection and online policy optimization, the robot's policy parameters are reset again to re-enhance its ability to learn and adapt to new sample distributions. The system readjusts the robot to its optimal learning state based on the real online environment to cope with online adjustments.

[0060] The next stage is the strategy update phase. In this phase, the robot optimizes its current strategy by adaptively adjusting policy constraints, ensuring it doesn't deviate too far from the validated behavioral strategy, and ultimately selects a safe direction for improvement. Here, "behavioral strategy" refers to the robot's actual behavioral patterns in the real environment. The system dynamically selects the most suitable policy constraint target for the current strategy reference to avoid overly aggressive updates that could lead to severe performance degradation or even safety hazards. In addition, this part includes a historical strategy set stored in memory, using strategy mixing to prevent catastrophic forgetting during strategy re-evaluation. When actually executing the strategy, the system queries all retained robot strategies and selects the best action to execute through evaluation. Through this entire process, the robot disaster relief response system can achieve stable and efficient strategy adjustment and safety verification during deployment and learning from offline to online, significantly improving the availability and safety of search and rescue robots.

[0061] See Figure 3 The figure shows a device for generalizable reinforcement learning execution from offline to online, provided in an embodiment of this application. As shown, the device comprises three parts:

[0062] Continuous Policy Re-emphasis Module: This section focuses on continuously re-emphasizing the policy. The resulting online policy possesses full learning capabilities and stable initial performance, enabling stable improvement of the online policy in the online environment. The input to the policy re-emphasis module is the control function neural network of the current multi-joint robot, including a basic timing trigger function that automatically fires after a certain number of rounds. The parameters of the input function are transmitted via bus or communication to the high-speed storage unit in the policy mixing module for storage. After initialization by accessing the sample pool in the policy constraint module, the newly initialized control policy returns to the robot's normal task execution flow. This module is a crucial foundation for eliminating initial biases, providing a solid learning foundation and ensuring that the policy network can understand and adapt to the latest online sample distribution in subsequent steps.

[0063] S302 Policy Constraint Module: This module is responsible for collecting and organizing the sample pool of online policy access, balancing policy conservatism and learning efficiency through adaptive policy constraints. In the task scenario, the multi-joint robot acquires environmental information through sensors, calls the decision pool to make joint decisions, and outputs the control actions of each joint, receiving feedback and state updates from the environment. This information is compressed into samples and stored in the sample pool of the policy constraint module. The policy constraint module obtains sample distribution information by updating the motion trajectories accessed by the robot's most recent behavioral policy in real time, and uses this information to constrain the learning of the control policy. In this way, the online policy can effectively improve without deviating from the previously learned good initial policy. This step is crucial to ensuring that the policy can accurately grasp and adapt to the online sample distribution.

[0064] S303 Policy Hybridization Module: This module maintains a set of historical policies. The multi-joint robot system uses the original policies fixed from continuous policy re-evaluation to propose joint actions based on the angles and torques of each joint during interaction with the environment. These joint control policies, defined by control functions, incorporate knowledge from historical data distributions. The policy mixer samples the Boltzmann distribution obtained from the value of the proposed joint actions, effectively blending historical policies to address various data distributions. The policy hybridization module supports control policies exceeding the network's capacity limitations, storing historical knowledge information in the historical policy set, and safely eliminating the impact of historical sample distributions on the efficiency of current learning.

[0065] It is evident that those skilled in the art will understand that the above-described embodiments of the present invention are all based on deep reinforcement learning and policies, focusing on the steps of a generalizable method from offline to online, or based on deep reinforcement learning and policies, focusing on the modules of a generalizable device for multi-joint robots from offline to online. These modules can be implemented using general-purpose computing devices, either centralized on a single computing device or deployed in a distributed framework on a network of multiple computer devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps can be executed in a different order than shown or described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

Claims

1. A generalizable reinforcement learning method for multi-joint robot control based on continuous policy re-enactment, characterized in that, include: Periodic strategy re-energization: Based on the execution of multi-joint robots in online scenarios during task rounds, relative time is used as the trigger condition. When this condition is reached during online learning, strategy re-energization is performed. Strategy re-energization resets the parameters of the last N layers of the multi-joint robot's control strategy network. The parameter N is determined by the network size and task characteristics. For the initialized control strategy network, a new control strategy is initialized and adapted to online learning based on the historical logs of robot action execution in the historical experience replay pool. Adaptive policy constraints: In the actual training process of the neural network representing the policy control function, historical behavioral policy data stored in the experience replay pool is analyzed. By modeling these action sequences, state transitions, and reward acquisition patterns, a constraint objective for the behavioral policy is constructed. By introducing a constraint loss term that minimizes the current learning policy and the behavioral policy, out-of-distribution errors that cause serious deviations in the learning policy are avoided. Policy mixing based on a policy pool: During each policy re-energization process, the old policy functions that are reset form a set containing historical control policies, which is stored in the policy pool. The set contains historical control functions defined by the neural network. When the robot interacts with the environment, a set of candidate actions is obtained by passing the task input information obtained by the robot's sensors to each function in the policy pool. The Boltzmann distribution of the execution action value is proposed by evaluating the "state-action" relationship based on the value of the "state-action" relationship, thereby realizing policy mixing and sampling the control policy that is most suitable for the current environment.

2. The offline-to-online generalizable reinforcement learning method for multi-joint robot control based on continuous policy re-enactment, as described in claim 1, is characterized in that... The specific implementation process of periodic strategy revitalization is as follows: S101 pre-constructs an offline policy, using an existing offline sequential decision dataset and an offline reinforcement learning algorithm based on "executor-critic" as a combination to train an offline reinforcement learning policy; In offline reinforcement learning, the value network and policy network are alternately optimized to complete the pre-training of the offline policy. The Q-optimization equation for the value network in offline reinforcement learning is as follows: in, Represents the mathematical expectation. The network parameters representing the value network Q are... These are samples from the offline dataset D, representing state-action-reward-next state respectively. It is the regression target of TD learning. It is the value network's estimation of the value of the current state-action sequence, and the policy network. The optimization equation is as follows: in, Represents the mathematical expectation. Representative Policy Network The parameters, It comes from the states and actions of samples in the offline dataset D. These are hyperparameters used to balance the weights of policy boosting and action cloning. It is the policy network's prediction of actions in the current state; S102 continuous policy revitalization mechanism, reactivating the learning ability of the policy network; Policy re-enhancement activates the learning ability of the policy network. Continuous policy re-enhancement includes two key steps: resetting the policy and initializing a new optimized policy. In the policy resetting step, the original input policy function parameters are stored, and the parameters of the last N layers of the network corresponding to the policy function are randomly discarded. In the initialization step, the policy function is initialized using an online experience replay pool, optimizing the following formula: in, Represents the mathematical expectation. It comes from the states and actions in the samples in the online experience replay pool B, and the value function parameters remain fixed during policy re-energization.

3. The offline-to-online generalizable reinforcement learning method for multi-joint robot control based on continuous policy re-enactment, as described in claim 2, is characterized in that... The multi-joint robot control problem is modeled as a Markov sequential decision model. The constructed offline dataset includes a "state-action-reward" model. In the multi-joint robot control problem, the state space consists of all possible poses and joint angles of the robot, and each state represents the robot's configuration at a certain moment. The state transition probability describes the probability that the robot will transition to a new state given the current state and the action taken. In multi-joint robot control, the probability is modeled based on physical dynamics and environmental interaction. The action space contains all possible control inputs. The reward function assigns a value to each state or state-action pair, indicating the merit of taking a certain action in that state. By collecting behavioral policies or expert policies and executing the collected state-action-reward sequences in the environment, an offline reinforcement learning dataset is obtained.

4. The offline-to-online generalizable reinforcement learning method for multi-joint robot control based on continuous policy re-enactment, as described in claim 1, is characterized in that... Stable online policy updates are achieved using adaptive policy constraints. During online policy learning, in addition to maximizing reward, adaptive policy constraints are applied to minimize the difference between the policy's access data distribution and the access data distribution of the behavioral policy in the online experience replay pool, using the distance metric of inter-distribution divergence. This ensures stable policy improvement. During online policy learning, the value function and policy function are optimized alternately. The optimization objective for the value function is as follows: in, Represents the mathematical expectation. The network parameters representing the value network Q are... These are samples from the online experience replay pool; the rest of their meaning remains consistent with the offline optimization objective. The optimization objective for the policy function is as follows: 。 5. The offline-to-online generalizable reinforcement learning method for multi-joint robot control based on continuous policy re-enactment, as described in claim 1, is characterized in that... A historical strategy set is constructed, and these strategies are then mixed. Based on a continuous strategy revitalization mechanism, the learning capability of the online learning strategy is restored by resetting the historical strategies. For the collected historical strategies, the action that best suits the current data distribution is selected through strategy mixing. The probability of each strategy proposing an action being selected is calculated as follows: in, Is the first choice The probability of a strategy proposing an action, where k is the total number of historical strategies. It is the value function that predicts the value of the action proposed by the j-th strategy. It is the temperature coefficient.

6. The offline-to-online generalizable reinforcement learning method for multi-joint robot control based on continuous policy re-enactment as described in claim 1, characterized in that, The actual deployment of the strategy is carried out by combining continuous strategy re-enhancement and adaptive strategy regularization constraints. At this stage, based on the policy set mixing of multi-joint robot control strategies, after the state perception obtained by the sensor is sent to the policy set, all control strategies will predict the possible robot joint angles and torque output actions, forming a set of candidate actions. Based on the Boltzmann distribution constructed by learning the predicted values ​​of these actions according to the value function, I can select actions that adapt to the current distribution to be executed in the actual environment. After the robot executes the action, the environmental samples are added to the experience replay pool, which further changes the distribution of behavioral policy samples in the experience replay pool and improves the target policy in the adaptive strategy constraint module. The optimized strategy is continuously added to the historical policy set through strategy re-enhancement.

7. A multi-joint robot control device based on continuous strategy re-vibration, characterized in that, Includes the following components: Periodic strategy re-emphasis component: Based on the execution of multi-joint robots in online scenarios during task rounds, relative time is used as the trigger condition. When this condition is reached during online learning, strategy re-emphasis is performed. Strategy re-emphasis resets the last N layers of network parameters in the control strategy network of the multi-joint robot. The parameter N is determined by the size of the network and the characteristics of the task. For the initialized control strategy network, a new control strategy is initialized and adapted to online learning based on the historical logs of robot action execution in the historical experience replay pool. Adaptive policy adjustment component: During actual training, historical behavioral policy data stored in the experience replay pool is analyzed. By modeling the action sequence, state transition, and reward acquisition pattern of the policy data, the constraint objective of the behavioral policy is constructed. By introducing a constraint loss term that minimizes the current learning policy and the behavioral policy, the out-of-distribution error that causes serious deviations in the learning policy is avoided. Policy Pool and Hybrid Policy Execution Component: When the robot interacts with the environment, a set of candidate actions is obtained by passing the task input information obtained by the robot's sensors to each function in the policy pool; the Boltzmann distribution of the execution action value is proposed by evaluating the "state-action" relationship based on the value of the historical policy through the value-based Q function, thereby realizing the hybridization of policies and sampling the control policy most suitable for the current environment.

8. The multi-joint robot control device based on continuous strategy re-vibration according to claim 7, characterized in that, The device is based on a computer hardware architecture, including a processor, memory, and input / output devices. The processor is responsible for executing the implementation process of each component, while the memory is used to store control program code, offline datasets, and real-time processed data. The input / output devices include sensors for monitoring the robot's status and environmental feedback, as well as communication interfaces for sending control commands. All these hardware components cooperate through a data transmission bus or network to jointly realize the control and learning of the multi-joint robot.

Images