Agent reinforcement learning method and device based on iterative policy constraint

Abstract

The application provides an agent reinforcement learning method and device based on an iterative policy constraint, comprising: performing policy offline learning on an agent based on a state of the agent in any application scenario; taking an optimized policy obtained through offline learning as an initial policy, and constructing an iterative policy constraint term; introducing the iterative policy constraint term on the basis of online reinforcement learning for maximizing a reward, to generate an optimized target of the agent; and performing policy online reinforcement learning on the agent based on the optimized target. Through iterative updating of the policy constraint, the application can not only avoid a decrease in policy performance in an early online fine-tuning stage of offline-to-online reinforcement learning, but also weaken the policy constraint in a later training stage, to obtain an optimal policy.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to an agent reinforcement learning method and apparatus based on iterative policy constraints. Background Technology

[0002] Online reinforcement learning, an increasingly important AI technology, is based on the idea that an agent learns a policy that maximizes cumulative rewards through continuous interaction with its environment. However, the initial exploration process during training is typically random, which is risky, costly, and inefficient, making it difficult to apply to real-world decision-making optimization scenarios (such as autonomous driving, data center energy optimization, and tuning of complex industrial equipment). Offline reinforcement learning and offline imitation learning methods are based on offline learning, where model training is entirely based on pre-collected offline datasets, without contacting the real system and requiring no additional high-precision simulations or interaction with the real system. However, offline policy learning methods are heavily influenced by dataset quality, making it difficult to learn optimal policies on low-quality datasets. Therefore, a direct solution is to pre-train a suboptimal policy using offline reinforcement learning or offline imitation learning methods, and then fine-tune the policy performance using online reinforcement learning methods. This approach eliminates the limitations imposed by dataset quality on offline policy learning and provides a superior initial policy for online reinforcement learning, skipping the early, inefficient, costly, and risky random exploration phase. Because it aligns with the real-world needs of decision optimization scenarios, this "offline pre-training, online fine-tuning" paradigm has garnered widespread attention from academia and industry, achieving significant breakthroughs in robot control and natural language processing (ChatGPT, GPT4), and holds promise for future applications in autonomous driving decision-making and industrial scenario optimization.

[0003] However, current offline-to-online reinforcement learning still faces many challenges, the biggest of which is distribution shift. This phenomenon is mainly concentrated in the early stages of online fine-tuning. In the early stages of fine-tuning, because online fine-tuning collects a lot of new data that has never appeared in the offline dataset, policy learning on this newly introduced data will result in large estimation errors due to the limited data distribution. This estimation error will affect the policy optimization process, causing a serious degradation in policy performance and destroying the better policies obtained from offline pre-training.

[0004] To address the performance degradation of agent policies due to initial distribution drift during online fine-tuning, existing offline-to-online reinforcement learning methods typically employ behavioral regularization or value function regularization to strictly restrict the online fine-tuning policy π to the distribution of the offline pre-trained policy or offline dataset, thus preventing severe performance degradation. Specifically, offline-to-online reinforcement learning methods based on behavioral regularization achieve policy constraints by limiting the KL (Kullback-Leibler) divergence between the online fine-tuning policy and the dataset distribution. Implicit policy constraints can then be achieved by solving the KKT (Karush-Kuhn-Tucker) conditions of the optimization problem. However, simply restricting the deviation between the online fine-tuning policy and the dataset distribution will, like offline policy learning, severely limit the performance of the fine-tuning policy to the quality of the dataset, making it difficult to obtain the optimal policy. Offline-to-online reinforcement learning methods based on value function regularization typically employ conservative Q-learning to pre-train the policy. Because the value function obtained through conservative Q-learning pre-training assigns lower value estimates to actions outside the data distribution, and since the value function reflects the policy performance, actions outside the data distribution are judged as poor-performing actions due to their lower value estimates. Therefore, in the online fine-tuning phase, actions outside the data distribution no longer participate in the policy optimization process, reducing the accumulation of errors caused by distribution drift. However, to effectively suppress the value function of all actions outside the data distribution, it is usually necessary to train multiple conservative value functions using methods such as value function aggregation, which undoubtedly consumes a great deal of computational resources and has extremely high application costs. Therefore, there is an urgent need to provide a new offline-to-online reinforcement learning method. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides an agent reinforcement learning method and apparatus based on iterative policy constraints. By iteratively updating policy constraints, it can avoid policy performance degradation during the early online fine-tuning phase of the transition from offline to online reinforcement learning, and can also reduce policy constraints in the later stages of training to obtain the optimal policy.

[0006] In a first aspect, the present invention provides an agent reinforcement learning method based on iterative policy constraints, the method comprising:

[0007] Based on the state of the agent in any application scenario, the agent performs offline policy learning.

[0008] Using the optimization strategy obtained from offline learning as the initial strategy, an iterative strategy constraint term is constructed; wherein, the iterative strategy constraint term is used to constrain the optimization strategy of each iteration within the confidence region of the optimization strategy of the previous iteration.

[0009] The iterative policy constraint term is introduced based on maximizing reward online reinforcement learning to generate the agent's optimization objective;

[0010] Based on the optimization objective, the agent undergoes online reinforcement learning of the policy.

[0011] According to the present invention, an agent reinforcement learning method based on iterative policy constraints is provided. The application scenarios of the agent include, but are not limited to, robot control task scenarios, autonomous driving task scenarios, and industrial power generation control task scenarios.

[0012] According to the present invention, an agent reinforcement learning method based on iterative policy constraints is provided, wherein the offline policy learning of the agent based on the agent's state in any application scenario includes:

[0013] Based on the state of the agent in any application scenario, the agent is subjected to offline reinforcement learning of policies.

[0014] or

[0015] Based on the state of the agent in any application scenario, the agent is subjected to offline policy imitation learning.

[0016] According to the present invention, an agent reinforcement learning method based on iterative policy constraints is provided, wherein the iterative policy constraint term is:

[0017] Where k ≥ 0 and k is an integer, (a|s) represents the decision action a given state s, π(a|s) is the candidate policy of the agent in the (k+1)th iteration under condition (a|s), and π k (a|s) is the optimization strategy of the agent in the k-th iteration under the condition (a|s), π0(a|s) is the optimization strategy obtained by offline learning under the condition (a|s), α is the conservatism factor, and f(·) is the regularization function corresponding to the f-divergence.

[0018] According to the agent reinforcement learning method based on iterative policy constraints provided by the present invention, the optimization objective is as follows:

[0019]

[0020] Where γ is the depreciation factor, s t and a t Let r represent the state and action of the agent at time t in any given application scenario. t Let a be the agent's policy reward at time t, (a t |s t ) represents the known state s t Decision action a in the following situation t,π(a t |s t )for( a t|s t The candidate policy for the agent in the (k+1)th iteration under the given condition, π k (a t |s t ) is (a t |s t The optimization strategy of the agent in the k-th iteration under the condition π k+1 The optimization strategy for the agent in the (k+1)th iteration is given.

[0021] According to the present invention, an agent reinforcement learning method based on iterative policy constraints is provided, wherein the step of performing online policy reinforcement learning on the agent based on the optimization objective includes:

[0022] Determine the equivalence of the optimization objective within the actor-critic framework;

[0023] The agent is then subjected to online reinforcement learning of its policies using the aforementioned equivalence formula.

[0024] According to the present invention, an agent reinforcement learning method based on iterative policy constraints is provided, and the equivalence formula is as follows:

[0025] Actionist:

[0026] critic:

[0027] Among them, s t+1 and a t+1 Let r represent the state and action of the agent at time t+1 in any given application scenario. t The policy reward for the agent at the current moment. Let P(·|s) be the state set. t ,a t ) is given (s t ,a t The state transition matrix at time (a) t+1 |s t+1 ) represents the known state s t+1 Decision action a in the following situation t+1 ,π(·|s t ) represents the action set, π k (·|s t+1 ) is (a t+1 |s t+1 Under the condition that the agent's policy in the k-th iteration contains the set of actions, Q(s) t+1 ,at+1 ) represents the agent in state s t+1 Under the condition of a t+1 The value function corresponding to the strategy. For the agent in state s during the k-th iteration t+1 Under the condition of a t+1 Let π be the value function corresponding to the strategy. k (a t+1 |s t+1 ) is (a t+1 |s t+1 The candidate strategy for the agent in the k-th iteration under the given condition, π k-1 (a t+1 |s t+1 ) is (a t+1 |s t+1 The optimization strategy for the agent in the k-th iteration under the given condition.

[0028] Secondly, the present invention provides an agent reinforcement learning device based on iterative policy constraints, the device comprising:

[0029] The offline learning module is used to perform offline policy learning on the agent based on the agent's state in any application scenario.

[0030] An iterative policy constraint construction module is used to construct iterative policy constraint terms using the optimization policy obtained from offline learning as the initial policy; wherein, the iterative policy constraint terms are used to constrain the optimization policy of each iteration within the confidence region of the optimization policy of the previous iteration.

[0031] An optimized target generation module is used to introduce the iterative policy constraint term based on maximizing reward online reinforcement learning, so as to generate the optimized target of the agent;

[0032] An online reinforcement learning module is used to perform online policy reinforcement learning on the agent based on the optimization objective.

[0033] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the agent reinforcement learning method based on iterative policy constraints as described in the first aspect.

[0034] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the agent reinforcement learning method based on iterative policy constraints as described in the first aspect.

[0035] This invention provides an agent reinforcement learning method and apparatus based on iterative policy constraints, comprising: offline policy learning for the agent based on the agent's state in any application scenario; constructing iterative policy constraint terms using the optimized policy obtained from offline learning as the initial policy; introducing iterative policy constraint terms based on online reinforcement learning that maximizes rewards to generate the agent's optimization objective; and performing online policy reinforcement learning for the agent based on the optimization objective. This invention, by iteratively updating policy constraints, can avoid policy performance degradation during the early online fine-tuning stage from offline to online reinforcement learning, and can also weaken policy constraints in the later stages of training to obtain the optimal policy. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating the agent reinforcement learning method based on iterative policy constraints provided by the present invention.

[0038] Figure 2 This is a schematic diagram of the structure of the intelligent agent reinforcement learning device based on iterative policy constraints provided by the present invention.

[0039] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention;

[0040] Figure label:

[0041] 310: Processor; 320: Communication interface; 330: Memory; 340: Communication bus. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0043] The technical terms involved in this invention are explained as follows:

[0044] Reinforcement Learning: Compared to supervised and unsupervised learning, reinforcement learning, as one of the hottest research topics in machine learning, does not require labeled input and output samples and tends to learn the optimal policy for completing a task. Its basic idea is that an agent receives rewards *r* through continuous interaction with the environment, and based on this, gradually forms an expectation *Q* of the cumulative reward, enabling it to make actions *a* that yield higher value based on the current state. * =max a Q, thus learning strategies to obtain the maximum reward. Where γ is the depreciation factor, r t The reward at time t.

[0045] Offline Reinforcement Learning: Offline reinforcement learning and online reinforcement learning are two major branches of reinforcement learning. Compared to online reinforcement learning, offline reinforcement learning is a data-driven, offline policy learning method that does not require online interaction between the agent and the environment. Instead, it uses a historical dataset that records the agent's "state-action-reward-state {s,a,r,s′}" transition information. The learning strategy is designed to maximize the cumulative reward. However, offline reinforcement learning is greatly affected by the quality of the dataset. When the dataset quality is low, it is difficult to learn a high-performing policy.

[0046] Offline Imitation Learning: Similar to offline reinforcement learning, offline imitation learning is a data-driven offline policy learning method, but it uses a supervised learning framework to learn the policy. Its basic idea is to use neural networks or other methods to build historical datasets. The mapping relationship between state s and action a allows the agent to perform a similar action to a after observing state s, thus achieving imitation of historical datasets; hence the name imitation learning. However, since imitation learning cannot further improve performance based on data, the quality of policy performance is severely constrained by the quality of the dataset.

[0047] Offline-to-Online Reinforcement Learning (OTL) is a policy training method that combines offline reinforcement / imitation learning and online reinforcement learning. First, a relatively optimal policy is obtained through offline reinforcement / imitation learning pre-training. Then, online reinforcement learning is used to further improve the policy obtained from the offline pre-training. This approach reduces the impact of dataset quality on offline reinforcement / imitation learning and provides a better initial policy for online reinforcement learning, skipping the early random exploration phase of online learning and improving learning efficiency.

[0048] The following is combined Figures 1-3 This invention describes the agent reinforcement learning method and apparatus based on iterative policy constraints provided by the present invention.

[0049] Firstly, this invention provides an agent reinforcement learning method based on iterative policy constraints, such as... Figure 1 As shown, the method includes:

[0050] S11. Based on the state of the agent in any application scenario, perform offline policy learning on the agent.

[0051] S12. Using the optimization strategy obtained from offline learning as the initial strategy, construct an iterative strategy constraint term; wherein, the iterative strategy constraint term is used to constrain the optimization strategy of each iteration within the confidence region of the optimization strategy of the previous iteration.

[0052] S13. Based on the online reinforcement learning that maximizes rewards, the iterative policy constraint term is introduced to generate the optimization objective of the agent;

[0053] S14. Based on the optimization objective, perform online reinforcement learning of the policy on the agent.

[0054] The intelligent agent reinforcement learning method of this invention adopts an offline-to-online reinforcement learning approach, that is, a suboptimal policy is learned in the offline stage, and then the policy performance is fine-tuned in the online reinforcement learning stage.

[0055] Specifically, in the online reinforcement learning phase of this invention, an iterative policy constraint term is set. The existence of this constraint term ensures that the optimized policy will be constrained within the confidence region of the suboptimal policy during the first round of online iteration, and the optimized policy will be constrained within the confidence region of the optimized policy in the previous round of online iteration during subsequent rounds of online iteration, thereby ensuring the monotonically increasing performance of the policy.

[0056] The biggest difference from existing methods is that the constraint term used to constrain the current iteration strategy in the optimization objective of this invention is the optimization strategy of the previous iteration, while the constraint term in existing methods is always the suboptimal strategy. Optimizing the strategy based on the above formula can both limit the deviation between the iteration strategy and the suboptimal strategy in the early stages of fine-tuning, preventing severe performance degradation, and allow the strategy constraint term to gradually deviate from the suboptimal strategy, thus giving the fine-tuning strategy sufficient degrees of freedom to find the optimal strategy.

[0057] The agent reinforcement learning method based on iterative policy constraints provided by this invention can avoid the decline in policy performance during the early online fine-tuning stage of the transition from offline to online reinforcement learning by iteratively updating policy constraints, and can also reduce policy constraints in the later stages of training to obtain the optimal policy.

[0058] Specifically, the application scenarios of the intelligent agent in S11 include, but are not limited to: robot control task scenarios, autonomous driving task scenarios, and industrial power generation control task scenarios. Offline learning includes, but is not limited to, offline reinforcement learning and offline imitation learning.

[0059] In a robot control task scenario, the agent's state 's' can include sensor data from various sensors, such as motor encoder data, images captured by cameras, and sensor data from infrared sensors, LiDAR, etc. The agent's action 'a' can include controllable commands such as motor torque commands, motor speed commands, and motor encoder position commands. 'r' is a reward function designed manually based on the robot's task execution (e.g., a robotic arm handling task).

[0060] In autonomous driving scenarios, the agent's state *s* can include high-precision map information, semantic segmentation information of the surrounding environment, the vehicle's current speed, position, acceleration, and heading information, as well as perception data from various sensors. The agent's action *a* can include controllable motion decisions such as braking force, throttle force, steering angle, and the position of the next target point. *r* is a reward function designed manually for the autonomous driving task. For example, a larger *r* indicates a longer safe driving distance, smoother driving, and more fuel-efficient driving.

[0061] In an industrial power generation control scenario, the agent's state 's' can include sensor data recorded by each unit (e.g., combustion chamber temperature and pressure, liquid flow rate, velocity, and pressure in pipelines) and the proportion of chemical components in emitted waste. The agent's action 'a' can include various controllable quantities of the generator set (e.g., whether valves are open or closed, the extent of valve opening, pipeline flow thresholds, etc.). 'r' is a reward function designed manually based on the power generation control task; for example, a higher power generation efficiency results in a larger 'r'.

[0062] Specifically, the iterative strategy constraint term in S12 is:

[0063] Where k ≥ 0 and k is an integer, (a|s) represents the decision action a given state s, π(a|s) is the candidate policy of the agent in the (k+1)th iteration under condition (a|s), and π k (a|s) is the optimization strategy of the agent in the k-th iteration under condition (a|s), π0(a|s) is the optimization strategy obtained by offline learning under condition (a|s), α is the conservatism factor, and f(·) is the regularization function corresponding to f-divergence, such as the logarithmic function.

[0064] Minimize this invention This makes π k+1 (a|s) and π k (a|s) should be as close as possible to π. k+1 (a|s) is constrained by π k (a|s) Attachment (confidence region).

[0065] Specifically, in step S13, an iterative policy constraint term is introduced into the standard online reinforcement learning optimization objective of maximizing reward, thereby obtaining the online reinforcement learning optimization objective of this invention, as follows:

[0066]

[0067] Where γ is the depreciation factor, s t and a t Let r represent the state and action of the agent at time t in any given application scenario. t Let a be the agent's policy reward at time t, (a t |s t ) represents the known state s t Decision action a in the following situation t ,π(a t |s t ) is (a t |s t The candidate policy for the agent in the (k+1)th iteration under the given condition, π k (a t |s t ) is (a t |s t The optimization strategy of the agent in the k-th iteration under the condition π k+1 Let t be the optimization strategy for the agent in the (k+1)th iteration, and t be the current time.

[0068] In other words, the optimization objective of this invention is to maximize the reward while keeping the strategies of adjacent iterations relatively close.

[0069] The design of the online reinforcement learning optimization objective in this invention has the following advantages:

[0070] (1) Only additionally introduced The computational cost introduced by this calculation is almost negligible, thus it has high computational efficiency.

[0071] (2) Only the policy π0 obtained from offline learning is needed, and its corresponding value function is not required. Therefore, it can be applied to all offline learning methods.

[0072] (3) Existing offline-to-online reinforcement learning methods are usually only improved for online reinforcement learning in a specific scenario and cannot be generalized to all offline-to-online reinforcement learning methods, resulting in poor universality. However, this invention has no such limitation and can be flexibly and easily introduced into all offline-to-online reinforcement learning methods, thus having high universality and application value.

[0073] Specifically, S14 includes: determining the equivalence of the optimization objective within the actor-critic framework;

[0074] The agent is then subjected to online reinforcement learning of its policies using the aforementioned equivalence formula.

[0075] Furthermore, the equivalence formula is as follows:

[0076] Actionist:

[0077] critic:

[0078] Among them, s t+1 and a t+1 Let r represent the state and action of the agent at time t+1 in any given application scenario. t The policy reward for the agent at the current moment. Let P(·|s) be the state set. t ,a t ) is given (s t ,a t The state transition matrix at time (a) t+1 |s t+1 ) represents the known state s t+1 Decision action a in the following situation t+1 ,π(·|s t ) represents the action set, π k (·|s t+1 ) is (a t+1 |s t+1 Under the condition that the agent's policy in the k-th iteration contains the set of actions, Q(s) t+1 ,a t+1 ) represents the agent in state st+1 Under the condition of a t+1 The value function corresponding to the strategy. For the agent in state s during the k-th iteration t+1 Under the condition of a t+1 Let π be the value function corresponding to the strategy. f (a t+1 |s t+1 ) is (a t+1 |s t+1 The candidate strategy for the agent in the k-th iteration under the given condition, π k-1 (a t+1 |s t+1 ) is (a t+1 |s t+1 The optimization strategy for the agent in the k-th iteration under the given condition.

[0079] The implementation of S14 of this invention only requires the calculation of two regularization terms in the standard actor-critic framework, so it can be easily and conveniently integrated into all online reinforcement learning methods based on the actor-critic framework, and has extremely high universality.

[0080] In summary, the present invention has the following advantages:

[0081] (1): It ensures both stable online fine-tuning and the acquisition of the optimal strategy.

[0082] (2): The additional computational cost of implementation is extremely low and negligible.

[0083] (3): It is applicable to all offline policy learning methods and almost all online reinforcement learning methods, and has extremely high universality.

[0084] To better apply the reinforcement learning method of this invention, an application example is provided, taking robot control as an example, as follows:

[0085] In the robot control task scenario, state s includes: the perception data of various sensors of the robot; action a includes motor torque command, motor speed command and motor encoder position command; the robot performs a robotic arm handling task, and r is a reward function, which is set according to the on-site requirements.

[0086] The sensor data of the robot includes, but is not limited to, motor encoder data, images captured by the camera, infrared sensor data, and lidar data.

[0087] Robot control decision-making methods based on reinforcement learning include:

[0088] S21: Based on the robot's current state, perform offline strategy learning on the robot;

[0089] S22: Using the optimization strategy obtained from offline learning as the initial strategy, construct an iterative strategy constraint term; wherein, the iterative strategy constraint term is used to constrain the optimization strategy of each iteration within the confidence region of the optimization strategy of the previous iteration.

[0090] S23: Based on the online reinforcement learning that maximizes rewards, the iterative policy constraint term is introduced to generate the policy optimization objective of the robot;

[0091] S24: Based on the policy optimization objective, perform online reinforcement learning on the robot to obtain the optimal policy for the robot.

[0092] Specifically, offline learning in S21 includes two types: offline reinforcement learning and offline simulation learning.

[0093] When offline reinforcement learning is used, the robot's current state is fed into the first pre-trained offline training model, and the corresponding optimization strategy can be learned.

[0094] The training process of the first offline training model includes:

[0095] Construct an offline reinforcement learning dataset for the robot; the samples in the offline reinforcement learning dataset record the transition information "current state-action-reward-next state";

[0096] Based on the aforementioned offline reinforcement learning dataset, the first offline training model is trained with the strategy of maximizing reward as the optimization objective.

[0097] When offline simulation learning is used, the robot's current state is fed into a pre-trained second offline training model, and the corresponding optimization strategy can be learned.

[0098] The training process of the second offline training model includes:

[0099] Construct an offline simulation learning dataset for the robot; the samples in the offline simulation learning dataset record "current state - current action";

[0100] Using the current state as the input to the initial neural network and the current action as the output of the initial neural network, a second offline training model that can reflect the mapping relationship between the state and the action is trained.

[0101] The iterative strategy constraint in S22 is:

[0102] Where k ≥ 0 and k is an integer, (a|s) represents the decision action a given state s, π(a|s) is the candidate strategy for the robot in the (k+1)th iteration under condition (a|s), and π k (a|s) is the optimization strategy of the robot in the k-th iteration under the condition (a|s), π0(a|s) is the optimization strategy obtained by offline learning under the condition (a|a), α is the conservatism factor, and f(·) is the regularization function corresponding to the f-divergence, such as the logarithmic function.

[0103] The strategy optimization objective in S23 is as follows:

[0104]

[0105] Where γ is the depreciation factor, s t and a t Let r represent the robot's state and action at time t, respectively. t Let be the robot's policy reward at time t, (a t |s t ) represents the known state s t Decision action a in the following situation t ,π(a t |s t ) is (a t |s t Candidate strategies for the robot in the (k+1)th iteration under the given condition, π k (a t |s t ) is (a t |s t The optimization strategy for the robot in the k-th iteration under the condition π k+1 Let t be the current time, representing the optimization strategy for the robot's (k+1)th iteration.

[0106] When k = 0, the above equation is equivalent to maximizing the reward function within the confidence region of the optimization policy π0. When k ≥ 1, the above equation will change the optimization policy π in the (k+1)th iteration. k+1 Optimization strategy π limited to the k-th iteration k Within the confidence region, the policy performance is guaranteed to increase monotonically.

[0107] In S24, the equivalence of the policy optimization objective under the actor-critic framework is used to perform online policy reinforcement learning for the robot;

[0108] The equivalence formula is as follows:

[0109] Actionist:

[0110] critic:

[0111] Among them, s t+1 and a t+1 Let r represent the robot's state and action at time t+1 in any given application scenario. t The strategy reward for the robot at the current moment. Let P(·|s) be the state set. t ,a t ) is given (s t ,a t The state transition matrix at time (a) t+1 |s t+1 ) represents the known state s t+1 Decision action a in the following situation t+1 ,π(·|s t ) represents the action set, π k (·|s t+1 ) is (a t+1 |s t+1 The set of actions included in the confidence region of the robot's policy in the k-th iteration under the given condition, Q(s) t+1 ,a t+1 ) represents the robot in state s t+1 Under the condition of a t+1 The value function corresponding to the strategy. For the robot in state s during the k-th iteration t+1 Under the condition of a t+1 Let π be the value function corresponding to the strategy. k (a t+1 |s t+1 ) is (a t+1 |s t+1 Candidate strategies for the robot in the k-th iteration under the given condition, π k-1 (a t+1 |s t+1 ) is (a t+1 |s t+1 The optimization strategy for the robot's k-th iteration under the given conditions.

[0112] Of course, the online reinforcement learning of the robot's policies is done on the basis of the robot's online reinforcement learning model; this robot's online reinforcement learning model is obtained by learning from a large amount of sample data using the above-mentioned online reinforcement learning method.

[0113] Secondly, the agent reinforcement learning device based on iterative policy constraints provided by the present invention will be described. The agent reinforcement learning device based on iterative policy constraints described below and the agent reinforcement learning method based on iterative policy constraints described above can be referred to in correspondence with each other. Figure 2 A schematic diagram illustrating the structure of agent reinforcement learning based on iterative policy constraints is shown, such as... Figure 2 As shown, the device includes:

[0114] Offline learning module 21 is used to perform offline policy learning on the agent based on the agent's state in any application scenario;

[0115] The iterative policy constraint construction module 22 is used to construct iterative policy constraint terms using the optimization policy obtained from offline learning as the initial policy; wherein, the iterative policy constraint terms are used to constrain the optimization policy of each iteration within the confidence region of the optimization policy of the previous iteration.

[0116] The optimization target generation module 23 is used to introduce the iterative policy constraint term based on the online reinforcement learning that maximizes rewards, so as to generate the optimization target of the agent;

[0117] The online reinforcement learning module 24 is used to perform online policy reinforcement learning on the agent based on the optimization objective.

[0118] Based on the above embodiments, as an optional embodiment, the application scenarios of the intelligent agent include, but are not limited to: robot control task scenarios, autonomous driving task scenarios, and industrial power generation control task scenarios.

[0119] Based on the above embodiments, as an optional embodiment, the offline learning module is used for:

[0120] Based on the state of the agent in any application scenario, the agent is subjected to offline reinforcement learning of policies.

[0121] or

[0122] Based on the state of the agent in any application scenario, the agent is subjected to offline policy imitation learning.

[0123] Based on the above embodiments, as an optional embodiment, the iterative strategy constraint term is:

[0124] Where k ≥ 0 and k is an integer, (a|s) represents the decision action a given state s, π(a|s) is the candidate policy of the agent in the (k+1)th iteration under condition (a|s), and π k (a|s) is the optimization strategy of the agent in the k-th iteration under the condition (a|s), π0(a|s) is the optimization strategy obtained by offline learning under the condition (a|s), α is the conservatism factor, and f(·) is the regularization function corresponding to the f-divergence.

[0125] Based on the above embodiments, as an optional embodiment, the optimization objective is as follows:

[0126]

[0127] Where γ is the depreciation factor, s t and a t Let r represent the state and action of the agent at time t in any given application scenario. t Let a be the agent's policy reward at time t, (a t |s t ) represents the known state s t Decision action a in the following situation t ,π(a t |s t ) is (a t |s t The candidate policy for the agent in the (k+1)th iteration under the given condition, π k (a t |s t ) is (a t |s t The optimization strategy of the agent in the k-th iteration under the condition π k+1 Let π be the optimization strategy for the agent in the (k+1)th iteration. k+1 (a t |s t ) is (a t |s t ) is the optimization strategy of the agent in the (k+1)th iteration under the condition, where t is the current time.

[0128] Based on the above embodiments, as an optional embodiment, the online reinforcement learning module includes:

[0129] A determining unit is used to determine the equivalence of the optimization objective within the actor-critic framework;

[0130] A learning unit is used to perform online policy reinforcement learning on the agent using the equivalence formula.

[0131] Based on the above embodiments, as an optional embodiment, the equivalent formula is as follows:

[0132] Actionist:

[0133] critic:

[0134]

[0135] Among them, s t+1 and a t+1Let r represent the state and action of the agent at time t+1 in any given application scenario. t The policy reward for the agent at the current moment. Let P(·|s) be the state set. t ,a t ) is given (s t ,a t The state transition matrix at time (a) t+1 |s t+1 ) represents the known state s t+1 Decision action a in the following situation t+1 ,π(·|s t ) represents the action set, π k (·|s t+1 ) is (a t+1 |s t+1 Under the condition that the agent's policy in the k-th iteration contains the set of actions, Q(s) t+1 ,a t+1 ) represents the agent in state s t+1 Under the condition of a t+1 The value function corresponding to the strategy. For the agent in state s during the k-th iteration t+1 Under the condition of a t+1 Let π be the value function corresponding to the strategy. k (a t+1 |s t+1 ) is (a t+1 |s t+1 The candidate strategy for the agent in the k-th iteration under the given condition, π k-1 (a t+1 |s t+1 ) is (a t+1 |s t+1 The optimization strategy for the agent in the k-th iteration under the given condition.

[0136] Thirdly, Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute an agent reinforcement learning method based on iterative policy constraints. This method includes: performing offline policy learning on the agent based on the agent's state in any application scenario; constructing iterative policy constraint terms using the optimized policy obtained from offline learning as the initial policy; wherein the iterative policy constraint terms are used to constrain the optimized policy of each iteration within the confidence region of the optimized policy of the previous iteration; introducing the iterative policy constraint terms based on online reinforcement learning to maximize reward to generate the agent's optimization objective; and performing online policy reinforcement learning on the agent based on the optimization objective.

[0137] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0138] Fourthly, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute an agent reinforcement learning method based on iterative policy constraints provided by the methods described above. The method includes: performing offline policy learning on the agent based on the state of the agent in any application scenario; constructing iterative policy constraint terms using the optimized policy obtained from offline learning as the initial policy; wherein the iterative policy constraint terms are used to constrain the optimized policy of each iteration within the confidence region of the optimized policy of the previous iteration; introducing the iterative policy constraint terms based on online reinforcement learning to maximize reward, so as to generate the optimization objective of the agent; and performing online policy reinforcement learning on the agent based on the optimization objective.

[0139] Fifthly, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements an agent reinforcement learning method based on iterative policy constraints provided by the methods described above. The method includes: performing offline policy learning on the agent based on the agent's state in any application scenario; constructing iterative policy constraint terms using the optimized policy obtained from offline learning as an initial policy; wherein the iterative policy constraint terms are used to constrain the optimized policy of each iteration within the confidence region of the optimized policy of the previous iteration; introducing the iterative policy constraint terms based on online reinforcement learning that maximizes rewards to generate the agent's optimization objective; and performing online policy reinforcement learning on the agent based on the optimization objective.

[0140] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An agent reinforcement learning method based on iterative policy constraints, characterized in that, The method includes: Based on the state of the agent in any application scenario, the agent performs offline policy learning. Using the optimization strategy obtained from offline learning as the initial strategy, an iterative strategy constraint term is constructed; wherein, the iterative strategy constraint term is used to constrain the optimization strategy of each iteration within the confidence region of the optimization strategy of the previous iteration. The iterative policy constraint term is introduced based on maximizing reward online reinforcement learning to generate the agent's optimization objective; Based on the optimization objective, the agent undergoes online policy reinforcement learning. The iterative strategy constraint is: ; in, and It is an integer. Represents a known state Decision-making actions under certain circumstances , for Under the condition of agent number 1 Candidate strategies for round iteration, for Under the condition of agent number 1 The optimization strategy of round-trip iteration, for The optimization strategy obtained through offline learning under certain conditions As a conservative factor, for - The regularization function corresponding to the divergence.

2. The agent reinforcement learning method based on iterative policy constraints according to claim 1, characterized in that, The application scenarios of the intelligent agent include, but are not limited to: robot control task scenarios, autonomous driving task scenarios, and industrial power generation control task scenarios.

3. The agent reinforcement learning method based on iterative policy constraints according to claim 1, characterized in that, The offline policy learning for the agent based on its state in any application scenario includes: Based on the state of the agent in any application scenario, the agent is subjected to offline reinforcement learning of policies. or, Based on the state of the agent in any application scenario, the agent is subjected to offline policy imitation learning.

4. The agent reinforcement learning method based on iterative policy constraints according to claim 1, characterized in that, The optimization objectives are as follows: ； in, As depreciation factor, and They are respectively The state and actions of the intelligent agent in any given application scenario. for The policy reward of the agent at any given moment Represents a known state Decision-making actions under certain circumstances , for Under the condition of agent number 1 Candidate strategies for round iteration, for Under the condition of agent number 1 The optimization strategy of round-trip iteration, For the first intelligent agent The optimization strategy of round-by-round iteration.

5. The agent reinforcement learning method based on iterative policy constraints according to claim 4, characterized in that, The step of performing online policy reinforcement learning on the agent based on the optimization objective includes: Determine the equivalence of the optimization objective within the actor-critic framework; The agent is then subjected to online reinforcement learning of its policies using the aforementioned equivalence formula.

6. The agent reinforcement learning method based on iterative policy constraints according to claim 5, characterized in that, The equivalent formula is as follows: Actionist: ; critic: ; in, and They are respectively The state and actions of the intelligent agent in any given application scenario. The policy reward for the agent at the current moment. For a set of states, For a given The state transition matrix at time, Represents a known state Decision-making actions under certain circumstances , For action sets, for Under the condition of agent number 1 The confidence region of the round-iteration strategy contains the set of actions. For the agent in state Under the condition of The value function corresponding to the strategy. For the first intelligent agent Round iteration in state Under the condition of The value function corresponding to the strategy. for Under the condition of agent number 1 Candidate strategies for round iteration, for Under the condition of agent number 1 The optimization strategy of round-by-round iteration.

7. An agent reinforcement learning device based on iterative policy constraints, characterized in that, The device includes: The offline learning module is used to perform offline policy learning on the agent based on the agent's state in any application scenario. An iterative policy constraint construction module is used to construct iterative policy constraint terms using the optimization policy obtained from offline learning as the initial policy; wherein, the iterative policy constraint terms are used to constrain the optimization policy of each iteration within the confidence region of the optimization policy of the previous iteration. An optimized target generation module is used to introduce the iterative policy constraint term based on maximizing reward online reinforcement learning, so as to generate the optimized target of the agent; An online reinforcement learning module is used to perform online policy reinforcement learning on the agent based on the optimization objective. The iterative strategy constraint is: ; in, and It is an integer. Represents a known state Decision-making actions under certain circumstances , for Under the condition of agent number 1 Candidate strategies for round iteration, for Under the condition of agent number 1 The optimization strategy of round-trip iteration, for The optimization strategy obtained through offline learning under certain conditions As a conservative factor, for - The regularization function corresponding to the divergence.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the agent reinforcement learning method based on iterative policy constraints as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the agent reinforcement learning method based on iterative policy constraints as described in any one of claims 1 to 6.

Images