Intrinsic reward-based reinforcement learning model training method

Abstract

The invention discloses a reinforcement learning model training method based on internal rewards, and particularly relates to the technical field of machine learning, a plurality of different internal reward functions are introduced by setting a reward pool, and the optimal internal reward function is automatically selected according to different tasks and learning processes, so that the learning efficiency is improved. Therefore, the exploration capability and the sampling efficiency of the intelligent agent are improved to the maximum extent, and unbiasedness of the optimal strategy is ensured. The proposed model algorithm is verified on the basis of multiple tasks of a Mini Grid environment, and experimental results show that the algorithm can effectively solve the tasks of sparse rewards and difficult exploration, and the model training speed is remarkably improved. According to the method, the target deviation problem and the task adaptation problem existing in the use of the existing internal reward method are solved, so that the application of the internal reward method is more universal, flexible and efficient.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a reinforcement learning model training method based on intrinsic rewards. Background Art

[0002] Reinforcement learning, one of the three major paradigms of machine learning, aims to enable intelligent agents to learn how to take optimal actions to achieve specific goals through interaction with the environment. In reinforcement learning, the agent gradually optimizes its decision-making strategy by trying different actions and receiving feedback information such as rewards or penalties from the environment. Its uniqueness lies in that the agent does not need explicit labeled samples, but learns autonomously and makes better decisions through trial and error. This makes reinforcement learning perform well in many complex tasks and fields, such as autonomous driving, game strategy optimization, and robot control.

[0003] However, existing reinforcement learning methods mainly rely on extrinsic rewards for training, which has the following problems: the general condition for model convergence is to visit all possible "state-action" pairs infinitely many times, but existing algorithms will fall into local optimal solutions after a finite number of iterations and cannot be further improved; in most real-world scenarios, the reward space of the task is very sparse, and it is impossible to provide a stable reward stream for the agent, making it difficult for the agent to explore valuable states, resulting in complete failure of model training; the model obtained by training only with task rewards has poor generalization ability and cannot make reasonable decisions when facing new and slightly different environments.

[0004] Intrinsic rewards refer to reward signals generated by the agent itself during the learning process, rather than reward signals from the external environment, which are highly similar to human "curiosity". By using intrinsic rewards, the agent can explore autonomously without explicit external reward signals, and it is easier to discover valuable behaviors and strategies. Intrinsic rewards can be regarded as the driving force of the agent itself to a certain extent, helping it to better learn and adapt to complex task environments. However, the use of intrinsic rewards has the problem of "target deviation", that is, the optimization of mixed rewards cannot guarantee the invariance of the optimal strategy (only for task rewards). In addition, the performance deviation of specific intrinsic reward methods in different task scenarios is extremely large, and there is a lack of effective selection methods.

[0005] Chinese patent CN116147627A discloses a mobile robot autonomous navigation method that combines deep reinforcement learning and intrinsic motivation. It uses visual sensors to obtain information from the environment, uses the D3QN algorithm to select the optimal action, and introduces a curiosity-based intrinsic motivation module to address the problem of sparse rewards in the navigation environment. The difference between the predicted state feature vector and the actual state feature vector is used as curiosity, so that the robot can maintain a high exploration efficiency when the reward is sparse. However, this patent restricts the use of a fixed intrinsic reward method and has limitations. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a reinforcement learning model training method based on intrinsic reward in response to the current status of the existing technology. A variety of different intrinsic reward functions are introduced by setting a reward pool, and the optimal intrinsic reward function is automatically selected according to the different tasks and learning processes, so as to automatically select the optimal intrinsic reward method for different tasks, maximize the exploration ability and sampling efficiency of the intelligent body, realize model training acceleration and ensure the invariance of the optimal strategy.

[0007] The technical solution adopted by the present invention to solve the above technical problems is: a reinforcement learning model training method based on intrinsic rewards, characterized in that it includes the steps of:

[0008] S1. Obtain the initialization strategy network, the initialization value network, the initialization intrinsic reward pool, and the queue for storing task rewards according to the learning model task;

[0009] S2. Obtain the intrinsic reward function of the current round from the intrinsic reward pool through the confidence interval upper bound algorithm. The confidence interval upper bound algorithm expression is as follows:

[0010]

[0011] In the formula, I t is the intrinsic reward function of the current round, I is the intrinsic reward, is the set of intrinsic reward functions, Q t (I) is the value estimate of the intrinsic reward function, c is a constant that controls the degree of exploration, t is time, and N t (I) is the number of times the intrinsic reward was selected before time t;

[0012] S3. Obtain the set through strategy sampling T steps And the set is calculated by the intrinsic reward function Each quadruple (s t ,a t ,E t ,s t+1) is the intrinsic reward, according to the current state s in each quadruple t Perform task return and advantage estimation operations processing;

[0013] S4. Obtain the updated policy network and value network through the gradient descent algorithm, and obtain the value estimate of the intrinsic reward function through the average value of the task reward estimate of the current round. The value estimate expression of the intrinsic reward function is as follows:

[0014]

[0015] Where W is the queue length, E is the external reward, is the prediction result of the value network for the task reward, is all states s in a single round t The average value of task reward estimates, φ is the parameter of the value network;

[0016] S5. Make a judgment based on the set number of training steps. If the current round is the last step, end the training. If not, return to step S2.

[0017] Furthermore, in the step S2, the external reward is maximized by the intrinsic reward function of the current round to obtain the maximum value of the external reward.

[0018] Furthermore, the intrinsic reward function of the current round of the maximization process is used to obtain all states s within the set number of steps. t The corresponding intrinsic reward is weightedly added by the extrinsic reward, and the weighted addition expression is as follows:

[0019]

[0020] In the formula, J E+I (θ) is the total reward, θ is the policy network, is the mathematical expectation, γ t is the discount factor, β t is the weight coefficient and β t =β0(1-κ) t , κ is the decay rate, E t For external rewards.

[0021] Furthermore, the four-tuple (s t ,a t ,E t ,s t+1 ), a t For action, s t+1 is the next state, and the current state s is transmitted through the agent tTo the strategy network and value network, output the action a through the strategy network t And make advantage estimation, and estimate the external returns through the value network.

[0022] Furthermore, the set number of training steps is the set number of steps for the agent to interact with the environment.

[0023] Compared with the prior art, the present invention has at least the following beneficial effects:

[0024] (1) The present invention provides a method for training a reinforcement learning model based on intrinsic rewards, which provides a universal and efficient method for using intrinsic rewards, automatically selects the optimal intrinsic reward method for different tasks, accelerates model training and ensures the invariance of the optimal strategy;

[0025] (2) By optimizing the algorithm, the entire model training is made more convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A flowchart of a reinforcement learning model training method based on intrinsic rewards of the present invention;

[0027] Figure 2 A framework diagram of a reinforcement learning model training method based on intrinsic rewards of the present invention;

[0028] Figure 3 An Empty-16x16 task environment diagram of a reinforcement learning model training method based on intrinsic rewards of the present invention;

[0029] Figure 4 A DoorKey-6x6 task environment diagram of a reinforcement learning model training method based on intrinsic rewards of the present invention;

[0030] Figure 5 A DoorKey-8x8 task environment diagram of a reinforcement learning model training method based on intrinsic rewards of the present invention;

[0031] Figure 6 This is the training result diagram in the Empty-16x16 task environment;

[0032] Figure 7 This is the training result diagram in the DoorKey-6x6 task environment;

[0033] Figure 8 This is the training result diagram in the DoorKey-8x8 task environment. DETAILED DESCRIPTION

[0034] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to these embodiments.

[0035] Embodiment 1

[0036] To solve the above problems, Figure 1 and Figure 2 As shown, the present invention provides a reinforcement learning model training method based on intrinsic rewards, characterized in that it includes the steps of:

[0037] S1. Obtain the initialization strategy network, the initialization value network, the initialization intrinsic reward pool, and the queue for storing task rewards according to the learning model task;

[0038] S2. Obtain the intrinsic reward function of the current round from the intrinsic reward pool through the confidence interval upper bound algorithm. The confidence interval upper bound algorithm expression is as follows:

[0039]

[0040] In the formula, I t is the intrinsic reward function of the current round, I is the intrinsic reward, is the set of intrinsic reward functions, Q t (I) is the value estimate of the intrinsic reward function, c is a constant that controls the degree of exploration, t is time, and N t (I) is the number of times the intrinsic reward was selected before time t;

[0041] S3. Obtain the set through strategy sampling T steps And the set is calculated by the intrinsic reward function Each quadruple (s t ,a t ,E t ,s t+1 ) is the intrinsic reward, according to the current state s in each quadruple t Perform task return and advantage estimation operations processing;

[0042] S4. Obtain the updated policy network and value network through the gradient descent algorithm, and obtain the value estimate of the intrinsic reward function through the average value of the task reward estimate of the current round. The value estimate expression of the intrinsic reward function is as follows:

[0043]

[0044] Where W is the queue length, E is the external reward, is the prediction result of the value network for the task reward, is all states s in a single round tThe average value of task reward estimates, φ is the parameter of the value network;

[0045] S5. Make a judgment based on the set number of training steps. If the current round is the last step, end the training. If not, return to step S2.

[0046] According to the corresponding learning model task, the training environment is initialized, and the policy network, value network, intrinsic reward pool, and queue for storing task rewards are initialized, and several intrinsic reward functions are selected, such as {I 1 ,…,I n} is used to build a reward pool, thereby initializing a queue of length W to store task rewards In each round, the intrinsic reward function of the current round in the queue storing the task reward is obtained through the confidence interval upper bound algorithm. The confidence interval upper bound algorithm expression is as follows:

[0047]

[0048] In the formula, I t is the intrinsic reward function of the current round, I is the intrinsic reward, is the set of intrinsic reward functions, Q t (I) is the value estimate of the intrinsic reward, c is a constant that controls the degree of exploration, t is time, and N t (I) is the number of times the intrinsic reward is selected before time t; the mechanism of intrinsic reward shaping is to promote agent exploration through intrinsic rewards, thereby increasing the probability of learning better strategies, that is, the probability of obtaining more extrinsic rewards. The optimization goal of this expression is to maximize the processing of extrinsic rewards, and to obtain the highest value of extrinsic rewards through update iterations. When the highest value of extrinsic rewards is obtained, this intrinsic reward function is selected; then the set is obtained through strategy sampling T steps And calculate the set through the intrinsic reward function Each quadruple (s t ,a t ,E t ,s t+1 ) is the intrinsic reward, according to the current state s in each quadruple t Perform task reward and advantage estimation operations, where s t is the current state, a t For action, s t+1 is the next state, E t is an external reward, the agent observes the current state s t And pass it to the policy network and value network, and output action a through the policy network tAnd perform advantage estimation, estimate the external reward through the value network, where the updated policy network and value network are obtained through the gradient descent algorithm, and the value estimate of the intrinsic reward function is obtained by the average value of the task reward estimate of the current round. The value estimate expression of the intrinsic reward function is as follows:

[0049]

[0050] Where W is the queue length, θ is the policy network, and E is the external reward. is the prediction result of the value network for the task reward, is the current state s in a single round t The average value of the task reward estimate, φ is the parameter of the value network; according to the selected intrinsic reward function, it is used to calculate all current states s in the round t The corresponding intrinsic reward is obtained by maximizing the intrinsic reward function of the current round to obtain all states s within the set number of steps. t The corresponding intrinsic rewards are weighted added through the extrinsic rewards. The weighted addition expression is as follows:

[0051]

[0052] In the formula, J E+I (θ) is the total reward, θ is the policy network, is the mathematical expectation, γ t is the discount factor, β t is the weight coefficient and β t =β0(1-κ) t , κ is the decay rate, E t For external rewards.

[0053] The judgment is made according to the set number of training steps. If the current round is the last step, the training is terminated. If not, it returns to step S2, where the set number of training steps is the set number of steps for the agent to interact with the environment. This method ensures that the choice of intrinsic rewards serves the extrinsic rewards. This method improves the exploration efficiency while ensuring the correctness of the strategy optimization direction.

[0054] To further illustrate the sampling efficiency of the present invention, Figures 3 to 5 As shown in the figure, the present invention uses MiniGrid as the experimental environment for experiments, where the triangle represents the agent and the square represents the target point. In the Empty-16x16 task, the agent needs to find the shortest path to reach the target point, while in the DoorKey-6x6 and DoorKey-8x8 tasks, the agent needs to first find the key to unlock the target and then reach the target point. In order to keep the external rewards of the tasks consistent, the external reward function formula is:

[0055]

[0056] In the formula, E t is the external reward, t0 is the current step number, t max By controlling the external reward, the sampling efficiency of the present invention is better verified.

[0057] The hyper parameters set by the present invention are shown in Table 1:

[0058] Table 1 Hyperparameter settings

[0059]

[0060]

[0061] According to the test results analysis, Figures 6 to 8 As shown in the figure, the A2C algorithm is used for single operation, the A2C algorithm is combined with the RE3 algorithm, i.e. A2C+RE3, and the two common operation modes are combined with the algorithm of the present invention and the A2C algorithm, i.e. A2C+AIRS. The three modes are compared. The experimental results show that A2C+RE3 surpasses the A2C algorithm in all three task environments. For the Empty-16x16 task, both A2C and A2C+RE3 can successfully solve the task, but A2C+RE3 achieves higher sampling efficiency; in the DoorKey-6x6 task, A2C+RE3 obtains higher reward value and faster convergence speed with the same number of training steps; in the DoorKey-8x8 task, the learning of A2C alone fails, while A2C+RE3 can successfully solve the task. The above experimental results show that the intrinsic reward method has a significant improvement in the exploration ability of the intelligent agent. However, judging from the experimental results, A2C+AIRS achieved the highest score and sampling efficiency in all three tasks. It can be seen that the algorithm model proposed in the present invention can effectively alleviate the target deviation problem in intrinsic reward applications, and can intelligently control the exploration degree of the intelligent agent, thereby maximizing the exploration and learning efficiency.

[0062] In summary, the present invention provides a reinforcement learning model training method based on intrinsic rewards. It is based on the confidence interval upper bound algorithm, introduces a variety of different intrinsic reward functions by setting a reward pool, and automatically selects the optimal intrinsic reward function according to the different tasks and learning processes, thereby maximizing the exploration ability and sampling efficiency of the intelligent agent, and ensuring the unbiasedness of the optimal strategy. The proposed model algorithm is verified based on a variety of tasks in the MiniGrid environment. The experimental results show that the algorithm can effectively solve the tasks of sparse rewards and difficult exploration, and significantly improve the model training speed. The present invention solves the target deviation problem and task adaptation problem existing in the use of existing intrinsic reward methods, making the application of intrinsic reward methods more universal, flexible and efficient.

[0063] It should be noted that all directional indications in the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0064] In addition, in the present invention, descriptions such as "first", "second", "one", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0065] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0066] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0067] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways without departing from the scope defined by the spirit of the present invention.

Claims

1. A reinforcement learning model training method based on intrinsic rewards, characterized in that: Includes steps: S1. Obtain the initialization strategy network, the initialization value network, the initialization intrinsic reward pool, and the queue for storing task rewards according to the learning model task; S2. Obtain the intrinsic reward function of the current round from the intrinsic reward pool through the confidence interval upper bound algorithm. The confidence interval upper bound algorithm expression is as follows: In the formula, I t is the intrinsic reward function of the current round, I is the intrinsic reward, is the set of intrinsic reward functions, Q t (I) is the value estimate of the intrinsic reward function, c is a constant that controls the degree of exploration, t is time, and N t (I) is the number of times the intrinsic reward was selected before time t; S3. Obtain the set through strategy sampling T steps And the set is calculated by the intrinsic reward function Each quadruple (s t ,a t ,E t ,s t+1 ) is the intrinsic reward, according to the current state s in each quadruple t Perform task return and advantage estimation operations processing; S4. Obtain the updated policy network and value network through the gradient descent algorithm, and obtain the value estimate of the intrinsic reward function through the average value of the task reward estimate of the current round. The value estimate expression of the intrinsic reward function is as follows: Where W is the queue length, E is the external reward, is the prediction result of the value network for the task reward, is all states s in a single round t The average value of task reward estimates, φ is the parameter of the value network; S5. Make a judgment based on the set number of training steps. If the current round is the last step, end the training. If not, return to step S2.

2. The method for training a reinforcement learning model based on intrinsic rewards according to claim 1, characterized in that: In the step S2, the external reward is maximized through the intrinsic reward function of the current round to obtain the maximum value of the external reward.

3. The method for training a reinforcement learning model based on intrinsic rewards according to claim 2, characterized in that: The intrinsic reward function of the current round that is maximized is used to obtain all states s within the set number of steps. t The corresponding intrinsic reward is weightedly added by the extrinsic reward, and the weighted addition expression is as follows: In the formula, J E+I (θ) is the total reward, θ is the policy network, is the mathematical expectation, γ t is the discount factor, β t is the weight coefficient and β t =β0(1-κ) t , κ is the decay rate, E t For external rewards.

4. The method for training a reinforcement learning model based on intrinsic rewards according to claim 1, characterized in that: The four-tuple (s t ,a t ,E t ,s t+1 ), a t For action, s t+1 is the next state, and the current state s is transmitted through the agent t To the strategy network and value network, output the action a through the strategy network t And make advantage estimation, and estimate the external returns through the value network.

5. The method for training a reinforcement learning model based on intrinsic rewards according to claim 1, characterized in that: The set number of training steps is the set number of steps for the agent to interact with the environment.

Images