Optimal strategy acquisition method and device based on double-layer deep reinforcement learning model

Abstract

The present application relates to an optimal strategy acquisition method and device based on a two-layer deep reinforcement learning model. The method comprises: setting a two-layer deep reinforcement learning model in a strategy model, including a reward function online shaping model for dividing a target into multiple sub-targets according to prior knowledge inspiration, mapping multiple specific states of the target task into multiple abstract states according to each sub-target, and calculating the online reward function through each abstract state, and a decision model for making action decisions according to each specific state, and when the decision model is trained, the online reward function provided by the reward function online shaping model is used to densify its reward, accelerate the model convergence speed, and improve the model training efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of deep reinforcement learning, and in particular to a method and device for acquiring an optimal strategy based on a double-layer deep reinforcement learning model. Background Art

[0002] Reinforcement learning is one of the three common types of machine learning algorithms. Reinforcement learning is between supervised learning and unsupervised learning. It obtains reward signals by interacting with the environment and uses these signals to reinforce the behavior it believes is correct. Reinforcement learning algorithms are particularly suitable for solving sequential decision-making problems and are the focus of this article.

[0003] In reinforcement learning, the agent observes a certain representation of the state of the environment from the environment, and based on this, selects an action according to the strategy to feed back to the environment. The next moment, the agent receives a numerical benefit signal from the environment, and the environment also enters a new state. The agent uses the collected experience to improve the strategy. This learning method of reinforcement learning makes it particularly suitable for solving sequential decision-making problems. Early reinforcement learning algorithms used tabular methods to solve the problem, which required manual extraction of environmental features and required that the state space and action space be discrete. When the environment to be observed becomes complex, the problem of too large a state space may arise, resulting in an explosion in the storage space required by the algorithm and an inability to solve it in a limited time.

[0004] In recent years, the Deepmind team has proposed a deep reinforcement learning algorithm, which uses deep learning to record environmental states and extract key features, combined with the decision-making ability of reinforcement learning, which has greatly promoted the development of reinforcement learning algorithms. For tabular reinforcement learning, the number of states that need to be stored increases exponentially with each additional dimension of input variables, while deep reinforcement learning only needs to add one input neuron, effectively dealing with the problem of state space explosion.

[0005] Based on the above advantages, deep reinforcement learning has become a key area of ​​modern reinforcement learning research, and has made remarkable progress in application scenarios such as StarCraft, unmanned driving, and drone path planning. While making progress, the theoretical and applied research of deep reinforcement learning still faces some serious problems.

[0006] Among them, the sparse reward problem is one of the core problems that restrict the application of deep reinforcement learning. In the actual application of reinforcement learning, the benefit signal is often scarce. Take the Go environment as an example. The intelligent agent cannot get rewards during the game and can only get the victory or defeat of the game after the game is over.

[0007] The sparse rewards of the training environment lead to the emergence of the following two sub-problems:

[0008] The first is the long-term credit allocation problem, that is, the problem of determining which behavior has a greater impact on the final reward. In the process of interaction between the reinforcement learning agent and the environment, the agent will receive a certain reward or punishment for performing a certain action in the environment. However, due to the sparse reward problem and the complexity of the high-dimensional environment, the reward or punishment obtained by the agent may be delayed. When the reward arrives late, it is difficult for the agent to determine which previous action caused the reward. This problem often causes confusion to the agent and leads to training failure.

[0009] The second is the problem of low training efficiency and slow convergence of the agent. The agent needs to interact with the environment a lot to continuously adjust its value function. Due to the existence of discount values ​​and a large number of non-target states, it takes time for the reward value to be transmitted back on the trajectory. When the environment is complex, the time and cost required for training may be unacceptable. Especially in real-life scenarios, the cost of interaction between the agent and the environment is often very high, which makes the training cost too high. Summary of the invention

[0010] Based on this, it is necessary to provide an optimal strategy acquisition method and device based on a two-layer deep reinforcement learning model, which can improve the reward density in a training environment and thus improve the training efficiency, in order to address the above technical problems.

[0011] A method for acquiring an optimal strategy based on a two-layer deep reinforcement learning model, the method comprising:

[0012] Obtaining a training sample set, wherein the training sample set includes multiple specific states related to a target task, actions performed by the agent in the specific states, new specific states to which the environment migrates after performing the actions, and environmental feedback rewards corresponding to each of the states;

[0013] Input the training sample set into the strategy model for training until the strategy model has the ability to output the optimal strategy for completing the target task, and obtain the trained optimal strategy model; wherein the optimal strategy model includes a reward function online shaping model and a decision model both of which are built based on the Markov decision process, in which the target task is divided into multiple sub-goals according to prior knowledge inspiration, and multiple specific states are mapped into multiple abstract states according to each of the sub-goals, and then an online reward function is obtained according to each abstract state shaping for use by the decision model, and the decision model includes a decision network, and the decision network is trained according to the environmental feedback reward and the online reward function so that it outputs the optimal strategy;

[0014] Obtain any specific state in the target task, and input the specific state into the trained optimal strategy model to obtain the optimal strategy for completing the target task with the specific state as the starting point.

[0015] In one embodiment, the target tasks include target tasks with engineering machinery as the application environment and target tasks with maze navigation as the application environment.

[0016] In one embodiment, in the online shaping model of the reward function, the target task is divided into a plurality of sub-goals according to prior knowledge inspiration, and a plurality of specific states are mapped into a plurality of abstract states according to each of the sub-goals, including:

[0017] Designing a state abstraction function according to each of the sub-goals, and mapping the specific state experienced by the agent before reaching the sub-goal into a corresponding abstract state according to the state abstraction function;

[0018] Each of the abstract states includes multiple specific states.

[0019] In one embodiment, the online reward function is further shaped according to each abstract state in the online reward function shaping model, including:

[0020] The abstract state transition reward is calculated based on the sum of the environmental feedback rewards obtained from multiple specific states when transferring from one abstract state to another;

[0021] The value function of the corresponding abstract state is calculated according to the abstract state transition reward, and the online reward function is calculated according to the value function.

[0022] In one embodiment, training the policy model includes:

[0023] Inputting a set of specific states and corresponding environmental feedback rewards in the training sample set into the reward function online shaping model and the decision model respectively;

[0024] In the reward function online shaping model, the environmental feedback reward is accumulated into the abstract state transition reward, and then the specific state is mapped into the corresponding abstract state according to the abstract function, and it is judged whether the abstract state is equal to the abstract state of the previous step. If they are not equal, it means that the current abstract state has been migrated, and the value function of the current abstract state is updated by using the abstract state transition reward. If they are equal, it means that the current abstract state has not been migrated, and the value function is not updated. The online reward function is calculated by using the value function, and the online reward function is provided to the decision model;

[0025] In the decision model, the next action is selected and executed according to the specific state, the online reward function and the environmental feedback reward, and the feedback is given to the environment;

[0026] The environment gives a new specific state and a corresponding environmental feedback reward according to the action output by the decision model, so that the reward function online shapes the model and the decision model undergoes a new round of training.

[0027] In one embodiment, before training the online reward function shaping model, the value functions corresponding to the abstract states are initialized and assigned initial values;

[0028] After completing the training of the online modeling model of the reward function, the value function corresponding to each of the abstract states is obtained as a fixed value.

[0029] In one embodiment, the decision model further includes a playback data set, and during each training, the transfer information is stored in the playback data set, and when the data in the playback data set reaches a preset amount, a portion of the data is extracted therefrom to update the network parameters of the decision model;

[0030] The transfer information includes the specific state of the current training, the output action, the sum of the reward and the online reward function based on the specific state and the output action environment feedback, and the specific state of the next step.

[0031] In one embodiment, in the decision model, the decision network adopts a deep Q-learning network.

[0032] An optimal strategy acquisition device based on a two-layer deep reinforcement learning model, the device comprising:

[0033] A training sample set acquisition module is used to acquire a training sample set, wherein the training sample set includes multiple specific states related to a target task, actions performed by the agent in the specific state, new specific states to which the environment migrates after performing the action, and environmental feedback rewards corresponding to each of the states;

[0034] An optimal strategy model training module, used for inputting the training sample set into the strategy model for training until the strategy model has the ability to output the optimal strategy for completing the target task, thereby obtaining the trained optimal strategy model; wherein the optimal strategy model includes a reward function online shaping model and a decision model both of which are built based on the Markov decision process, in which the target task is divided into a plurality of sub-goals according to prior knowledge inspiration, and a plurality of specific states are mapped into a plurality of abstract states according to each of the sub-goals, and then an online reward function is shaped according to each of the abstract states for use by the decision model, and the decision model includes a decision network, and the decision network is trained according to the environmental feedback reward and the online reward function so that it outputs the optimal strategy;

[0035] The optimal strategy acquisition module is used to acquire any specific state in the target task and input the specific state into the trained optimal strategy model to obtain the optimal strategy for completing the target task with the specific state as the starting point.

[0036] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0037] Obtaining a training sample set, wherein the training sample set includes multiple specific states related to a target task, actions performed by the agent in the specific states, new specific states to which the environment migrates after performing the actions, and environmental feedback rewards corresponding to each of the states;

[0038] Input the training sample set into the strategy model for training until the strategy model has the ability to output the optimal strategy for completing the target task, and obtain the trained optimal strategy model; wherein the optimal strategy model includes a reward function online shaping model and a decision model both of which are built based on the Markov decision process, in which the target task is divided into multiple sub-goals according to prior knowledge inspiration, and multiple specific states are mapped into multiple abstract states according to each of the sub-goals, and then an online reward function is obtained according to each abstract state shaping for use by the decision model, and the decision model includes a decision network, and the decision network is trained according to the environmental feedback reward and the online reward function so that it outputs the optimal strategy;

[0039] Obtain any specific state in the target task, and input the specific state into the trained optimal strategy model to obtain the optimal strategy for completing the target task with the specific state as the starting point.

[0040] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0041] Obtaining a training sample set, wherein the training sample set includes multiple specific states related to a target task, actions performed by the agent in the specific states, new specific states to which the environment migrates after performing the actions, and environmental feedback rewards corresponding to each of the states;

[0042] Input the training sample set into the strategy model for training until the strategy model has the ability to output the optimal strategy for completing the target task, and obtain the trained optimal strategy model; wherein the optimal strategy model includes a reward function online shaping model and a decision model both of which are built based on the Markov decision process, in which the target task is divided into multiple sub-goals according to prior knowledge inspiration, and multiple specific states are mapped into multiple abstract states according to each of the sub-goals, and then an online reward function is obtained according to each abstract state shaping for use by the decision model, and the decision model includes a decision network, and the decision network is trained according to the environmental feedback reward and the online reward function so that it outputs the optimal strategy;

[0043] Obtain any specific state in the target task, and input the specific state into the trained optimal strategy model to obtain the optimal strategy for completing the target task with the specific state as the starting point.

[0044] The above-mentioned optimal strategy acquisition method and device based on the double-layer deep reinforcement learning model, by setting a double-layer deep reinforcement learning model in the optimal strategy model, includes a reward function online shaping model for dividing the target into multiple sub-goals according to prior knowledge inspiration, mapping multiple specific states of the target task into multiple abstract states according to each sub-goal, and calculating the online reward function through each abstract state, and a decision model for making action decisions according to each specific state. When the decision model is trained, the online reward function provided by the reward function online shaping model is used to densify its reward, thereby accelerating the model convergence speed and improving the model training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a flow chart of an optimal strategy acquisition method based on a double-layer deep reinforcement learning model in one embodiment;

[0046] Figure 2 A schematic diagram of a maze learning environment in one embodiment;

[0047] Figure 3 A schematic diagram of a maze sub-goal in one embodiment;

[0048] Figure 4 A schematic diagram of state mapping of sub-goals in an embodiment;

[0049] Figure 5 A schematic diagram of an algorithm flow in an optimal strategy model in an embodiment;

[0050] Figure 6 A schematic diagram of state mapping in an embodiment;

[0051] Figure 7It is a structural block diagram of an optimal strategy acquisition device based on a double-layer deep reinforcement learning model in one embodiment;

[0052] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] like Figure 1 As shown, a method for obtaining an optimal strategy based on a two-layer deep reinforcement learning model is provided, comprising the following steps:

[0055] Step S100, obtaining a training sample set, which includes multiple specific states related to a target task, the actions performed by the agent in the specific state, the new specific state to which the environment migrates after performing the action, and the environmental feedback reward corresponding to each state;

[0056] Step S110, inputting the training sample set into the strategy model for training until the strategy model has the ability to output the optimal strategy for completing the target task, and obtaining the trained optimal strategy model; wherein the optimal strategy model includes a reward function online shaping model and a decision model both of which are built based on the Markov decision process, in which the target task is divided into multiple sub-goals according to prior knowledge inspiration in the reward function online shaping model, and multiple specific states are mapped into multiple abstract states according to each sub-goal, and then an online reward function is shaped according to each abstract state for use by the decision model, and the decision model includes a decision network, and the decision network is trained according to the environmental feedback reward and the online reward function so that it outputs the optimal strategy;

[0057] Step S120, obtaining any specific state in the target task, and inputting the specific state into the trained optimal strategy model to obtain the optimal strategy for completing the target task with the specific state as the starting point.

[0058] It should be noted that the optimal strategy acquisition method provided in this application can be applied to most problems that can be solved by deep reinforcement learning models, and the background of the application is different for the reinforcement learning model, that is, the different environmental backgrounds. In this embodiment, the target task mentioned in step S100 can be a target task with engineering machinery as the application environment, or a target task with maze navigation as the application environment. These two target tasks are two aspects of the problem that can be solved based on reinforcement learning, one is the control of the optimal strategy for the target, and the other is the output of the optimal route for the target.

[0059] Specifically, taking engineering machinery as the background, the control of a robotic arm set up next to an assembly line is taken as an example. When it is necessary to accurately control the arm based on reinforcement learning so that it can grab an item on the assembly line with the optimal trajectory. First, it is necessary to set each state of the robotic arm before reaching the target, and each current state is based on the current state after the robotic arm has performed a certain action in the previous state. Whether the current state is beneficial to the robotic arm in reaching the target is expressed by a corresponding reward fed back by the environment. The purpose of reinforcement learning is to determine the optimal trajectory from the initial state of the robotic arm to the final target by obtaining the highest reward, that is, the optimal control strategy. Therefore, in the application environment of the robotic arm, the specific states of the robotic arm are represented by spatial coordinates (x, y, z) and the angle of rotation of the robotic arm, and its action is to move in the spatial coordinates or rotate the arm, and according to the execution of the action, another state can be reached.

[0060] Specifically, taking maze navigation as an example, with the goal of getting out of the maze, the specific state of the environment is the coordinates (x, y) represented by the chessboard, and the action is to move left, right, forward, and backward. Similarly, when moving from the previous position to the current position, the environment will also give a corresponding reward.

[0061] This embodiment constructs a maze environment with a height of 5 grids and a length of 15 grids. The agent learning to walk through the maze is designated by a gray square, the end point is represented by a gray circle, and the wall is designated by the black squares in the sixth and eleventh columns. When the agent hits the wall in the next step, the agent will remain in place, indicating that it has hit the wall. Figure 2 As shown in the figure, there are two walls in the sixth and eleventh columns, each with a door for the agent to pass through. The agent can only get a +1 reward when it reaches the end, which means that the scene has complex terrain, a large exploration space and sparse rewards. In this scene, if the strategy is not good enough, the agent needs to walk a long distance, sometimes even hundreds or thousands of steps, to reach the end to get the reward.

[0062] Next, we need to design sub-goals, such as Figure 3 As shown in the figure, since the maze space is divided into three spaces by two walls, the agent must pass through the door on the wall to switch between the three spaces. If it passes through other positions of the wall, it will be trapped in place. The operator's arrival at the door position is an event that has a significant impact on the environment in the experiment. Therefore, this embodiment sets the sub-goal at the position of the two doors. When the agent reaches the door position, the abstract state changes. The sub-goal recognition function is designed by judging whether the operator's current position is the same as the sub-goal position.

[0063] Then, according to the prior knowledge, two sub-goals {sg1, sg2} are set at the doors on the two walls. After obtaining the sub-goals and their identification functions, the original specific state space can be reconstructed into an abstract state space of Z = {z1, z2, z3}. This abstract space has only three abstract states. The state before reaching sg1 is Z1, the state before reaching sg2 is Z2, and the state after reaching sg2 is Z3. The mapping process is as follows: Figure 4 shown.

[0064] From the above two examples, we can see that in step S100, the training sample set given has different ways of representing specific states depending on the application environment. When training the policy model, it is necessary to first build the application environment, list all the specific states of the agent in the environment, and give the environmental feedback reward corresponding to each state, that is, the feedback reward matrix.

[0065] Usually in reinforcement learning, the environment feedback reward is set to be given only when the agent reaches the goal, which leads to the problem of sparse rewards, which is not conducive to the convergence of the model. In some more complex environments, the number of specific states may be huge, resulting in a lot of time spent on training such a model. In order to solve this problem, Figure 5 As shown, in this method, the optimal strategy model includes two layers of structures based on the Markov decision process. One layer of structure shapes rewards by solving the value function of the abstract state, which is called the reward function online shaping model in this method. The other layer of structure is used to select and execute actions, and in this layer of structure, the reward function output by the reward function online shaping model is used to densify the reward and accelerate the convergence speed. In this method, this structure is called the strategy model. In this way, the combination of online reward shaping and offline reinforcement learning can reduce the correlation of samples and reduce the impact of the changing reward function on the specific reinforcement learning of the lower layer. The algorithm framework is shown in the figure. In the reward function online shaping model, the state value function can be calculated using the tabular reinforcement learning method; the state value function can also be calculated using the deep reinforcement learning method, which depends on the size of the abstract state space. In this article, the tabular reinforcement learning method is introduced. Next, the two layers of structure are introduced in detail.

[0066] When humans learn a complex task, they often cannot directly obtain the desired results through simple one or two steps, so people will first set some simple sub-goals, and eventually learn to solve a complex task by solving a series of simple sub-goals. Take the example of humans operating a robot arm to grab a product. First, the robot arm must be aimed at the product, then the robot arm must be moved to a position where the product can be grabbed in the direction of the aiming direction, and then the command is pressed to close the robot arm to clamp the product. In such a process, the sub-goal of aiming at the product appears first, and then the sub-goal of moving to a fixed position appears. After solving these two sub-goals, the task of grabbing the product can be successfully achieved.

[0067] Based on the above inspiration, in the online modeling of the reward function, the target task is divided into multiple sub-goals based on prior inspiration, that is, by using the prior knowledge of the target environment to find significant events in the environment as sub-goals. The definition of a sub-goal here is to obtain the event with the most significant impact on the entire environment based on prior knowledge. In this embodiment, the prior knowledge can be given by professionals who understand the application environment.

[0068] In this embodiment, the reward function divides the target task into multiple sub-goals according to the prior knowledge inspiration in the online shaping model, and maps multiple specific states into multiple abstract states according to each of the sub-goals, including: designing a state abstraction function according to each sub-goal, and mapping the specific state experienced by the agent before reaching the sub-goal into a corresponding abstract state according to the state abstraction function. Each abstract state contains multiple specific states.

[0069] Each sub-goal is also a specific state for the intelligent agent, so in the model, each sub-goal is represented by the corresponding specific state.

[0070] Specifically, after determining the sub-goal sequence {sg0,sg1···sg n}, a total of n sub-goals are obtained, and the state abstract function Z = f(s) is designed based on the sub-goals. The abstract function converts the agent into a state when it reaches the sub-goal sg i Specific states of previous experience i Mapped to abstract state z i . Usually reaches the sub-goal sg i The specific state s experienced by the previous agent i There will be many, and here they are defined as a set S i , where s i ∈S i . S i The specific states in S are often related to the strategy and environment transition probability. During the initial training, due to the uncertainty of the agent and the environment, S iThe elements in will change. The agent learns through continuous trial and error. After the learned strategy gradually converges, the specific state set S that it experiences before reaching the sub-goal is i According to the above state aggregation method, a new abstract state space z is obtained through the mapping function on the original specific state space i ∈Z, such as Figure 6 Shown is a state map.

[0071] After setting n sub-goals based on prior knowledge, we can determine that the number of abstract states is n+1. Before the agent reaches sub-goal 1, it is in abstract state z0. When it reaches the second sub-goal, the state of the agent changes from z0 to z1. And so on. When it reaches the ith sub-goal, the state of the agent changes from z0 to z1. i-1 Transform to z i , which actually constructs a new abstract Markov decision model In the model, Z is a set of abstract states, O is a set of abstract actions, and the selected abstract action o i Contains multiple specific actions a. is the transition probability between abstract states, It is the sum of the rewards obtained from multiple specific states when transferring from one abstract state to another, as shown in the formula:

[0072]

[0073] In formula (1), k is the number of specific states that the agent passes through before reaching the next sub-goal, and the function accumulates rewards with a discount factor γ.

[0074] Next, we define the value function of the new abstract Markov decision (MDP) model. The value function is a function of the state (or state and action tuple) that is used to evaluate how good the current agent is in a given state. It is defined by the expected value of the reward G. The reward G here is the reward of all state transitions in the process of transitioning from the initial abstract state to the final state in the abstract MDP model. The sum of .

[0075]

[0076] The expected reward that the agent can get depends on the action chosen by the agent, so the value function is related to a specific behavior, called a strategy. In the training of this layer model, that is, the reward function online shaping model, the strategy adopted is obtained by training another layer of strategy model. The initial strategy is a random strategy. When the agent is in an abstract state z, it takes N actions {a1, a2, ···, a n}, and then the state transitions to z', then the action set {a1,a2,···,a n} constitutes an abstract action o i ,As the specific neural network training progresses, the strategy gradually improves, and the value function learned by the ,reward function online shaping model becomes closer to the optimal value.

[0077] The value function formula is as follows:

[0078]

[0079] The value function is recursive and can be used to get from the abstract state z i to z i+1 The N-step temporal difference learning algorithm between is estimated from experience, and the formula is as follows:

[0080]

[0081] When the abstract Markov decision process has accumulated a certain amount of experience, the value function V(Z) of the abstract state under the current strategy can be evaluated according to Formula 4.

[0082] V(Z) calculated using the abstract state Markov decision model can be used as a whole specific state set S i The state potential energy is:

[0083] φ(s)=V(g(s))=V(Z) (5)

[0084] In formula (5), g represents the aggregation function that maps from the specific state set to the abstract state, which is obtained by learning.

[0085] Use the learned potential energy function to shape the reward function, as shown in the formula:

[0086] F(z t ,z t+1 )=γV(z t+1 )-V(z t ) (6)

[0087] It can be seen from formula (4) that the value function of each abstract state is only related to the abstract state transition reward. Therefore, the online reward function obtained according to each abstract state includes: calculating the abstract state transition reward according to the sum of the environmental feedback rewards obtained from multiple specific states when transferring from one abstract state to another; calculating the value function of the corresponding abstract state according to the abstract state transition reward (using formula (4)), and calculating the online reward function according to the value function (using formula (6)).

[0088] In this embodiment, training the policy model includes:

[0089] A set of specific states and corresponding environmental feedback rewards in the training sample set are input into the online shaping model of the reward function and the decision model respectively. In the online shaping model of the reward function, the environmental feedback reward is accumulated into the abstract state migration reward, and then the specific state is mapped into the corresponding abstract state according to the abstract function, and it is judged whether the abstract state is equal to the abstract state of the previous step. If they are not equal, it means that the current abstract state has migrated, and the value function of the current abstract state is updated using the abstract state migration reward. If they are equal, it means that the current abstract state has not migrated, and the value function is not updated. The online reward function is calculated using the value function, and the online reward function is provided to the decision model. In the decision model, the next action is selected and executed according to the specific state, the online reward function and the environmental feedback reward, and fed back to the environment. The environment gives a new specific state and the corresponding environmental feedback reward according to the action output by the decision model, so that the online shaping model of the reward function and the decision model can be trained for a new round.

[0090] Specifically, before training the online reward function shaping model, the value function corresponding to each abstract state is initialized and assigned an initial value of "0", as shown in Table 1. After completing the training of the online reward function shaping model, the value function corresponding to each abstract state is a fixed value.

[0091] Table 1 Initial abstract state value function table

[0092]

[0093] Specifically, when training the policy model, the agent obtains the specific state S from the environment t , and environmental feedback reward r t They are input into the online reward function shaping model and the decision model respectively. In the online reward function shaping model, the reward r t Accumulated to abstract state transition reward Then, we use the abstract function Z t =f(S t ) The specific state S t Mapped into abstract state Z t . Determine the current state Z t Is it equal to the state Z of the previous step? t-1 , if they are not equal, it proves that the abstract state has been transferred, using the reward Update the state value function.

[0094]

[0095] If Z tand Z t-1 If they are equal, the state value function update step is skipped. Then the reward function F(z t-1 ,z t ). Combine it with the environmental feedback reward r t The sum is used for training the decision model.

[0096] At the same time, the decision model obtains the environmental state S t The decision model includes the decision network and the playback data set, and the obtained environment state S t Input decision network is used to select action a t And execute, and feed back to the environment, so as to obtain a new environment state S t+1 and environmental rewards t+1 . The above transfer information <S t ,a t ,r t+1 +F(z t ,z t+1 ),S t+1 > Store in the replay buffer. When the data storage in the replay buffer reaches a certain amount, a batch of data is extracted from it to update the network parameters of the policy model.

[0097] In this embodiment, the reward function online shaping model and the decision model are trained simultaneously. The strategy used in the reward function online shaping model is provided by the decision model. When the decision model learns through rewards, the rewards include the online reward function provided by the reward function online shaping model and the environment feedback reward, which further densifies the rewards in the decision model, accelerates its convergence, and improves training efficiency.

[0098] In this embodiment, in the decision model, the decision network uses a deep Q learning network. In this way, in the optimal strategy model, an algorithm named DQN-KRS can be used, and its algorithm pseudo code is as follows:

[0099]

[0100]

[0101] In step S120, after the strategy model is trained, it is only necessary to input a specific state in the corresponding environment into it, and the optimal strategy for the agent to reach the goal from this state can be obtained. For different environmental backgrounds, if it is an application scenario of a robotic arm, the current state of the robotic arm (the state includes the position coordinates and the current angle of the arm) is input into the model so that the robotic arm can grab the target object with the shortest movement trajectory. In the application scenario of maze navigation, the current specific state of the chess piece (that is, the position of the chess piece) is input into the model, and the optimal route for the chess piece to reach the exit of the maze can be obtained.

[0102] The above-mentioned optimal strategy acquisition method based on the double-layer deep reinforcement learning model sets a double-layer deep reinforcement learning model in the strategy model, which includes a reward function online shaping model for dividing the target into multiple sub-goals according to prior knowledge inspiration, mapping multiple specific states of the target task into multiple abstract states according to each sub-goal, and calculating the online reward function through each abstract state, and a decision model for making action decisions according to each specific state. When the decision model is trained, the online reward function provided by the reward function online shaping model is used to densify its reward, thereby accelerating the model convergence speed and improving the model training efficiency.

[0103] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0104] In one embodiment, Figure 7 As shown, an optimal strategy acquisition device based on a two-layer deep reinforcement learning model is provided, comprising: a training sample set acquisition module 200, an optimal strategy model training module 210 and an optimal strategy acquisition module 220, wherein:

[0105] The training sample set acquisition module 200 is used to acquire a training sample set, wherein the training sample set includes multiple specific states related to a target task, actions performed by the agent in the specific state, new specific states to which the environment migrates after performing the action, and environmental feedback rewards corresponding to each of the states;

[0106] The optimal strategy model training module 210 is used to input the training sample set into the strategy model for training until the strategy model has the ability to output the optimal strategy for completing the target task, thereby obtaining the trained optimal strategy model; wherein the optimal strategy model includes a reward function online shaping model and a decision model both of which are built based on the Markov decision process, in which the target task is divided into a plurality of sub-goals according to prior knowledge inspiration, and a plurality of specific states are mapped into a plurality of abstract states according to each of the sub-goals, and then an online reward function is obtained according to each of the abstract states for use by the decision model, and the decision model includes a decision network, and the decision network is trained according to the environmental feedback reward and the online reward function so that it outputs the optimal strategy;

[0107] The optimal strategy acquisition module 220 is used to acquire any specific state in the target task, and input the specific state into the trained optimal strategy model to obtain the optimal strategy for completing the target task with the specific state as the starting point.

[0108] For the specific definition of the optimal strategy acquisition device based on the double-layer deep reinforcement learning model, please refer to the definition of the optimal strategy acquisition method based on the double-layer deep reinforcement learning model mentioned above, which will not be repeated here. Each module in the above-mentioned optimal strategy acquisition device based on the double-layer deep reinforcement learning model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0109] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an optimal strategy acquisition method based on a double-layer deep reinforcement learning model is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0110] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0111] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0112] Obtaining a training sample set, wherein the training sample set includes multiple specific states related to a target task, actions performed by the agent in the specific states, new specific states to which the environment migrates after performing the actions, and environmental feedback rewards corresponding to each of the states;

[0113] Input the training sample set into the strategy model for training until the strategy model has the ability to output the optimal strategy for completing the target task, and obtain the trained optimal strategy model; wherein the optimal strategy model includes a reward function online shaping model and a decision model both of which are built based on the Markov decision process, in which the target task is divided into multiple sub-goals according to prior knowledge inspiration, and multiple specific states are mapped into multiple abstract states according to each of the sub-goals, and then an online reward function is obtained according to each abstract state shaping for use by the decision model, and the decision model includes a decision network, and the decision network is trained according to the environmental feedback reward and the online reward function so that it outputs the optimal strategy;

[0114] Obtain any specific state in the target task, and input the specific state into the trained optimal strategy model to obtain the optimal strategy for completing the target task with the specific state as the starting point.

[0115] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0116] Obtaining a training sample set, wherein the training sample set includes multiple specific states related to a target task, actions performed by the agent in the specific states, new specific states to which the environment migrates after performing the actions, and environmental feedback rewards corresponding to each of the states;

[0117] Input the training sample set into the strategy model for training until the strategy model has the ability to output the optimal strategy for completing the target task, and obtain the trained optimal strategy model; wherein the optimal strategy model includes a reward function online shaping model and a decision model both of which are built based on the Markov decision process, in which the target task is divided into multiple sub-goals according to prior knowledge inspiration, and multiple specific states are mapped into multiple abstract states according to each of the sub-goals, and then an online reward function is obtained according to each abstract state shaping for use by the decision model, and the decision model includes a decision network, and the decision network is trained according to the environmental feedback reward and the online reward function so that it outputs the optimal strategy;

[0118] Obtain any specific state in the target task, and input the specific state into the trained optimal strategy model to obtain the optimal strategy for completing the target task with the specific state as the starting point.

[0119] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0120] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. The optimal strategy acquisition method based on the two-layer deep reinforcement learning model is characterized by: The method comprises: Acquire a training sample set, wherein the training sample set includes multiple specific states related to a target task, actions performed by the agent in the specific state, new specific states to which the environment migrates after performing the actions, and environmental feedback rewards corresponding to each of the states, wherein the target task includes a target task with engineering machinery as the application environment and a target task with maze navigation as the application environment; The training sample set is input into the strategy model for training until the strategy model has the ability to output the optimal strategy to complete the target task, and the trained optimal strategy model is obtained; wherein, the optimal strategy model includes a reward function online shaping model and a decision model both of which are built based on the Markov decision process, in which the target task is divided into multiple sub-goals according to prior knowledge inspiration, and multiple specific states are mapped into multiple abstract states according to each sub-goal, and then an online reward function is shaped according to each abstract state for use by the decision model, the decision model includes a decision network, and the decision network is trained according to the environmental feedback reward and the online reward function so that it outputs the optimal strategy, wherein in the reward function online shaping model, the online reward function is shaped according to each abstract state, including: calculating the abstract state transition reward according to the sum of the environmental feedback rewards obtained from multiple specific states experienced when transferring from one abstract state to another, calculating the value function of the corresponding abstract state according to the abstract state transition reward, and calculating the online reward function according to the value function Further, when training the strategy model, it includes: inputting a group of specific states and corresponding environmental feedback rewards in the training sample set into the reward function online shaping model and the decision model respectively, in the reward function online shaping model, accumulating the environmental feedback rewards into the abstract state migration rewards, and then mapping the specific state into the corresponding abstract state according to the abstract function, and judging whether the abstract state is equal to the abstract state of the previous step, if not equal, it means that the current abstract state has migrated, and updating the value function of the current abstract state by using the abstract state migration reward, if equal, it means that the current abstract state has not migrated, then the value function is not updated, and the online reward function is calculated by using the value function, and the online reward function is provided to the decision model, in the decision model, the next action is selected and executed according to the specific state, the online reward function and the environmental feedback reward, and fed back to the environment, the environment gives a new specific state and the corresponding environmental feedback reward according to the action output by the decision model, so that the reward function online shaping model and the decision model undergo a new round of training; Obtain any specific state in the target task, and input the specific state into the trained optimal strategy model to obtain the optimal strategy for completing the target task with the specific state as the starting point.

2. The optimal strategy acquisition method according to claim 1, characterized in that: In the online shaping model of the reward function, the target task is divided into a plurality of sub-goals according to prior knowledge inspiration, and a plurality of specific states are mapped into a plurality of abstract states according to each of the sub-goals, including: Designing a state abstraction function according to each of the sub-goals, and mapping the specific state experienced by the agent before reaching the sub-goal into a corresponding abstract state according to the state abstraction function; Each of the abstract states includes multiple specific states.

3. The optimal strategy acquisition method according to claim 2, characterized in that: Before training the online reward function shaping model, the value functions corresponding to the abstract states are initialized and assigned initial values; After completing the training of the online modeling model of the reward function, the value function corresponding to each of the abstract states is obtained as a fixed value.

4. The optimal strategy acquisition method according to claim 3, characterized in that: The decision model also includes a playback data set. During each training, the transfer information is stored in the playback data set. When the data in the playback data set reaches a preset amount, a portion of the data is extracted from the playback data set to update the network parameters of the decision model. The transfer information includes the specific state of the current training, the output action, the sum of the reward and the online reward function based on the specific state and the output action environment feedback, and the specific state of the next step.

5. The optimal strategy acquisition method according to claim 4, characterized in that: In the decision model, the decision network adopts a deep Q learning network.

6. An optimal strategy acquisition device based on a two-layer deep reinforcement learning model, characterized in that: The device comprises: A training sample set acquisition module, used to acquire a training sample set, wherein the training sample set includes a plurality of specific states related to a target task, and an environmental feedback reward corresponding to each of the states; The optimal strategy model training module is used to input the training sample set into the strategy model for training until the strategy model has the ability to output the optimal strategy for completing the target task, thereby obtaining the trained optimal strategy model; wherein the optimal strategy model includes a reward function online shaping model and a decision model both of which are built based on the Markov decision process, in which the target task is divided into multiple sub-goals according to prior knowledge inspiration, and multiple specific states are mapped into multiple abstract states according to each of the sub-goals, and then an online reward function is shaped according to each abstract state for use by the decision model, the decision model includes a decision network, and the decision network is trained according to the environmental feedback reward and the online reward function so that it outputs the optimal strategy, wherein the online reward function is shaped according to each abstract state in the reward function online shaping model, including: calculating an abstract state transition reward according to the sum of the environmental feedback rewards obtained from multiple specific states experienced when transferring from one abstract state to another, calculating a value function of the corresponding abstract state according to the abstract state transition reward, and calculating the value function according to the value function. The online reward function, further, when training the strategy model, includes: inputting a group of specific states in the training sample set and the corresponding environmental feedback rewards into the online shaping model of the reward function and the decision model respectively; in the online shaping model of the reward function, the environmental feedback rewards are accumulated into the abstract state migration rewards; and then the specific states are mapped into the corresponding abstract states according to the abstract function, and it is determined whether the abstract state is equal to the abstract state of the previous step; if they are not equal, it means that the current abstract state has migrated; the value function of the current abstract state is updated by using the abstract state migration rewards; if they are equal, it means that the current abstract state has not migrated, and the value function is not updated; the online reward function is calculated by using the value function, and the online reward function is provided to the decision model; in the decision model, the next action is selected and executed according to the specific state, the online reward function and the environmental feedback rewards, and the feedback is given to the environment; the environment gives a new specific state and the corresponding environmental feedback rewards according to the action output by the decision model, so that the online shaping model of the reward function and the decision model are trained for a new round; The optimal strategy acquisition module is used to acquire any specific state in the target task and input the specific state into the trained optimal strategy model to obtain the optimal strategy for completing the target task with the specific state as the starting point.

Images