Multi-style highly controllable strategy learning method and system based on reinforcement learning

Abstract

The invention provides a multi-style highly controllable strategy learning method and system based on reinforcement learning, and the method comprises the steps: obtaining the meta-behaviors of an intelligent agent in an environment, carrying out the reward shaping of each meta-behavior, each meta-behavior comprises a corresponding style parameter, and the style parameter controls the reward scale after completing the meta-behaviors; in the interaction process of the agent and the environment, according to the number of meta behaviors, randomly generating a group of style parameters, sending a style parameter and state combination into a model of the agent for reasoning to obtain an action, and after the action is fed back to the environment, generating an award obtained by the agent at the current moment and a state at the next moment; and carrying out reinforcement learning on style parameters, states, actions and rewards in the interaction process of the intelligent agent and the environment to obtain a multi-style high-controllability strategy model. According to the method, a single reinforcement learning model can have strategies of multiple styles at the same time, the strategies of the model can be controlled by adjusting the parameters of the multiple styles, and the controllability of the strategies is improved.

Description

Technical Field

[0001] The present invention relates to the field of AI technology, and in particular to a multi-style highly controllable strategy learning method and system based on reinforcement learning. Background Art

[0002] The AI ​​strategy based on reinforcement learning technology has a high upper limit in terms of capabilities, but has a single mode and is difficult to control in terms of style during reasoning. This results in poor generalization of AI strategies based on reinforcement learning, and the inability to have diverse strategies at the same time. It is easy to be discovered and targeted for weaknesses, making it difficult to adapt to different application scenarios.

[0003] If multiple AI strategies are required, the existing technical solution is to first define the required strategies, then design the reward shaping schemes corresponding to the strategies, and then train different reward shaping schemes based on reinforcement learning methods to obtain different models, and then deploy the corresponding models according to the specific business scenarios.

[0004] Patent document CN115430154A discloses a method and system for training game AI for different players, which constructs the game AI's response strategy, sampling strategy and reward model according to the opponent type M. Game AI uses sampling strategy to interact with opponents and collects interaction data of various types of opponents; step S3: use interaction data to calculate M opponent reward models. Use the maximum and minimum Q learning algorithm, game model and reward model to learn the Nash equilibrium strategy of the mth type of opponent as a response strategy, and repeat the learning to obtain the response strategy of all types of opponents. This scheme requires a detailed design of the reward shaping scheme required for the strategy, and the process of designing the reward shaping scheme is relatively difficult, because inappropriate reward shaping process may cause the intelligent agent to learn the wrong strategy or behavior. Moreover, even if all reward shaping processes are designed correctly and trained, the large number of models obtained need to be accurately delivered for different business scenarios. This greatly increases the difficulty of training, the complexity of the business and the maintenance cost, and reduces the reliability of the strategy. Summary of the invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a multi-style highly controllable strategy learning method and system based on reinforcement learning.

[0006] A multi-style highly controllable strategy learning method based on reinforcement learning provided by the present invention includes: Meta-behavior acquisition step: acquiring the meta-behavior of the agent in the environment, and shaping rewards for each meta-behavior. Each meta-behavior includes a corresponding style parameter, and the style parameter controls the reward scale after completing these meta-behaviors. Style generation step: During the interaction between the agent and the environment, a set of style parameters is randomly generated according to the number of meta-behaviors. The style parameters and state combination are sent to the agent's model for reasoning to obtain actions. After the actions are fed back to the environment, the reward obtained by the agent at the current moment and the state at the next moment are generated. Reinforcement learning steps: Perform reinforcement learning on the style parameters, states, actions, and rewards during the interaction between the agent and the environment to obtain a multi-style highly controllable strategy model.

[0007] Preferably, the meta-behavior includes the basic actions of the agent, and the reward for the agent to complete the meta-behavior = the reward for completing the first n Basic rewards for individual actions a n ×No. n Style parameters for meta-behaviors w n .

[0008] Preferably, the range of the style parameter includes [-1, 1]; When the style parameter is -1, it means that executing this meta-behavior will get the opposite reward, thus forming the opposite tendency; When the style parameter is 0, it means that there is no reward for performing the meta-behavior, thus indicating that the performance of the behavior is neither encouraged nor supported; When the style parameter is 1, it indicates that the reward for this meta-behavior is positively increased to the maximum extent.

[0009] Preferably, in the reinforcement learning step, the style parameters are encoded by a style encoder to obtain style features, which are combined with the features of the state and sent to a training network for learning to generate action probabilities and values; When the style parameters change, the characteristics of the state do not change, and the change in strategy is only fitted by the subsequent training network parameters.

[0010] Preferably, the multi-style highly controllable strategy model adopts a proximal strategy optimization algorithm to limit the degree of strategy change by limiting the action probability ratio between the current strategy and the old strategy. The action probability ratio between the current strategy and the old strategy is as follows:

[0011] in s t 、c t 、a t Represents the time step t The following states, style parameters and actions; and The current strategy and the old strategy are in states t and style parameters c t Take action a t The probability of; the restricted ratio is ,in ε is a hyperparameter that controls the degree of restriction; The policy objective function is defined as:

[0012] in represents the expectation of a sample batch, A t is the time step calculated by the generalized advantage estimation algorithm t The advantage estimate of The value function objective is defined as:

[0013] Among them, the target return , is the value function of the current strategy, is the value function of the old strategy; Through continuous optimization and , let the training network be in different states s and style parameters c There is an optimal strategy under all the above conditions, and different style parameters and corresponding rewards are fitted by the value function.

[0014] A multi-style highly controllable strategy learning system based on reinforcement learning provided by the present invention includes: Meta-behavior acquisition module: obtains the meta-behavior of the agent in the environment and shapes the reward for each meta-behavior. Each meta-behavior includes a corresponding style parameter, which controls the reward scale after completing these meta-behaviors. Style generation module: In the process of interaction between the agent and the environment, a set of style parameters is randomly generated according to the number of meta-behaviors. The style parameters and state combination are sent to the model of the agent for inference to obtain actions. After the actions are fed back to the environment, the reward obtained by the agent at the current moment and the state at the next moment are generated; Reinforcement learning module: Perform reinforcement learning on the style parameters, states, actions, and rewards during the interaction between the agent and the environment to obtain a multi-style highly controllable strategy model.

[0015] Preferably, the meta-behavior includes the basic actions of the agent, and the reward for the agent to complete the meta-behavior = the reward for completing the first n Basic rewards for individual actions an ×No. n Style parameters for meta-behaviors w n .

[0016] Preferably, the range of the style parameter includes [-1, 1]; When the style parameter is -1, it means that executing this meta-behavior will get the opposite reward, thus forming the opposite tendency; When the style parameter is 0, it means that there is no reward for performing the meta-behavior, thus indicating that the performance of the behavior is neither encouraged nor supported; When the style parameter is 1, it indicates that the reward for this meta-behavior is positively increased to the maximum extent.

[0017] Preferably, in the reinforcement learning module, the style parameters are encoded by a style encoder to obtain style features, which are combined with the features of the state and sent to a training network for learning to generate action probabilities and values; When the style parameters change, the characteristics of the state do not change, and the change in strategy is only fitted by the subsequent training network parameters.

[0018] Preferably, the multi-style highly controllable strategy model adopts a proximal strategy optimization algorithm to limit the degree of strategy change by limiting the action probability ratio between the current strategy and the old strategy. The action probability ratio between the current strategy and the old strategy is as follows:

[0019] in s t 、c t 、a t Represents the time step t The following states, style parameters and actions; and The current strategy and the old strategy are in state s t and style parameters c t Take action a t The probability of; the restricted ratio is ,in ε is a hyperparameter that controls the degree of restriction; The policy objective function is defined as:

[0020] in represents the expectation of a sample batch, A tis the time step calculated by the generalized advantage estimation algorithm t The advantage estimate of The value function objective is defined as:

[0021] Among them, the target return , is the value function of the current strategy, is the value function of the old strategy; Through continuous optimization and , let the training network be in different states s and style parameters c There is an optimal strategy under all the above conditions, and different style parameters and corresponding rewards are fitted by the value function.

[0022] Compared with the prior art, the present invention has the following beneficial effects: The present invention can enable a single reinforcement learning model to have diversified strategies by arbitrarily matching style parameters, and can set different style parameters for delivery according to specific application scenarios, which greatly increases the generalization and practicality of the model.

[0023] The multi-style parameters of the present invention can be changed in real time during reasoning, so the model obtained by the multi-style learning method also has the characteristics of high reliability, can adapt to changes in scenes and make changes in strategies at any time, and reduces the training cost and human maintenance cost of the reinforcement learning model.

[0024] The multi-style training method of the present invention has higher efficiency, which is also helpful in reducing training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 It is a schematic diagram of the overall framework of the present invention; Figure 2 Schematic diagram for meta-behavior and reward shaping; Figure 3 Schematic diagram of the style generator; Figure 4 Diagram of a network training for a multi-style strategy; Figure 5 Schematic diagram of the multi-style strategy screening solution. DETAILED DESCRIPTION

[0026] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0027] Reinforcement learning (RL): Reinforcement learning is a machine learning method that allows an agent to learn how to make optimal decisions by interacting with the environment. The goal of reinforcement learning is to find a strategy that allows the agent to obtain the maximum cumulative reward within a certain period of time.

[0028] Training environment: Or simply environment, in reinforcement learning, the environment is the world that the agent interacts with. The environment receives the agent's actions and returns new states and rewards based on these actions. The environment can be any system that the agent can interact with, such as a game, the physical world of a robot, or even a stock market model.

[0029] Agent: In reinforcement learning, an agent is an entity that can perceive the environment and perform actions in the environment. The goal of the agent is to maximize the total reward by learning how to choose actions.

[0030] State: A state is a description of the environment at a certain moment in time, which contains all the necessary information that the agent needs to make a decision. For example, in a board game, the state may include the type of chess piece at each position on the board.

[0031] Action: An action is a basic operation that an agent can perform in a given state. An agent interacts with the environment by performing actions. For example, if the agent is a character in a game, possible actions include moving left, moving right, jumping, etc.

[0032] Reward: Reward is the feedback from the environment to the agent's actions. When the agent performs an action, the environment returns a reward to tell the agent whether the action is good or bad. Rewards can be positive (indicating encouragement of the action) or negative (indicating a desire to inhibit the action). The goal of the agent is to maximize the total reward by learning how to choose actions.

[0033] Training epoch: The training environment does not run indefinitely, but starts and ends in units of training epochs. At the beginning of each training epoch, the environment will be initialized to a certain state; at the end of each training epoch, done will be used as a signal.

[0034] Reward shaping: Also known as reward setting, reward shaping is a reinforcement learning technique that guides the agent to learn faster by designing the reward function.

[0035] Meta-behavior: The main behavior of the agent in the training environment. Usually, most of the agent's behaviors in the environment can be composed of meta-behaviors. For example, the behavior of the agent jumping left can be completed by the combination of the two meta-behaviors of the agent moving left and jumping.

[0036] like Figure 1 As shown, the present invention provides a multi-style highly controllable strategy learning method based on reinforcement learning, comprising: Meta-behavior acquisition step: obtain the meta-behavior of the agent in the environment, and shape the reward for each meta-behavior. Each meta-behavior includes a corresponding style parameter, and the style parameter controls the reward scale after completing these meta-behaviors.

[0037] In the present invention, the behaviors of the agent to be trained in the environment are first summarized, and a series of meta-behaviors are designed. These meta-behaviors can be completed by the basic actions of the agent. Then, rewards are shaped for each meta-behavior, that is, a reward value will be obtained after completing the meta-behavior. Each meta-behavior has a corresponding style parameter, which controls the reward scale after completing these meta-behaviors, so each style parameter can represent the tendency of the corresponding meta-behavior.

[0038] Style generation step: During the interaction between the agent and the environment, a set of style parameters is randomly generated according to the number of meta-behaviors. The style parameters and state combination are sent to the agent's model for inference to obtain actions. After the actions are fed back to the environment, the reward obtained by the agent at the current moment and the state at the next moment are generated.

[0039] The inclinations of different meta-behaviors of the agent can freely combine into a large number of strategic styles. In the process of interaction between the agent and the environment, the style generator will randomly generate a set of style parameters according to the number of meta-behaviors. This style parameter will be combined with the state and then sent to the agent's model for reasoning to obtain the action. After the action is fed back to the environment, the reward that the agent can obtain at the current moment and the state at the next moment are generated. In this process, the reward will be affected by multiple style parameters.

[0040] Reinforcement learning steps: Perform reinforcement learning on the style parameters, states, actions, and rewards in the process of interaction between the agent and the environment, fit multiple style parameters and meta-behavioral tendencies, and finally train a multi-style, highly controllable strategy model.

[0041] Different multi-style parameter settings can form multi-style strategies, so that a single reinforcement learning model can have multiple styles of strategies at the same time, which increases the generalization of the model and reduces the redundancy of the model. By adjusting the multi-style parameters, the model's strategy can be controlled, such as controlling the style of the current strategy, the timing of style switching, etc., which increases the controllability of the strategy and reduces the complexity of the business. With the help of data statistics, multi-style parameter configurations that can meet specific business needs can be automatically selected, reducing manpower investment.

[0042] The following is an example of a basketball game: We first need to design a series of meta-behaviors for the agent in this environment. The principles for designing meta-behaviors are: (1) The correlation between meta-behaviors should be as small as possible; (2) By combining meta-behaviors, we can cover as many behaviors as possible that the agent can perform in the environment; (3) Under the above conditions, the fewer the number of meta-behaviors, the better.

[0043] This is to make the subsequent process simpler and greatly increase the success rate of multi-style strategy training. Then we need to shape the reward for each meta-behavior, that is, how much reward will be obtained for completing such a meta-behavior. The process is as follows: Figure 2 As shown. a n Indicates completion of n The basic reward for each behavior. w n Indicates the style parameter corresponding to the meta-behavior. The basic reward and the style parameter are multiplied to get the final reward of the meta-behavior. Meta-behaviors can be combined to form a variety of behaviors. For example, "two-point shooting" and "area 1 activities" can be combined to form the behavior of "making two-point shooting in area 1". Through the adjustment of style parameters, the tendency of these behavior combinations also has different degrees, thus forming a variety of strategic styles.

[0044] During the reinforcement learning phase, the environment will interact with the training end to continuously generate data for the latest strategy. At the beginning of each training period of the environment, a style generator will generate style parameters used in that training period. The style generator can generate style parameters that correspond to the reward configuration one by one. The style parameters can be multiplied by the corresponding rewards to control the value of the reward obtained when the strategy performs the behavior during training. The range of style parameters can usually be set to [-1, 1]. When the style parameter is -1, it means that the strategy will receive the opposite reward for executing the behavior, thus forming an opposite tendency; when the style parameter is 0, it means that no reward will be obtained for executing the line, which means that the behavior is neither encouraged nor supported; when the style parameter is 1, it means that the reward for the behavior is positively increased to the greatest extent. Style parameters can be generated by random sampling or by setting specific rules, such as Figure 3 As shown. For example, you can randomly generate any value between -1 and 1, or generate it at intervals of 0.1, or only generate values ​​between 0 and 1, etc. Through the style generator, the reinforcement learning strategy can obtain data under different reward settings, thus providing a basis for the subsequent reinforcement learning algorithm to fit the optimal strategy under different style parameters.

[0045] The generated style parameters are encoded by a style encoder during training to obtain style features, which are then merged with the features of the original input state and sent to the subsequent network for learning. This process will eventually generate action probabilities and values. When the style parameters change, the features of the state do not change, and the changes in the strategy are only fitted by the subsequent network parameters, which is conducive to the learning of state and style parameters. This process is as follows Figure 4 shown.

[0046] In order to efficiently train models with multi-style policies, we use the Proximal Policy Optimization (PPO) algorithm as our main algorithm. PPO is a popular policy gradient algorithm that performs well in various tasks. The PPO algorithm limits the degree of policy change by limiting the action probability ratio between the current policy and the old policy. The action probability ratio between the current policy and the old policy is as follows:

[0047] in s t 、c t 、a t Represents the time step t The following states, style parameters and actions; and The current strategy and the old strategy are in state s t and style parameters c t Take action a t The probability of; the restricted ratio is ,in ε is a hyperparameter used to control the degree of restriction. This restriction can prevent the strategy from fluctuating greatly, thus avoiding instability in the training process.

[0048] The policy objective function is defined as:

[0049] in represents the expectation of a sample batch, A t is the time step calculated by the Generalized Advantage Estimator (GAE) t Advantage estimation; The value function objective is defined as:

[0050] Among them, the target return , is the value function of the current strategy, is the value function of the old strategy; Through continuous optimization and , let the training network be in different states s and style parameters c The optimal strategy is available under all situations. Different style parameters and corresponding rewards are fitted by the value function. The value function can guide the optimal behavior of the strategy under different styles, thus ensuring the convergence of the algorithm. At the same time, by setting different style parameters in the input to call different behavior styles, the controllability of the strategy is ensured.

[0051] During training, multiple style parameters can generally be generated randomly, thereby increasing the stability and generalization of the strategy. When in use, the desired style parameter combination can be set according to the specific scenario or business needs. The setting of style parameters can be heuristically set, or the desired strategy can be selected through data analysis methods. The heuristic method is relatively simple and can usually be configured based on the experience of algorithm engineers or planners, but it usually requires a lot of testing to get the desired strategy. The style parameters of the required strategy can be obtained more accurately through data analysis methods, which requires specifying the behavioral indicators of the desired strategy first. For example, in this application scenario, the behavioral indicators can be the average score, number of layups, number of three-pointers, number of blocks, etc. of the agent in each game. Then we traverse the style parameter combinations at a certain granularity and test them in the environment to obtain the statistical indicators of game behavior under different style parameter configurations, thereby obtaining the style parameters of the strategy that is closest to the target indicator. Taking the application in basketball games as an example, such as Figure 5 As shown in the figure, we can first randomly sample two sets of style parameters, and then let the model use these two style parameters to fight, get the statistical results of the battle, and send the results to the database. The database records the statistical indicators of all styles that appear, and then sets the values ​​of game indicators according to the needs of planning or scenarios, such as the average number of 5 three-pointers, 6 layups, etc. per game. Then, according to specific game indicators, the most matching style parameters are found in the database, and this style parameter is the target style parameter. In this way, different style parameters can be selected in a targeted manner. Since the style parameters can be changed at any time in the game, this method realizes real-time control of the strategy, greatly increasing the practicality and flexibility of AI Bot.

[0052] The present invention also provides a multi-style highly controllable strategy learning system based on reinforcement learning. The multi-style highly controllable strategy learning system based on reinforcement learning can be implemented by executing the process steps of the multi-style highly controllable strategy learning method based on reinforcement learning, that is, those skilled in the art can understand the multi-style highly controllable strategy learning method based on reinforcement learning as a preferred implementation mode of the multi-style highly controllable strategy learning system based on reinforcement learning.

[0053] The present invention provides a multi-style highly controllable strategy learning system based on reinforcement learning, comprising: Meta-behavior acquisition module: obtains the meta-behavior of the intelligent agent in the environment, and shapes the reward for each meta-behavior. Each meta-behavior includes a corresponding style parameter, and the style parameter controls the reward scale after completing these meta-behaviors.

[0054] Style generation module: During the interaction between the agent and the environment, a set of style parameters is randomly generated according to the number of meta-behaviors. The style parameters and state combination are sent to the agent's model for inference to obtain actions. After the actions are fed back to the environment, the reward obtained by the agent at the current moment and the state at the next moment are generated.

[0055] Reinforcement learning module: Perform reinforcement learning on the style parameters, states, actions, and rewards during the interaction between the agent and the environment to obtain a multi-style highly controllable strategy model.

[0056] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0057] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A multi-style highly controllable strategy learning method based on reinforcement learning, characterized in that: include: Meta-behavior acquisition step: acquiring the meta-behavior of the agent in the environment, and shaping rewards for each meta-behavior. Each meta-behavior includes a corresponding style parameter, and the style parameter controls the reward scale after completing these meta-behaviors. Style generation step: During the interaction between the agent and the environment, a set of style parameters is randomly generated according to the number of meta-behaviors. The style parameters and state combination are sent to the agent's model for reasoning to obtain actions. After the actions are fed back to the environment, the reward obtained by the agent at the current moment and the state at the next moment are generated. Reinforcement learning steps: Perform reinforcement learning on the style parameters, states, actions, and rewards during the interaction between the agent and the environment to obtain a multi-style highly controllable strategy model.

2. The multi-style highly controllable strategy learning method based on reinforcement learning according to claim 1 is characterized in that: The meta-behavior includes the basic actions of the agent, and the reward for the agent to complete the meta-behavior = the reward for completing the first n Basic rewards for individual actions a n ×No. n Style parameters for meta-behaviors w n .

3. The multi-style highly controllable strategy learning method based on reinforcement learning according to claim 1 is characterized in that: The range of the style parameter includes [-1, 1]; When the style parameter is -1, it means that executing this meta-behavior will get the opposite reward, thus forming the opposite tendency; When the style parameter is 0, it means that there is no reward for performing the meta-behavior, thus indicating that the performance of the behavior is neither encouraged nor supported; When the style parameter is 1, it indicates that the reward for this meta-behavior is positively increased to the maximum extent.

4. The multi-style highly controllable strategy learning method based on reinforcement learning according to claim 1 is characterized in that: In the reinforcement learning step, the style parameters are encoded by a style encoder to obtain style features, which are combined with the features of the state and sent to the training network for learning to generate action probabilities and values; When the style parameters change, the characteristics of the state do not change, and the change in strategy is only fitted by the subsequent training network parameters.

5. The multi-style highly controllable strategy learning method based on reinforcement learning according to claim 4 is characterized in that: The multi-style highly controllable strategy model adopts a proximal strategy optimization algorithm to limit the degree of strategy change by limiting the action probability ratio between the current strategy and the old strategy. The action probability ratio between the current strategy and the old strategy is as follows: in s t 、c t 、a t Represents the time step t The following states, style parameters and actions; and The current strategy and the old strategy are in state s t and style parameters c t Take action a t The probability of; the restricted ratio is ,in ε is a hyperparameter that controls the degree of restriction; The policy objective function is defined as: in represents the expectation of a sample batch, A t is the time step calculated by the generalized advantage estimation algorithm t The advantage estimate of The value function objective is defined as: Among them, the target return , is the value function of the current strategy, is the value function of the old strategy; Through continuous optimization and , let the training network be in different states s and style parameters c There is an optimal strategy under all the above conditions, and different style parameters and corresponding rewards are fitted by the value function.

6. A multi-style highly controllable strategy learning system based on reinforcement learning, characterized in that: include: Meta-behavior acquisition module: obtains the meta-behavior of the agent in the environment and shapes the reward for each meta-behavior. Each meta-behavior includes a corresponding style parameter, which controls the reward scale after completing these meta-behaviors. Style generation module: In the process of interaction between the agent and the environment, a set of style parameters is randomly generated according to the number of meta-behaviors. The style parameters and state combination are sent to the model of the agent for inference to obtain actions. After the actions are fed back to the environment, the reward obtained by the agent at the current moment and the state at the next moment are generated; Reinforcement learning module: Perform reinforcement learning on the style parameters, states, actions, and rewards during the interaction between the agent and the environment to obtain a multi-style highly controllable strategy model.

7. The multi-style highly controllable strategy learning system based on reinforcement learning according to claim 6 is characterized in that: The meta-behavior includes the basic actions of the agent, and the reward for the agent to complete the meta-behavior = the reward for completing the first n Basic rewards for individual actions a n ×No. n Style parameters for meta-behaviors w n .

8. The multi-style highly controllable strategy learning system based on reinforcement learning according to claim 6 is characterized in that: The range of the style parameter includes [-1, 1]; When the style parameter is -1, it means that executing this meta-behavior will get the opposite reward, thus forming the opposite tendency; When the style parameter is 0, it means that there is no reward for performing the meta-behavior, thus indicating that the performance of the behavior is neither encouraged nor supported; When the style parameter is 1, it indicates that the reward for this meta-behavior is positively increased to the maximum extent.

9. The multi-style highly controllable strategy learning system based on reinforcement learning according to claim 6, characterized in that: In the reinforcement learning module, the style parameters are encoded by the style encoder to obtain the style features, which are combined with the features of the state and sent to the training network for learning to generate action probabilities and values; When the style parameters change, the characteristics of the state do not change, and the change in strategy is only fitted by the subsequent training network parameters.

10. The multi-style highly controllable strategy learning system based on reinforcement learning according to claim 9, characterized in that: The multi-style highly controllable strategy model adopts a proximal strategy optimization algorithm to limit the degree of strategy change by limiting the action probability ratio between the current strategy and the old strategy. The action probability ratio between the current strategy and the old strategy is as follows: in s t 、c t 、a t Represents the time step t The following states, style parameters and actions; and The current strategy and the old strategy are in state s t and style parameters c t Take action a t The probability of; the restricted ratio is ,in ε is a hyperparameter that controls the degree of restriction; The policy objective function is defined as: in represents the expectation of a sample batch, A t is the time step calculated by the generalized advantage estimation algorithm t The advantage estimate of The value function objective is defined as: Among them, the target return , is the value function of the current strategy, is the value function of the old strategy; Through continuous optimization and , let the training network be in different states s and style parameters c There is an optimal strategy under all the above conditions, and different style parameters and corresponding rewards are fitted by the value function.

Images