Multi-style high-controllable policy learning method and system based on reinforcement learning

Abstract

The application provides a multi-style high-controllable policy learning method and system based on reinforcement learning, acquires meta behaviors of an agent in an environment, rewards and shapes each meta behavior, each meta behavior includes a corresponding style parameter, the style parameter controls the reward scale after completing the meta behaviors; in the process of interaction between the agent and the environment, a group of style parameters are randomly generated according to the number of meta behaviors, the style parameters and state combinations are sent into the model of the agent for reasoning to obtain actions, the actions are fed back to the environment, and the reward obtained by the agent at the current moment and the state at the next moment are generated; the style parameters, the state, the actions and the reward in the process of interaction between the agent and the environment are subjected to reinforcement learning, and a multi-style high-controllable policy model is obtained. The application can enable a single reinforcement learning model to have multiple style strategies at the same time, the strategies of the model can be controlled through adjustment of the multi-style parameters, and the controllability of the strategies is increased.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of AI, in particular to a multi-style high-controllable policy learning method and system based on reinforcement learning. BACKGROUND

[0002] The AI policy obtained based on the reinforcement learning technology has a high upper limit in terms of ability, but has the problems of single mode and difficulty in style control in reasoning. This leads to poor generalization of the AI policy based on reinforcement learning, and the policy cannot have diversity at the same time, is easy to be found and targeted, and is difficult to adapt to different application scenarios.

[0003] If multiple AI policies are needed, the prior art solution is to first define the required policy, then design a reward shaping scheme corresponding to the policy, then train different reward shaping schemes based on the reinforcement learning method, obtain different models, and then deploy the corresponding model according to the specific business scenario.

[0004] Patent document CN115430154A discloses a game AI training method and system for different players, which constructs a response strategy, a sampling strategy and a reward model of the game AI according to the opponent type M. The game AI interacts with the opponent using the sampling strategy and collects interactive data of various types of opponents. Step S3: Calculate the M opponent reward models using the interactive data. Use the max-min Q learning algorithm, the game model and the reward model to learn the Nash equilibrium strategy of the mth type of opponent as the response strategy, and repeatedly learn to obtain the response strategy of all types of opponents. This solution needs to design a reward shaping scheme required by the strategy in detail, and the process of designing the reward shaping scheme is relatively difficult, because an inappropriate reward shaping process may cause the agent to learn incorrect strategies or behaviors. Moreover, even if all the reward shaping processes are designed correctly and trained, a large number of models need to be accurately deployed for different business scenarios. This greatly increases the difficulty of training, the complexity of the business and the maintenance cost, and reduces the reliability of the strategy. SUMMARY

[0005] In view of the defects in the prior art, the purpose of the present application is to provide a multi-style high-controllable policy learning method and system based on reinforcement learning.

[0006] According to the multi-style high-controllable policy learning method based on reinforcement learning provided by the present application, the following steps are included:

[0007] Meta-behavior acquisition step: acquire the meta-behavior of the agent in the environment, and reward shaping is performed on each meta-behavior, each meta-behavior includes a corresponding style parameter, and the style parameter controls the reward scale after completing the meta-behavior;

[0008] Style generation step: in the process of interaction between the agent and the environment, a set of style parameters are randomly generated according to the number of meta behaviors, the style parameters are combined with the state and sent to the model of the agent for reasoning to obtain the action, and the action is fed back to the environment to generate the reward obtained by the agent at the current time and the state at the next time;

[0009] Reinforcement learning step: reinforcement learning is performed on the style parameters, state, action and reward in the process of interaction between the agent and the environment to obtain a multi-style high-controllable policy model.

[0010] Preferably, the meta behavior includes a basic action of the agent, and the reward of the agent completing the meta behavior = the basic reward of completing the first n meta behavior a n × the style parameter of the first n meta behavior w n

[0011] Preferably, the range of the style parameter includes [-1, 1];

[0012] When the style parameter is -1, it means that the execution of the meta behavior will obtain an opposite reward, thereby forming an opposite tendency;

[0013] When the style parameter is 0, it means that there is no reward for executing the meta behavior, thereby indicating that the execution of the behavior is neither encouraged nor supported;

[0014] When the style parameter is 1, it means that the reward for the meta behavior has a positive addition to the maximum extent.

[0015] Preferably, in the reinforcement learning step, the style parameter is encoded by a style encoder to obtain a style feature, and the style feature is combined with the feature of the state and sent to a training network for learning to generate an action probability and a value;

[0016] When the style parameter changes, the feature of the state does not change, and the change of the policy is only fitted by the subsequent training network parameters.

[0017] Preferably, the multi-style high-controllable policy model adopts a proximal policy optimization algorithm, and the degree of change of the policy is limited by limiting the action probability ratio between the current policy and the old policy, and the action probability ratio between the current policy and the old policy is as follows:

[0018]

[0019] Wherein s t 、c t 、a t ​Representing time steps t The status, style parameters, and actions under these conditions; and These are the current policy and the old policy in the state. s t and style parameters c t Take action below a t The probability; the ratio after restriction is ,in ε These are hyperparameters used to control the degree of restriction;

[0020] The policy objective function is defined as follows:

[0021]

[0022] in This represents the expected value of a particular sample batch. A t The time steps are calculated using the generalized dominance estimation algorithm. t Advantage estimation;

[0023] The objective of the value function is defined as follows:

[0024]

[0025] Among them, target return , Let the value function of the current strategy be... It is the value function of the old strategy;

[0026] Through continuous optimization and Allow the training network to operate in different states. s and style parameters c Each strategy has an optimal approach, and different style parameters and corresponding rewards are fitted by a value function.

[0027] According to the present invention, a multi-style highly controllable policy learning system based on reinforcement learning includes:

[0028] Meta-behavior acquisition module: Acquires the agent's meta-behaviors in the environment, shapes the rewards for each meta-behavior, and each meta-behavior includes a corresponding style parameter, which controls the reward scale after completing these meta-behaviors;

[0029] Style generation module: During the interaction between the agent and the environment, a set of style parameters is randomly generated based on the number of meta-behaviors. The style parameters and the state combination are fed into the agent's model for reasoning to obtain actions. After the actions are fed back to the environment, the reward obtained by the agent at the current moment and the state at the next moment are generated.

[0030] The reinforcement learning module: reinforcement learning is performed on the style parameters, states, actions and rewards in the process of interaction between the agent and the environment, to obtain a multi-style high-controllable policy model.

[0031] Preferably, the meta-behavior includes a basic action of the agent, and the reward for the agent completing the meta-behavior = the basic reward for completing the first n meta-behavior a n × the style parameter of the first n meta-behavior w n .

[0032] Preferably, the range of the style parameter includes [-1, 1];

[0033] When the style parameter is -1, it means that performing the meta-behavior will obtain an opposite reward, thereby forming an opposite tendency;

[0034] When the style parameter is 0, it means that performing the meta-behavior has no reward, thereby indicating that performing the behavior is neither encouraged nor supported;

[0035] When the style parameter is 1, it means that the reward for the meta-behavior has a positive addition to the maximum extent.

[0036] Preferably, in the reinforcement learning module, the style parameter is encoded by a style encoder to obtain a style feature, and the style feature is merged with the features of the state and sent to a training network for learning to generate an action probability and a value;

[0037] When the style parameter changes, the features of the state do not change, and the change of the policy is only fitted by the subsequent training network parameters.

[0038] Preferably, the multi-style high-controllable policy model adopts a proximal policy optimization algorithm, and the degree of change of the policy is limited by limiting the ratio of the action probability between the current policy and the old policy, and the ratio of the action probability between the current policy and the old policy is as follows:

[0039]

[0040] Wherein s t 、c t 、a t s t, s t, a t respectively represent the state, the style parameter and the action at time step t t; and s t, s t respectively represent the state and the style parameter at time step s t tc t The probability of taking the following action a t The ratio after limiting is Wherein ε Is a hyperparameter, used to control the degree of limitation;

[0041] The policy objective function is defined as:

[0042]

[0043] Wherein Represents the expectation of a certain sample batch, A t Is the advantage estimate of the time step t Calculated by the generalized advantage estimation algorithm;

[0044] The value function objective is defined as:

[0045]

[0046] Wherein, the target return , Is the value function of the current policy, Is the value function of the old policy;

[0047] By continuously optimizing And , the training network has the optimal policy under different states s And style parameters c Different style parameters and corresponding rewards are fitted by the value function.

[0048] Compared with the prior art, the present application has the following beneficial effects:

[0049] The present application can make a single reinforcement learning model have diversified strategies by arbitrarily collocating style parameters, and can set different style parameters for deployment according to specific application scenarios, greatly increasing the generalization and practicality of the model.

[0050] The multiple style parameters of the present application can be changed in real time during inference, so the model obtained by the multiple style learning method also has the characteristics of high reliability, can adapt to changes in the scene and make changes in the strategy at any time, and reduces the training cost and manpower maintenance cost of the reinforcement learning model.

[0051] The multiple style training method of the present application has higher efficiency, which is also helpful for reducing the training cost. BRIEF DESCRIPTION OF DRAWINGS

[0052] Other features, objects, and advantages of the application will become more apparent from the following detailed description when read in conjunction with the accompanying drawings, in which:

[0053] Figure 1 is a schematic diagram of the overall framework of the present application;

[0054] Figure 2 is a schematic diagram of meta-behavior and reward shaping;

[0055] Figure 3 is a schematic diagram of style generator;

[0056] Figure 4 is a schematic diagram of multi-style policy training network;

[0057] Figure 5 is a schematic diagram of multi-style policy screening scheme. DETAILED DESCRIPTION

[0058] The present application will be described in detail below with specific embodiments. The following examples will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of changes and improvements can be made. These are within the scope of the present application.

[0059] Reinforcement Learning (RL): Reinforcement learning is a machine learning method that allows an agent to learn how to make optimal decisions through interaction with the environment. The goal of reinforcement learning is to find a policy that maximizes the cumulative reward of the agent over time.

[0060] Training Environment (Environment): or simply environment, in reinforcement learning, the environment is the world that the agent interacts with. The environment receives the action of the agent and returns a new state and reward according to the action. The environment can be any system that can interact with the agent, such as a game, a physical world of a robot, or even a stock market model, etc.

[0061] Agent: In reinforcement learning, an agent is an entity that can perceive the environment and perform actions in the environment. The goal of the agent is to maximize the total reward by learning how to choose actions.

[0062] State: State is a description of the environment at a certain time, which contains all the necessary information for the agent to make decisions. For example, in a chess game, the state may include the type of chess pieces on each position on the board.

[0063] Action: An action is a basic operation that an agent can perform in a given state. An agent interacts with the environment by performing actions. For example, if the agent is a character in a game, possible actions include moving left, moving right, jumping, etc.

[0064] Reward: A reward is the feedback from the environment to the agent's action. After an agent performs an action, the environment returns a reward to tell the agent how good or bad the action is. The reward can be positive (indicating encouraging the action) or negative (indicating discouraging the action). The goal of the agent is to maximize the total reward by learning how to choose actions.

[0065] Episode: A training environment does not run indefinitely, but starts and ends in episodes. At the beginning of each episode, the environment will be initialized to a certain state; at the end of each episode, done will be signaled.

[0066] Reward shaping: Also known as reward setting, reward shaping is a reinforcement learning technique that guides the agent to learn faster by designing a reward function.

[0067] Meta-behavior: The main behavior of an agent in a training environment, which is usually composed of a combination of meta-behaviors. For example, the action of jumping left can be completed by combining the left movement and jumping actions of the agent.

[0068] As shown in Figure 1 The present application provides a multi-style high-controllable policy learning method based on reinforcement learning, which comprises:

[0069] Meta-behavior acquisition step: Acquire the meta-behavior of the agent in the environment, and perform reward shaping on each meta-behavior. Each meta-behavior includes a corresponding style parameter, which controls the reward scale after completing the meta-behavior.

[0070] In the present application, the behavior of the agent to be trained in the environment is first summarized, and a series of meta-behaviors are designed, which can be completed by the basic actions of the agent. Then, reward shaping is performed on each meta-behavior, i.e. completing the meta-behavior will obtain a reward value. Each meta-behavior has a corresponding style parameter, which controls the reward scale after completing the meta-behavior, so each style parameter can represent the tendency of the corresponding meta-behavior.

[0071] Style generation step: in the process of interaction between the agent and the environment, a set of style parameters is randomly generated according to the number of meta behaviors, the style parameters are combined with the state to be fed into the model of the agent for reasoning to obtain the action, and the action is fed back to the environment to generate the reward obtained by the agent at the current time and the state at the next time.

[0072] The tendency of different meta behaviors of the agent can be freely combined to form a large number of strategy styles, in the process of interaction between the agent and the environment, the style generator will randomly generate a set of style parameters according to the number of meta behaviors, this style parameter will be combined with the state, and then fed into the model of the agent for reasoning to obtain the action, and the action is fed back to the environment to generate the reward obtained by the agent at the current time and the state at the next time. In this process, the reward is affected by the multi-style parameter.

[0073] Reinforcement learning step: reinforcement learning is performed on the style parameters, states, actions and rewards in the process of interaction between the agent and the environment, and the multi-style parameters and meta behavior tendencies are fitted, and finally a multi-style, highly controllable strategy model is trained.

[0074] Different multi-style parameter settings can form multi-style strategies, so that a single reinforcement learning model can have multiple style strategies at the same time, increasing the generalization of the model and reducing the redundancy of the model. By adjusting the multi-style parameters, the strategy of the model can be controlled, such as controlling the style of the current strategy, the switching time of the style, etc., increasing the controllability of the strategy and reducing the complexity of the business. With the help of data statistics, multi-style parameter configurations that can achieve specific business requirements can be automatically selected, reducing the labor input.

[0075] The following takes a basketball game as an example:

[0076] We first need to design a series of meta behaviors of the agent in the environment, the principles for designing meta behaviors are: (1) the correlation between meta behaviors is as little as possible; (2) through the combination of meta behaviors, the behaviors that the agent can make in the environment are covered as much as possible; (3) under the above conditions, the number of meta behaviors is as small as possible.

[0077] This is to make the subsequent process simpler and greatly increase the success rate of multi-style strategy training. Then we need to shape the reward for each meta behavior, that is, how much reward will be obtained by completing such a meta behavior, the process is as Figure 2 shown. Among them a n represents the basic reward obtained by completing the n th meta behavior, w nThe style parameter corresponding to the meta-behavior is represented, and the final reward of the meta-behavior is obtained by multiplying the basic reward and the style parameter. The meta-behaviors can be combined to form rich behaviors, for example, the combination of "two-point shot" and "region 1 activity" can form the behavior of "two-point shot in region 1", and through the adjustment of the style parameter, the tendency of these behavior combinations also has different degrees, thereby forming rich strategy styles.

[0078] In the reinforcement learning phase, the environment will interact with the training end to continuously generate the latest strategy data. At the beginning of each training period of the environment, the style generator will generate the style parameter used in this training period. The style generator can generate a style parameter corresponding to the reward configuration, and the style parameter can control the reward value obtained by the strategy when executing the behavior in training by multiplying the corresponding reward. The range of the style parameter can usually be set to [-1, 1], when the style parameter is -1, it means that the strategy executing the behavior will obtain the opposite reward, thereby forming an opposite tendency; when the style parameter is 0, it means that the reward obtained by executing the behavior is 0, thereby indicating that it neither encourages nor supports the execution of the behavior; when the style parameter is 1, it means that the reward of the behavior has the maximum positive addition. The style parameter can be generated in the form of random sampling, or can be generated by setting specific rules, such as Figure 3 For example, any value between -1 and 1 can be randomly generated, values can be generated at intervals of 0.1, or only values between 0 and 1 can be generated, etc. Through the style generator, the reinforcement learning strategy can obtain data under different reward settings, thereby providing a basis for fitting the optimal strategy under different style parameters for subsequent reinforcement learning algorithms.

[0079] The generated style parameter is encoded by a style encoder in training to obtain a style feature, which is merged with the original input state feature and sent to the subsequent network for learning. This process finally generates action probability and value. When the style parameter changes, the state feature does not change, and the change of the strategy is only fitted by the subsequent network parameters, which can be beneficial to the learning of the state and the style parameter, and the process is as shown in Figure 4 .

[0080] In order to efficiently train a model with a multi-style strategy, we use the Proximal Policy Optimization (PPO) algorithm as our main algorithm, which is a popular policy gradient algorithm and performs well in various tasks. The PPO algorithm limits the degree of change of the strategy by limiting the action probability ratio between the current policy and the old policy, and the action probability ratio between the current policy and the old policy is as follows:

[0081]

[0082] wherein s t 、c t 、a t respectively represent the state, style parameter and action at time step t ; and are the probabilities of the current policy and the old policy taking action s t and style parameter c t at state a t The ratio after restriction is wherein ε is a hyperparameter used to control the degree of restriction. By such a restriction method, the policy can be prevented from making large fluctuations, thereby avoiding instability in the training process.

[0083] The policy objective function is defined as:

[0084]

[0085] wherein represents the expectation of a certain sample batch, A t is the advantage estimation of time step t calculated by the Generalized Advantage Estimator (GAE);

[0086] The value function objective is defined as:

[0087]

[0088] wherein, the target return , is the value function of the current policy, is the value function of the old policy;

[0089] By continuously optimizing and , the training network has the optimal policy at different states s and style parameters c . Different style parameters and corresponding rewards are fitted by the value function. The value function can guide the optimal behavior of the policy under different styles, thereby ensuring the convergence of the algorithm. At the same time, by setting different style parameters in the input to call different behavior styles, the controllability of the policy is ensured.

[0090] In training, multi-style parameters can be randomly generated in general, so as to increase the stability and generalization of the strategy. In use, the expected style parameter combination can be set according to specific scene or business requirements. The setting of the style parameter can be set in a heuristic manner, or the desired strategy can be selected by a data analysis method. The heuristic method is relatively simple, and can be configured based on the experience of algorithm engineers or planners, but usually a large number of tests are required to obtain the desired strategy. The method of data analysis can accurately obtain the style parameters of the required strategy, which requires the behavior indicators of the desired strategy to be specified first. For example, in this application scenario, the behavior indicators can be the average score of the agent in each game, the number of dunks, the number of three-point shots, the number of blocks, and the like. Then we traverse the style parameter combination with a certain granularity, and test in the environment to obtain the game behavior statistical indicators under different style parameter configurations, so as to obtain the style parameters of the strategy closest to the target indicators. For example, as shown in the basketball game application, we can first randomly sample two sets of style parameters, and then let the model use the two sets of style parameters to fight, obtain the statistical results of the fight, and deliver the results to the database. The database records the statistical indicators of all the styles, and then sets the values of the game indicators according to the requirements of the planner or the scene, for example, requiring an average of 5 three-point shots and 6 dunks per game. Then find the most matching style parameters in the database according to the specific game indicators, which are the target style parameters. In this way, different style parameters can be selected. Since the style parameters can be changed at any time in the game, this method realizes real-time control of the strategy, greatly increasing the practicality and flexibility of the AI Bot. Figure 5

[0091] The application also provides a multi-style high-controllable strategy learning system based on reinforcement learning, which can be realized by executing the process steps of the multi-style high-controllable strategy learning method based on reinforcement learning, that is, the multi-style high-controllable strategy learning method based on reinforcement learning can be understood by those skilled in the art as the preferred embodiment of the multi-style high-controllable strategy learning system based on reinforcement learning.

[0092] The application provides a multi-style high-controllable strategy learning system based on reinforcement learning, which comprises:

[0093] The meta-behavior acquisition module acquires the meta-behavior of the agent in the environment, rewards and shapes each meta-behavior, and each meta-behavior comprises a corresponding style parameter, and the style parameter controls the reward scale after the completion of the meta-behavior.

[0094] ​Style generation module: in the process of the agent interacting with the environment, a set of style parameters is randomly generated according to the number of meta behaviors, the style parameters are input into the model of the agent combined with the state to obtain an action, and the action is fed back to the environment to obtain the reward obtained by the agent at the current time and the state at the next time.

[0095] Reinforcement learning module: the style parameters, state, action and reward in the process of the agent interacting with the environment are subjected to reinforcement learning to obtain a multi-style high-controllable strategy model.

[0096] Those skilled in the art know that, in addition to implementing the system provided by the present application and each device, module and unit thereof in a pure computer readable program code manner, the system provided by the present application and each device, module and unit thereof can also be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers to achieve the same functions by logically programming the method steps. Therefore, the system provided by the present application and each device, module and unit thereof can be considered as a hardware component, and the devices, modules and units included therein for achieving various functions can also be considered as structures within the hardware component; the devices, modules and units for achieving various functions can also be considered as both software modules for implementing methods and structures within the hardware component.

[0097] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the specific embodiments described above, and various changes or modifications can be made by those skilled in the art within the scope of the claims, which do not affect the essential content of the present application. In the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A multi-style high-controllable policy learning method based on reinforcement learning, characterized in that, The method comprises the following steps: a meta-behavior acquisition step: acquiring meta-behaviors of an agent in an environment, and performing reward shaping on each meta-behavior, each meta-behavior comprising a corresponding style parameter, the style parameter controlling a reward scale after the meta-behaviors are completed; a style generation step: in the process of interaction between the agent and the environment, a set of style parameters are randomly generated according to the number of meta-behaviors, the style parameters and state combinations are fed into a model of the agent to obtain actions, and the actions are fed back to the environment to generate a reward obtained by the agent at the current time and a state at the next time; a reinforcement learning step: performing reinforcement learning on the style parameters, the state, the actions and the reward in the process of interaction between the agent and the environment to obtain a multi-style high-controllable policy model; The meta-behavior includes a basic action of the agent, and a reward for the agent completing the meta-behavior = a basic reward for completing the n a n × the style parameter of the n w n th meta-behavior​​ in the reinforcement learning step, the style parameters are encoded by a style encoder to obtain style features, and the style features are combined with features of the state and fed into a training network to learn to generate action probabilities and values; when the style parameters change, the features of the state do not change, and the change of the policy is only fitted by subsequent training network parameters; the multi-style high-controllable policy model adopts a proximal policy optimization algorithm, and the degree of change of the policy is limited by limiting the action probability ratio between the current policy and the old policy, the action probability ratio between the current policy and the old policy being as follows: where s t 、c t 、a t s, φ, a represent the state, style parameter and action at time step t πθ and πθold are the current and old policy respectively, and s t φs, φsold are the style parameters at state s and sold respectively c t πθ (s, φs) and πθold (s, φs) are the probabilities of taking action a under the current and old policy respectively a t the ratio after restriction is where ε is a hyperparameter that controls the degree of restriction​​ a policy objective function is defined as: wherein represents the expectation of a certain sample batch, A t is the time step t advantage estimate calculated by the generalized advantage estimation algorithm; a value function objective is defined as: where the target return , is the value function of the current policy, is the value function of the old policy; By continuously optimizing and , the training network has the optimal strategy in different states s and style parameters c , and different style parameters and corresponding rewards are fitted by the value function.

2. The method of claim 1, wherein, the range of the style parameters comprises [-1, 1]; when the style parameter is -1, it indicates that the meta-behavior will obtain an opposite reward, thereby forming an opposite tendency; when the style parameter is 0, it indicates that the meta-behavior has no reward, thereby indicating that the meta-behavior is neither encouraged nor supported; when the style parameter is 1, it indicates that the meta-behavior has the maximum reward with a positive addition.

3. A multi-style high-controllable policy learning system based on reinforcement learning, characterized in that, The method comprises the following steps: a meta-behavior acquisition module: acquiring meta-behaviors of an agent in an environment, and performing reward shaping on each meta-behavior, each meta-behavior comprising a corresponding style parameter, the style parameter controlling a reward scale after the meta-behaviors are completed; a style generation module: in the process of interaction between the agent and the environment, a set of style parameters are randomly generated according to the number of meta-behaviors, the style parameters and state combinations are fed into a model of the agent to obtain actions, and the actions are fed back to the environment to generate a reward obtained by the agent at the current time and a state at the next time; a reinforcement learning module: performing reinforcement learning on the style parameters, the state, the actions and the reward in the process of interaction between the agent and the environment to obtain a multi-style high-controllable policy model; The meta-behavior includes a basic action of the agent, and a reward for the agent completing the meta-behavior = a basic reward for completing the n a n n w n th meta-behavior​​​ in the reinforcement learning module, the style parameters are encoded by a style encoder to obtain style features, and the style features are combined with features of the state and fed into a training network to learn to generate action probabilities and values; when the style parameters change, the features of the state do not change, and the change of the policy is only fitted by subsequent training network parameters; the multi-style high-controllable policy model adopts a proximal policy optimization algorithm, and the degree of change of the policy is limited by limiting the action probability ratio between the current policy and the old policy, the action probability ratio between the current policy and the old policy being as follows: in s t 、c t 、a t Representing time steps t The status, style parameters, and actions under these conditions; and These are the current policy and the old policy in the state. s t and style parameters c t Take action below a t The probability; the ratio after restriction is ,in ε These are hyperparameters used to control the degree of restriction; a policy objective function is defined as: wherein represents the expectation of a certain sample batch, A t is the advantage estimate of the time step t computed by the generalized advantage estimation algorithm. The value function objective is defined as: where the target return , is the value function of the current policy, is the value function of the old policy; By continuously optimizing and , the training network has the optimal strategy in different states s and style parameters c , and different style parameters and corresponding rewards are fitted by the value function.

4. The multi-style high-controllable policy learning system based on reinforcement learning according to claim 3, characterized in that, The range of the style parameter includes [-1, 1]; When the style parameter is -1, it indicates that performing the meta-behavior will obtain an opposite reward, thereby forming an opposite tendency; When the style parameter is 0, it indicates that there is no reward for performing the behavior, thereby indicating that performing the behavior is neither encouraged nor supported; When the style parameter is 1, it indicates that the maximum reward for the meta-behavior has a positive addition.

Images