Reinforcement learning method, device, computer equipment and storage medium for intelligent agent

Abstract

The present disclosure provides a reinforcement learning method, apparatus, computer equipment and storage medium for an intelligent agent, wherein the method comprises: obtaining task requirements of a target task; determining, based on the task requirements of the target task, the state space and action space corresponding to the intelligent agent for executing the target task; centrally training the intelligent agent based on the state space, action space and a baseline algorithm in a pre-built intelligent agent reinforcement learning framework to determine the value network corresponding to the intelligent agent; and training the policy network corresponding to the intelligent agent based on the value network corresponding to the intelligent agent to control the intelligent agent to execute the target task based on the trained policy network.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a reinforcement learning method, device, computer equipment and storage medium for an intelligent agent. Background Art

[0002] Multi-agent game is an important issue in the field of artificial intelligence. The multi-agent reinforcement learning technology used at its underlying layer has broad application potential in game games, intelligent transportation, intelligent medical care, intelligent manufacturing and other fields.

[0003] In related technologies, when training an intelligent agent, developers are often required to manually define the environment space and state space, and then select a baseline algorithm to train the intelligent agent. After the training of the intelligent agent is completed, the training results of the intelligent agent can be checked through simulation experiments, and the problems that arise during the simulation experiment can be optimized. However, this process often takes a lot of time, so how to efficiently train the intelligent agent has become an urgent problem to be solved in related fields. Summary of the invention

[0004] The embodiments of the present disclosure at least provide a method, apparatus, computer device and storage medium for reinforcement learning of an intelligent agent.

[0005] In a first aspect, an embodiment of the present disclosure provides a reinforcement learning method for an intelligent agent, comprising:

[0006] Obtain the task requirements of the target task;

[0007] Based on the task requirements of the target task, determining the state space and action space corresponding to the agent for performing the target task;

[0008] Centrally training the agent based on the state space, the action space, and a baseline algorithm in a pre-built agent reinforcement learning framework to determine a value network corresponding to the agent;

[0009] Based on the value network corresponding to the agent, the policy network corresponding to the agent is trained to control the agent to perform the target task based on the trained policy network.

[0010] In this way, the whole process from state space and action space definition, centralized training of agents, to policy network training can be completed conveniently. The pre-built agent reinforcement learning framework can reduce the amount of code development in the whole process, and corresponding extensions can be made based on the pre-built agent reinforcement learning framework, thereby providing developers with a complete and easy-to-use solution for solving multi-agent game problems, thereby improving the efficiency of agent reinforcement learning.

[0011] In a possible implementation, the state type of the state space includes one or more of global observation information, local observation information, and mixed observation information; the action type of the action space includes one or more of discrete action, continuous action, and mixed action.

[0012] In this way, by reasonably defining the representation methods of state space and action space, a widely applicable access standard can be established for the intelligent agent reinforcement learning framework, so that the constructed intelligent agent reinforcement learning framework can be applicable to different task scenarios to adapt to different task requirements.

[0013] In a possible implementation, the agent reinforcement learning framework includes at least one pre-packaged functional module;

[0014] The step of training the strategy network corresponding to the agent based on the value network corresponding to the agent includes:

[0015] Based on the task requirements of the target task, determining a target functional module for executing the target task;

[0016] Based on the value network corresponding to the agent and the target function module, the strategy network corresponding to the agent is trained.

[0017] In this way, by encapsulating functional modules for the agent reinforcement learning framework, the amount of code development for developers during the reinforcement learning training process can be reduced, thereby improving the efficiency of training the agent.

[0018] In a possible implementation manner, the target function module includes a first target function module and a second target function module:

[0019] The second target function module is an agent action masking module, which is used to mask the target type action output by the policy network so that the target type action is not executed by the agent;

[0020] The first objective function module is a normalization module, which is used to normalize the value function corresponding to the value network.

[0021] In a possible implementation, the training of the strategy network corresponding to the agent based on the value network corresponding to the agent and the target function module includes:

[0022] The value network corresponding to the agent is optimized based on the first objective function module, and the strategy network corresponding to the agent is trained based on the optimized value network.

[0023] In this way, by normalizing the value function corresponding to the value network, the training process of the value network can be accelerated.

[0024] In a possible implementation, the training of the strategy network corresponding to the agent based on the value network corresponding to the agent and the target function module includes:

[0025] For any intelligent agent, the strategy network of the intelligent agent is optimized based on the second objective function module;

[0026] Based on the optimized policy network, determine the output value of the value network corresponding to the agent, and determine the loss value corresponding to the policy network corresponding to the agent based on the output value of the value network;

[0027] The policy network corresponding to the agent is trained based on the loss value.

[0028] In this way, the target type action can be shielded through the agent action masking module.

[0029] In one possible implementation, the input of the policy network is the local observation information of the corresponding intelligent agent, and the output is the probability value of the corresponding intelligent agent executing each action in the action space; the input of the value network is the global observation information of each intelligent agent, and the output is the evaluation value of each action or state of the intelligent agent.

[0030] In a possible implementation, the agent reinforcement learning framework includes at least one pre-packaged optimization module;

[0031] The step of training the strategy network corresponding to the agent based on the value network corresponding to the agent includes:

[0032] Based on the optimization instruction, determining a first target optimization module for optimizing the value network;

[0033] The value network corresponding to the agent is optimized based on the first target optimization module, and the strategy network corresponding to the agent is trained based on the optimized value network.

[0034] In a possible implementation, the agent reinforcement learning framework includes at least one pre-packaged optimization module;

[0035] After training the policy network corresponding to the agent, the method further includes:

[0036] Based on the target task, testing the trained agent;

[0037] Based on the test result of the target task, determining a second target optimization module for optimizing the value network;

[0038] Based on the second objective optimization module, optimizing the value network;

[0039] Based on the optimized value network, the strategy network corresponding to the agent is optimized and trained.

[0040] In this way, the value network for training the policy network can be adapted to different application scenarios, so that each agent can better perform the target task.

[0041] In one possible implementation, the optimization module includes an agent state masking module, which is used to set the state parameters corresponding to the target agent that needs to perform agent masking operation as target parameters to characterize that the target agent does not participate in the training process of the value network, wherein the state parameters are used for state evaluation through the value network.

[0042] In this way, it is possible to avoid the influence of the dead agent on the value network training process, thereby ensuring the network accuracy of the value network.

[0043] In one possible implementation, the optimization module includes a joint value function decomposition module, which is used to establish a local value network for each intelligent agent, and based on each local value network and a preset joint strategy, construct a combined hybrid network to train the strategy network based on the output of the hybrid network.

[0044] In this way, the multi-agent reinforcement learning framework can be applied to more scenarios.

[0045] In a possible implementation, the method further includes optimizing the policy network according to the following method:

[0046] Performing a truncation operation on the output value of the policy network to obtain a target output value in the output value that meets the requirements of a preset numerical range;

[0047] The target output value is activated based on the activation function to instruct the agent to perform the target task according to the target output value after the activation process.

[0048] In this way, the problem that the probability value distribution of each action output by the policy network is too sharp due to inappropriate initialization parameter setting of the policy network, thereby limiting the degree of exploration of the environment by the intelligent agent when performing the target task can be avoided.

[0049] In a second aspect, the present disclosure also provides a reinforcement learning device for an intelligent agent, comprising:

[0050] An acquisition module is used to obtain the task requirements of the target task;

[0051] A first determination module is used to determine the state space and action space corresponding to the agent for performing the target task based on the task requirements of the target task;

[0052] A second determination module is used to perform centralized training on the agent based on the state space, the action space and a baseline algorithm in a pre-built agent reinforcement learning framework to determine a value network corresponding to the agent;

[0053] A training module is used to train the strategy network corresponding to the agent based on the value network corresponding to the agent, so as to control the agent to perform the target task based on the trained strategy network.

[0054] In a possible implementation, the state type of the state space includes one or more of global observation information, local observation information, and mixed observation information; the action type of the action space includes one or more of discrete action, continuous action, and mixed action.

[0055] In a possible implementation, the agent reinforcement learning framework includes at least one pre-packaged functional module;

[0056] The training module, when training the strategy network corresponding to the agent based on the value network corresponding to the agent, is used to:

[0057] Based on the task requirements of the target task, determining a target functional module for executing the target task;

[0058] Based on the value network corresponding to the agent and the target function module, the strategy network corresponding to the agent is trained.

[0059] In a possible implementation manner, the target function module includes a first target function module and a second target function module:

[0060] The second target function module is an agent action masking module, which is used to mask the target type action output by the policy network so that the target type action is not executed by the agent;

[0061] The first objective function module is a normalization module, which is used to normalize the value function corresponding to the value network.

[0062] In a possible implementation, the training module, when training the policy network corresponding to the agent based on the value network corresponding to the agent and the target function module, is used to:

[0063] The value network corresponding to the agent is optimized based on the first objective function module, and the strategy network corresponding to the agent is trained based on the optimized value network.

[0064] In a possible implementation, the training module, when training the policy network corresponding to the agent based on the value network corresponding to the agent and the target function module, is used to:

[0065] For any intelligent agent, the strategy network of the intelligent agent is optimized based on the second objective function module;

[0066] Based on the optimized policy network, determine the output value of the value network corresponding to the agent, and determine the loss value corresponding to the policy network corresponding to the agent based on the output value of the value network;

[0067] The policy network corresponding to the agent is trained based on the loss value.

[0068] In one possible implementation, the input of the policy network is the local observation information of the corresponding intelligent agent, and the output is the probability value of the corresponding intelligent agent executing each action in the action space; the input of the value network is the global observation information of each intelligent agent, and the output is the evaluation value of each action or state of the intelligent agent.

[0069] In a possible implementation, the agent reinforcement learning framework includes at least one pre-packaged optimization module;

[0070] The training module, when training the strategy network corresponding to the agent based on the value network corresponding to the agent, is used to:

[0071] Based on the optimization instruction, determining a first target optimization module for optimizing the value network;

[0072] The value network corresponding to the agent is optimized based on the first target optimization module, and the strategy network corresponding to the agent is trained based on the optimized value network.

[0073] In a possible implementation, the agent reinforcement learning framework includes at least one pre-packaged optimization module;

[0074] After training the policy network corresponding to the agent, the training module is further used to:

[0075] Based on the target task, testing the trained agent;

[0076] Based on the test result of the target task, determining a second target optimization module for optimizing the value network;

[0077] Based on the second objective optimization module, optimizing the value network;

[0078] Based on the optimized value network, the strategy network corresponding to the agent is optimized and trained.

[0079] In one possible implementation, the optimization module includes an agent state masking module, which is used to set the state parameters corresponding to the target agent that needs to perform agent masking operation as target parameters to characterize that the target agent does not participate in the training process of the value network, wherein the state parameters are used for state evaluation through the value network.

[0080] In one possible implementation, the optimization module includes a joint value function decomposition module, which is used to establish a local value network for each intelligent agent, and based on each local value network and a preset joint strategy, construct a combined hybrid network to train the strategy network based on the output of the hybrid network.

[0081] In a possible implementation manner, the training module is further used to optimize the policy network according to the following steps:

[0082] Performing a truncation operation on the output value of the policy network to obtain a target output value in the output value that meets the requirements of a preset numerical range;

[0083] The target output value is activated based on the activation function to instruct the agent to perform the target task according to the target output value after the activation process.

[0084] In a third aspect, an embodiment of the present disclosure further provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are performed.

[0085] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are executed.

[0086] For a description of the effects of the above-mentioned reinforcement learning device, computer equipment and storage medium of the intelligent agent, please refer to the description of the reinforcement learning method of the above-mentioned intelligent agent, which will not be repeated here.

[0087] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0089] Figure 1 A flow chart of a reinforcement learning method for an intelligent agent provided by an embodiment of the present disclosure is shown;

[0090] Figure 2 A flowchart showing a specific method for training a policy network corresponding to an intelligent agent in a reinforcement learning method for an intelligent agent provided by an embodiment of the present disclosure;

[0091] Figure 3 A flowchart showing another specific method for training a policy network in the reinforcement learning method of an intelligent agent provided in an embodiment of the present disclosure is shown;

[0092] Figure 4 A flowchart showing another specific method for training a policy network corresponding to an intelligent agent in the reinforcement learning method of the intelligent agent provided by an embodiment of the present disclosure;

[0093] Figure 5 A flowchart showing a specific method for optimizing and training a policy network corresponding to an intelligent agent in a reinforcement learning method for an intelligent agent provided by an embodiment of the present disclosure;

[0094] Figure 6 A flowchart showing a specific method for optimizing a strategy network in a reinforcement learning method for an intelligent agent provided in an embodiment of the present disclosure is shown;

[0095] Figure 7 A schematic diagram of the architecture of a reinforcement learning device for an intelligent agent provided by an embodiment of the present disclosure is shown;

[0096] Figure 8A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0097] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0098] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0099] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0100] Research has found that when training an intelligent agent, developers often need to manually define the environment space and state space, and then select a baseline algorithm to train the intelligent agent. After the intelligent agent training is completed, the training results of the intelligent agent can be checked through simulation experiments, and the problems that arise during the simulation experiment can be optimized. However, this process often takes a lot of time, so how to efficiently train the intelligent agent has become an urgent problem to be solved in related fields.

[0101] Based on the above research, the present disclosure provides a reinforcement learning method, device, computer equipment and storage medium for an intelligent agent, and pre-builds an intelligent agent reinforcement learning framework. Based on the task requirements of the target task, the state space and action space corresponding to the intelligent agent used to perform the target task can be determined, and the baseline algorithm in the intelligent agent reinforcement learning framework can be used to centrally train the intelligent agent, so as to train the policy network corresponding to the intelligent agent according to the obtained value network corresponding to the intelligent agent, and obtain a policy network that can control the intelligent agent to perform the target task. In this way, the whole process of defining the state space and action space, the centralized training of the intelligent agent and the training of the policy network can be completed, and the pre-built intelligent agent reinforcement learning framework can reduce the amount of code development in the whole process, and corresponding extensions can be made on the basis of the pre-built intelligent agent reinforcement learning framework, so as to provide developers with a complete and easy-to-use solution for solving multi-agent game problems, so as to improve the efficiency of intelligent agent reinforcement learning.

[0102] To facilitate understanding of the present embodiment, a reinforcement learning method for an intelligent agent disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the reinforcement learning method for an intelligent agent provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities, and the computer device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the reinforcement learning method for the intelligent agent can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0103] See also Figure 1 FIG. 1 is a flow chart of a reinforcement learning method for an intelligent agent provided by an embodiment of the present disclosure, wherein the method includes S101 to S104, wherein:

[0104] S101: Obtain task requirements of the target task.

[0105] S102: Based on the task requirements of the target task, determine the state space and action space corresponding to the agent used to perform the target task.

[0106] S103: Centrally train the agent based on the state space, action space, and a baseline algorithm in a pre-built agent reinforcement learning framework to determine a value network corresponding to the agent.

[0107] S104: Based on the value network corresponding to the agent, the policy network corresponding to the agent is trained to control the agent to perform the target task based on the trained policy network.

[0108] The following is a detailed description of the above steps.

[0109] With respect to S101, the task requirement of the target task represents the goal to be achieved by training the intelligent agent in this reinforcement learning process.

[0110] In practical applications, agent design is an important part of multiplayer cooperative game design. Agents have certain decision-making intelligence, and can use their decision-making intelligence to control artificial intelligence players and real players in the game. For example, a human-computer game scenario may include multiple artificial intelligence players, each of which can be regarded as an agent. The target task may be to enable the multiple artificial intelligence players to win the human-computer game, and the task requirement corresponding to the target task is to enable the artificial intelligence players to perform appropriate game operations (release game skills at the right time, etc.).

[0111] S102: Based on the task requirements of the target task, determine the state space and action space corresponding to the agent used to perform the target task.

[0112] Here, the state type of the state space includes one or more of global observation information, local observation information and mixed observation information; the action type of the action space includes one or more of discrete action, continuous action and mixed action.

[0113] Among them, the global observation information is the observation information of the overall environment in the application scenario, the local observation information is the information observed by a single intelligent agent, the mixed observation information is the information observed by multiple intelligent agents (such as the information observed by two adjacent intelligent agents), the discrete action indicates that adjacent actions are discrete (such as in the Go game scenario, only one piece can be placed at a time), the continuous action indicates that adjacent actions are continuous (such as controlling the walking of the target character in the game scenario), and the mixed action indicates the simultaneous existence of discrete actions and continuous actions.

[0114] In practical applications, corresponding state spaces and action spaces can be set in advance for different types of task requirements based on the task requirements in different application scenarios, so that when using the intelligent agent reinforcement learning framework, the corresponding state space and action space can be matched according to the identified task requirements.

[0115] Specifically, according to the state type of the state space, the representation methods of the state space that can be defined include local observation information of each intelligent agent, global observation information, local observation information and global observation information of each intelligent agent, local observation information and global observation information of each intelligent agent with the overlapping parts removed, and mixed observation information.

[0116] Exemplarily, still taking the human-computer game scenario as an example, based on the pre-set task requirements and the association between the state space and the action space, it can be determined that the actions in the action space under the scenario are mixed actions, and the observation information in the state space under the scenario is determined to be global observation information.

[0117] In this way, by reasonably defining the representation method of the state space, a widely applicable access standard can be established for the intelligent agent reinforcement learning framework, so that the constructed intelligent agent reinforcement learning framework can be applicable to different task scenarios to adapt to different task requirements.

[0118] S103: Centrally train the agent based on the state space, action space, and a baseline algorithm in a pre-built agent reinforcement learning framework to determine a value network corresponding to the agent.

[0119] Here, the input of the value network can be the global observation information of each intelligent agent, and the output is the evaluation value of each action or state of the intelligent agent. The input of the strategy network can be the local observation information of the corresponding intelligent agent, and the output is the probability value of the corresponding intelligent agent executing each action in the action space.

[0120] Specifically, the general algorithm framework in the agent reinforcement learning framework can adopt a learning paradigm of centralized training and decentralized execution. In this paradigm, the input value of each agent is a fusion of the local observation of the agent and the global observation. The value network is used to evaluate the actions of each agent. The trained value network is used to train each agent to learn the corresponding strategy network. After centralized training of multiple agents, since there is only one value network corresponding to multiple agents, this value network can also be called a joint value network.

[0121] Furthermore, in the agent reinforcement learning framework, the policy network and the value network can be configured as follows:

[0122] 1. Policy Network

[0123] Here, the input of the policy network is the local observation information observed by the corresponding agent, and the dimensions are (B, A, N local state ), where B represents the number of samples during training, A represents the number of agents, and N local stateIndicates the dimension of the local observation information input by the agent. In the single-agent application scenario, A=1, and in the multi-agent application scenario, A>1, and the local observation information of different agents is different. The output dimension of the agent strategy network is (B, A, N action ), which represents the probability value logits of the agent executing each action in the corresponding action space, where N action Indicates the number of actions that the agent can choose.

[0124] Specifically, after obtaining the probability values ​​corresponding to each action output by the strategy network, the action that the agent needs to perform can be determined based on the probability values ​​corresponding to each action, such as executing the target action with the largest probability value.

[0125] Furthermore, when collecting data from the policy network, the Boltzmann distribution sampling method can be used according to the probability value of each action to obtain the action performed by each agent, so that when the value network evaluates the action of the policy network, the action with the highest probability is used as the action performed by each agent.

[0126] 2. Value Network

[0127] Here, the input of the value network is the local observation information observed by the corresponding agent, and the dimension is (B, A, N state ), where N state It represents the dimension after the fusion of the local observation information and the global observation information of the agent (the fusion process is the process of data splicing). In the application scenario of a single agent, A=1, and in the application scenario of multiple agents, A>1, and the fused information corresponding to different agents is different from the fused information corresponding to other agents.

[0128] Specifically, the value network can be configured as an action evaluation network Q network, and the evaluation value of the action performed by each agent can be determined through the value network; in addition, the value network can also be configured as a state evaluation network V network, and the evaluation value of the state of each agent at this time can be determined through the value network. The evaluation value represents the degree of influence of the current action or state on the agent's execution of the target task in the application scenario, and can be obtained through a preset calculation formula or neural network fitting.

[0129] It should be noted that the essence of the agent reinforcement learning framework is to process code, and the essence of the above-mentioned value network and policy network is function. The use of neural networks can better fit the function, so the corresponding neural network is used to approximate its real function. In addition, when optimizing the value network and the policy network, optimization methods such as stochastic gradient descent can be used to optimize network parameters, so that the algorithms in the agent reinforcement learning framework are richer, and corresponding algorithms can be selected according to different task requirements.

[0130] In this way, since the relevant settings for the number of agents are added when the agent reinforcement learning framework is pre-constructed, when the agent reinforcement learning framework is actually used, through the parameter configuration of the policy network and the value network, only a small amount of changes in the parameter settings of each network and the input of information of each network are needed to achieve the switching of multi-agent and single-agent task modes, so that the use of the agent reinforcement learning framework can be widely used in different scenarios.

[0131] S104: Based on the value network corresponding to the agent, the policy network corresponding to the agent is trained to control the agent to perform the target task based on the trained policy network.

[0132] In a possible implementation, Figure 2 As shown, the policy network corresponding to the agent can be trained by the following steps:

[0133] S201: Based on the task requirements of the target task, determine a target functional module for executing the target task.

[0134] Here, the target function module includes a first target function module and a second target function module, wherein the first target function module is a normalization module and the second target function module is an agent action masking module.

[0135] Next, the normalization module and the agent action masking module will be introduced in detail:

[0136] Module 1: Normalization module

[0137] Here, the normalization module can be used to normalize the value function corresponding to the value network, thereby speeding up the training process of the value network.

[0138] Specifically, the outputs of general value functions and advantage functions (a type of value function) are greatly affected by states and actions. Therefore, in order to facilitate the fitting of the value network to the value function, the output values ​​of the value function can be normalized, such as subtracting the mean of the output values ​​and then dividing them by the standard deviation of the output values.

[0139] Module 2: Agent Action Masking Module

[0140] Here, the agent action masking module can mask the target type action output by the policy network so that the target type action is not executed by the agent.

[0141] Specifically, since the agent's selectable actions change during the training process, for the output of the policy network, the logits value corresponding to the non-selectable target type action can be set to a large negative constant value to ensure that the probability of executing this action output after the network activation operation is zero, thereby shielding the target type action.

[0142] S202: Based on the value network corresponding to the agent and the target function module, the strategy network corresponding to the agent is trained.

[0143] Here, the method of training the policy network corresponding to the agent is different depending on the target function module used.

[0144] In a possible implementation, when training the policy network corresponding to the agent, the value network corresponding to the agent can be optimized based on the first objective function module, and the policy network corresponding to the agent can be trained based on the optimized value network.

[0145] Here, by using the normalization module to normalize the value function, the training speed of the value network can be accelerated.

[0146] In another possible implementation, Figure 3 As shown, the policy network can be trained by the following steps:

[0147] S2021: For any intelligent agent, optimize the strategy network of the intelligent agent based on the second objective function module.

[0148] S2022: Based on the optimized policy network, determine the output value of the value network corresponding to the agent, and determine the loss value corresponding to the policy network corresponding to the agent based on the output value of the value network.

[0149] S2023: Training the policy network corresponding to the agent based on the loss value.

[0150] Specifically, after using the agent action masking module to optimize the output of the policy network, the actions performed by the agent can be affected by changing the probability values ​​of each action output by the policy network, so that the evaluation value obtained by the value network after evaluation also changes accordingly, so that the policy network can be trained according to the adjusted evaluation value to obtain a trained policy network.

[0151] In practical applications, the agent reinforcement learning framework includes at least one pre-packaged optimization module, and the optimization module includes an agent state masking module and a joint value function decomposition module.

[0152] The following will introduce the agent state masking module and the joint value function decomposition module in detail:

[0153] Module 1: Agent state masking module

[0154] Here, the agent state masking module is used to set the state parameters corresponding to the target agent that needs to perform agent masking operation as target parameters to represent that the target agent does not participate in the training process of the value network, wherein the state parameters are used for state evaluation through the value network.

[0155] In practical applications, since each intelligent agent may die before completing the target task in the application scenario (that is, no longer continue to perform the target task), if no corresponding processing is performed, it may affect the value network training process and thus affect the network accuracy of the value network.

[0156] Specifically, after detecting that any agent is in a dead state, the state parameter corresponding to the agent can be set to 0 (i.e., the target parameter), and the agent's identity can be concatenated with the zeroed state parameter to generate a vector for input into the policy network corresponding to the agent, to indicate that the agent is no longer involved in executing the target task, and the state parameters are all set to zero. The subsequent value network is also unable to complete the evaluation of the policy network corresponding to the agent, thereby completing the masking operation on the agent.

[0157] Module 2: Joint value function decomposition module

[0158] Here, the joint value network decomposition module is used to establish a local value network for each agent, and based on each local value network and a preset joint strategy, construct a combined hybrid network to train the strategy network based on the output of the hybrid network.

[0159] In practical applications, the joint value network is a value network corresponding to each of the above-mentioned agents. However, the use of the joint value network may not meet the task requirements in different scenarios. Based on this, a joint value function decomposition module can be added to the multi-agent reinforcement learning framework to make the multi-agent reinforcement learning framework applicable to more scenarios.

[0160] Specifically, for any agent, a local value network representing the local value function of the agent can be established using a recurrent neural network structure, wherein the input of the local value network is the local observation information at the current moment and the action performed by the agent at the previous moment, and the output is the evaluation value of the action performed by the agent at the previous moment. After establishing the local value networks corresponding to each agent, the local value networks can be combined using a preset combination strategy to obtain a hybrid network.

[0161] In a possible implementation, the network type of the hybrid network may be a feedforward neural network, the input of the hybrid network is the output of each local value network, the output of each local value network may be linearly mixed, and the output of the hybrid network may be activated by an absolute value activation function to obtain an evaluation value of the output of the hybrid network. The network parameters of the hybrid network (such as weights and other parameters) may be generated by a separate super network and restricted to linear functions that meet the constraints, thereby speeding up the training of the hybrid network and providing a reliable evaluation standard for the optimization of the subsequent policy network.

[0162] In a possible implementation, Figure 4 As shown, the policy network corresponding to the agent can also be trained by the following steps:

[0163] S401: Based on the optimization instruction, determine a first target optimization module for optimizing the value network.

[0164] S402: Optimizing the value network corresponding to the agent based on the first objective optimization module, and training the strategy network corresponding to the agent based on the optimized value network.

[0165] Here, the optimization instruction includes a target identifier of the optimization module, and according to the target identifier, the first target optimization module for optimizing the value network can be determined.

[0166] Specifically, the first objective optimization module can be the above-mentioned agent state masking module, and / or the joint value function decomposition module. Through the optimization processing of the first objective optimization module, the value network for training the policy network can be adapted to different application scenarios, so that each agent can better perform the target task.

[0167] In another possible implementation, after the policy network corresponding to the agent is trained, Figure 5 As shown, the policy network can also be optimized and trained by the following steps:

[0168] S501: Based on the target task, the trained intelligent agent is tested.

[0169] S502: Based on the test result of the target task, determine a second target optimization module for optimizing the value network.

[0170] Here, when testing the intelligent agent, the state of the intelligent agent when actually executing the target task can be simulated in a simulation environment, so that the corresponding evaluation parameters can be obtained, and the second target optimization module can be determined based on the corresponding relationship between the values ​​of the preset evaluation parameters and the optimization modules; or, the optimization instructions for the test results can be obtained to determine the target identifier in the optimization instructions and determine the second target optimization module.

[0171] S503: Based on the second objective optimization module, optimize the value network.

[0172] S504: Based on the optimized value network, optimize and train the strategy network corresponding to the agent.

[0173] Specifically, the second objective optimization module can be the above-mentioned agent state masking module, and / or the joint value function decomposition module. Through the optimization processing of the second objective optimization module, the value network for training the policy network can be adapted to different application scenarios, so that each agent can better perform the target task.

[0174] In practical applications, when using the multi-agent reinforcement learning framework for reinforcement learning, if the initialization parameters of the policy network are not set appropriately, the probability value distribution of each action output by the policy network may be too sharp, thereby limiting the degree of exploration of the environment by the agent when performing the target task.

[0175] In a possible implementation, Figure 6 As shown, the policy network can be optimized by the following steps:

[0176] S601: performing a truncation operation on the output value of the policy network to obtain a target output value of the output value that meets the requirements of a preset numerical range.

[0177] S602: Activate the target output value based on the activation function to instruct the agent to perform the target task according to the activated target output value.

[0178] Specifically, the output value of the policy network can be truncated to retain the target output value required by the preset numerical range, and the probability value can be activated using an activation function to obtain the category distribution corresponding to each action, so as to obtain the action that the agent needs to perform by sampling from the category distribution. The preset numerical range can be obtained according to the test simulation results, such as (-5, 2).

[0179] Furthermore, the network parameters of the policy network can be shared among the agents, that is, each agent uses the same network results and network parameters, but uses a unique hot vector constructed by the identifier corresponding to each agent to distinguish different agents when inputting the policy network, thereby improving the efficiency of using the policy network.

[0180] The reinforcement learning method for an intelligent agent provided in the embodiment of the present disclosure pre-builds an intelligent agent reinforcement learning framework, and can determine the state space and action space corresponding to the intelligent agent used to perform the target task based on the task requirements of the target task, and use the baseline algorithm in the intelligent agent reinforcement learning framework to centrally train the intelligent agent, so as to train the policy network corresponding to the intelligent agent according to the obtained value network corresponding to the intelligent agent, and obtain a policy network that can control the intelligent agent to perform the target task. In this way, the whole process of defining the state space and action space, the centralized training of the intelligent agent, and the training of the policy network can be completed conveniently, and through the pre-built intelligent agent reinforcement learning framework, the amount of code development in the whole process can be reduced, and corresponding expansion can be performed on the basis of the pre-built intelligent agent reinforcement learning framework, so as to provide developers with a complete and easy-to-use solution for solving multi-agent game problems, so as to improve the efficiency of intelligent agent reinforcement learning.

[0181] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0182] Based on the same inventive concept, the embodiments of the present disclosure also provide a reinforcement learning device for an intelligent agent corresponding to the reinforcement learning method for the intelligent agent. Since the principle of solving the problem by the device in the embodiments of the present disclosure is similar to the reinforcement learning method for the intelligent agent in the embodiments of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0183] Reference Figure 7 As shown, it is a schematic diagram of the architecture of a reinforcement learning device for an intelligent agent provided by an embodiment of the present disclosure, the device includes: an acquisition module 701, a first determination module 702, a second determination module 703, and a training module 704; wherein,

[0184] An acquisition module 701 is used to acquire the task requirements of a target task;

[0185] A first determination module 702 is used to determine the state space and action space corresponding to the agent for performing the target task based on the task requirements of the target task;

[0186] A second determination module 703 is used to perform centralized training on the agent based on the state space, the action space and the baseline algorithm in the pre-built agent reinforcement learning framework to determine the value network corresponding to the agent;

[0187] The training module 704 is used to train the policy network corresponding to the agent based on the value network corresponding to the agent, so as to control the agent to perform the target task based on the trained policy network.

[0188] In a possible implementation, the state type of the state space includes one or more of global observation information, local observation information, and mixed observation information; the action type of the action space includes one or more of discrete action, continuous action, and mixed action.

[0189] In a possible implementation, the agent reinforcement learning framework includes at least one pre-packaged functional module;

[0190] The training module 704, when training the strategy network corresponding to the agent based on the value network corresponding to the agent, is used to:

[0191] Based on the task requirements of the target task, determining a target functional module for executing the target task;

[0192] Based on the value network corresponding to the agent and the target function module, the strategy network corresponding to the agent is trained.

[0193] In a possible implementation manner, the target function module includes a first target function module and a second target function module:

[0194] The second target function module is an agent action masking module, which is used to mask the target type action output by the policy network so that the target type action is not executed by the agent;

[0195] The first objective function module is a normalization module, which is used to normalize the value function corresponding to the value network.

[0196] In a possible implementation, the training module 704, when training the policy network corresponding to the agent based on the value network corresponding to the agent and the target function module, is used to:

[0197] The value network corresponding to the agent is optimized based on the first objective function module, and the strategy network corresponding to the agent is trained based on the optimized value network.

[0198] In a possible implementation, the training module 704, when training the policy network corresponding to the agent based on the value network corresponding to the agent and the target function module, is used to:

[0199] For any intelligent agent, the strategy network of the intelligent agent is optimized based on the second objective function module;

[0200] Based on the optimized policy network, determine the output value of the value network corresponding to the agent, and determine the loss value corresponding to the policy network corresponding to the agent based on the output value of the value network;

[0201] The policy network corresponding to the agent is trained based on the loss value.

[0202] In one possible implementation, the input of the policy network is the local observation information of the corresponding intelligent agent, and the output is the probability value of the corresponding intelligent agent executing each action in the action space; the input of the value network is the global observation information of each intelligent agent, and the output is the evaluation value of each action or state of the intelligent agent.

[0203] In a possible implementation, the agent reinforcement learning framework includes at least one pre-packaged optimization module;

[0204] The training module 704, when training the strategy network corresponding to the agent based on the value network corresponding to the agent, is used to:

[0205] Based on the optimization instruction, determining a first target optimization module for optimizing the value network;

[0206] The value network corresponding to the agent is optimized based on the first target optimization module, and the strategy network corresponding to the agent is trained based on the optimized value network.

[0207] In a possible implementation, the agent reinforcement learning framework includes at least one pre-packaged optimization module;

[0208] After training the policy network corresponding to the agent, the training module 704 is further used to:

[0209] Based on the target task, testing the trained agent;

[0210] Based on the test result of the target task, determining a second target optimization module for optimizing the value network;

[0211] Based on the second objective optimization module, optimizing the value network;

[0212] Based on the optimized value network, the strategy network corresponding to the agent is optimized and trained.

[0213] In one possible implementation, the optimization module includes an agent state masking module, which is used to set the state parameters corresponding to the target agent that needs to perform agent masking operation as target parameters to characterize that the target agent does not participate in the training process of the value network, wherein the state parameters are used for state evaluation through the value network.

[0214] In one possible implementation, the optimization module includes a joint value function decomposition module, which is used to establish a local value network for each intelligent agent, and based on each local value network and a preset joint strategy, construct a combined hybrid network to train the strategy network based on the output of the hybrid network.

[0215] In a possible implementation manner, the training module 704 is further configured to optimize the policy network according to the following steps:

[0216] Performing a truncation operation on the output value of the policy network to obtain a target output value in the output value that meets the requirements of a preset numerical range;

[0217] The target output value is activated based on the activation function to instruct the agent to perform the target task according to the target output value after the activation process.

[0218] The reinforcement learning device for an intelligent agent provided in the embodiment of the present disclosure has a pre-built intelligent agent reinforcement learning framework. Based on the task requirements of the target task, the state space and action space corresponding to the intelligent agent used to perform the target task can be determined, and the baseline algorithm in the intelligent agent reinforcement learning framework can be used to centrally train the intelligent agent, so as to train the policy network corresponding to the intelligent agent according to the obtained value network corresponding to the intelligent agent, and obtain a policy network that can control the intelligent agent to perform the target task. In this way, the whole process of defining the state space and action space, the centralized training of the intelligent agent, and the training of the policy network can be completed. Through the pre-built intelligent agent reinforcement learning framework, the amount of code development in the whole process can be reduced, and corresponding expansion can be performed on the basis of the pre-built intelligent agent reinforcement learning framework, so as to provide developers with a complete and easy-to-use solution for solving multi-agent game problems, so as to improve the efficiency of intelligent agent reinforcement learning.

[0219] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0220] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Figure 8 As shown, it is a schematic diagram of the structure of a computer device 800 provided in an embodiment of the present disclosure, including a processor 801, a memory 802, and a bus 803. Among them, the memory 802 is used to store execution instructions, including a memory 8021 and an external memory 8022; the memory 8021 here is also called an internal memory, which is used to temporarily store the operation data in the processor 801, and the data exchanged with the external memory 8022 such as a hard disk. The processor 801 exchanges data with the external memory 8022 through the memory 8021. When the computer device 800 is running, the processor 801 communicates with the memory 802 through the bus 803, so that the processor 801 executes the following instructions:

[0221] Obtain the task requirements of the target task;

[0222] Based on the task requirements of the target task, determining the state space and action space corresponding to the agent for performing the target task;

[0223] Centrally training the agent based on the state space, the action space, and a baseline algorithm in a pre-built agent reinforcement learning framework to determine a value network corresponding to the agent;

[0224] Based on the value network corresponding to the agent, the policy network corresponding to the agent is trained to control the agent to perform the target task based on the trained policy network.

[0225] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the agent reinforcement learning method described in the above method embodiment are executed. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0226] The present disclosure also provides a computer program product that carries a program code. The program code includes instructions that can be used to execute the steps of the reinforcement learning method for the intelligent agent described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0227] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0228] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0229] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0230] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0231] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0232] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A reinforcement learning method for an intelligent agent, characterized in that: include: Obtaining task requirements of a target task; wherein the target task is to enable an artificial intelligence player to win a human-computer game, and the task requirements of the target task are to enable the artificial intelligence player to perform game operations; Based on the task requirements of the target task, determine the state space and action space corresponding to the agent used to perform the target task; wherein the agent is used to control the artificial intelligence player to play a game with a real player; the state type of the state space includes one or more of global observation information, local observation information and mixed observation information, the local observation information is the information observed by a single agent in the game scene, and the mixed observation information is the information observed by multiple agents in the game scene; the action type of the action space is used to indicate that the game action performed by the agent is a discrete action or a continuous action; the action type of the action space includes one or more of discrete actions, continuous actions and mixed actions; Centrally training the agent based on the state space, the action space, and a baseline algorithm in a pre-built agent reinforcement learning framework to determine a value network corresponding to the agent; Based on the value network corresponding to the agent, the policy network corresponding to the agent is trained to control the agent to perform the target task based on the trained policy network.

2. The method according to claim 1, characterized in that The agent reinforcement learning framework includes at least one pre-packaged functional module; The step of training the strategy network corresponding to the agent based on the value network corresponding to the agent includes: Based on the task requirements of the target task, determining a target functional module for executing the target task; Based on the value network corresponding to the agent and the target function module, the strategy network corresponding to the agent is trained.

3. The method according to claim 2, characterized in that The target function module includes a first target function module and a second target function module: The second target function module is an agent action masking module, which is used to mask the target type action output by the policy network so that the target type action is not executed by the agent; The first objective function module is a normalization module, which is used to normalize the value function corresponding to the value network.

4. The method according to claim 2 or 3, characterized in that: The step of training the strategy network corresponding to the agent based on the value network corresponding to the agent and the target function module includes: The value network corresponding to the agent is optimized based on the first objective function module, and the strategy network corresponding to the agent is trained based on the optimized value network.

5. The method according to claim 2 or 3, characterized in that: The step of training the strategy network corresponding to the agent based on the value network corresponding to the agent and the target function module includes: For any intelligent agent, the strategy network of the intelligent agent is optimized based on the second objective function module; Based on the optimized policy network, determine the output value of the value network corresponding to the agent, and determine the loss value corresponding to the policy network corresponding to the agent based on the output value of the value network; The policy network corresponding to the agent is trained based on the loss value.

6. The method according to claim 1, characterized in that The input of the strategy network is the local observation information of the corresponding intelligent agent, and the output is the probability value of the corresponding intelligent agent executing each action in the action space; the input of the value network is the global observation information of each intelligent agent, and the output is the evaluation value of each action or state of the intelligent agent.

7. The method according to claim 1, characterized in that The agent reinforcement learning framework includes at least one pre-packaged optimization module; The step of training the strategy network corresponding to the agent based on the value network corresponding to the agent includes: Based on the optimization instruction, determining a first target optimization module for optimizing the value network; The value network corresponding to the agent is optimized based on the first target optimization module, and the strategy network corresponding to the agent is trained based on the optimized value network.

8. The method according to claim 1, characterized in that The agent reinforcement learning framework includes at least one pre-packaged optimization module; After training the policy network corresponding to the agent, the method further includes: Based on the target task, testing the trained agent; Based on the test result of the target task, determining a second target optimization module for optimizing the value network; Based on the second objective optimization module, optimizing the value network; Based on the optimized value network, the strategy network corresponding to the agent is optimized and trained.

9. The method according to claim 7 or 8, characterized in that: The optimization module includes an agent state masking module, which is used to set the state parameters corresponding to the target agent that needs to perform agent masking operation as target parameters to represent that the target agent does not participate in the training process of the value network, wherein the state parameters are used for state evaluation through the value network.

10. The method according to claim 7 or 8, characterized in that: The optimization module includes a joint value function decomposition module, which is used to establish a local value network for each intelligent agent, and based on each local value network and a preset joint strategy, construct a combined hybrid network to train the strategy network based on the output of the hybrid network.

11. The method according to claim 1, characterized in that The method further comprises optimizing the policy network according to the following method: Performing a truncation operation on the output value of the policy network to obtain a target output value in the output value that meets the requirements of a preset numerical range; The target output value is activated based on the activation function to instruct the agent to perform the target task according to the target output value after the activation process.

12. A reinforcement learning device for an intelligent agent, characterized in that: include: An acquisition module, used to acquire the task requirements of a target task; wherein the target task is to enable an artificial intelligence player to win a human-computer game, and the task requirements of the target task are to enable the artificial intelligence player to perform game operations; A first determination module is used to determine the state space and action space corresponding to the agent for performing the target task based on the task requirements of the target task; The second determination module is used to perform centralized training on the agent based on the state space, action space and the baseline algorithm in the pre-built agent reinforcement learning framework to determine the value network corresponding to the agent; wherein the agent is used to control the artificial intelligence player to play a game with a real player; the state type of the state space includes one or more of global observation information, local observation information and mixed observation information, the local observation information is the information observed by a single agent in the game scene, and the mixed observation information is the information observed by multiple agents in the game scene; the action type of the action space is used to indicate that the game action performed by the agent is a discrete action or a continuous action; the action type of the action space includes one or more of discrete actions, continuous actions and mixed actions; A training module is used to train the strategy network corresponding to the agent based on the value network corresponding to the agent, so as to control the agent to perform the target task based on the trained strategy network.

13. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the reinforcement learning method for an intelligent agent as described in any one of claims 1 to 11 are performed.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the reinforcement learning method for an intelligent agent as described in any one of claims 1 to 11.

Images