Multi-agent control method and device for changing environment based on reinforcement learning

Abstract

The present invention provides a method and device for controlling multi-agents in a changing environment based on reinforcement learning, including: inputting current environment information and the implicit state of each agent into the agent network to obtain the observed action value of each agent output by the agent network; the observed action value is used to characterize the estimated value of all actions currently performed by the corresponding agent; based on the observed action value of each agent, each agent is controlled to perform actions; the agent network is obtained by reinforcement learning based on sample environment information and sample implicit states of sample agents in combination with a self-weighted network; the self-weighted network is used to determine the joint action value of actions performed by all sample agents; in the reinforcement learning process, the input of the self-weighted network includes the sample observed action values ​​of all sample agents output by the agent network. The method and device provided by the present invention can avoid the influence of the dynamic change of the number of agents in a changing environment on the multi-agent control and training process.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for controlling multiple intelligent agents in a changing environment based on reinforcement learning. Background Art

[0002] In recent years, with the development of deep reinforcement learning algorithms in the field of multi-agent control, cooperative multi-agent reinforcement learning algorithms have become an important method for artificial intelligence control problems in multi-unit combat tasks. Cooperative multi-agent reinforcement learning means that multiple agents can cooperate with each other, continuously learn from the environment, continuously improve their own behavior, and gradually approach the ultimate collective goal.

[0003] A key issue in cooperative multi-agent reinforcement learning is the impact of complex changing environments on training. In actual control, the number of agents in a multi-agent system will change dynamically. Taking a multi-unit battle game as an example, the number of agents will decrease due to enemy attacks. The dynamic change in the number of agents will affect the calculation process of the multi-agent system, reducing the convergence speed during training and the performance of the strategy after convergence. Summary of the invention

[0004] The present invention provides a method and device for controlling a multi-agent system in a changing environment based on reinforcement learning, so as to solve the defect that the prior art cannot cope with the changing environment and avoid the influence of the changing environment on the multi-agent system.

[0005] The present invention provides a multi-agent control method in a changing environment based on reinforcement learning, comprising:

[0006] Input the current environment information and the implicit state of each agent into the agent network to obtain the observed action value of each agent output by the agent network; the observed action value is used to represent the estimated value of all actions currently performed by the corresponding agent;

[0007] Based on the observed action value of each agent, control each agent to perform actions;

[0008] Among them, the intelligent agent network is obtained by reinforcement learning based on sample environment information and sample implicit states of sample agents in conjunction with a self-weighted network; the self-weighted network is used to determine the joint action value of actions performed by all sample agents; in the reinforcement learning process, the input of the self-weighted network includes the sample observation action values ​​of all sample agents output by the intelligent agent network.

[0009] According to the reinforcement learning-based changing environment multi-agent control method provided by the present invention, the agent network includes multiple flow networks, each flow network corresponds to an action subspace in the action space of the agent; the flow network is used to determine the estimated value of the action currently executed by the agent in the corresponding action subspace; the action subspace is divided according to the function of the action, and different action subspaces have no intersection; the observed action value includes the estimated value of all actions in the action space.

[0010] According to the multi-agent control method for a changing environment based on reinforcement learning provided by the present invention, the joint action value is determined based on the following steps:

[0011] Based on the mapping parameters of the self-weighted network, nonlinear mapping is performed on the sample observation action values ​​of each sample intelligent agent to obtain the value mapping results of each sample intelligent agent;

[0012] The joint action value is determined based on the value mapping results of each sample agent and the value weight of each sample agent in the self-weighted network.

[0013] According to the multi-agent control method for a changing environment based on reinforcement learning provided by the present invention, the value weight of each sample agent in the self-weighted network is determined based on the following steps:

[0014] Based on the encoding parameters in the self-weighted network, the local observation information of each sample agent and the global state of the sample environment are encoded respectively to obtain a first intermediate variable and a second intermediate variable;

[0015] Based on the first intermediate variable and the second intermediate variable, the value weight of each sample intelligent agent is determined.

[0016] According to the multi-agent control method for a changing environment based on reinforcement learning provided by the present invention, the agent network is determined based on the following steps:

[0017] Based on a plurality of randomly selected groups of transfer history data and a preset loss function, jointly training the initial agent network and the initial self-weighted network to obtain the agent network;

[0018] The joint training process includes multiple training stages, and the update target of the loss function of any training stage is calculated based on the stage self-weighted network and the stage agent network when the previous training stage of any training stage is completed; each group of transfer history data includes the joint observations, joint actions and the global state of the sample environment at each historical moment before each joint training; the joint observations and joint actions respectively represent the local observation information and execution actions of all sample agents.

[0019] According to the multi-agent control method for a changing environment based on reinforcement learning provided by the present invention, the update target is determined based on the following steps:

[0020] Determine a joint action value at a next historical moment based on a joint observation at a next historical moment, the stage agent network, and the stage self-weighted network;

[0021] The update target is determined based on the joint reward obtained from the sample environment and the joint action value.

[0022] According to the multi-agent control method for a changing environment based on reinforcement learning provided by the present invention, the sample observed action value includes the sample estimated value of all actions performed by the sample agent;

[0023] The determining of the joint action value at the next historical moment based on the joint observation at the next historical moment, the stage agent network, and the stage self-weighted network includes:

[0024] Based on the joint observation at the next historical moment and the agent network at the stage, an estimated target value of all sample agents at the next historical moment is obtained; the estimated target value is the maximum value of all sample estimated values ​​of the corresponding sample agents;

[0025] Based on the estimated target value and the stage self-weighted network, the joint action value is obtained.

[0026] The present invention also provides a multi-agent control device in a changing environment based on reinforcement learning, comprising:

[0027] An input module is used to input the current environment information and the implicit state of each agent into the agent network to obtain the observed action value of each agent output by the agent network; the observed action value is used to represent the estimated value of all actions currently performed by the corresponding agent;

[0028] A control module is used to control each agent to perform actions based on the observed action value of each agent;

[0029] Among them, the intelligent agent network is obtained by reinforcement learning based on sample environment information and sample implicit states of sample agents in conjunction with a self-weighted network; the self-weighted network is used to determine the joint action value of actions performed by all sample agents; in the reinforcement learning process, the input of the self-weighted network includes the sample observation action values ​​of all sample agents output by the intelligent agent network.

[0030] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the multi-agent control method for a changing environment based on reinforcement learning as described above are implemented.

[0031] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described methods for controlling multi-agents in a changing environment based on reinforcement learning.

[0032] The reinforcement learning-based changing environment multi-agent control method and device provided by the present invention are obtained by combining the agent network with a self-weighted network for reinforcement learning, so that the agent network can learn the cooperation strategy, thereby realizing distributed control of the agent execution actions based on the agent network, and the self-weighted network is used to determine the joint action value of all sample agents executing actions, thereby avoiding the influence of the dynamic change of the number of agents in the changing environment on the training process, and ensuring the convergence speed during training and the strategy performance after convergence. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 It is a flow chart of the multi-agent control method in a changing environment based on reinforcement learning provided by the present invention;

[0035] Figure 2 It is a structural diagram of the intelligent agent network provided by the present invention;

[0036] Figure 3 It is a schematic diagram of the structure of the self-weighted network provided by the present invention;

[0037] Figure 4 It is a flow chart of the method for determining an intelligent agent network based on reinforcement learning provided by the present invention;

[0038] Figure 5 is a curve showing the change of the winning rate of the multi-agent system provided by the present invention in a battle with the number of training steps;

[0039] Figure 6 It is a structural schematic diagram of a multi-agent control device for a changing environment based on reinforcement learning provided by the present invention;

[0040] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The centralized training distributed execution framework is widely used in cooperative multi-agent reinforcement learning algorithms. Under this framework, multiple agents in the system are treated as a whole during training, and the agent strategy is obtained by optimizing the joint action value function of the entire system; in actual execution, the agents execute the strategy in a distributed manner without performing additional operations such as communication. When the observation range and communication capabilities are limited, the algorithm based on this framework has stronger versatility and better combat performance.

[0043] In this regard, an embodiment of the present invention provides a multi-agent control method in a changing environment based on reinforcement learning based on a centralized training distributed execution framework. Figure 1 is a flow chart of the multi-agent control method in a changing environment based on reinforcement learning provided by the present invention, such as Figure 1 As shown, the method includes:

[0044] Step 110, input the current environment information and the implicit state of each agent into the agent network, and obtain the observed action value of each agent output by the agent network; the observed action value is used to represent the estimated value of all actions currently performed by the corresponding agent;

[0045] Step 120, controlling each agent to perform an action based on the observed action value of each agent;

[0046] Among them, the agent network is obtained by reinforcement learning based on the sample environment information and the sample implicit state of the sample agent in conjunction with the self-weighted network; the self-weighted network is used to determine the joint action value of the actions performed by all sample agents; in the reinforcement learning process, the input of the self-weighted network includes the sample observation action values ​​of all sample agents output by the agent network.

[0047] Specifically, there are multiple friendly units and multiple enemy units in a multi-unit battle task. In order to coordinate and control multiple friendly units to achieve the final victory in the battle, the embodiment of the present invention applies each agent to each friendly unit, determines the observed action value of each agent based on the agent network, and then determines the action to be performed by each agent based on the observed action value of each agent, and then controls each agent to perform the determined action in the environment. Here, the observed action value is used to represent the estimated value of all actions currently performed by the corresponding agent, and all actions may include moving up, moving right, attacking, etc.

[0048] In order to enable each agent to execute strategies in a distributed manner when the observation range and communication capabilities are limited, the embodiment of the present invention only inputs the current environment information and the implicit state of each agent into the agent network, without the need for the agent to communicate and share information with other agents, and can obtain the observed action value of each agent output by the agent network, which is used for subsequent individual control of each agent. Here, the current environment information may include the normalized life, shield, relative coordinates and other information of all units in the current environment, and the implicit state of each agent may be, for example, the execution action and identity information of the previous moment.

[0049] In addition, before executing step 110, it is necessary to pre-train the agent network. Specifically, the agent network can be trained in the following manner: First, a large amount of sample environment information and the sample implicit states of all sample agents in each sample environment are collected. Then, in order to maximize the joint benefits obtained by multiple agents in a cooperative relationship, reinforcement learning is performed based on the sample environment information and the sample implicit states of the sample agents, and in conjunction with the self-weighted network, to obtain the agent network. Here, the self-weighted network can be used to determine the joint action value generated by the actions performed by all sample agents in each sample environment.

[0050] Considering that in multi-unit combat tasks, the operating environment of multiple agents is usually dynamically changing. For example, the number of agents is reduced due to the attack of enemy units, and the dynamic change of the number of agents will affect the calculation of the joint action value of the multi-agent system, reducing the convergence speed in training and the performance of the strategy after convergence. To address this problem, in the joint training process, the embodiment of the present invention uses the sample observation action value of all sample agents in each sample environment output by the agent network as the input of the self-weighted network. At this time, if the number of sample agents in the sample environment changes, the output dimension of the agent network will also change accordingly, thereby causing the input dimension of the self-weighted network to change dynamically with the number of agents, thereby ensuring the accuracy of the joint action value calculated by the self-weighted network.

[0051] For example, if the number of sample agents in the sample environment is reduced from 6 to 5, the agent network only outputs the sample observation action values ​​of the 5 sample agents, and inputs the sample observation action values ​​of these 5 sample agents into the self-weighted network, so that the self-weighted network only needs to calculate the joint action value of these 5 sample agents.

[0052] The method provided by the embodiment of the present invention is obtained by reinforcement learning through the intelligent agent network in conjunction with the self-weighted network, so that the intelligent agent network can learn the cooperation strategy, thereby realizing distributed control of the intelligent agent execution action based on the intelligent agent network, and the self-weighted network is used to determine the joint action value of all sample intelligent agents to perform actions, thereby avoiding the impact of the dynamic change of the number of agents in the changing environment on the training process, and ensuring the convergence speed during training and the strategy performance after convergence.

[0053] Based on any of the above embodiments, the agent network includes multiple flow networks, each flow network corresponds to an action subspace in the action space of the agent; the flow network is used to determine the estimated value of the action currently executed by the agent in the corresponding action subspace; the action subspace is divided according to the function of the action, and different action subspaces have no intersection; the observed action value includes the estimated value of all actions in the action space.

[0054] Specifically, considering that in a multi-agent operating environment, in addition to changes in the number of agents, the action space of the agents will also change dynamically. For example, the number of enemy units decreases due to the attack of the agents, which reduces the number of targets that the agents can attack, that is, the reduction of the action space of the agents. The dynamic changes in the action space of the agents will affect the calculation of the observed action value of the agents, reducing the speed of convergence during training and the performance of the strategies after convergence.

[0055] In this regard, the embodiment of the present invention divides the action space of the agent into multiple action subspaces according to the function of the agent's action, and there is no intersection between different action subspaces. On this basis, the agent network is set to include multiple flow networks, each flow network corresponds to an action subspace, which is used to evaluate the estimated value of the action currently executed by the agent in the corresponding action subspace, so that each flow network can be adaptively adjusted according to the dynamic changes in the corresponding action subspace, without interfering with each other, thereby avoiding the impact of the changing environment on the agent network. After the estimated value of all actions in the action space is obtained through each flow network, the observed action value output by the agent network can be obtained.

[0056] Furthermore, the action space can be divided into a unit-oriented attack action subspace and an environment-oriented movement action subspace, and the agent network provides corresponding flow networks to evaluate the value of actions in the action subspace, that is, the attack flow network is used to evaluate the estimated attack value of the agent's attack action on all enemy units in the current environment, and the movement flow network is used to evaluate the estimated movement value of the agent's current movement action. Here, the movement action may include the agent's movement action in all directions, and the embodiment of the present invention does not specifically limit this.

[0057] On this basis, if the number of enemy units in the current environment changes, the attack action subspace of the agent will change, while the mobile action subspace will remain unchanged. Then the output dimension of the attack flow network will also change, while the output dimension of the mobile flow network will remain unchanged. For example, if the number of enemy units in the current environment decreases from 5 to 3, the attack action subspace of the agent will decrease. Then the attack flow network will only output the estimated attack value of each agent performing an attack action on 3 enemy units, while the output dimension of the mobile flow network will still be the estimated mobile value of each agent performing a mobile action.

[0058] Based on any of the above embodiments, local observation information of the agent is obtained according to the multi-agent operating environment, including current environment information, the agent's execution action at the previous moment, and the agent's identity information, and then, the observation information related to each action subspace is determined as the input of the corresponding flow network. For the i-th agent, the observation information related to the two action subspaces is the environment-oriented observation and the unit-oriented observation, wherein the environment-oriented observation The information included includes: normalized life, shield, relative coordinates and other information of all units in the current environment, the actions performed by the agent at the last moment and the identity of the agent; observations of the unit (i→j),t The information included is the normalized health, shield, relative coordinates, etc. of the attack target, i.e. the enemy unit j.

[0059] Figure 2 is a schematic diagram of the structure of the intelligent agent network provided by the present invention, such as Figure 2 As shown in Figure 1, the agent network includes an environment-oriented flow network, namely the mobile flow network, and a unit-oriented flow network, namely the attack flow network. The mobile flow network uses three fully connected layers: With a Gated Recurrent Unit (GRU) layer The subscript 1 indicates that it contains only one GRU layer, and the superscript e indicates that the network belongs to the environment-oriented flow network. The calculation formula is as follows:

[0060] z i,t+1 =σ(W z ·[hi,t ,x i,t ])

[0061] r i,t+1 =σ(W r ·[h i,t ,x i,t ])

[0062]

[0063]

[0064] Among them, h i,t represents the implicit state of the agent that requires additional input to the GRU layer, x i,t Represents observations towards the environment Through the first fully connected layer The calculated eigenvector; z i,t is the update gate of the GRU layer, r i,t is the reset gate of the GRU layer, is the candidate hidden layer of the GRU layer, and finally the hidden state h at the next moment is calculated by the above formula i,t+1 ; This hidden state is used in the GRU layer calculation at the next moment on the one hand, and as the second fully connected layer at time t on the other hand The input is used to calculate the estimated movement value of the movement action; the operator [x, y] represents the serial concatenation operation of two vectors x and y, and σ(·) represents the Sigmoid nonlinear activation function, which is calculated as follows:

[0065]

[0066] Tanh(·) is the hyperbolic tangent nonlinear activation function, and its calculation formula is as follows:

[0067]

[0068] The attack flow network uses two fully connected layers, where the input of the second fully connected layer consists of two parts: The output of the first fully connected layer in the attack flow network The output of the agent network can be 64 neurons in each network.

[0069] Based on any of the above embodiments, for the i-th agent, the method of controlling its execution of actions is:

[0070] First, the local observation information of the agent is obtained according to the multi-agent operating environment; the implicit state of its own output at the previous moment is obtained, and the two are used as the input of the agent network to calculate the estimated value of all actions performed by the agent in the mobile action subspace and the attack action subspace, that is, the observed action value Q i (o i,t ,u;θ i ),u∈U i,t ;

[0071] Among them, θ i is the parameter of the agent network, U i,t is the action space of agent i at time t. Preferably, all agents in the multi-agent system share the same agent network parameters, i.e., θ1 = ... = θ n ;

[0072] According to the observed action value, an action is selected from the action space according to the action selection strategy to control the agent to execute in the multi-agent operating environment; preferably, the action selection strategy of the agent is an ε-greedy strategy:

[0073]

[0074] Among them, u i,t is the action selected by the ith agent at time t, N ui,t is the number of optional actions available to the agent at the current moment, ε is the exploration rate, random() is the random number function, and randint() is the random integer function.

[0075] Based on any of the above embodiments, the joint action value is determined based on the following steps:

[0076] Based on the mapping parameters of the self-weighted network, the sample observation action values ​​of each sample agent are nonlinearly mapped to obtain the value mapping results of each sample agent;

[0077] Based on the value mapping results of each sample agent and the value weight of each sample agent in the self-weighted network, the joint action value is determined.

[0078] Specifically, considering the reputation allocation problem in cooperative multi-agent reinforcement learning, that is, in a cooperative scenario, the multi-agent system can only obtain a joint reward from the environment to evaluate its joint actions. The algorithm needs to use a reasonable reputation allocation method to evaluate the contribution of each agent's actions to the entire system based on the reward. Through reasonable reputation allocation, agents should be able to cooperate to maximize the joint reward.

[0079] Therefore, in order to solve the credit distribution problem, the embodiment of the present invention provides an efficient network structure to represent the joint action value function of the multi-agent system, that is, using a self-weighted network to nonlinearly map the sample observation action value of each sample agent, and calculate its value weight in the joint action value, and perform weighted summation on the mapped sample observation action value, so as to obtain the joint action value generated by all sample agents in the sample environment. On this basis, the agent network and the self-weighted network are jointly trained, so that the agent network can learn the action strategy that maximizes the joint action value, thereby enabling the agent to perform actions that maximize the joint reward.

[0080] Here, the value weight assigned by the self-weighting network to each sample agent can be directly determined based on the local observation information of each sample agent and the global state of the sample environment, or it can be determined based on the encoding result after first encoding the local observation information of each sample agent and the global state of the sample environment. The mapped sample observation action value is the value mapping result of each sample agent in the embodiment of the present invention. The nonlinear mapping method can be based on the mapping parameters of the self-weighting network, and the sample observation action value of each sample agent is nonlinearly mapped. The mapping parameters can be derived from one or more neural networks inside the self-weighting network, and the embodiment of the present invention does not specifically limit this.

[0081] Based on any of the above embodiments, the value weight of each sample agent in the self-weighted network is determined based on the following steps:

[0082] Based on the encoding parameters in the self-weighted network, the local observation information of each sample agent and the global state of the sample environment are encoded respectively to obtain a first intermediate variable and a second intermediate variable;

[0083] Based on the first intermediate variable and the second intermediate variable, the value weight of each sample intelligent agent is determined.

[0084] Specifically, the value weight of each sample agent in the self-weighted network can be obtained in the following manner: first, the local observation information of each sample agent and the global state of the sample environment are encoded respectively using the encoding parameters in the self-weighted network, so as to obtain the first intermediate variable and the second intermediate variable respectively; here, the encoding parameters used for encoding the local observation information and the global state can be encoding parameters from the same neural network or encoding parameters from different neural networks, which is not specifically limited in the embodiment of the present invention;

[0085] Then, based on the encoded first intermediate variable and the second intermediate variable, the value weight of each sample intelligent agent can be determined for calculation of the subsequent joint action value; here, the value weight can be determined by simple combinations such as concatenating and adding the first intermediate variable and the second intermediate variable, or by simply combining the first intermediate variable and the second intermediate variable and then encoding the combination result.

[0086] Based on any of the above embodiments, the self-weighted network is used to calculate the joint action value Q(o) of the multi-agent system. k ,u k θ joint ), the network estimates the multi-agent system under joint observation o by inputting the sample observation action value of all sample agents. t ={o 1,t ,...,o n,t} to perform the joint action u t = {u 1,t ,...,u n,t}Expected cumulative discount rewards

[0087] In this embodiment, the self-weighted network takes the sample observation action value Q of all sample agents as i (o i,t ,u;θ i ), i∈[1,n] as the main input to jointly observe o t ={o 1,t ,...,o n,t} as auxiliary input. Since this network only appears during training, the global state of the sample environment can be used as the input of a set of hypernetworks to calculate all weights and bias terms in the self-weighted network. The number of neurons in the hypernetwork can be set to 64.

[0088] Figure 3 It is a schematic diagram of the structure of the self-weighted network provided by the present invention, such as Figure 3 As shown in Figure 2, the network uses a flexible calculation method to perform nonlinear mapping on the sample observation action value of each sample agent to obtain q′ i,t , and calculate its value weight k in the joint action value i,t For the i-th sample agent, its sample observation action value Q i (o i,t ,u;θ i ) through two neural networks and Get the value mapping result q′ i,t, the number of neurons in the two networks can be set to 32. The activation function of the first network uses the Exponential Linear Unit (ELU), and its calculation formula is as follows:

[0089]

[0090] Wherein, α is a hyperparameter that controls the slope and is set to 1 in this embodiment.

[0091] Local observation information i,t Through the neural network NN o Calculate the intermediate variables With the global state s t Through neural network NN s The calculated intermediate variable h s Perform serial splicing operations, and pass through a neural network [W k ,b k ] Calculate the value weight k i,t .

[0092] The self-weighted network calculates the value mapping result and the corresponding value weight based on the two parts of the input, and calculates the joint action value of the multi-agent system according to the following formula:

[0093]

[0094] Among them, v t is the global state s t The global bias term calculated by a fully connected neural network.

[0095] Based on any of the above embodiments, the agent network is determined based on the following steps:

[0096] Based on multiple groups of randomly selected transfer history data and a preset loss function, the initial agent network and the initial self-weighted network are jointly trained to obtain an agent network;

[0097] The joint training process includes multiple training stages, and the update target of the loss function of any training stage is calculated based on the stage self-weighted network and the stage agent network when the previous training stage of any training stage is completed; each group of transfer history data includes the joint observations, joint actions and the global state of the sample environment at each historical moment before each joint training; the joint observations and joint actions respectively represent the local observation information and execution actions of all sample agents.

[0098] Specifically, in order to obtain faster convergence speed and better final performance, the intelligent agent network can be trained in the following way: first, by initializing the network parameters, the initial intelligent agent network and the initial self-weighted network are obtained, and then the initial intelligent agent network and the initial self-weighted network are jointly iteratively trained; considering that the sample agents will continuously perform actions during the iterative training process, the local observation information and execution actions of all sample agents in the sample environment at each moment and the global state of the sample environment can be recorded, and then multiple groups are randomly selected from all the recorded transfer history data as training samples for a new round of iterative training, and a new round of joint iterative training is carried out on this basis until the termination condition of the iterative training is reached, and the final intelligent agent network can be obtained.

[0099] In addition, the joint training process may include multiple training stages, and each training stage may be trained using a different loss function, that is, the update target of the loss function of any training stage may be calculated based on the stage self-weighted network and the stage agent network obtained when the previous training stage of the training stage is completed. For example, when the joint training process proceeds to the second training stage, since the initial agent network and the initial self-weighted network are updated to the stage self-weighted network and the stage agent network when the first training stage is completed, the update target of the loss function of the second training stage may be calculated based on the stage self-weighted network and the stage agent network.

[0100] Based on any of the above embodiments, the update target is determined based on the following steps:

[0101] Determine the joint action value at the next historical moment based on the joint observation at the next historical moment, the stage agent network, and the stage self-weighted network;

[0102] Based on the joint reward obtained from the sample environment and the joint action value, the update target is determined.

[0103] Specifically, for each joint training, multiple groups of transfer history data are randomly selected as multiple training samples, and each group of transfer history data corresponds to an update target. Since each group of transfer history data includes a joint observation, joint action, and global state of the sample environment at a historical moment, the selected multiple groups of transfer history data can be arranged in chronological order. When calculating the update target corresponding to any group of transfer history data, the corresponding joint observation can be taken from the next group of transfer history data, that is, the joint observation at the next historical moment, and then the joint action value at the next historical moment is determined based on the joint observation, the stage agent network, and the stage self-weighted network. Then, based on the joint action value and the joint reward obtained from the sample environment, the update target corresponding to the group of transfer history data is obtained.

[0104] Based on any of the above embodiments, the sample observed action value includes the sample estimated value of all actions performed by the sample agent;

[0105] Based on the joint observation of the next historical moment, the stage agent network, and the stage self-weighted network, the joint action value of the next historical moment is determined, including:

[0106] Based on the joint observation of the next historical moment and the stage agent network, the estimated target value of all sample agents at the next historical moment is obtained; the estimated target value is the maximum value of all sample estimated values ​​of the corresponding sample agent;

[0107] Based on the estimated target value and the stage self-weighted network, the joint action value is obtained.

[0108] Specifically, the joint action value at the next historical moment can be obtained in the following way: first, the local observation information of all sample agents at the next historical moment can be determined based on the joint observation at the next historical moment, and the sample observation action value of all sample agents at the next historical moment can be obtained by inputting it into the stage agent network. Since the sample observation action value includes the sample estimated value corresponding to all actions performed by the corresponding sample agent, the maximum value can be determined from all the sample estimated values ​​corresponding to each sample agent as the estimated target value of each sample agent at the next historical moment; on this basis, the estimated target value of all sample agents at the next historical moment can be input into the stage self-weighted network to obtain the joint action value.

[0109] Based on any of the above embodiments, Figure 4 is a flow chart of the method for determining an intelligent agent network based on reinforcement learning provided by the present invention, such as Figure 4 As shown, the method comprises the following steps:

[0110] Step S100: Based on the multi-agent operating environment, define the local observation information of each agent in the multi-agent system. i,t Initialize the parameters, experience pool and exploration rate of each network: Initialize the agent network parameters θ i and the parameters of the agent network at each stage The network structures of the two are the same; the self-weighted network parameters θ are initialized joint and the parameters of the self-weighted network in the stage The network structures of the two are the same; the experience pool D and the exploration rate ε are initialized; where i is the agent number. This step is the initialization step of the algorithm.

[0111] In this embodiment, the multi-agent operating environment is a multi-unit battle environment. For the i-th agent, the action space can be divided into a mobile action subspace facing the environment in four directions and an attack action subspace facing the unit against a specific enemy unit. Correspondingly, there are observation subspaces facing the environment. and the observation u facing the enemy unit j (i→j),t The operating environment obtains the next state of each unit in the scene according to the joint action selected by the multi-agent system, and feeds back the joint reward to the multi-agent system. The joint reward includes immediate reward and final reward.

[0112] The immediate reward is the difference in damage between the enemy and us. If our unit causes damage to the enemy unit, the environment will feedback a positive reward equal to the damage value as the multi-agent system; if the enemy unit causes damage to our unit, the environment will feedback a negative joint reward equal to the opposite of the damage value as the multi-agent system. The final reward reflects the outcome of the battle. If we win, the environment will feedback a positive joint reward, and if we lose, the environment will feedback a negative joint reward. In this embodiment, the specific value of the final reward is set to + / -10.

[0113] The agent network is used to calculate the observed action value of the agent in a distributed manner. Each agent network controls a self-unit in the environment and selects an appropriate action for the unit by inputting the local observation information of the unit. The self-weighted network is used to characterize the joint observation action value function of the multi-agent system. In practice, the multi-agent system uses the joint action value function to describe the expected cumulative reward that the system can obtain by performing joint actions under joint observations.

[0114] Step S200, based on the agent network and action selection strategy, the action of each agent is obtained according to the local observation information of the agent and executed in the environment.

[0115] Each agent in the multi-agent system selects actions for the units it controls in a distributed manner based on its local observation information and action selection strategy. The multi-unit combat environment collects and executes all actions and feeds back the joint observations of the next moment to the multi-agent system. i,t+1 Combined with the reward r in step S100 t+1 .

[0116] Since the present invention adopts a centralized training distributed execution framework, the multi-agent system adopts a distributed execution mode in the process of interacting with the environment. Ignoring the self-weighted network, each agent only uses the agent network and action selection strategy to select actions.

[0117] Step S300, collecting transfer history data including decision transfer history (o t ,u t ,r t+1,o t+1 ) and state transition history (s t ,s t+1 ), and stored in the experience pool D; where o t 、u t and t are the joint observation, joint action and global state of the multi-agent system at time t respectively;

[0118] Step S400, sample b groups of data from the experience pool D, calculate the update target of the multi-agent system based on the stage self-weighted network and the stage agent network, train the self-weighted network and the agent network according to the preset loss function and update their parameters.

[0119] Due to the existence of the GRU layer in the agent network, the data sampled from the experience pool D is based on a game. Each time sampling, 32 sets of transfer history data from different games are randomly selected from the experience pool;

[0120] The calculation formula for the update target is:

[0121]

[0122] Among them, r k+1 is the joint reward obtained by the multi-agent system from the environment, the subscript k represents the sample index extracted from the experience pool D; γ is the discount coefficient, and its value is 0.99; is the joint action value calculated by the multi-agent system based on the stage self-weighted network, is the self-weighted network parameter of the previous stage, which is determined by the global state s at the next moment k+1 Calculated by a set of hypernetworks. represents the target action, and its calculation formula is:

[0123]

[0124] This action means to jointly observe o at the next moment k+1 In the following example, each agent takes an action by maximizing the observed action value, where the observed action value is calculated based on the stage agent network.

[0125] The loss function is calculated as:

[0126]

[0127] Among them, Q(o k ,u k θ joint ) is the joint action value calculated based on the self-weighted network, where θ joint is the self-weighted network parameter of the stage, which is determined by the global state s kCalculated by a set of hypernetworks.

[0128] In this embodiment, the optimizer of the loss function in the training step S400 is RMSProp, and the optimization of the parameters adopts the RMSProp optimizer. For each parameter ω in the parameter set j , and its update rule is:

[0129]

[0130]

[0131] ω t+1 =ω t +Δω t

[0132] Among them, ω t is the parameter at time t; g t is the parameter ω at time t j The gradient of t is the exponential average of the squared gradient at time t; η is the learning rate of the optimizer, and its value is set to 0.0005; ρ is the smoothing parameter, and its value is set to 0.99; ∈ is the numerical stability term, and its value is set to 0.00001.

[0133] Since the present invention adopts a centralized training distributed execution framework, a centralized training method is adopted in the training process of the self-weighted network and the intelligent agent network. Based on the sampled data, the self-weighted network takes the intelligent agent observation action value calculated by the intelligent agent network as input, calculates the loss function, calculates the gradient according to the specified optimizer, and updates the parameters of the intelligent agent network and the self-weighted network at the same time.

[0134] Step S500: replace the parameters of the stage self-weighted network and the stage agent network with the parameters of the current self-weighted network and the current agent network according to the preset stage network replacement interval, and attenuate the exploration rate according to the preset parameters.

[0135] In this embodiment, the preset interval for replacing the stage network is 200 rounds. After each 200 rounds of training, the algorithm replaces the parameters of the two stage networks with the parameters of the current two networks, i.e., the parameters of the stage self-weighted network. and the parameters of the agent network at each stage

[0136] The exploration rate ε is decayed after each round of training. After 50,000 rounds of training, the exploration rate decays from the initial 1.0 to 0.05 and remains unchanged.

[0137] Step S600, repeat steps S200-S500 until the preset number of training rounds is reached, and the last updated agent network parameters are used as the training result.

[0138] In this embodiment, the preset training length is in time steps, and the training stops after sampling 2 million time steps. After the training is completed, the agent network parameters are retained, and the agent network is used in subsequent executions to select actions for the agent in a distributed manner.

[0139] Based on any of the above embodiments, the agent network has multiple different flow networks, each of which corresponds to a different action subspace of the agent action space. For an action subspace of fixed size, the input and output dimensions of the flow network remain unchanged; for an action subspace of varying size, the input dimension of the flow network remains fixed, and the output dimension changes as the size of the action subspace changes in the changing environment. The output of the flow network is the value of the agent performing actions in different action subspaces, and the input of the flow network is local observation information related to this type of action.

[0140] Based on any of the above embodiments, for the problem of cooperative multi-agent control, when the observation range of each agent is limited, coordinated decision-making is taken to eliminate the enemy and win the victory. The present invention provides a method for controlling agents in a changing environment based on reinforcement learning based on a centralized training distributed execution framework. The method enables the multi-agent system to learn cooperative strategies through training and to perform actions in a distributed manner in actual execution. Moreover, the method can be applied in complex changing environments, wherein the self-weighted network can solve the problem of reputation distribution and the problem that the dynamic changes in the number of agents in a changing environment affect the training process, and the agent network can solve the problem that the dynamic changes in the size of the action space of agents in a changing environment affect the training process.

[0141] Based on any of the above embodiments, the method provided by the present invention is applied to the multi-unit confrontation environment of game A. In the first scenario, our side controls 8 remote soldiers "Marine" and the enemy side controls 8 remote soldiers "Marine" (8m); in the second scenario, our side controls 5 remote soldiers "Marine" and the enemy side controls 6 remote soldiers "Marine" (5m_vs_6m); in the third scenario, our side controls 3 remote soldiers "Hunters" and 5 melee soldiers "Fanatics", and the enemy side controls 3 remote soldiers "Hunters" and 6 melee soldiers "Fanatics" (3s5z_vs_3s6z). In the latter two scenarios, although the enemy has only one melee soldier, it will cause great suppression to our side in the actual game. In the third scenario, due to the inclusion of two different types of units, the difficulty of controlling the multi-agent system is significantly increased.

[0142] Figure 5 is a curve showing the change of the winning rate of the multi-agent system in the battle with the number of training steps provided by the present invention, such as Figure 5As shown, the winning rate in the figure was obtained by averaging after repeating the experiment three times. In the first scenario, the number and types of units on both sides are the same. The multi-agent system of the present invention can quickly learn a better strategy and achieve a winning rate close to 100%; in the second scenario, despite the disadvantage in quantity, the multi-agent system trained based on this method can still achieve an 82% winning rate at the end of training; in the third scenario, the control of heterogeneous agents and the disadvantage in quantity of our side greatly increase the control difficulty, but the multi-agent system provided by the present invention can still achieve a winning rate of 51% at the end of training. The experimental results show that the method implemented in the present invention achieves excellent multi-agent system control effect.

[0143] The following is a description of the changing environment multi-agent control device based on reinforcement learning provided by the present invention. The changing environment multi-agent control device based on reinforcement learning described below and the changing environment multi-agent control method based on reinforcement learning described above can be referenced to each other.

[0144] Figure 6 is a schematic diagram of the structure of the multi-agent control device for a changing environment based on reinforcement learning provided by the present invention. Figure 6 As shown, the device comprises:

[0145] Input module 610, used to input current environment information and implicit states of each agent into the agent network, and obtain the observed action value of each agent output by the agent network; the observed action value is used to represent the estimated value of all actions currently performed by the corresponding agent;

[0146] A control module 620, for controlling each agent to perform an action based on the observed action value of each agent;

[0147] Among them, the agent network is obtained by reinforcement learning based on the sample environment information and the sample implicit state of the sample agent in conjunction with the self-weighted network; the self-weighted network is used to determine the joint action value of the actions performed by all sample agents; in the reinforcement learning process, the input of the self-weighted network includes the sample observation action values ​​of all sample agents output by the agent network.

[0148] The device provided by the embodiment of the present invention is obtained by reinforcement learning through an intelligent agent network in conjunction with a self-weighted network, so that the intelligent agent network can learn cooperation strategies, thereby enabling distributed control of the intelligent agents' execution of actions based on the intelligent agent network, and the self-weighted network is used to determine the joint action value of all sample intelligent agents' execution of actions, thereby avoiding the impact of dynamic changes in the number of agents in a changing environment on the training process, and ensuring the convergence speed during training and the strategy performance after convergence.

[0149] Based on any of the above embodiments, the agent network includes multiple flow networks, each flow network corresponds to an action subspace in the action space of the agent; the flow network is used to determine the estimated value of the action currently executed by the agent in the corresponding action subspace; the action subspace is divided according to the function of the action, and different action subspaces have no intersection; the observed action value includes the estimated value of all actions in the action space.

[0150] Based on any of the above embodiments, the joint action value is determined based on the following steps:

[0151] Based on the mapping parameters of the self-weighted network, the sample observation action values ​​of each sample agent are nonlinearly mapped to obtain the value mapping results of each sample agent;

[0152] Based on the value mapping results of each sample agent and the value weight of each sample agent in the self-weighted network, the joint action value is determined.

[0153] Based on any of the above embodiments, the value weight of each sample agent in the self-weighted network is determined based on the following steps:

[0154] Based on the encoding parameters in the self-weighted network, the local observation information of each sample agent and the global state of the sample environment are encoded respectively to obtain a first intermediate variable and a second intermediate variable;

[0155] Based on the first intermediate variable and the second intermediate variable, the value weight of each sample intelligent agent is determined.

[0156] Based on any of the above embodiments, the agent network is determined based on the following steps:

[0157] Based on multiple groups of randomly selected transfer history data and a preset loss function, the initial agent network and the initial self-weighted network are jointly trained to obtain an agent network;

[0158] The joint training process includes multiple training stages, and the update target of the loss function of any training stage is calculated based on the stage self-weighted network and the stage agent network when the previous training stage of any training stage is completed; each group of transfer history data includes the joint observations, joint actions and the global state of the sample environment at each historical moment before each joint training; the joint observations and joint actions respectively represent the local observation information and execution actions of all sample agents.

[0159] Based on any of the above embodiments, the update target is determined based on the following steps:

[0160] Determine the joint action value at the next historical moment based on the joint observation at the next historical moment, the stage agent network, and the stage self-weighted network;

[0161] Based on the joint reward obtained from the sample environment and the joint action value, the update target is determined.

[0162] Based on any of the above embodiments, the sample observed action value includes the sample estimated value of all actions performed by the sample agent;

[0163] Based on the joint observation of the next historical moment, the stage agent network, and the stage self-weighted network, the joint action value of the next historical moment is determined, including:

[0164] Based on the joint observation of the next historical moment and the stage agent network, the estimated target value of all sample agents at the next historical moment is obtained; the estimated target value is the maximum value of all sample estimated values ​​of the corresponding sample agent;

[0165] Based on the estimated target value and the stage self-weighted network, the joint action value is obtained.

[0166] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the multi-agent control method for a changing environment based on reinforcement learning, including: inputting the current environment information and the implicit state of each agent into the agent network to obtain the observed action value of each agent output by the agent network; the observed action value is used to characterize the estimated value of all actions currently performed by the corresponding agent; based on the observed action value of each agent, each agent is controlled to perform an action; wherein the agent network is obtained by reinforcement learning based on the sample environment information and the sample implicit state of the sample agent in combination with the self-weighted network; the self-weighted network is used to determine the joint action value of all sample agents performing actions; in the reinforcement learning process, the input of the self-weighted network includes the sample observed action value of all sample agents output by the agent network.

[0167] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0168] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the changing environment multi-agent control method based on reinforcement learning provided by the above-mentioned methods, including: inputting the current environment information and the implicit state of each agent into the agent network to obtain the observed action value of each agent output by the agent network; the observed action value is used to characterize the estimated value of all actions currently performed by the corresponding agent; based on the observed action value of each agent, each agent is controlled to perform actions; wherein the agent network is obtained by reinforcement learning based on sample environment information and sample implicit states of sample agents in combination with a self-weighted network; the self-weighted network is used to determine the joint action value of all sample agents performing actions; in the reinforcement learning process, the input of the self-weighted network includes the sample observed action values ​​of all sample agents output by the agent network.

[0169] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the reinforcement learning-based changing environment multi-agent control method provided by the above-mentioned methods, including: inputting current environmental information and the implicit state of each agent into the agent network to obtain the observed action value of each agent output by the agent network; the observed action value is used to characterize the estimated value of all actions currently performed by the corresponding agent; based on the observed action value of each agent, each agent is controlled to perform actions; wherein the agent network is obtained by reinforcement learning based on sample environmental information and sample implicit states of sample agents in combination with a self-weighted network; the self-weighted network is used to determine the joint action value of all sample agents performing actions; in the reinforcement learning process, the input of the self-weighted network includes the sample observed action values ​​of all sample agents output by the agent network.

[0170] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0171] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for controlling multi-agents in a changing environment based on reinforcement learning, applied to a multi-unit battle game based on cooperative multi-agent learning, characterized in that: include: Inputting the dynamically changing current environment information and the implicit state of each agent into the agent network, and obtaining the observed action value of each agent output by the agent network; The observed action value is used to represent the estimated value of all actions currently performed by the corresponding agent; Based on the observed action value of each agent, control each agent to perform actions; The current environment information includes the normalized life, shield, and relative coordinates of all units in the current environment. All units include multiple friendly units and multiple enemy units in a multi-unit battle game. Each intelligent agent is applied to each friendly unit. The implicit state of each intelligent agent includes the execution action and identity information at the last moment. All actions include move up, move right, and attack. The agent network is obtained by performing reinforcement learning based on sample environment information and sample implicit states of sample agents in combination with a self-weighted network; the self-weighted network is used to determine the joint action value of actions performed by all sample agents; during the reinforcement learning process, the input of the self-weighted network includes the sample observed action values ​​of all sample agents output by the agent network; The agent network includes a plurality of flow networks, each flow network corresponds to an action subspace in the dynamically changing action space of the agent; the flow network is used to determine the estimated value of the action currently executed by the agent in the corresponding action subspace; the action subspace is divided according to the function of the action, and different action subspaces have no intersection; the observed action value includes the estimated value of all actions in the action space; the action subspace includes a unit-oriented attack action subspace and an environment-oriented movement action subspace, in the flow network, the attack flow network corresponding to the attack action subspace is used to evaluate the estimated attack value of the agent's attack action on all enemy units in the current environment, and the movement flow network corresponding to the movement action subspace is used to evaluate the estimated movement value of the agent's current movement action; The joint action value is determined based on the following steps: Based on the mapping parameters of the self-weighted network, nonlinear mapping is performed on the sample observation action values ​​of each sample intelligent agent to obtain the value mapping results of each sample intelligent agent; The joint action value is determined based on the value mapping results of each sample agent and the value weight of each sample agent in the self-weighted network.

2. The method for controlling multi-agents in a changing environment based on reinforcement learning according to claim 1, characterized in that: The value weight of each sample agent in the self-weighted network is determined based on the following steps: Based on the encoding parameters in the self-weighted network, the local observation information of each sample agent and the global state of the sample environment are encoded respectively to obtain a first intermediate variable and a second intermediate variable; Based on the first intermediate variable and the second intermediate variable, the value weight of each sample intelligent agent is determined.

3. The method for controlling multi-agents in a changing environment based on reinforcement learning according to claim 1, characterized in that: The agent network is determined based on the following steps: Based on a plurality of randomly selected groups of transfer history data and a preset loss function, jointly training the initial agent network and the initial self-weighted network to obtain the agent network; The joint training process includes multiple training stages, and the update target of the loss function of any training stage is calculated based on the stage self-weighted network and the stage agent network when the previous training stage of any training stage is completed; each group of transfer history data includes the joint observations, joint actions and the global state of the sample environment at each historical moment before each joint training; the joint observations and joint actions respectively represent the local observation information and execution actions of all sample agents.

4. The method for controlling multi-agents in a changing environment based on reinforcement learning according to claim 3, characterized in that: The update target is determined based on the following steps: Determine a joint action value at a next historical moment based on a joint observation at a next historical moment, the stage agent network, and the stage self-weighted network; The update target is determined based on the joint reward obtained from the sample environment and the joint action value.

5. The method for controlling multi-agents in a changing environment based on reinforcement learning according to claim 4, characterized in that: The sample observed action value includes the sample estimated value of all actions performed by the sample agent; The determining of the joint action value at the next historical moment based on the joint observation at the next historical moment, the stage agent network, and the stage self-weighted network includes: Based on the joint observation at the next historical moment and the agent network at the stage, an estimated target value of all sample agents at the next historical moment is obtained; the estimated target value is the maximum value of all sample estimated values ​​of the corresponding sample agents; Based on the estimated target value and the stage self-weighted network, the joint action value is obtained.

6. A multi-agent control device for changing environments based on reinforcement learning, applied to a multi-unit battle game based on cooperative multi-agent learning, characterized in that: include: An input module, used to input the dynamically changing current environment information and the implicit state of each agent into the agent network, and obtain the observed action value of each agent output by the agent network; The observed action value is used to represent the estimated value of all actions currently performed by the corresponding agent; A control module is used to control each agent to perform actions based on the observed action value of each agent; The current environment information includes the normalized life, shield, and relative coordinates of all units in the current environment. All units include multiple friendly units and multiple enemy units in a multi-unit battle game. Each intelligent agent is applied to each friendly unit. The implicit state of each intelligent agent includes the execution action and identity information at the last moment. All actions include move up, move right, and attack. The agent network is obtained by performing reinforcement learning based on sample environment information and sample implicit states of sample agents in combination with a self-weighted network; the self-weighted network is used to determine the joint action value of actions performed by all sample agents; during the reinforcement learning process, the input of the self-weighted network includes the sample observed action values ​​of all sample agents output by the agent network; The agent network includes a plurality of flow networks, each flow network corresponds to an action subspace in the dynamically changing action space of the agent; the flow network is used to determine the estimated value of the action currently executed by the agent in the corresponding action subspace; the action subspace is divided according to the function of the action, and different action subspaces have no intersection; the observed action value includes the estimated value of all actions in the action space; the action subspace includes a unit-oriented attack action subspace and an environment-oriented movement action subspace, in the flow network, the attack flow network corresponding to the attack action subspace is used to evaluate the estimated attack value of the agent's attack action on all enemy units in the current environment, and the movement flow network corresponding to the movement action subspace is used to evaluate the estimated movement value of the agent's current movement action; The joint action value is determined based on the following steps: Based on the mapping parameters of the self-weighted network, nonlinear mapping is performed on the sample observation action values ​​of each sample intelligent agent to obtain the value mapping results of each sample intelligent agent; The joint action value is determined based on the value mapping results of each sample agent and the value weight of each sample agent in the self-weighted network.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the multi-agent control method in a changing environment based on reinforcement learning are implemented as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for controlling a multi-agent in a changing environment based on reinforcement learning are implemented as described in any one of claims 1 to 5.

Images