Cooperative multi-agent control method and device based on value function supervision
By introducing formation-level agents to guide platform-level agent action selection in a multi-agent system, and combining value function supervision and training algorithms, the performance instability problem in the distributed execution phase is solved, and the stability and adaptability of the system are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 709TH RESEARCH INSTITUTE CHINA STATE SHIPBUILDING CORP LTD
- Filing Date
- 2022-09-28
- Publication Date
- 2026-05-12
AI Technical Summary
Existing multi-agent reinforcement learning suffers from performance instability during distributed execution, and the accuracy of platform-level agent value functions is highly dependent during centralized training, making it difficult to adapt to large-scale complex problems.
In the distributed execution phase, a formation-level agent is introduced to guide the platform-level agent in selecting actions. A cooperative multi-agent control method with value function supervision is adopted, and the controller is trained by combining VDN and AC algorithms to reduce the dependence on centralized training.
It improves the stability and adaptability of multi-agent systems, reduces the reliance on the accuracy of platform-level agent value functions during centralized training, and enhances the efficiency and accuracy of decision-making.
Smart Images
Figure CN115755593B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multi-agent reinforcement learning technology, and more specifically, relates to a cooperative multi-agent control method and device based on value function supervision. Background Technology
[0002] In recent years, the groundbreaking progress made in single-agent deep reinforcement learning technology has also promoted the development and research of multi-agent reinforcement learning. In real-world situations, there are often multiple agents making independent decisions. Therefore, the research on multi-agent reinforcement learning has significant application value.
[0003] The interaction process between fully cooperative multi-agent systems and the environment is shown in the appendix. Figure 1 As shown, there are n independent and autonomous agents in the environment. The process is mainly as follows: (1) At time t, agent i perceives the current environmental state s. t To obtain its own local observation information (2) The agent, based on the current local observation information The current strategy involves selecting an action from action space A. Forming joint actions (3) When the joint actions of multiple agents act on the environment, the environment transitions to a new state s. t+1 And give a global reward value r. t This process repeats itself. Here, reward refers to the evaluative feedback signal received by the agent from the environment during its interaction with it. Through reinforcement learning, the agent determines how to take a series of actions in the environment to maximize long-term cumulative rewards.
[0004] Multi-agent reinforcement learning can be categorized into three training architectures: centralized learning, independent learning, and a combination of both (centralized training and distributed execution). Independent learning involves using a reinforcement learning algorithm on each agent individually, treating other agents as part of the environment. A representative example is IQL (Independent Q-Learning). During the learning process, each agent independently executes the Q-learning algorithm. These algorithms are relatively easy to implement and perform well in small-scale problems with discrete state-action spaces. However, because they don't consider the interactions between agents, they cannot handle large-scale, complex problems. Centralized learning involves pooling the states and actions of all agents to form a global state space and joint action space, using a single-agent reinforcement learning approach. This method addresses environmental instability by considering the global context, making it easier to train and enabling better collaboration among multiple agents. However, this method assumes no barriers to information exchange between agents and requires gathering global situational awareness data to make decisions before sending them to each agent for execution, resulting in efficiency delays.
[0005] The "centralized training-distributed execution" approach is currently the most commonly used training architecture for multi-agent reinforcement learning algorithms. It combines the advantages of centralized and independent learning. During training, all agents are trained using a centralized architecture, and each agent can obtain information from other agents through unrestricted open channels. In the execution phase after training, each agent makes action decisions based on its own local observations and limited communication. However, because this method introduces global information during training, it suffers from performance instability during distributed execution. Summary of the Invention
[0006] To address the aforementioned problems, this invention provides a cooperative multi-agent control method based on value function supervision. It proposes introducing formation-level agents during the distributed execution phase, guiding agents to select actions and improving stability. Furthermore, the introduction of formation-level macro-decision actions during the execution phase enhances the adaptability of the multi-agent system and reduces dependence on the accuracy of platform-level agent value functions during centralized training.
[0007] To achieve the above objectives, according to one aspect of the present invention, a cooperative multi-agent control method based on value function supervision is provided, comprising the following steps:
[0008] S1 initializes the simulation environment, loads the formation-level controller model parameters, loads the platform-level controller model parameters, and each platform shares the same platform-level controller model parameters;
[0009] S2 uses a formation-level controller to select formation-level decision-making action commands;
[0010] S3 uses a platform-level controller to select platform-level decision-making action instructions under the supervision of formation-level decision-making action instructions;
[0011] The S4 simulation environment executes joint action instructions u = (u1, ..., u2) of multiple intelligent agents. n ), where n is the number of platform-level agents. Repeat steps S2 to S4 above to complete the control of multiple agents.
[0012] In one embodiment of the present invention, step S2 specifically includes:
[0013] S21 obtains the formation-level state vector O at time steps K;
[0014] S22 inputs the formation-level state vector O, and the formation-level controller outputs the formation-level decision-making action command, i.e., the platform-level agent value function supervision vector. For the strategy function of the formation-level controller, It is its parameter, where m is the number of actions that the platform-level intelligent agent can perform.
[0015] In one embodiment of the present invention, step S3 specifically includes:
[0016] S31 At each time step, for each platform-level agent i, obtain the environmental state o observed by agent i. i ;
[0017] S32 inputs the local observation state o of each platform-level agent i. i The platform-level controller outputs the local state-action value vector Q of the platform-level agent. i =(Q(o) i ,a1),…,Q(o i ,a j ),…,Q(o i ,a m ), where a1~a m represents the actions that a platform-level intelligent agent can execute; m represents the number of actions that a platform-level intelligent agent can execute; Q(o i ,a j ) = Q(o = o i a=a j |θ), where state o i Choose action a j The state-action value at time; Q(o,a|θ) is the value function of the platform-level controller, where θ is its parameter;
[0018] S33 updates the local state-action vector Q of the platform-level agent using the formation-level decision-making action instruction W. i Value
[0019] S34 is selected based on a greedy strategy. The action with the largest median value is used as the action instruction for platform-level agent i.
[0020] In one embodiment of the present invention, the value function network of the platform-level controller in step S3 is trained using the existing multi-agent reinforcement learning method VDN (Value Decomposition Networks) algorithm to update the network parameters.
[0021] In one embodiment of the present invention, the training method for the value function network of the platform-level controller specifically includes:
[0022] Step 1.1 Initialize the simulation environment and initialize the parameter θ of the platform-level controller value function network Q(o,a|θ);
[0023] Step 1.2 At each time step, for each platform-level agent i, obtain the environmental state o observed by agent i. i ;
[0024] Step 1.3 Input the local observation state of the agent. i The platform-level controller outputs the platform-level agent's state-action value function Q. i =(Q(o) i ,a1),……,Q(o i ,a m )), Q(o i ,a j ) = Q(o = o i a=a j |θ), where m is the number of actions that the platform-level intelligent agent can execute;
[0025] Step 1.4 Update the local state-action value vector Q of the platform-level agent using the formation-level decision action instruction W. i Value
[0026] Step 1.5 Select according to greedy strategy The action with the largest median value is used as the action instruction for platform-level agent i.
[0027] Step 1.6 The simulation environment executes the joint action u = (u1, ..., u2) of multiple agents. n), where n is the number of platform-level agents, and returns the global reward r, with the next time-instance state being the observed state o′ of each agent = (o 1′ ,……, n′ At the next moment, save the (o,u,r,o′) tuple to the training data buffer D, repeat steps 1.2 to 1.6, collect data and put it into D, and stop when the termination condition or the maximum number of times is reached;
[0028] Step 1.7 After storing a certain number of training data in the data buffer D, randomly select M sample data from D and update the value function network Q using the VDN method. L The parameter θ;
[0029] Step 1.8: The learning process ends when it converges or reaches the maximum number of iterations; otherwise, return to step 1.2.
[0030] In one embodiment of the present invention, step 1.7 uses the VDN method to update the value function network Q. L The parameter θ specifically includes:
[0031] Calculate the global value function using the following formula:
[0032]
[0033] Q total (s,u) Global value function, where s is the global environment state vector, u is the joint action, and Q... i (o i ,u i Let Q be the local state-action value function of agent i. i (o i ,u i ) = Q(o = o i ,a j =u i |θ);
[0034] The loss function is constructed as follows:
[0035]
[0036] Among them, y j =r j +γargmax u′ Q total (s′,u′), where s′ is the global state at the next time step and u′ is the joint action at the next time step, θ is updated by the following formula, where η is the learning rate:
[0037]
[0038] In one embodiment of the present invention, after the platform-level controller in step S3 is trained, the policy network of the formation-level controller in step S2 is trained using the AC (Actor-Critic) algorithm to update the network parameters.
[0039] In one embodiment of the present invention, the method for training the policy network of the formation-level controller specifically includes:
[0040] Step 2.1 Initialize the simulation environment and initialize the policy network of the formation-level controller. parameters Initialize the evaluation network Load the pre-trained platform-level controller value function network Q(o) i ,a j The parameter θ of |θ);
[0041] Step 2.2 At time steps K, obtain the formation-level state vector O and input it into the policy network π of the formation-level controller. H Output formation-level decision-making action instructions W = (w1, w2, ..., w m );
[0042] Step 2.3: At each time step, for each agent i, obtain the environmental state o observed by agent i. i The input is fed into the platform-level controller value function network, and the output is the platform-level agent's state-action value function Q. i =(Q(o) i ,a1),……,Q(o i ,a M ));
[0043] Step 2.4: Calculate the supervised Q-value based on the formation-level decision results: Select based on greedy strategy The action with the largest median value is used as the action instruction for platform-level agent i.
[0044] Step 2.5: Combine the actions u = (u1, ..., u) generated by all agents. n The input is given to the environment, executed, and the reward r is returned.
[0045] Repeat steps 2.1-2.5, updating the parameter ψ of the evaluation network V(ψ) at time steps K, until the current simulation environment ends;
[0046]
[0047] Where: L(ψ)=(r K +V(O′|ψ)-V(O|ψ)) 2 , O′ is the formation-level state vector for the next moment;
[0048] Step 2.6: Update the policy network of the formation-level controller. parameters
[0049]
[0050] Where η is the learning rate;
[0051] Step 2.7: The learning process ends when it converges or reaches the maximum number of iterations; otherwise, return to step 2.2.
[0052] In one embodiment of the present invention, the simulation environment is the 5sV6z scenario in StarCraft II.
[0053] According to another aspect of the present invention, a cooperative multi-agent control device based on value function supervision is also provided, including at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, wherein the instructions, after being executed by the processor, are used to complete the cooperative multi-agent control method based on value function supervision.
[0054] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:
[0055] (1) In the distributed execution phase, this invention introduces a formation-level decision-making agent to guide the decision-making of the platform-level agent, thereby improving stability;
[0056] (2) The decision-making of the formation-level intelligent agent of this invention hides the learned macro strategy, which is more in line with the actual command and decision-making process;
[0057] (3) The present invention introduces formation-level macro decision-making actions in the execution phase, which can improve the adaptability of multiple agents and reduce the dependence on the accuracy of platform-level agent value functions during centralized training. Attached Figure Description
[0058] Appendix Figure 1 A schematic diagram of the interaction process in fully cooperative multi-agent reinforcement learning;
[0059] Appendix Figure 2 A flowchart illustrating the cooperative multi-agent control method based on value function supervision in this invention;
[0060] Appendix Figure 3 A schematic diagram of a multi-agent simulation environment in an embodiment of the present invention - StarCraft II 5sV6z. Detailed Implementation
[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0062] In order to solve the problems existing in the current technology, such as Figure 2 As shown, this invention provides a cooperative multi-agent control method based on value function supervision, comprising the following steps:
[0063] S1 initializes the simulation environment, loads the formation-level controller model parameters, loads the platform-level controller model parameters, and each platform shares the same platform-level controller model parameters;
[0064] S2 uses a formation-level controller to select formation-level decision-making action commands;
[0065] Step S2 specifically includes:
[0066] S21 obtains the formation-level state vector O at time steps K;
[0067] S22 inputs the formation-level state vector O, and the formation-level controller outputs the formation-level decision-making action command, i.e., the platform-level agent value function supervision vector. For the strategy function of the formation-level controller, It is its parameter, where m is the number of actions that the platform-level intelligent agent can perform.
[0068] S3 uses a platform-level controller to select platform-level decision-making action instructions under the supervision of formation-level decision-making action instructions;
[0069] Step S3 specifically includes:
[0070] S31 At each time step, for each platform-level agent i, obtain the environmental state o observed by agent i. i ;
[0071] S32 inputs the local observation state o of each platform-level agent i. i The platform-level controller outputs the local state-action value vector Q of the platform-level agent. i =(Q(o) i ,a1),…,Q(o i ,a j ),…,Q(o i ,a m ), where a1~a mrepresents the actions that a platform-level intelligent agent can execute; m represents the number of actions that a platform-level intelligent agent can execute; Q(o i ,a j ) = Q(o = o i a=a j |θ), where state o i Choose action a j The state-action value at time; Q(o,a|θ) is the value function of the platform-level controller, where θ is its parameter;
[0072] S33 updates the local state-action vector Q of the platform-level agent using the formation-level decision-making action instruction W. i Value
[0073] S34 is selected based on a greedy strategy. The action with the largest median value is used as the action instruction for platform-level agent i.
[0074] Specifically, in step S3, the value function network of the platform-level controller is trained using the existing multi-agent reinforcement learning method VDN (Value Decomposition Networks) algorithm to update the network parameters.
[0075] The S4 simulation environment executes joint action instructions u = (u1, ..., u2) of multiple intelligent agents. n ), where n is the number of platform-level agents. Repeat steps S2 to S4 above to complete the control of multiple agents.
[0076] Furthermore, the training method for the value function network of the platform-level controller specifically includes:
[0077] Step 1.1 Initialize the simulation environment and initialize the parameter θ of the platform-level controller value function network Q(o,a|θ);
[0078] Step 1.2 At each time step, for each platform-level agent i, obtain the environmental state o observed by agent i. i ;
[0079] Step 1.3 Input the local observation state of the agent. i The platform-level controller outputs the platform-level agent's state-action value function Q. i =(Q(o) i ,a1),……,Q(o i ,a m )), Q(o i ,a j ) = Q(o = o i a=a j|θ), where m is the number of actions that the platform-level intelligent agent can execute;
[0080] Step 1.4 Update the local state-action value vector Q of the platform-level agent using the formation-level decision action instruction W. i Value is
[0081] Step 1.5 Select according to the greedy strategy The action with the largest median value is used as the action instruction for platform-level agent i.
[0082] Step 1.6 The simulation environment executes the joint action u = (u1, ..., u2) of multiple agents. n ), where n is the number of platform-level agents, and returns the global reward r, with the next time-instance state being the observed state o′ of each agent = (o 1′ ,……,o n′ At the next moment, save the (o,u,r,o′) tuple to the training data buffer D, repeat steps 1.2 to 1.6, collect data and put it into D, and stop when the termination condition or the maximum number of times is reached;
[0083] Step 1.7 After storing a certain number of training data in the data buffer D, randomly select M sample data from D and update the value function network Q using the VDN method. L The parameter θ;
[0084] Step 1.8: The learning process ends when it converges or reaches the maximum number of iterations; otherwise, return to step 1.2.
[0085] Furthermore, in step 1.7, the value function network Q is updated using the VDN method. L The parameter θ specifically includes:
[0086] Calculate the global value function using the following formula:
[0087]
[0088] Q total (s,u) Global value function, where s is the global environment state vector, u is the joint action, and Q... i (o i ,u i Let Q be the local state-action value function of agent i. i (o i ,u i ) = Q(o = o i ,a j =u i |θ);
[0089] The loss function is constructed as follows:
[0090]
[0091] Among them, y j =r j +γargmax u′ Q total (s′,u′), where s′ is the global state at the next time step and u′ is the joint action at the next time step, θ is updated by the following formula, where η is the learning rate:
[0092]
[0093] Furthermore, after the platform-level controller in step S3 is trained, the policy network of the formation-level controller in step S2 is trained using the AC (Actor-Critic) algorithm to update the network parameters.
[0094] Furthermore, the method for training the policy network of the formation-level controller specifically includes:
[0095] Step 2.1 Initialize the simulation environment and initialize the policy network of the formation-level controller. parameters Initialize the evaluation network V(ψ), and load the pre-trained platform-level controller value function network Q(o). i ,a j The parameter θ of |θ);
[0096] Step 2.2 At time steps K, obtain the formation-level state vector O and input it into the policy network π of the formation-level controller. H Output formation-level decision-making action instructions W = (w1, w2, ..., w m );
[0097] Step 2.3: At each time step, for each agent i, obtain the environmental state o observed by agent i. i The input is fed into the platform-level controller value function network, and the output is the platform-level agent's state-action value function Q. i =(Q(o) i ,a1),……,Q(o i ,a M ));
[0098] Step 2.4: Calculate the supervised Q-value based on the formation-level decision results: Select based on greedy strategy The action with the largest median value is used as the action instruction for platform-level agent i.
[0099] Step 2.5: Combine the actions u = (u1, ..., u) generated by all agents. n The input is given to the environment, executed, and the reward r is returned.
[0100] Repeat steps 2.1-2.5, updating the parameter ψ of the evaluation network V(ψ) at time steps K, until the current simulation environment ends;
[0101]
[0102] Where: L(ψ)=(r K +V(O′|ψ)-V(O|ψ)) 2 , O′ is the formation-level state vector for the next moment;
[0103] Step 2.6: Update the policy network of the formation-level controller. parameters
[0104]
[0105] Where η is the learning rate;
[0106] Step 2.7: The learning process ends when it converges or reaches the maximum number of iterations; otherwise, return to step 2.2.
[0107] The following detailed description of a cooperative multi-agent control method based on value function supervision proposed in this invention, with reference to specific embodiments, is provided below. Figure 3 As shown, for a typical multi-agent control task, such as the 5sv6z scenario in StarCraft II, the objective of the task is to learn from scratch how to control five friendly units to cooperate in attacking six enemy units. Figure 3 This describes the specific scenario of the simulation environment. There are five controlled agents in this simulation environment. For each agent, the input is a 104-dimensional local observation state, and the output is one of 12 discrete actions. During formation-level decision-making, the input is the combined state of the five agents, and the output is the formation-level decision action. During platform-level decision-making, the input is the local observation state of each agent, and the output is one of the 12 discrete actions.
[0108] Furthermore, the present invention also provides a cooperative multi-agent control device based on value function supervision, including at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the cooperative multi-agent control method based on value function supervision.
[0109] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A cooperative multi-agent control method based on value function supervision, characterized in that, Includes the following steps: S1 initializes the simulation environment, loads the formation-level controller model parameters, loads the platform-level controller model parameters, and each platform shares the same platform-level controller model parameters; S2 uses a formation-level controller to select formation-level decision-making action commands; S3 uses a platform-level controller to select platform-level decision-making action instructions under the supervision of formation-level decision-making action instructions; after the platform-level controller is trained, the policy network of the formation-level controller in step S2 is trained using the AC (Actor-Critic) algorithm to update the network parameters; the specific training method for the policy network of the formation-level controller includes: Step 2.1 Initialize the simulation environment and initialize the policy network of the formation-level controller. parameters Initialize the evaluation network Load the pre-trained platform-level controller value function network parameters ; Step 2.2 Every time step Obtain the formation-level state vector The policy network input to the formation-level controller Output formation-level decision-making action commands ; Step 2.3: At each time step, for each agent... i Acquire intelligent agents i Observed local observation state of the agent The input is fed into the platform-level controller value function network, and the output is the platform-level agent's local state-action value function. ; Step 2.4: Calculate the post-monitoring results based on the formation-level decision-making results. value: Select according to greedy strategy The action with the largest median value serves as a platform-level intelligent agent. Action instructions ; Step 2.5: Combine the actions generated by all agents The input is given to the environment, executed, and returns a reward r. Repeat steps 2.1-2.5, at intervals of [time step value]. Update the evaluation network parameter This continues until the current simulation environment ends; in: , , This is the formation-level state vector for the next time step; Step 2.6: Update the policy network of the formation-level controller. parameters : in, The learning rate; Step 2.7: The learning process ends when it converges or reaches the maximum number of iterations; otherwise, return to step 2.
2. The S4 simulation environment executes joint action instructions from multiple intelligent agents. , Given the number of platform-level agents, repeat steps S2 to S4 above to complete the control of multiple agents.
2. The cooperative multi-agent control method based on value function supervision as described in claim 1, characterized in that, Step S2 specifically includes: S21 time step Obtain the formation-level state vector ; S22 Input formation-level state vector The formation-level controller outputs formation-level decision-making action commands, i.e., platform-level agent value function supervision vectors. , For the strategy function of the formation-level controller, These are its parameters. This represents the number of actions that a platform-level intelligent agent can perform.
3. The cooperative multi-agent control method based on value function supervision as described in claim 1 or 2, characterized in that, Step S3 specifically includes: S31 At each time step, for each platform-level agent i Acquire intelligent agents i Observed local observation state of the agent ; S32 for each platform-level intelligent agent i Input the local observation state of the intelligent agent The platform-level controller outputs the local state-action value vector of the platform-level agent. ,in, Actions that can be performed by platform-level intelligent agents; The number of actions that a platform-level intelligent agent can perform; , for state Choose action State-action value at time; For the value function of the platform-level controller, Its parameters; S33 uses formation-level decision-making commands. Update the local state-action value vector of the platform-level agent Value is , ; S34 is selected based on a greedy strategy. The action with the largest median value serves as a platform-level intelligent agent. Action instructions .
4. The cooperative multi-agent control method based on value function supervision as described in claim 1 or 2, characterized in that, In step S3, the value function network of the platform-level controller is trained using the existing multi-agent reinforcement learning method VDN (Value Decomposition Networks) algorithm to update the network parameters.
5. The cooperative multi-agent control method based on value function supervision as described in claim 4, characterized in that, The training method for the value function network of the platform-level controller specifically includes: Step 1.1 Initialize the simulation environment and initialize the platform-level controller value function network. parameters ; Step 1.2 At each time step, for each platform-level agent... i Acquire intelligent agents i Observed local observation state of the agent ; Step 1.3 Input the local observation state of the agent The platform-level controller outputs the local state-action value function of the platform-level agent. , , The number of actions that a platform-level intelligent agent can perform; Step 1.4 Use formation-level decision-making action instructions Update the local state-action value vector of the platform-level agent Value , ; Step 1.5 Select according to greedy strategy The action with the largest median value serves as a platform-level intelligent agent. Action instructions ; Step 1.6 The simulation environment executes the joint actions of multiple intelligent agents. , The number of platform-level intelligent agents is returned, along with the global reward. The observed state of each agent in the next moment. Save in the next moment Add the tuples to the training data buffer D, and repeat steps 1.2 to 1.6 to collect data and put it into D. Stop when the termination condition or the maximum number of times is reached. Step 1.7 After storing a certain amount of training data in the training data buffer D, randomly select M sample data from D and update the value function network using the VDN method. parameters ; Step 1.8: The learning process ends when it converges or reaches the maximum number of iterations; otherwise, return to step 1.
2.
6. The cooperative multi-agent control method based on value function supervision as described in claim 5, characterized in that, In step 1.7, the value function network is updated using the VDN method. parameters Specifically, it includes: Calculate the global value function using the following formula: in Global value functions This is the global environment state vector. For joint operations, For intelligent agents Local state-action value function, ; The loss function is constructed as follows: in, ,in This represents the global state at the next moment. For the next joint action, update using the following formula ,in Learning rate: 。 7. The cooperative multi-agent control method based on value function supervision as described in claim 1 or 2, characterized in that, The simulation environment is the 5sV6z scenario from StarCraft II.
8. A cooperative multi-agent control device based on value function supervision, characterized in that: It includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the cooperative multi-agent control method based on value function supervision as described in any one of claims 1-7.