A multi-agent federated collaboration method based on deep reinforcement learning
By employing deep reinforcement learning and federated learning methods in multi-agent collaboration, the problems of high collaboration overhead and low efficiency are solved, achieving efficient collaboration and improved learning efficiency among agents.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- YANGTZE DELTA REGION INST (QUZHOU) UNIV OF ELECTRONIC SCI & TECH OF CHINA
- Filing Date
- 2020-12-17
- Publication Date
- 2026-07-31
AI Technical Summary
Multi-agent collaboration suffers from high collaboration overhead, low collaboration efficiency, and the curse of dimensionality, making it particularly difficult to efficiently process information and make decisions in complex environments.
A multi-agent federated cooperation method based on deep reinforcement learning is adopted. By dividing the agents into tasks and neural network models, federated learning technology is used to aggregate and update parameters, thereby reducing cooperation overhead and improving cooperation efficiency.
It effectively improves the learning efficiency of multi-agent collaboration, reduces collaboration overhead, and has a certain generalization ability, making it suitable for multi-agent collaboration in different scenarios.
Smart Images
Figure CN112465151B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence. Specifically, it addresses the problems of high collaboration overhead and low collaboration efficiency in the process of multi-agent collaboration under complex tasks. It designs a multi-agent federated collaboration method based on deep reinforcement learning. By combining federated learning technology, it reduces the overhead in the multi-agent collaboration process based on deep reinforcement learning and improves the efficiency of multi-agent collaboration. Background Technology
[0002] In recent years, artificial intelligence technology has developed rapidly and has been widely applied. Among them, intelligent agents based on deep reinforcement learning are one of the key research directions. They perceive information about their surrounding environment, make intelligent decisions, and thus interact with the environment to complete corresponding tasks.
[0003] In real-world problems, the decision-making power of a single agent is far from sufficient. When using a centralized agent to solve a problem, various resource and conditional limitations arise, making it impossible for a single agent to cope with complex real-world environments. However, using multiple agents to collaborate can effectively improve the completeness of information and the efficiency of decision-making.
[0004] However, multi-agent collaboration currently faces the following challenges:
[0005] First, the environment of a multi-agent system is typically partially observable. In a multi-agent environment, each agent's local environment is usually different, resulting in different observational information received by each agent, with each observation containing some local environmental information. Establishing a communication mechanism between agents can aggregate the local environmental information known by each agent, forming a general description of the overall environment, thereby improving algorithm performance. However, this places higher demands on the performance of the communication information processing module, requiring it to filter and extract effective information from the complex observational data and form an efficient representation of the overall environment, which often incurs considerable overhead.
[0006] Secondly, the environment in a multi-agent environment is non-static. In a single-agent environment, an agent only needs to consider the consequences of its own actions. However, in a multi-agent environment, the environmental feedback received by an agent is not only related to its own policy but also to the policies of other agents. If agents communicate with each other independently, the overall time required to complete the task will continuously increase.
[0007] Finally, multi-agent models are prone to the curse of dimensionality. When the number of agents is large, the overall model size becomes enormous, and reinforcement learning algorithms need to learn from a large amount of sampled data, making the model training process very slow. Therefore, it is particularly important to find ways to optimize the training of a single agent model by utilizing the models of multiple agents.
[0008] Therefore, it is of great significance to design a multi-agent collaboration technology under complex conditions to improve the efficiency of multi-agent collaboration and enhance its generalization ability, in order to address the current problems faced by multi-agent collaboration. Summary of the Invention
[0009] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multi-agent federated collaboration method based on deep reinforcement learning. This method allows an agent to explore and make decisions about the environment through deep reinforcement learning while simultaneously acquiring the learning experience of other agents using federated learning techniques. This effectively improves the learning efficiency of the agent and reduces the collaboration overhead between agents.
[0010] The objective of this invention is achieved through the following technical solution: a multi-agent federated cooperation method based on deep reinforcement learning, comprising the following steps:
[0011] S1. Divide the task and the agent, establish a deep reinforcement learning model for each agent, and set the state space of the agent in the environment, the action space that the agent can decide, and the reward of the agent's actions in the environment.
[0012] S2. Based on the deep reinforcement learning algorithm, a corresponding neural network is established for each agent. Agents in the same task area establish the same neural network model, while agents in different task areas can establish different neural network models.
[0013] S3. The agent decides its next action based on the assigned goal and its current state, interacts with the environment to obtain its next state, stores the decision experience in the experience pool, and updates the local neural network model in a timely manner according to the stochastic gradient descent method.
[0014] S4. After the loop reaches the set number of iterations, the local neural network model parameters are transmitted to the collaborative platform for subsequent federated learning.
[0015] S5. Aggregate the parameters uploaded by all agents and return the results to each agent for parameter updates; The collaboration platform divides agents according to the type of task and the enhanced agent model, aggregates the parameters of enhanced agents with similar tasks and the same model, and returns the corresponding aggregation results.
[0016] S6. Each agent performs a soft update on the received return results and the local neural network parameters to obtain the latest local model parameters.
[0017] S7. Repeat S3-S6 until the target task is completed.
[0018] Furthermore, the specific implementation method of dividing tasks and agents in step S1 is as follows: the multiple agents are divided into multiple task regions, each task region has one enhanced agent and multiple ordinary agents; the agent with the strongest ability in each task region is the enhanced agent, and the remaining agents are ordinary agents.
[0019] Furthermore, the deep reinforcement learning model established in step S1 is as follows: There are N agents in the space, numbered from 1 to n, with the i-th agent numbered i; the state space is S, the action space is A, the reward value is R, and the triplet...<S,A,R> This determines the specific multi-agent model;
[0020] (1) State space: Under the same state space, if the state space of the i-th agent is S i The state space of the j-th agent is S. j Then S i =S j , 1≤i≤N, 1≤j≤N, i≠j; under different state space conditions, the agent state space S is established according to the specific scenario. i and S j ;
[0021] (2) Behavior space: A behavior space is established for each agent. Agents with the same state space have the same behavior space. Under different state space conditions, the agent behavior space is established according to the specific scenario.
[0022] (3) Reward value: The reward value R of a multi-agent system includes the following three rewards:
[0023] a) Individual instant reward r1: The environment provides an instant reward to the current agent. Let the current state be s. i The next state is s i+1 The desired state is s e If |s i+1 -s e |<|s i -s e If |, it is determined that the target is close and rewarded with p; otherwise, it is determined that the target is far away and rewarded with -p.
[0024] b) Global instant reward r2: The reward given to all agents by the current environment. Let the current state of the global agents be s. t , Let be the state of the j-th agent at time t, and let s be the next state of the global agent. t+1 When the task is completed, the state of the global agent is s. e If |s t+1 -s e |<|s t -s e If |, it is determined that the agent is close to the target and a reward of q is given to each agent; otherwise, it is determined that the agent is far from the target and a reward of -q is given to each agent.
[0025] Where p and q are positive numbers set by the user, satisfying q≤p;
[0026] c) Global Final Reward r3: This is the reward given to all agents globally when the multi-agent system completes the objective; no reward is given in other cases. The global final reward is related to the total time; that is, the longer the total time, the smaller the final reward, and the shorter the total time, the larger the final reward. Therefore, the global final reward is expressed as... Where T is the total time, α and β are coefficients, α < 0, β > 0, set by the user, and satisfying β ≥ 10(p+q), ensuring that r3 is tens of times (p+q) or more, prompting the agent to learn the currently known approximate optimal policy;
[0027] Therefore, the reward value for the agent is:
[0028] Furthermore, while multi-agent systems select actions to obtain immediate rewards, they also need to consider future gains. Therefore, the agent's reward value G at time t is... t Represented as
[0029]
[0030] Among them, R t+k+1 This represents the agent's reward value at time t+k+1. γ represents the discount factor. The larger the γ is, the more important future rewards are; the smaller the γ is, the more important current rewards are.
[0031] (4) Model Composition: The agent model consists of an interaction network Q(θ), an evaluation network Q′(θ′), an experience pool, and a gradient update function. θ and θ′ refer to the neural network coefficients of the interaction network and the evaluation network, respectively, and are composed of the weights and biases of each layer. Their roles and relationships are as follows:
[0032] The interactive network is responsible for interacting with the environment in real time. At time t, the input state of the i-th agent is s. i The output action is a i After the agent completes its action, the environment will provide the next state s.i+1 and reward R i The agent will use the quadruple i a i R i s i+1 Store in the experience pool;
[0033] The gradient update function uses gradient descent, updating the gradient of the interactive network by obtaining quadruples from the experience pool and the output of the evaluation network. The formula for gradient descent is min(R... i +γmax a′ Q′(s i+1 ,a′;θ′)-Q(s i a i ;θ)) 2 Where a′ represents the result of completing action a i Arrival in state s i+1 The next optional action;
[0034] After a certain number of iterations, the evaluation network obtains neural network parameters from the interaction network and updates some of its own neural network parameters to ensure the accuracy of the evaluation network's gradient update and the convergence of the algorithm.
[0035] This process is repeated iteratively until the agent completes the target task or reaches the predetermined number of iterations, thus completing the training of the deep reinforcement learning model for the agent.
[0036] Furthermore, the parameter aggregation algorithm in step S5 includes the following sub-steps:
[0037] S51. Ordinary agents within the task area upload local parameter samples to the enhanced agent;
[0038] S52. The enhanced agent calculates the center point of the parameter samples and calculates the Mahalanobis distance from each parameter sample to the center point.
[0039] S53. Remove nodes whose Mahalanobis distance to the center point is greater than a preset threshold;
[0040] S54. Perform a weighted average on the remaining nodes and return the result to the ordinary agent;
[0041] S55. Ordinary intelligent agents update local model parameters using a soft update method.
[0042] Furthermore, the specific implementation method for calculating the center point in step S52 is as follows: Let θ be the model parameter sample uploaded by the i-th ordinary intelligent agent. i Each time, the enhanced intelligent agent or parameter aggregation collaboration platform receives a set of model parameter samples, Θ = {θ}. i The center point θ of the sample is obtained by calculating the average value. avg ,Right now n is the number of model parameter samples;
[0043] The Mahalanobis distance from the parameter sample to the center point is calculated as follows: First, calculate the covariance matrix ∑ X The formula is as follows:
[0044]
[0045] Where X is of size n×m, that is, n model parameter samples, each sample has m dimensions;
[0046] Then the i-th model parameter sample θ i With center point θ avg Mahalanobis distance D ma (θ i θ avg )for:
[0047]
[0048] Further, step S54 is specifically implemented as follows: scoring each ordinary agent based on its capability and credibility, and assigning it different weights; let the sample weight of the model parameters of the i-th ordinary agent be w. i Let Φ be the set of model parameter samples after parameter filtering, then the weighted average result θ agg for:
[0049]
[0050] Finally, the result θ agg Returned to each ordinary intelligent agent.
[0051] Furthermore, the specific implementation method of step S55 is as follows: the ordinary intelligent agent receives the updated parameter θ returned by the enhanced intelligent agent or the parameter aggregation and cooperation platform. agg Then, the local neural network model is updated using a soft update method, that is, θ is adjusted by a weight τ. agg Add the current local neural model parameters θ t In the middle, the updated neural network parameters θ t+1 For: θ t+1 =(1-τ)θ t +τθ agg This completes one federated learning process, where τ∈[0,1]; when τ is 0, it indicates that the parameter θ will not be updated. agg Integrate into the local model; when τ is 1, it means that the local model directly copies and updates the parameter θ. aggBased on experience, the weight τ is initially set to a relatively large value to speed up the training process. Then, during the training iterations, the weight is gradually reduced to ensure convergence stability.
[0052] The beneficial effects of this invention are: This invention enhances the collaboration between agents based on deep reinforcement learning through federated learning. Multiple agents, while exploring the environment and making decisions through deep reinforcement learning, simultaneously acquire learning experience from other agents using federated learning technology, thereby effectively improving agent learning efficiency and reducing collaboration overhead. Because the method of this invention has a certain generalization ability, it can be applied to multi-agent collaboration in different scenarios. Attached Figure Description
[0053] Figure 1 A diagram illustrating a multi-agent collaborative scenario;
[0054] Figure 2 This is a flowchart of the multi-agent federated cooperation method based on deep reinforcement learning of the present invention;
[0055] Figure 3 This is a flowchart of the parameter aggregation algorithm of the present invention. Detailed Implementation
[0056] The technical solution of the present invention will be further described below with reference to the accompanying drawings.
[0057] The intelligent agents described in this invention possess physical characteristics and can refer to high-performance and ordinary intelligent detection devices in network security testing, or multiple unmanned intelligent devices or systems, such as drones, autonomous vehicles, and sensor nodes. In network security testing, intelligent detection devices or nodes can be considered intelligent agents. A single intelligent detection device, limited by its own computing power and deployment in a local environment, has a limited security protection effect. Therefore, they need to cooperate through multi-agent collaboration to improve overall security protection capabilities and achieve a global defense effect. In multiple unmanned intelligent devices or systems, cooperation is required to complete certain tasks, involving collaboration between a master coordinator and subordinates. For example, in the field of multi-drone cooperative control, drones can be divided into leaders and followers, and they need to cooperate effectively to complete tasks such as target tracking and path planning. This also falls under multi-agent collaboration and has significant application value in both military and civilian fields. Multi-agent collaboration scenarios include... Figure 1As shown in the diagram, in this scenario, multiple agents need to collaborate to complete a task, and there is a parameter aggregation and collaboration platform. Agents can be divided into enhanced agents (E-Agents) and ordinary agents. Based on the agents' capabilities, such as communication and computing resources, some agents with stronger capabilities can be designated as enhanced agents. Depending on the specific circumstances, complex tasks can be divided into multiple sub-tasks, i.e., task regions. Multiple ordinary agents and one enhanced agent are deployed within each task region. Agents within the same task region have the same model, while agent models in different task regions may not be the same.
[0058] This invention deploys deep reinforcement learning algorithms on all agents and employs federated learning on the enhanced agents and a parameter aggregation collaboration platform. Each agent builds its own neural network model and interacts with the environment to collaboratively complete a designated task. The enhanced agents can obtain information from other agents within their region, aggregating their parameter information and returning the results to accelerate the updating of local neural model parameters for other agents, guiding effective collaboration among multiple agents to quickly achieve the goal. Simultaneously, the parameter aggregation collaboration platform acquires the model parameters of enhanced agents within each task region and implements federated learning for enhanced agents with similar task types. Using federated learning, the enhanced agents and the collaboration platform aggregate the parameters uploaded by the agents, removing outliers and anomalies, and then performing a weighted average of the remaining parameters. The result is returned to each agent for parameter updates. The agents soft-update the received results with their local neural network parameters, adding the results to their local parameters with a certain weight. In this way, each agent can learn from the experience of other agents, optimize its local model, achieve better collaborative results, and jointly complete the task.
[0059] like Figure 2 As shown, the present invention provides a multi-agent federated cooperation method based on deep reinforcement learning, comprising the following steps:
[0060] S1. Divide the task and the agent, establish a deep reinforcement learning model for each agent, and set the state space of the agent in the environment, the action space that the agent can decide, and the reward of the agent's actions in the environment.
[0061] The specific implementation method for dividing tasks and agents is as follows: divide multiple agents into multiple task regions, with one enhanced agent and multiple ordinary agents in each task region; designate the most capable agent in each task region as the enhanced agent, and the remaining agents as ordinary agents.
[0062] The deep reinforcement learning model is established as follows: There are N agents in the space, numbered from 1 to n, with the i-th agent numbered i; the state space is S, the action space is A, the reward value is R, and the triplet...<S,A,R> This determines the specific multi-agent model;
[0063] (1) State Space: Each agent needs to establish its own state space. However, since in most cases, multiple agents explore space and complete tasks in a large environment, agents in the same task area generally have the same state space, while agents in different task areas may have different state spaces. Therefore, under the same state space, if the state space of the i-th agent is S... i The state space of the j-th agent is S. j Then S i =S j , 1≤i≤N, 1≤j≤N, i≠j, N is the number of agents; under different state space conditions, the agent state space S is established according to the specific scenario. i and S j ;
[0064] (2) Behavior space: A behavior space is established for each agent. Agents with the same state space have the same behavior space. Under different state space conditions, the agent behavior space is established according to the specific scenario.
[0065] (3) Reward value: The reward value R of a multi-agent system includes the following three rewards:
[0066] a) Individual instant reward r1: The environment provides an instant reward to the current agent. Let the current state be s. i The next state is s i+1 The desired state is s e If |s i+1 -s e |<|s i -s e If |, it is determined that the target is close and rewarded with p; otherwise, it is determined that the target is far away and rewarded with -p.
[0067] b) Global instant reward r2: The reward given to all agents by the current environment. Let the current state of the global agents be s. t , Let be the state of the j-th agent at time t, and let s be the next state of the global agent. t+1 When the task is completed, the state of the global agent is s. e If |s t+1 -s e |<|s t -s eIf |, it is determined that the agent is close to the target and a reward of q is given to each agent; otherwise, it is determined that the agent is far from the target and a reward of -q is given to each agent.
[0068] Where p and q are positive numbers set by the user, satisfying q≤p;
[0069] c) Global Final Reward r3: This is the reward given to all agents globally when the multi-agent system completes the objective; no reward is given in other cases. The global final reward is related to the total time (number of steps); that is, the longer the total time, the smaller the final reward, and the shorter the total time, the larger the final reward. Therefore, the global final reward is expressed as... Where T is the total time, α and β are coefficients, α < 0, β > 0, set by the user, and can be adjusted according to specific needs to satisfy β ≥ 10(p+q), ensuring that r3 is tens of times (p+q) or more, prompting the agent to learn the currently known approximate optimal strategy;
[0070] Therefore, the reward value for the agent is:
[0071] Furthermore, while multi-agent systems select actions to obtain immediate rewards, they also need to consider future gains. Therefore, the agent's reward value G at time t is... t Represented as
[0072]
[0073] Among them, R t+k+1 This represents the agent's reward value at time t+k+1. (This is an unlimited discount model, therefore) i has no upper limit; γ represents the discount coefficient. The larger γ is, the more future rewards are valued, and the smaller γ is, the more current rewards are valued.
[0074] (4) Model Composition: The agent model consists of an interaction network Q(θ), an evaluation network Q′(θ′), an experience pool, and a gradient update function. θ and θ′ refer to the neural network coefficients of the interaction network and the evaluation network, respectively, and are composed of the weights and biases of each layer. Their roles and relationships are as follows:
[0075] The interactive network is responsible for interacting with the environment in real time. At time t, the input state of the i-th agent is s. i The output action is a i After the agent completes its action, the environment will provide the next state s. i+1 and reward R i The agent will use the quadruple i a i R i s i+1 Store in the experience pool;
[0076] The gradient update function uses gradient descent, updating the gradient of the interactive network by obtaining quadruples from the experience pool and the output of the evaluation network. The formula for gradient descent is min(R... i +γmax a′ Q′(s i+1 ,a′;θ′)-Q(s i a i ;θ)) 2 Where a′ represents the result of completing action a i Arrival in state s i+1 The next optional action;
[0077] After a certain number of iterations, the evaluation network obtains neural network parameters from the interaction network and updates some of its own neural network parameters to ensure the accuracy of the evaluation network's gradient update and the convergence of the algorithm.
[0078] This process is repeated iteratively until the agent completes the target task or reaches the predetermined number of iterations, thus completing the training of the deep reinforcement learning model for the agent.
[0079] S2. Based on the deep reinforcement learning algorithm, a corresponding neural network is established for each agent. Agents in the same task area establish the same neural network model, while agents in different task areas can establish different neural network models.
[0080] S3. The agent decides its next action based on the assigned goal and its current state, interacts with the environment to obtain its next state, stores the decision experience in the experience pool, and updates the local neural network model in a timely manner according to the stochastic gradient descent method.
[0081] S4. After the loop reaches the set number of iterations, the local neural network model parameters are transmitted to the collaborative platform for subsequent federated learning.
[0082] S5. Aggregate the parameters uploaded by all agents and return the results to each agent for parameter updates; The collaboration platform divides agents according to the type of task and the enhanced agent model, aggregates the parameters of enhanced agents with similar tasks and the same model, and returns the corresponding aggregation results.
[0083] This section utilizes federated learning, a collaborative platform, and enhanced agents to process the parameters uploaded by the agents and return the results to the agents, accelerating the learning process. The main function of federated learning is to aggregate parameters, improving the overall accuracy of the model parameters. The parameter aggregation algorithm flow is as follows: Figure 3 As shown, it includes the following sub-steps:
[0084] S51. Ordinary agents within the task area upload local parameter samples to the enhanced agent;
[0085] S52. The enhanced agent calculates the center point of the parameter samples and calculates the Mahalanobis distance from each parameter sample to the center point.
[0086] The specific method for calculating the center point is as follows: Let θ be the model parameter sample uploaded by the i-th ordinary agent. i Each time, the enhanced intelligent agent or parameter aggregation collaboration platform receives a set of model parameter samples, Θ = {θ}. i The center point θ of the sample is obtained by calculating the average value. avg ,Right now n is the number of model parameter samples;
[0087] The Mahalanobis distance from the parameter sample to the center point is calculated as follows: First, calculate the covariance matrix ∑ X The formula is as follows:
[0088]
[0089] Where X is of size n×m, that is, n model parameter samples, each sample has m dimensions;
[0090] Then the i-th model parameter sample θ i With center point θ avg Mahalanobis distance D ma (θ i θ avg )for:
[0091]
[0092] S53. Remove nodes whose Mahalanobis distance to the center point is greater than a preset threshold; set a removal threshold ∈ D, if D ma If >∈, then the model parameter sample is removed; if D ma If the threshold ≤ ∈, then the model parameter sample is retained. The value of the threshold ∈ can be flexibly adjusted according to actual needs. The smaller ∈ is, the fewer samples are effectively included in the aggregation; the larger ∈ is, the more samples are effectively included in the aggregation.
[0093] Steps S52 and S53 mainly involve preprocessing the model parameter samples uploaded by the agent, filtering outliers, removing outliers, and ensuring the convergence of the final result.
[0094] S54. Perform a weighted average on the remaining nodes and return the result to the ordinary agent. The specific implementation method is as follows: score each ordinary agent based on its ability and credibility, assigning different weights; let the sample weight of the model parameters of the i-th ordinary agent be w. iLet Φ be the set of model parameter samples after parameter filtering, then the weighted average result θ agg for:
[0095]
[0096] Finally, the result θ agg Returned to each ordinary intelligent agent.
[0097] S55. Ordinary agents update their local model parameters using a soft update method; specifically, the ordinary agent receives the updated parameters θ returned by the enhanced agent or the parameter aggregation and collaboration platform. agg Then, the local neural network model is updated using a soft update method, that is, θ is adjusted by a weight τ. agg Add the current local neural model parameters θ t In the middle, the updated neural network parameters θ t+1 For: θ t+1 =(1-τ)θ t +τθ agg This completes one federated learning process, where τ∈[0,1]; when τ is 0, it indicates that the parameter θ will not be updated. agg Integrate into the local model; when τ is 1, it means that the local model directly copies and updates the parameter θ. agg Based on experience, the weight τ is initially set to a relatively large value to speed up the training process. Then, during the training iterations, the weight is gradually reduced to ensure convergence stability.
[0098] S6. Each agent performs a soft update on the received return results and the local neural network parameters to obtain the latest local model parameters.
[0099] S7. Repeat S3-S6 until the target task is completed.
[0100] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.
Claims
1. A multi-agent federated cooperation method based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Divide tasks and intelligent agents. Intelligent agents refer to high-performance intelligent detection equipment and ordinary intelligent detection equipment in network security detection. Or it can refer to multiple unmanned intelligent devices or systems, including drones, autonomous vehicles or sensor nodes; establish a deep reinforcement learning model for each intelligent agent, set the state space of the intelligent agent in the environment, the behavior space that the intelligent agent can decide, and the environmental rewards for the intelligent agent's behavior. Multiple agents are divided into multiple task regions. Each task region contains one enhanced agent and multiple ordinary agents. The most capable agent in each task region is designated as the enhanced agent, and the remaining agents are designated as ordinary agents. The deep reinforcement learning model is established as follows: There are N agents in the space, and the agents are numbered from... arrive , No. The agent's ID is ; Let the state space be , the action space be , the reward value be , and the triple determine the specific multi-agent model; (1) State space: Under the same state space, if the state space of the i-th agent is The state space of the j-th agent is ,but In different state space scenarios, the agent's state space is established according to the specific scenario. and ; (2) Behavior space: A behavior space is established for each agent. Agents with the same state space have the same behavior space. Under different state space conditions, the agent behavior space is established according to the specific scenario. (3) a reward value, a reward value of the multi-agent system includes the following three rewards: a) Individual Instant Rewards The environment provides real-time feedback as a reward to the current agent; let the current state be... The next state is The desired state is ,like If it is determined to be close to the target, a reward will be given. Otherwise, it means moving away from the target, and a reward will be given. ; b) Global Instant Rewards The reward given to all agents by the current environment, where the current global agent state is denoted as . , , , Let be the state of the j-th agent at time t, and let the next state of the global agent be... When the task is completed, the state of the global agent is ,like If the target is nearby, then each agent is rewarded. Otherwise, to distance themselves from the target, each agent is rewarded. ; in, q is a positive number set by the user, satisfying ; c) Global Final Reward The global final reward is given to all agents when the multi-agent system completes its objective; otherwise, no reward is given. The global final reward is related to the total time; the longer the total time, the smaller the final reward, and vice versa. Therefore, the global final reward is expressed as: ,in Total time For coefficients, , It is set by the user and meets the requirements. ,Assure for This is tens of times or more than the current optimal strategy, prompting the agent to learn the currently known near-optimal strategy. Therefore, the reward value for the agent is: ; Furthermore, multi-agent systems, while selecting actions to obtain immediate rewards, also need to consider future benefits; therefore, agents in... Reward value at any moment Represented as ； in, This represents the agent's reward value at time t+k+1. ; Represents the discount factor. The larger the value, the more important future rewards are. The smaller the value, the more important the current reward is. (4) Model composition: The model of the intelligent agent consists of an interaction network. Evaluation Network It consists of an experience pool and a gradient update function, among which These refer to the neural network coefficients of the interaction network and the evaluation network, respectively, which are composed of the weights and biases of each layer of the network; their functions and relationships are as follows: Interactive networks are responsible for interacting with the environment in real time. At time i, the input state of the i-th agent is The output action is After the agent completes its action, the environment will provide the next state. and rewards The agent will use the quadruple Store in the experience pool; The gradient update function uses gradient descent, updating the gradient of the interactive network by obtaining quadruples from the experience pool and the output of the evaluation network. The formula for gradient descent is: ,in To complete the action Arrival Status The next optional action; After a certain number of iterations, the evaluation network obtains neural network parameters from the interaction network and updates some of its own neural network parameters to ensure the accuracy of the evaluation network's gradient update and the convergence of the algorithm. This process is repeated iteratively until the agent completes the target task or reaches the predetermined number of iterations, thus completing the training of the deep reinforcement learning model for the agent. S2. Based on the deep reinforcement learning algorithm, establish a neural network for each agent; S3. The agent decides its next action based on the assigned goal and its current state, interacts with the environment to obtain its next state, stores the decision experience in the experience pool, and updates the local neural network model in a timely manner according to the stochastic gradient descent method. S4. After the loop reaches the set number of iterations, the local neural network model parameters are transmitted to the collaborative platform for subsequent federated learning. S5. Aggregate the parameters uploaded by all agents and return the results to each agent for parameter updates; The collaboration platform divides agents according to the type of task and the enhanced agent model, aggregates the parameters of enhanced agents with similar tasks and the same model, and returns the corresponding aggregation results. S6. Each agent performs a soft update on the received return results and the local neural network parameters to obtain the latest local model parameters. S7. Repeat S3-S6 until the target task is completed.
2. The multi-agent federated cooperation method based on deep reinforcement learning according to claim 1, characterized in that, The parameter aggregation algorithm in step S5 includes the following sub-steps: S51. Ordinary agents within the task area upload local parameter samples to the enhanced agent; S52. The enhanced agent calculates the center point of the parameter samples and calculates the Mahalanobis distance from each parameter sample to the center point. S53. Remove nodes whose Mahalanobis distance to the center point is greater than a preset threshold; S54. Perform a weighted average on the remaining nodes and return the result to the ordinary agent; S55. Ordinary intelligent agents update local model parameters using a soft update method.
3. The multi-agent federated cooperation method based on deep reinforcement learning according to claim 2, characterized in that, The specific implementation method for calculating the center point in step S52 is as follows: Let the model parameter sample uploaded by the i-th ordinary intelligent agent be denoted as . Each time, the enhanced intelligent agent or parameter aggregation collaboration platform receives a set of model parameter samples. The center point of the sample is obtained by calculating the average value. ,Right now n is the number of model parameter samples; The Mahalanobis distance from the parameter sample to the center point is calculated as follows: First, the covariance matrix is calculated. The formula is as follows: ； in, Size is ,Right now There are model parameter samples, each sample has dimension; Then the first Sample of model parameters With the center point Mahalanobis distance for: 。 4. The multi-agent federated cooperation method based on deep reinforcement learning according to claim 3, characterized in that, The specific implementation method of step S54 is as follows: Scoring the ordinary intelligent agents based on their capabilities and credibility, and assigning them different weights; let the... The model parameter sample weights of a typical agent are: Let the set of model parameter samples after parameter filtering be... The result of the weighted average for: ； Finally, the results Returned to each ordinary intelligent agent.
5. A multi-agent federated cooperation method based on deep reinforcement learning according to claim 4, characterized in that, The specific implementation method of step S55 is as follows: the ordinary intelligent agent receives the updated parameters returned by the enhanced intelligent agent or the parameter aggregation and collaboration platform. Then, the local neural network model is updated using a soft update method, that is, by weight. Will Add local current neural model parameters In the middle, the updated neural network parameters for: To complete a federated learning process, in which ;when A value of 0 indicates that the parameter will not be updated. Integrate into the local model; when When the value is 1, it means that the parameters are directly copied and updated in the local model. .