Multi-agent reinforcement learning communication method, terminal device and storage medium
Through the multi-agent reinforcement learning communication method, combined with signal gradient optimization and extended obedience constraints, the decision-making bias of agent games in mixed-motive scenarios is solved, and effective communication and decision optimization of multi-agent systems are achieved.
Patent Information
- Application Number
- CN202310397744.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-06
- Publication Date
- 2025-09-16
- Estimated Expiration
- 2043-04-06
AI Technical Summary
Existing multi-agent reinforcement learning communication algorithms only consider fully cooperative scenarios and fail to effectively handle the game between agents in mixed-motivation scenarios, resulting in decision-making deviations.
A multi-agent reinforcement learning communication method is adopted. Through environment modeling, trajectory data update and signal gradient optimization, combined with extended obedience constraints, the network model of the signal sender and signal receiver is updated. The expected future benefits of the signal strategy in the sequential game are considered to achieve effective communication under mixed motivations.
Effective communication between agents is achieved in mixed-motivation scenarios, which solves the problem of decision-making bias in existing technologies and improves the decision-making optimization capability of multi-agent systems.
Smart Images
Figure CN116455754B_ABST
Abstract
Description
Technical Field
[0001] The present invention discloses a multi-agent reinforcement learning communication method, terminal equipment and storage medium, belonging to the technical field of multi-agent reinforcement learning. Background Art
[0002] Multi-agent systems are complex systems composed of multiple agents, each capable of perception, interaction, and autonomous decision-making. Depending on the relationships between agents (competition / cooperation), they can be categorized as competitive, cooperative, or hybrid. Reinforcement learning, a machine learning method that optimizes decision-making through trial-and-error learning, has the advantage of being independent of any prior knowledge or external environmental models. In recent years, multi-agent reinforcement learning technology has rapidly developed and has found widespread application in real-world applications such as self-driving cars, computer game scenarios, robot swarms, and sensor networks.
[0003] Currently, multi-agent reinforcement learning communication algorithms typically only consider communication between multiple agents in fully cooperative scenarios. However, in reality, the behavior between agents is often mixed, encompassing both cooperative and competitive behavior. In these mixed-motivation scenarios, different agents aim to optimize different objectives, and the interactions between these agents with different goals involve a game. Existing techniques only consider communication in fully cooperative scenarios, ignoring the game between agents, which can lead to poor equilibrium in the evolution of decision-making. Summary of the Invention
[0004] The purpose of this application is to provide a multi-agent reinforcement learning communication method, terminal device and storage medium to solve the technical problem that the existing technology only considers the communication of multiple agents in a fully cooperative scenario, does not consider the game between multiple agents, and leads to deviations in decision-making.
[0005] A first aspect of the present invention provides a multi-agent reinforcement learning communication method, comprising:
[0006] Step 1: Model an intelligent agent model based on the environment, where the intelligent agent includes a signal sender and a signal receiver, and set the hyperparameters of the intelligent agent model;
[0007] Step 2: Obtain trajectory data generated by the agent's interaction in the environment;
[0008] Step 3: Based on the trajectory data, the action strategy network, evaluation network, and target evaluation network corresponding to each agent are updated, and the signal network of the signal sender is updated based on the signal gradient and the extended compliance constraint;
[0009] Step 4: Adjust the hyperparameters of the agent and repeat steps 2-3 to select the agent model with the best performance among multiple sets of hyperparameter groups.
[0010] Preferably, according to the signal gradient and the extended compliance constraint, an optimization method is used to calculate and update the signal network of the signal sender, specifically including:
[0011] When the signal strategy of the signal sender satisfies the extended obedience constraint, the update gradient of the signal network is the signal gradient;
[0012] When the signal strategy of the signal sender does not satisfy the extended compliance constraint, the update gradient of the signal network is the gradient of the signal gradient and the extended compliance constraint with respect to the parameters of the signal network;
[0013] After the gradient calculation is completed, the parameters of the signal network of the signal sender are updated using the ascent method.
[0014] Preferably, the signal set of the signal sender used in the extended compliance constraint is determined based on the signal that maximizes the expected posterior benefit of the signal receiver.
[0015] Preferably, the calculation of the gradient of the signal network is determined based on the influence of the signal sent by the signal sender on the signal receiver in the agent model;
[0016] The influence of the signal receiver includes the action strategy of the signal receiver.
[0017] Preferably, updating the action strategy network, evaluation network, and target evaluation network corresponding to each agent according to the trajectory data specifically includes:
[0018] Updating the action strategy network, evaluation network, and target evaluation network corresponding to the signal receiver;
[0019] Updating the action strategy network, evaluation network, target evaluation network and signal network corresponding to the signal sender;
[0020] The evaluation network and the target evaluation network maintained by the signal sender for the signal receiver are updated.
[0021] Preferably, updating the action strategy network, evaluation network and target evaluation network corresponding to the signal receiver specifically includes:
[0022] updating the action policy network of the signal receiver using a policy gradient method according to the trajectory data of the signal receiver;
[0023] Calculating a temporal difference error using the trajectory data and the target evaluation network, and updating the evaluation network;
[0024] The target evaluation network is updated in a soft update manner.
[0025] Preferably, the trajectory data is collected and stored in an experience replay pool;
[0026] The experience replay pool includes the experience replay pool of the signal receiver and the experience replay pool of the signal sender.
[0027] A terminal device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.
[0028] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.
[0029] The multi-agent reinforcement learning communication method, terminal device, and storage medium of the present invention have the following beneficial effects:
[0030] (1) The present invention is a multi-agent communication system, a method for multi-agent communication in mixed-motivation scenarios implemented by reinforcement learning. By extending the obedience constraints in computational economics information design to sequential games and explicitly considering the impact of signaling strategies on future profit expectations in sequential games, the agents can learn to generate effective communication in mixed-motivation scenarios.
[0031] (2) This paper proposes a new model for modeling the communication process between a signal sender and a signal receiver, as well as their interaction in an environment. This random process is called a Markov signaling game. This method resolves the unreasonable assumptions in current research on extending Bayesian persuasion to sequential decision making.
[0032] (3) The present invention derives the gradient of the signal sender's expected benefit in sequential decision-making with respect to the strategy parameters of its own signal transmission, namely, the signal gradient.
[0033] (4) In actual learning, the present invention updates the signal strategy of the sender based on its expected benefit, and its gradient is calculated as the result of the signal gradient, while also satisfying the extended obedience constraint. Thus, a method for multi-agent communication in a mixed-motivation scenario using reinforcement learning is implemented. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 Schematic diagram of the steps of the multi-agent reinforcement learning communication method of the present invention. DETAILED DESCRIPTION
[0035] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.
[0036] A first aspect of the embodiments of the present invention provides a multi-agent reinforcement learning communication method, specifically comprising:
[0037] Step S1: Building an intelligent agent model based on the environment, where the intelligent agent includes a signal sender and a signal receiver, and setting hyperparameters of the intelligent agent model;
[0038] Step S2: enabling the agent to interact in the environment and collecting trajectory data of the agent;
[0039] Step S3: updating the action strategy network, evaluation network, and target evaluation network corresponding to each agent according to the trajectory data, and calculating and updating the signal network of the signal sender using an optimization method based on the signal gradient and the extended compliance constraint;
[0040] Step S4: Adjust the hyperparameters of the agent and repeat steps 2-3 until the reward curve of each agent converges or reaches the specified maximum number of training times, and select the agent model with the best performance among multiple sets of hyperparameter groups.
[0041] In the embodiment of the present invention, the extended obedience constraint in Bayesian persuasion is determined by the probability of the signal sender sending a signal, the probability and expectation of the signal receiver receiving the signal sent by the signal sender and taking an action, and the probability and expectation of the signal receiver receiving the signal sent by other signal senders and taking an action, under each environmental state s, as shown in formula (1):
[0042]
[0043] Where, is the probability of the environment state s∈S occurring in equilibrium given the signaling strategy and the joint action strategy of all persons; is the signal strategy, specifically the probability that the signal sender sends a joint signal σ when the environment state is s, defined as σ j is the signal sent by the signal sender to the jth signal receiver, where η is the parameter of the signal strategy; is the action strategy of the jth signal receiver, specifically, the jth signal receiver receives the signal σ j When , take action a j The probability of θ is the parameter of the action strategy; W j(*) is the expected benefit of the j-th signal receiver in a given state S and joint action a; When the jth signal receiver receives the signal σ′, it takes action a j probability.
[0044] In the embodiment of the present invention, based on the judgment result, the signal network of the signal sender is updated using a corresponding update method, specifically including:
[0045] When the extension obedience constraint is satisfied, the gradient used by the signal network to update is only the signal gradient, and the specific calculation formula is as follows:
[0046]
[0047] Where, is the expected benefit of the signal sender in state s∈S, is the expected benefit of the signal sender in a given state S and joint action a, π θ (a|, s, σ) is a joint strategy, defined as in is the action strategy of the signal sender, specifically, when the signal sender observes the environment state S and the joint signal σ sent by itself, it takes action a i probability.
[0048] When the extension obedience constraint is not satisfied, the gradient used by the signal network to update is the signal gradient plus the gradient of the extension obedience constraint on the signal network parameters, as shown in formula (3):
[0049]
[0050] Where λ is the Lagrange multiplier, which is a predefined hyperparameter. After the gradient is calculated, the parameters of the sender signal network are updated using the gradient ascent method.
[0051] In the embodiment of the present invention, based on the trajectory data of each agent, the action strategy network, evaluation network, and target evaluation network corresponding to each agent are updated, specifically including:
[0052] According to the trajectory data of the signal receiver, the action policy network of the receiver is updated using the policy gradient method, as shown in formula (4):
[0053]
[0054] in The signal receiver receives the signal σ j The expected return when is the advantage function, and the specific formula is in The output of the evaluation network for the signal receiver is the signal receiver's estimation of the received σ j After taking a j Expected returns;
[0055] The time difference error is calculated using the trajectory data and the target evaluation network, and the evaluation network is updated. The time difference error is specifically where r j is the reward obtained by signal receiver j, and are the outputs of the target evaluation network and the evaluation network respectively.
[0056] The target evaluation network is updated in a soft update mode, and the update formula is μ′ k+1 ←(1-δ)μ′ k +δμ k , where μ and μ′ represent the parameters of the evaluation network and the target evaluation network respectively; the subscripts represent the number of update iterations, and δ is a hyperparameter, which is the update amplitude of the target evaluation network; the updates of the signal sender's action strategy network, evaluation network, and target evaluation network are similar.
[0057] A second aspect of the present invention provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0058] A third aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0059] The method of the present invention will be described in detail below with more specific examples.
[0060] Step 1: Determine the target task for the multi-agent system in a mixed-motivation scenario and create a simulation environment based on the target task. Create a communication topology between each agent and set the state transition probability function for the environment. Set the type, observation interval, and reward function for each agent. Specifically, each agent type includes a signal sender and a signal receiver. Define a Markov signaling game. A Markov game can be represented by a nine-tuple, which is:
[0061]
[0062] Where i represents the number of the signal sender, J represents the set of signal receivers, and S represents the state set of the simulation environment that the agent interacts with. It is the set of signals that the signal sender can send to each signal receiver. Specifically, the channel between the signal sender and each signal receiver is private and cannot be observed by other agents. i represents the set of actions that the signal sender can take in the simulation environment, represents the set of actions that each signal receiver can take in the simulation environment, represents the reward function of the signal sender, represents the reward function of the signal receiver, and T:S×A→Δ(S) represents the state transition function of the simulation environment.
[0063] The present invention is applicable to communication problems in incomplete information sequence games. The sender possesses private information about the payoffs of their decisions, giving them an information advantage over the receiver. Each agent in the game is self-interested and optimizes only its own expected payoff. Creating a simulation environment requires determining the communication topology between agents and the state transition probability function of the environment. The information between the sender and each receiver with whom it can communicate in the communication topology is private, and other receivers cannot observe the signals sent by the sender to the current receiver.
[0064] Algorithms have already incorporated mechanism design into multi-agent algorithms to address the proper distribution of rewards between oneself and others. However, further research is needed on information design. Currently, information design in economics has been extensively studied in the context of a single signal sender persuading a single receiver, with explicit constraint expressions. Specifically, the signal sender's recommended action strategy is: if the receiver's expected payoff increases compared to not following the advice, then the receiver is guaranteed to follow the advice. The signal sender selects the best-payoff recommendation from a set of recommendations that the receiver is guaranteed to follow, thereby leveraging information advantages to improve their own expected payoff. This proposed algorithm analyzes a communication model between self-interested agents operating under mixed motivations. This one-on-one scenario is known as Bayesian persuasion.
[0065] Currently, Bayesian persuasion requires the assumption that the agent's game only occurs once, which is inconsistent with the sequential decision-making assumption among agents in multi-agent reinforcement learning algorithms. Further analysis is needed to incorporate this assumption. Some researchers have extended this approach to the context of sequential decision-making and proposed several models. However, these models all assume that the sender's information advantage does not affect state transitions, that is, the probability of state transitions is independent of the sender's unique private information. This assumption does not conform to the information advantage of the signaler in multi-agent reinforcement learning. Furthermore, Bayesian persuasion, while based on the perspective of evolutionary equilibrium, makes some reasonable assumptions to simplify the problem. However, these assumptions do not generally conform to the evolutionary process and cannot guide the signaler's strategy updates during evolution. For example, the commitment assumption requires the sender to determine their signaling strategy before the interaction begins, communicate this strategy to the receiver, and promise not to change it. This does not conform to the learning required in multi-agent interactions. Another situation is that Bayesian persuasion uses an analysis similar to the revelation principle. It is believed that from the perspective of the results, the process of "the sender sends a signal and the receiver takes an action based on the signal" can be transformed into "the sender directly recommends the receiver to take an action" without loss of generality. However, in common multi-agent scenarios, the number of states is greater than the number of actions. If the signal space is selected as the action space rather than the state space, then during the evolution process, the signal sender's ability to transmit information will decrease.
[0066] The present invention aims to extend Bayesian persuasion in economic information design to sequential games and eliminate the assumptions analyzed from evolutionary results, introduce it into multi-agent communication algorithms, and fill the gap in communication between agents in mixed-motive scenarios.
[0067] Build a model for each agent in the multi-agent system and set the model's hyperparameters;
[0068] Each agent type includes a signal sender and a signal receiver. Like the agents in single-agent reinforcement learning, each signal receiver maintains an action strategy network, an evaluation network, and a target evaluation network. The networks maintained by the above signal receivers are used to participate in the interaction and learning of the simulation environment to obtain higher benefits in the environment.
[0069] Each signal sender maintains an action strategy network, an evaluation network and a target evaluation network, and maintains an evaluation network and a target evaluation network for each signal receiver. In addition, a signal network is also maintained. The networks maintained by the above signal senders are used to estimate and judge the expected benefits of the signal receiver in each state.
[0070] Each agent maintains a private experience replay pool to store data obtained from its interactions in the simulation environment.
[0071] Set hyperparameters for each agent's model, including: the learning rate of each network, the optimizer type, the number of network layers, the type of activation function, the amplitude of the soft update of the target evaluation network, the increase ratio of the reward function, the type of agent exploration, and specific parameters (such as the entropy coefficient in the regularization term for introducing entropy into the policy network), the value of the Lagrangian variable λ in the Lagrangian function, the slack amount ∈, and the batch capacity, etc.
[0072] Step 2: Multiple agents interact in the simulation environment, obtain trajectory data, and store the corresponding amount of trajectory data in the experience replay pool of each agent. The corresponding amount of trajectory data stored by the information receiver is The amount of trajectory data stored by the sender is (..., s t ,σ t , a t , r t , ...).
[0073] The sequential process of communication and interaction between agents in the simulation environment is as follows: Signal sender agent After observing the state of the environment s∈S, according to Send signal σ j ∈∑ j Give each signal receiver j∈J. Then all agents select their own actions from their own action sets, where the action strategy of the sender is Indicates that the sender sees the environment state s and the signal it sends to everyone Then select a i The probability of , and the action strategy of the receiver j∈J is Here, the signal sender must consider the signal it sends when making decisions. This allows the signal sender to adapt when seeing the same state but sending different signals. Similar to the state-value function and state-action-value function in the Markov decision process, the Markov signaling game also has a quantity that the sender can estimate the expected long-term benefit: and in represents the future cumulative reward of the signal sender under the current circumstances, and we assume that at any moment, as long as the agents choose any action under the current state of the environment, no matter what the signal sent by the sender is, everyone’s reward is certain, so
[0074] In this method, the obedience constraint in Bayesian persuasion is extended to sequential decision making, and the result is in is the probability of state s∈S occurring given a signaling strategy and everyone's joint action strategy. Bayesian persuasion is analyzed from the perspective of evolutionary equilibrium outcomes, making a commitment assumption and employing an analysis similar to the revelation principle to simplify the problem. However, in the context of reinforcement learning sequential decision-making, constraints more consistent with the evolutionary process are required to guide the update of the sender's signaling strategy. The obedience constraints extended by this invention eliminate these assumptions.
[0075] Step 3: Use the trajectory data to update the network of each agent's receiver and each agent's sender respectively;
[0076] Step 3.1: Each signal receiver updates the action policy network, the evaluation network, and the target evaluation network.
[0077] The action policy network update gradient of each signal receiver is:
[0078]
[0079] in is the advantage function; specifically, the advantage function is obtained by the following method:
[0080] The signal receiver takes a batch of data from its own experience replay pool and inputs the σ in this batch of data. j and a j To the first evaluation network, get this batch of And according to and Calculate the advantage function
[0081] The evaluation network and target evaluation network of each signal receiver are updated as follows:
[0082] The signal receiver takes a batch of data from the experience replay pool and inputs the σ in this batch of data. j and a j To the evaluation network and target evaluation network, we can get the batch of and
[0083] The loss of the evaluation network is The evaluation network is updated using the gradient of this loss to the network parameters.
[0084] The update of the target evaluation network is μ′ k+1 ←(1-δ)μ′ k +δμ k , where μ and μ′ represent the parameters of the evaluation network and the target evaluation network respectively; the subscripts represent the number of update iterations, and δ is a hyperparameter, which is the amplitude of the first target evaluation network update.
[0085] Step 3.2: Each signal sender updates its own action policy network, evaluation network, target evaluation network, and signal network, as well as the evaluation network and target evaluation network maintained for each signal receiver. The update method is the same as step 3.1.
[0086] The signal sender takes a batch of data from the experience replay pool and selects σ j′ , determine whether this batch of data meets the expansion obedience constraint:
[0087]
[0088] Where W j (s,a j ) The evaluation network output ∈ maintained by the signal sender for the jth signal receiver is the hyperparameter relaxation amount. If the extension obeys the constraint, the updated gradient of the sender's signal network is
[0089] Otherwise, the gradient of the signal network update is:
[0090]
[0091] in The calculation of is based on the result of the signal gradient, which is obtained as follows:
[0092]
[0093] After calculating the gradient of the signal network, the gradient ascent method is used to update the network.
[0094] Specifically, the optimization goal of the signal sender's signaling strategy is its own expected payoff. However, unlike existing reward mechanisms, the signal sender's influence on the signal receiver occurs not only during the signal receiver's policy update phase but also during the trajectory generation phase in sequential decision-making scenarios. In other words, assuming the signal receiver's strategy is fixed, the signal sender's signaling strategy will affect the sender's own expected payoff, so the signal sender needs to explicitly factor this into its update method. Because expected payoffs must take into account future payoffs, calculating the gradient involves sequentially coupled iterations. Therefore, the derivation of the signal gradient is similar to the classic single-agent policy gradient algorithm. This method derives the gradient of the sender's expected payoff in sequential decision-making with respect to the parameters of its own signaling strategy, and is named the signal gradient.
[0095] This invention is an algorithm for multi-agent communication in a hybrid-motivation scenario using reinforcement learning. It is a combination of the second and third innovations and is named signal gradient that satisfies the extended obedience constraint. In actual learning, the update target of the sender's signal strategy is its own expected benefit, and its gradient is calculated as the result of the signal gradient, while also satisfying the extended obedience constraint. Therefore, the updated solution is the Lagrangian function of this constrained optimization problem, and the gradient is The first of these The calculation of the signal gradient is based on the result of the signal gradient. The second term σ j′ The value is randomly selected from ∑ j The expected estimate in the above formula is the average of a batch of trajectory generation data.
[0096] Step 4: Adjust the hyperparameters of the agent and repeat steps 2-3 until the reward curve of each agent converges or reaches the specified maximum number of training times, and select the agent model with the best performance among multiple sets of hyperparameter groups.
[0097] Step 4.1: Repeat step 3 until the reward curve of each agent converges or the preset maximum number of training times is reached. Test the performance of the multi-agent and perform quantitative evaluation.
[0098] Step 4.2 adjusts the hyperparameters in step 2 and repeats steps 3 to 4.1 until the quantitative evaluation in step 4.1 meets the preset standards and the corresponding hyperparameters are obtained.
[0099] The setting of the present invention limits the environment to a single signal sender. If there are multiple information senders and the game between the senders in sending information is not considered, the update formula of each sender is the same as that of the case with only a single sender.
[0100] The present invention transforms the optimization problem with extended obedience constraints into an unconstrained optimization problem through the Lagrange multiplier method to solve it iteratively. If we consider adding a regularization term to the optimization objective of the sender signal network This can be interpreted as the sender improving others' expected posterior benefits while optimizing his own benefits.
[0101] This invention is an algorithm for multi-agent communication in mixed-motivation scenarios using reinforcement learning. By extending the obedience constraints used in computational economics information design to sequential games and explicitly considering the impact of signaling strategies on expected future returns in sequential games, the agents can learn to effectively communicate in mixed-motivation scenarios. The invention has applications in the field of automatic control, including human-computer interaction systems, autonomous driving systems, navigation systems, and recommendation systems, where each agent corresponds to a target object in the invention's application field.
[0102] The above embodiment merely represents one embodiment of the present invention, and its description is relatively specific and detailed, but it should not be construed as restricting the scope of the present invention. It should be noted that a person of ordinary skill in the art may make a number of variations and improvements without departing from the concept of the present invention, and these variations and improvements fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the appended claims.
Claims
1. A multi-agent reinforcement learning communication method, characterized in that: include: Step 1: Model an intelligent agent model based on the environment, where the intelligent agent includes a signal sender and a signal receiver, and set the hyperparameters of the intelligent agent model; Step 2: Obtain trajectory data generated by the agent's interaction in the environment; Step 3: Based on the trajectory data, the action strategy network, evaluation network, and target evaluation network corresponding to each agent are updated, and the signal network of the signal sender is updated based on the signal gradient and the extended compliance constraint; The extended obedience constraint is determined based on the obedience constraint in Bayesian persuasion, specifically by each environmental state s Under the above conditions, the probability of a signal sender sending a signal, the probability and expectation of a signal receiver receiving a signal sent by the signal sender and taking an action, and the probability and expectation of a signal receiver receiving a signal sent by another signal sender and taking an action are determined; The signal network of the signal sender is updated according to the signal gradient and the extended compliance constraint, specifically including: When the signal strategy of the signal sender satisfies the extended obedience constraint, the update gradient of the signal network is the signal gradient; When the signal strategy of the signal sender does not satisfy the extended compliance constraint, the update gradient of the signal network is determined according to the signal gradient and the gradient of the parameters of the signal network according to the extended compliance constraint; After the gradient calculation is completed, the parameters of the signal network of the signal sender are updated using the ascent method; Step 4: Adjust the hyperparameters of the agent and repeat steps 2-3 to select the agent model with the best performance among multiple sets of hyperparameter groups.
2. The multi-agent reinforcement learning communication method according to claim 1, characterized in that: The signal set of the signal sender used in the extended obedience constraint is determined based on the signal that maximizes the posterior benefit expectation of the signal receiver.
3. The multi-agent reinforcement learning communication method according to claim 1, characterized in that: The calculation of the gradient of the signal network is determined based on the influence of the signal sent by the signal sender on the signal receiver in the agent model; The influence of the signal receiver includes the action strategy of the signal receiver.
4. The multi-agent reinforcement learning communication method according to claim 1, characterized in that: Based on the trajectory data, the action strategy network, evaluation network, and target evaluation network corresponding to each agent are updated, specifically including: Updating the action strategy network, evaluation network, and target evaluation network corresponding to the signal receiver; Updating the action strategy network, evaluation network, target evaluation network and signal network corresponding to the signal sender; The evaluation network and the target evaluation network maintained by the signal sender for the signal receiver are updated.
5. The multi-agent reinforcement learning communication method according to claim 4, characterized in that: Updating the action strategy network, evaluation network, and target evaluation network corresponding to the signal receiver, specifically including: updating the action policy network of the signal receiver using a policy gradient method according to the trajectory data of the signal receiver; Calculating a temporal difference error using the trajectory data and the target evaluation network, and updating the evaluation network; The target evaluation network is updated in a soft update manner.
6. The multi-agent reinforcement learning communication method according to claim 1, characterized in that: The trajectory data is collected and stored in an experience replay pool; The experience replay pool includes the experience replay pool of the signal receiver and the experience replay pool of the signal sender.
7. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.