Multi-agent communication strategy learning method based on human feedback

Abstract

The invention provides a multi-agent communication strategy learning method based on human feedback. The method comprises the steps that S1, a multi-agent reinforcement learning model is constructed, modeling is conducted on a multi-agent system, a dynamic communication topological structure in the multi-agent system is constructed, and the overall calculation process of the model is determined; s2, an offline pre-training process of the multi-agent system is carried out, training is carried out only through historical trajectory data in the process, interaction is not participated, and a suboptimal behavior strategy is rapidly learned; s3, collecting human feedback reward data and historical training data of the multi-agent system based on the specific task scene model; and S4, dynamically adjusting a communication strategy, carrying out auxiliary online training according to reward data fed back by human beings, helping the multi-agent system to better carry out communication decision making, and improving learning efficiency. Human feedback data is applied to multi-agent system training, and the limitation of a traditional multi-agent reinforcement learning method is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a multi-agent communication strategy learning method based on human feedback. Background Art

[0002] In recent years, the rapid advancement of artificial intelligence technology has provided machines with unprecedented autonomy. These AI-enhanced intelligent machines are not only able to perform complex tasks, but also independently formulate strategies and complete tasks in multiple fields such as traffic control, surgical robotics, drone formations, and collaborative robots. These achievements do not rely on pre-set programs by humans, but on the decision-making ability of the intelligent system itself. As more and more intelligent machines are integrated into human society, people's lives are greatly enriched and facilitated, such as semi-autonomous driving technology and remote-controlled minimally invasive surgery.

[0003] Traditional multi-agent communication systems usually use fixed communication strategies or rules to determine the communication mode and frequency between agents. These rules may be based on static parameter settings, such as fixed communication frequencies or routing tables, or based on simple heuristic algorithms to adjust communication parameters. However, these methods often lack adaptability to dynamic changes in the system and user needs. A more common method is to use reinforcement learning to conduct trial and error learning. Reinforcement learning is a machine learning method that learns which actions to take in a specific environment to maximize expected benefits through trial and error and feedback. In the field of multi-agent communication, reinforcement learning can be used to optimize communication strategies so that agents can adaptively select the best communication methods and parameter settings in complex and dynamic environments.

[0004] Multi-agent systems usually need to process a large amount of state information and communication data. As the number of agents increases and the complexity of the tasks increases, the dimension and size of the data will also increase dramatically. This requires the agents to spend a lot of time processing and analyzing data during the learning process, resulting in a long learning time.

[0005] In communication learning, the agent needs to strike a balance between exploration and exploitation. If the agent focuses too much on utilizing existing experience and knowledge and lacks exploration of unknown areas, it is easy to fall into local optimality.

[0006] In traditional multi-agent communication methods, communication between agents is often based on fixed rules or simple algorithms, while ignoring human feedback information. Summary of the invention

[0007] The purpose of the present invention is to provide a multi-agent communication strategy learning method based on human feedback, so as to solve the technical problems of multi-agent system in the communication learning process such as long learning time, easy falling into local optimum and lack of human feedback information.

[0008] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows:

[0009] A multi-agent communication strategy learning method based on human feedback includes the following steps:

[0010] Step S1: construct a multi-agent reinforcement learning model, model the multi-agent system, build a dynamic communication topology in the multi-agent system, and determine the overall calculation process of the model;

[0011] Step S2: Perform an offline pre-training process for the multi-agent system. This process is trained only through historical trajectory data, does not participate in interaction, and quickly learns a suboptimal behavior strategy;

[0012] Step S3: Collect human feedback reward data and historical training data of the multi-agent system based on a specific task scenario model;

[0013] Step S4: Dynamically adjust the communication strategy and assist online training based on human feedback reward data to help the multi-agent system make better communication decisions and improve learning efficiency.

[0014] Furthermore, the step S1 comprises the following steps:

[0015] Step S11: Using the partially observable Dec-POMDP model to build a multi-agent reinforcement learning model, the topological structure of the multi-agent system communication is modeled as a dynamic communication graph with weights;

[0016] Step S12: Calculate the importance weight vector of the communication group for the current agent and the hidden state of the current time step based on the global attention implicit graph;

[0017] Step S13: Screen valuable agents to join the communication group through the gated GRU of the communication screening network;

[0018] Step S14: The dominant message encoder completes message encoding by combining the learned importance weight vector with the dominant message.

[0019] Furthermore, the multi-agent reinforcement learning model in step S11 includes an agent, an MLP layer, a global attention implicit graph generation module, a communication screening network, an advantage message encoder and a global critic network.

[0020] Further, step S12 includes the following steps:

[0021] Step S121: vectorizing the feature representation of the agent by using GNN in an embedding manner;

[0022] Step S122: Calculate the importance weight vector;

[0023] Potential communication relationship between agent i and agent j

[0024]

[0025] Where LeakyReLU(·) represents the activation function, a is a single-layer feedforward neural network, T represents transposition, W is a learnable parameter matrix, || represents a vector concatenation operation, represents the embedding representation of agent i, represents the embedding representation of agent j;

[0026] The importance of agent j to agent i

[0027]

[0028] In potential communication collection The importance weight vector of all agents in agent i Calculated by the following formula:

[0029]

[0030] Step S123: Calculate the hidden state of the current time step

[0031] The weight of the additional information that agent j can bring to agent i

[0032]

[0033] Among them, W ′ is a trainable parameter matrix, b is the bias term, and tanh(·) represents the activation function;

[0034] The hidden state at the current time step is calculated as follows:

[0035]

[0036] Further, step S13 includes the following steps:

[0037] Reset Gate and update gate The calculation formula is as follows:

[0038]

[0039] Among them, σ() represents the activation function, W r , U r , W z, U z are the network parameters involved in training, for action;

[0040] By resetting the gate The key features of the weight vector learned by the graph attention mechanism can be screened out and integrated into the historical state; if the weight in the reset gate is close to 0, it means resetting the corresponding hidden state element to 0, that is, discarding the hidden state of the previous time step; if the element value is close to 1, then the hidden state of the previous time step is retained.

[0041] Further, step S14 includes the following steps:

[0042] Step S141: Calculate the advantage function value of agent j at time step t The advantage function is actually the difference between the Q function (action value) and the V function (state value);

[0043]

[0044] Step S142: Calculate the message code generated by agent i at time step t, and the calculation method is shown in the following formula:

[0045]

[0046] Among them, α ji Represents the weight coefficient, which indicates the importance of agent j’s contribution to agent i.

[0047] Further, step S2 includes the following steps:

[0048] Step S21: Initialize the conditional variational autoencoder and Initialize Q network Q i , the perturbation function ξ i , target Q network Target perturbation network

[0049] Step S22: Using an offline database To fit the conditional variational autoencoder and

[0050] Step S23: From the offline database Sampling mini-batch;

[0051] Step S24: Update target network and

[0052] Further, step S4 includes the following steps:

[0053] Step S41: designing an online feedback reward model training based on human preference to fine-tune the model, and introducing human feedback signals to guide the action decision-making of the multi-agent system during the interaction between the multi-agent system and the environment;

[0054] Step S42: Design a human preference feedback resampling method. Using this method, the agent resamples from the buffer of human feedback historical experience each time it updates its strategy, so as to learn human feedback rewards more efficiently.

[0055] Compared with the prior art, the present invention has the following beneficial technical effects:

[0056] 1) The present invention applies human feedback data to the training of multi-agent systems, which solves the difficulties caused by the characteristics of the multi-agent task environment, such as a large number of agents, complex strategies, and dynamic changes in the environment, and effectively improves the limitations of traditional multi-agent reinforcement learning methods.

[0057] 2) This invention proposes a multi-agent communication strategy learning method that is closer to human values. By integrating human cognitive ability and agent computing ability, the advantages of the decision-making process are complementary, thereby obtaining more efficient decision results. Based on artificial intelligence, this invention has conducted in-depth research in multi-agent reinforcement learning and has great practical value.

[0058] 3) Based on multi-agent reinforcement learning, this invention integrates human feedback rewards into traditional multi-agent system task training, greatly reducing the training time cost and greatly improving the ability of the multi-agent system to complete tasks. Effective experiments have proved that this method has good performance effects, a wide range of applications, and low running time. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0060] Figure 1 A schematic diagram of the overall process of a multi-agent communication strategy learning method based on human feedback provided by an embodiment of the present invention;

[0061] Figure 2 A schematic diagram of a multi-agent reinforcement learning model provided by an embodiment of the present invention;

[0062] Figure 3 A schematic diagram of a weighted dynamic communication topology provided in an embodiment of the present invention;

[0063] Figure 4 A schematic diagram of a communication screening network structure provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] Humans play an important role in multi-agent systems and have intuitive feelings and evaluations of system performance, response speed, energy consumption, etc. Therefore, it is of great significance to use human feedback as a guiding signal to optimize multi-agent communication strategies. Human feedback can be qualitative (such as user satisfaction surveys) or quantitative (such as communication delay, packet loss rate and other indicators).

[0066] The present invention proposes a multi-agent communication strategy learning method based on human feedback, which includes four main steps: building a multi-agent reinforcement learning model, offline pre-training of the multi-agent system, collecting human feedback reward data, and dynamically adjusting the communication strategy. Figure 1 As shown, each step is described in detail as follows:

[0067] Step S1: Construct a multi-agent reinforcement learning model, model the multi-agent system, build a dynamic communication topology in the multi-agent system, and determine the overall calculation process of the model.

[0068] Specifically, step S1 includes the following steps:

[0069] Step S11: A multi-agent reinforcement learning model is constructed using a partially observable Dec-POMDP (Decentralized Partially Observable Markov Decision Process) model, and the topological structure of the multi-agent system communication is modeled as a weighted dynamic communication graph.

[0070] like Figure 2 As shown in the figure, the multi-agent reinforcement learning model includes an agent, an MLP layer (Multilayer Perceptron), a global attention implicit image generation module, a communication screening network, an advantage message encoder and a global critic network.

[0071] At a time step t, each agent i obtains a local observation and the action taken at the previous time step The local observation and actions Pass it into the MLP layer to get a hidden state Based on Identify which agents are in view; if agent i can see agent j, add these agents to the potential communication set The global attention implicit graph generation module first integrates the historical information of the agent And the hidden state obtained by MLP layer encoding As input, get the hidden state of the current time step Under the partially observable setting, the agent uses the global attention implicit graph generation module to obtain the potential communication set The importance weight vector Ω for the current agent i .

[0072] The communication filtering network is based on the hidden state of the importance weight vector features transferred from the global attention implicit image generation module. Filter out agents with larger importance weights to obtain potential communication sets Subset of That is, the communication group that finally communicates with agent i is obtained. The advantage message encoder converts the importance weight vector Ω i As input, we get the advantage message encoding of agent i This vector reflects the weighted contribution of other agents in the communication group to agent i, and finally uses the obtained information to bias the Q value of other agents as an incentive. In order to calculate the Q value function of agent i, the global critic network will pass the hidden state vector obtained by the communication screening network And the weighted contribution features from other agents are used as input to obtain the Q-value function.

[0073] The present invention uses a weighted dynamic communication graph as the topological structure of agent communication. Considering that the importance of agent i to agent j is not equivalent to the importance of agent j to agent i, a unidirectional weighted edge is used to define the importance between agents. Therefore, the constructed graph is a directed graph. Figure 3 As shown in the figure, the arrows indicate that there is communication between the agents, and the numbers on each edge represent the weight coefficient α ij , which indicates the importance of agent i’s contribution to agent j. The larger the value, the greater the contribution value.

[0074] Step S12: Calculate the importance weight vector of the communication group for the current agent and the hidden state of the current time step based on the global attention implicit graph.

[0075] Step S121: Use GNN (Graph Neural Network) to vectorize the feature representation of the agent by embedding.

[0076] For a potential communication set with n agents For example, the embedding representation vector of the agent at time step t is expressed as follows:

[0077]

[0078] Among them, the embedding of each agent is represented as a K-dimensional vector In the global attention network, after the features of the agent in the input graph are processed internally by the neural network, the output of the neural network is still a K-dimensional vector:

[0079]

[0080] The embedding representations of agent i and agent j at time step t are and

[0081] Step S122: Calculate the importance weight vector.

[0082] Using the graph attention mechanism, the embedding representation of agent i is and the embedding representation of agent j After linear transformation, splicing is performed; the spliced ​​result is passed through a feedforward neural network with LeakyReLU as the nonlinear activation function to obtain the potential communication relationship between agent i and agent j

[0083]

[0084] Where LeakyReLU(·) represents the activation function, a is a single-layer feedforward neural network, T represents transpose, W is a learnable parameter matrix, and || represents a vector concatenation operation.

[0085] For potential communication collection For the remaining n-1 agents in the set, the potential communication relationship with agent i is calculated respectively, and then the influence weights of all agents in the set are normalized using the softmax activation function to obtain the importance of agent j to agent i.

[0086]

[0087] In potential communication collection The importance weight vector of all agents in agent i Calculated by the following formula:

[0088]

[0089] Step S123: Calculate the hidden state of the current time step.

[0090] After obtaining the importance weight vector, a feedforward neural network with a hyperbolic activation function tanh is defined to calculate the weight of the additional information that agent j can bring to agent i This weight indicates how much information agent j is worth giving to agent i. The calculation method is as follows:

[0091]

[0092] Among them, W ′ is a trainable parameter matrix, b is the bias term, and tanh(·) represents the activation function.

[0093] Will get Multiply it by the embedding representation of agent j, then perform weighted summation with the importance weight vector, and finally perform concatenation calculation to obtain the aggregate embedding representation of agent i at time step t That is, the hidden state of the current time step: The calculation method is as follows:

[0094]

[0095] Step S13: Use the gated GRU (Gated Recurrent Unit) of the communication screening network to screen valuable agents to join the communication group.

[0096] The communication filtering network embeds the aggregated embedding obtained by the global attention implicit image generation module and the importance weight vector As input, the aggregate embedding representation combines the latent communication set For the importance weight feature of the current agent communication, the screening network removes agents with no communication value from the potential communication set based on this feature. The communication set obtained by screening is the potential communication set A subset of express.

[0097] Communication filtering network structure such as Figure 4 As shown, the communication filtering network first embeds The action is obtained through action strategy network reasoning and the importance weight vector The reset gate at the current moment is calculated based on the historical state characteristics, the action output by the agent, and the learned weights. and update gate The calculation formula is as follows:

[0098]

[0099] Among them, σ() represents the activation function, W r , U r , W z , U z are the network parameters involved in training.

[0100] The reset gate and update gate are used to achieve the effect of selective memory, and more valuable historical information is obtained. The GRU gating unit can gradually forget the worthless state features, which is manifested in the screening of the state features of the agent. From a macro perspective, the agents that are worthless to the current agent will not appear in its communication group, which is further manifested in that there is no edge connecting the two agents in the communication topology graph.

[0101] Resetting the gate helps capture short-term dependencies in time series. The key features of the weight vector learned by the graph attention mechanism can be screened out and integrated into the historical state. If the weight in the reset gate is close to 0, it means resetting the corresponding hidden state element to 0, that is, discarding the hidden state of the previous time step. If the element value is close to 1, the hidden state of the previous time step is retained. The larger the weight value, the more critical the feature is, and it needs to be written into the hidden vector of the historical state; otherwise, it will be gradually forgotten. The final feature vector of the current time step saves the information of the current unit and passes it to the network. The calculation method is as follows:

[0102]

[0103] Among them, tanh(·) represents the activation function, W h , U h is the network parameter involved in the training, and ⊙ is the Hadamard Product calculation.

[0104] After integrating the key weight information features learned in the current time step, the gate is updated Select the valid historical information to be retained, and use the updated historical state features as part of the input at the next moment. The larger the weight of the update gate, the more important the weight information at the current moment. The calculation process is as follows:

[0105]

[0106] Step S14: The advantage message encoder completes message encoding by combining the learned importance weight vector with the advantage message, and then implements communication through the encoded message to help the intelligent agent learn the optimal behavior strategy.

[0107] Step S141: Calculate the advantage function value of agent j at time step t The advantage function is actually the difference between the Q function (action value) and the V function (state value). If the value calculated by the advantage function is a positive number, it means that taking action a in state s is better than the current strategy. On the contrary, if it is a negative number, it means that it is worse than the current strategy.

[0108]

[0109] Step S142: Calculate the message code generated by agent i at time step t, and the calculation method is shown in the following formula:

[0110]

[0111] Among them, α ji Represents the weight coefficient, which indicates the importance of agent j’s contribution to agent i.

[0112] Through the above calculations, agent i can extract features from the dominant messages received by each agent in the communication group according to the dominant message coding, obtain the dominant message coding value, and effectively use the useful information to make decisions.

[0113] Step S2: Perform offline pre-training of the multi-agent system. This process is trained only through historical trajectory data, does not participate in interaction, and quickly learns suboptimal behavior strategies.

[0114] Specifically, step S2 includes the following steps:

[0115] Step S21: Initialize the conditional variational autoencoder and Initialize Q network Q i , the perturbation function ξ i , target Q network Target perturbation network

[0116] Among them, the autoencoder Indicates the encoder Mapping state s and action a to the mean μ in latent space 1 and standard deviation σ 1 , thus obtaining a distribution of latent variables, and then sampling from this distribution to obtain the latent variable z 1 , decoder Then the latent variable z 1and state s as input to generate an action a that meets the conditions.

[0117] Autoencoder Indicates the encoder With current state s, action a and next state s ′ As input, it is mapped to the mean μ in the latent space 2 and standard deviation σ 2 , thus obtaining a distribution of latent variables, and then sampling from this distribution to obtain the latent variable z 2 , decoder Then the current state s, the next state s and the potential variable z 2 As input, generate an action a that meets the conditions.

[0118] Step S22: Using an offline database To fit the conditional variational autoencoder and

[0119] Step S23: From the offline database Sampling mini-batch;

[0120] Step S24: Update target network and

[0121] Step S3: Collect human feedback reward data and historical training data of the multi-agent system based on a specific task scenario model, or historical trajectories generated by a model trained by an expert;

[0122] Step S4: Dynamically adjust the communication strategy and assist online training based on human feedback reward data to help the multi-agent system make better communication decisions and improve learning efficiency.

[0123] Specifically, step S4 includes the following steps:

[0124] Step S41: Design an online feedback reward model training based on human preferences to fine-tune the model. In the process of interaction between the multi-agent system and the environment, introduce human feedback signals to guide the action decisions of the multi-agent system.

[0125] Due to the dynamics and uncertainty of human feedback rewards, it is impossible to design an explicit reward function. Therefore, a supervised learning method is used to update the function approximator to continuously fit human preferences. It is assumed that humans provide this feedback based on the implicit function H(·,·):S×A→R, where S represents the state set and A represents the action set. The H function takes the state and action as input and outputs a real value R. The real value can be understood as a feedback evaluation of humans taking a specific action under a specific state. This implicit function is implicitly understood by humans and does not actually exist. It can only be fitted through learning methods. Consider the learning problem here to calculate the evaluation of this function, using It is assumed that the agent determines the action execution according to a fixed action selection policy, which selects the next action as the action that maximizes the expected human feedback, that is:

[0126] Considering that the agent takes time to perform actions and observe human feedback, the agent uses its own experience in = (s, a, Δ t_state ) as the input of H, and in the form of human feedback out=(h,Δ t_feedback ) as its output result, where Δ t_state is the agent from state s t-1 Take action t To state s t The time interval, Δ t_feedback is the time interval at which human feedback is observed, and h is the feedback scalar value. Based on these observations, the agent's task is to calculate the evaluation H value. It is worth noting that due to the delay in human feedback, each observed out corresponds to several recently observed action trajectories, and some action trajectories may not have corresponding feedback information at all. The above settings are encoded in the learning process, and the following loss function is used to judge quality.

[0127]

[0128] Where ω(·) is a scalar-valued weighted function. It is worth noting that when no human feedback reward is observed, the function value is zero; Δ t_feedback When is particularly large, the function value is close to zero, indicating that the human feedback reward has a negligible guiding effect on the current state-action pair. Assume that humans intend to apply their feedback only to the most recent agent behavior. Specifically, using the human feedback delay probability distribution P delay Sampling to get the weight ω(Δ t_state ,Δ t_feedback ). Finally, by finding a To minimize the expected value of loss to achieve the learning goal The implementation is as follows:

[0129]

[0130] Among them, E in,out Indicates the desired operation on input and output.

[0131] To learn The present invention adopts stochastic gradient descent (SGD) to update The update method is shown in the following formula:

[0132]

[0133] Among them, t is the number of iteration steps, η t-1 is the learning rate, is the gradient corresponding to the loss l.

[0134] Step S42: Design a human preference feedback resampling method. Using this method, the agent resamples from the buffer of human feedback historical experience each time it updates its strategy, so as to learn human feedback rewards more efficiently, effectively alleviating the sparsity of human preference rewards and the problem of low learning efficiency.

[0135] To learn faster with limited real-time human interactions the agent receives SGD updates are performed at a fixed rate that often exceeds the rate at which humans are typically able to provide feedback to the agent. Stored in the feedback playback buffer D hf , and repeatedly samples from this buffer with replacement to perform learning. t represents the state of the agent at time step t, a t represents the action taken by the agent at time step t, r t represents the immediate reward obtained by the agent from the environment at time step t, represents the human feedback reward obtained by the agent at time step t, D hf Contains the historical experience of all human feedback information. Of course, the weight ω(Δ t_state ,Δ t_feedback )=0 is discarded, and only meaningful feedback historical experience is stored.

[0136] By D hf , and continuously perform SGD updates at a fixed rate. However, in order to ensure that each human feedback has an immediate impact on the behavior of the agent, not only is it sampled from D at a fixed rate, but also from hf, and once the human provides new feedback and the weights ω(Δ t_state ,Δ t_feedback )≠0, the feedback information is used as sampled data to perform SGD updates. Of course, the fixed rate training_rate here is a hyperparameter of the model. In actual training, it is observed that human feedback usually provides a feedback reward signal every 40 time steps, that is, every 2000 time steps, humans can provide 50 feedback reward signals.

[0137] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A multi-agent communication strategy learning method based on human feedback, characterized in that: The method comprises the following steps: Step S1: construct a multi-agent reinforcement learning model, model the multi-agent system, build a dynamic communication topology in the multi-agent system, and determine the overall calculation process of the model; Step S2: Perform an offline pre-training process for the multi-agent system. This process is trained only through historical trajectory data, does not participate in interaction, and quickly learns a suboptimal behavior strategy; Step S3: Collect human feedback reward data and historical training data of the multi-agent system based on a specific task scenario model; Step S4: Dynamically adjust the communication strategy and perform auxiliary online training based on human feedback reward data to help the multi-agent system make better communication decisions and improve learning efficiency.

2. The multi-agent communication strategy learning method based on human feedback according to claim 1 is characterized in that: The step S1 comprises: Step S11: Using the partially observable Dec-POMDP model to build a multi-agent reinforcement learning model, the topological structure of the multi-agent system communication is modeled as a dynamic communication graph with weights; Step S12: Calculate the importance weight vector of the communication group for the current agent and the hidden state of the current time step based on the global attention implicit graph; Step S13: Screen valuable agents to join the communication group through the gated GRU of the communication screening network; Step S14: Use the dominant message encoder to complete message encoding by combining the learned importance weight vector with the dominant message.

3. The multi-agent communication strategy learning method based on human feedback according to claim 2 is characterized in that: The multi-agent reinforcement learning model in step S11 includes an agent, an MLP layer, a global attention implicit image generation module, a communication screening network, an advantage message encoder and a global critic network.

4. The multi-agent communication strategy learning method based on human feedback according to claim 3 is characterized in that: Step S12 further comprises: Step S121: vectorizing the feature representation of the agent by using GNN in an embedding manner; Step S122: Calculate the importance weight vector; Potential communication relationship between agent i and agent j for: Where LeakyReLU(·) represents the activation function, a is a single-layer feedforward neural network, T represents transposition, W is a learnable parameter matrix, || represents a vector concatenation operation, represents the embedding representation of agent i, represents the embedding representation of agent j; The importance of agent j to agent i for: In potential communication collection The importance weight vector of all agents in agent i Calculated by the following formula: Step S123: Calculate the hidden state of the current time step The weight of the additional information that agent j can bring to agent i Among them, W ′ is a trainable parameter matrix, b is the bias term, and tanh(·) represents the activation function; The hidden state at the current time step is calculated as follows:

5. The multi-agent communication strategy learning method based on human feedback according to claim 4 is characterized in that: Step S13 further comprises: Reset Gate and update gate The calculation formula is as follows: Among them, σ() represents the activation function, W r , U r , W z , U z are the network parameters involved in training, for action; By resetting the gate The key features of the weight vector learned by the graph attention mechanism can be screened out and integrated into the historical state; if the weight in the reset gate is close to 0, it means resetting the corresponding hidden state element to 0, that is, discarding the hidden state of the previous time step; if the element value is close to 1, then the hidden state of the previous time step is retained.

6. The multi-agent communication strategy learning method based on human feedback according to claim 5, characterized in that: Step S14 further comprises: Step S141: Calculate the advantage function value of agent j at time step t The advantage function is actually the difference between the Q function (action value) and the V function (state value); Step S142: Calculate the message code generated by agent i at time step t, and the calculation method is shown in the following formula: Among them, α ji Represents the weight coefficient, which indicates the importance of agent j’s contribution to agent i.

7. The multi-agent communication strategy learning method based on human feedback according to claim 6, characterized in that: Step S2 further comprises: Step S21: Initialize the conditional variational autoencoder and Initialize Q network Q i , the perturbation function ξ i , target Q network Target perturbation network Step S22: Using an offline database Fitting a Conditional Variational Autoencoder and Step S23: From the offline database Sampling mini-batch; Step S24: Update target network and 8. The multi-agent communication strategy learning method based on human feedback according to claim 7, characterized in that: Step S4 further comprises: Step S41: designing an online feedback reward model training based on human preferences, fine-tuning the model, and introducing human feedback signals to guide the action decisions of the multi-agent system during the interaction between the multi-agent system and the environment; Step S42: Design a human preference feedback resampling method. Using this method, the agent resamples from the buffer of human feedback historical experience each time it updates its strategy, so as to learn human feedback rewards more efficiently.

Images