Multi-agent learning method, device and equipment based on dynamic weighted mean field

Abstract

The application discloses a kind of multi-agent learning method, device and equipment based on dynamic weighted average field, method includes: according to the action of dynamic weighted average action corresponding to agent selects action;Current position information of agent is obtained Calculation aggregation factor, the weight of agent is dynamically updated using improved PSO algorithm in combination with aggregation factor;According to the action update dynamic weighted average action with weight, execute joint action to obtain next state, according to the reward update action and state, and according to the weight and reward update individual weight optimal value and group weight optimal value;Experience replay buffer is sampled from experience data including current state, joint action, reward, next state and dynamic weighted average action and updates the Q function of agent, according to the updated Q function and specified learning rate updates the target network of agent.The application can guide multi-agent to tend to group performance optimization, and also ensure that individual agent has sufficient exploration ability.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and specifically to a multi-agent learning method, apparatus, and device based on a dynamic weighted average field. Background Technology

[0002] Multi-agent reinforcement learning (MARL) is currently widely studied and has been applied in various practical scenarios, such as unmanned swarm control and medical image analysis. In a multi-agent environment, agents need to interact with the environment and other agents; the decision-making behavior of a single agent can affect other agents, leading to instability in the learning process. As the number of agents increases, the state space and action space grow exponentially, resulting in a significant increase in computational complexity and ultimately causing the algorithm's performance to fall short of expectations.

[0003] Some researchers have proposed efficient algorithms to address the aforementioned problems. Based on mean-field theory, two mean-field multi-agent reinforcement learning algorithms (Mean Field Actor-Critic, MF-AC and Mean Field Independent Q-learning, MF-Q) are constructed. These algorithms approximate the learning among swarm agents as learning between two entities: one is a single agent, and the other is a virtual entity composed of the average effect of the entire (local) swarm. Although mean-field multi-agent reinforcement learning (MF MARL) can effectively alleviate the challenges of collaboration among massive numbers of agents, its "averaging" method does not fully consider the impact of the heterogeneity of individual agents on swarm decision-making.

[0004] Currently, researchers have proposed solutions to address the varying importance of agents and practical needs, aiming to better adapt to different scenarios and improve algorithm performance. Based on the idea that "not all agents influence the decisions of one agent," an adaptive mean field MARL based on an attention mechanism (AM) is constructed. Alternatively, a multi-agent reinforcement learning framework with a weighted mean field is implemented using reward attribution decomposition, employing multi-head attention to calculate weights to form a weighted mean field Q-function. Using the attention mechanism to implement weighted mean field reinforcement learning allows the drone to select more valuable neighborhood information. Weighted behavior is formed based on the weighted mean field method to highlight the importance of different agents. However, methods that determine weights through AM or neural networks (NN) lack explicit consideration of the relationship between individuals and the group, have shortcomings in the interpretability of weights, and suffer from high computational complexity due to extensive parameterization. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a multi-agent learning method, device and equipment based on dynamic weighted average field, which improves the average field multi-agent reinforcement learning based on a simple and efficient optimization algorithm. It can not only highlight the differences between agents through the dynamic weights constructed by the optimization algorithm, but also effectively ensure the implementation efficiency of the algorithm.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A multi-agent learning method based on a dynamically weighted average field includes the following steps:

[0008] S101: Initialize the parameters of each agent, including the dynamic weighted average action, weight, optimal individual weight value and optimal group weight value for each agent;

[0009] S102: Obtain the initial state of each agent;

[0010] S103: Using a specified strategy, select the action of each agent based on the dynamic weighted average action corresponding to each agent, thereby obtaining the joint action of all agents;

[0011] S104: Obtain the position information of each agent in the current state, calculate the aggregation factor of each agent based on the position information, update the weight of each agent and perform exponential normalization based on the aggregation factor, the optimal value of individual weight and the optimal value of group weight;

[0012] S105: Update the dynamic weighted average action corresponding to each agent according to the updated weight and action of each agent, execute the joint action of all agents to obtain the next state of each agent, update the reward of each agent according to the action and state of each agent, store the current state, joint action, reward, next state and dynamic weighted average action as experience data in the experience replay buffer, and update the individual weight optimal value and the group weight optimal value according to the updated weight and reward of each agent.

[0013] S106: Sample experience data from the experience replay buffer and update the Q function of each agent. Update the target network of each agent according to the updated Q function and the specified learning rate. Return to step S103 until the learning ends.

[0014] Furthermore, when using a specified strategy to select the action of each agent based on the dynamic weighted average action corresponding to each agent, specifically, in the current state of each agent, an action is selected for each agent through the Boltzmann policy and the dynamic weighted average action corresponding to each agent.

[0015] Furthermore, when calculating the aggregation factor for each agent based on location information, specifically, the negative value of the distance between each agent and the center of its corresponding neighboring agents is calculated based on the location information of each agent. After mapping the negative value of the distance to a specified interval, the aggregation factor of the corresponding agent is calculated using the mapping result. The mathematical expression is as follows:

[0016]

[0017]

[0018]

[0019]

[0020] In the formula, As an aggregation factor, It is time Time-based intelligent agents The position vector, It is time Time-based intelligent agents neighborhood Middle intelligence agent The position vector, It is time Time-based intelligent agents The weights; At any moment At that time, intelligent agent The negative value of the distance to the center of its neighboring agent is calculated based on the weights; Indicates will The mapping result without changing the relative size of the original data. ; express The final mapping result, ; and These are different coefficients.

[0021] Furthermore, when updating the weights of each agent based on its aggregation factor, optimal individual weight, and optimal group average weight, the calculation formula is as follows:

[0022]

[0023]

[0024] In the formula, For intelligent agents At any moment The weight adjustment rate; Inertia weight; For intelligent agents At any moment The weights; and It is a random factor; and Is A random number that takes values ​​within a range; Aggregation factor; It is an intelligent agent The optimal value of individual weights, It is the optimal group weight value for all agents; random factor and The calculation formula is as follows:

[0025]

[0026]

[0027] in, and It's experience points. It is the maximum time step.

[0028] Furthermore, when updating the dynamically weighted average action corresponding to each agent based on the updated weights and actions of each agent, the mathematical expression is as follows:

[0029]

[0030] in, Represents intelligent agents The corresponding dynamic weighted average action, Indicating in intelligent agents neighborhood Middle intelligence agent The weight, It is an intelligent agent The action.

[0031] Furthermore, when updating the optimal individual weight and the optimal group weight based on the updated weight and reward of each agent, the reward of each agent is compared with the reward of the optimal individual weight. If the reward is higher than the reward of the optimal individual weight, the optimal individual weight of the corresponding agent is updated to the updated weight of the corresponding agent. At the same time, the average reward of all agents is calculated and compared with the average reward of the optimal group weight. If the average reward is higher than the average reward of the optimal group weight, the optimal group weight is updated based on the updated weight of all agents.

[0032] Furthermore, when updating the Q-function of each agent, the mathematical expression is as follows:

[0033]

[0034] In the formula, Indicates the agent's current state At any moment The Q function, Indicates the agent's current state At any moment The Q function, For learning rate, It is a reward discount factor. It is an intelligent agent The reward It is an intelligent agent At any moment The weighted average field value function is expressed mathematically as follows:

[0035]

[0036] In the formula, It is an intelligent agent At any moment strategy, Indicates the agent's state in the next state At any moment The Q function, This represents the expectation operation. Indicates the current state. Indicating agents in joint actions The action, Represents intelligent agents The corresponding dynamic weighted average action, Indicates the next state. Indicating agents in joint actions All actions of intelligent agents other than Represents intelligent agents All other intelligent agents at time A joint strategy.

[0037] Furthermore, when updating the target network for each agent based on the updated Q-function and the specified learning rate, the following steps are included:

[0038] If multi-agent learning uses the Actor-Critic algorithm, then according to the agent... At any moment The target value is calculated using a weighted average field value function. Then, based on the target value and the updated Q-function, the loss is minimized to update the critic network. The policy gradient is then calculated based on the updated Q-function to update the actor network. Finally, each agent is updated using the corresponding learning rate parameter. The target network;

[0039] If multi-agent learning uses the Independent Q-learning algorithm, then according to the agents... At any moment The target value is calculated using a weighted average field value function. Then, based on the target value and the updated Q function, the loss is minimized to update the Q network. Finally, each agent is updated using the corresponding learning rate parameter. The target network.

[0040] This invention also proposes a multi-agent learning device based on a dynamically weighted average field, comprising:

[0041] An initialization module is used to initialize the parameters of each agent, including the dynamic weighted average action, weight, optimal individual weight value, and optimal group weight value for each agent.

[0042] The data acquisition module is used to acquire the initial state of each agent;

[0043] The data processing module is used to select the action of each agent based on the dynamic weighted average action corresponding to each agent using a specified strategy, thereby obtaining the joint action of all agents; it is also used to obtain the position information of each agent in the current state, calculate the aggregation factor of each agent based on the position information, update the weight of each agent and perform exponential normalization based on the aggregation factor, the optimal individual weight value and the optimal group weight value; it is also used to update the dynamic weighted average action corresponding to each agent based on the updated weight and action of each agent, execute the joint action of all agents to obtain the next state of each agent, update the reward of each agent based on the action and state of each agent, store the current state, joint action, reward, next state and dynamic weighted average action as experience data in the experience replay buffer, and update the optimal individual weight value and the optimal group weight value based on the updated weight and reward of each agent;

[0044] The learning and training module is used to sample experience data from the experience replay buffer and update the Q function of each agent, and update the target network of each agent based on the updated Q function and the specified learning rate.

[0045] The present invention also proposes an electronic device, including a processor and a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and the computer program is executed by the processor to implement the steps of the multi-agent learning method based on a dynamically weighted average field.

[0046] Compared with the prior art, the advantages of the present invention are as follows:

[0047] This invention comprehensively considers both individual and group levels, combining an improved PSO algorithm with mean-field multi-agent reinforcement learning. It leverages the characteristic of PSO to balance individual and group interests during optimization to adjust agent weights, resulting in weights that are dynamic, have mean fusion properties, and are interpretable. Weights are determined from both individual and group dimensions, focusing on both the individual decision-making ability of agents and the collaborative ability of the group: individual performance is reflected through a reward mechanism, and group interaction characteristics are characterized by aggregated states. While guiding the multi-agent group towards optimal group performance, it also ensures that individual agents have sufficient exploratory capabilities.

[0048] This invention aims to dynamically determine the parameters of a balanced population for exploration and utilization. Simultaneously, it constructs a clustering factor based on the current physical location of each agent and selectively adjusts the weights to better adapt to the current scenario and efficiently complete the task. This not only highlights the differences between agents through the dynamic weights constructed by the optimization algorithm but also effectively ensures the algorithm's implementation efficiency. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the method framework of an embodiment of the present invention.

[0050] Figure 2 This is a flowchart of a method according to an embodiment of the present invention.

[0051] Figure 3 This is a schematic diagram illustrating the analysis under different conditions for the proposed aggregation factor. Figure 3 (a) represents case 1. Figure 3 (b) represents case 2. Figure 3 (c) represents case 3. Figure 3 (d) indicates case 4.

[0052] Figure 4 A visualization comparison chart for training in a Battle scenario. Figure 4 (a) represents the training results of the MF-AC method. Figure 4 (b) represents the training results of the MF-Q method. Figure 4 (c) represents the training result of the DWMF-AC method in this embodiment of the invention. Figure 4 (d) represents the training result of the DWMF-Q method in this embodiment of the invention.

[0053] Figure 5 These are the experimental results of Battle when the obstacle ratio is 0. Figure 5 (a) is a comparison of the overall reward of the DWMF-AC method of this invention with existing methods. Figure 5 (b) is a comparison of the number of wins of the DWMF-AC method of this invention with existing methods. Figure 5(c) is a comparison of the overall reward between the DWMF-Q method of this invention and existing methods. Figure 5 (d) is a comparison of the number of defeats of the DWMF-Q method in this embodiment of the invention with that of existing methods.

[0054] Figure 6 These are the experimental results of Battle when the obstacle ratio is 0.01. Figure 6 (a) is a comparison of the overall reward of the DWMF-AC method of this invention with existing methods. Figure 6 (b) is a comparison of the number of wins of the DWMF-AC method of this invention with existing methods. Figure 6 (c) is a comparison of the overall reward between the DWMF-Q method of this invention and existing methods. Figure 6 (d) is a comparison of the number of defeats of the DWMF-Q method in this embodiment of the invention with that of existing methods.

[0055] Figure 7 These are the experimental results of Battle when the obstacle ratio is 0.02. Figure 7 (a) is a comparison of the overall reward of the DWMF-AC method of this invention with existing methods. Figure 7 (b) is a comparison of the number of wins of the DWMF-AC method of this invention with existing methods. Figure 7 (c) is a comparison of the overall reward between the DWMF-Q method of this invention and existing methods. Figure 7 (d) is a comparison of the number of defeats of the DWMF-Q method in this embodiment of the invention with that of existing methods.

[0056] Figure 8 This section describes the behavior visualization and win rate of the DWMF-AC and DWMF-Q methods in this embodiment of the invention when the obstacle ratio is 0. Figure 8 (a) Visualization of DWMF-AC behavior when the obstacle ratio is 0; Figure 8 (b) Visualization of DWMF-Q behavior when the obstacle ratio is 0; Figure 8 (c) is the win rate when the obstacle ratio is 0.

[0057] Figure 9 This section visualizes the behavior and win rate of the DWMF-AC and DWMF-Q methods in this embodiment of the invention when the obstacle ratio is 0.01. Figure 9 (a) Visualization of DWMF-AC behavior when the obstacle ratio is 0.01; Figure 9 (b) Visualization of DWMF-Q behavior when the obstacle ratio is 0.01; Figure 9 (c) is the win rate when the obstacle ratio is 0.01.

[0058] Figure 10 This section visualizes the behavior and win rate of the DWMF-AC and DWMF-Q methods in this embodiment of the invention when the obstacle ratio is 0.02. Figure 10 (a) Visualization of DWMF-AC behavior when the obstacle ratio is 0.02; Figure 10 (b) Visualization of DWMF-Q behavior when the obstacle ratio is 0.02; Figure 10 (c) is the win rate when the obstacle ratio is 0.02.

[0059] Figure 11 It is a weighted visual comparison chart. Figure 11 (a) shows the weight visualization of the MFACA method. Figure 11 (b) shows the weight visualization of the MFQA method. Figure 11 (c) shows the weight visualization of the DWMF-AC method in this embodiment of the invention. Figure 11 (d) represents the visualization of the weights of the DWMF-Q method in this embodiment of the invention.

[0060] Figure 12 These are the results of the Battle 144 vs 144 experiment. Figure 12 (a) is a comparison of the overall reward of the DWMF-AC method of this invention with existing methods. Figure 12 (b) is a comparison of the number of wins of the DWMF-AC method of this invention with existing methods. Figure 12 (c) is a comparison of the overall reward between the DWMF-Q method of this invention and existing methods. Figure 12 (d) is a comparison of the number of defeats of the DWMF-Q method in this embodiment of the invention with that of existing methods.

[0061] Figure 13 These are the results of the Battle 400 vs 400 experiment. Figure 13 (a) is a comparison of the overall reward of the DWMF-AC method of this invention with existing methods. Figure 13 (b) is a comparison of the number of wins of the DWMF-AC method of this invention with existing methods. Figure 13 (c) is a comparison of the overall reward between the DWMF-Q method of this invention and existing methods. Figure 13 (d) is a comparison of the number of defeats of the DWMF-Q method in this embodiment of the invention with that of existing methods.

[0062] Figure 14 These are the experimental results from Pursuit. Figure 14 (a) is a comparison of the overall reward of the DWMF-AC method of this invention with existing methods. Figure 14(b) is a comparison of the number of labels in the DWMF-AC method of this invention with existing methods. Figure 14 (c) is a comparison of the overall reward between the DWMF-Q method of this invention and existing methods. Figure 14 (d) is a comparison of the number of markers between the DWMF-Q method of this invention and existing methods.

[0063] Figure 15 This is a visualization of the behavior of the DWMF-AC and DWMF-Q methods in the Pursuit scenario according to embodiments of the present invention. Figure 15 (a) shows the behavior of the DWMF-AC method in an embodiment of the present invention. Figure 15 (b) shows the visualization of the behavior of the DWMF-Q method in the embodiment of the present invention. Detailed Implementation

[0064] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0065] Multi-agent systems generalize classic decision-making problems by enabling multiple agents (usually robots) to interact and achieve common or conflicting goals within a shared environment. Multi-agent learning is one of the most effective techniques for studying these systems. A simplified explanation of the multi-agent learning process is as follows:

[0066] A random game involving N agents It can be formally represented as a tuple ,in Representing the state space, It is an intelligent agent Action space; intelligent agent The reward function is defined as The state transition probability is , Representing the state space The set of probability distributions on the [aspect]; reward discount factor .

[0067] The agent selects actions according to the policy. The strategy is defined as The joint policy of all agents is represented as: Given an initial state Under the joint strategy intelligent agent The value function can be expressed as the expected cumulative discount future reward:

[0068]

[0069] In joint strategy Below, intelligent agents The Q function can be expressed as:

[0070]

[0071] In the formula, Represents the state at the next moment; value function It can be obtained through the Q function in equation (2):

[0072]

[0073] By performing joint actions by all intelligent agents The Q function is incorporated into equation (2), and the expectation of the joint action is taken in equation (3), thereby realizing the extension from single-agent game to N-agent game.

[0074] Currently, mean-field (MF) multi-agent reinforcement learning (MARL) methods applied to large-scale multi-agent scenarios neglect agent heterogeneity due to their mean-based approach, leading to the "lazy agent" phenomenon and limiting the emergent nature of group collaboration. Furthermore, the weight updates in weighted mean-field MARL lack interpretability. To address these issues, this embodiment proposes a dynamic weighted mean-field multi-agent reinforcement learning (DWMF) method, applicable to robot control in multi-agent systems composed of controlled robots as agents. This method features dynamic weights, mean fusion characteristics, and interpretability, determining weights from both individual and group dimensions. It considers both individual agent decision-making capabilities and group collaboration: reflecting individual performance through a reward mechanism and characterizing group interaction characteristics through aggregated states. Specifically, as... Figure 1 As shown, this method combines the improved PSO algorithm with mean-field multi-agent reinforcement learning. It utilizes the characteristic of PSO to balance individual and group interests during the optimization process to adjust the agent weights. To balance the exploration and utilization of the group, the parameters of the improved PSO are dynamically determined. At the same time, a cohesion factor is constructed based on the current physical position of each agent to selectively modify the weights, so as to better adapt to the current scenario and complete the task efficiently.

[0075] The method in this embodiment can be improved based on the Actor-Critic (AC) algorithm and the Independent Q-learning (IQL) algorithm to obtain the Dynamically Weighted Average Field Actor-Critic method (DWMF-AC) and the Dynamically Weighted Average Field Independent Q-learning method (DWMF-Q), such as... Figure 2 As shown, both methods include the following steps:

[0076] S101: Parameter initialization: Initialize the parameters of each agent, including the dynamic weighted average action, weight, optimal individual weight value and optimal group weight value for each agent;

[0077] Specifically, during parameter initialization, if it is based on a dynamically weighted average field Actor-Critic, then initialize... , , , , , , , , , If it is based on dynamically weighted average field Independent Q-learning, then initialize... , , , , , , , .

[0078] In this embodiment, the initial optimal weights of each agent are set according to their initial positions. and weight This assigns greater weights to agents closer to the group's centroid, aiming to enhance group effects, promote group attention, and improve cooperation among agents. Initial optimal group weights. We set this as the mean, because we expect all agents to be excellent, so we assume that each agent has the same importance and level of excellence.

[0079] S102: Obtain the initial state of each agent. .

[0080] S103: Using a specified strategy, select the action for each agent based on the dynamically weighted average action corresponding to each agent. Specifically, in the current state of each agent, select the action based on the Boltzmann policy and the dynamically weighted average action corresponding to each agent. For each intelligent agent Select Action Thus, the joint actions of all intelligent agents are obtained. .

[0081] S104: Obtain the position information of each agent in the current state, calculate the aggregation factor of each agent based on the position information, use the improved PSO algorithm, update the weight of each agent based on the aggregation factor, the optimal value of the individual weight and the optimal value of the group weight, and perform exponential normalization.

[0082] In this embodiment, the fitness function of PSO is set as the reward function. When the agent's current reward is higher than the highest historical individual reward, the agent's reward is updated. That is, the optimal value of the individual weights of the agent. Update the weights to reflect the agent's updated weights; update the weights when the group's average reward is higher than the historical highest group average reward. The optimal value of the group weight Update the weights to reflect the current group's updated weights. The formula for dynamic weight update is shown below:

[0083]

[0084] In the formula, For intelligent agents At any moment The weight adjustment rate; in order to distinguish With respect to the value function, the subsequent weight adjustment rate is defined as ; Inertia weight, ; For intelligent agents At any moment The weights; and It is a random factor; and Is A random number that takes values ​​within a range; As an aggregation factor, express The mapping results At any moment At that time, intelligent agent The negative value of the distance to the center of its neighboring agent is calculated based on the weights.

[0085] To better balance exploration and utilization, this embodiment... and Improvements were made to focus on individual exploration in the early stages and on the collective interests of the agents in the later stages.

[0086]

[0087] In the formula, and Through and The value is determined empirically. ; It is the maximum time step.

[0088] This embodiment proposes a clustering factor. This ensures stable weight updates, prevents the "lazy agent" phenomenon, and balances exploration and exploitation.

[0089] In the context of this embodiment, the position of the agent is a dynamic weight, dividing the agents into those closer to the group's centroid ( ) and farther ( Two types of intelligent agents. At the present moment There are two possibilities: moving away from the group's center of mass or moving closer to it (physically moving away from or closer to the group's center of mass). When or At that time, equations (4) and (5) can determine that a relatively faster rate should be used ( Relatively large, see equations (4)(5), (8)~(10)) decrease or increase In both of the above cases, None of the above values ​​were optimal or even better, indicating that the individual or group rewards were higher than the corresponding historical rewards. In each scenario, agents are expected to efficiently complete tasks through collaboration. Therefore, agents closer to the group's centroid are given greater flexibility, allowing them to play a more important role within the group. For agents farther from the group's centroid... ,when or At that time, at a relatively slower rate ( Relatively small, see equations (4)(5), (8)~(10)) decrease or increase In both of the above scenarios, it also indicates that the individual or group reward is higher than the corresponding historical reward, but... Possibly on the edge of the group, in order to avoid Drastic changes lead to unstable algorithm training; therefore, it is updated in a more conservative manner while retaining their exploratory role. In the early iterations, each agent is in the exploratory phase, and updating their weights more quickly allows for better exploration of group behavior patterns. In the later iterations, regardless of whether the scenario is cooperative, competitive, or adversarial, ideally, the agents tend towards a stable cooperative or competitive state. and A small difference results in a small speed increment. The aggregation factor was further reduced proportionally. This can avoid large fluctuations in weight updates and ensure the stability of the algorithm's convergence process.

[0090]

[0091] In the formula, It is time Time-based intelligent agents The position vector; Mapping to without changing the relative size of the original data ; Finally mapped to ; The degree of curve smoothness is determined to avoid problems during training. The fluctuations are too large. The value and It should be a relatively small value. .

[0092] Figure 3 illustrates the proposed implementation in four different scenarios. as well as The importance of this. In Case 1 (Figure 3(a)), agent aggregation affects their behavior, leading to the "lazy agent" phenomenon. Improved PSO enhances the agents' exploration capabilities. In Case 2 (Figure 3(b)), the red and blue sides mostly approach each other, but a few outlier agents do not contact or approach each other. In this case, even if the reward for outlier agents is high, their weight should not be too large, therefore, through... To reduce the impact of outliers on the group, in scenario 3 (Figure 3(c)), the red and blue agents are far apart, and each group exhibits clustering. To prevent agents from remaining in this state for an extended period, when the individual reward and group reward of an agent change, [the system] will [take action]. Adjusting the weights of each agent at a relatively fast rate encourages the group to change its strategy promptly, enhancing its exploration capabilities and avoiding getting trapped in local optima. In scenario 4 (Figure 3(d)), most agents are far apart, receiving only negative rewards similar to a step penalty, while a few agents are close together and can gain high positive rewards by defeating each other. In this case, it is intuitive to assign greater weights to outlier agents, using these agents to correct the group's behavior.

[0093] To demonstrate that using a dynamically weighted average field can effectively improve algorithm efficiency, we prove that "active agents" receive higher weights than "lazy agents." The following reasoning is employed:

[0094] Theorem 1. Assume the set of weights for all agents is... , The set of "active agents" is , The set of "lazy agents" is as follows: , ,and As the algorithm iterates, the weight update increment for the "active agent" is... The weight update increment for the "lazy agent" is... ,and .

[0095] Proof. Using equations (4) and (5), the weight update can be written as follows: ,make ,but:

[0096]

[0097] make , ,but ,as well as Further proof See equation (12).

[0098]

[0099] It is known that , , , Then the inequality is proved. Similarly, we can obtain... .therefore, .

[0100]

[0101] After calculating the weights of all agents using equations (4) and (5), exponential normalization (softmax) is performed to ensure that the weights are non-negative and their sum is 1. This normalization step does not affect the above proof.

[0102] S105: Update the dynamic weighted average action corresponding to each agent according to the updated weight and action of each agent, execute the joint action of all agents to obtain the next state of each agent, update the reward of each agent according to the action and state of each agent, store the current state, joint action, reward, next state and dynamic weighted average action as experience data in the experience replay buffer, and update the individual weight optimal value and the group weight optimal value according to the updated weight and reward of each agent.

[0103] In this embodiment, when storing the experience data, which consists of the current state, joint action, reward, next state, and dynamically weighted average action, into the experience replay buffer, specifically, it stores... To the experience playback buffer ,in , ,in, For the current state, For joint operations, For rewards, For the next state, This is a dynamic weighted average action.

[0104] In this embodiment, when updating the optimal individual weight and the optimal group weight based on the updated weight and reward of each agent, specifically, the reward of each agent is... Compared with the reward of the individual weight optimal value, if the reward The reward that exceeds the individual weight's optimal value will correspond to the agent. i The optimal value of the individual weights is updated to the agent's value. i Updated weights Simultaneously calculate the average reward of all agents. Then compare it with the average reward of the optimal value of the group weight. If the average reward is... The average reward for values ​​above the optimal group weight is calculated based on the updated weights of all agents. Update the optimal value of the group weight.

[0105] S106: Sample experience data from the experience replay buffer and update the Q function of each agent. Update the target network of each agent according to the updated Q function and the specified learning rate. Return to step S103 until the learning ends.

[0106] In this embodiment, when sampling experience data from the experience replay buffer, specifically from the experience replay buffer... Medium sampling K Minimum Batch Experience .

[0107] For the Q-function, DWMF enables agents to dynamically determine their weights based on their own capabilities and the aggregation state of the group. This not only improves the efficiency of cooperation among agents but also helps agents adapt to the current situation more effectively. This embodiment considers joint action and decomposes the Q-function through pairwise local interactions:

[0108]

[0109] In the formula, Indicating in intelligent agents intelligent agents in the neighborhood The weights, and satisfying Dynamic weighted average action By intelligent agents neighborhood Decision, and Add a small fluctuation :

[0110]

[0111] In the formula, Can be understood as an intelligent agent The empirical distribution of actions taken by the neighbors. Through relevant formulas and theoretical proofs, it can be proven that the Q-function constructed using DWMF can converge. According to Taylor's theorem, It can be represented as:

[0112]

[0113] In the formula, The remainder of the Taylor polynomial. The first-order term in equation (15) is due to the first-order term in equation (14). And it is 0. When When the L-smoothing condition is met, The proof is as follows:

[0114] Theorem 2. When the Q function can be additively decomposed according to equation (13), and relative to the action If it is L-smooth, then the DWMF Q function ( )and The error between them can be defined as:

[0115]

[0116] Proof. First, due to the intelligent agent status and actions Since the agent's index can be considered a fixed parameter and is independent of the derivation, the expression for the pairwise Q-function can be written as: Assume Q(a) is L-smooth, and its gradient... If a property has Lipschitz continuity, then there exists a property for all and Constants that are all valid :

[0117]

[0118] Due to the Hessian matrix It is a real symmetric matrix, which can be diagonalized. There exists an orthogonal matrix... Can be diagonalized ,get Then we can further obtain:

[0119]

[0120] In equation (14), and ,but:

[0121]

[0122] In the formula, It is a vector The Each component corresponds to a specific action. ,satisfy .but:

[0123]

[0124] Therefore, during the learning phase, based on experience The weighted average field Q-function is updated as follows:

[0125]

[0126] In the formula, Indicates the agent's current state At any moment The Q function, Indicates the agent's current state At any moment The Q function, For learning rate, It is a reward discount factor. It is an intelligent agent The reward ; Intelligent agents At any moment Weighted average field value function for:

[0127]

[0128] In the formula, It is an intelligent agent At any moment strategy, Indicates the agent's state in the next state At any moment The Q function, This represents the expectation operation. Indicates the current state. Indicating agents in joint actions The action, Represents intelligent agents The corresponding dynamic weighted average action, Indicates the next state. Indicating agents in joint actions All actions of intelligent agents other than Represents intelligent agents All other intelligent agents at time A joint strategy.

[0129] In DWMF-Q, the agent Training is performed by minimizing the loss function:

[0130]

[0131] In the formula, the Q function is parameterized as ; This is the target value for DWMF. The gradient is as follows:

[0132]

[0133] The loss function of the DWMF-AC Critic network is defined by equation (24). The DWMF-AC policy uses a weighted network... The construction of neural networks. Policy networks (Actors) Training is performed using the sampled policy gradient:

[0134]

[0135] Therefore, in this embodiment, updating the target network for each agent based on the updated Q-function and the specified learning rate includes:

[0136] If multi-agent learning uses the Actor-Critic algorithm, then according to formula (23), based on the agents... At any moment The target value is calculated using the weighted average field value function. , Then, using formula (24), the minimum loss is calculated based on the target value and the updated Q function to update the critic network, and using formula (26), the policy gradient is calculated based on the updated Q function to update the actor network. Finally, the corresponding learning rate parameter is used. and Update the intelligent agent The target network, i.e. and ;

[0137] If multi-agent learning uses the Independent Q-learning algorithm, then according to formula (23), based on the agents... At any moment The target value is calculated using the weighted average field value function. Then, using formula (24), the minimum loss is calculated based on the target value and the updated Q function to update the Q network. Finally, the corresponding learning rate parameter is used. Update the intelligent agent The target network, i.e. .

[0138] The pseudocode for the above steps is as follows, with the pseudocode for the Actor-Critic method based on the dynamic weighted average field implemented through the above steps shown in Table 1; the pseudocode for the Independent Q-learning method based on the dynamic weighted average field implemented through the above steps is shown in Table 2.

[0139]

[0140] Through the above steps, the method of this embodiment can effectively solve the problem of "lazy agents", as shown in the following example:

[0141] The MAgent general experimental platform was used, employing the Battle (adversarial and cooperative environment) and Pursuit (cooperative environment) scenarios. In each environment, agent actions are discrete. In Battle, red and blue teams compete, with rewards including single-step penalties and bonuses for defeating the opposing agent. The map size is set to 40x40, with an obstacle ratio of 0, 0.01, or 0.02. The map size can also be set to 60x60 or 100x100, with an obstacle ratio of 0. Obstacles in this environment appear randomly according to a certain ratio. In Pursuit, the red agent's task is to mark blue agents attempting to avoid the marks. Because blue agents are faster than red agents, red agents need to cooperate to better complete the task. The map size is set to 45x45, with an adversarial mode of 25 vs. 50, i.e., predator (red agent) versus prey (blue agent). In this environment, the appearance of obstacles is fixed. When a predator marks its prey, the predator receives a reward of 1 point, while the prey receives a reward of -1 point.

[0142] The method of this embodiment is compared with the training visualization results of existing research in the Battle scenario, such as... Figure 4 As shown, Figure 4 (a) shows the MF-AC results, indicating that during training, the agents of both sides aggregate separately (over multiple rounds or time steps). Figure 4 (b) In the MF-Q results shown, prolonged aggregation affects agent behavior: it hinders some agents' directional movement or other actions, leading to the phenomenon of "lazy agents." For example, some red agents move closer to blue agents and have already engaged in conflict. However, since all agents are assigned equal importance (average), intuitively, those agents that should be more important are assigned relatively lower importance, ultimately affecting the algorithm's performance. The method in this embodiment, while guiding the multi-agent group towards optimal group performance, also ensures that individual agents possess sufficient exploratory capabilities, such as... Figure 4 (c) and Figure 4 As shown in (d).

[0143] To further verify the effectiveness of the method in this embodiment, tests were conducted in multiple scenarios, taking into full account the relationships between agents and the roles of agents. The environmental requirements for agents included adversarial, cooperative, competitive, and obstacle avoidance.

[0144] Battle 64 vs 64 (with different obstacle ratios). In the Battle, agents between groups are required to compete against each other, while agents within a group are expected to cooperate. Table 3 shows the performance of each method with different obstacle ratios, and the evaluation metrics include total reward, average number of eliminations, average number of defeats, and run time per episode (seconds / episode, s / E). DWMF-AC and DWMF-Q achieve optimal or near-optimal evaluation metrics. Although the total reward decreases with increasing obstacle ratio, the method in this embodiment still has certain advantages. (See Table 3 and...) Figures 5 to 7 It can be seen that the total reward is strongly correlated with the average number of defeats, which is related to the reward settings of the environment. However, when evaluating the effectiveness of an algorithm, it is also necessary to consider factors such as running efficiency, the ratio of defeats to eliminations, and the rationality of the weights. Agent groups using DWMF-AC and DWMF-Q as policy models have a lower average number of eliminations. The method in this embodiment is similar in running efficiency to AC, MF-AC, IQL, and MF-Q, indicating that improving the weights based on simple and efficient optimization algorithms is effective. However, the AM-based method has a significant disadvantage in running efficiency, which is attributed to the larger number of AM parameters, increasing computational complexity. Furthermore, from... Figures 5 to 7 It can be seen that the convergence curves of DWMF-AC and DWMF-Q are more stable and the error band is relatively small, while the comparative method has poor stability and a slower convergence speed.

[0145] like Figures 8 to 10 As shown, although the blue agent learned a suboptimal strategy, moving to the boundary to avoid confrontation (at which point, this strategy yields a relatively high reward), the red agent using DWMF-AC not only successfully chased the blue agent but also employed an encirclement strategy to defeat more enemy agents while ensuring the survival of more agents. The red agent using DWMF-Q also learned to separate the blue agents, preventing them from clustering together. Against the comparison model, the win rate was maintained above 70%, demonstrating a certain advantage. This non-mean dynamic weight update strategy not only more effectively reflects the relative importance and contribution of individual agents in the group but also guides the agent group to achieve better adversarial and cooperative strategies in complex Battle scenarios, thus verifying the effectiveness of the method in this embodiment.

[0146]

[0147] MF-AC and MF-Q reflect the importance of each agent within the group through the mean, ignoring the differences between agents in dynamic environments. Figure 11The visualization uses MFACA and MFQA agent weights. Although there are some differences in the weights of each agent, they are not significant and do not highlight the capabilities of different agents in local observation and local environments. Through DWMF-AC and DWMF-Q weight visualization, it can be seen that red agents that come into contact with or are close to the blue team have relatively higher weights. This reflects the differences between agents, better corrects group behavior, avoids low agent participation in decision-making or interaction, and encourages agents to move closer to the blue team or complete tasks better. Table 3 also shows that this differentiation improves runtime and adversarial efficiency.

[0148] To further analyze the scalability of the mean-field theory-based algorithm, the effectiveness of each method was analyzed in two scenarios: 144 vs 144 and 400 vs 400. Table 4 shows that in the Battle environment, the reward is more influenced by the number of defeated enemies. Even if more agents survive, the group reward is not necessarily the highest. Within a round, due to the large number of agents, there may be some ineffective actions (moving away from the opponent's agent, not performing counter-actions when approaching the opponent's agent, etc.), leading to lower rewards. This invention still shows significant advantages in scenarios with a large number of agents, and its operating efficiency is similar to MF-AC and MF-Q, while AM-based methods are significantly affected. Figure 12 and Figure 13 As shown, DWMF-AC and DWMF-Q converge faster and are relatively more stable.

[0149]

[0150] Pursuit 25 vs 50. In Pursuit, cooperative behavior exists between agents. Table 3 shows the performance of different methods in this environment, with evaluation metrics including total reward, average number of labels, and runtime per round. It can be seen that DWMF-AC and DWMF-Q have significant advantages in this environment. This illustrates that the method in this embodiment enables the red agent to learn to label the blue agent through cooperation. Figure 15 It can be intuitively seen that the red agents mark individual blue agents through pairwise or multi-agent cooperation. Both DWMF-AC and DWMF-Q exhibit clear cooperative marking behavior, which is an effective strategy to compensate for their slow movement. See... Figure 14 Compared to the baseline, DWMF-AC and DWMF-Q curves converge faster and have higher values.

[0151]

[0152] Example 2

[0153] This invention provides a multi-agent learning device based on a dynamically weighted average field for executing the method of Embodiment 1, comprising:

[0154] An initialization module is used to initialize the parameters of each agent, including the dynamic weighted average action, weight, optimal individual weight value, and optimal group weight value for each agent.

[0155] The data acquisition module is used to acquire the initial state of each agent;

[0156] The data processing module is used to select the action of each agent based on the dynamic weighted average action corresponding to each agent using a specified strategy, thereby obtaining the joint action of all agents; it is also used to obtain the position information of each agent in the current state, calculate the aggregation factor of each agent based on the position information, update the weight of each agent and perform exponential normalization based on the aggregation factor, the optimal individual weight value and the optimal group weight value; it is also used to update the dynamic weighted average action corresponding to each agent based on the updated weight and action of each agent, execute the joint action of all agents to obtain the next state of each agent, update the reward of each agent based on the action and state of each agent, store the current state, joint action, reward, next state and dynamic weighted average action as experience data in the experience replay buffer, and update the optimal individual weight value and the optimal group weight value based on the updated weight and reward of each agent;

[0157] The learning and training module is used to sample experience data from the experience replay buffer and update the Q function of each agent, and update the target network of each agent based on the updated Q function and the specified learning rate.

[0158] This embodiment also proposes an electronic device, including a processor and a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and the computer program is executed by the processor to implement the steps of the multi-agent learning method based on a dynamic weighted average field described in Embodiment 1.

[0159] In summary, this invention proposes a multi-agent learning method, apparatus, and device based on a dynamic weighted average field. It utilizes the Particle Swarm Optimization (PSO) algorithm's characteristic of considering both individual interests and group benefits during the optimization process to determine dynamic weights. Considering the heterogeneity of individual agents' influence on group decision-making, dynamic parameters are introduced, and a contraction factor is designed based on the group's aggregation level to construct an interpretable weighted average field. This balances the algorithm's exploration and utilization, enhancing the decision-making ability of individual agents and the collaborative ability of the agent group. Ultimately, this optimizes the efficiency of multi-agent tasks and alleviates the "lazy agent" problem. Furthermore, the convergence of this invention is theoretically proven, and its effectiveness is verified in adversarial, cooperative, or competitive interaction scenarios.

[0160] The main contributions of this invention are as follows:

[0161] (1) This invention elaborates on the "lazy agent" phenomenon that occurs during the training process of policy models based on mean-field multi-agent reinforcement learning—that is, some agents exhibit relatively low participation in decision-making or interaction. The root cause is that the model assigns relatively low weights to agents that should receive higher attention. At the same time, existing methods based on weighted mean fields lack interpretability in terms of weight design.

[0162] (2) This invention determines the weights from both individual and group dimensions: a reward mechanism is used to reflect individual performance, and the group aggregation state is used to characterize interaction characteristics. The designed weights are dynamic, incorporating both the mean method and interpretability.

[0163] (3) Research shows that when the dynamically weighted average field Q-function satisfies the L-smoothing assumption, its gradient It possesses Lipschitz-continuous properties, which provide a theoretical basis for the convergence analysis of the algorithm. The effectiveness of the invention was verified through experiments in MAgent environments with adversarial, cooperative, or competitive characteristics.

[0164] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multi-agent learning method based on a dynamically weighted average field, characterized in that, A method for robot control applied in a multi-agent system consisting of controlled robots as intelligent agents includes the following steps: S101: Initialize the parameters of each agent, including the dynamic weighted average action, weight, optimal individual weight value and optimal group weight value corresponding to each agent, wherein the agent is a controlled robot and the controlled robot is a mobile robot; S102: Obtain the initial state of each agent, the initial state including the initial physical position, the initial optimal value of individual weights, and the weights; S103: Using a specified strategy, select the action of each agent based on the dynamic weighted average action corresponding to each agent, wherein the action is movement, thereby obtaining the joint action of all agents; S104: Obtain the physical location information of each agent in the current state, calculate the aggregation factor of each agent based on the physical location information, update the weight of each agent and perform exponential normalization based on the aggregation factor, the optimal value of individual weight and the optimal value of group weight; S105: Update the dynamic weighted average action corresponding to each agent according to the updated weight and action of each agent, execute the joint action of all agents, so that each controlled robot as an agent executes the corresponding action in the joint action, obtain the next state of each agent, update the reward of each agent according to the action and state of each agent, store the current state, joint action, reward, next state and dynamic weighted average action as experience data in the experience replay buffer, and update the individual weight optimal value and the group weight optimal value according to the updated weight and reward of each agent. S106: Sample experience data from the experience replay buffer and update the Q function of each agent. Update the target network of each agent according to the updated Q function and the specified learning rate. Return to step S103 until the learning ends. When calculating the aggregation factor for each agent based on physical location information, specifically, the negative value of the distance between each agent and the center of its corresponding neighboring agents is calculated based on the physical location information of each agent. After mapping the negative value of the distance to a specified interval, the aggregation factor of the corresponding agent is calculated using the mapping result. The mathematical expression is as follows: In the formula, As an aggregation factor, It is time Time-based intelligent agents The position vector, It is time Time-based intelligent agents neighborhood Middle intelligence agent The position vector, It is time Time-based intelligent agents The weights; At any moment At that time, intelligent agent The negative value of the distance to the center of its neighboring agent is calculated based on the weights; Indicates will The mapping result without changing the relative size of the original data. ; express The final mapping result, ; and These are different coefficients; When updating the weights of each agent based on its aggregation factor, optimal individual weight, and optimal group average weight, the calculation formula is as follows: In the formula, For intelligent agents At any moment The weight adjustment rate; Inertia weight; For intelligent agents At any moment The weights; and It is a random factor; and Is A random number that takes values ​​within a range; Aggregation factor; It is an intelligent agent The optimal value of individual weights, It is the optimal group weight value for all agents; random factor and The calculation formula is as follows: in, and It's experience points. It is the maximum time step.

2. The multi-agent learning method based on a dynamically weighted average field according to claim 1, characterized in that, When using a specified strategy to select the action of each agent based on the dynamic weighted average action corresponding to each agent, specifically, in the current state of each agent, an action is selected for each agent through the Boltzmann policy and the dynamic weighted average action corresponding to each agent.

3. The multi-agent learning method based on a dynamically weighted average field according to claim 1, characterized in that, When updating the dynamically weighted average action for each agent based on the updated weights and actions, the mathematical expression is as follows: in, Represents intelligent agents The corresponding dynamic weighted average action, Indicating in intelligent agents neighborhood Middle intelligence agent The weight, It is an intelligent agent The action.

4. The multi-agent learning method based on a dynamically weighted average field according to claim 1, characterized in that, When updating the optimal individual weight and the optimal group weight based on the updated weight and reward of each agent, the reward of each agent is compared with the reward of the optimal individual weight. If the reward is higher than the reward of the optimal individual weight, the optimal individual weight of the corresponding agent is updated to the updated weight of the corresponding agent. At the same time, the average reward of all agents is calculated and compared with the average reward of the optimal group weight. If the average reward is higher than the average reward of the optimal group weight, the optimal group weight is updated based on the updated weight of all agents.

5. The multi-agent learning method based on a dynamically weighted average field according to claim 1, characterized in that, When updating the Q-function of each agent, the mathematical expression is as follows: In the formula, Indicates the agent's current state At any moment The Q function, Indicates the agent's current state At any moment The Q function, For learning rate, It is a reward discount factor. It is an intelligent agent The reward It is an intelligent agent At any moment The weighted average field value function is expressed mathematically as follows: In the formula, It is an intelligent agent At any moment strategy, Indicates the agent's state in the next state At any moment The Q function, This represents the expectation operation. Indicates the current state. Indicating agents in joint actions The action, Represents intelligent agents The corresponding dynamic weighted average action, Indicates the next state. Indicating agents in joint actions All actions of intelligent agents other than Represents intelligent agents All other intelligent agents at time A joint strategy.

6. The multi-agent learning method based on a dynamically weighted average field according to claim 5, characterized in that, When updating the target network for each agent based on the updated Q-function and the specified learning rate, the following is included: If multi-agent learning uses the Actor-Critic algorithm, then according to the agent... At any moment The target value is calculated using a weighted average field value function. Then, based on the target value and the updated Q-function, the loss is minimized to update the critic network. The policy gradient is then calculated based on the updated Q-function to update the actor network. Finally, each agent is updated using the corresponding learning rate parameter. The target network; If multi-agent learning uses the Independent Q-learning algorithm, then according to the agents... At any moment The target value is calculated using a weighted average field value function. Then, based on the target value and the updated Q function, the loss is minimized to update the Q network. Finally, each agent is updated using the corresponding learning rate parameter. The target network.

7. A multi-agent learning device based on a dynamic weighted average field for performing the method according to any one of claims 1 to 6, applied to robot control in a multi-agent system composed of controlled robots as agents, characterized in that, The device includes: An initialization module is used to initialize the parameters of each agent. The parameters include the dynamic weighted average action, weight, optimal individual weight value, and optimal group weight value corresponding to each agent. The agent is a controlled robot, and the controlled robot is a mobile robot. The data acquisition module is used to acquire the initial state of each agent, which includes the initial physical position, the initial optimal value of individual weights, and the weights. The data processing module is used to select the action of each agent based on the dynamic weighted average action corresponding to each agent using a specified strategy. The action is movement, thereby obtaining the joint action of all agents. It is also used to acquire the physical position information of each agent in its current state, calculate the aggregation factor of each agent based on the physical position information, update the weight of each agent and perform exponential normalization based on the aggregation factor, the optimal individual weight value, and the optimal group weight value. Furthermore, it is used to update the dynamic weighted average action corresponding to each agent based on the updated weight and action, execute the joint action of all agents, so that each controlled robot, acting as an agent, executes the corresponding action in the joint action, obtains the next state of each agent, updates the reward of each agent based on the action and state, stores the current state, joint action, reward, next state, and dynamic weighted average action as experience data in the experience replay buffer, and updates the optimal individual weight value and the optimal group weight value based on the updated weight and reward of each agent. The learning and training module is used to sample experience data from the experience replay buffer and update the Q function of each agent, and update the target network of each agent based on the updated Q function and the specified learning rate.

8. An electronic device, characterized in that, The device includes a processor and a computer-readable storage medium storing a computer program, which is executed by the processor to implement the steps of the multi-agent learning method based on a dynamically weighted average field as described in any one of claims 1 to 6.

Images