Multi-agent collaborative reinforcement learning method, device, equipment, medium and product

Abstract

The invention discloses a multi-agent collaborative reinforcement learning method, device and equipment, a medium and a product. The method comprises the following steps: constructing an experience pool according to historical action parameters of multiple agents, selecting target experience data from the experience pool, and determining next-step execution strategy parameters of the multiple agents according to the target experience data and a multi-agent collaborative network; global return data and local return data are determined according to the next execution strategy parameters of the multiple agents; and performing parameter updating on the multi-agent collaborative network based on the global evaluation network and the local evaluation network according to the action parameters, the strategy parameters, the global return data and the local return data of the multiple agents. According to the method, the optimization efficiency of the intelligent agent strategy is maximized through the double-evaluation network architecture, the global evaluation network is responsible for capturing the macroscopic situation of multi-agent interaction, the local evaluation network focuses on strategy optimization of an individual behavior level, accurate updating of the strategy is achieved through the synergistic effect of the global evaluation network and the local evaluation network, and then the cooperation ability of the intelligent agents is improved.

Description

Technical Field

[0001] This invention relates to the field of multi-agent cooperative technology, and in particular to a reinforcement learning method, apparatus, device, medium and product for multi-agent cooperative learning. Background Technology

[0002] With the development of artificial intelligence and autonomous systems, multi-agent cooperative control has shown broad application prospects in fields such as intelligent traffic scheduling, unmanned swarm formation, intelligent manufacturing, robot collaboration, and smart city management. In these complex and dynamic environments, multiple agents need to achieve efficient cooperation through effective decision-making mechanisms to accomplish tasks that are difficult for a single agent to achieve.

[0003] In recent years, reinforcement learning (RL) has been widely used in decision optimization of multi-agent systems due to its characteristics of not requiring prior models and being able to learn optimal strategies autonomously through trial and error. In particular, deep reinforcement learning (DRL), combined with the powerful function approximation capabilities of neural networks, enables agents to effectively explore and learn in high-dimensional state and action spaces.

[0004] However, directly applying traditional single-agent reinforcement learning methods in multi-agent scenarios faces numerous challenges. For example, the current learning process of an agent is easily disturbed by the dynamic evolution of other agents' policies, leading to instability. A common improvement strategy is to decompose the central value function into local value functions for each agent, and then the agent selects actions based on these local value functions. However, this is typically used in environments where only global rewards are available, not individual rewards. In other words, the main problem is that simply applying basic reinforcement learning and experience replay algorithms to production scenarios, while achieving practical application, fails to consider the complexity of multi-agent training in real-world production environments. Summary of the Invention

[0005] This invention provides a method, apparatus, device, medium, and product for multi-agent collaborative reinforcement learning to address the issues of efficiency and stability in multi-agent collaborative reinforcement learning.

[0006] According to one aspect of the present invention, a multi-agent cooperative reinforcement learning method is provided, comprising:

[0007] An experience pool is constructed based on the historical action parameters of the multi-agent, target experience data is selected from the experience pool, and the next execution strategy parameters of the multi-agent are determined based on the target experience data and the multi-agent cooperative network.

[0008] The global and local reward data are determined based on the next step execution strategy parameters described by the multi-agent;

[0009] The multi-agent cooperative network updates parameters based on the action parameters, policy parameters, global reward data, and local reward data of the multi-agents using a global evaluation network and a local evaluation network. The global evaluation network optimizes the interaction policy parameters of the multi-agents in the cooperative network based on global reward data, while the local evaluation network optimizes the policy parameters of individual agents in the cooperative network based on local reward data. The cooperative network supports the agents in collaboratively executing the target task.

[0010] According to another aspect of the present invention, a multi-agent cooperative reinforcement learning device is provided, comprising:

[0011] The experience selection module is used to construct an experience pool based on the historical action parameters of the multi-agent, select target experience data from the experience pool, and determine the next execution strategy parameters of the multi-agent based on the target experience data and the multi-agent cooperative network.

[0012] The reward data determination module is used to determine global reward data and local reward data based on the next step execution strategy parameters of the multi-agent;

[0013] A dual-network parameter update module is used to update the parameters of a multi-agent cooperative network based on the action parameters, policy parameters, global reward data, and local reward data of the multi-agents, using a global evaluation network and a local evaluation network. The global evaluation network is used to optimize the interaction policy parameters of the multi-agents in the cooperative network based on global reward data, and the local evaluation network is used to optimize the policy parameters of individual agents in the cooperative network based on local reward data. The cooperative network supports the agents in collaboratively executing a target task.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the multi-agent cooperative reinforcement learning method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the multi-agent cooperative reinforcement learning method according to any embodiment of the present invention.

[0019] According to another aspect of this application, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the multi-agent cooperative reinforcement learning method described in any embodiment of the present invention.

[0020] The technical solution of this invention maximizes the optimization efficiency of agent policies through a dual evaluation network architecture. The global evaluation network is responsible for capturing the macro-level situation of multi-agent interactions, while the local evaluation network focuses on policy optimization at the individual behavior level. The two work together to achieve accurate policy updates, thereby improving the collaborative ability of agents.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a multi-agent cooperative reinforcement learning method provided according to an embodiment of the present invention;

[0024] Figure 2 This is a flowchart of another multi-agent cooperative reinforcement learning method provided according to an embodiment of the present invention;

[0025] Figure 3 This is a diagram of the overall architecture of multi-agent collaborative reinforcement learning that combines dynamic decomposition of deep deterministic policy gradient (DD-MADDPG) and dual priority experience replay mechanism (T2-PER).

[0026] Figure 4 This is an overview diagram of the simulation environment for four simulation scenarios;

[0027] Figure 5 This is a comparison chart of the first global reward change curve;

[0028] Figure 6 This is a comparison chart of the second global reward change curve;

[0029] Figure 7 This is a schematic diagram of the structure of a multi-agent cooperative reinforcement learning device according to an embodiment of the present invention;

[0030] Figure 8 This is a schematic diagram of the structure of an electronic device that implements the multi-agent cooperative reinforcement learning method of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "candidate," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Figure 1 This invention provides a flowchart of a multi-agent cooperative reinforcement learning method. This embodiment is applicable to situations where multi-agent cooperative reinforcement learning is being optimized. The method can be executed by a multi-agent cooperative reinforcement learning device, which can be implemented in hardware and / or software and can be configured in a server with computing power. Figure 1 As shown, the method includes:

[0034] S110. Construct an experience pool based on the historical action parameters of the multi-agent, select target experience data from the experience pool, and determine the next execution strategy parameters of the multi-agent based on the target experience data and the multi-agent collaborative network.

[0035] This embodiment is based on a multi-agent cooperative network with centralized training and decentralized execution. The multi-agent cooperative network is used to support agents to perform collaborative execution of target tasks. That is, multiple agents formulate strategies and execute actions according to the multi-agent cooperative network to achieve the goal of completing the target task through multi-agent collaboration.

[0036] The agent acquires joint observations from the environment at time t. Each agent i extracts individual observations from the joint observations. And based on their individual strategies Generate corresponding actions Multi-agent cooperative systems execute joint actions based on multi-agent cooperative network models. Then, the environment transitions to the next state. At the same time, the system receives a global reward. Local rewards The specific parameter definitions are as follows: ; ; ; N represents the number of agents.

[0037] An experience pool (also known as an experience replay buffer, DD) is a data structure that stores historical interaction data. Each piece of experience data in the experience pool is typically a tuple. For example, an agent's interaction with its environment generates multiple tuples of experience data. , , , , That is, each piece of experience data in the experience pool includes the joint observation, joint action parameters, local reward data, global reward data, and the joint observation corresponding to the next time step.

[0038] During training, small batches of target experience data are randomly sampled from the experience pool to update the policy network and value network. In multi-agent cooperative systems, acquiring experience data from the experience pool is a crucial step in the training process, especially in deep reinforcement learning frameworks (such as MADDPG, MAPPO, QMIX, etc.), which improves learning efficiency and stability.

[0039] In Multi-Agent Reinforcement Learning (MARL), the primary purpose of acquiring target experience data from the experience pool is to update the neural network parameters of the agents. These parameters determine how each agent perceives the environment, selects actions, and evaluates the value of its actions. For example, after sampling target experience data from the experience pool, the following two types of core parameters are updated using gradient descent (or ascent) algorithms: 1. Policy network parameters: These parameters control the policy functions of each agent and determine the parameters for the next policy action; 2. Value network parameters: These parameters evaluate the quality of the current state and action. Common forms include joint action value functions, state value functions, and advantage functions. The update method minimizes the error between the predicted and target values. For example, the policy functions of each agent can be selected according to the actual scenario, and this is not a restriction.

[0040] S120. Determine the global reward data and local reward data based on the next execution strategy parameters of the multi-agent system.

[0041] Multi-agent systems refer to systems composed of multiple agents (such as robots, drones, and software agents) that collaborate to complete a task. The next execution strategy parameters refer to the parameters corresponding to the policy function that each agent will adopt (usually the weights of a neural network used to determine the next action). Global reward data is determined based on the global reward obtained by all agents as a whole, reflecting the team's task completion status (e.g., whether an object was successfully moved, or whether an opponent was defeated). Local reward data is determined based on the local reward obtained by each agent, which may be based on individual behavioral performance (e.g., distance traveled, energy consumption, number of collisions). The relationship between global reward data and global reward data is determined based on a certain mathematical mapping relationship, which can be determined according to the actual situation and is not limited here. Similarly, the relationship between local reward data and local reward data is also determined based on a certain mathematical mapping relationship.

[0042] For example, each agent predicts its next action based on the next execution policy parameters, and then evaluates the global reward data and local reward data. The global reward data is used to determine the global reward data, and the local reward data is used to determine the local reward data. For instance, the system predicts what each agent will do next based on the policy it is about to use (defined by parameters), and thereby estimates how much total team benefit (global reward) and each member's individual score (local reward) this collaboration will bring.

[0043] S130. Based on the global evaluation network and the local evaluation network, the parameters of the multi-agent cooperative network are updated according to the action parameters, policy parameters, global reward data and local reward data of the multi-agents.

[0044] Among them, the multi-agent cooperative network is a network that determines the action data of each agent. That is, the agent relies on the agent cooperative network to determine the next execution strategy parameters, and then determines the action data.

[0045] In this embodiment, a local evaluation network is added to the existing MADDPG algorithm framework. This local evaluation network is specifically used to quantify the effect of the agent's current strategy on its own behavior. Simultaneously, the original centralized critic network of MADDPG is reconstructed into a global evaluation network, whose function focuses on evaluating the impact of the agent's strategy on the group's collaborative effectiveness. Specifically, this dual-network approach combines individual behavior with overall collaboration. At the beginning of training, due to the agent's lack of experience, it pays more attention to the local evaluation network and less attention to the global evaluation network. As training progresses, the agent gains more experience and is better able to complete individual tasks. It then gradually reduces its focus on the local evaluation network and continues to increase its focus on the global evaluation network.

[0046] Specifically, the global evaluation network is used to optimize the interaction policy parameters of multiple agents in a multi-agent cooperative network based on global reward data, while the local evaluation network is used to optimize the policy parameters of a single agent in a multi-agent cooperative network based on local reward network.

[0047] For example, the global gradient of the expected return in the global evaluation network is determined based on the policy parameters of the multi-agent network, the corresponding joint action parameters, and the acquired global return data; the local gradient of the expected return in the local evaluation network is determined based on the policy parameters of each agent, the corresponding individual action parameters, and the acquired local return data; and the parameter update of the multi-agent cooperative network is guided by the specific values ​​of the global gradient and the local gradient.

[0048] This paper proposes a novel multi-agent learning framework, Dynamic Decomposition of Deep Deterministic Policy Gradient (DD-MADDPG), which employs a dual-evaluation network architecture to maximize the optimization efficiency of agent policies. By simultaneously considering both global and local evaluation metrics, a dynamic policy update mechanism is achieved. This design enables agents to effectively collaborate with each other while minimizing the impact of other agents' policies on the current training process. Specifically, the global evaluation network captures the macro-level situation of multi-agent interactions, while the local evaluation network focuses on policy optimization at the individual behavior level. Their synergy, through a dynamic weight adjustment mechanism, achieves precise exploration and utilization of the policy space.

[0049] In one feasible embodiment, S130 includes:

[0050] Determine the centralized action value function, and based on the centralized action value function, determine the global rate of change of the global reward when multiple agents change their actions;

[0051] Determine the individual action value function, and based on the individual action value function, determine the local rate of change of the local reward when the target agent changes its action;

[0052] Determine the action change information of the action parameters under the corresponding individual observation when the individual policy parameters of the target agent change;

[0053] Determine the global policy gradient based on the global rate of change and action change information;

[0054] Determine the local policy gradient based on local rate of change and action change information;

[0055] The gradient parameters of the expected reward of the multi-agent system are determined based on the fusion results of the global policy gradient and the local policy gradient.

[0056] The parameters of the multi-agent cooperative network are updated based on the direction of change of the gradient parameters.

[0057] Specifically, the centralized action value function is used to determine the loss based on global return data, and the specific form of loss can be adjusted according to the actual situation. The individual action value function can determine the loss based on local return data, and the specific form of loss can also be adjusted according to the actual situation.

[0058] Action change information represents the change in the probability of the i-th agent choosing an action under the current observation when the policy parameters of the i-th agent undergo a small change. The global policy gradient is the average value of the product of the global rate of change and the action change information obtained by sampling each piece of experience data in the experience pool; the local policy gradient is the average value of the product of the local rate of change and the action change information obtained by sampling each piece of experience data in the experience pool.

[0059] The gradient parameter of the expected reward of the multi-agent network indicates the direction and rate of change of the reward loss relative to the parameters of the multi-agent cooperative network, while the parameter update of the multi-agent cooperative network is to adjust the parameters in the opposite direction of the gradient in order to minimize the loss function.

[0060] In a feasible embodiment, the gradient parameters of the expected reward of the multi-agent are determined based on the fusion result of the global policy gradient and the local policy gradient, including:

[0061] Determine the global weights corresponding to the global policy gradient and the local weights corresponding to the local policy gradient;

[0062] The gradient parameters of the expected reward of the multi-agent system are determined by summing the product of the global policy gradient and the global weights, and the product of the local policy gradient and the local weights.

[0063] In this process, the local weights are updated using a progressive decay mechanism, and the sum of the global weights and the local weights is 1.

[0064] For example, global weights and local weights are determined according to the following formula: ; Here, `decay` is a hyperparameter, which is always less than 1. The initial value is 1, and a gradual decay mechanism is used.

[0065] In a feasible example, the gradient parameters of the expected reward of the multi-agent system are determined according to the following formula:

[0066] ;

[0067] in, Represents the global weight. Indicates local weights, Indicates joint observation, This represents the individual observations extracted by the i-th agent from the joint observations. This represents the individual policy parameters of the i-th agent. This represents the action parameters generated by the i-th agent based on the individual policy parameters. Represents the joint action parameters of multiple agents. Represents the relevant motion parameters of the i-th agent. This represents the value function of centralized actions. This represents the value function of an individual action. Indicates information about changes in action. Indicates the global rate of change. This represents the local rate of change.

[0068] Indicates that the i-th agent is in In this environment, actions were taken based on the strategy parameters. Then, the expected Q-value obtained by the i-th agent, that is, the local reward data of the i-th agent, This refers to multi-agent systems. In a joint environment, joint actions were taken based on a subset of the policy parameters. The Q-value obtained later is the global reward data corresponding to the multi-agent system.

[0069] In this embodiment, MADDPG is extended to DD-MADDPG, and the gradient of the update policy of the i-th agent is the gradient parameter of the expected reward expressed by the above formula; where, That is, the i-th agent in According to The actions taken It is by (Parameters) are defined. D is the experience pool. and These are the dynamic fusion weight coefficients of the global evaluation network and the local evaluation network, which represent the weight allocation of the policy network's attention to the value assessment of the two types of evaluation networks.

[0070] For example, in reinforcement learning, an agent's goal is to maximize its expected cumulative reward. In multi-agent reinforcement learning, the gradient of the agent's expected reward is a core driving force in the training process, determining how each agent adjusts its policy to improve long-term performance. To optimize the agent's execution policy, this embodiment calculates the gradient of the policy parameters according to the above formula. This gradient determines the trend of change in expected reward when the policy parameters of the i-th agent are changed, and the direction of parameter adjustment for the agent cooperative network is determined based on this trend. The value of the gradient can indicate the direction of policy update, for example, using gradient ascent.

[0071] In the above gradient determination formula, This indicates that the expected value is calculated by sampling the empirical data in the experience pool D. The policy grid of the i-th agent is represented with respect to its parameters. The gradient, which represents the value when the i-th agent observes the environment. And select an action The sensitivity of the probability density output by the policy to the parameters. This represents the relevant parameters in the policy network of the i-th agent; This represents the centralized action value function with respect to the action of the i-th agent. The partial derivative of , which represents the change in action of the i-th agent. The extent of the impact on the overall system's global return data (Q value); The individual action value function represents the action of the i-th agent. The partial derivative of , which represents the change in action of the i-th agent. The degree of impact on the corresponding local return data (Q value).

[0072] In a feasible implementation, the centralized action value function updates the policy parameters according to the following process:

[0073] The expected total global reward is determined based on the global reward data of the multi-agent at the current moment and the predicted global reward data after performing joint action parameters under joint observation at the next moment.

[0074] The first loss is determined based on the difference between the current global reward data and the expected total global reward after the multi-agents execute joint action parameters under the joint observation at the current moment;

[0075] Update the strategy parameters based on the first loss.

[0076] Specifically, the expected total global return equals the sum of the global reward data and the predicted global return data. Furthermore, the expected total global return equals the sum of the global reward data and the predicted global return data multiplied by a discount factor. The specific value of the discount factor can be adjusted according to actual circumstances and is not limited here.

[0077] The first loss is equal to the expected value of the square of the difference between the current global return data and the expected total global return. That is, the average of the squares of the differences between the current global return data and the expected total global return obtained by sampling each empirical data in the empirical pool is used as the expected value.

[0078] For example, the update strategy parameters for the centralized action value function are:

[0079] ;

[0080] in, Indicates the expected total return. Indicates the joint observation at the next moment. Indicates the parameters of the joint action at the next moment. , This represents global reward data. Indicates the discount factor. This represents a subset of the policy parameter set. This represents the predicted global reward data after multiple agents execute joint action parameters under joint observation; that is, the current global reward data of the current neural network for the joint action global reward. This indicates the predicted global return value for the next time step using the current network's prediction data; that is, using the future global return data predicted by the target network. This is the subset of parameters corresponding to the next moment.

[0081] The loss parameters of the centralized action value function are defined by the updated strategy parameters. By minimizing the squared error between the current global reward data and the predicted future global reward data, the network learns to accurately evaluate the value of team collaboration behavior, thereby guiding the agent to make better decisions.

[0082] In a feasible implementation, the individual action value function updates the policy parameters according to the following process:

[0083] The expected total local reward is determined based on the local reward data of the target agent at the current moment and the predicted local reward data after performing the corresponding action parameters under individual observation at the next moment.

[0084] The second loss is determined based on the difference between the current local reward data and the expected total local reward after the target agent performs the corresponding action parameters under the individual observation at the current moment;

[0085] Update the strategy parameters based on the second loss.

[0086] Specifically, the expected total local return equals the sum of the local reward data and the predicted local return data. Furthermore, the expected total local return equals the sum of the results of multiplying the local reward data and the predicted local return data by a discount factor. The specific value of the discount factor can be adjusted according to actual circumstances and is not limited here.

[0087] The second loss is equal to the expected value of the square of the difference between the current local return data and the expected total local return. That is, the average of the squares of the differences between the current local return data and the expected total local return is obtained by sampling each empirical data in the empirical pool, and the expected value is taken as the average value.

[0088] For example, the update policy parameters for the individual action value function are:

[0089] ;

[0090] in, Indicates the expected total local return. This represents the local reward data for the i-th agent. This represents the individual observation extracted by the i-th agent from the joint observation at the next moment. This represents the individual policy parameters of the i-th agent at the next time step. This represents the action parameters generated by the i-th agent at the next moment based on the individual policy parameters. Indicates the discount factor. This represents the predicted local reward data obtained by the i-th agent when performing the corresponding action parameters under individual observation; that is, the current local reward data of the neural network for the agent's individual action. This represents the predicted value of the local reward data for the next moment using the current network prediction, i.e., the predicted local reward data for each agent in the future using the target network prediction.

[0091] The loss parameters of the individual action value function are defined by the above-mentioned update strategy parameters. By minimizing the squared error between the current local reward data and the predicted future local reward data, the network learns to accurately evaluate the value of individual actions, thereby guiding the agent to make better decisions.

[0092] The technical solution of this embodiment maximizes the optimization efficiency of agent policies through a dual evaluation network architecture. The global evaluation network is responsible for capturing the macro-level situation of multi-agent interactions, while the local evaluation network focuses on policy optimization at the individual behavior level. The two work together to achieve accurate policy updates, thereby improving the collaborative ability of agents.

[0093] Figure 2 This is a flowchart of another multi-agent collaborative reinforcement learning method provided by an embodiment of the present invention. This embodiment further refines the empirical data selection process in the above embodiments. Figure 2 As shown, the method includes:

[0094] S210. Determine the global reward data and global return data for each experience data based on the historical action parameters of the multi-agent, and determine the temporal difference error for each experience data based on the global reward data and global return data.

[0095] The DD-MADDPG method described in the above embodiments can reduce the interference of other agent policies on the current agent training, but it does not fully consider the dynamic characteristics of agent policies evolving over time. In multi-agent systems, since the policies of each agent are continuously iterated and updated, directly reusing old data from the historical experience pool for training can lead to suboptimal convergence in policy optimization. To address this, this embodiment proposes a priority experience sampling mechanism (T2-PER) that integrates temporal difference error (TD error) and sample timeliness to replace the traditional random sampling method.

[0096] For example, the agent interacts with the environment to generate multiple tuples of experience data ( , , , , The temporal difference error of each empirical data tuple is determined based on the global reward data and global return data of the corresponding empirical data tuple at the next time step. For example, the temporal difference error is determined based on a first difference between the global return data corresponding to the empirical data at the next time step and the global return data corresponding to the empirical data at the current time step. Further, a second difference is determined based on the global reward data corresponding to the empirical data at the next time step and the global reward data corresponding to the empirical data at the current time step. The temporal difference error is then determined based on the weighted value of the first and second differences.

[0097] In one feasible embodiment, determining the temporal difference error of each empirical data point based on global reward data and global return data includes:

[0098] Determine the global reward data for each experience data point in the next time step;

[0099] Determine the current global reward data for each empirical data point at the current moment under the joint observation and joint action parameters;

[0100] Determine the next global reward data for each empirical data point under the joint observation and joint action parameters at the next time step;

[0101] The temporal difference error for each empirical data point is determined based on the global reward data, the current global reward data, and the next global reward data.

[0102] The time-series difference error is determined by comparing the sum of the global reward data and the next global return data at the next time step with the current global return data. Further, the time-series difference error is determined by comparing the sum of the global reward data and the next global return data multiplied by a discount factor at the next time step with the current global return data. The specific value of the discount factor can be adjusted according to actual circumstances and is not limited here.

[0103] For example, the timing difference error is determined according to the following formula:

[0104] ;

[0105] in, This represents the time-series difference error of the i-th empirical data point. Let represent the global reward data for the i-th experience data at time t+1, i.e., the global reward data for the next time step. Indicates the discount factor. This represents the global reward data for the i-th empirical data at time t, under the joint observation and joint action parameters, i.e., the current global reward data. Let represent the global return data of the i-th empirical data at time t+1 under the joint observation and joint action parameters, i.e. the next global return data. The two Q values ​​represent the global Q values ​​of empirical sample i at t and t+1, respectively.

[0106] This embodiment improves the accuracy of determining the time-series difference error for each empirical data point, so as to accurately represent the return value corresponding to each empirical data point.

[0107] S220. Determine the priority of each empirical data point based on its timing difference error and storage time.

[0108] In addition to introducing the temporal difference error of each empirical data, the timeliness of the sample is also taken into consideration when determining the priority of the empirical samples. The timeliness of the sample is determined based on the storage time of the empirical data. The longer the storage time, the lower the timeliness of the empirical data sample.

[0109] Specifically, storage time is defined as the time sequence parameter of when experience data is stored in the experience pool. The later the experience data is stored, the larger its storage time value, thereby increasing the priority of recent experience and decreasing the priority of early experience.

[0110] For example, the timing difference error is positively correlated with the priority of each empirical data point, while the storage time is negatively correlated with the priority of each empirical data point. The final priority is determined by combining the first relationship between timing difference error and the priority of each empirical data point, and the second relationship between storage time and the priority of each empirical data point. For example, the combination result can be a product combination or an additive combination, which is not limited here. The first relationship is a positive correlation, and the second relationship is a negative correlation. The specific form of the correlation is not limited here.

[0111] For example, in the above example, the experience data stored in the experience pool is in the form of tuples. Furthermore, the priority of each piece of experience data is also added to the tuple. For example, the experience data tuple is ( , , , , , ).

[0112] In one feasible embodiment, the priority of each empirical data point is determined based on its temporal differential error and storage time, including:

[0113] The impact of storage time on error is determined based on the storage time and influencing factors of each empirical data point.

[0114] The priority of each empirical data point is determined based on the fusion result of storage time impact error and timing difference error.

[0115] Specifically, the impact of storage time on error is negatively correlated with storage time; that is, the longer the storage time of any empirical data, the smaller the impact of storage time on error. For example, a first value is determined with a natural number as the base and the negative of the product of storage time and the impact factor as the exponent. The difference between the value 1 and the first value is used to determine the impact of storage time on error. The priority of each empirical data is determined by the product of the impact of storage time on error and the time-series difference error. Furthermore, the priority of each empirical data is determined by the product of the square of the impact of storage time on error and the time-series difference error.

[0116] For example, priority is determined according to the following formula:

[0117] ;

[0118] in, This indicates the priority of the i-th empirical data point. Indicates the impact factor. This represents the storage time of the i-th piece of experience data. The initial storage time of each piece of experience data when it is first added to the experience pool is 1, and it increases continuously during training. K is an influencing factor that determines the rate at which the utility of experience decays in policy updates. The larger K is, the stronger the bias towards recent experience data. Its specific value can be adjusted according to the actual situation and is not restricted here. For example... .

[0119] This embodiment improves the accuracy of prioritizing empirical data based on temporal difference error and storage time, thereby increasing the contribution of sampling results to training when sampling empirical data.

[0120] S230. Select the target experience data according to the priority of each experience data in the experience pool.

[0121] For example, the experience data in the experience pool are sorted according to priority, and target experience data are selected according to the sorting result. For instance, a preset number of experience data with the highest priority are selected as the target experience data for the current training batch. The specific value of the preset number can be set according to the actual situation and is not limited here.

[0122] In one feasible embodiment, target experience data is selected according to the priority of each experience data in the experience pool, including:

[0123] The selection probability of each piece of experience data is determined based on the priority of each piece of experience data in the experience pool, and the target experience data is selected based on the selection probability.

[0124] The probability of selecting each empirical data point is determined based on its proportion within the sum of priority control values ​​for all empirical data, where the priority control value is determined by the data's priority. This priority control value is determined based on the data's priority and priority control parameters. For example, the probability of selecting the i-th empirical data point is: K represents the total number of experience data in the experience pool. This indicates the priority control parameter.

[0125] By calculating the temporal difference error of the empirical data and its storage duration in the empirical replay pool, the sample priority is dynamically determined, and a sampling probability distribution is generated based on the priority weights. The probability of the i-th empirical data point being selected is: ,index Used to control the degree of prioritization, for example, when When the value is 0, the system degenerates into a uniform sampling mode, where all empirical samples have equal selection probabilities. The specific value can be adjusted according to the priority requirements for selection, but there is no restriction on its specific value here.

[0126] S240. Determine the next execution strategy parameters of the multi-agent network based on the target experience data and the multi-agent cooperative network, and determine the global reward data and local reward data based on the next execution strategy parameters of the multi-agent network.

[0127] S250 updates the parameters of the multi-agent cooperative network based on the action parameters, policy parameters, global reward data, and local reward data of the multi-agents, using a global evaluation network and a local evaluation network.

[0128] This embodiment proposes a dual-priority experience replay mechanism (T2-PER) that integrates temporal difference error (TD error) and sample timeliness based on the DD-MADDPG algorithm framework. This mechanism dynamically determines the priority weight of experience samples by jointly considering both TD error and sample timeliness indicators, and constructs a sample sampling probability distribution model based on these priority weights. This method significantly reduces the probability of reusing outdated experience data, thereby effectively suppressing the interference effect of the outdated historical experience of other agents on the current agent's policy optimization during multi-agent collaborative training. Specifically, TD error serves as a quantitative representation of policy value deviation, while sample timeliness reflects the dynamic effectiveness of experience data. The synergistic optimization of these two aspects achieves accurate selection of high-value samples in the experience pool and a comprehensive improvement in policy training efficiency.

[0129] The technical solution of this embodiment optimizes the efficiency and stability of multi-agent collaborative training by dynamically adjusting the reuse probability of experience data in the experience pool through the temporal difference error and timeliness characteristics of quantified sample experience data.

[0130] This invention proposes a multi-agent reinforcement learning method based on dynamic decomposition of deep deterministic policy gradients. It integrates a dual-priority experience replay mechanism that considers both temporal difference error and sample timeliness, addressing the coordination problem in multi-agent learning under continuous space and discrete time conditions. It also solves problems such as low utilization of existing experience and the impact of long-term experience on short-term training. Figure 3 The diagram shows the overall architecture of a multi-agent collaborative reinforcement learning system that combines Dynamic Decomposition of Deep Deterministic Policy Gradient (DD-MADDPG) and Double Priority Experience Replay (T2-PER). The overall workflow of DD-MADDPG and T2-PER is as follows:

[0131] Starting from iteration 1: From time t=1 to the end of this iteration: Agent i performs a series of actions to obtain global reward data and local reward data; determines the temporal difference error of the experience data based on the global reward data, calculates the experience priority based on the temporal difference error and storage time, and stores the experience data carrying the priority into the experience pool.

[0132] If the number of iterations is an integer multiple of d, then calculate the selection probability of each piece of empirical data, and extract a batch of target empirical data from the experience pool based on the selection probability.

[0133] After extracting the target experience data, for each agent, the action to be executed is determined according to the policy network and all actions are executed.

[0134] Based on the update strategies for the centralized action value function and the individual action value function, the global evaluation network and the local evaluation network are updated by minimizing the loss function. For example, the network parameters are updated using the Adam method: when If convergence is not achieved, update the partial first-order moment estimate based on the gradient of the expected return, update the partial second-order primitive moment estimate, calculate the bias-corrected first-order moment estimate, calculate the bias-corrected second-order primitive moment estimate, and finally update the parameters based on the corrected moment estimates. Update the network parameters based on the gradient of the expected return. Repeat the above steps until iteration T ends.

[0135] DD-MADDPG and T2-PRE were applied to simulation experiments, and the architecture of this embodiment was evaluated based on the performance of the control algorithm.

[0136] Specifically, to verify the optimization effects of the above framework and reinforcement learning experience replay mechanism, the effectiveness of DD-MADDPG and T2-PER was evaluated in the MPE (a time-discrete, spatially continuous two-dimensional multi-agent environment developed by OpenAI) simulation environment. The Adam optimizer was used to train the parameters of the neural network. Experimental results show that compared with the classic MADDPG algorithm, Dynamic Decomposition of Deep Deterministic Policy Gradient (DD-MADDPG) significantly improves training stability and obtains higher cumulative rewards. The dual-priority experience replay mechanism (T2-PER) based on temporal difference error and sample timeliness can further enhance the learning stability under the DD-MADDPG framework. The combined DD-MADDPG and T2-PER scheme has obvious performance advantages.

[0137] Experimental simulations were conducted on the multi-agent collaborative reinforcement learning architecture combining DD-MADDPG and T2-PER according to embodiments of the present invention to verify the efficiency and accuracy of reinforcement learning in this embodiment.

[0138] For example, the four simulation scenarios are: Cooperative Navigation (where agents need to coordinate to cover N landmarks, obtaining global rewards by minimizing landmark distances while incurring collision penalties to avoid obstacles), Keep-Away (where N agents compete with M adversaries around a single landmark; agents need to approach the landmark, while adversaries gain benefits by driving agents away), Predator-Prey (where high-speed green agents (prey) need to avoid being surrounded by low-speed red adversaries (predators); obstacle settings increase the complexity of path planning), and Physical Deception (where N green agents need to implement a deception strategy by distributing coverage of multiple landmarks under conditions of information asymmetry (adversaries do not know the identity of the target landmark), while ensuring that at least one agent approaches the target landmark, while adversaries need to autonomously identify the real target). These environments respectively verify core capabilities such as multi-agent cooperative coverage, adversarial game, dynamic obstacle avoidance, and distributed deception. Figure 4 The diagram shows an overview of the simulation environment for four simulation scenarios.

[0139] In the simulation experiments, the performance of the scheme was evaluated from two aspects: learning stability and global reward, using the cumulative reward curve during agent training and the average reward and its 95% confidence interval for the last 10,000 training rounds. The baseline comparison algorithms included MADDPG, DDPG, TD3, and DE-MADDPG. All models used the Adam optimizer, and the learning rates for the policy network and the critic network were set separately. and The policy network and commentator network are both structured as 3 fully connected layers (input layer + 2 hidden layers + output layer) and 4 fully connected layers (input layer + 3 hidden layers + output layer), respectively. Each hidden layer contains 64 neurons, and the activation function is ReLU. The empirical replay pool capacity is... Batch size 256, discount factor . Figure 4 The AD was trained for 100,000, 50,000, 300,000, and 300,000 rounds respectively.

[0140] In experimental evaluation, firstly, neither our proposed scheme nor the control algorithm employed any improved temporal difference algorithm. Based on experimental data from 10 independent random seeds, we calculated the mean global reward for the last 10,000 training rounds and labeled its 95% confidence interval. This statistical method quantifies the algorithm's convergence stability and sensitivity to random initialization; the confidence interval range reflects the dispersion of policy performance under different seeds, such as... Figure 5The figure shown is a comparison of the first global reward change curve, demonstrating the efficiency and robustness of DD-MADDPG in multi-agent cooperative optimization. Secondly, the DD-MADDPG combined T2-PER algorithm is validated in two environments: cooperative navigation and area defense, and compared with the combined TD-PER and combined S-BUFFER algorithms. Figure 6 The figure shows a comparison of the second global reward change curve. The T2-PER mechanism can significantly accelerate policy convergence, reaching a stable state within 10,000 training epochs, while the TD-PER and S-BUFFER methods require approximately 20,000 epochs to converge. The average reward of T2-PER is higher than that of other methods in both environments, and its reward standard deviation is also lower.

[0141] In summary, to address the issue of training oscillation caused by existing technologies not fully considering the mutual influence between multiple agents during training, the solution of this invention reduces the mutual influence between agents and also takes into account the correlation of global information by dynamically decomposing the gradient of the deep deterministic policy. Secondly, to address the issue of existing technologies not fully considering the storage time of past experience, this paper improves the training effect by fusing temporal difference error and sample timeliness.

[0142] In multi-agent cooperative reinforcement learning, this invention proposes a novel multi-agent learning framework, namely Dynamic Decomposition Deep Deterministic Policy Gradient (DD-MADDPG). It employs a training strategy that maximizes agent optimization using two evaluation networks, considering both global and local agent evaluation strategies to reduce the influence of other agents' policies on the current agent's training. Building upon DD-MADDPG, it integrates a dual-priority experience replay mechanism prioritizing both temporal difference error and sample timeliness, further improving and enhancing training stability and reward.

[0143] Figure 7 This is a schematic diagram of the structure of a multi-agent cooperative reinforcement learning device provided in an embodiment of the present invention. Figure 7 As shown, the device includes:

[0144] The experience selection module 710 is used to construct an experience pool based on the historical action parameters of the multi-agent, select target experience data from the experience pool, and determine the next execution strategy parameters of the multi-agent based on the target experience data and the multi-agent cooperative network.

[0145] The reward data determination module 720 is used to determine global reward data and local reward data based on the next execution strategy parameters of the multi-agent;

[0146] The dual-network parameter update module 730 is used to update the parameters of the multi-agent cooperative network based on the action parameters, policy parameters, global reward data, and local reward data of the multi-agents, using a global evaluation network and a local evaluation network. The global evaluation network is used to optimize the interaction policy parameters of the multi-agents in the multi-agent cooperative network based on the global reward data, and the local evaluation network is used to optimize the policy parameters of a single agent in the multi-agent cooperative network based on the local reward network. The multi-agent cooperative network is used to support agents in coordinating the execution of target tasks.

[0147] The technical solution of this embodiment maximizes the optimization efficiency of agent policies through a dual evaluation network architecture. The global evaluation network is responsible for capturing the macro-level situation of multi-agent interactions, while the local evaluation network focuses on policy optimization at the individual behavior level. The two work together to achieve accurate policy updates, thereby improving the collaborative ability of agents.

[0148] Optional, dual network parameter update module, including:

[0149] A global rate of change determination unit is used to determine a centralized action value function and, based on the centralized action value function, to determine the global rate of change of the global reward corresponding to the change of actions of multiple agents.

[0150] The local change rate determination unit is used to determine the individual action value function and, based on the individual action value function, determine the local change rate of the local reward corresponding to the change of the target agent's action.

[0151] The action change information determination unit is used to determine the action change information of the action parameters under the corresponding individual observation when the individual policy parameters of the target agent change.

[0152] A global policy gradient determination unit is used to determine the global policy gradient based on the global rate of change and the action change information.

[0153] A local policy gradient determination unit is used to determine the local policy gradient based on the local rate of change and the action change information.

[0154] A gradient fusion unit is used to determine the gradient parameters of the expected reward of the multi-agent based on the fusion result of the global policy gradient and the local policy gradient.

[0155] The parameter update unit is used to update the parameters of the multi-agent cooperative network according to the direction of change of the gradient parameters.

[0156] Optionally, the centralized action value function updates policy parameters according to the following process:

[0157] The expected total global reward is determined based on the global reward data of the multi-agent at the current moment and the predicted global reward data after executing joint action parameters under joint observation at the next moment.

[0158] The first loss is determined based on the difference between the current global reward data after the multi-agent performs joint action parameters under the joint observation at the current moment and the expected total global reward.

[0159] Update the strategy parameters based on the first loss.

[0160] Optionally, the policy parameters of the individual action value function are updated according to the following process:

[0161] The expected total local reward is determined based on the local reward data of the target agent at the current moment and the predicted local reward data after performing the corresponding action parameters under individual observation at the next moment.

[0162] The second loss is determined based on the difference between the current local reward data after the target agent performs the corresponding action parameters under the individual observation at the current moment and the expected total local reward.

[0163] The policy parameters are updated based on the second loss.

[0164] Optional, gradient fusion unit, specifically used for:

[0165] Determine the global weights corresponding to the global policy gradient and the local weights corresponding to the local policy gradient;

[0166] The gradient parameters of the expected reward of the multi-agent are determined based on the sum of the product of the global policy gradient and the global weights, and the product of the local policy gradient and the local weights.

[0167] The local weights are updated using a gradual decay mechanism, and the sum of the global weights and the local weights is 1.

[0168] Optional, experience selection module, including:

[0169] The temporal difference error determination unit is used to determine the global reward data and global return data of each experience data based on the historical action parameters of the multi-agent, and to determine the temporal difference error of each experience data based on the global reward data and global return data.

[0170] The priority determination unit is used to determine the priority of each empirical data based on the time-series differential error and storage time of each empirical data.

[0171] The target experience data selection unit is used to select target experience data according to the priority of each experience data in the experience pool.

[0172] Optionally, a timing differential error determination unit is used specifically for:

[0173] Determine the global reward data for each empirical data point at the next time step; determine the current global reward data for each empirical data point at the current time step under the joint observation and joint action parameters; determine the next global reward data for each empirical data point at the next time step under the joint observation and joint action parameters; and determine the temporal difference error for each empirical data point based on the global reward data, the current global reward data, and the next global reward data.

[0174] Optional, priority determination unit, specifically used for:

[0175] The impact of storage time on error is determined based on the storage time and influencing factors of each empirical data point.

[0176] The priority of each empirical data point is determined based on the fusion result of the storage time impact error and the timing difference error.

[0177] Optional, the target experience data selection unit is specifically used for:

[0178] The selection probability of each piece of experience data is determined based on the priority of each piece of experience data in the experience pool, and the target experience data is selected based on the selection probability.

[0179] The probability of selecting each empirical data point is determined based on its proportion within the sum of priority control values ​​for all empirical data, determined by the priority of that empirical data point. The priority control value is determined based on the priority of the empirical data point and its priority control parameter.

[0180] The multi-agent cooperative reinforcement learning device provided in the embodiments of the present invention can execute the multi-agent cooperative reinforcement learning method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0181] The acquisition, storage, use, and processing of data in this application comply with relevant national laws and regulations and do not violate public order and good morals.

[0182] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0183] Figure 8A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0184] like Figure 8 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0185] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0186] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods described above, such as multi-agent cooperative reinforcement learning methods.

[0187] In some embodiments, the multi-agent cooperative reinforcement learning method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the multi-agent cooperative reinforcement learning method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the multi-agent cooperative reinforcement learning method by any other suitable means (e.g., by means of firmware).

[0188] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific reference products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0189] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0190] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0191] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0192] The systems and technologies described herein can be implemented in computing systems that include back-end components (e.g., as data servers), or computing systems that include switching components (e.g., application servers), or computing systems that include front-end components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such back-end, switching, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0193] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0194] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0195] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multi-agent cooperative reinforcement learning method as provided in any embodiment of this application.

[0196] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0197] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0198] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multi-agent collaborative reinforcement learning method, characterized in that, The method includes: An experience pool is constructed based on the historical action parameters of the multi-agent, target experience data is selected from the experience pool, and the next execution strategy parameters of the multi-agent are determined based on the target experience data and the multi-agent cooperative network. The global reward data and local reward data are determined based on the next step execution strategy parameters of the multi-agent; The multi-agent cooperative network updates parameters based on the action parameters, policy parameters, global reward data, and local reward data of the multi-agents using a global evaluation network and a local evaluation network. The global evaluation network optimizes the interaction policy parameters of the multi-agents in the multi-agent cooperative network based on the global reward data, while the local evaluation network optimizes the policy parameters of individual agents in the multi-agent cooperative network based on the local reward network. The multi-agent cooperative network supports the agents in collaboratively executing the target task.

2. The method according to claim 1, characterized in that, The parameters of the multi-agent cooperative network are updated based on the action parameters, policy parameters, global reward data, and local reward data of the multi-agents, using both global and local evaluation networks. Determine a centralized action value function, and based on the centralized action value function, determine the global rate of change of the global reward when multiple agents change their actions; Determine the individual action value function, and determine the local rate of change of the local reward when the target agent changes its action based on the individual action value function; Determine the action change information of the action parameters under the corresponding individual observation when the individual policy parameters of the target agent change; The global policy gradient is determined based on the global rate of change and the action change information; The local policy gradient is determined based on the local rate of change and the action change information; The gradient parameters of the expected reward of the multi-agent are determined based on the fusion result of the global policy gradient and the local policy gradient. The parameters of the multi-agent cooperative network are updated according to the direction of change of the gradient parameters.

3. The method according to claim 2, characterized in that, The centralized action value function updates the policy parameters according to the following process: The expected total global reward is determined based on the global reward data of the multi-agent at the current moment and the predicted global reward data after executing joint action parameters under joint observation at the next moment. The first loss is determined based on the difference between the current global reward data after the multi-agent performs joint action parameters under the joint observation at the current moment and the expected total global reward. Update the strategy parameters based on the first loss.

4. The method according to claim 2, characterized in that, The individual action value function updates its policy parameters according to the following process: The expected total local reward is determined based on the local reward data of the target agent at the current moment and the predicted local reward data after performing the corresponding action parameters under individual observation at the next moment. The second loss is determined based on the difference between the current local reward data after the target agent performs the corresponding action parameters under the individual observation at the current moment and the expected total local reward. Update the strategy parameters based on the second loss.

5. The method according to claim 2, characterized in that, The gradient parameters of the expected reward of the multi-agent are determined based on the fusion result of the global policy gradient and the local policy gradient, including: Determine the global weights corresponding to the global policy gradient and the local weights corresponding to the local policy gradient; The gradient parameters of the expected reward of the multi-agent are determined based on the sum of the product of the global policy gradient and the global weights, and the product of the local policy gradient and the local weights. The local weights are updated using a gradual decay mechanism, and the sum of the global weights and the local weights is 1.

6. The method according to claim 1, characterized in that, An experience pool is constructed based on the historical action parameters of multiple agents, and target experience data is selected from the experience pool, including: The global reward data and global return data for each experience data are determined based on the historical action parameters of the multi-agent, and the temporal difference error for each experience data is determined based on the global reward data and global return data. The priority of each empirical data point is determined based on its timing difference error and storage time. Target experience data is selected based on the priority of each experience data in the experience pool.

7. The method according to claim 6, characterized in that, The temporal difference error for each empirical data point is determined based on the global reward data and the global return data, including: Determine the global reward data for each experience data point in the next time step; Determine the current global reward data for each empirical data point at the current moment under the joint observation and joint action parameters; Determine the next global reward data for each empirical data point under the joint observation and joint action parameters at the next time step; The temporal difference error for each empirical data point is determined based on the global reward data, the current global reward data, and the next global reward data.

8. The method according to claim 7, characterized in that, The priority of each empirical data point is determined based on its time-series differential error and storage time, including: The impact of storage time on error is determined based on the storage time and influencing factors of each empirical data point. The priority of each empirical data point is determined based on the fusion result of the storage time impact error and the timing difference error.

9. The method according to claim 8, characterized in that, Target experience data is selected based on the priority of each experience data in the experience pool, including: The selection probability of each piece of experience data is determined based on the priority of each piece of experience data in the experience pool, and the target experience data is selected based on the selection probability. The probability of selecting each piece of empirical data is determined based on the proportion of the priority control value determined by the priority of that empirical data in the sum of the priority control values ​​of all empirical data. The priority control value is determined based on the priority of the empirical data and the priority control parameter.

10. A multi-agent collaborative reinforcement learning device, characterized in that, The device includes: The experience selection module is used to construct an experience pool based on the historical action parameters of the multi-agent, select target experience data from the experience pool, and determine the next execution strategy parameters of the multi-agent based on the target experience data and the multi-agent cooperative network. The reward data determination module is used to determine global reward data and local reward data based on the next step execution strategy parameters of the multi-agent; A dual-network parameter update module is used to update the parameters of a multi-agent cooperative network based on the action parameters, policy parameters, global reward data, and local reward data of the multi-agents, using a global evaluation network and a local evaluation network. The global evaluation network is used to optimize the interaction policy parameters of the multi-agents in the multi-agent cooperative network based on the global reward data, and the local evaluation network is used to optimize the policy parameters of individual agents in the multi-agent cooperative network based on the local reward network. The multi-agent cooperative network supports the agents in collaboratively executing a target task.

11. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the multi-agent cooperative reinforcement learning method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the multi-agent cooperative reinforcement learning method according to any one of claims 1-9.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the multi-agent cooperative reinforcement learning method according to any one of claims 1-9.

Images