A multi-agent reinforcement learning intelligent decision-making method and device

Abstract

This invention provides a multi-agent reinforcement learning intelligent decision-making method and apparatus. The method includes: determining the state vectors of multiple agents within a target problem at the current time step; inputting the state vectors of neighboring agents into a graph attention network included in the algorithm model of the target agent to obtain corresponding influence weights; performing a weighted average of the state vectors of neighboring agents based on the influence weights to obtain the corresponding mean field vector; inputting the state vector and mean field vector of the target agent into an actor network included in the algorithm model of the target agent to obtain the corresponding processing decision for the target agent, thereby controlling the target agent to execute the corresponding action according to the processing decision at the current time step. The method provided by this invention is applicable to large-scale agent intelligent decision-making, significantly improving the efficiency and accuracy of multi-agent reinforcement learning intelligent decision-making, and effectively enhancing the agent decision-making level.

Description

Technical Field

[0001] This invention relates to the field of intelligent decision-making technology, specifically to a multi-agent reinforcement learning intelligent decision-making method and apparatus. Additionally, it relates to an electronic device and a processor-readable storage medium. Background Technology

[0002] Multi-agent reinforcement learning (MAL) is a machine learning algorithm that enables intelligent decision-making in scenarios involving multiple agents, each controlling one agent. In different problems, agents may have complex relationships involving cooperation, competition, and resource sharing. The main challenge lies in the exceptional complexity of decision-making when the number of agents is large, leading to poor efficiency and accuracy in MML. Current technologies address this challenge primarily through two approaches. One is the mean-field method, which calculates an average of the characteristic behaviors of all agents and transforms the pairwise interactions between agents into interactions between that agent and this average, thus reducing computational complexity. While this method simplifies computation, its indiscriminate treatment of all agents in calculating the average loses some information about the problem at hand. Specifically, the interaction strength between different agents varies; for example, spatially adjacent agents often interact more strongly. The performance of this method is limited, especially as the number of agents increases.

[0003] Therefore, designing a multi-agent reinforcement learning scheme that can effectively reduce computational complexity, thus making it suitable for scenarios with a large number of agents, while minimizing information loss in the problem to be solved has become an urgent problem to be solved. Summary of the Invention

[0004] To address this, the present invention provides a multi-agent reinforcement learning intelligent decision-making method and apparatus to overcome the shortcomings of existing multi-agent reinforcement learning intelligent decision-making schemes, which have high limitations and result in poor intelligent decision-making efficiency and accuracy.

[0005] In a first aspect, the present invention provides a multi-agent reinforcement learning intelligent decision-making method, comprising: determining the state vectors of the units in which multiple agents in the target problem reside at the current time step;

[0006] For each agent, the state vectors of neighboring agents associated with the target agent are input into the graph attention network contained in the algorithm model of the target agent. The influence weights between the unit where the target agent is located and the units where its neighboring agents are located at the current time step are obtained. The state vectors of the neighboring agents are weighted and averaged based on the influence weights to obtain the mean field vector corresponding to the target agent.

[0007] The state vector and mean field vector of the target agent at the current time step are input into the actor network contained in the algorithm model of the target agent to obtain the processing decision corresponding to the target agent at the current time step, so as to control the target agent to perform the corresponding action according to the processing decision at the current time step;

[0008] The algorithm model includes an actor network and a graph attention network, both of which are trained based on the sample state and the sample decision results corresponding to the sample state.

[0009] Furthermore, the graph attention network includes an actor key network and an actor query network; the traversal of each agent, inputting the state vectors of neighboring agents associated with the target agent into the graph attention network contained in the algorithm model of the target agent, to obtain the influence weights between the unit where the target agent is located and the units where its neighboring agents are located at the current time step, specifically includes:

[0010] For each agent, the state vector of the target agent currently being traversed is input into the actor key network contained in the algorithm model of the target agent to obtain the actor key vector of the target agent; and the state vector of the unit where the neighboring agent is located is input into the actor query network contained in the algorithm model of the target agent to obtain the actor query vector of the neighboring agent.

[0011] The influence weights of multiple neighboring agents on the target agent are calculated based on the agent key vector and the agent query vector of the target agent. The agent includes the target agent and its neighboring agents, and the target agent is any one of the multiple agents. The magnitude of the influence weight represents the strength of the association between the target agent and its neighboring agents.

[0012] Furthermore, before determining the state vector of each agent's unit in the target problem at the current time step, the method further includes: pre-determining the association relationship of multiple agents in the target problem to be processed, and constructing a corresponding agent relationship graph based on the association relationship using a graph structure method; wherein, each agent controls a unit in the target problem and corresponds to a node in the agent relationship graph; the association relationship includes the spatial distribution relationship and logical association relationship of the agents.

[0013] Furthermore, determining the state vectors of the units where multiple agents in the target problem reside at the current time step specifically includes: obtaining multi-dimensional state information corresponding to the units where each agent resides at the current time step based on the agent relationship graph, and generating corresponding state vectors based on the multi-dimensional state information.

[0014] Furthermore, the influence weights include weight values ​​representing the degree of influence of the units containing the multiple adjacent agents on the unit containing the target agent;

[0015] The step of weighting the state vectors of the neighboring agents based on the influence weights to obtain the mean field vector corresponding to the target agent specifically includes:

[0016] The average field vector corresponding to the target agent is obtained by performing a weighted average process on the state vectors of the units where the neighboring agents are located and their corresponding weight values.

[0017] Furthermore, before traversing each agent, the following is also included:

[0018] Obtain the sample states and their decision results used to train the algorithm model;

[0019] Based on the sample state and its sample decision results, and using the pre-set evaluator network, evaluator query network and evaluator key network in the agent, the actor network and graph attention network included in the algorithm model are trained to adjust the network parameters and obtain an algorithm model that meets the preset conditions.

[0020] Furthermore, the calculation based on the actor key vector and the actor query vector of the target agent to obtain the influence weights of multiple neighboring agents on the target agent specifically includes:

[0021] Calculate the inner product of the target agent's actor key vector and the target agent's actor query vector to obtain the inner product calculation result;

[0022] Based on the inner product calculation results, the influence weights of the multiple neighboring agents on the target agent are determined.

[0023] Secondly, the present invention also provides a multi-agent reinforcement learning intelligent decision-making device, comprising:

[0024] The state vector determination module is used to determine the state vectors of the units containing multiple agents within the target problem at the current time step;

[0025] The mean field vector determination module is used to traverse each agent, input the state vectors of neighboring agents associated with the target agent into the graph attention network contained in the algorithm model of the target agent, obtain the influence weights between the unit where the target agent is located and the units where its neighboring agents are located at the current time step, and perform weighted averaging on the state vectors of the neighboring agents based on the influence weights to obtain the mean field vector corresponding to the target agent.

[0026] The processing decision acquisition module is used to input the state vector and the mean field vector of the target agent at the current time step into the actor network contained in the algorithm model of the target agent to obtain the processing decision corresponding to the target agent at the current time step, so as to control the target agent to perform the corresponding action according to the processing decision at the current time step;

[0027] The algorithm model includes an actor network and a graph attention network, both of which are trained based on the sample state and the sample decision results corresponding to the sample state.

[0028] Furthermore, the graph attention network includes an actor key network and an actor query network; the mean field vector determination module is specifically used for:

[0029] For each agent, the state vector of the target agent currently being traversed is input into the actor key network contained in the algorithm model of the target agent to obtain the actor key vector of the target agent; and the state vector of the unit where the neighboring agent is located is input into the actor query network contained in the algorithm model of the target agent to obtain the actor query vector of the neighboring agent.

[0030] The influence weights of multiple neighboring agents on the target agent are calculated based on the agent key vector and the agent query vector of the target agent. The agent includes the target agent and its neighboring agents, and the target agent is any one of the multiple agents. The magnitude of the influence weight represents the strength of the association between the target agent and its neighboring agents.

[0031] Furthermore, before determining the state vector of each agent's unit within the target problem at the current time step, the system further includes: an agent relationship graph construction module, used to predetermine the association relationships among multiple agents in the target problem to be processed, and construct a corresponding agent relationship graph based on the association relationships using a graph structure method; wherein, each agent controls a unit in the target problem and corresponds to a node in the agent relationship graph; the association relationships include the spatial distribution relationship and logical association relationship of the agents.

[0032] Furthermore, the state vector determination module is specifically used to: obtain multi-dimensional state information corresponding to the unit where each agent is located at the current time step based on the agent relationship graph, and generate corresponding state vectors based on the multi-dimensional state information.

[0033] Furthermore, the influence weights include weight values ​​representing the degree of influence of the units containing the multiple adjacent agents on the unit containing the target agent;

[0034] The mean field vector determination module is specifically used for:

[0035] The average field vector corresponding to the target agent is obtained by performing a weighted average process on the state vectors of the units where the neighboring agents are located and their corresponding weight values.

[0036] Furthermore, before traversing each agent, the following is also included:

[0037] The algorithm model training module is used to obtain the sample states and sample decision results for training the algorithm model; based on the sample states and sample decision results, and using the preset evaluator network, evaluator query network and evaluator key network in the agent, the agent network and graph attention network included in the algorithm model are trained to adjust the network parameters and obtain an algorithm model that meets the preset conditions.

[0038] Furthermore, the calculation based on the actor key vector and the actor query vector of the target agent to obtain the influence weights of multiple neighboring agents on the target agent specifically includes:

[0039] Calculate the inner product of the target agent's actor key vector and the target agent's actor query vector to obtain the inner product calculation result;

[0040] Based on the inner product calculation results, the influence weights of the multiple neighboring agents on the target agent are determined.

[0041] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the multi-agent reinforcement learning intelligent decision-making method as described in any of the above claims.

[0042] Fourthly, the present invention also provides a processor-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the multi-agent reinforcement learning intelligent decision-making method as described in any of the preceding claims.

[0043] The multi-agent reinforcement learning intelligent decision-making method provided by this invention determines the state vectors of multiple agents within the target problem at the current time step, and inputs the state vectors of neighboring agents associated with the target agent into the graph attention network included in the algorithm model of the target agent to obtain influence weights. Based on the influence weights, the state vectors of neighboring agents are weighted and averaged to obtain the corresponding mean field vector. The state vector and mean field vector of the target agent are then input into the actor network included in the algorithm model of the target agent to obtain the corresponding processing decision for the target agent, thereby controlling the target agent to execute the corresponding action according to the processing decision at the current time step. This method is applicable to large-scale agent intelligent decision-making, significantly improving the efficiency and accuracy of multi-agent reinforcement learning intelligent decision-making, and thus effectively improving the agent decision-making level. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the multi-agent reinforcement learning intelligent decision-making method provided in an embodiment of the present invention;

[0046] Figure 2 This is a complete flowchart of the multi-agent reinforcement learning intelligent decision-making method provided in this embodiment of the invention;

[0047] Figure 3 This is a schematic diagram illustrating the mutual influence relationships between intelligent agents provided in an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram of the weight calculation process based on graph attention network provided in an embodiment of the present invention;

[0049] Figure 5This is a schematic diagram illustrating the average field vector calculation process based on graph attention network provided in an embodiment of the present invention;

[0050] Figure 6 This is a schematic diagram of the algorithm model training process provided in the embodiments of the present invention;

[0051] Figure 7 This is a flowchart illustrating an embodiment of the present invention based on a preset algorithm platform.

[0052] Figure 8 This is a schematic diagram of the structure of the multi-agent reinforcement learning intelligent decision-making device provided in an embodiment of the present invention;

[0053] Figure 9 This is a schematic diagram of the physical structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] The following section first describes in detail the embodiments of the multi-agent reinforcement learning intelligent decision-making method described in this invention. For example... Figure 1 As shown, this is a flowchart illustrating the multi-agent reinforcement learning intelligent decision-making method provided in an embodiment of the present invention. The specific implementation process includes the following steps:

[0056] Step 101: Determine the state vectors of the units containing multiple agents within the target problem at the current time step.

[0057] In this embodiment of the invention, the target problem can be a decision-making problem involving a large number of intelligent agents with a large number of units. For example, in the traffic light control problem, each intersection can be considered as a unit corresponding to an intelligent agent, and there are hundreds or thousands of intelligent agents in the entire city; in the ride-hailing dispatch problem, each ride-hailing vehicle can be considered as a unit corresponding to an intelligent agent, and there are hundreds or thousands of vehicles under a platform; in the resource allocation problem in a city, each block can be considered as a unit corresponding to an intelligent agent, and there are hundreds or thousands of blocks in a city, and so on.

[0058] Before performing this step, it is necessary to pre-determine the relationships between multiple agents in the target problem to be processed. These relationships include the spatial distribution and logical connections between the agents. A graph model can be used to model these relationships, resulting in a corresponding agent relationship graph. Each agent controls a unit within the target problem and corresponds to a node in the agent relationship graph. During the specific implementation of this step, based on the agent relationship graph, multi-dimensional state information corresponding to the unit where each agent is located at the current time step can be obtained, and a corresponding state vector can be generated based on this multi-dimensional state information.

[0059] Specifically, in real-world problems, when the number of agents is large, the relationships between them are complex. Some agents may have cooperative or competitive relationships, while others may not have obvious mutual influence. This invention uses a graph model to model the mutual influence relationships between agents, such as... Figure 3 As shown, when considering a target problem involving a total of N units, there are N agents, each controlling one unit. Therefore, a graph G containing N nodes is constructed (i.e., as shown in the diagram). Figure 3 The diagram shows the agent relationship, where 301 is one of the agents, and each agent has a corresponding state vector 302 and action vector 303. The edge weight between the i-th node and the j-th node is e. ij The edge weight is either a mask variable or a switch variable. When there is a mutual influence relationship between two agents, the corresponding edge weight is 1. When the edge weight is 1, e ij w ij =w ij Otherwise, the corresponding edge weight is 0. When the edge weight is 0, e ij w ij =0. For the i-th agent (e.g., the target agent), its neighboring agents That is:

[0060]

[0061] In practice, agents that typically have mutual influence relationships include agents that are spatially adjacent and agents that are logically related in terms of tasks.

[0062] Step 102: Traverse each agent and input the state vectors of neighboring agents associated with the target agent into the graph attention network contained in the algorithm model of the target agent to obtain the influence weights between the unit where the target agent is located and the units where its neighboring agents are located at the current time step. Based on the influence weights, perform a weighted average processing on the state vectors of the neighboring agents to obtain the mean field vector corresponding to the target agent.

[0063] like Figure 5 As shown, during the algorithm model training process, the graph attention network 401 may include the actor key network and actor query network corresponding to the actor (i.e., the actor network), and the evaluator key network and evaluator query network corresponding to the evaluator (i.e., the evaluator network). However, in the actual deployment and application of this application, only the actor network obtained during training and its corresponding actor key network and actor query network are needed; no other networks are required, and gradient descent updates of network parameters are no longer involved. That is, by utilizing the graph attention network, including the actor key network and actor query network, the mean field vector of the state can be obtained, so that the mean field vector of the state can be subsequently input into the actor network to obtain the corresponding processing decision.

[0064] like Figure 5As shown, in this step, the process of traversing each agent and inputting the state vector 305 of the neighboring agents 304 associated with the target agent 301 into the graph attention network 401 included in the algorithm model of the target agent to obtain the influence weight between the unit where the target agent is located and the units where its neighboring agents are located at the current time step can be implemented as follows: traversing each agent and inputting the state vector of the target agent currently being traversed into the actor key network included in the algorithm model of the target agent to obtain the actor key vector of the target agent; and inputting the state vector of the target agent currently being traversed into the graph attention network 401 included in the algorithm model of the target agent to obtain the actor key vector of the target agent; and inputting the state vector of the target agent ... The state vector of the unit containing the neighboring agent is input into the actor query network included in the algorithm model of the target agent to obtain the actor query vector of the neighboring agent. Based on the actor key vector and the actor query vector of the target agent, the influence weights of multiple neighboring agents on the target agent are calculated. The agent includes the target agent and its neighboring agents, and the target agent is any one of the multiple agents. The magnitude of the influence weight represents the strength of the association between the traversed target agent and its neighboring agents. Specifically, the process of calculating the influence weights of multiple neighboring agents on the target agent based on the actor key vector and the actor query vector of the target agent includes: calculating the inner product of the actor key vector and the actor query vector of the target agent to obtain the inner product calculation result; and determining the influence weights of multiple neighboring agents on the target agent based on the inner product calculation result. The specific implementation process of obtaining the average field vector corresponding to the target agent by weighted averaging the state vectors of the neighboring agents based on the influence weights may include: weighted averaging the state vectors of the units where the neighboring agents are located and their corresponding weight values ​​to obtain the average field vector corresponding to the target agent.

[0065] This application utilizes a graph attention mechanism to calculate the strength of the interaction between each pair of agents. Then, using this strength as a weight (i.e., influence weight), it performs a weighted average of the state vectors 302 and action vectors 303 corresponding to all agents to obtain an average field vector. Finally, each agent makes a decision based on this average field vector to obtain its action, thereby reducing computational complexity and increasing the scale of agents. The specific method is described below:

[0066] like Figure 2 As shown, the definition of the large-scale intelligent decision-making problem solved by this invention is given first: Let the total number of units in the target problem be N, and each unit be controlled by an agent. The decision problem involves time steps of length T, and the state of each unit at the t-th time step is:

[0067] (S (1) (t),S (2) (t),…S (N) (t))

[0068] Each of S (i) (t) represents an s-dimensional vector describing the various state characteristics of the unit at that time step, where s is an integer greater than or equal to 1. On the other hand, the decision problem refers to making a decision for each unit at each time step. The decision for each unit at the t-th time step is:

[0069] (A (1) (t),A (2) (t),…A (N) (t))

[0070] Each A (i) (t) are all a-dimensional vectors, describing the actions of the agent controlling the unit at this time step, where a is an integer greater than or equal to 1.

[0071] During the algorithm model training phase, each agent comprises an actor network and an evaluator network. Both the actor network and the evaluator network are composed of neural networks with learnable parameters. The evaluator network is used for auxiliary parameter tuning during the algorithm model training process. For the i-th agent, the actor network takes the state vector of the unit where the agent (e.g., the target agent) is located and the average field of the states of the units where other agents are located as inputs, and outputs a decision vector. The other agents are neighboring agents that have an influence relationship with the target agent. Its functional form is denoted as:

[0072]

[0073] in Let π be the mean field vector of the state. i For an actor network, this is the function that maps states to actions. The calculation method is as follows:

[0074]

[0075] in For agent i, let w be a neighboring agent. ij Let be the influence weight between agents i and j, and:

[0076]

[0077] The sum of weights is used for normalization. Similarly, the critic network takes the state of the cell in which the agent (e.g., the target agent) is located, the average field of the states of other agents' cells, and the average field of the target agent's decisions and the decisions of other agents as inputs. It outputs an evaluation of the quality of the agent's decisions, where the other agents are neighboring agents that have an influence relationship with the target agent. Its functional form is denoted as:

[0078]

[0079] in These are the mean field vectors of the state and the mean field vectors of the action, respectively, and are calculated as follows:

[0080]

[0081] Where u ij Let be the influence weight between agents i and j, and:

[0082]

[0083] The sum of weights is used for normalization.

[0084] In the above process of calculating the influence weights based on the graph attention mechanism, the mean field vector of the calculated actor network is... And calculate the average field vector of the evaluator network. and In essence, this involves calculating the influence weights w among multiple agents. ij and u ij In this invention, based on the agent relationship graph constructed above, graph attention mechanism is used to calculate the two sets of influence weights. For example... Figure 4 As shown, each agent includes a pair of actor query networks (309 and 310) and a pair of actor key networks (311 and 312), and a pair of evaluator query networks (306 and 307) and a pair of evaluator key networks (308 and 313). Each network is a parameter-learnable neural network. 301 is the state vector of the target agent, and 305 is the state vector of the neighboring agents. The actor query network is the query network of the actor, the actor key network is the key network of the actor, the evaluator query network is the query network of the evaluator, and the evaluator key network is the key network of the evaluator.

[0085] At a certain time step, it is necessary to calculate the weight w between the i-th agent and the j-th agent. ij and u ijThe steps are as follows: Input the state vector of the unit where the i-th agent is located into the actor key network and the evaluator key network respectively to obtain the actor key vector and evaluator key vector of the i-th agent respectively; Input the state vector of the unit where the j-th agent is located into the actor query network and the evaluator query network respectively to obtain the actor query vector and evaluator query vector of the j-th agent respectively; Calculate the inner product of the actor key vector of the i-th agent and the actor query vector of the j-th agent to obtain the influence weight w. ij Calculate the inner product of the evaluator key vector of the i-th agent and the evaluator query vector of the j-th agent to obtain the influence weight u. ij By iterating through each agent and its combinations with all neighboring agents, all influence weights can be obtained and the corresponding mean field vector can be calculated. For example, a target problem includes three neighboring agents A, B, and C that have influence relationships; when the i-th agent is agent A, the j-th agent can be agent B or agent C; where, when the j-th agent is agent B, the influence weight w ij The influence weight w is represented by the state of agent B on agent A. When the j-th agent is agent C, the influence weight w is... ij Let w represent the influence strength of agent C's state on agent A; when the i-th agent is agent B, the j-th agent can be either agent A or agent C; where, when the j-th agent is agent A, the influence weight w ij The influence weight w is represented by the state of agent A on agent B. When the j-th agent is agent C, the influence weight w is... ij Let w represent the influence strength of agent C's state on agent B; when the i-th agent is agent C, the j-th agent can be either agent A or agent B; where, when the j-th agent is agent A, the influence weight w ij The influence weight w is represented by the state of agent A on agent C. When the j-th agent is agent B, the influence weight w is... ij This represents the strength of the influence of agent B's state on agent C.

[0086] It should be noted that, in this invention, before traversing each agent, sample states and their decision results for training the algorithm model can be obtained in advance. Based on these sample states and decision results, and utilizing the pre-defined evaluator network, evaluator query network, and evaluator key network within the agent, the actor network and graph attention network included in the algorithm model are trained to adjust network parameters and obtain an algorithm model that meets preset conditions. The algorithm model continuously collects experience in its interaction with the environment, improves its processing decisions, and ultimately yields a mature and high-performance algorithm model.

[0087] In algorithm model training, to ensure smooth updates of neural network parameters, an online network and a target network design can be adopted. All networks within each agent—namely, the actor network, the evaluator network, a pair of actor query networks and a pair of actor key networks, as well as a pair of evaluator query networks and a pair of evaluator key networks—each contain corresponding online and target networks. The specific training steps are as follows: Figure 6 The steps are as follows: First, randomly initialize the parameters of each online network and the memory space of the experience cache pool. Considering the homogeneity and similarity among agents, these network parameters are shared among all agents. Second, copy each online network parameter to obtain the corresponding target network. Third, initialize the environment corresponding to the problem to be solved. Fourth, obtain the state of each agent's unit from the environment; calculate the mean field vector using a graph attention mechanism. Fifth, input the state of each agent and the mean field vector into each online actor network, add micro-perturbations to obtain the current policy. Sixth, execute the obtained policy, interact with the environment, and add a sample to the experience cache pool. Seventh, when the number of experiences in the experience cache pool reaches a preset value, randomly select a small batch from the experience cache pool. The sample is used to calculate the mean field vector using graph attention mechanism. Reference values ​​for long-term returns are calculated by each target evaluator network, and estimated values ​​for long-term returns are calculated by each online evaluator network. Step 8: If not, jump back to step 4. Step 9: Minimize the long-term return estimation error using gradient descent, updating the parameters of each online network one by one. Step 10: Maximize the long-term return estimate using the policy gradient method, updating the parameters of each online network one by one. Step 11: Slowly update the parameter values ​​of each target network one by one by using soft replacement between each group of target networks and online networks. Step 12: When the preset number of training iterations is reached, training is complete, and the algorithm model is obtained; otherwise, jump back to step 3. Step 13: When the model training is complete and applied to a practical problem, the evaluator network can be removed, retaining only the actor network.

[0088] Step 103: Input the state vector and mean field vector of the target agent at the current time step into the actor network included in the algorithm model of the target agent to obtain the processing decision corresponding to the target agent at the current time step, so as to control the target agent to perform the corresponding action according to the processing decision at the current time step. The actor network and graph attention network included in the algorithm model are both trained based on the sample state and the sample decision results corresponding to the sample state.

[0089] For example, if the target problem corresponds to four adjacent agents A, B, C, and D, and A is the target agent, then the graph attention network can calculate the influence weights based on the state vectors of agents B, C, and D, and perform a weighted average to obtain the average field vector of the influence of agents B, C, and D on agent A. In this step, the state vector of agent A and the average field vector of the influence of agents B, C, and D on agent A can be input into the actor network included in the algorithm model of the target agent to obtain the processing decision corresponding to the target agent at the current time step.

[0090] All embodiments of the present invention are based on a preset algorithm platform (i.e., a simulation algorithm platform) for the target problem to be solved, such as... Figure 7 As shown. For a specific target problem to be solved, a corresponding preset algorithm platform can be designed. This preset algorithm platform contains the dynamic characteristics, constraints, evolution laws, etc. of the target problem, and sometimes it can also incorporate other collected real data.

[0091] In a specific implementation process, if the objective problem is to solve the resource allocation problem in event G occurring in city F, the specific scenarios are shown in the table below:

[0092] area F City event G Incident population 7191638 Number of regions (units) 3130 Number of points of interest 39411 Resource quantity 71,916 doses / day Duration 1512 time steps

[0093] First, a simulation environment for the propagation of event G is constructed. Data on the population structure characteristics, spatial distribution, and mobility characteristics of city F are input into a pre-defined algorithm platform to build the corresponding simulation environment. Simultaneously, the actual situation in the simulation environment is initialized, and subsequent steps are implemented based on this simulation environment. Next, each region is treated as a unit, and each unit is controlled by an agent, totaling 3130 agents. An agent relationship graph is established based on the population flow characteristics in city F. Specifically, if the frequency of population flow between two regions exceeds a certain threshold, the edge weight between the corresponding nodes in these two regions is 1, making them neighboring regions (i.e., adjacent units) in the agent relationship graph; conversely, if the frequency of population flow between two regions is less than this threshold, the edge weight between the corresponding nodes in these two regions is 0, and they are not neighboring regions in the agent relationship graph.

[0094] The next step is the algorithm model training process. The actor network and evaluator network are initialized, as are the key network (including the actor key network and the evaluator key network) and the query network (including the actor query network and the evaluator query network). The algorithm model is then trained. During training, the following steps are executed iteratively: The community infection state and changes in community infection for each region (i.e., unit) at the current time step are obtained from the simulation environment, and these values ​​are used as the state vectors of the corresponding regions. The mean field vector is calculated using the graph attention mechanism, and the influence weights between all regions and their neighboring regions at the current time step are calculated. The mean field vectors of all neighboring regions of each region are then calculated using these influence weights. The state vector and mean field vector of the current time step are input into the reinforcement learning actor to obtain the resource allocation decision between regions at the current time step. The representation vector of the current time step is input into the reinforcement learning evaluator network to obtain an evaluation of the quality of the state at that time step under the current decision. This evaluation, combined with the reward function value, is used to update the gradient of the reinforcement learning actor network and the evaluator network. Training ends when the number of executions reaches the preset limit or the performance of the obtained algorithm model meets the requirements, and an algorithm model containing the trained network parameters is obtained.

[0095] The next step is deployment and application. During deployment and application, only the actor network obtained during training, along with its corresponding actor key network and actor query network, are needed. No other networks are required, and gradient descent updates of network parameters are no longer involved. The following steps are executed iteratively during deployment and application: The community infection status and changes in community infection at the current time step for each region are obtained from the simulation environment, and these values ​​are used as the state vector for the corresponding region. Based on the graph attention mechanism described above, the average field vector is calculated using the trained actor key network and actor query network to obtain the weights between all regions and their neighboring regions at the current time step. The average field vector of all neighboring regions for each region is then calculated using these weights. The state vector and average field vector of the current time step are input into the trained actor network to obtain the resource allocation decision (i.e., processing decision) between regions at the current time step. Deployment and application end when the number of executions reaches a preset upper limit, yielding the resource allocation result (i.e., the corresponding action executed) under the method decision of this invention.

[0096] In a specific implementation, the user wants to use this method to solve the resource allocation problem in event X that occurs in city S. The specific situation is shown in the table below:

[0097] area S City type X Event population 6635035 Number of regions 3555 Number of points of interest 40964 Resource quantity 66,350 doses / day Duration 1512 time steps

[0098] First, a simulation environment for the propagation of event X is constructed. Data on the population structure characteristics, spatial distribution, and mobility characteristics of city S are input into a pre-defined algorithm platform to build the corresponding simulation environment. Simultaneously, the infection situation in the simulation environment is initialized, and subsequent steps are implemented based on this environment. Next, each region is treated as a unit, and each unit is controlled by an agent, totaling 3555 agents. An agent relationship graph is established based on the population flow characteristics in city S. Specifically, if the frequency of population flow between two regions exceeds a certain threshold, the edge weight between the corresponding nodes in these two regions is 1, making them neighboring regions in the agent relationship graph; conversely, if the frequency of population flow between two regions is less than the threshold, the edge weight between the corresponding nodes in these two regions is 0, and they are not neighboring regions in the agent relationship graph.

[0099] The next step is the algorithm model training process. The actor network and evaluator network are initialized, as are the key network (including the actor key network and the evaluator key network) and the query network (including the actor query network and the evaluator query network). The following steps are executed iteratively during training: The community infection state and changes in community infection at the current time step for each region are obtained from the simulation environment, and these values ​​are used as the state vectors of the corresponding regions; The influence weights between all regions and their neighboring regions at the current time step are calculated using the graph attention average field method, and the average field vector of all neighboring regions of each region is calculated using these influence weights; The state vector and average field vector of the current time step are input into the reinforcement learning actor network to obtain the resource allocation decision between regions at the current time step; The representation vector of the current time step is input into the reinforcement learning evaluator network to obtain the evaluation of the quality of the state at that time step under the current decision, and the gradient is used to update the reinforcement learning actor network and evaluator network by combining this evaluation with the reward function value; Training ends when the number of executions reaches a preset limit or the performance of the obtained algorithm model meets the requirements, resulting in an algorithm model containing the trained network parameters.

[0100] The next step is the deployment and application process. During deployment and application, only the actors obtained during training and their corresponding actor key networks and actor query networks are needed; no other networks are required, and gradient descent updates of network parameters are no longer involved. The deployment and application process iteratively executes the following steps: Obtain the community infection status and changes in community infection for each region at the current time step from the simulation environment, and use these values ​​as the state vector of the corresponding region; Based on the graph attention average field method described above, use the trained actor key network and actor query network to calculate the weights between all regions and their neighboring regions at the current time step, and use the obtained weights to calculate the average field vector of all neighboring regions for each region; Input the state vector and average field vector of the current time step into the trained actor network to obtain the resource allocation decision (i.e., processing decision) between regions at the current time step; When the number of executions reaches a preset upper limit, the deployment and application process ends, and the resource allocation result (i.e., the corresponding action executed) under the method decision of this invention is obtained.

[0101] The multi-agent reinforcement learning intelligent decision-making method described in this invention determines the state vectors of multiple agents within the target problem at the current time step, and inputs the state vectors of neighboring agents associated with the target agent into the graph attention network included in the algorithm model of the target agent to obtain influence weights. Based on the influence weights, the state vectors of neighboring agents are weighted and averaged to obtain the corresponding mean field vector. The state vector and mean field vector of the target agent are then input into the actor network included in the algorithm model of the target agent to obtain the corresponding processing decision for the target agent, thereby controlling the target agent to execute the corresponding action according to the processing decision at the current time step. This method is applicable to large-scale agent intelligent decision-making, significantly improving the efficiency and accuracy of multi-agent reinforcement learning intelligent decision-making, and thus effectively improving the agent decision-making level.

[0102] Corresponding to the multi-agent reinforcement learning intelligent decision-making method provided above, this invention also provides a multi-agent reinforcement learning intelligent decision-making device. Since the embodiments of this device are similar to the method embodiments described above, the description is relatively simple. For relevant details, please refer to the description in the method embodiment section above. The embodiments of the multi-agent reinforcement learning intelligent decision-making device described below are merely illustrative. Please refer to... Figure 8 The diagram shown is a structural schematic of a multi-agent reinforcement learning intelligent decision-making device provided in an embodiment of the present invention. The multi-agent reinforcement learning intelligent decision-making device of the present invention specifically includes the following parts:

[0103] The state vector determination module 801 is used to determine the state vectors of the units containing multiple agents within the target problem at the current time step.

[0104] The mean field vector determination module 802 is used to traverse each agent, input the state vectors of neighboring agents associated with the target agent into the graph attention network contained in the algorithm model of the target agent, obtain the influence weights between the unit where the target agent is located and the units where its neighboring agents are located at the current time step, and perform weighted averaging on the state vectors of the neighboring agents based on the influence weights to obtain the mean field vector corresponding to the target agent.

[0105] The processing decision acquisition module 803 is used to input the state vector and the mean field vector of the target agent at the current time step into the actor network contained in the algorithm model of the target agent to obtain the processing decision corresponding to the target agent at the current time step, so as to control the target agent to perform the corresponding action according to the processing decision at the current time step;

[0106] The algorithm model includes an actor network and a graph attention network, both of which are trained based on the sample state and the sample decision results corresponding to the sample state.

[0107] Furthermore, the graph attention network includes an actor key network and an actor query network; the mean field vector determination module is specifically used for:

[0108] For each agent, the state vector of the target agent currently being traversed is input into the actor key network contained in the algorithm model of the target agent to obtain the actor key vector of the target agent; and the state vector of the unit where the neighboring agent is located is input into the actor query network contained in the algorithm model of the target agent to obtain the actor query vector of the neighboring agent.

[0109] The influence weights of multiple neighboring agents on the target agent are calculated based on the agent key vector and the agent query vector of the target agent. The agent includes the target agent and its neighboring agents, and the target agent is any one of the multiple agents. The magnitude of the influence weight represents the strength of the association between the target agent and its neighboring agents.

[0110] Furthermore, before determining the state vector of each agent's unit within the target problem at the current time step, the system further includes: an agent relationship graph construction module, used to predetermine the association relationships among multiple agents in the target problem to be processed, and construct a corresponding agent relationship graph based on the association relationships using a graph structure method; wherein, each agent controls a unit in the target problem and corresponds to a node in the agent relationship graph; the association relationships include the spatial distribution relationship and logical association relationship of the agents.

[0111] Furthermore, the state vector determination module is specifically used to: obtain multi-dimensional state information corresponding to the unit where each agent is located at the current time step based on the agent relationship graph, and generate corresponding state vectors based on the multi-dimensional state information.

[0112] Furthermore, the influence weights include weight values ​​representing the degree of influence of the units containing the multiple adjacent agents on the unit containing the target agent;

[0113] The mean field vector determination module is specifically used for:

[0114] The average field vector corresponding to the target agent is obtained by performing a weighted average process on the state vectors of the units where the neighboring agents are located and their corresponding weight values.

[0115] Furthermore, before traversing each agent, the following is also included:

[0116] The algorithm model training module is used to obtain the sample states and sample decision results for training the algorithm model; based on the sample states and sample decision results, and using the preset evaluator network, evaluator query network and evaluator key network in the agent, the agent network and graph attention network included in the algorithm model are trained to adjust the network parameters and obtain an algorithm model that meets the preset conditions.

[0117] Furthermore, the calculation based on the actor key vector and the actor query vector of the target agent to obtain the influence weights of multiple neighboring agents on the target agent specifically includes:

[0118] Calculate the inner product of the target agent's actor key vector and the target agent's actor query vector to obtain the inner product calculation result;

[0119] Based on the inner product calculation results, the influence weights of the multiple neighboring agents on the target agent are determined.

[0120] The multi-agent reinforcement learning intelligent decision-making device described in this invention determines the state vectors of multiple agents within the target problem at the current time step, and inputs the state vectors of neighboring agents associated with the target agent into the graph attention network included in the algorithm model of the target agent to obtain influence weights. Based on the influence weights, the state vectors of neighboring agents are weighted and averaged to obtain the corresponding mean field vector. The state vector and mean field vector of the target agent are then input into the actor network included in the algorithm model of the target agent to obtain the corresponding processing decision for the target agent, thereby controlling the target agent to execute the corresponding action according to the processing decision at the current time step. It is applicable to large-scale agent intelligent decision-making, significantly improving the efficiency and accuracy of multi-agent reinforcement learning intelligent decision-making, and thus effectively improving the agent decision-making level.

[0121] Corresponding to the multi-agent reinforcement learning intelligent decision-making method provided above, this invention also provides an electronic device. Since the embodiments of this electronic device are similar to the method embodiments described above, the description is relatively simple. For relevant details, please refer to the description in the method embodiment section above. The electronic device described below is merely illustrative. Figure 9 The diagram shown is a schematic representation of the physical structure of an electronic device disclosed in an embodiment of the present invention. The electronic device may include a processor 901, a memory 902, and a communication bus 903. The processor 901 and the memory 902 communicate with each other via the communication bus 903 and communicate with external devices via a communication interface 904. The processor 901 can call logical instructions in the memory 902 to execute a multi-agent reinforcement learning intelligent decision-making method. This method includes: determining the state vectors of multiple agents within the target problem at the current time step; traversing each agent, inputting the state vectors of neighboring agents associated with the target agent into the graph attention network included in the algorithm model of the target agent, obtaining the influence weights between the target agent's unit and its neighboring units at the current time step, and performing a weighted average of the state vectors of the neighboring agents based on the influence weights to obtain the mean field vector corresponding to the target agent; inputting the target agent's state vector and the mean field vector at the current time step into the actor network included in the algorithm model of the target agent, obtaining the processing decision corresponding to the target agent at the current time step, so as to control the target agent to perform corresponding actions according to the processing decision at the current time step; wherein the actor network and graph attention network included in the algorithm model are both trained based on sample states and the sample decision results corresponding to the sample states.

[0122] Furthermore, the logical instructions in the aforementioned memory 902 can be implemented as software functional modules and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as memory chips, USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] On the other hand, embodiments of the present invention also provide a computer program product, the computer program product including a computer program stored on a processor-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the multi-agent reinforcement learning intelligent decision-making method provided in the above-described method embodiments. The method includes: determining the state vectors of multiple agents within the target problem at the current time step; traversing each agent, inputting the state vectors of neighboring agents associated with the target agent into the graph attention network included in the algorithm model of the target agent, obtaining the influence weights between the target agent's unit and its neighboring agent units at the current time step, and performing a weighted average of the state vectors of the neighboring agents based on the influence weights to obtain the mean field vector corresponding to the target agent; inputting the target agent's state vector and the mean field vector at the current time step into the actor network included in the algorithm model of the target agent, obtaining the processing decision corresponding to the target agent at the current time step, so as to control the target agent to perform corresponding actions according to the processing decision at the current time step; wherein, the actor network and graph attention network included in the algorithm model are both trained based on sample states and the sample decision results corresponding to the sample states.

[0124] In another aspect, embodiments of the present invention also provide a processor-readable storage medium storing a computer program, which, when executed by a processor, implements the multi-agent reinforcement learning intelligent decision-making method provided in the above embodiments. The method includes: determining the state vectors of multiple agents within a target problem at the current time step; traversing each agent, inputting the state vectors of neighboring agents associated with the target agent into a graph attention network included in the algorithm model of the target agent, obtaining the influence weights between the target agent's unit and its neighboring agent units at the current time step, and performing a weighted average processing on the state vectors of the neighboring agents based on the influence weights to obtain the average field vector corresponding to the target agent; inputting the target agent's state vector and the average field vector at the current time step into an actor network included in the algorithm model of the target agent, obtaining a processing decision corresponding to the target agent at the current time step, so as to control the target agent to perform corresponding actions according to the processing decision at the current time step; wherein the actor network and graph attention network included in the algorithm model are both trained based on sample states and the sample decision results corresponding to the sample states.

[0125] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0126] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-agent reinforcement learning intelligent decision-making method, characterized in that, include: Determine the state vectors of the units containing multiple agents within the target problem at the current time step; For each agent, the state vectors of neighboring agents associated with the target agent are input into the graph attention network contained in the algorithm model of the target agent. The influence weights between the unit where the target agent is located and the units where its neighboring agents are located at the current time step are obtained. The state vectors of the neighboring agents are weighted and averaged based on the influence weights to obtain the mean field vector corresponding to the target agent. The state vector and mean field vector of the target agent at the current time step are input into the actor network contained in the algorithm model of the target agent to obtain the processing decision corresponding to the target agent at the current time step, so as to control the target agent to perform the corresponding action according to the processing decision at the current time step; The algorithm model includes an actor network and a graph attention network, both of which are trained based on the sample state and the sample decision results corresponding to the sample state. The influence weights include weight values ​​representing the degree of influence of the units containing the multiple adjacent agents on the unit containing the target agent; The step of weighting the state vectors of the neighboring agents based on the influence weights to obtain the mean field vector corresponding to the target agent specifically includes: The average field vector corresponding to the target agent is obtained by performing a weighted average process on the state vectors of the units where the neighboring agents are located and their corresponding weight values.

2. The multi-agent reinforcement learning intelligent decision-making method according to claim 1, characterized in that, The graph attention network includes an actor key network and an actor query network; The process of traversing each agent involves inputting the state vectors of neighboring agents associated with the target agent into the graph attention network contained in the algorithm model of the target agent, thereby obtaining the influence weights between the unit where the target agent is located at the current time step and the units where its neighboring agents are located. Specifically, this includes: For each agent, the state vector of the target agent currently being traversed is input into the actor key network contained in the algorithm model of the target agent to obtain the actor key vector of the target agent; and the state vector of the unit where the neighboring agent is located is input into the actor query network contained in the algorithm model of the target agent to obtain the actor query vector of the neighboring agent. The influence weights of multiple neighboring agents on the target agent are calculated based on the agent key vector and the agent query vector of the target agent. The agent includes the target agent and its neighboring agents, and the target agent is any one of the multiple agents. The magnitude of the influence weight represents the strength of the association between the target agent and its neighboring agents.

3. The multi-agent reinforcement learning intelligent decision-making method according to claim 1, characterized in that, Before determining the state vector of each agent in the target problem at the current time step, the method further includes: pre-determining the relationship between multiple agents in the target problem to be processed, and constructing a corresponding agent relationship graph based on the relationship using a graph structure; wherein each agent controls a unit in the target problem and corresponds to a node in the agent relationship graph; the relationship includes the spatial distribution relationship and logical relationship of the agents.

4. The multi-agent reinforcement learning intelligent decision-making method according to claim 3, characterized in that, The step of determining the state vectors of the units where multiple agents in the target problem are located at the current time step specifically includes: obtaining multi-dimensional state information corresponding to the units where each agent is located at the current time step based on the agent relationship graph, and generating corresponding state vectors based on the multi-dimensional state information.

5. The multi-agent reinforcement learning intelligent decision-making method according to claim 1, characterized in that, Before iterating through each agent, the process also includes: Obtain the sample states and their decision results used to train the algorithm model; Based on the sample state and its sample decision results, and using the pre-set evaluator network, evaluator query network and evaluator key network in the agent, the actor network and graph attention network included in the algorithm model are trained to adjust the network parameters and obtain an algorithm model that meets the preset conditions.

6. The multi-agent reinforcement learning intelligent decision-making method according to claim 2, characterized in that, The process of calculating the influence weights of multiple neighboring agents on the target agent based on the agent key vector and the agent query vector of the target agent specifically includes: Calculate the inner product of the target agent's actor key vector and the target agent's actor query vector to obtain the inner product calculation result; Based on the inner product calculation results, the influence weights of the multiple neighboring agents on the target agent are determined.

7. A multi-agent reinforcement learning intelligent decision-making device, characterized in that, include: The state vector determination module is used to determine the state vectors of the units containing multiple agents within the target problem at the current time step; The mean field vector determination module is used to traverse each agent, input the state vectors of neighboring agents associated with the target agent into the graph attention network contained in the algorithm model of the target agent, obtain the influence weights between the unit where the target agent is located and the units where its neighboring agents are located at the current time step, and perform weighted averaging on the state vectors of the neighboring agents based on the influence weights to obtain the mean field vector corresponding to the target agent. The processing decision acquisition module is used to input the state vector and the mean field vector of the target agent at the current time step into the actor network contained in the algorithm model of the target agent to obtain the processing decision corresponding to the target agent at the current time step, so as to control the target agent to perform the corresponding action according to the processing decision at the current time step; The algorithm model includes an actor network and a graph attention network, both of which are trained based on the sample state and the sample decision results corresponding to the sample state. The influence weights include weight values ​​representing the degree of influence of the units containing the multiple adjacent agents on the unit containing the target agent; The mean field vector determination module is specifically used for: The average field vector corresponding to the target agent is obtained by performing a weighted average process on the state vectors of the units where the neighboring agents are located and their corresponding weight values.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the multi-agent reinforcement learning intelligent decision-making method as described in any one of claims 1 to 6.

9. A processor-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-agent reinforcement learning intelligent decision-making method as described in any one of claims 1 to 6.

Images