A multi-agent interaction method and system based on group game

Abstract

The present invention discloses a multi-agent interaction method and system based on group game, the method comprising: initializing preset parameters required for agent reinforcement learning, initializing the Q table of the agent, and initializing the learning cycle; obtaining the solution threshold of the ordinary differential equation, and attenuating the solution threshold of the ordinary differential equation; initializing the system state, adopting a greedy strategy to select the joint action of the agent; each agent executes the selected action, and receives observation samples and rewards; constructing a group game based on the Q table of the agent, and solving the approximate equilibrium strategy using the inherited BNN group game dynamic algorithm; the agent updates the Q table based on the expected reward obtained. The present invention does not need to accurately solve the equilibrium strategy of each stage of the game, and obtains the approximate equilibrium with relatively low computational complexity. By designing an acceleration mechanism for solving ordinary differential equations, the optimal trade-off between equilibrium complexity and equilibrium accuracy, as well as the optimal performance of the multi-agent system, is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer application technology, and in particular to a multi-agent interaction method and system based on group game and a computer-readable storage medium. Background Art

[0002] Many real-world problems (such as multi-robot control, drone swarms, and autonomous driving) involve complex interactions between multiple agents. Multiple agents hope to learn the optimal strategy in a complex environment during the interaction process. Non-cooperative multi-agent reinforcement learning generally refers to a scenario in which agents only optimize their own goals during the interaction with the environment. In this scenario, each agent seeks to maximize its own reward. How to enable agents to learn an equilibrium strategy without motivation to change their current strategy is a challenging research topic in the current field of multi-agent reinforcement learning. Existing solutions usually use an equilibrium-based multi-agent reinforcement learning framework to encourage multi-agents to adopt equilibrium strategies in each stage of the game. This learning framework is based on Q-learning (Q-learning, a model-independent reinforcement learning algorithm based on the Markov decision process). First, the interaction between multiple agents is modeled as a random game, and then the equilibrium strategy of the corresponding regular game is calculated at each state of the random game, and the Q function of the agent (a function used to characterize the degree of goodness of the agent state-action combination) is updated according to the selected equilibrium strategy.

[0003] Although the above-mentioned equilibrium-based learning framework can theoretically promote the convergence of intelligent agents to equilibrium strategies, it also has certain shortcomings, that is, some multi-agent reinforcement learning algorithms based on game theory often use precise equilibrium points with high computational complexity to update the Q function. For example, the classic Nash-Q algorithm (an algorithm that extends the Minimax-Q algorithm from zero-sum games to multi-person general-sum games, and can converge to the Nash equilibrium point in cooperative equilibrium or adversarial equilibrium environments) uses precise Nash equilibrium to update strategies, and the CE-Q algorithm uses precise correlated equilibrium to update strategies. Since the computational complexity of precise equilibrium increases sharply with the increase in the number of intelligent agents, existing methods cannot be well applied to environments with multiple intelligent agents.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a multi-agent interaction method, system, system and computer-readable storage medium based on group game, aiming to solve the problem that the methods of the prior art cannot be well applied to environments with multiple agents.

[0006] To achieve the above object, the present invention provides a multi-agent interaction method based on group game, and the multi-agent interaction method based on group game comprises the following steps:

[0007] Initialize the preset parameters required for the agent's reinforcement learning, initialize the agent's Q table, and initialize the learning cycle;

[0008] Obtaining a threshold value for solving the ordinary differential equation, and performing attenuation processing on the threshold value for solving the ordinary differential equation;

[0009] Initialize the system state and use a greedy strategy to select the joint action of the agent;

[0010] Each agent performs the selected action and receives observation samples and rewards;

[0011] Construct a group game based on the Q table of intelligent agents, and use the inheritance-based BNN group game dynamic algorithm to solve the approximate equilibrium strategy;

[0012] The agent updates the Q table based on the expected reward obtained.

[0013] The multi-agent interaction method based on group game, wherein the agent updates the Q table based on the expected reward obtained, and then further includes:

[0014] Determine whether the current system state is a termination state;

[0015] If the current system state is not a terminal state, the current system state is transferred to the next system state;

[0016] If the current system state is the termination state, determine whether the current learning cycle has reached the maximum learning cycle;

[0017] If the current learning cycle does not reach the maximum learning cycle, the learning cycle iterator is updated;

[0018] If the current learning cycle reaches the maximum learning cycle, the current process ends.

[0019] The multi-agent interaction method based on group game, wherein the preset parameters include learning rate, discount factor and exploration factor;

[0020] The initialization of preset parameters required for the agent reinforcement learning, initialization of the agent's Q table, and initialization of the learning cycle specifically include:

[0021] Initialize the learning rate, discount factor and exploration factor, initialize the threshold for solving ordinary differential equations, initialize the agent's Q table to zero, and initialize the learning cycle e;

[0022] Among them, the learning rate is used to control the learning speed of reinforcement learning; the discount factor represents the influence of the future reward generated by the current action of the agent on the current Q table update; the exploration factor represents the probability of the agent trying a new action in the visited state; the Q table of the agent is used to represent the long-term expected return corresponding to different state-action pairs of the agent; the value range of the learning cycle e is 1, 2, ..., N eps , where N eps Represents the maximum number of training cycles.

[0023] The multi-agent interaction method based on group game, wherein the step of obtaining the solution threshold of the ordinary differential equation and performing attenuation processing on the solution threshold of the ordinary differential equation, specifically includes:

[0024] Get the solution threshold of the ordinary differential equation and attenuate the solution threshold by δ e =κδ e-1 , where δ e represents the threshold for solving the ordinary differential equation for the learning period e, δ e-1 represents the solution threshold of the ordinary differential equation in the learning cycle e-1, and κ represents the attenuation factor.

[0025] The multi-agent interaction method based on group game, wherein the system state is initialized and the joint action of the agent is selected by a greedy strategy, specifically including:

[0026] Define the system state as the environment state currently observed by the agent, and initialize the system state s = s ini ;

[0027] A greedy strategy is used to select the joint action a of the agents, where the joint action a represents the action set of multiple agents. A joint action is randomly selected for each agent with probability ε, and a joint action that maximizes the Q value is selected for the agent with probability (1-ε), where ε represents a random number between 0 and 1.

[0028] The multi-agent interaction method based on group game, wherein each agent performs the selected action and receives observation samples and rewards, specifically includes:

[0029] Each agent performs the selected action and receives observation samples (s, a, R i , s′), where R i represents the reward received by agent i from the environment, and s′ represents the next system state.

[0030] The multi-agent interaction method based on group game, wherein the group game based on the Q table of the agent is constructed, and the approximate equilibrium strategy is solved by using the BNN group game dynamic algorithm based on inheritance, specifically includes:

[0031] Construct a regular game based on the agent's Q value at the next system state s':

[0032]

[0033] in, represents the action space of agent i, Qi(s′) represents the value of the Q function of agent i in the next state s′, represents the set of all agents;

[0034] The regular game based on the Q value of the agent is constructed into a corresponding group game, which is characterized by two reward functions:

[0035]

[0036]

[0037] in, It represents the reward obtained by selecting the kth pure strategy meta-agent in the meta-agent corresponding to agent i when the population state is x(s); represents the average return of the population corresponding to agent i; i and j represent the subscripts of the agents; k represents the subscript of the pure strategy of the agent; a -i represents the strategies of all agents except agent i, Indicates a -i The value space of Q i (s, k, a -i ) indicates that agent i is in system state s, agent i chooses the kth pure strategy, and the joint strategy of other agents is a -i The corresponding Q value when Indicates that when the system state is s, the population corresponding to agent j selects pure strategy a j The proportion of meta-agents; It indicates the proportion of meta-agents that choose the kth pure strategy in the population corresponding to agent i when the system state is s;

[0038] The inheritance-based BNN group game dynamic algorithm is used to solve the approximate equilibrium strategy in the s′ state:

[0039]

[0040] Among them, F BNN represents the iterative function for solving ordinary differential equations, x * (s′) represents the equilibrium strategy of the agent corresponding to the system state s′;

[0041] The ordinary differential equation to be solved is expressed as:

[0042]

[0043] in, express The derivative with respect to τ is represents the excess reward obtained by selecting the kth pure strategy meta-agent in the population corresponding to agent i, It represents the excess reward obtained by the meta-agent that selects the lth pure strategy in the population corresponding to agent i.

[0044] The multi-agent interaction method based on group game, wherein the agent updates the Q table based on the expected reward obtained, specifically includes:

[0045] Each agent gets the expected reward Θ in state s′ i (s′), and use the expected reward to update the Q table as:

[0046] Q i (s, a) = (1-α) Q i (s,a)+α[r i +γΘ i (s′)];

[0047] Among them, Q i (s, a) represents the Q value corresponding to the agent i and the state-action pair (s, a), α represents the learning rate, r i represents the reward of agent i, and γ represents the discount factor;

[0048] When the state of the environment is changed by the current action of the agent, the system state s is transformed to the next system state s′.

[0049] In addition, to achieve the above-mentioned purpose, the present invention also provides a system, wherein the system includes: a memory, a processor, and a multi-agent interaction program based on group game stored on the memory and executable on the processor, wherein the multi-agent interaction program based on group game, when executed by the processor, implements the steps of the multi-agent interaction method based on group game as described above.

[0050] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multi-agent interaction program based on group game, and when the multi-agent interaction program based on group game is executed by a processor, the steps of the multi-agent interaction method based on group game as described above are implemented.

[0051] The present invention discloses a multi-agent interaction method and system based on group game, which is used to solve the problem of difficult trade-off between the complexity of solving equilibrium strategies and system performance in multi-agent systems, for example, it can promote the development of applications such as vehicle path planning and road network coordination in autonomous driving. First, the regular game of each stage in non-cooperative multi-agent reinforcement learning is modeled as a group game. In the group game, each agent is expanded into a group of "meta-agents". Each meta-agent is a mathematical abstraction when the actual agent selects a certain pure strategy state. The mixed strategy of a single agent is represented by the proportion of available pure strategies selected by a group of meta-agents. Then, the equilibrium strategy in the group game can be obtained by solving ordinary differential equations to obtain this theoretical guarantee. The system efficiently calculates the equilibrium strategy corresponding to each state, and controls the trade-off between equilibrium complexity and accuracy by designing two acceleration mechanisms.

[0052] The present invention introduces group game into the multi-agent reinforcement learning framework. The equilibrium strategy of the agent can be obtained by solving a series of ordinary differential equations. Since the solution accuracy of ordinary differential equations is controllable, the multi-agent interaction method and system proposed in the present invention achieve a better trade-off between the solution accuracy and solution speed of the equilibrium strategy. Experimental results show that the multi-agent interaction system proposed in the present invention has better performance in a multiple-agent environment than the existing multi-agent reinforcement learning system. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flow chart of a preferred embodiment of the multi-agent interaction method based on group game of the present invention;

[0054] Figure 2 It is a flow chart of strategy update of a multi-agent interaction system based on group game in a preferred embodiment of the multi-agent interaction method based on group game of the present invention;

[0055] Figure 3 It is a schematic diagram of group game composition in a preferred embodiment of the multi-agent interaction method based on group game of the present invention;

[0056] Figure 4 It is a schematic diagram of Q table update based on group state evolution in a preferred embodiment of the multi-agent interaction method based on group game of the present invention;

[0057] Figure 5 Schematic diagram of the operating environment of a preferred embodiment of the system of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0059] The multi-agent interaction method based on group game described in the preferred embodiment of the present invention is as follows: Figure 1 and Figure 2 As shown, the multi-agent interaction method based on group game includes the following steps:

[0060] Step S10: Initialize the preset parameters required for the agent reinforcement learning, initialize the agent's Q table, and initialize the learning cycle.

[0061] Specifically, the preset parameters include learning rate, discount factor and exploration factor; initializing the learning rate, discount factor and exploration factor, initializing the solution threshold of the ordinary differential equation, initializing the Q table (also called Q function) of the agent to zero, and initializing the learning cycle e.

[0062] Among them, the learning rate is used to control the learning speed of reinforcement learning; the discount factor represents the influence of the future reward generated by the current action of the agent on the current Q table update; the exploration factor represents the probability of the agent trying a new action in the visited state; the Q table of the agent is used to represent the long-term expected return corresponding to different state-action pairs of the agent; the value range of the learning cycle e is 1, 2, ..., N eps , where N eps Represents the maximum number of training cycles.

[0063] Step S20: Obtain a threshold value for solving the ordinary differential equation, and perform attenuation processing on the threshold value for solving the ordinary differential equation.

[0064] Specifically, the solution threshold of the ordinary differential equation is obtained, and the solution threshold is attenuated by δ e =κδ e-1 , where δ e represents the threshold for solving the ordinary differential equation for the learning period e, δ e-1 represents the solution threshold of the ordinary differential equation in the learning cycle e-1, and κ represents the attenuation factor.

[0065] Step S30: Initialize the system state and use a greedy strategy to select the joint action of the agent.

[0066] Specifically, the system state is defined as the state of the environment currently observed by the agent (usually defined by a set of characteristics of the environment), and the system state s is initialized to s ini ; until s becomes the system termination state s term (For example, the current game is over).

[0067] A greedy strategy is used to select the joint action a of the agents, where the joint action a represents the action set of multiple agents. A joint action is randomly selected for each agent with probability ε, and a joint action that maximizes the Q value is selected for the agent with probability (1-ε), where ε represents a random number between 0 and 1.

[0068] Step S40: Each agent performs the selected action and receives observation samples and rewards.

[0069] Specifically, each agent performs the selected action and receives observation samples (s, a, R i , s′), where R i Represents the reward received by agent i from the environment, and s′ represents the next system state; where the reward has different definitions in different environments. For example, in autonomous driving, the reward can be defined as the congestion level of a certain intersection.

[0070] Step S50: construct a group game based on the Q table of the intelligent agent, and use the inheritance-based BNN group game dynamic algorithm to solve the approximate equilibrium strategy.

[0071] Specifically, a regular game based on the agent's Q value is constructed in the next system state s':

[0072]

[0073] in, represents the action (i.e. pure strategy) space of agent i, Q i (s′) represents the value of the Q function of agent i in the next state s′, represents the set of all agents; since the reward R i It cannot represent the long-term expected return of agent i. The Q function of the agent used in the present invention is used as the return function in the regular game to solve the equilibrium.

[0074] The regular game based on the Q value of the agent is constructed into a corresponding group game, which is characterized by two reward functions:

[0075]

[0076]

[0077] in, It represents the reward obtained by selecting the kth pure strategy meta-agent in the meta-agent corresponding to agent i when the population state is x(s); represents the average return of the population corresponding to agent i; i and j represent the subscripts of the agents; k represents the subscript of the pure strategy of the agent; a -i represents the strategies of all agents except agent i, Indicates a -i The value space of Q i (s, k, a -i ) indicates that agent i is in system state s, agent i chooses the kth pure strategy, and the joint strategy of other agents is a -i The corresponding Q value when Indicates that when the system state is s, the population corresponding to agent j selects pure strategy a j The proportion of meta-agents; It indicates the proportion of meta-agents that choose the kth pure strategy in the population corresponding to agent i when the system state is s;

[0078] The approximate equilibrium strategy in the S′ state is solved by using the inheritance-based BNN (Brown-von Neumann-Nash) group game dynamic algorithm:

[0079]

[0080] Among them, F BNN represents the iterative function for solving ordinary differential equations, x * (s′) represents the equilibrium strategy of the agent corresponding to the system state s′;

[0081] The ordinary differential equation to be solved is expressed as:

[0082]

[0083] in, express The derivative with respect to τ is represents the excess reward obtained by selecting the kth pure strategy meta-agent in the population corresponding to agent i, It represents the excess reward obtained by the meta-agent that selects the lth pure strategy in the population corresponding to agent i.

[0084] Step S60: The agent updates the Q table based on the expected reward obtained.

[0085] Specifically, each agent gets the expected reward Θ in state s′ i (s′), and use the expected reward to update the Q table as:

[0086] Q i (s, a) = (1-α) Q i (s,a)+α[r i +γΘ i (s′)];

[0087] Among them, Q i(s, a) represents the Q value corresponding to the agent i and the state-action pair (s, a), α represents the learning rate, r i represents the reward of agent i, and γ represents the discount factor;

[0088] When the state of the environment is changed by the current action of the agent, the system state s is converted to the next system state S′, that is, the state of the environment is changed by the current action of the agent and transferred to the new system state s′.

[0089] Furthermore, if Figure 2 As shown, after the agent updates the Q table based on the expected reward, it determines whether the current system state is a terminal state; if the current system state is not a terminal state, the current system state is transferred to the next system state, and continues to jump to each agent to perform the selected action, and receives the observation sample and reward and the next process; if the current system state is a terminal state, it determines whether the current learning cycle has reached the maximum learning cycle; if the current learning cycle has not reached the maximum learning cycle, the learning cycle iterator is updated, and continues to jump to the attenuation of the solution threshold of the ordinary differential equation and the next process; if the current learning cycle reaches the maximum learning cycle, the current process ends.

[0090] like Figure 3 As shown in the figure, a population society usually contains multiple populations, and each population is composed of multiple meta-agents. Each meta-agent can select an action from the optional actions of its corresponding agent as its strategy. The proportion of meta-agents in the entire population selecting different strategies represents the mixed strategy of the corresponding agent.

[0091] like Figure 4 As shown, by introducing group game, the present invention can find the equilibrium strategy of the intelligent agent in each state by means of group state evolution, and by adjusting the number of evolutionary iterations, the approximate equilibrium in each state can be obtained.

[0092] The main purpose of the present invention is to provide a multi-agent interaction method and system based on group game. Its main feature is that it does not accurately solve the equilibrium strategy of the game at each stage, but obtains approximate equilibrium with relatively low computational complexity. By designing an acceleration mechanism for solving ordinary differential equations, it achieves the optimal trade-off between equilibrium complexity and equilibrium accuracy, as well as the optimal performance of the multi-agent system.

[0093] Furthermore, the present invention can also modify the definition of the multi-agent reinforcement learning environment, including states, actions, and rewards. Use other group dynamics algorithms to solve the equilibrium strategy corresponding to the stage game, and use the same game theory reinforcement learning framework as the present invention to update the agent strategy. Modify the threshold initialization method and update method for solving ordinary differential equations. Modify the initial value update method for solving ordinary differential equations.

[0094] Furthermore, if Figure 5 As shown, based on the above-mentioned multi-agent interaction method and system based on group game, the present invention also provides a system accordingly, and the system includes a processor 10, a memory 20 and a display 30. Figure 5 Only some of the components of the system are shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may be implemented instead.

[0095] In some embodiments, the memory 20 may be an internal storage unit of the system, such as a hard disk or memory of the system. In other embodiments, the memory 20 may also be an external storage device of the system, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the system. Further, the memory 20 may also include both an internal storage unit of the system and an external storage device. The memory 20 is used to store application software and various types of data installed in the system, such as the program code of the installation system, etc. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a multi-agent interaction program 40 based on group game is stored on the memory 20, and the multi-agent interaction program 40 based on group game can be executed by the processor 10, thereby realizing the multi-agent interaction method based on group game in the present application.

[0096] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the multi-agent interaction method based on group game.

[0097] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information in the system and to display a visual user interface. The components 10-30 of the system communicate with each other via a system bus.

[0098] In one embodiment, when the processor 10 executes the group game-based multi-agent interaction program 40 in the memory 20, the steps of the group game-based multi-agent interaction method described above are implemented.

[0099] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multi-agent interaction program based on group game, and when the multi-agent interaction program based on group game is executed by a processor, the steps of the multi-agent interaction method based on group game as described above are implemented.

[0100] In summary, the multi-agent interaction method and system based on group game of the present invention has the following advantages compared with the prior art:

[0101] (1) Compared with the existing equilibrium-based multi-agent system, the present invention achieves the control of the trade-off between equilibrium solution speed and solution accuracy by introducing group game into the equilibrium strategy solution of stage game, increases the flexibility of equilibrium solution, and improves the overall performance of the multi-agent system.

[0102] (2) Compared with the existing group game equilibrium strategy solution method, the present invention uses the idea of ​​inheritance and controllable threshold to speed up the equilibrium solution speed under the same system state. Specifically, since the solution speed of ordinary differential equations depends on the initial value and the solution threshold, when the agent visits the same system state twice in a row, the present invention uses the equilibrium state obtained in the previous visit as the initial value for the second visit. Compared with the existing group dynamic solution equilibrium method, this mechanism improves the equilibrium solution speed and further increases the scalability of the entire multi-agent system.

[0103] (3) The existing equilibrium-based multi-agent reinforcement learning method treats each agent as a separate individual. The present invention uses the agent's Q function as the agent's reward function in each stage of the game, and the regular game in each stage is equivalent to a group game. Each population is divided into multiple "meta-agents", and the behavior of each agent is characterized by the collective behavior of the "meta-agents". This allows the system proposed by the present invention to obtain the equilibrium strategy of the agent in each stage of the game by solving ordinary differential equations, reducing the complexity of equilibrium solution and achieving a better trade-off between complexity and performance.

[0104] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0105] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0106] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multi-agent interaction method based on group game, characterized in that: The multi-agent interaction method based on group game includes: Initialize the preset parameters required for the agent's reinforcement learning, initialize the agent's Q table, and initialize the learning cycle; Obtaining a threshold value for solving the ordinary differential equation, and performing attenuation processing on the threshold value for solving the ordinary differential equation; Initialize the system state and use a greedy strategy to select the joint action of the agent; Each agent performs the selected action and receives observation samples and rewards; Construct a group game based on the Q table of intelligent agents, and use the inheritance-based BNN group game dynamic algorithm to solve the approximate equilibrium strategy; The agent updates the Q table based on the expected reward obtained; The method of constructing a group game based on the Q table of an intelligent agent and solving the approximate equilibrium strategy by using the inheritance-based BNN group game dynamic algorithm specifically includes: In the next system state The following constructs a regular game based on the Q value of the agent: ； in, Representing an Agent i The action space, Representing an Agent i The Q function in the next state The value below represents the set of all agents; The regular game based on the Q value of the agent is constructed into a corresponding group game, which is characterized by two reward functions: ； ； in, Indicates that the population status is When the intelligent agent i The corresponding meta-agent selects k The reward obtained by a pure strategy meta-agent; Representing an Agent i The average return obtained by the corresponding population; i and j The subscript representing the agent; k The subscript representing the pure strategy of the agent; In addition to the intelligent agent i The strategies of all agents except express The value space of Representing an Agent i In the system status s , Agent i Select k pure strategy, and the joint strategy of other agents is The corresponding Q value when Indicates the system status is s When the intelligent agent j Choose a pure strategy in the corresponding population The proportion of meta-agents; Indicates the system status is s When the intelligent agent i Select the first k The proportion of meta-agents with pure strategies; Solving the problem using the inheritance-based BNN group game dynamic algorithm Approximate equilibrium strategy under the state: ； in, represents the iterative function for solving ordinary differential equations, Indicates system status The corresponding equilibrium strategy of the intelligent agent; The ordinary differential equation to be solved is expressed as: ； in, express right The derivative of Representing intelligent agent i Select the first k The excess reward obtained by a pure strategy meta-agent is Representing an Agent i Select the first l The excess reward obtained by a pure strategy meta-agent.

2. The multi-agent interaction method based on group game according to claim 1, characterized in that: The agent updates the Q table based on the expected reward obtained, and then further includes: Determine whether the current system state is a termination state; If the current system state is not a terminal state, the current system state is transferred to the next system state; If the current system state is the termination state, determine whether the current learning cycle has reached the maximum learning cycle; If the current learning cycle does not reach the maximum learning cycle, the learning cycle iterator is updated; If the current learning cycle reaches the maximum learning cycle, the current process ends.

3. The multi-agent interaction method based on group game according to claim 1, characterized in that: The preset parameters include learning rate, discount factor and exploration factor; The initialization of preset parameters required for the agent reinforcement learning, initialization of the agent's Q table, and initialization of the learning cycle specifically include: Initialize the learning rate, discount factor, and exploration factor, initialize the threshold for solving ordinary differential equations, initialize the agent's Q table to zero, and initialize the learning cycle e ; Among them, the learning rate is used to control the learning speed of reinforcement learning; the discount factor represents the influence of the future reward generated by the current action of the agent on the current Q table update; the exploration factor represents the probability of the agent trying a new action in the visited state; the Q table of the agent is used to represent the long-term expected return corresponding to different state-action pairs of the agent; the learning cycle e The value range is 1, 2, ..., N eps ,in N eps Represents the maximum number of training cycles.

4. The multi-agent interaction method based on group game according to claim 3 is characterized in that: The obtaining of the solution threshold of the ordinary differential equation and performing attenuation processing on the solution threshold of the ordinary differential equation specifically includes: Get the solution threshold of the ordinary differential equation and attenuate the solution threshold ,in, Represents the learning cycle e The threshold value for solving the ordinary differential equation, Represents the learning cycle e -1 is the threshold for solving ordinary differential equations, Represents the attenuation factor.

5. The multi-agent interaction method based on group game according to claim 4 is characterized in that: The system state is initialized, and the joint action of the agent is selected using a greedy strategy, specifically including: Define the system state as the environment state currently observed by the agent and initialize the system state ; Use a greedy strategy to select the joint action of the agent , joint action Represents the action set of multiple agents; with probability Randomly select a joint action for each agent with probability (1- ) is the joint action selected by the agent to maximize the Q value, where: Represents a random number between 0 and 1.

6. The multi-agent interaction method based on group game according to claim 5, characterized in that: Each agent performs the selected action and receives observation samples and rewards, including: Each agent performs the selected action and receives observation samples ,in, Representing an Agent i The rewards received from the environment, Indicates the next system state.

7. The multi-agent interaction method based on group game according to claim 1, characterized in that: The agent updates the Q table based on the expected reward obtained, specifically including: Each agent gets Expected reward under state , and use the expected reward to update the Q table as: ； in, Representing an Agent i and state-action pairs The corresponding Q value is represents the learning rate, Representing an Agent i Rewards, represents the discount factor; When the state of the environment is changed by the current action of the agent, the system state s Transition to next system state .

8. A system, characterized in that: The system includes: a memory, a processor, and a multi-agent interaction program based on group game stored in the memory and executable on the processor. When the multi-agent interaction program based on group game is executed by the processor, the steps of the multi-agent interaction method based on group game as described in any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multi-agent interaction program based on group game, and when the multi-agent interaction program based on group game is executed by the processor, the steps of the multi-agent interaction method based on group game as described in any one of claims 1-7 are implemented.

Images