A model-based two-team zero-sum game multi-agent reinforcement learning method

Abstract

This invention discloses a model-based multi-agent reinforcement learning method for a two-team zero-sum game. The method includes: constructing multiple agents using a reinforcement learning model for a game problem involving two robot teams; solving for the optimal actions and optimal strategies of the agents, wherein the solution process is decomposed into an exploration phase and a planning phase. In the exploration phase, the agents randomly explore the environment and collect multiple samples from the exploration trajectories. Based on the collected samples, a state transition matrix and a Markov game model are constructed for the environment. In the planning phase, trajectory data is collected from the Markov game model. For each sample in the trajectory data, the equilibrium strategy for the next state of each agent is obtained by solving a matrix game problem. Then, the optimal action of each agent is obtained by solving an optimization problem. This invention significantly reduces the dependence on sampling of the real environment, reduces sampling complexity, and effectively improves the stability of the model training process.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and more specifically, to a model-based two-team zero-sum game multi-agent reinforcement learning method. Background Technology

[0002] In recent years, multi-agent reinforcement learning (MARL) algorithms have been increasingly applied to solve optimization problems in various multi-agent scenarios. Multi-agent game settings can be categorized into three types: cooperative, competitive, and a hybrid of both. Leveraging its powerful decision-making capabilities, MARL algorithms have achieved significant success in both cooperative and competitive problems. Some algorithms have also explored cooperative-competitive problems that combine both approaches.

[0003] Hybrid games include a common and worthy type of game to study: two-team zero-sum games. These games describe many common agent-based hybrid game scenarios, such as robot soccer or drone formation offense and defense. In these games, agents cooperate with other agents in their own team through a communication network while competing against the opposing team. The large number of agents involved, the complex relationships between them, and the communication problems between agents make solving two-team games extremely difficult.

[0004] Currently, some research has proposed Decompositional Multi-Agent Min-Max Q Learning (FM3Q) to solve the distributed min-max strategy for two teams of players. Some studies have attempted to solve this type of game using model-free algorithms and have provided limited sample analysis. However, model-free algorithms require a large number of samples, which is both unreasonable and challenging in real-world environments. Furthermore, when each agent trains using newly collected samples, the environment's stability is very poor, negatively impacting the policy learning process.

[0005] In contrast, model-based multi-agent reinforcement learning methods offer superior sample efficiency. Existing research has employed model-based approaches and offline frameworks to handle cooperative and two-player competitive scenarios. Algorithms such as Multi-Agent Multi-Game Entity Transformer (MAGENTA) and Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization (OMIGA) utilize offline settings to alleviate data collection bottlenecks, while studies like Multi-Agent Pessimistic Model-Based Reinforcement Learning (MA-PMBRL) and Iteration with Lower Confidence Bounds for Zero-Sum Markov Games (VI-LCB-Game) explore optimal sample complexity. However, the application of model-based MARL in two-team zero-sum Markov games is not only limited but also lacks a corresponding theoretical foundation. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a model-based multi-agent reinforcement learning method for two-team zero-sum games. This method includes the following steps: For the game theory problem of two robot teams competing, a reinforcement learning model is used to construct multiple agents, where each robot is regarded as an agent. The optimal action and optimal policy of the intelligent agent are determined based on a given optimization problem. The solution process is decomposed into an exploration phase and a planning phase. During the exploration phase, the agent randomly explores the environment and collects data from its exploration trajectory. One sample; Based on the collected samples, construct the state transition matrix in the environment. And Markov game models: ; During the planning phase, from the model Trajectory data collected in the middle and for Each sample in The next state of each agent is obtained by solving a matrix game problem. Equilibrium strategy Then, by solving the given optimization problem, the results for each agent can be obtained. Optimal action value function ; in, Represents the global state space. and These are the intelligent agents in Team 1. And the intelligent agents in Team 2 The action space, the joint action space is defined as... and , and These are the reward distributions for Team 1 and Team 2, and The communication networks for Team 1 and Team 2 are respectively. As a discount factor, It is the sample size. This indicates the number of agents in Team 1. It is the number of agents in Team 2. Indicates the state at time t. This represents the state at time t+1. This represents the action of agent i in team 1 at time t. Let k represent the action of agent j in team 2 at time t, and k be the iteration index.

[0007] Compared with existing technologies, the advantages of this invention are that the model-based multi-agent reinforcement learning method for two-team zero-sum games separates the exploration and planning phases. In the exploration phase, fewer samples are needed to construct the environment model, allowing agents to freely collect samples from the environment model for planning, thus ensuring supplementary planning of agent strategies. In the planning phase, any efficient optimization algorithm can be used, further improving algorithm performance. This invention designs a model-based decentralized value iteration method for two-team zero-sum Markov games, which can reduce sample complexity in complex scenarios and provides finite sample analysis for the algorithm, resulting in superior sample efficiency.

[0008] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0009] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0010] Figure 1 This is a flowchart of a model-based two-team zero-sum game multi-agent reinforcement learning method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a robot soccer game according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the process of a model-based two-team zero-sum game multi-agent reinforcement learning method according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the training loss convergence curve (with mean error band) according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the value function error convergence curve (with mean error band) according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the Nash distance convergence curve (with mean error band) according to an embodiment of the present invention; Figure 7 This is a schematic diagram of strategy entropy comparison (with mean error band) according to an embodiment of the present invention. Detailed Implementation

[0011] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0012] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0013] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0014] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0015] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0016] In general, existing model-free reinforcement learning algorithms suffer from low sample efficiency, difficulties in collecting real-world samples, and poor stability of the training environment when solving two-team zero-sum games. This invention designs a model-based multi-agent reinforcement learning method for two-team zero-sum games, decomposing the policy-solving process into an exploration phase (allowing agents to explore and build a model) and a planning phase (planning the policy based on the model). This achieves lower sample complexity, smaller steady-state error, and faster convergence rate. This invention provides, for the first time, a finite-sample analysis of model-based multi-agent reinforcement learning in two-team zero-sum Markov games, providing a benchmark and theoretical guarantee for this type of algorithm.

[0017] See Figure 1 As shown, the provided model-based two-team zero-sum game multi-agent reinforcement learning method includes the following steps: Step S1: For the game problem of two robot teams competing, multiple intelligent agents are constructed using a reinforcement learning model.

[0018] See Figure 2 As shown, taking a robot soccer match scenario as an example, the playing field is a rectangular plane, and two robot teams (Team 1 and Team 2) each consist of multiple autonomous mobile robots, each robot being considered an intelligent agent. The goal of the match is to shoot the ball into the opponent's goal. This invention provides a model-based zero-sum reinforcement learning decision-making method for Team 1, where Team 1 is the team (coordinating offense and defense), and Team 2 is the opposing team (the adversary). The zero-sum relationship between the two sides is reflected in the fact that the cumulative reward of Team 1 is equal to the negative of the cumulative reward of Team 2, that is, one side scoring is equivalent to the other side losing a point.

[0019] The environment is a robot soccer field, and the observations include the goal area, boundaries, the real-time position and speed of the ball, the poses of each robot (position, orientation, and speed), the current score, and the match time.

[0020] Every moment The global state can be composed of the following information: the position and velocity of the ball; the position, orientation angle, linear velocity, angular velocity, and whether each robot in Team 1 and Team 2 is holding the ball or can kick it; the current score; and the remaining time in the game.

[0021] Action space includes movement direction, movement speed, kicking action (direct shot, pass) or interception, etc.

[0022] Each agent's rewards include a shot score reward, a shot attempt reward, a successful pass reward, an interception reward (corresponding to the opposing team's shot score reward, shot attempt reward, and successful pass reward; a successful interception is positive, and a failed interception is negative, ensuring that the sum of the rewards for both teams is zero), and a position reward (the sum of the position rewards for both teams is zero).

[0023] In one embodiment, Team 1's joint strategy To start from the global state Mapping to the joint distribution of robot actions The present invention aims to solve the equilibrium strategy of Team 1 under the optimal confrontation of the opponent, that is, the optimal response strategy of Team 1 when it is known that Team 2 will maximize its own reward.

[0024] Step S2 involves decomposing the policy solution process into an exploration phase and a planning phase to determine the optimal action values ​​for each agent.

[0025] See Figure 3 As shown, in the Model-Based Decentralized Value Iteration (MDVI) algorithm, the solution process is divided into two stages: the exploration stage and the planning stage.

[0026] During the exploration phase, the agent randomly explores the environment and collects data from its exploration trajectory. One sample (one sample refers to) Subsequently, based on these samples, state transitions in the environment are constructed. and Markov game models ,in, Represents the global state space. and These are Team 1 intelligent agents And the intelligent agents in Team 2 The action space, correspondingly, , Team 1 intelligent agent And the intelligent agents in Team 2 The specific actions, the joint action space is defined as and . Here is the state transition matrix. and It is the reward distribution, sampled values. , For specific rewards. , The communication networks for teams 1 and 2 are respectively. This is the discount factor.

[0027] During the planning phase, from the model Trajectory data collected in the middle , which includes One sample. For Each sample in Both can be solved by solving a matrix game problem. (This matrix game problem can be efficiently solved as a linear programming problem) to obtain the next state of each agent. Equilibrium strategy Next, calculate the target value. Each agent can be obtained by solving the optimization problem. Reachable optimal action value function : (1) Among them, solution belong And suffer constraint( It is the upper bound of the absolute value of the reward function. Represents the value under the model (e.g., It is the action value function under the model. It is Team 1 in (Joint equilibrium strategy under the following conditions) This represents the estimate from the optimization algorithm (e.g., yes (estimates) The first step of the algorithm The next iteration; This indicates that the value comes from the first... One sample (equivalent to) (Time sample) Indicates the intelligent agent in Team 1 The value of (since the solution only needs to focus on team 1, "1" can be omitted); This represents the optimal value. This indicates the number of agents in Team 1. This indicates the number of agents in Team 2.

[0028] In practical applications, many decentralized algorithms can solve the problem. Ultimately, after... After the iteration, the optimal action value is obtained. and optimal strategy .

[0029] To verify the effectiveness of this invention, the convergence of the algorithm is proven and supported by evidence, and a finite sample analysis is conducted at the theoretical level. Before proceeding, several assumptions need to be made.

[0030] Assumption 1 ( -Mixed Sample Path): It was collected from a sample path of a Markov game and satisfies the exponential... - Mixed properties. Furthermore, state-action distribution. It is stable. -The mixed properties are defined as follows: set up It is a random process. express The set, Indicates by generated -Algebra. Then, for all ,exist Make (2) in yes - Future events in algebra.

[0031] Assumption 2 (Concentration Factor): Let For agents from distribution Departure process The new distribution obtained after the step. Then there exists a constant. Make (3) Assumption 3 (Single-step decentralized computation error): In the first step... In each iteration, for each agent There exists a constant Make (4) Assumption 4 (Linear Function Class): The function class used in Algorithm 1 It can be described as a parametric linear function class. ,in It is a state-action sample eigenvectors. Therefore, by Obtained from The matrix formed It is at full capacity.

[0032] Assumption 5 (Function Class Capacity): Let Represents function class The pseudo-dimensionality. Then the function class used in Algorithm 1. The pseudo-dimensionality satisfies .

[0033] Assumption 6 (Property of the Consensus Matrix Sequence): For any consensus matrix The following relationship must be satisfied.

[0034] (Decentralization property) If And the side ( It is a communication network diagram. (edge ​​set), then .

[0035] (Double randomness) ,and .

[0036] (Joint spectral property) There exists an integer Make ,in ( (where is the largest singular value of the matrix).

[0037] To facilitate the timeline The above sample set For simplified analysis, let's assume the sample set... The sample was collected from a single sample path. Since the game under discussion is a Markov process, this sample possesses... - The mixed properties are reasonable. Hypothesis 2 measures the distribution from a fixed distribution. The distribution of future states from the starting point and the distribution from the sample The similarity between the distributions of future states at the start ensures that the sample data distribution fully covers the true distribution. Assumption 3 characterizes the computational error of the distributed optimization algorithm due to a finite number of iterations. Function class function value The boundedness of the property guarantees the rationality of this assumption. Because by The constraints are such that Assumption 4 is easily satisfied. The full-rank condition ensures the strong convexity of the fitting problem. In Assumption 5, The capacity of the function class is described. Since the function class used is linear, its complexity metric is always finite. Assumption 6 defines a consensus matrix sequence. It uses communication networks Based on this, its double randomness and joint spectrum properties guarantee the convergence of the decentralized algorithm on time-varying communication networks.

[0038] The main conclusions of convergence and finite sample analysis are given below.

[0039] Theorem 1 (Value Function Error Bound of MDVI): Under the premise of satisfying Assumptions 1-6, for any and There are at least The probability exists:

[0040] (5) in: (6)

[0041]

[0042] (7) in, As given by Assumption 5, Given by Assumption 1, As given by hypothesis 2.

[0043] It is worth noting that, The upper bound of the error bound is the error between the actual action-value function and the action-value function calculated by MDVI under the model. Therefore, its upper bound directly reflects and measures the convergence of the algorithm. The obtained error bound consists of three main parts: decentralized optimization error (DOE, caused by the solution process of the distributed optimization algorithm), iteration error (IE, generated during the iteration process), and model estimation error (MEE, caused by the difference between the model and the actual environment). For any If the number of samples satisfy: (8) The algorithm can then converge. Furthermore, the sample complexity is: (9) in This indicates that the complexity is ignored when considering logarithmic factors.

[0044] Furthermore, simulations were performed to verify the effectiveness of the invention. The simulation process included a two-stage, two-team zero-sum Markov game, but the scope of protection of the invention is not limited to the following embodiments.

[0045] In one simulation example, teams 1 and 2 each contain There are 12 agents. Each agent's action space contains 2 discrete actions, and they compete against agents in the opposing team. Based on the actions taken by both sides in the first phase, the game proceeds to the corresponding subgame in the second phase. The final payoff for each team is the sum of the payoffs of all agents in that team throughout the two phases of the game.

[0046] See Figure 4 and Figure 5 As shown, the convergence of the model-free method (using...) Compared to samples from multiple periods, the MDVI algorithm proposed in this invention only requires... With the necessary samples for a given period, convergence can be achieved, and a smaller steady-state training loss can be obtained. Compared with model-free algorithms At the same time, it also achieved a better steady-state value function error (). Compared with model-free ).

[0047] See Figure 6 and Figure 7 As shown, MDVI also works with a smaller sample size ( Each cycle, compared to the model-free method. Within a few cycles, a very close policy entropy can be obtained, and the Nash distance also converges to an acceptable range, reaching Nash equilibrium.

[0048] In the actual simulation implementation, the DIGing algorithm is used as the solver for the internal matrix game, and the convergence tolerance is set to... Each iteration requires a solution. A matrix game. Step size. The initial value is set to Subsequently, with the decay rate Gradually decaying, with a minimum step size lower limit of 1. The number of sample periods generated from the learning model is One, discount factor .

[0049] In summary, compared with the prior art, the present invention has the following main advantages: (1) This invention decouples the solution process into an exploration phase and a planning phase. In the exploration phase, only a small number of samples need to be collected from the physical environment according to the sample complexity requirements. In the planning phase, a large number of virtual samples are generated by the model for strategy optimization, thereby greatly reducing the dependence on sampling of the real environment, reducing sampling complexity, and effectively improving the stability of the training process. (2) This invention rigorously proves that there is an upper bound on the error between the action value function obtained by the algorithm and the optimal action value function. Both theoretical and experimental results show that, compared with the model-free algorithm, this invention has a faster convergence rate and a smaller steady-state error.

[0050] (3) This invention provides a finite sample analysis for the two-team zero-sum Markov game problem for the first time, clarifies the sample complexity of solving the problem, and fills a theoretical gap in this field. It significantly reduces the sampling requirements of real-world environments and provides a theoretical bound on sample complexity, making it more feasible, reasonable and efficient to train multi-agent teams in real physical systems.

[0051] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0052] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0053] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0054] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0055] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0056] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0057] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0058] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A model-based multi-agent reinforcement learning method for two teams playing zero-sum games, comprising the following steps: For the game theory problem of two robot teams competing, a reinforcement learning model is used to construct multiple agents, where each robot is regarded as an agent. Solving for the optimal action and optimal policy of the agent involves decomposing the solution process into an exploration phase and a planning phase: In the exploration phase, the agent explores the environment randomly and collects samples from the exploration trajectories; Based on the collected samples, a state transition matrix in the environment is constructed and a Markov game model: ； During the planning phase, from the model Trajectory data collected in the middle and for Each sample in The next state of each agent is obtained by solving a matrix game problem. Equilibrium strategy Then, by solving the given optimization problem, the results for each agent can be obtained. Optimal action value function ; in, Represents the global state space. and These are the intelligent agents in Team 1. And the intelligent agents in Team 2 The action space, the joint action space is defined as... and , and These are the reward distributions for Team 1 and Team 2, and The communication networks for Team 1 and Team 2 are respectively. As a discount factor, It is the sample size. This indicates the number of agents in Team 1. It is the number of agents in Team 2. Indicates the state at time t. This represents the state at time t+1. This represents the action of agent i in team 1 at time t. Let k represent the action of agent j in team 2 at time t, and k be the iteration index.

2. The method of claim 1, wherein, The matrix game problem can be represented as: 。 3. The method of claim 1, wherein, The optimization problem is set as follows: Among them, the target value , belong And suffer constraint, It is the upper bound of the absolute value of the reward function. This represents the value under the model. It is Team 1 in The joint equilibrium strategy under; Indicates the estimated value; k represents the first value. iteration Indicates the intelligent agent in Team 1 The value; This represents the optimal value.

4. The method of claim 1, wherein, For each intelligent agent, the environment is a robot soccer match field, and the observations include the goal area, boundaries, the real-time position and speed of the ball, the poses of each robot, the current score, and the match time.

5. The method of claim 4, wherein, Every moment The global state includes: the position and speed of the ball, the position, orientation angle, linear velocity, angular velocity, whether the robot is holding the ball, whether it can kick the ball, the current score, and the remaining time in the game.

6. The method of claim 1, wherein, The action space includes movement direction, movement speed, kicking action, or interception.

7. The method of claim 1, wherein, The rewards for each agent include one or more of the following: scoring reward, shot attempt reward, successful pass reward, interception reward, and position reward.

8. The method of claim 1, wherein, The optimization problem is solved using a centralized algorithm. After multiple iterations, the optimal action value and the optimal strategy are obtained.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer device comprising a memory and a processor, having stored on the memory a computer program capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images