Two-stage credit distribution method based on multi-game unit cooperation mode modeling

Abstract

The invention belongs to the technical field of computer application, and particularly relates to a two-stage credit distribution method for modeling based on a multi-game-unit cooperation mode. According to the method, firstly, cooperation modes of a plurality of game units are modeled into all non-empty subset problems of a complete set, and each subset is regarded as a cooperation alliance; secondly, calculating the marginal contribution of each unit under different alliances by using an anti-factual reasoning method; and finally, through an approximation method based on weighted sampling, efficiently calculating expected contributions of each unit under all possible cooperation modes, and taking the expected contributions as credit values of the units. The credit distribution framework constructed by the invention can realize'global-local-individual 'two-stage credit distribution, and can be seamlessly integrated into a current mainstream algorithm, so that the calculation complexity is remarkably reduced while the credit distribution precision is ensured.

Description

Technical Field

[0001] This invention belongs to the field of computer application technology, specifically relating to a two-stage credit allocation method based on multi-game unit cooperative modeling. Background Technology

[0002] With the development of intelligent technology in recent years, a large number of complex tasks that require multiple game units to work together have emerged in the gaming field. These tasks are often characterized by dynamism, uncertainty and multi-objectives, and the collaborative mode of a single individual or fixed rules is no longer suitable. For example, in multiplayer online battle arena (MOBA) games, teams are composed of game units with different roles such as tanks, mages, marksmen, and supports. Through coordinated actions like vision detection, skill combos, focused fire, and tactical maneuvering, they must overcome enemy defenses and destroy key facilities. A single game unit operating in isolation will lead to team defeat due to a lack of coordination. In real-time strategy (RTS) games, mining, construction, and combat units need to work efficiently together. Mining units ensure resource supply, construction units build fortifications, and combat units defend against enemy attacks. Inefficient operation or disjointed coordination of a single unit can affect the overall course of the battle. In team survival games, reconnaissance, combat, and supply units need to cooperate. Reconnaissance units scout safe routes, combat units eliminate enemy threats, and supply units provide logistical support. Simple pre-set action logic cannot handle unexpected situations such as terrain changes and enemy ambushes. These tasks place urgent demands on the precision of coordination between game units, environmental adaptability, and task completion efficiency.

[0003] To address the challenges of collaborative optimization in the aforementioned scenarios, multi-agent reinforcement learning (MAL) technology, which combines reinforcement learning with multi-agent systems, has emerged. The core idea of ​​reinforcement learning is to continuously interact with the environment, perceive its state, and execute actions. Based on reward signals from the environment, the strategy is constantly adjusted to ultimately optimize the objective. MAL extends this framework to multi-agent interaction scenarios. Its core objective is to enable multiple agents in a shared environment to jointly achieve a unified task objective defined by a global reward function through mutual perception, interaction, and collaborative learning. Among these, the "centralized training and distributed execution" paradigm has become the mainstream technology in the field of cooperative multi-agent reinforcement learning because it balances policy optimization effectiveness with actual execution efficiency. This paradigm uses a centralized architecture during the training phase, aggregating local information from all agents and global environmental information to provide ample data support for policy optimization. During the execution phase, it uses a distributed architecture, where each game unit makes independent decisions based on its own perception information, without needing to share global information in real time. This effectively reduces communication overhead and decision latency, making it well-suited to the actual needs of game scenarios.

[0004] However, the effectiveness of cooperation is largely limited by the core bottleneck of credit allocation. Credit allocation refers to how to reasonably attribute and distribute global reward signals to various game units based on individual contributions, thereby providing precise guidance for individual strategy updates. However, in real-world cooperative scenarios, due to complex interactive relationships between individuals, limitations in the observability of environmental perception, and the non-linear mapping between global rewards and individual contributions, achieving accurate and efficient credit allocation is often extremely challenging. For similar problems, Rashid et al. introduced a hybrid network satisfying the individual-global-optimality condition to achieve a nonlinear combination of the local Q-function into the global Q-function (Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning, ICML 2018); Son et al. proposed a provable, more generalized value decomposition method to try to overcome the monotonicity constraint of hybrid weights (Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning, ICML 2019); Shen et al. proposed a risk-sensitive individual-global optimization principle that balances collaborative stability and risk control (RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization, NeurIPS 2023). Although the above value decomposition methods alleviate the credit allocation problem to some extent, they generally cannot accurately characterize the complex nonlinear relationship between the two, resulting in insufficient value decomposition. Furthermore, most of these methods are black-box decomposition processes, lacking interpretability of the distributed credit allocation results, making it difficult to trace the source and logic of individual contributions, which is detrimental to the optimization and debugging of subsequent strategies. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies and address the problems of inaccurate credit allocation, difficulty in modeling complex collaborative relationships, and exponential growth in computational complexity with the number of cooperating units, this invention proposes a two-stage credit allocation method based on modeling the cooperative modes of multiple game units. This method aims to fairly and accurately quantify the contribution of individuals to the overall team reward. First, the cooperative modes of multiple game units are modeled as a problem of all non-empty subsets of the entire set, with each subset considered as a cooperative alliance. Second, counterfactual reasoning is used to calculate the marginal contribution of each game unit under different alliances. Finally, an approximation method based on weighted sampling is used to efficiently calculate the expected contribution of each game unit under all possible cooperative modes, which is then used as its credit value. The credit allocation framework constructed in this invention can achieve a two-stage credit allocation process of "global-local-individual" and can be seamlessly integrated into current mainstream algorithms, significantly reducing computational complexity while maintaining the accuracy of credit allocation.

[0006] This invention is roughly divided into two parts:

[0007] (1) Construct a model that can characterize the collaborative relationships between individuals, and accurately calculate the contribution of each game unit based on this model. This method formally defines all possible cooperative modes as non-empty subsets of the entire set, with each subset representing a specific cooperative alliance. For a specific game unit in the system, its contribution in any cooperative alliance is defined as the change in the overall value contribution of the alliance before and after the game unit joins the alliance, i.e., its marginal contribution. In order to fairly measure the overall role of the game unit, it is necessary to quantify its role in different alliances, avoid treating it as an independent individual and ignoring the problem of complex collaborative interactions, and realize a two-stage credit allocation of "global-local-individual".

[0008] (2) To address the computational complexity issue caused by the exponential growth of the number of alliances with the number of collaborations, a weighted sampling-based approximation method is designed. This method simulates and estimates the marginal contribution under different collaboration scenarios by randomly sampling a certain number of collaborative alliances and simultaneously sampling the joint actions that individuals outside the alliances may take. By weighting these sampling results according to their corresponding alliance weights and the probability of occurrence of the sampled actions, a reliable approximation of the individual credit value can be obtained efficiently. This method successfully reduces the computational complexity from exponential to linear, which is related to the number of samplings, enabling the algorithm to be applied to large-scale scenarios, thereby guiding the optimization and updating of the policy network of each game unit.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0010] A two-stage credit allocation method based on multi-game unit cooperation modeling, comprising the following steps:

[0011] Step (1) Modeling the cooperation mechanism and calculating counterfactual contributions

[0012] (1.1) First, perform cooperative alliance modeling and weight allocation, including... A collection of game units All non-empty subsets are defined as cooperative unions, i.e., each non-empty subset... This corresponds to a collaboration mode, in which Indicates alliance The scale. For any individual The alliances it can participate in are derived from subsets that do not include itself. A reasonable weight is assigned to each collaborative alliance to reflect its importance in the overall collaboration. The weight calculation formula is as follows:

[0013] (1)

[0014] in, For the alliance In the set The probability distribution in the distribution, i.e., the weights. Indicates from Select from each game unit The number of combinations that make up an alliance, and the weighting method ensures that the importance of alliances of different sizes is reasonably reflected.

[0015] To address the limitations of traditional counterfactual methods that rely on default actions or simulators, this paper calculates the alliance value by marginalizing the action probability distribution of individuals outside the alliance, thus avoiding additional approximation errors and user-defined parameters. For alliances... Its value Defined as:

[0016] (2)

[0017] in, For the alliance The combined actions of individuals (including a set of possible actions in the action space such as movement and attack). For any joint action by individuals outside the alliance, This is the global state. For the action value function of the central commentator, The probability of joint actions by individuals outside the alliance is determined as follows:

[0018] (3)

[0019] in, For individuals Based on its own trajectory The formula quantifies the probability of actions by summing the probabilities of all possible actions of individuals outside the alliance. This method does not rely on simulators or preset default actions, thus accurately quantifying the alliance's actions. Contribution to overall value.

[0020] (1.2) Calculation of individual marginal contribution, for any game unit Assuming it participates in an alliance (i.e., the alliance does not initially include) ), then the game unit In the league Marginal contribution Defined as:

[0021] (4)

[0022] Substituting formula (2) into formula (4) and expanding, we get:

[0023] (5)

[0024] in, This is the product of the individual action probabilities within the alliance. Game Unit Ultimate individual credit The expected marginal contribution of a component across all coalitions, weighted by coalition weights, is calculated using the following formula:

[0025] (6)

[0026] in, It expresses expectations; by integrating contributions from different cooperation methods, this formula achieves a two-stage credit allocation of "global-local-individual", fairly and accurately quantifying the actual role of each individual.

[0027] Step (2) Weighted Sampling Approximation and Framework Integration

[0028] (2.1) Credit approximation calculation based on weighted sampling: To address the computational complexity issue caused by the increase in the number of alliances as the number of game units grows, a weighted sampling approximation method is designed to reduce computational costs through a finite number of samplings. The first step is two-stage sampling: sampling from all non-empty alliances according to their weights... sampling Alliance ( This is a hyperparameter, and its preferred value is equal to the number of collaborations. (balancing accuracy and efficiency); for the first Sampling Alliance From outside the alliance Sampling joint actions in the action space Calculate its action probability Then, the sampling contribution is calculated: the first In the second sampling, the alliance value and game unit marginal contribution They are respectively:

[0029] (7)

[0030] (8)

[0031] Finally, approximate individual credit is calculated by assigning weights to each sample and combining the probability of the coalition occurring with the probability of the action to arrive at the final approximate individual credit. for:

[0032] (9)

[0033] in, Indicates alliance Compared to the entire league The probability of occurrence, This represents the relative probability of the marginal contribution of this sampling.

[0034] Based on the above approximate credit calculation results, this invention can be adapted to different training frameworks. Currently, the mainstream frameworks include two types: value decomposition methods and Actor-Critic methods. The method proposed in this invention has strong adaptability. The first type is an integration of value decomposition methods: it only needs to integrate the alliance value... The calculation method is slightly adapted and adjusted, while the other calculation methods remain unchanged, that is, the combined action in formulas (2) and (7) is adjusted. Replace with individual Q value The alliance value has been changed to:

[0035] (10)

[0036] in, For the value decomposition method, For the alliance Joint Q-value of individual samples The joint Q-value for out-of-consortium sampling. Let the probability of the joint Q-value be sampled. The agent network updates by minimizing the following loss:

[0037] (11)

[0038] in, For individuals Network parameters, To predict the individual Q-value.

[0039] The second approach is the Actor-Critic method integration: combining approximate individual credit... As the reward signal for the policy gradient, the update gradient of the individual policy is:

[0040] (12)

[0041] in, For policy network parameters, For individuals Based on its own trajectory The probability of an action.

[0042] (2.2) The entire training process is divided into two alternating phases:

[0043] Phase 1: Game units interact with the environment and collect data. And store it in the experience playback buffer, where For trajectory, For joint operations, This represents the global state at the current time step. For global rewards, For the global state at the next time step; sample batch data from the buffer (batch size is...). The central commentator network parameters are updated using temporal difference (TD) loss. :

[0044] (12)

[0045] in, For the target value, As a discount factor, For the trajectory at the next time step, For the joint action at the next time step, For the target commenter's network parameters, The overall action value output by the central commentator.

[0046] Phase Two: Individual Credit Allocation and Strategy Update, Based on Updated Central Commentator Output The approximate individual credit of each game unit is calculated using the weighted sampling method in step (2.1). Depending on the integration method type, the strategy network of each game unit is updated using formula (11) or formula (12).

[0047] The beneficial effects of this invention are:

[0048] The purpose of this invention is to model cooperation by treating the cooperation of game units as a set of all non-empty alliances. Combined with counterfactual marginal contribution calculation, it comprehensively considers the complex interactions between individuals, avoiding the shortcomings of existing methods that rely on monotonic value relationships or ignore complex collaborations. This achieves a two-stage credit allocation of "global-local-individual," effectively suppressing the "lazy agent" phenomenon. Furthermore, through a weighted sampling approximation method, the original exponential computational complexity is reduced to the polynomial level, remaining feasible in large-scale scenarios. The number of samplings can be flexibly adjusted, balancing accuracy and efficiency. It can be seamlessly integrated into two mainstream methods without significant modifications to the original network structure, adapting to various collaborative tasks and possessing good versatility and scalability. Attached Figure Description

[0049] Figure 1 This forms the overall framework for the two-stage credit allocation of this invention.

[0050] Figure 2 This is a schematic diagram of the credit allocation module structure of the present invention.

[0051] Figure 3 This is a flowchart of the training process for central commentators in this invention.

[0052] Figure 4 This is a flowchart of the individual strategy training process in this invention. Detailed Implementation

[0053] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0054] This invention can be used for credit allocation in collaborative reinforcement learning tasks, and the overall framework is as follows: Figure 1 As shown, the core structure of the credit allocation module is as follows: Figure 2 As shown, the training process for the central commentator and each local agent is as follows: Figure 3 and Figure 4 As shown.

[0055] This embodiment describes a scenario in the StarCraft II (SMAC) environment where "3 Marines + 2 Marauders" coordinate a raid on an enemy base. The mission involves five independent "multi-game units" (3 Marines, A1-A3, and 2 Marauders, B1-B2), who must coordinate their movements, attacks, and focused fire to breach enemy defenses and destroy the core structure within a time limit (120 seconds). The battlefield environment is partially observable (each game unit can only perceive enemy units, terrain obstacles, and friendly positions within an 8x8 grid around itself). This embodiment integrates the two-stage credit allocation method of this invention into the Actor-Critic method. The specific implementation schemes discussed are only for illustrating the implementation of this invention and do not limit its scope. The core implementation process of this invention is divided into two main modules: "Central Commentator Training" and "Two-Stage Credit Allocation and Individual Strategy Update," which are implemented as follows:

[0056] (1) Training for Central Commentators:

[0057] The input layer of the central commentator network is a global state of dimension 128. The hidden layers are 3 fully connected layers (512, 256, and 128 neurons, ReLU activation function), and the output layer is a 1-dimensional global action value. The implementation process is as follows: After starting the SMAC environment, the five game units select actions based on the action probabilities output by their own policy networks to form a joint action. After execution, the environment returns to the global state. Global Rewards Next global state and trajectory data experience group Store in the experience replay buffer; when the buffer size reaches 10... 4 At each time, 256 sets of experience data are randomly sampled for batch training, and the target commentator network (parameters) is used. Calculate the maximum joint Q-value of the next state using the formula. Calculate the target value; then calculate the temporal difference loss of the current commentator network according to formula (12), minimize the loss through the Adam optimizer and update (parameters). Every 200 training steps, the current parameters are synchronized to the target network and updated. This is to ensure the stability of the target value calculation.

[0058] (2) Two-stage credit allocation and individual strategy updates:

[0059] Combine the 5 game units All non-empty subsets are considered cooperative unions Calculate the alliance weight according to formula (1). The total number of game units. (For alliance size). For example, an alliance ( The weight is 1.2, for the entire league. ( The weight is set to 0.2 to ensure a reasonable representation of the importance of alliances of different sizes. Then, counterfactual marginal contribution is calculated, based on game units. For example, filtering does not include All alliances are calculated using formula (2); then the value of each alliance is calculated using formula (4). Marginal contributions in each alliance, such as In the league The marginal contribution in The above calculation involves too many alliances. Therefore, this invention uses weighted sampling approximation, sampling 5 alliances according to their alliance weights. For each sampled alliance, a set of joint actions is sampled from the game unit action space outside the alliance and the action probability is calculated. The approximate marginal contribution is calculated according to formulas (7) and (8), and then the approximate individual credit of the game unit is calculated using formula (9).

[0060] The input layer of the individual policy network (each game unit is deployed independently) consists of 64-dimensional local observations. The hidden layers are two fully connected layers (256 and 128 neurons, ReLU activation function), and the output layer is a 5-dimensional action probability distribution. (Corresponding to movement, attack, defense, standby, and skill actions). This network updates individual strategy parameters based on approximate individual credit values: the alliance value calculation result is mapped to the advantage value function of the individual strategy network, and the individual strategy network loss is calculated according to formula (12). For individual policy network parameters, the expectation is... (Based on empirical replay buffer sampled data); the loss is minimized using the Adam optimizer, and the policy network parameters are updated along the gradient direction, such as... Losses when taking an attack action Adjusted by gradient descent The action probabilities output by the policy network converge towards higher credit contributions; the individual policy networks and the central commentator network are iteratively updated simultaneously, and the process is repeated until completion. Step training.

Claims

1. A two-stage credit allocation method based on multi-game unit cooperation modeling, characterized in that, The steps are as follows: Step (1) Modeling the cooperation mechanism and calculating counterfactual contributions (1.1) First, perform cooperative alliance modeling and weight allocation, including... A collection of game units All non-empty subsets are defined as cooperative unions, i.e., each non-empty subset... This corresponds to a collaboration mode, in which Indicates alliance The scale; for any individual The alliances it can participate in are derived from subsets that do not include itself; each cooperative alliance is assigned a reasonable weight to reflect its importance in the overall cooperation, and the weight calculation formula is as follows: （1）； in, For the alliance In the set The probability distribution in the distribution, i.e., the weights, Indicates from Select from each game unit The number of combinations that make up an alliance; The alliance value is calculated by marginalizing the action probability distribution of individuals outside the alliance; for the alliance Its value Defined as: （2）； in, For the alliance The combined actions of individuals, including the set of possible actions in the action space such as movement and attack. For any joint action by individuals outside the alliance, This is the global state. For the action value function of the central commentator, The probability of joint actions by individuals outside the alliance is determined as follows: （3）； in, For individuals Based on its own trajectory The probability of the action; (1.2) Calculation of individual marginal contribution, for any game unit Assuming it participates in an alliance That is, the alliance did not initially include Then the game unit In the league Marginal contribution Defined as: （4）； Substituting formula (2) into formula (4) and expanding, we get: （5）； in, The product of individual action probabilities within the alliance; game unit Ultimate individual credit The expected marginal contribution of a component across all coalitions, weighted by coalition weights, is calculated using the following formula: ϕ (a)= E p(C∣N) Φ a (C) （6）； in, Expressing expectations; Step (2) Weighted Sampling Approximation and Framework Integration (2.1) Design of a weighted sampling approximation method: First, a two-stage sampling method is used: from all non-empty alliances, weighted sampling is performed. sampling Alliance For the first Sampling Alliance From outside the alliance Sampling joint actions in the action space Calculate its action probability Then, the sampling contribution is calculated: the first In the second sampling, the alliance value and game unit marginal contribution They are respectively: （7）； （8）； Finally, approximate individual credit is calculated by assigning weights to each sample and combining the probability of the coalition occurring with the probability of the action to arrive at the final approximate individual credit. for: （9）； in, Indicates alliance Compared to the entire league The probability of occurrence, This represents the relative probability of the marginal contribution of this sampling. (2.2) The entire training process is divided into two alternating phases: Phase 1: Interaction between game units and the environment; Phase Two: Individual Credit Allocation and Strategy Updates 2. The two-stage credit allocation method based on multi-game unit cooperation modeling as described in claim 1, characterized in that, In step (2.1), based on the approximate credit calculation results, two training frameworks, the value decomposition method and the Actor-Critic method, are adapted. The first method is value decomposition integration: it only requires integrating the alliance value. The calculation method is adapted and adjusted, while the other calculation methods remain unchanged, that is, the combined action in formulas (2) and (7) is adjusted. Replace with individual Q value The alliance value has been changed to: （10）； in, For the value decomposition method, For the alliance Joint Q-value of individual samples The joint Q-value for out-of-consortium sampling. To sample the probability of the joint Q-value occurring, the agent network updates by minimizing the following loss: （11）； in, For individuals Network parameters, To predict the individual's Q-value; The second approach is the Actor-Critic method integration: combining approximate individual credit... As the reward signal for the policy gradient, the update gradient of the individual policy is: g k = E π ∑ a ∇ θ log π a u a ∣ τ a ϕ ̂ (a) （12）； in, For policy network parameters, For individuals Based on its own trajectory The probability of an action.

3. The two-stage credit allocation method based on multi-game unit cooperation modeling as described in claim 1, characterized in that, Step (2.2) is as follows: Phase 1: Game units interact with the environment and collect data. And store it in the experience playback buffer, where For trajectory, For joint operations, This represents the global state at the current time step. For global rewards, This represents the global state at the next time step. Sample batch data from the buffer (batch size is...) The parameters of the central commentator network are updated using temporal difference loss. : （12）； in, For the target value, As a discount factor, For the trajectory at the next time step, For the joint action at the next time step, For the target commenter's network parameters, The overall value of the actions delivered by the central commentators; Phase Two: Individual Credit Allocation and Strategy Update, Based on Updated Central Commentator Output The approximate individual credit of each game unit is calculated using the weighted sampling method in step (2.1). Depending on the integration method type, the strategy network of each game unit is updated using formula (11) or formula (12).

Images