A hybrid strategy optimization system and method based on game sample equilibrium

Abstract

The application relates to a mixed strategy optimization system and method based on game sample equilibrium, which comprises at least one independent and parallel environment interaction module, at least one strategy generation module and a training optimization module. The environment interaction module is used for interacting with a multi-agent environment, obtaining environment state observation information and executing a received joint action to obtain an environment feedback reward; the strategy generation module is used for receiving the environment state observation information, generating a joint action according to the environment state observation information and sending the joint action to the corresponding environment interaction module, and storing experience samples generated by interaction into an experience pool; the training optimization module is used for extracting a balanced training sample set special for each agent group from the experience pool through a game sample equilibrium sampling algorithm; and after centralized updating of strategy network model parameters of each agent group based on the balanced training sample set, the updated strategy network model parameters are synchronously distributed to each strategy generation module.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a hybrid strategy optimization system and method based on game sample equilibrium. Background Technology

[0002] In a single-agent environment, the state space describes all possible states of the environment in which the agent exists. The action space describes all possible actions the agent can perform. The size of these two spaces depends on the complexity of the environment. However, in a multi-agent environment, each agent has its own state and action space. The state space consists of the states of all agents, and the action space consists of the actions each agent can perform. This causes the dimensions of the state and action spaces in a multi-agent environment to increase with the number of agents. If there are N agents, the state space of the environment consists of the joint states of all N agents, and the action space also expands due to multiple agents performing actions simultaneously. This results in the state and action spaces of a multi-agent environment typically being larger and more complex than those of a single-agent environment. Therefore, for more complex multi-agent game environments, policy acceleration optimization can improve the efficiency of acquiring learning samples and accelerate the convergence speed of the algorithm, thus enabling multi-agent policies to better cope with complex environments.

[0003] However, in the process of multi-agent policy learning, the mismatch between the speed of sample generation and policy learning, as well as the incomplete evaluation of game samples, leads to asynchronous policy learning among agents, making it difficult to reach game equilibrium in a short time. On the one hand, the speed of generating experience samples for reinforcement learning agents is often mismatched with the speed of experience sampling and training, resulting in a lack of learnable experience and slow updating of old experience, thus causing inefficient agent training.

[0004] Figure 4 This study demonstrates the asynchronous nature of game experience generation and sampling training speeds in both single-agent and multi-agent environments. Figure 4 In this process, the agent generates learnable experience samples by interacting with the environment and stores these experience samples in an experience pool of 400. This experience generation process is performed on the CPU and runs in a single-threaded manner. Figure 4 The experience sampling training process is performed on the GPU, with computational acceleration achieved by extracting data from an experience pool of 400. The state and action spaces of a multi-agent environment are more dimensional than those of a single-agent environment. The complexity of multi-agent environments increases the computational load on the CPU performing environment interactions in a single thread, limiting the rate of experience sample generation. GPU-based experience sampling training is more efficient than CPU-based experience generation. Figure 4This reflects the asynchronous data throughput of the two hardware components at different computational stages. This mismatch between generation and consumption rates results in a limited number of new samples available for GPU learning in the experience pool of 400, while older samples within the experience pool are not promptly covered and updated. At the sample utilization level in the game environment, the same experience samples have different values ​​for different agent groups. Inappropriate sample evaluation and sampling methods make it difficult for the strategies of each agent group to be optimized synchronously, leading to a unilateral disadvantage. The imbalance in data throughput rates at the hardware level and the limitations of the evaluation methods at the sampling level combine to cause overall inefficient agent training, ultimately leading to the failure of synchronous learning of game strategies.

[0005] Figure 5 It demonstrates the reward fluctuations and competitive game relationship between the defender and the intruder during the strategy learning process in the unmanned surface vessel target protection mission. Figure 5 In the diagram, the horizontal axis represents training time, reflecting the dynamic evolution of the strategy as the game progresses. The vertical axis represents reward, reflecting the immediate performance or score achieved by the player at the corresponding moment. Figure 5 It contains two curves representing the average reward learning curve for the defender and the average reward learning curve for the invading side, respectively.

[0006] like Figure 5 As shown, in the unmanned surface vessel (USV) target defense mission, both the defender and the intruder learn strategies simultaneously. During the time interval t1 to t2, the intruder's average reward learning curve is at a higher position, indicating that the intruder's strategy is consistently superior. This results in most samples in the experience pool being favorable to the intruder's strategy learning. Therefore, if sample evaluation and sampling are not conducted from the perspective of both competing parties, the intruder's strategic advantage will become increasingly larger in subsequent learning, leading to a one-sided policy dominance. This situation violates the game training objective of mutual practice and iterative optimization, ultimately preventing either the intruder or the defender from learning the optimal strategy. Figure 5 By demonstrating the strategy imbalance caused by incomplete game experience analysis, this invention proves that the sampling algorithm proposed in this paper, which is based on agent grouping for differentiated evaluation, is necessary for solving the sample distribution imbalance and achieving synchronous optimization of game strategies.

[0007] In accelerating agent policy optimization, existing methods primarily improve policy learning efficiency through distributed parallel training and empirical importance evaluation. Regarding distributed parallel training, most works leverage the multi-threading capabilities of CPUs and the multi-node parallelism of distributed servers to increase the number of interactive agent environments, thereby enhancing the ability to acquire training experience in a short time. These methods have been widely applied in single-agent policy learning; however, the complexity of multi-agent environments, especially the intricate relationships between different groups in cooperative-competitive tasks, makes existing distributed parallel methods difficult to apply directly. Regarding sample importance evaluation, most current single-agent and multi-agent reinforcement learning methods primarily evaluate empirically from the perspectives of training phase priority and the importance of team collaboration. In adversarial game environments, the competitive relationships between groups of agents result in different importance for the same sample to different groups (e.g., Figure 5 As shown in the figure, this poses new challenges to the evaluation of game samples, and current work has basically not considered how to evaluate samples for different groups. Driven by the above problems and the current state of research in the field, the technical problem to be solved by this invention is how to accelerate the sample generation speed and improve the sample utilization efficiency in the policy learning process, overcome the large state space and the imbalance of empirical samples, and achieve accelerated policy optimization. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a hybrid strategy optimization system based on game sample equilibrium from a first aspect. The system adopts a distributed parallel architecture that separates environment interaction and strategy generation. The system includes at least one independent parallel environment interaction module, at least one strategy generation module, and a training and optimization module. The environment interaction module interacts with the multi-agent environment, acquires environmental state observation information, and executes joint actions received from the policy generation module to obtain rewards from environmental feedback. A single policy generation module connects to multiple environment interaction modules. The policy generation module receives environmental state observation information from its corresponding connected environment interaction modules, generates joint actions accordingly, sends them to the corresponding environment interaction modules, and stores the experience samples generated by the interaction, including environmental state observation information, joint actions, and environmental feedback rewards, into an experience pool. The training optimization module incorporates a game-theoretic sample equilibrium sampling algorithm based on agent grouping for differentiated evaluation. The training optimization module extracts a unique equilibrium training sample set for each agent group from the experience pool using this algorithm. Based on the equilibrium training sample set, it centrally updates the policy network model parameters for each agent group and then synchronously distributes the updated policy network model parameters to each policy generation module. The central update includes: adjusting weights when calculating the loss function based on the sampling probability of each sample in the equilibrium training sample set.

[0009] According to a preferred embodiment, the training optimization module includes a differential priority calculation unit to implement a game sample equilibrium sampling algorithm; the differential priority calculation unit is configured to: calculate the time difference priority of experience samples in the experience pool using the state value evaluation network; calculate the importance assessment of the grouping strategy of the experience samples to different groups based on the specific policy objectives of each agent group; and fuse the time difference priority with the grouping strategy importance assessment to obtain the sampling priority of each agent group corresponding to the experience samples.

[0010] According to a preferred embodiment, the differential priority calculation unit in the training optimization module performs the following steps when calculating the importance assessment of the grouping strategy: extracting the cumulative reward and key time indicators of the task of interacting with the environment from the experience samples of the agent group; and calculating the importance assessment of the grouping strategy for a specific agent group based on the differential requirements of the agent group for time indicators in the cooperative-competitive game, combined with the cumulative reward, using the grouping differential function.

[0011] According to a preferred embodiment, the training optimization module further includes a group-independent sampling unit; the group-independent sampling unit is configured to: construct completely independent priority sampling data structures for multiple agent groups in the game environment; before extracting the exclusive balanced training sample set for each agent group, the group-independent sampling unit updates and stores the calculated sampling priority of each agent group into the corresponding priority sampling data structure of the agent group; and extract the exclusive balanced training sample set by performing hierarchical interval sampling on different priority sampling data structures.

[0012] According to a preferred embodiment, the training optimization module includes a state value evaluation unit and a policy optimization unit; the state value evaluation unit is configured with a state value evaluation network, which uses the global state space information of the environment as input to perform centralized state value evaluation to calculate the temporal difference loss; the policy optimization unit is configured with a policy network, which uses the local observation information of a single agent as input, combines the state value evaluation to calculate the cumulative policy advantage for gradient update; and agents within the same agent group share the policy network model parameters in the policy network.

[0013] According to a preferred embodiment, the policy optimization unit performs gradient updates based on the proximal policy optimization algorithm; the training optimization module uses the extracted balanced training sample set specific to each agent group, and through the loss function of the proximal policy optimization algorithm, controls the policy optimization unit to independently calculate and update the policy network model parameters of each agent group.

[0014] According to a preferred embodiment, the step of the policy generation module generating a joint action includes: obtaining the local observation information of each agent constituting the environmental state observation information at the current time step transmitted by the environmental interaction module; mapping the local observation information of the agents to corresponding individual policy actions based on a preset policy network; and aggregating the individual policy actions generated by all agents in the current time step to generate a joint action for sending to the environmental interaction module.

[0015] This invention provides a hybrid policy optimization method based on game-theoretic sample equilibrium from a second aspect. The method includes: acquiring environmental state observation information and executing joint actions to obtain rewards from environmental feedback; generating joint actions based on environmental state observation information and storing the experience samples generated by the interaction, including environmental state observation information, joint actions, and environmental feedback rewards, into an experience pool; extracting a set of equilibrium training samples specific to each agent group from the experience pool using a game-theoretic sample equilibrium sampling algorithm; and centrally updating the policy network model parameters of each agent group based on the equilibrium training sample set; the central update includes: adjusting the weights according to the sampling probability of each sample in the equilibrium training sample set when calculating the loss function.

[0016] According to a preferred embodiment, the step of weight correction includes: calculating the importance sampling weights of each sample in the balanced training sample set, and weighting the loss function using the importance sampling weights.

[0017] According to a preferred embodiment, the step of weighting the loss function using importance sampling weights includes: multiplying the temporal difference error of the corresponding sample by the importance sampling weight of the sample.

[0018] According to a preferred embodiment, the method further includes: calculating the temporal differential priority of experience samples in the experience pool using a state value assessment network; calculating the importance assessment of the experience samples to the grouping strategies of different groups in a differentiated manner according to the specific policy objectives of each agent group; and fusing the temporal differential priority with the grouping strategy importance assessment to obtain the sampling priority of each agent group corresponding to the experience samples.

[0019] According to a preferred embodiment, the method further includes: constructing completely independent priority sampling data structures for multiple agent groups in the game environment; before extracting the exclusive equilibrium training sample set for each agent group, updating and storing the calculated sampling priority of each agent group into the priority sampling data structure of the corresponding agent group; and extracting the exclusive equilibrium training sample set by performing hierarchical interval sampling on different priority sampling data structures. Attached Figure Description

[0020] Figure 1This is a schematic diagram illustrating the overall approach to hybrid strategy optimization based on game sample equilibrium provided by the present invention. Figure 2 This is a schematic diagram of the logic modules of the hybrid strategy optimization system based on game sample equilibrium provided by the present invention; Figure 3 This is a schematic diagram of the equilibrium process of a game sample that integrates time difference and strategy importance, provided by the present invention. Figure 4 This is a schematic diagram illustrating the asynchronous speed of game experience generation and sampling training provided by the present invention; Figure 5 This is a schematic diagram illustrating the incomplete game theory experience analysis provided by this invention; Figure 6 This is a schematic diagram of the algorithm process of the game sample sampling algorithm based on SumTree provided by the present invention; Figure 7 This is a schematic diagram of the cooperative-competitive hybrid strategy acceleration optimization algorithm based on game sample equilibrium provided by the present invention; Figure 8 This is a schematic diagram of a three-agent collaborative-competitive task environment provided by the present invention; Figure 9 This is a schematic diagram comparing the average reward learning curves of the PPO algorithm and the MSGEE algorithm provided by this invention in a predator-prey environment; Figure 10 This is a schematic diagram comparing the average reward learning curves of the MADDPG algorithm and the MSGEE algorithm provided by this invention in a predator-prey environment; Figure 11 This is a schematic diagram comparing the average reward learning curves of the SVDE algorithm and the MSGEE algorithm provided by this invention in a predator-prey environment; Figure 12 This is a schematic diagram comparing the average reward learning curves of the MSGEE algorithm and the PPO algorithm in a robot soccer environment provided by this invention. Figure 13 This is a schematic diagram comparing the average reward learning curves of the MADDPG algorithm and the MSGEE algorithm provided by this invention in a robot soccer environment; Figure 14 This is a schematic diagram comparing the average reward learning curves of the SVDE algorithm and the MSGEE algorithm provided by this invention in a robot soccer environment; Figure 15 This is a schematic diagram comparing the average reward learning curves of the PPO algorithm and the MSGEE algorithm provided by this invention in a multi-unmanned surface vessel target guarding environment; Figure 16This is a schematic diagram comparing the average reward learning curves of the MADDPG algorithm and the MSGEE algorithm provided by this invention in a multi-unmanned surface vessel target guarding environment; Figure 17 This is a schematic diagram comparing the average reward learning curves of the SVDE algorithm and the MSGEE algorithm provided by this invention in a multi-unmanned surface vessel target guarding environment; Figure 18 This is a schematic diagram of the average reward learning curves of the MSGEE method and the MSGEE-NMS variant method in a multi-unmanned surface vessel target guarding environment provided by the present invention; Figure 19 This is a schematic diagram of the average reward learning curves of the MSGEE method and the MSGEE-TD variant method in a multi-unmanned surface vessel target guarding environment provided by the present invention; Figure 20 This is a schematic diagram of the average reward learning curves of the MSGEE method and the MSGEE-UNIFORM variant method in a multi-unmanned surface vessel target guarding environment provided by the present invention; Figure 21 This is a schematic diagram of the average reward learning curves of the MSGEE method and the MSGEE-NMS variant method in the predator-prey environment provided by this invention; Figure 22 This is a schematic diagram of the average reward learning curves of the MSGEE method and the MSGEE-TD variant method in the predator-prey environment provided by this invention; Figure 23 This is a schematic diagram of the average reward learning curves of the MSGEE method and the MSGEE-UNIFORM variant method in the predator-prey environment provided by this invention; Figure 24 This is a schematic diagram of the average reward learning curves of the MSGEE method and the MSGEE-NMS variant method in the robot soccer environment provided by this invention; Figure 25 This is a schematic diagram of the average reward learning curves of the MSGEE method and the MSGEE-TD variant method in the robot soccer environment provided by this invention; Figure 26 This is a schematic diagram of the average reward learning curves of the MSGEE method and the MSGEE-UNIFORM variant method in the robot soccer environment provided by this invention.

[0021] List of reference numerals 100: Environment interaction module; 200: Policy generation module; 300: Training optimization module; 310: Differentiated priority calculation unit; 320: Group independent sampling unit; 330: State value evaluation unit; 340: Policy optimization unit; 400: Experience pool. Detailed Implementation

[0022] The following is a detailed explanation with reference to the accompanying drawings.

[0023] This invention explains and illustrates the Multi-Scene Multi-Agent Mixed Cooperative-Competitive Markov Game (MSMMG), the Predator-Prey model, the Robot Soccer model, and the Multi-Unmanned Surface Vessel Target Guardian model.

[0024] The MSMMG process can be defined as a seven-tuple. M represents the number of multi-agent game scenarios, N represents the number of agents in each game scenario, and S represents the global state space of the scenario. This represents the state transition probability of a game scenario. K indicates that all agents in each game scenario are divided into K groups. This represents the set of local observation information of all agents. It is a joint action of all intelligent agents.

[0025] Let m represent the set of reward functions for all groups. In each scenario m, the sum of the rewards for all groups is zero. .

[0026] During the MSMMG process, at each decision time t, the agent... Each game scenario m will be derived from its state space. Obtain your own observation information And based on the strategies shared within the group. Actions to be performed All agents simultaneously perform joint actions. And receive the reward for the current moment. For each intelligent agent In other words, its learning objective is to continuously optimize strategies. Maximize the cumulative reward of its group Where T represents the maximum time step and γ represents the reward decay factor. (Agent) The objective function can be defined as: .

[0027] In the above formula, This represents the joint strategy of all groups except the agent group.

[0028] Predator-Prey Model: The environment consists of a predator group and a prey group. The average reward learning curves of the two groups of agents are symmetrical. Predators gain positive rewards by capturing prey, such as 1 hunting point for each prey captured. Prey reduce negative rewards by taking evasive actions. During the experimental training phase, the maximum length of each round is set to 500 time steps.

[0029] Robot Soccer Model: In this environment, agents are divided into two teams, blue and purple. The goal for both sides is to cooperate to kick the ball into the opponent's goal to gain a positive reward, while simultaneously defending to avoid being penalized for scoring. The average reward learning curve for both sides exhibits an alternating lead during training. In this experimental environment, the maximum timesteps per round are set to 1000.

[0030] Multi-UAV Target Guarding Model: This environment includes two groups of agents: a guard and an intruder. Both sides learn their strategies simultaneously during training. The intruder group's goal is to make contact with the guarded target. When the intruder collides with the guarded target, it is considered a successful contact, and the intruder group scores 1 point. The maximum time step for each round in this model is set to 500 steps.

[0031] Complex Unmanned Surface Vehicle (USV) Game Scenario: This environment is built using Unity software. Agent teams improve their total reward through two conventional methods. One method is acquiring reward points in the environment; each agent touching a reward point adds 2 points to the team's reward. The other method is eliminating opponent USVs; each eliminated opponent USV adds 5 points to the team's reward. When all opponent USVs are eliminated, the team receives an additional 10 points and ends the current round. StarCraft Multi-Agent Challenge (SMAC, Homogeneous Game): This includes multiple micro-management adversarial maps, such as 5m_vs_6m (your 5 infantry versus the enemy's 6 infantry, at a numerical disadvantage), 10m_vs_11m, and the large-scale 25m (25 infantry on each side). The agent's action space includes discrete up, down, left, and right movement actions, as well as actions to attack enemies with specific numbers.

[0032] The basic idea of ​​this invention is: To address the challenges of slow sample generation and incomplete evaluation in multi-agent adversarial game tasks, which hinders policy optimization, this invention proposes a multi-scene gaming experience evaluation (MSGEE) method based on game sample equilibrium, also known as a hybrid policy optimization system and method based on game sample equilibrium. This method improves the speed of experience sample generation through a multi-scenario accelerated generation architecture for game samples and enhances sample training efficiency by performing grouped equilibrium evaluation of game samples, thereby achieving accelerated policy optimization. The overall idea is as follows: Figure 1 As shown in the diagram. Specifically, MSGEE first separates the scenario interaction process from the policy generation process, while considering the game relationships between different groups, to achieve parallel generation of multi-scenario game samples. Then, it evaluates the empirical samples multiple times from the perspective of different game groups through time difference and policy effectiveness, and performs multiple sampling using SumTree. Finally, based on the evaluation results of multi-scenario game samples, it achieves efficient training of multi-agent policies by sharing intra-group game samples and policy models.

[0033] The main contributions of this invention include: 1) Regarding game sample generation, this invention designs an accelerated generation architecture for game experience samples based on multi-scenario parallel deduction. Parallel generation of game samples is achieved through a policy generation module 200 and an environment interaction module 100. Compared to current distributed learning methods that only consider single agents and collaborative agents, the multi-scenario game sample accelerated generation architecture forms the policy generation module 200 by integrating policy networks (Actor networks) corresponding to K agent groups. While considering multiple sets of game adversarial processes, it separates the scenario interaction process from the policy generation process, thereby accelerating the generation of samples for multiple sets of game adversarial processes.

[0034] 2) Regarding game sample equilibrium, this invention proposes a game sample equilibrium model that integrates time difference and strategy importance. It evaluates multiple sets of game samples multiple times from both time difference and strategy importance perspectives, supporting strategy training optimization for different groups. Compared to current methods that evaluate samples based on training phase priority and team collaboration importance, this invention considers the differences in importance of the same experience samples to different game groups, improving sample effectiveness and utilization through group evaluation and group sampling.

[0035] 3) Regarding multi-agent policy training optimization, this invention designs a cooperative-competitive hybrid policy acceleration optimization method based on game sample equilibrium. This method combines a centrally trained proximal policy optimization network with a multi-scenario sample generation architecture, utilizing multi-scenario game samples for policy training. By sharing intra-group game samples and policy models, it achieves efficient training of multi-agent policies. In experiments, this invention's method outperforms benchmark methods in predator-prey, robot soccer, and multi-UAV target defense tasks. In particular, in multi-UAV target defense, it reaches game equilibrium in approximately 20,000 rounds (episodes), which is about 5,000-10,000 rounds (episodes) faster than the benchmark method.

[0036] Example 1 This invention provides a detailed description of a hybrid strategy optimization system and method based on game sample equilibrium.

[0037] Compared to general multi-agent environments, multi-agent game environments present significant challenges to the training and optimization of agent policies due to their higher complexity. Furthermore, the simultaneous learning of multiple agent policies places higher demands on the efficiency of generating experience samples. Existing work primarily utilizes distributed parallel frameworks, leveraging CPU multi-threading and the parallel computing capabilities of distributed servers to increase the number of experience samples acquired in a short time. In existing technologies, SEED separates M environment interaction modules 100 from the training and optimization module 300 during agent policy learning. The environment interaction modules 100 interact with the agent environment, sending observation information to the training and optimization module 300. The task of policy generation in the environment interaction modules 100 is completely transferred and centralized in the training and optimization module 300, which is responsible for generating policies, sending policy actions to the environment interaction modules 100, and updating the policy model network. This structure requires the training and optimization module 300 to simultaneously handle action policy generation during interaction and subsequent experience training and updates. This setup places a significant burden on the training and optimization module 300, and the number of environment interaction modules 100 that a single training and optimization module 300 can support is very limited.

[0038] Methods such as IMPALA integrate the environment interaction and policy generation modules 200, meaning the policy generation module 200 is included in each environment interaction module 100 and centrally learned in the training and optimization module 300. While performing environment interaction tasks, the environment interaction module 100 needs to run a policy model for action inference. This architecture results in significant memory consumption and computational overhead for each environment interaction node. The policy model in the environment interaction module 100 requires frequent batch updates, which reduces the overall sample generation speed of the framework. High-frequency model parameter synchronization increases network communication bandwidth consumption. The allocation of computational resources between environment simulation and policy inference tasks leads to a decrease in the efficiency of producing experience samples per unit time. This integrated architecture limits the parallel scaling of the environment interaction module 100.

[0039] The existing SVDE algorithm combines the advantages of the above framework and significantly improves the efficiency of generating experience samples by layering the environment interaction module, policy generation module, and training optimization module.

[0040] Building upon the aforementioned work, this invention proposes a framework for accelerating game sample generation in adversarial game environments, based on parallel multi-scenario deduction. Figure 1The overall framework construction approach is demonstrated. Considering the characteristic of multiple agents learning policies simultaneously in a game environment, this framework simplifies the policy generation module 200 by sharing a policy network within each group. Simultaneously, by separating the scene interaction process from the policy generation process, it accelerates the generation of experience samples for the policy learning process of multiple agents. The hybrid policy optimization system based on game sample equilibrium of this invention adopts a distributed parallel architecture that separates environment interaction and policy generation. The system includes at least one independent parallel environment interaction module 100, at least one policy generation module 200, and a training and optimization module 300.

[0041] At least one environment interaction module 100 establishes a bidirectional data communication connection with a corresponding policy generation module 200. A single policy generation module 200 may be connected to multiple environment interaction modules 100.

[0042] like Figure 2 As shown, the data output terminals of the strategy generation module 200 establish unidirectional communication connections with the data writing terminals of the experience pool 400. The data reading terminals of the experience pool 400 establish unidirectional or bidirectional communication connections with the training and optimization module 300.

[0043] like Figure 2 As shown, the parameter sending end of the training optimization module 300 establishes a broadcast or point-to-point communication connection with the model update interface of the policy generation module 200.

[0044] like Figure 2 As shown, the data output terminal of the state value assessment unit 330 is unidirectionally connected to the data input terminal of the differential priority calculation unit 310. The output terminal of the differential priority calculation unit 310 is unidirectionally connected to the structure update input terminal of the group independent sampling unit 320. The data distribution terminal of the group independent sampling unit 320 establishes parallel communication connections with the training input terminals of both the state value assessment unit 330 and the policy optimization unit 340. The value calculation output terminal of the state value assessment unit 330 establishes a communication connection with the calculation input terminal of the policy optimization unit 340. As the final output terminal of the training optimization module 300, the parameter synchronization interface of the policy optimization unit 340 is communicatively connected to the network parameter update interface of the policy generation module 200 outside the system.

[0045] The environment interaction module 100 is used to interact with the multi-agent environment, obtain environmental state observation information, and execute joint actions received from the policy generation module 200 in order to obtain rewards from environmental feedback.

[0046] Preferably, a graph convolutional network can be further connected after the environment interaction module 100 for dynamic feature extraction. This network models entities and environmental obstacles in the multi-agent environment as graph nodes, and extracts features with spatial topological relationships through multi-layer graph convolution operations to improve the perception accuracy of the agent in complex environments.

[0047] Specifically, the environment interaction module 100 is responsible for interacting with a specific environment, that is, obtaining global state, local observation information, and reward information from the environment, and delivering the joint actions output by the policy generation module 200 to the various agents in the environment for execution. M×M environment interaction modules 100 can be set up in parallel within the entire multi-scenario game sample acceleration generation framework, with each module operating independently. For the environment interaction module... It obtains the current global state from the environment at time t. Joint observation with all intelligent agents : .

[0048] The joint action of all agents is obtained through the corresponding policy generation module m. : .

[0049] After the joint action is executed, obtain the rewards from the environment for all agents. : .

[0050] In the above formula, N and K represent the number of agents and agent groups in a single environment interaction module 100, respectively.

[0051] The policy generation module 200 is used to receive environmental state observation information from its corresponding connected environmental interaction module 100. It should be noted that in the environmental state observation information, the local observation information of each agent is dedicated to inputting into the policy generation module 200 for decentralized action generation, while the global state space information is dedicated to being sent to the training optimization module 300 for centralized value evaluation. Based on this, joint actions are generated and sent to the corresponding environmental interaction module 100. The experience samples generated by the interaction, including environmental state observation information, joint actions, and environmental feedback rewards, are stored in the experience pool 400.

[0052] According to a preferred embodiment, the step of generating a joint action by the policy generation module 200 includes: acquiring the local observation information of each agent constituting the environmental state observation information at the current time step transmitted by the environment interaction module 100; mapping the local observation information of the agents to corresponding individual policy actions based on a preset policy network; and aggregating the individual policy actions generated by all agents in the current time step to generate a joint action for sending to the environment interaction module 100.

[0053] The policy generation module 200 is responsible for generating policy actions for different groups of agents, that is, obtaining the observation information of each agent from the environment interaction module 100 and outputting the joint actions of all agents.

[0054] Figure 1 This paper presents the overall approach to hybrid strategy optimization based on game sample equilibrium, including a multi-scenario accelerated generation architecture for game samples based on parallel extrapolation in multiple scenarios, a game sample equilibrium sampling algorithm that integrates time difference and policy importance, and a process for centralized updating and synchronous distribution of policy network model parameters.

[0055] The game sample acceleration generation module based on multi-scenario parallel simulation adopts a distributed parallel architecture that separates environment interaction and strategy generation. Figure 1 The underlying layer deploys M×M independent parallel environment interaction modules 100, and the middle layer deploys M policy generation modules 200. Each policy generation module 200 is connected to M environment interaction modules 100. This process can be effectively accelerated using GPUs. Each policy generation module 200 contains K policy generation networks labeled Actor 1 to Actor K, corresponding to the K intelligent agent groups in the game scenario. The data stream from the environment interaction module 100 to the policy generation module 200 represents the global state, local observation information, and environmental feedback rewards. Figure 1 The middle label is to Each agent group shares the policy network model parameters. M policy generation modules 200 and M×M environment interaction modules 100 jointly execute the multi-scenario experience sample generation process. The environment interaction module 100 interacts with the multi-agent environment to obtain environmental state observation information and executes joint actions. The policy generation module 200 receives the environmental state observation information and generates joint actions, sending them to the corresponding environment interaction module 100. For example... Figure 1 As shown, the data stream from the strategy generation module 200 downwards to the environment interaction module 100 represents the generated joint action. Figure 1 The middle label is to The experience samples generated from interactions in each scenario are uniformly aggregated into black arrows labeled "Episodes" and output upwards. The strategy generation module 200 uniformly sends the experience samples generated from interactions in each scenario to the experience pool 400 for storage and experience evaluation.

[0056] Furthermore, the experience pool 400 can be logically divided into a regular sample pool and a high-win-rate sample pool. When extracting a balanced training sample set, the training optimization module 300 performs mixed sampling from the two sample pools according to a preset ratio, and adopts a pseudo-reward function based on edit distance bias in the high-win-rate sample pool, thereby enhancing the agent's efficiency in utilizing key success experiences.

[0057] Experience pool 400 receives the experience sample stream of Episodes. For example... Figure 1 and Figure 3 As shown, the input sample sequences in experience pool 400 are represented as those containing letter labels. The data blocks. The sample importance assessment layer extracts corresponding assessment data for each experience sample, such as... Figure 1 and Figure 3 As shown, each sample square points upwards to the area containing the letter label. The physical computation steps for evaluating the importance of a sample are actually executed by the State Value Evaluation Network (Critic Network) called by the Differentiated Priority Calculation Unit 310 within the Training Optimization Module 300. The Experience Pool 400 only serves as a passive storage structure, providing a data interface to receive and update the final sampling priority of the corresponding experience samples. A SumTree sampling data structure is configured above the nodes. The SumTree sampling data structure presents a binary tree topology where the bottom-level nodes converge upwards to the plus sign nodes. The Experience Pool 400 uses the SumTree sampling data structure to extract a balanced training sample set specific to each agent group, and labels it with letters. The data flow arrows are output to the training optimization module 300, such as... Figure 1 and Figure 3 As shown. The above design not only overcomes the shortcomings of the existing technology, but also... Figure 3 The framework is extended to game-theoretic scenarios, avoiding the problem of high complexity of the policy generation module due to the increase in the number of agents.

[0058] The training optimization module 300 incorporates a game-theoretic sample equilibrium sampling algorithm for differentiated evaluation based on agent grouping. The module extracts a balanced training sample set specific to each agent group from the experience pool 400 using this algorithm. After centrally updating the policy network model parameters for each agent group based on the balanced training sample set, the updated parameters are synchronously distributed to each policy generation module 200. Because non-uniform sampling alters the sample distribution, importance sampling weights are used for gradient correction when calculating the loss function, multiplying the TD-error of the corresponding sample by the importance sampling weight.

[0059] like Figure 1 As shown, the training and optimization module 300 performs experience evaluation and policy network model parameter updates to accelerate optimization using a cooperative-competitive hybrid strategy based on game sample equilibrium. The training and optimization module 300 calculates the importance of experience sample sequences in the experience pool 400 by combining time difference and policy effectiveness. The training and optimization module 300 calculates the importance of experience samples in the experience pool 400 from the perspective of different game groups. The evaluation process uses a state value evaluation network to calculate the time difference priority of experience samples, calculates the importance of grouped policies by combining the specific policy objectives and survival time differences of each agent group, and obtains the sampling priority of each agent group for the experience samples through data fusion.

[0060] like Figure 1 As shown, the training optimization module 300 configures the Actor network and the Critic network. The training optimization module 300 centrally updates the parameters of the Actor network and the Critic network for each agent group based on a balanced training sample set. When calculating the network loss function corresponding to the central update, the extracted sample distribution deviates from the true probability distribution of the experience pool 400 because the balanced training sample set uses a non-uniform game-theoretic sample equalization sampling algorithm. To eliminate this distribution bias caused by biased sampling and prevent the Actor network and the Critic network from getting stuck in local optima or diverging during training, the training optimization module 300 dynamically adjusts the weights of the network loss function according to the sampling probability of each sample in the balanced training sample set, thereby ensuring the unbiasedness of gradient updates. Specifically, the training optimization module 300 calculates the importance sampling weights of each sample in the balanced training sample set and uses these importance sampling weights to weight and adjust the loss functions of the Actor network and the Critic network.

[0061] Preferably, when adjusting the weighted loss function of the Critic network, the training optimization module 300 directly multiplies the temporal difference error (TD-error) of the corresponding sample extracted from the experience pool 400 by the importance sampling weight calculated for that sample, so as to achieve accurate sample distribution compensation.

[0062] Training optimization module 300 will update the policy network model parameters The parameters are synchronously distributed to each strategy generation module 200, thus achieving a closed loop of parameter synchronization and strategy optimization.

[0063] Specifically, the training and optimization module 300 is responsible for training and optimizing all policy models. This invention primarily uses the on-policy Proximal Policy Optimization (PPO) algorithm for policy training. During training, experience samples obtained from multiple parallel environment interaction modules 100 are used to train each policy model group through multiple sampling methods, thereby ensuring the sample utilization rate for training and optimizing different policy groups. After several rounds of training, the training and optimization module 300 distributes the trained model parameters to each policy generation module 200, synchronizing the parameters of each policy generation module 200.

[0064] The samples generated in accelerated multi-scenario learning inevitably contain low-quality or even invalid samples. Traditional reinforcement learning methods using uniform sampling obviously reduce training efficiency, and how to evaluate sample quality has always been a cutting-edge research problem in the field of reinforcement learning. In game-theoretic scenarios, learning multi-agent policies means that the same sample has different importance for different groups (see...). Figure 5 Differentiated evaluation of samples from different groups is crucial for efficient policy learning. Therefore, this invention proposes a game-theoretic sample equilibrium method that integrates time difference and policy importance. By using differentiated evaluation and multiple sampling, it provides high-quality samples for training and optimizing policy models for different groups, thereby improving the efficiency of policy training and optimization.

[0065] According to a preferred embodiment, the training optimization module 300 includes a differential priority calculation unit 310 to implement a game sample equilibrium sampling algorithm. The differential priority calculation unit 310 uses the state value evaluation network to calculate the temporal differential priority of experience samples in the experience pool 400; based on the specific policy objectives of each agent group, it differentially calculates the importance assessment of the experience samples to the grouping strategies of different groups; and it fuses the temporal differential priority with the grouping strategy importance assessment to obtain the sampling priority of each agent group corresponding to the experience samples.

[0066] Temporal Difference Error (TD-error) is widely used in typical reinforcement learning algorithms such as DQN, DDPG, SAC, and PPO, and is generally used as the loss function for backpropagation updates. TD-error directly reflects the importance of empirical samples to the current network training and optimization; a larger TD-error indicates more parameters that the network can update, and a more effective approach to training and optimizing the current policy. Most existing reinforcement learning sample evaluation methods use TD-error as the basis for evaluating sample importance. Addressing this limitation, this invention first calculates the priority of different empirical samples based on TD-error.

[0067] like Figure 3 As shown, each sample in the experience pool of 400 contains sample data generated from T interactions in a scenario. For each agent group in the experience samples, its sampling priority is calculated using the following function. : .

[0068] Where γ represents the discount factor for the reward, Let represent the state value evaluation network for the k-th agent group. The sampling priority reflects the importance of empirical samples for the policy training of the agent group, and the calculation method of the sampling priority is shared for each agent group.

[0069] The differential priority calculation unit 310 in the training optimization module 300 performs the following steps when calculating the importance assessment of the grouping strategy: extracting the cumulative reward and key time indicators of the task of interacting with the environment for a specific agent group from the experience samples; and calculating the importance assessment of the grouping strategy for a specific agent group based on the differential requirements of the agent group for time indicators in the cooperative-competitive game, combined with the cumulative reward, using the grouping differential function.

[0070] The aforementioned TD-error only reflects the important role of empirical samples in network optimization convergence and directly expresses the importance of agent policies. However, in multi-agent hybrid tasks, different groups of agents have different policy objectives, which directly leads to different policy importance of empirical samples for different groups. Therefore, this invention uses a group-differentiated policy importance evaluation calculation method to calculate the policy importance of empirical samples for different groups, starting from the policy objectives of different agent groups. For each agent group, its policy importance... Calculated using the following formula: .

[0071] In the above formula, This represents the cumulative reward for the agent group in the empirical samples, and the cumulative reward is positively correlated with the importance of the policy. and These represent the existence time and sampling number of the empirical sample, respectively, and are negatively correlated with the importance of the strategy. This paper primarily considers the impact of the length T of the empirical samples on different group strategies, representing a differentiation function that emphasizes the importance of group strategies. For example, in unmanned surface vessel (USV) target defense missions, a larger time step T of the empirical samples is advantageous to the defender and disadvantageous to the intruder. For the defender... For the intruder, .

[0072] In other implementation scenarios, specifically for robot soccer, It can be defined as a monotonically increasing function based on goal difference, encouraging attack by giving higher weight to fast-scoring samples; for the predator-prey scenario, the predator group Set as a decreasing function of capture time (e.g., 1 / T), for the prey group Set as an increasing function of survival time (e.g.) This allows for accurate assessment of different competitive objectives.

[0073] The above steps achieve the calculation of the temporal difference priority of empirical samples and the importance of the grouping strategy, resulting in the final sampling priority. We obtain the result by summing the two aspects: .

[0074] To improve the sampling efficiency of experience samples, this invention uses a SumTree to sample an experience pool of 400. The leaf nodes of the SumTree store the sampling importance weights of the experience samples, and the root node stores the sum of the weights of all leaf nodes. Considering the differential weights among different groups of agents, for an experience pool of L samples of 400, this invention sets up K SumTrees and extracts a balanced training sample set for each agent group through multiple sampling processes. For the group of agents, its balanced training sample set Dk is based on... Figure 6 The algorithm shown performs sampling.

[0075] The specific sampling process is as follows: the total weight interval stored in the root node of the SumTree is evenly divided into... The step size for each line segment is calculated using the following formula: .

[0076] Then, uniform random numbers are generated independently within each step interval, and the corresponding high-priority experience samples are quickly identified and extracted by searching downwards through a tree-like hierarchical structure.

[0077] Figure 6This demonstrates the complete process of a game sample sampling algorithm based on SumTree. The algorithm's input parameters include the experience pool. Experience pool length Number of samples And the set of SumTree node weight arrays for K groups of intelligent agents. After the algorithm completes execution, it will output a balanced training sample set specific to each agent group. .

[0078] The algorithm's execution process begins with initialization, starting with setting the node weight array of the SumTree. Set all values ​​to zero and clear the target balanced training sample set. The algorithm then proceeds to the priority calculation and tree structure construction phase. It employs a two-layer nested loop: the outer loop controls the agent group, with variable K iterating from 0 to K-1; the inner loop controls the samples in the experience pool of 400, with variable K... Iterate from 0 to L-1. For each experience... The algorithm calculates the importance of this experience to the agent group based on the formulas for calculating time difference priority, policy importance, and final sampling priority. Then, the algorithm assigns this value to the corresponding leaf node, making... and initialize node indexes. Next, the algorithm enters a loop with the condition `index / / 2>0`, continuously updating the weight of the parent node upwards and performing the calculation: Tree k [(index-1) / / 2]=Tree k [((index-1) / / 2)*2+1]+Tree k [((index-1) / / 2)*2+2]. After each calculation, set index=(index-1) / / 2, until the weights of the entire tree are updated and the loop exits.

[0079] After updating the weights of all nodes, the algorithm enters the hierarchical sampling phase. The algorithm restarts the outer loop, at which point the variables of the agent group... Iterate from 1 to K. For each agent group, first calculate the line segment step size, let... Then, the inner sampling loop is entered, and the variable b is iterated from 0 to... In each iteration, the algorithm operates within the interval... A value IM is uniformly sampled from the root node, and then... Next, the entry condition is... Downward search loop: If the weight of the left child node satisfies Then search to the left, let Otherwise, search to the right, and let After exiting the search loop and locking the bottom-most leaf node, the algorithm performs the assignment operation. The extracted experience samples are then stored in the training set for that group. This sampling process is repeated until the sample extraction cycle for all groups is completely completed.

[0080] Considering the simultaneous learning of strategies by multiple agents in multi-agent game adversarial tasks, this invention proposes a cooperative-competitive hybrid strategy acceleration optimization method based on game sample equilibrium to reduce learning complexity and improve learning efficiency, building upon the Actor-Critic framework. According to a preferred embodiment, the training optimization module 300 further includes a group-independent sampling unit 320. The group-independent sampling unit 320 is configured to construct completely independent priority sampling data structures for each group of agents in the game environment. Before extracting the equilibrium training sample set specific to each agent group, the group-independent sampling unit 320 updates and stores the calculated sampling priorities of each agent group into the corresponding priority sampling data structure; by performing hierarchical interval sampling on different priority sampling data structures, a specific equilibrium training sample set is extracted.

[0081] like Figure 1 As shown, for N agents that can be divided into K groups in a hybrid task, this embodiment adopts a policy training method that shares network parameters within the group. Furthermore, agents within the same group share the policy network model parameters in the policy network. That is: This represents the set of Actor networks for all groups.

[0082] This represents the set of Critic networks for all groups.

[0083] and Let π and V represent the sets of network parameters, respectively. The policy optimization unit 340 is configured with a policy network, which uses local observation information from a single agent as input, combined with state value assessment, to calculate the cumulative policy advantage for gradient updates. At each time step t, belonging to... Group's intelligent agents Its actions Through π k The network performs the calculations: .

[0084] According to a preferred embodiment, the policy optimization unit 340 performs gradient updates based on the proximal policy optimization algorithm. In multi-agent policy training, the network parameters of all groups are trained and optimized according to the PPO algorithm. Unlike standard PPO, this invention uses an experience pool of 400 samples to improve sample utilization, and the network parameters of all groups are trained in a distributed manner. The training optimization module 300 utilizes the extracted balanced training sample set specific to each agent group, and controls the policy optimization unit 340 to independently calculate and update the policy network model parameters of each agent group through the loss function of the proximal policy optimization algorithm. To eliminate the data distribution bias caused by the game sample balanced sampling algorithm (non-uniform sampling), the training optimization module 300 uses importance sampling weights to weight and correct the gradient when calculating the policy network loss and the state value evaluation network loss, thereby strictly ensuring the unbiasedness of the online gradient of the multi-agent policy and the stability of centralized updates. For agent groups, the sample set... Its Actor network The objective function is: .

[0085] .

[0086] In the above formula, ϵ∈[0,1] is mainly used to control the magnitude of the update between the old and new strategies. express The number of agents in the group Indicates the length of the sample set. The cumulative policy advantage of agent group k at time t is mainly calculated using the following formula: .

[0087] In the above formula, γ represents the discount factor for the reward. and This represents the state value of agent group k.

[0088] According to a preferred embodiment, the training optimization module 300 includes a state value evaluation unit 330 and a policy optimization unit 340. The state value evaluation unit 330 is configured with a state value evaluation network for performing centralized state value evaluation to calculate the time difference loss, using global state space information of the environment as input.

[0089] Since the training of the Critic network directly guides the updating of the Actor network, and the Critic network does not participate in policy generation during the policy generation phase, this embodiment directly uses the global state as the input to the state value evaluation network, thereby improving the accuracy of state value evaluation. For the agent group, the loss function of its Critic network is expressed as the loss function of the agent group on the sample set. Time difference in: .

[0090] Figure 7 The complete Algorithm 2 is shown in the paper, which accelerates the optimization process based on a cooperative-competitive hybrid strategy of game sample equilibrium.

[0091] like Figure 7 As shown, the input parameters of Algorithm 2 include the number of agents N, the number of agent groups K, the multi-scenario parallel parameters M, and the experience pool. Policy network set π and π old The algorithm outputs a set of trained and optimized policies, π*, along with a set of state value evaluation networks, V.

[0092] Algorithm 2 first enters the initialization phase, randomly initializing the policy network parameters. And copy the parameters to the policy network set π old At the same time, the parameters of the state value evaluation network are randomly initialized. Then, the experience pool of 400 is cleared. Subsequently, Algorithm 2 starts the outermost loop that controls the number of rounds, with the variable episode iterating from 1 to max_episode_num. Within each round, the algorithm collects experience through environmental interaction using three nested loops: the first two loops are controlled by variables m1 and m2 respectively, which traverse multiple scenes in parallel, with boundaries from 1 to M; the third loop is a time step loop, with the variable t iterating from 1 to T.

[0093] Within each time step t, Algorithm 2 first obtains the current agent perception information set from the environment interaction module 100. Next, the agent traversal loop is started, with variable i ranging from 1 to N, to calculate the action of agent i at the current moment. After the motion calculation is completed, the combined motion is executed. And obtain the current reward set from the environment. and new sets of sensory information Then, the generated quadruple will be... Stored in experience pool 400. At this point, the nested loops of time step t and parallel parameters m2 and m1 have ended sequentially.

[0094] After the environmental interaction data collection is completed, Algorithm 2 obtains a balanced training sample set for each agent group from the experience pool 400 according to Algorithm 1. Then, the calculation and update phase of network parameters begins. Algorithm 2 starts the agent group update loop, iterating through the variable k from 1 to K. For each agent group... First, calculate the cumulative advantage set A of the group's strategies. k Then it enters the internal training step loop, with variables... Traverse from 1 to In a single training step, the computational agent group... Network loss strategy The parameters of the policy network are updated using gradient descent. Next, the intelligent agent group is calculated. State value assessment of network loss The parameters of the state value evaluation network are updated using gradient descent. After the internal training step loop ends, the assignment operation is performed. After updating all agent groups, the loop for that round ends, and the algorithm continues to execute the next round until the maximum number of rounds is reached.

[0095] Example 2 This embodiment is a further explanation of embodiment 1, and repeated content will not be repeated.

[0096] This embodiment evaluates the algorithm's performance through experimental analysis in several typical multi-agent game environments. Specifically, this embodiment first constructs multiple comparative experimental environments based on a benchmark environment and sets benchmark algorithms and performance evaluation metrics. Then, the performance of MSGEE is analyzed through comparative experiments.

[0097] 1) Experimental Environment To verify the effectiveness of the method proposed in this invention, this embodiment constructs three multi-scenario experimental environments based on Unity3D, focusing on the training and optimization task of a multi-agent cooperative-competitive hybrid strategy. These environments are described below.

[0098] (1) Predator-prey multi-scenario experimental environment Figure 8 As shown, based on Unity3D's parallel scene generation capabilities, this embodiment constructs a multi-scene predator-prey training environment, meaning that one environment instance contains M independent and parallel predator-prey task scenarios. Specifically, as... Figure 8 As shown in the single-task scenario, the experimental environment is set as a two-dimensional space with physical boundaries. Specifically, three predator agents and three prey agents are configured within the scenario, and two obstacles are distributed throughout the environment. In terms of interaction rules, the predators aim to capture the prey through cooperative encirclement, while the prey must use the obstacles to avoid them. The movement trajectories of each agent are constrained by the environmental boundaries and cannot penetrate the obstacles. Due to the limitations of distributed server hardware resources, the above setup simplifies the implementation of the method proposed in this invention, enabling parallel environmental interaction within a single machine.

[0099] (2) Multi-scenario experimental environment for robot soccer Similar to the predator-prey multi-scenario experimental environment, the strategies of the blue and purple sides are learned simultaneously in the robot soccer task. In this embodiment, the parallel interaction process of multiple robot soccer scenarios is realized through Unity3D.

[0100] (3) Multi-unmanned surface vessel target guarding multi-scenario experimental environment In this invention, the strategies of the defender and attacker are learned simultaneously, and like the two environments mentioned above, multiple scenes interact in parallel using Unity3D.

[0101] 2) Benchmark Algorithm In the comparative experiment, this embodiment sets up three benchmark algorithms, which are described in detail below: (1) Proximal Policy Optimization (PPO) algorithm PPO exhibits superior policy learning capabilities in the single-agent domain. In this embodiment, PPO is extended to multi-agent collaborative-competitive hybrid policy learning tasks. During policy training, each agent group shares the PPO network, meaning each group trains a common policy generation network. Furthermore, each group trains independently, using only its own perceptual information and not sharing perceptual information or behavioral policies.

[0102] (2) Multi-agent deep deterministic policy gradient (MADDPG) MADDPG is an extension of the DDPG algorithm in the multi-agent domain and is widely used in multi-agent adversarial game tasks. Similar to PPO, in the comparative experiments, each group of agents shares the MADDPG network, and each agent uses its own local perceptual information to generate policies. The difference is that during policy training, MADDPG uses the perceptual information and behavioral policies of all agents, which is similar to the method proposed in this invention using global state during training.

[0103] (3) Scalable Value-Decomposition Exploration (SVDE) Building upon multi-agent value decomposition methods, SVDE combines distributed experience generation and experience evaluation to accelerate policy training and optimization, achieving leading performance in multiple benchmark environments. SVDE focuses on fully cooperative tasks; therefore, similar to PPO and MADDPG, each agent group shares the SVDE policy generation network, while groups train independently. Furthermore, experience is sampled only once, and this sampled data is used by all agent groups for simultaneous policy training and optimization. In addition to setting benchmark algorithms for comparison, this embodiment also designs four evaluation metrics to assess the performance of MSGEE and the benchmark algorithms. Details are as follows: (1) Average reward: The average reward of each agent group in each episode during the five training sessions. It is mainly used to evaluate the training performance of each method in the benchmark environment. In Table 1, the rows represent the active evaluator (or attacker) and the columns represent the benchmark opponent (or defender), thus constructing a standardized adversarial evaluation coordinate system.

[0104] (2) Average score: This represents the average hunting score of the predator group per episode (an episode contains 500 time steps) in the predator-prey relationship. The predator group receives +1 point for each prey it captures. In the corresponding confrontation results table, the specific numerical distribution of this indicator follows the pattern: rows correspond to the predator group being evaluated, and columns correspond to the prey group it faces.

[0105] (3) Goal win rate: This mainly evaluates the performance of each method in the robot soccer environment, representing the winning probability of the blue team's strategy model in the process of competing against the purple method. In the corresponding competition result table, the rows represent the strategy models that actively conduct win rate evaluations, and the columns represent the opposing models that compete against them.

[0106] (4) Intrusion Score: In a multi-UAV target defense environment, the number of times the intruder group intrudes into the defending target in each episode (the intruder group's score increases by 1 if a collision occurs with the defending target). The maximum time step for each episode is 500. The average intrusion score is calculated using a trained model. In the adversarial matrix, this score is obtained by the intruder group (represented by rows) against the defender group (represented by columns). A higher value indicates a better strategy for the intruder side.

[0107] 3) Performance Comparison This embodiment trains MSGEE and benchmark methods in three experimental environments. The performance of each method during the training phase is evaluated using average reward, and the extrapolation performance is assessed using average score, goal winning percentage, and defense success rate. In the predator-prey environment, the maximum episode length is 20,000, with each episode containing 500 time steps. In the robot soccer environment, the episode length is set to 10,000, with a maximum time step of 1,000 per episode. In the multi-UAV target defense environment, the maximum episode length for each method is 40,000, with a maximum time step of 500 per episode. Figures 9 to 14 and Figures 15 to 17 The average reward learning curves for each method are shown in three experimental environments. In each figure, the shaded area represents the variance and the solid line represents the mean.

[0108] like Figures 9 to 20 As shown, MSGEE outperforms all benchmark methods in terms of average reward learning curve across three experimental environments. Figure 9 , Figure 10 and Figure 11 The average reward learning curves for predators and prey in a predator-prey environment are shown for different algorithms. In the above figures, the horizontal axis represents the training rounds, and the vertical axis represents the corresponding reward value; the algorithms compared include the MSGEE method proposed in this invention, as well as PPO, MADDPG, and SVDE.

[0109] Specifically, in the predator-prey relationship (see...) Figure 9 , Figure 10 and Figure 11 In MSGEE, predators gain positive rewards by capturing prey, while prey reduces penalties by avoiding them. Therefore, the average reward learning curves for both predators and prey are symmetrical around the horizontal axis. Compared to PPO and MADDPG, MSGEE's average reward learning curve is closer to the horizontal axis starting from approximately 5000 episodes, indicating a better balance between the predator and prey groups. Compared to SVDE, although the two models alternately lead in the first 13000 episodes, MSGEE consistently maintains better convergence performance throughout the subsequent training.

[0110] In the robot soccer environment, Figure 12 , Figure 13 and Figure 14 The average reward learning curves of the MSGEE algorithm, PPO algorithm, and MADDPG algorithm are compared. The average reward learning curves of the MADDPG algorithm and MSGEE algorithm are compared. The average reward learning curves of the SVDE algorithm and MSGEE algorithm are compared. Figure 12 , Figure 13 and Figure 14 The x-axis represents the number of training rounds, and the y-axis represents the corresponding reward value.

[0111] In robot soccer (see...) Figure 12 , Figure 13 and Figure 14 Both the blue and purple sides need to score goals to gain positive rewards and avoid being penalized by the opponent's goals. Therefore, the average reward learning curves for both sides alternate in leading. Compared to PPO and MADDPG, MSGEE did not have a significant advantage in the first 5500 rounds (episode) of training, but maintained its learning level in the subsequent training phases. In contrast, PPO and MADDPG showed an imbalance in the later stages where the blue side completely led, preventing the purple side from improving. Compared to SVDE, although the difference in the average reward learning curves between the blue and purple sides was not significant in the final stage of training, SVDE became unstable in the middle of training (approximately 4000-7000 rounds), indicating that the blue side had a clear strategic advantage.

[0112] In environments with multiple unmanned surface vessels (USVs) guarding targets Figure 15 This diagram illustrates a comparison of the average reward learning curves of the PPO algorithm and the MSGEE algorithm in a multi-unmanned surface vessel target guarding environment. Figure 16 This diagram illustrates a comparison of the average reward learning curves of the MADDPG algorithm and the MSGEE algorithm in a multi-unmanned surface vessel target guarding environment. Figure 17 The diagram illustrates a comparison of the average reward learning curves of the SVDE and MSGEE algorithms in a multi-unmanned surface vessel (USV) target guarding environment. The horizontal axis of each graph represents the number of training steps, and the vertical axis represents the corresponding average reward value.

[0113] In the defense of multiple unmanned surface vessels (see...) Figures 15 to 17 ),from Figures 15 to 17 The curve trends show that, similar to the predator-prey relationship, MSGEE exhibits better convergence performance of the average reward learning curves for both the defender and attacker compared to the baseline method. For example... Figure 15 As shown, when faced with highly dynamic and complex guarding tasks, the PPO and MADDPG algorithms, due to the lack of an effective sample balancing mechanism, exhibit extremely slow or even stalled convergence of their average reward learning curves; while MSGEE can learn quickly and eventually converge to a higher reward range of over 100. Figure 16 As shown, compared to MSGEE-TD, which only uses temporal difference for grouped sample evaluation and sampling, and MSGEE-UNIFORM, which directly performs uniform sampling without sample evaluation, the complete MSGEE algorithm has the fastest convergence speed and the highest final reward, proving the effectiveness of fusing temporal difference and policy importance. Figure 17As shown, compared with the SVDE algorithm, MSGEE not only has a higher reward value at final convergence, but also a smaller range of curve oscillation in the later stage of training (i.e., a narrower variance shadow area), demonstrating better policy training stability.

[0114] In addition to comparing the average reward training performance of each method in the benchmark environment, this embodiment also evaluates the specific performance of each method by extrapolating the policy models of each agent group after training. Table 1 shows the average score, goal winning rate, and defense success rate of MSGEE and the benchmark algorithm in three experimental environments. The data in the table shows that MSGEE's extrapolation performance is better than the benchmark method in all three environments. The reading of the table data should refer to the coordinate system description in the aforementioned indicator definition, that is, to obtain the performance comparison data of specific game parties by cross-locating the rows and columns. Specifically, in the predator-prey game, when facing prey controlled by other models, the predator group controlled by MSGEE consistently has a higher average score than PPO, MADDPG, and SVDE, and achieves the highest score of 87.24 when facing prey controlled by PPO. In addition, when facing predators controlled by each model, the prey group controlled by MSGEE has the smallest loss score compared to the benchmark method, indicating that the predation and avoidance strategies learned by MSGEE are superior to the benchmark method. In the robot soccer environment, the blue or purple team controlled by MSGEE achieved the highest win rate against opponents controlled by the baseline method. In multi-UAV target defense, the invading team controlled by MSGEE scored higher against the defending team controlled by the baseline method.

[0115] Table 1: Adversarial Results of Benchmark Models in Experimental Environment

[0116] In summary, comparative experiments in three environments demonstrate that the proposed multi-agent policy acceleration optimization method outperforms benchmark methods. During model training, by comparing the average reward learning curve with the benchmark method, MSGEE exhibits a faster policy convergence time and its average reward learning curve is closer to policy equilibrium. Furthermore, during model derivation, various indicators show that the policy learned by MSGEE enables any group of agents to gain an advantage in competition with the benchmark model. These comparative experimental results validate the performance of the proposed cooperative-competitive hybrid policy acceleration optimization method based on multi-scenario game sample evaluation. Compared to benchmark reinforcement learning methods (PPO), benchmark multi-agent reinforcement learning methods (MADDPG), and multi-agent acceleration optimization methods (SVDE), it can more efficiently learn cooperative-competitive hybrid policies in multi-agent game tasks.

[0117] 4) Ablation analysis By comparing and analyzing with benchmark methods, the effectiveness of the multi-scenario game sample evaluation method proposed in this invention is verified. To further analyze the specific impact of accelerated multi-scenario sample generation and grouped experience evaluation on model performance, this embodiment designs multiple ablation experiments. Specifically, this embodiment first designs several variant methods, which are described below: (1) MSGEE contains all the modules mentioned above.

[0118] (2) MSGEE-NMS uses the multi-scenario game sample acceleration generation architecture mentioned above, that is, the number of interactive scenarios in each experimental environment is M=1.

[0119] (3) MSGEE-TD evaluates samples only through time difference and retains the grouped sample sampling process.

[0120] (4) MSGEE-UNIFORM does not perform sample evaluation, uniform sampling is used to obtain the samples required for training, and sampling is performed only once for different agent groups.

[0121] To analyze and evaluate the performance of the variant methods, this embodiment trains MSGEE and its three variants in three benchmark environments, with the same model parameters and training parameter settings as in the aforementioned comparative experiments. The performance of each method is evaluated during the training phase using the average reward of the agent group, and the policy level of each variant is evaluated during the model extrapolation phase using the average score, goal win rate, and number of intrusions. Figures 18 to 26 The average reward learning curves of MSGEE and its variants are shown in three experimental settings.

[0122] like Figures 18 to 26 As shown, MSGEE outperforms all variant methods in terms of average reward learning curve across three experimental environments. In multi-UAV target guarding (see...),... Figures 18 to 20 ), Figures 18 to 20 The average reward learning curves of the MSGEE algorithm of this invention are compared with those of various ablation variants. Figure 18 A comparison between MSGEE and MSGEE-NMS. Figure 19 For the comparison between MSGEE and MSGEE-TD, Figure 20 This section compares MSGEE with MSGEE-UNIFORM. The horizontal axis of each graph represents the number of training steps, and the vertical axis represents the corresponding average reward value. Similar to the learning results in Predator-Prey and Robot Soccer, the average reward learning curves for MSGEE and MSGEE-NMS (see...) Figure 18 The difference is not significant. For example... Figure 18As shown, the average rewards for both sides converge to positive values ​​because the environmental reward function, based on the sparse adversarial penalty, assigns positive dense base rewards with higher weights to survival time and exploration steps for both sides. This contrasts with MSGEE-TD (see...). Figure 19 ) and MSGEE-UNIFORM (see Figure 20 Compared to the two variant methods, MSGEE can achieve a stable multi-agent adversarial strategy earlier. Figure 19 As shown, compared with MSGEE-TD which relies solely on time difference, MSGEE can significantly reduce curve oscillations during training and achieve stable multi-agent adversarial strategies earlier; such as Figure 20 As shown, compared with MSGEE-UNIFORM, which does not perform sample evaluation and directly performs uniform sampling, MSGEE not only converges faster, but also has a significantly higher average reward value at the final convergence, which fully demonstrates the core role of the game sample equilibrium evaluation method proposed in this invention in improving the final performance of the strategy.

[0123] Specifically, in a predator-prey environment, Figures 21 to 23 The average reward learning curves of the MSGEE algorithm of this invention are compared with those of various ablation variants. Figure 21 A comparison between MSGEE and MSGEE-NMS. Figure 22 For the comparison between MSGEE and MSGEE-TD, Figure 23 A comparison between MSGEE and MSGEE-UNIFORM. Figures 21 to 23 The horizontal axis represents the training episode, and the vertical axis represents the corresponding reward value.

[0124] from Figures 21 to 23 The learning results show that the average reward learning curves of MSGEE and the three variant methods can all converge to equilibrium, and MSGEE has a better final reward. Figure 21 As shown, while MSGEE-NMS achieves better convergence of the final average reward learning curve compared to MSGEE-NMS, both methods tend to reach policy equilibrium for both the predator and prey groups around the 2000-5500 episode stage. Compared to MSGEE-TD (see...),... Figure 22 ) and MSGEE-UNIFORM (see Figure 23 Compared to the two variant methods, MSGEE not only has a final reward that is more in line with the predator-prey strategy balance, but it also converges earlier. MSGEE begins to converge significantly and approach the horizontal axis after about 2,500 rounds; while MSGEE-TD and MSGEE-UNIFORM exhibit violent oscillations in the early stages of training and only begin to converge gradually after about 10,000 rounds.

[0125] In the robot soccer environment, Figure 24 , Figure 25 and Figure 26 This paper demonstrates a comparison of the MSGEE algorithm of this invention with ablation variant methods in terms of policy balance. Figure 24 A comparison between MSGEE and MSGEE-NMS. Figure 25 For the comparison between MSGEE and MSGEE-TD, Figure 26 A comparison between MSGEE and MSGEE-UNIFORM. Figures 24 to 26 The horizontal axis represents the training episodes, and the vertical axis represents the corresponding average reward value.

[0126] In robot soccer ( Figures 24 to 26 Both the blue and purple teams receive rewards for scoring goals. Figure 24 As shown, the average reward learning curves of MSGEE and MSGEE-NMS highly overlap, exhibiting consistent convergence performance and policy balance. Figure 25 As shown, compared to MSGEE-TD, MSGEE exhibits stronger training stability, with its reward curve showing less oscillation in the later stages of training and a more stable convergence speed. And as... Figure 26 As shown, in the MSGEE-UNIFORM method using uniform sampling, after approximately 7000 rounds (episodes) of training, the reward curves of the blue and purple sides diverge significantly, indicating that the purple side's strategy overwhelms the blue side's, causing both sides' strategies to ultimately fail to reach the expected game equilibrium. In contrast, the MSGEE algorithm of this invention can effectively maintain the long-term game equilibrium of both sides' strategies.

[0127] In addition to comparing the average reward learning performance of MSGEE and its variants during the training phase, this embodiment also evaluates the specific performance of each method by extrapolating the trained policy model. Table 2 shows the average score, goal winning rate, and invasion score of MSGEE and its variants in three experimental environments. Rows in Table 2 represent the invading side, and columns represent the defending side. The data shows that MSGEE outperforms the variants in extrapolation across all three environments. The interpretation of the table data should refer to the coordinate system explanation in the aforementioned indicator definitions; that is, performance comparison data for specific game players is obtained through the cross-location of rows and columns.

[0128] Table 2: Adversarial Results of Various Variant Models in Three Experimental Environments

[0129] Specifically, in the predator-prey scenario, the predator group controlled by MSGEE consistently achieved a higher average score than MSGEE-TD and MSGEE-UNIFORM. In robot soccer, the blue or purple team controlled by MSGEE achieved the highest win rate against opponents controlled by variant methods. In the multi-UAV target defense environment, the intruder controlled by MSGEE achieved a higher score against the defender controlled by variant methods. Notably, the average reward convergence thresholds were similar across the three environments for the same number of rounds. This demonstrates that while significantly increasing sample throughput per unit time and greatly reducing the training time of real models through a multi-scenario parallel acceleration architecture, this invention still maintains a high-quality policy convergence level comparable to sequential interactions in a single scenario. This overcomes the technical bottleneck of traditional parallel acceleration, which easily leads to divergence in game strategies.

[0130] In summary, ablation experiments in three environments demonstrate that MSGEE, equipped with all the functional modules proposed in this invention, outperforms its variants in both model training and model inference. In particular, MSGEE performs better than both methods that perform grouped sample evaluation sampling only through temporal difference (MSGEE-TD) and those that perform uniform sampling directly without sample evaluation (MSGEE-UNIFORM). This proves that the game sample equilibrium evaluation method proposed in this invention, which integrates temporal difference and policy importance, can more effectively learn policies during the learning phase.

[0131] This invention addresses the challenge of accelerating the optimization of multi-agent collaborative-competitive hybrid policy learning processes by proposing a method based on game sample equilibrium. This method improves policy training efficiency under multi-agent game conditions through parallel generation of game samples across multiple scenarios and grouped sample evaluation sampling.

[0132] The training optimization module 300 employs a sampling probability-based weight adjustment algorithm when centrally updating and calculating the network loss function. Specifically, it uses the importance sampling weights multiplied by the time difference error (TD-error) to weight the Critic network loss function. This successfully overcomes the sample distribution bias problem caused by non-uniform sampling, effectively suppresses the oscillations and divergences of the Actor network and Critic network in game adversarial training, and ensures the stability and high quality of the final generated hybrid strategy.

[0133] This invention designs an accelerated multi-scenario generation architecture for game samples. By separating the scenario interaction process from the strategy generation process and considering the game relationships between different groups, it achieves parallel generation of game samples for multiple scenarios. Regarding game sample equilibrium, this invention proposes a game sample equilibrium evaluation method that integrates time difference and strategy importance. It evaluates and samples multiple groups of game experience samples from both time difference and strategy importance perspectives, supporting strategy training optimization for different agent groups and improving sample utilization efficiency. For multi-agent strategy training optimization, this invention designs a cooperative-competitive hybrid strategy acceleration optimization algorithm based on game sample equilibrium. This combines a near-end strategy optimization algorithm with the multi-scenario game sample generation architecture, utilizing multi-scenario game samples for strategy training. By sharing intra-group game samples and strategy models, it achieves efficient strategy training. In experiments, the method of this invention outperforms benchmark methods in multiple task environments, especially in multi-UAV target defense, where it reaches game equilibrium in approximately 20,000 rounds (episodes), accelerating the process by approximately 5,000-10,000 rounds (episodes) compared to the benchmark method.

[0134] To address the technical problems of low sample output efficiency and imbalance of competition strategies in multi-agent collaborative-competitive hybrid tasks, existing technologies (such as the single sampling strategy in publication CN116226662A which leads to slow convergence in complex environments, or the limited simple red-blue adversarial logic in publication CN110991545A) provide a collaborative hybrid strategy optimization technology solution.

[0135] At the data generation level, this invention employs a distributed parallel architecture that separates environment interaction and policy generation, improving sample throughput in complex multi-agent state spaces and shortening data collection time in real physical time. At the data utilization level, addressing the technical characteristic that the same sample has different learning values ​​for different camps in a game-theoretic environment, this invention proposes a differentiated game sample equilibrium algorithm that integrates time difference priority and grouping strategy importance (combined with dynamic task time indicators). This game sample equilibrium algorithm differs from single error evaluation, achieving experience extraction from multiple camp perspectives. At the algorithm optimization level, this invention integrates centralized training and decentralized execution (CTDE) architectures within the proximal policy optimization (PPO) framework, utilizing importance sampling weights. By guiding value evaluation through global state and combining weight correction to eliminate data distribution bias caused by non-uniform sampling, it ensures the unbiasedness of online gradient updates and the stability of multi-camp parameter synchronization. This design solves the training oscillation defect caused by prior art such as CN116796814A, which only uses a priority replay sampling algorithm without gradient weight rebalancing.

[0136] This invention combines multi-scenario parallel accelerated output, game faction differentiation evaluation, and unbiased policy gradient update, avoiding the policy divergence problem caused by accelerated learning in non-stationary game environments, and achieving technical effects that cannot be obtained by existing comparison file splicing and combination.

Claims

1. A hybrid strategy optimization system based on game sample equilibrium, characterized in that, The system employs a distributed parallel architecture that separates environment interaction from policy generation. The system includes: At least one independent parallel environment interaction module (100) is used to interact with the multi-agent environment, obtain environmental state observation information and execute joint actions received from the policy generation module (200) to obtain rewards from environmental feedback; At least one strategy generation module (200), wherein a single strategy generation module (200) is connected to a plurality of environment interaction modules (100); the strategy generation module (200) is used to receive the environmental state observation information from the corresponding connected environment interaction module (100), generate the joint action accordingly and send it to the corresponding environment interaction module (100), and store the experience sample generated by the interaction, including the environmental state observation information, the joint action and the environmental feedback reward, into the experience pool (400). The training optimization module (300) has a built-in game sample equilibrium sampling algorithm for differentiated evaluation based on agent grouping. The training optimization module (300) is used to extract the equilibrium training sample set exclusive to each agent group from the experience pool (400) through the game sample equilibrium sampling algorithm; and after centrally updating the policy network model parameters of each agent group based on the equilibrium training sample set, the updated policy network model parameters are synchronously distributed to each policy generation module (200). The centralized update includes: adjusting the weights when calculating the loss function based on the sampling probability of each sample in the balanced training sample set.

2. The system according to claim 1, characterized in that, The training optimization module (300) includes a differential priority calculation unit (310) to implement the game sample equilibrium sampling algorithm; The differentiated priority calculation unit (310) is configured as follows: The temporal difference priority of the experience samples in the experience pool (400) is calculated using a state value assessment network; Based on the specific policy objectives of each agent group, the importance assessment of the empirical samples to the grouping strategies of different groups is calculated differentially. The time difference priority is fused with the importance assessment of the grouping strategy to obtain the sampling priority of each agent group corresponding to the empirical sample.

3. The system according to claim 1 or 2, characterized in that, The differential priority calculation unit (310) in the training optimization module (300) performs the following steps when evaluating the importance of the differential calculation grouping strategy: Extract the cumulative reward and key time indicators of the task of interacting with the environment from the experience samples of the agent group. Based on the differentiated time requirements of agent groups in cooperative-competitive games, and combined with the cumulative reward, the importance assessment of the grouping strategy for a specific agent group is calculated using the grouping differentiation function.

4. The system according to any one of claims 1 to 3, characterized in that, The training optimization module (300) also includes a grouped independent sampling unit (320); The grouped independent sampling unit (320) is configured as follows: Construct completely independent priority sampling data structures for multiple groups of agents in a game environment; Before extracting the balanced training sample set specific to each agent group, the group independent sampling unit (320) updates and stores the calculated sampling priority of each agent group into the priority sampling data structure of the corresponding agent group. By performing hierarchical interval sampling on different priority sampling data structures, the exclusive balanced training sample set is extracted.

5. The system according to any one of claims 1 to 4, characterized in that, The training optimization module (300) includes a state value evaluation unit (330) and a policy optimization unit (340). The state value assessment unit (330) is equipped with a state value assessment network, which is used to perform centralized state value assessment to calculate time difference loss by taking the global state space information of the environment as input. The policy optimization unit (340) is configured with a policy network, which uses the local observation information of a single agent as input and combines the state value assessment to calculate the cumulative advantage of the policy for gradient update; and agents within the same agent group share the policy network model parameters in the policy network.

6. The system according to any one of claims 1 to 5, characterized in that, The policy optimization unit (340) performs gradient updates based on the near-end policy optimization algorithm; The training optimization module (300) uses the extracted balanced training sample set specific to each agent group and controls the policy optimization unit (340) to independently calculate and update the policy network model parameters of each agent group through the loss function of the near-end policy optimization algorithm.

7. The system according to any one of claims 1 to 6, characterized in that, The steps of the strategy generation module (200) in generating the joint action include: Obtain the local observation information of each agent constituting the environmental state observation information at the current time step transmitted by the environmental interaction module (100); Based on a preset policy network, the local observation information of the agent is mapped to corresponding individual policy actions; The individual policy actions generated by all agents within the current time step are vector-aggregated to generate the joint action to be sent to the environment interaction module (100).

8. A hybrid strategy optimization method based on game sample equilibrium, characterized in that, The method includes: Acquire environmental status observation information and execute joint actions to obtain rewards from environmental feedback; The joint action is generated based on the environmental state observation information, and the experience samples generated by the interaction, including the environmental state observation information, the joint action and the environmental feedback reward, are stored in the experience pool (400). The game sample equilibrium sampling algorithm is used to extract the equilibrium training sample set specific to each agent group from the experience pool (400); and the policy network model parameters of each agent group are updated in a centralized manner based on the equilibrium training sample set. The centralized update includes: adjusting the weights when calculating the loss function based on the sampling probability of each sample in the balanced training sample set.

9. The method according to claim 8, characterized in that, The method further includes: The temporal difference priority of the experience samples in the experience pool (400) is calculated using a state value assessment network; Based on the specific policy objectives of each agent group, the importance assessment of the empirical samples to the grouping strategies of different groups is calculated differentially. The time difference priority is fused with the importance assessment of the grouping strategy to obtain the sampling priority of each agent group corresponding to the empirical sample.

10. The method according to claim 8 or 9, characterized in that, The method further includes: Construct completely independent priority sampling data structures for multiple groups of agents in a game environment; Before extracting the balanced training sample set specific to each agent group, the calculated sampling priority of each agent group is updated and stored in the priority sampling data structure of the corresponding agent group. By performing hierarchical interval sampling on different priority sampling data structures, the exclusive balanced training sample set is extracted.

Images