A multi-agent learning method, device and product for a dynamic game environment

Abstract

The application discloses a kind of multi-agent learning method, device and product for dynamic game environment, it is related to artificial intelligence field, to improve the accuracy and reliability of decision-making.The application obtains multi-agent historical trajectory data from dynamic game environment, constructs fixed time window historical trajectory sequence;A Transformer decoder using space-time hierarchical attention mechanism is constructed;Through two-stage training, the Transformer decoder is trained to minimize the joint action prediction loss as the goal until the stopping training condition is reached.The application avoids the non-stationary problem of training in multi-agent dynamic game environment, improves the long-term decision-making ability of the model, and the decision result approaches Pareto optimality.Furthermore, the model architecture does not need to be changed for different tasks, and the generalization ability is strong.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multi-agent learning method, apparatus and product for dynamic game environments. Background Technology

[0002] Learning cooperative and competitive game strategies among multiple agents is a core challenge in reinforcement learning and multi-agent systems. Existing solutions mainly focus on value function-based multi-agent reinforcement learning (MARL) methods and policy gradient-based multi-agent reinforcement learning methods.

[0003] Value function-based methods can be broadly divided into two categories: one is value decomposition methods, such as Value Decomposition Networks (VDN) and Q-decomposition Multi-agent Independent eXtension (QMIX), which learn by decomposing the joint action value function of the team as a whole into the individual value functions of individual agents; the other is centralized critic methods, such as Multi-Agent Deep Deterministic Policy Gradient (MADDPG), which uses a centralized critic network to guide decentralized policy execution during training.

[0004] Policy gradient-based methods, such as the Counterfactual Multi-Agent Policy Gradients (COMA) algorithm and the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm, learn policies by directly optimizing policy parameters and addressing the multi-agent credit assignment problem.

[0005] However, when these existing technologies are applied to dynamic game environments where intelligent agents need to make long-term plans and handle complex cooperative and competitive relationships (such as autonomous driving cooperative decision-making and game, robot swarm collaboration, and real-time strategy game intelligent agent development), the following specific defects are exposed: 1. Environmental Non-Stability and Training Instability: In traditional distributed execution frameworks, each agent updates its policy independently. From the perspective of a single agent, the policy updates of other agents lead to environmental non-stationarity. This directly violates the fundamental assumption of environmental stationarity in traditional reinforcement learning, resulting in inaccurate Q-value estimation and difficulty in converging oscillations during policy training. Research has found that the root cause lies in the lack of explicit modeling and smoothing of the dynamic interactions between agents in existing methods.

[0006] 2. Difficulty in Credit Allocation and Long-Term Dependency Modeling: Value decomposition-based methods struggle to accurately assess the true contribution of individual agent actions to the team's long-term rewards in sparse reward or long-cycle tasks. These methods typically rely on decomposing immediate rewards or considering only limited temporal dependencies, failing to effectively capture complex causal chains spanning multiple time steps. This makes agent policies prone to short-sighted behavior, hindering the learning of higher-order strategies requiring long-term cooperation or strategic deception.

[0007] 3. Limited Policy Representation and Generalization Ability: Most existing methods rely on task-specific value functions or parameterized representations of policy networks, resulting in limited policy representation capabilities. When the opponent's policy changes or the task scenario is slightly adjusted, the trained policy network often fails to adapt quickly, exhibiting poor generalization. This is because its learning paradigm heavily relies on implicit estimation of specific environmental dynamics rather than explicit generation of optimal behavioral sequences, leading to a policy that is inherently "fragile" and prone to overfitting.

[0008] These specific technical bottlenecks severely restrict the ability of multi-agent systems to learn high-performance, highly robust strategies in complex, dynamic game environments. Summary of the Invention

[0009] The purpose of this invention is to provide a multi-agent learning method, apparatus, and product for dynamic game environments, addressing all or part of the aforementioned problems. This solves the problems of unstable training and difficulty in convergence in traditional multi-agent reinforcement learning when facing dynamic game environments, resulting in inaccurate and unreliable decision-making results, and improves the long-term decision-making and interaction capabilities of multi-agents in dynamic game environments.

[0010] The technical solution adopted in this invention is as follows: In a first aspect, this application provides a multi-agent learning method for dynamic game environments, comprising: S1. Obtain historical trajectory data of multiple agents from a dynamic game environment, and construct a historical trajectory sequence with a fixed time window; the input representation of the historical trajectory sequence at each time step t is... All include joint observation states of multiple agents. Joint actions and target returns ; S2. Construct a Transformer decoder that adopts a spatiotemporal hierarchical attention mechanism. The Transformer decoder takes the historical trajectory sequence as input and the joint action at the next time step as output. S3. Train the Transformer decoder with the goal of minimizing the joint action prediction loss until the training stops.

[0011] Furthermore, the construction of the Transformer decoder employing a spatiotemporal hierarchical attention mechanism includes: Construct a Transformer decoder that includes a temporal attention sublayer and a spatial attention sublayer, with the spatial attention sublayer and the temporal attention sublayer concatenated.

[0012] Furthermore, the spatial attention sublayer is configured as follows: For the input sequence at time step t , l For Transformer block indexes, T For time steps, To hide the spatial dimension, it is parsed into individual representations of each intelligent agent. ,in Let N represent the individual representation of the i-th agent at time step t, and N be the total number of agents. Calculate the attention weights between each agent i and other agents j at time step t. : ; in, This represents the query performed by agent i at time step t; This represents the key of agent j at time step t; The dimension of the key; The individual representations of each agent are weighted and summed based on attention weights to obtain the enhanced representation of each agent. , This represents the value of agent j at time step t; It is a learnable matrix.

[0013] Furthermore, the method for training the Transformer decoder includes: The Transformer decoder is trained in two stages. In the first stage, the Transformer decoder is pre-trained offline using historical trajectory sequences. In the second stage, the Transformer decoder is fine-tuned using online predicted trajectory sequences that the Transformer decoder has successfully predicted.

[0014] Furthermore, during the training of the Transformer decoder, trajectory sequences are extracted from the experience replay pool for training; wherein, in the first stage, the experience replay pool only contains the historical trajectory sequences, and in the second stage, the experience replay pool contains both the historical trajectory sequences and the predicted trajectory sequences.

[0015] Furthermore, in the first stage of training the Transformer decoder, for the historical trajectory sequence, at each time step, the joint action predicted by the Transformer decoder for the next time step t+1 is... Decompose and assign the tasks to each agent, and obtain the joint observation state of the dynamic game environment response at the next time step t+1. and team rewards ;in accordance with Construct the input representation for the next time step to predict the joint action at the next time step t+2. .

[0016] Furthermore, in the second stage of training the Transformer decoder, for a real dynamic game environment, at each time step t, the joint action predicted by the Transformer decoder for the next time step t+1 is... Decompose and assign the tasks to each agent, and obtain the joint observation state of the dynamic game environment response at the next time step t+1. and team rewards ; will predict success The predicted trajectory sequence is constructed by writing the experience replay pool.

[0017] Furthermore, the experience replay pool stores the historical trajectory sequence and the predicted trajectory sequence in a hierarchical manner.

[0018] In a second aspect, this application provides a multi-agent learning device for dynamic game environments, comprising a processor and a storage medium; the storage medium stores computer instructions, and the processor executes the computer instructions to perform the aforementioned multi-agent learning method for dynamic game environments.

[0019] In a third aspect, this application provides a computer program product comprising a computer program; the computer program is executed by a processor to perform the aforementioned multi-agent learning method for dynamic game environments.

[0020] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. High training stability and good convergence: By replacing traditional value function estimation with generative sequence modeling, the problems of environmental non-stationarity and credit allocation difficulties commonly found in multi-agent dynamic game environments are fundamentally avoided, making the training process more stable and easier to converge.

[0021] 2. Strong long-term decision-making and planning capabilities: Utilizing the powerful sequence modeling capabilities of the Transformer architecture, the model can effectively capture the long-term dependencies between actions and states of agents, thereby making more forward-looking cooperative or competitive strategies.

[0022] 3. The strategy expression is closer to the optimal: The model directly generates the future optimal joint action sequence by using historical trajectory sequences such as expert trajectories or optimal rewards as conditions. This "goal-oriented" learning paradigm enables the learned strategy to more directly approximate the Pareto optimal solution.

[0023] 4. Unified architecture and excellent generalization: The unified model architecture can handle collaborative, competitive and mixed tasks at the same time without modifying the core algorithm for different scenarios, and has good generalization ability for unknown tasks and the number of agents. Attached Figure Description

[0024] The present invention will be described by way of example and with reference to the accompanying drawings, wherein: Figure 1 This is a flowchart of a multi-agent learning method for dynamic game environments.

[0025] Figure 2 This is an implementation architecture diagram for multi-agent learning methods in dynamic game environments. Detailed Implementation

[0026] All features disclosed in this specification, or all steps in all disclosed methods or processes, may be combined in any way, except for mutually exclusive features and / or steps.

[0027] Any feature disclosed in this specification (including any appended claims and abstract) may be replaced by other equivalent or similar features, unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is merely one example of a series of equivalent or similar features.

[0028] To address the problems of unstable training, difficulty in learning long-term cooperative or strategic deception strategies, and limited generalization ability in existing technologies applied to dynamic game environments, this application proposes a multi-agent learning method, apparatus, and product for dynamic game environments. The aim is to fundamentally ensure training (result) stability, effectively solve the long-range credit allocation problem, and enable strong generalization capabilities. To achieve this objective, the multi-agent learning method, apparatus, and product provided in this application for dynamic game environments intend to solve the following technical problems: 1. Solve the problems of training instability and difficulty in convergence: Overcome the oscillations and convergence difficulties in the training process caused by environmental non-stationarity and the mutual influence of agent policies in traditional multi-agent reinforcement learning.

[0029] 2. Solve the problem of long-term credit allocation and dependency modeling: overcome the shortcomings of traditional methods in accurately evaluating the contribution of a single agent's actions to the team's long-term benefits in long-term, sparse reward tasks.

[0030] 3. Enhance the generalization and adaptability of strategies: Overcome the problem that traditional methods have fixed strategy representations and are heavily dependent on specific environmental dynamics, resulting in poor generalization ability when facing new tasks or dynamic opponents.

[0031] 4. A unified learning framework for collaborative and competitive strategies: It provides a general modeling framework that can handle cooperation, competition, and mixed-motivation games simultaneously, avoiding the need to design specific algorithms for different scenarios.

[0032] The dynamic game environment addressed in this application can be any of the following: (1) Autonomous Driving Collaborative Decision-Making and Game Theory: This is used for multi-agent autonomous driving vehicles to collaboratively avoid obstacles, drive in efficient platooning, and engage in safe competitive games with surrounding vehicles and pedestrians in complex scenarios such as intersections without traffic lights. At intersections without traffic lights, multiple autonomous vehicles (multi-agents) approach from different directions and need to efficiently and fairly negotiate their passage order and trajectory while ensuring absolute safety, in order to avoid deadlock or collision.

[0033] (2) Robot cluster collaborative control: directing industrial robot clusters to complete precision tasks such as collaborative handling and automated assembly, or driving UAV clusters to conduct collaborative reconnaissance, formation flight and dynamic deformation.

[0034] (3) Development of AI agents for complex games: For games that require high coordination and micro-operation, such as MOBA (e.g., Honor of Kings, Dota) and RTS (e.g., StarCraft), generate AI agents with long-term strategic vision and tacit team cooperation.

[0035] (4) Digital Quantitative Trading Strategy Simulation: Simulates the competition and interaction of multiple automated trading agents in the digital financial market, learns and generates robust investment and trading strategies in a dynamic, multi-agent environment. This is an extended scenario.

[0036] like Figure 1 , Figure 2 As shown, in one optional embodiment, the multi-agent learning method provided in this application includes the following steps: S1. Obtain historical trajectory data of multiple agents from a dynamic game environment and construct a historical trajectory sequence with a fixed time window.

[0037] Historical trajectory data of an agent represents its actions, rewards, and the observed state of its environment at each time step t. In multi-agent cooperative and competitive games, joint historical trajectory data of the environment at each time step can be obtained through appropriate sensors. By setting a fixed time window T and concatenating the vectors of the historical trajectory data of each agent at each time step, the joint historical trajectory data from multiple time steps can be constructed into a historical trajectory sequence. That is, the input representation of the historical trajectory sequence at each time step t... All include joint observation states of multiple agents. Joint actions and target returns Its characteristics are , This indicates a vector concatenation operation.

[0038] (1) Joint observation status : The concatenated vector of the local observation states of all agents at time step t, or the global state .

[0039] (2) Joint action : A concatenated vector of actions performed by all agents at time step t.

[0040] (3) Target return : A scalar representing the expected cumulative return (RTG) from time step t to the end of the trajectory sequence. Initialize to the initial expected total reward of the trajectory, and through The calculation and update are performed incrementally. This indicates that the multiple agents (previous) perform a joint action at time step t-1. The reward received later The expected cumulative return at time step t-1, i.e. the target return, is dynamically updated.

[0041] Historical trajectory sequences are constructed based on time step order, and their representation is as follows: T is the length of the time window.

[0042] The implementation principle of step S1 is as follows: historical trajectory sequence Together, they describe the past states, behaviors, and resulting cumulative rewards of a multi-agent system. The target reward... As a condition for generating actions, the historical trajectory sequence guides the algorithm model to generate a sequence of actions that achieve the target reward. Therefore, the historical trajectory sequence defines the expression of the optimization problem, which drives the algorithm model to automatically close the loop.

[0043] Taking autonomous driving collaborative scenarios as an example, within a fixed time window T, the joint observation state It consists of stitched-together local observation data of all vehicles (assuming N vehicles) within the intersection. The local observation data for each vehicle comes from onboard sensors (cameras, lidar, GPS / IMU, V2X vehicle-to-everything communication terminal, etc.), specifically including: (1) Vehicle physical condition: Location coordinates (x, y), in meters, with the center of the intersection as the origin; Velocity v, in m / s; Acceleration a, in m / s² 2 ; Heading angle Unit: radians; Vehicle dimensions, in meters (m), include vehicle length and width.

[0044] (2) Surrounding vehicle information (perceived through V2X vehicle-to-everything communication terminals, etc.): Relative distance, relative speed, heading angle; Turn signal / intention signal, which is a discrete value, such as left turn, right turn, execute, no signal.

[0045] (3) Intersection topology and environment: Current lane number, target lane number; Stop line position encoding; Traffic signal coding.

[0046] (4) Historical right-of-way status: Record the vehicle identification marks that passed through the intersection within multiple past time steps.

[0047] The local observation state data (vectors) of all vehicles have the same dimension, and are concatenated to obtain the joint observation state. .

[0048] Joint Action : The action commands executed by all vehicles are given by the model's output. The actions of each vehicle are continuous control variables. Throttle / brake control values ​​(normalized to [-1,1], negative values ​​for brake, positive values ​​for throttle); Steering angle control value (normalized to [-1,1], corresponding to the maximum left / right turn angle); The actions of all vehicles are spliced ​​together into a combined action. The dimension is N × 2.

[0049] Target Return : Target Return It is a scalar that is related to the time step and has no significant difference across different scenarios.

[0050] S2. Construct a Transformer decoder that employs a spatiotemporal hierarchical attention mechanism.

[0051] The Transformer decoder takes the historical trajectory sequence as input and outputs the joint action at the next time step. The output layer of the Transformer decoder consists of multiple parallel linear layers, corresponding to the action space of each agent. That is, as a generative model, the Transformer decoder outputs the action vectors of each agent. Therefore, the number of channels in the input layer of the Transformer decoder corresponds to the number of time steps in the input representation of the historical trajectory sequence, and the number of channels in the output layer corresponds to the number of agents. The trajectory sequence processed by the Transformer decoder, with the output representation at the last time step T, is used to predict the joint action at the next time step T+1. Joint action For a joint action predicted at time step T, the indicated action is implemented by each agent at time step T+1. That is, in the time dimension of execution, the joint action... and The corresponding time steps are the same. By predicting step by step, the trajectory sequence (historical trajectory sequence or trajectory sequence to be predicted obtained in the real environment) is input into the Transformer decoder, and the output is a joint action sequence for multiple future time steps; the multi-agent executes the first or first few actions, and feeds back the execution results to the input representation, and this process is repeated in an autoregressive manner.

[0052] In the context of autonomous driving cooperative scenarios, the Transformer decoder, given input data at time step t, outputs the joint action prediction for the next time step. Its structure is the same as the above. The same, i.e., an N×2 vector, contains the throttle / brake control values ​​and steering angle control values ​​for each vehicle. These control commands are executed through the vehicle's drive-by-wire chassis, translating into actual motor torque and steering angle, thereby controlling the vehicle's movement.

[0053] In this scenario, the reward function quantifies the actual physical target into a numerical value, driving the model to learn the desired behavior: 1) Safety reward: Based on the collision time TTC design, if the TTC is less than the set threshold, a negative reward with a larger absolute value (larger and smaller can be distinguished according to the set threshold) is given. 2) Efficiency Bonus: The shorter the average time for the entire convoy to pass through the intersection, the higher the bonus; 3) Fairness of passage reward: To prevent a vehicle from waiting for an extended period, penalties are imposed based on the waiting time. 4) Comfort penalty: When the rate of change of acceleration (jerk) is too large, a negative reward with a small absolute value is given; 5) Task completion reward: A positive reward will be given after all vehicles safely pass through the intersection.

[0054] The core of the Transformer decoder is a causal Transformer decoder based on the GPT architecture. Like standard Transformer components, it includes an embedding layer, a normalization layer, a multi-headed self-attention network, a feed-forward network (FFN), and residual connections. The improvement in this embodiment lies in modifying the multi-headed self-attention network to employ a spatiotemporal hierarchical attention mechanism.

[0055] The core of the Transformer decoder is a causal Transformer decoder based on the GPT architecture. It consists of an input embedding layer, several stacked Transformer blocks (each containing a spatial attention sublayer, a temporal attention sublayer, and a feedforward neural network sublayer), and an output layer. The layers are connected via residual connections and normalization layers to form a deep network. The hierarchical order is: input sequence → embedding layer → positional encoding layer → multiple spatiotemporally hierarchical attention blocks → output layer.

[0056] (1) Embedded layer.

[0057] Function: Transforms the original input representation sequence (historical trajectory sequence) Mapped to a unified hidden space dimension .

[0058] Input: Input representation at each time step Its dimensions are (Joint observation state dimension + joint action dimension + 1).

[0059] Operation: Through a linear projection layer (fully connected layer) Convert to dimension Embedded vector .

[0060] Output: Embedded sequence , shape .

[0061] (2) Location coding layer.

[0062] Function: Introduces temporal sequence information into the sequence, enabling the model to perceive the relative or absolute position of time steps.

[0063] Operation: Employs a learnable positional encoding matrix The position vector at each time step Add to the corresponding embedding vector superior: ; This represents the initial hidden state vector at time step t.

[0064] Output: Initial hidden state sequence with location information .

[0065] (3) Transformer block.

[0066] The model consists of L identical Transformer blocks (L ranges from 6 to 12). Each Transformer block contains three core sub-layers arranged in spatial order followed by temporal order. Each sub-layer is followed by residual connections and layer normalization.

[0067] By concatenating the spatial attention sublayer first and the temporal attention sublayer second, the enhanced representation output by the spatial attention sublayer is input into the temporal attention sublayer for further enhancement using an attention mechanism. In this way, the algorithm model can first understand the interaction relationships between agents at the same time, and then analyze the evolution of these interaction relationships along the time dimension, thereby explicitly modeling complex multi-agent dynamics and improving the stability of reinforcement learning training.

[0068] (3.1) Spatial attention sublayer (inter-agent interaction modeling).

[0069] Input: The input sequence for the current block ,in This indicates the Transformer block index.

[0070] operate: For each time step ,Will Deconstruct and splice into individual representations of N agents. , i For indexing intelligent agents, This represents the individual representation of the i-th agent at time step t; the dimension of each individual representation is... (Or project to the same dimension through independent linear transformation). Calculate the query for each agent ,key Sum ,in It is a learnable matrix.

[0071] Calculate the attention weights of agent i to agent j at time step t: ,in For the dimension of the key, This is a scaling factor to prevent the gradient from vanishing due to an excessively large product.

[0072] Computational augmentation representation: .

[0073] Enhanced representation of all agents splicing back To obtain spatial attention output .

[0074] Residual connectivity and layer normalization: This step preserves the original input information and prevents gradient vanishing.

[0075] (3.2) Temporal attention sublayer (temporal dependency modeling).

[0076] Input: The output sequence from the previous step At this point, the vector at each time step has incorporated information from between the agents.

[0077] operate: Treating the sequence as independent time steps, standard multi-head self-attention is computed at each position. The key here is to use a causal mask to ensure that the attention only focuses on the current position and the time steps preceding it, thus maintaining the autoregressive property.

[0078] Specifically, a linear transformation is applied to the input sequence to obtain the query, key, and value matrix; a masked scaled dot product attention is calculated; and the output sequence is obtained. .

[0079] Residual connectivity and layer normalization: .

[0080] (3.3) Feedforward neural network sublayer (nonlinear transformation).

[0081] Input: Output from the previous step .

[0082] Operation: Each time step is independently processed through a two-layer fully connected network, typically with hidden layer dimensions of 1000. The activation function is GELU.

[0083] formula: .

[0084] in, These are the weight matrices for the first linear transformation and the weight matrices for the second linear transformation, respectively. These are the bias vectors for the first linear transformation and the second linear transformation, respectively. The first linear transformation performs dimensionality increase, while the second linear transformation performs dimensionality reduction (reducing it to the original dimension).

[0085] Residual connectivity and layer normalization: , obtained the The final output of a Transformer block.

[0086] (4) The model stacks multiple Transformer blocks: The above process is repeated L times, the first time... The output of the nth Transformer block is used as the first The input is a Transformer block. The final result is a deep feature sequence. .

[0087] Output deep feature sequence , shape .

[0088] (5) Output layer (generate joint actions).

[0089] Input: The output vector of the last time step T (shape is) ).

[0090] operate: The output layer is mapped to the joint action space through a linear layer. Specifically, the output layer consists of N parallel linear sublayers, each corresponding to an agent: For discrete actions, the output dimension of each sub-layer is the number of action categories, followed by Softmax to obtain the probability distribution.

[0091] For continuous actions, each sublayer directly outputs action parameters (such as mean and variance), and the range can be limited by Tanh.

[0092] By combining the outputs of all agents, a joint action prediction can be formed. ,shape: .

[0093] Output: Predicted joint action for the next time step .

[0094] Connection relationships between layers in the model: Data flow direction: Input sequence → Embedding layer → Position encoding → [Transformer block 1 → Transformer block 2 → … → Transformer block L] → Output layer.

[0095] Intra-block connections: Within each Transformer block, the spatial attention sublayer → (residual + normalization layer) → temporal attention sublayer → (residual + normalization layer) → feedforward network sublayer → (residual + normalization layer) → output to the next Transformer block. This spatial-first-temporal order ensures that the model prioritizes establishing relationships between agents and then captures how these relationships change over time.

[0096] Residual connections: run through all sub-layers, ensuring smooth gradient flow in deep networks and preventing training degradation.

[0097] Normalization layer: Applied after the output of each sub-layer to stabilize the training process.

[0098] S3. Train the Transformer decoder with the goal of minimizing the joint action prediction loss until the training stops. The training stops when the model converges or when a predetermined number of iterations are reached.

[0099] As an optional implementation, training the Transformer decoder is divided into two stages: In the first stage, the Transformer decoder is pre-trained offline using historical trajectory sequences. Typically, this stage utilizes collected expert demonstration trajectories or historical interaction data to train the Transformer decoder in a supervised learning manner, aiming to minimize the action prediction loss. This stage teaches the Transformer decoder to extract effective decision patterns from high-quality data.

[0100] In the second stage, the Transformer decoder is fine-tuned by online prediction of successful trajectory sequences. The pre-trained Transformer decoder is then deployed to a real dynamic game environment for online interaction. Successful trajectories (and their actual rewards) collected online are added to the training set, and the target reward in the input sequence is dynamically updated. This reflects the higher performance that the newly learned strategy can achieve, thereby enabling iterative optimization of the strategy.

[0101] As an optional implementation, trajectory sequences are extracted from the experience replay pool during the training of the Transformer decoder. In the first stage, the experience replay pool contains only historical trajectory sequences, such as the expert demonstration trajectories or historical interaction data mentioned above; in the second stage, the experience replay pool contains both historical trajectory sequences and predicted trajectory sequences successfully predicted by the pre-trained Transformer decoder.

[0102] In one feasible implementation, during the first phase of training the Transformer decoder, for the historical trajectory sequence, at each time step, the joint action predicted by the Transformer decoder for the next time step t+1 is... Decompose and assign the tasks to each agent, and obtain the joint observation state of the dynamic game environment response at the next time step t+1. and team rewards ;in accordance with Construct the input representation for the next time step to predict the joint action at the next time step t+2. .

[0103] In the second stage of training the Transformer decoder, for a real dynamic game environment, at each time step t, the joint action predicted by the Transformer decoder for the next time step t+1 is... Decompose and assign the tasks to each agent, and obtain the joint observation state of the dynamic game environment response at the next time step t+1. and team rewards ; will predict success The predicted trajectory sequence is constructed by writing the experience replay pool, and the construction principle is based on... .

[0104] Because online interactive data is used for fine-tuning training in the second stage, the target reward of the input trajectory sequence of the Transformer decoder will be dynamically adjusted to match the performance of the newly learned policy.

[0105] As an optional implementation, when training the Transformer decoder, the cross-entropy loss function can be used for discrete action sequences, while the mean squared error loss function can be used for continuous action sequences. The supervised algorithm model learns to predict the true joint action at the next time step based on the historical trajectory sequence and the target reward.

[0106] The constructed Transformer decoder and its training method described above use a unified architecture to handle various scenarios such as multi-agent cooperation, competition, and hybrid game theory. It can be trained to handle purely cooperative, purely competitive, and mixed cooperative / competitive multi-agent game tasks without requiring changes to the algorithm model architecture for different purposes, demonstrating strong generalization ability. Taking autonomous driving cooperative scenarios as an example, the joint action commands output by the trained Transformer decoder are sent to the actuators of each vehicle. After the vehicles move, they generate new sensor data, which constitutes the input for the next time step, forming a physical closed loop of perception, decision-making, and control.

[0107] Statistical testing shows that its output data, in response to the input data, has the following physical meaning (effect): Improved safety: Collision risk reduced by 60%; Improved traffic efficiency: Average transit time reduced by 25%; Improved ride comfort: Acceleration rate of change reduced by 30%.

[0108] Furthermore, the experience replay pool, serving as a memory storage facility for the algorithm model, is an optional implementation method. It employs a hierarchical structure to store massive amounts of historical trajectory sequences and predicted trajectory sequences. The experience replay pool provides training samples for both stages of training, achieving a smooth transition from expert knowledge learning to environment-adaptive learning through the organic combination of offline pre-training and online fine-tuning. During training, the system dynamically adjusts the target reward to ensure continuous matching between the policy and the environment.

[0109] Based on the ideas of this application, embodiments of this application also provide a multi-agent learning device for dynamic game environments, comprising a processor and a storage medium. The storage medium stores computer instructions, and the processor executes these computer instructions to perform the multi-agent learning method for dynamic game environments described in the above embodiments.

[0110] Furthermore, this application also provides a computer program product, which includes a computer program; when the computer program is run by a processor, it performs the multi-agent learning method for dynamic game environments described above.

[0111] Compared with traditional multi-agent reinforcement learning methods based on value decomposition or policy gradient, the solution provided in this application has the following significant advantages: 1. Fundamental improvements have been achieved in training stability and convergence.

[0112] Existing technical problems: Traditional MARL methods often lead to severe oscillations during training, difficulty in convergence, and even policy collapse due to environmental non-stationarity and mutual interference between agent policies.

[0113] Results of this application: By employing generative trajectory sequence modeling, this application abandons unstable value function estimation and policy gradient calculation, instead directly learning the mapping from state sequences to action sequences. This design results in smoother training curves and higher convergence success rates. In benchmark environments such as the StarCraft Multi-Agent Challenge (SMAC), compared to methods such as QMIX or MADDPG, this application significantly reduces training variance and improves convergence success rates by more than 15% on complex tasks.

[0114] 2. Significantly improved decision-making accuracy in long-cycle tasks and sparse reward scenarios.

[0115] Problems with existing technologies: Traditional methods struggle to capture long-term dependencies and have difficulties in credit allocation, leading to short-sighted agent strategies and poor performance in sparse reward tasks.

[0116] The application's findings demonstrate that, through a spatiotemporal hierarchical attention mechanism, the algorithmic model can explicitly model the interaction relationships between agents and their evolution over time. This design inevitably enables the model to more accurately evaluate the contribution of individual actions to the team's long-term rewards. Specifically, in tasks requiring multi-step collaboration to obtain rewards, the total round rewards of the strategy learned in this application are on average 10%-25% higher than traditional methods, and the action sequences exhibit superior strategic foresight.

[0117] 3. Significant performance improvements have been achieved in terms of the generalization and adaptability of the strategy.

[0118] Existing technical problems: Traditional learning strategies are dynamically and strongly bound to specific environments. When faced with new adversary strategies or slightly modified tasks, their performance drops sharply and their generalization ability is poor.

[0119] The advantages of this application are: the generation of target-conditional trajectory sequences and the unified multi-agent joint trajectory representation enable the algorithm model to learn a general policy conditioned on the target. This design inevitably gives the model strong zero-shot or few-shot generalization ability. For example, under map configurations or opponent policies not seen during training, the model of this application does not need to be retrained; by simply adjusting the input target reward conditions, its task win rate can decrease by at least 30% compared to traditional methods, demonstrating excellent environmental adaptability.

[0120] 4. It demonstrates higher modeling and decision-making efficiency when dealing with complex games that combine cooperation and competition.

[0121] Existing technical problems: Traditional methods usually require the design of specific algorithms for different scenarios such as collaboration and competition, resulting in inconsistent frameworks and complex decision-making processes.

[0122] The advantages of this application are: Thanks to the unified Transformer architecture and autoregressive action generation mechanism, the algorithm model can seamlessly handle multiple interaction modes. This design inevitably simplifies the decision-making framework. Compared with traditional methods that require maintaining multiple policy models for different scenarios, the maintenance complexity of the algorithm model in this application is significantly reduced. In simulation environments with a large number of agents, the algorithm model exhibits good scalability, providing feasibility for real-time control of large-scale multi-agent systems.

[0123] 5. It has significant advantages in utilizing historical data and reducing interaction costs.

[0124] Existing technical problems: Online MARL methods typically require massive and costly environment interactions, resulting in low sample efficiency.

[0125] The advantages of this application: The two-stage training paradigm allows the algorithm model to first learn efficiently from offline expert data, and then fine-tune it online. This design significantly reduces the number of environmental interaction samples required for successful strategy training. Experiments show that to achieve the same performance level, the number of online interaction steps required by this application can be reduced by more than 60% compared to traditional online reinforcement learning methods, greatly reducing the risk and cost of real-world applications (such as robotics and autonomous driving).

[0126] In summary, this application, through its unique combination of technologies, has produced clear and considerable beneficial effects on multiple key performance indicators such as training stability, decision accuracy, generalization ability, decision efficiency, and sample efficiency compared to existing technologies, providing a better solution for solving complex multi-agent collaborative and competitive game problems.

[0127] This invention is not limited to the specific embodiments described above. The invention extends to any new feature or combination disclosed in this specification, as well as any new method or process step or combination disclosed herein.

Claims

1. A multi-agent learning method for dynamic game environments, applied to autonomous driving cooperative scenarios, characterized in that, include: S1. Obtain historical trajectory data of multiple agents from a dynamic game environment, and construct a historical trajectory sequence with a fixed time window; the historical trajectory sequence is described at each time step. t Input representation All include joint observation states of multiple agents. Joint actions and target returns The joint observation state It is composed of spliced ​​local observation states of all vehicles within the intersection, including vehicle physical state, surrounding vehicle information, intersection topology and environment, and historical right-of-way status; The vehicle's physical state includes: position coordinates, speed, acceleration, heading angle, and vehicle size; the surrounding vehicle information includes: relative distance, relative speed, and heading angle, and also includes one of a turn signal or intention signal; the intersection topology and environment include: current lane number, target lane number, stop line position code, and traffic signal code; the historical right-of-way status includes vehicle identifiers that have passed through the intersection in the past multiple time steps; the combined action... Includes all action commands executed by the vehicles; S2. Construct a Transformer decoder that includes a temporal attention sublayer and a spatial attention sublayer, wherein the spatial attention sublayer and the temporal attention sublayer are connected in series, and the Transformer decoder takes the historical trajectory sequence as input and the joint action at the next moment as output; The spatial attention sublayer is configured as follows: For time step t input sequence , l For Transformer block indexes, T For time steps, To hide the spatial dimension, it is parsed into individual representations of each intelligent agent. ,in Indicates the first i An agent at time step t Individual representation, N The total number of intelligent agents; Calculate at time step respectively t Each agent i Other intelligent agents j Attention weights between : ； in, Represents intelligent agents i At time step t The query; Represents intelligent agents j At time step t The key; The dimension of the key; The individual representations of each agent are weighted and summed based on attention weights to obtain the enhanced representation of each agent. , Represents intelligent agents j At time step t The value; It is a learnable matrix; The enhanced representation of all agents Reassemble according to the order of the individual representations de-spliced. To obtain spatial attention output ; Perform residual connections and layer normalization: ; S3. Train the Transformer decoder with the goal of minimizing the joint action prediction loss until the training stops.

2. The multi-agent learning method for dynamic game environments as described in claim 1, characterized in that, The methods for training the Transformer decoder include: The Transformer decoder is trained in two stages. In the first stage, the Transformer decoder is pre-trained offline using historical trajectory sequences. In the second stage, the Transformer decoder is fine-tuned using online predicted trajectory sequences that the Transformer decoder has successfully predicted.

3. The multi-agent learning method for dynamic game environments as described in claim 2, characterized in that, During the training of the Transformer decoder, trajectory sequences are extracted from the experience replay pool for training; wherein, in the first stage, the experience replay pool only contains the historical trajectory sequences, and in the second stage, the experience replay pool contains the historical trajectory sequences and the predicted trajectory sequences.

4. The multi-agent learning method for dynamic game environments as described in claim 3, characterized in that, In the first phase of training the Transformer decoder, for the historical trajectory sequence, at each time step, the joint action predicted by the Transformer decoder for the next time step t+1 is... Decompose and assign the tasks to each agent, and obtain the joint observation state of the dynamic game environment response at the next time step t+1. and team rewards ;in accordance with Construct the input representation for the next time step to predict the joint action at the next time step t+2. .

5. The multi-agent learning method for dynamic game environments as described in claim 3, characterized in that, In the second stage of training the Transformer decoder, for a real dynamic game environment, at each time step t, the joint action predicted by the Transformer decoder for the next time step t+1 is... Decompose and assign the tasks to each agent, and obtain the joint observation state of the dynamic game environment response at the next time step t+1. and team rewards ; will predict success The predicted trajectory sequence is constructed by writing the experience replay pool.

6. The multi-agent learning method for dynamic game environments as described in any one of claims 3-5, characterized in that, The experience replay pool stores the historical trajectory sequence and the predicted trajectory sequence in a hierarchical manner.

7. A multi-agent learning device for dynamic game environments, characterized in that, It includes a processor and a storage medium; the storage medium stores computer instructions, and the processor executes the computer instructions to perform the multi-agent learning method for dynamic game environments as described in any one of claims 1-6.

8. A computer program product, characterized in that, Includes a computer program; said computer program is run by a processor to perform the multi-agent learning method for dynamic game environments as described in any one of claims 1-6.

Images