Game confrontation strategy generation method based on efficient hierarchical diffusion offline reinforcement learning

Abstract

The invention discloses a game confrontation strategy generation method based on efficient hierarchical diffusion offline reinforcement learning. The method comprises the steps of extracting structure information from offline game tracks, constructing a topological graph by analyzing similarity of state representation, obtaining tree partitions of a state community, and adaptively constructing a multi-scale diffusion hierarchical structure for each game track; in the conditional diffusion link, the top-layer model generates a game sub-target sequence according to the overall game target, the lower-layer model generates an intermediate-layer game sub-target sequence or a bottom-layer game state strategy sequence, and each sequence is subjected to conditional generation based on the corresponding upper-layer game sub-target; meanwhile, the structural information gain of each game state strategy community is calculated according to hierarchical division, and the structural information gain is used as a classifier to be freely guided; establishing an exploration regularizer based on the structure entropy; and outputting a game strategy by the game confrontation strategy generation model. According to the method, the structure information of the offline game data is fully utilized, and a flexible and stable game strategy learning framework is constructed.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a method for generating game confrontation strategies based on efficient hierarchical diffusion offline reinforcement learning. Background Art

[0002] Against the backdrop of the rapid evolution of competitive gaming technology, generating accurate and efficient competitive strategies has become a core challenge for improving competitive capabilities and gaining a competitive advantage. Traditional game strategy development relies heavily on expert experience and limited historical game data, making it difficult to adequately address the complex and ever-changing state space in competitive scenarios, resulting in insufficient strategy generalization. With the advancement of big data and artificial intelligence technologies, offline reinforcement learning has gradually been introduced into the gaming field, providing a new approach to optimizing game strategies.

[0003] Offline reinforcement learning, also known as batch reinforcement learning, leverages pre-collected data on game behavior to learn strategies, eliminating the need for real-time data interaction during the game and effectively avoiding the risks associated with online exploration. For example, in esports, historical match data contains multi-dimensional information such as player control habits, hero selection, and map resource distribution. Offline reinforcement learning can analyze this data to uncover differences in the win rates of different strategies in various scenarios, generating more targeted strategies for new games and avoiding missed opportunities due to blind trial and error.

[0004] However, offline reinforcement learning faces numerous challenges in generating game strategies. The out-of-distribution (OOD) state and action problem is particularly prominent. The complexity of game scenarios makes it difficult for offline datasets to cover all possible game states. When a strategy encounters states not covered by the training data, temporal difference-based methods are prone to significant extrapolation errors. For example, in a chess game, if a model encounters an unconventional lineup it has never seen before, it may fail to identify the threat level of that state and formulate an incorrect strategy, leading to a loss.

[0005] The problem of multimodal strategy coverage cannot be ignored. There are significant differences in the operating styles and strategic preferences of different commanders. Historical data often contain a large number of suboptimal strategies. Traditional unimodal strategy representations are difficult to capture these multimodal characteristics, resulting in the generated strategies being unable to flexibly respond to diverse confrontation scenarios.

[0006] To address these issues, researchers introduced a diffusion-based generative approach. Diffusion probability models, with their powerful distribution matching and sequence generation capabilities, can transform strategy optimization into a generative modeling problem by synthesizing high-fidelity adversarial trajectories aligned with offline datasets, effectively mitigating the risk of out-of-distribution actions. However, this approach has significant limitations in long-term game decision-making. First, the variance in value estimates increases exponentially with the number of rounds, leading to inaccurate assessments of the strategy's long-term benefits. Second, the iterative denoising process is computationally expensive, making it difficult to rapidly generate strategies in real-time adversarial scenarios, limiting its practical application value.

[0007] The emergence of hierarchical reinforcement learning (HRL) offers a new approach for improving the efficiency of long-term game decision-making. It decomposes complex adversarial tasks into high-level sub-goals and low-level actions, enabling hierarchical strategy generation. However, existing hierarchical diffusion methods often employ a manually defined two-layer fixed structure with rigid sub-goal time spans. This makes them unable to adapt to the dynamic and changing tempo of the game, resulting in insufficient strategy flexibility and a tendency for accumulated approximation errors to cause decision drift. Summary of the Invention

[0008] To solve the above problems, the present invention proposes a game confrontation strategy generation method based on efficient hierarchical diffusion offline reinforcement learning. Targeting long-term games, confrontation scenarios with complex state spaces and sparse feedback, the present invention fully utilizes the structural information of offline game data to construct a flexible and stable strategy learning framework, breaking through the existing technical bottlenecks and providing a new path for strategy optimization in the game field.

[0009] To achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a method for generating game confrontation strategies based on efficient hierarchical diffusion offline reinforcement learning, comprising the following steps:

[0010] S10, constructing a game confrontation strategy generation model, including the following steps:

[0011] S101 extracts structural information from offline trajectories, constructs a topological graph by analyzing the similarity of state representations, and obtains a tree-like partition of state communities. Based on this, a multi-scale diffusion hierarchy is adaptively constructed for each trajectory, enabling flexible trajectory generation at different time scales.

[0012] S102, during the conditional diffusion process, the top-level model generates a sequence of game sub-goals according to the overall game goal. The lower-level model then generates a sequence of intermediate-level game sub-goals or a sequence of bottom-level game state strategies. Each sequence is conditionally generated based on the corresponding upper-level game sub-goal. The structural information gain of each game state strategy community is calculated according to the hierarchical division, and this gain is then used as a free guide for the classifier to enhance the generation of conditional sequences.

[0013] S103, establishes an exploration regularizer based on structural entropy to encourage policies to cover a wide range of state spaces, alleviate the extrapolation error caused by the difference in offline and online policy distributions, and improve the model's generalization ability;

[0014] S20, inputting the game data to be tested into the game confrontation strategy generation model, and the model outputs the game confrontation strategy.

[0015] Furthermore, constructing a multi-scale diffusion hierarchy includes the following steps:

[0016] Given an offline game adversarial dataset Extract all game state elements s to form a vertex set S, and build weighted edges between each vertex and its first k neighbors according to their feature similarity, thus forming a k-nearest neighbor state graph Then the HCSE optimization algorithm is applied to derive the height of The optimal coding tree The tree minimizes Dimensional structural entropy Coding Tree express The tree community partitioning of α, where each node α corresponds to a state community at a specific granularity level The parent-child relationship between nodes reflects the inclusion relationship of the state characteristics of different game groups;

[0017] Based on the state S Community division in offline game dataset Each trajectory τ0 in is adaptively segmented hierarchically;

[0018] By The hierarchical sub-goals are taken into control to build a hierarchical multi-scale diffusion model; on each layer h>1, a length of The sub-goal sequence For each sub-goal Its related next-level sub-goal sequence recursion;

[0019] When h=1, the sub-goal The next level sub-goal sequence Equivalent to the state-action segment

[0020] Following the control-as-reasoning paradigm, a binary optimality variable O is introduced here i , used to indicate the trajectory segment Is it relative to the subgoal? is optimal;

[0021] Given each offline trajectory τ0 Layer trajectory segmentation and sub-target hierarchy, performing conditional probability decomposition;

[0022] Through conditional probability decomposition, the conditional generation problem of offline trajectories is transformed into a multi-layer hierarchical diffusion process with dynamic time scales between different layers.

[0023] Furthermore, by maximizing the upper bound of dynamic uncertainty To determine the parameter k, we need to ensure that the coding tree The tree structure has sufficient structural expression capabilities.

[0024] Furthermore, based on the state S Community division in offline game dataset Each trajectory τ0 in is adaptively segmented hierarchically, including:

[0025] Layer by layer Extract all tree nodes U with the same height h h The state community partition defined by the middle node divides the trajectory τ0 to ensure that each generated trajectory segment It is continuous in time and consists of states in the same community.

[0026] Furthermore, the conditional probability is decomposed into:

[0027]

[0028] in, It captures the cumulative reward over the entire trajectory τ0, represents the sub-goal satisfaction constraint imposed by the Dirac delta function, and Represents the h-th layer sub-goal sequence For the The prior probability distribution of the layer sub-target sequence, is the conditional probability distribution of the i-th sub-target sequence in the h-th layer, For the Conditional information of the layer sub-target sequence, is the total number of layers in the hierarchical diffusion model, representing the hierarchical depth of task decomposition, and h is the current layer index.

[0029] Furthermore, in the process of conditional diffusion:

[0030] For the diffusion model at the hth layer, each target sequence Represented as initial data Representation based on sub-goal subgoals or state-action sequences of conditions;

[0031] Then a series of noisy versions are constructed iteratively through the forward diffusion process

[0032] Use a shared diffusion model at layer h Diffusion Model Use classifier free bootstrapping to jointly predict the noise term ∈~N(0,I) and estimate the posterior distribution at each diffusion step K

[0033] According to the conditional probability decomposition, the conditional input of the top diffusion model It is defined as an exponential function of the cumulative reward over the entire trajectory τ0; at the bottom layer, a signal based on structural information is calculated; for each sequence Identify the coding tree The corresponding tree node α of height h in All state elements in belong to community V α ;Conditional input It is then defined as the information gain of node α;

[0034] This gain term quantifies the probability of determining the Single-step random state transitions occur in lower-level fragments Additional information required within.

[0035] Furthermore, an exploration regularizer based on structural entropy is established, including:

[0036] For each trajectory τ0, the state transition (s t ,s t+1 ), and uses the lowest level diffusion model Estimated sampling probability Sampling probability instead of G s The feature similarity between the state elements in S is used to construct a new topological structure on S, that is, a complete and weighted state graph G' s ; In Figure G' s In the above example, the degree of each state s∈S is defined as its diffusion visit probability, which is obtained by marginalizing the transition probability Get; define G' s About the encoding tree The structural entropy of

[0037] For the encoding tree and complete state diagram Structural entropy and Shannon entropy Satisfying the following inequality, we establish The variational lower bound of is obtained, and the theoretical connection between it and the Shannon entropy of the state distribution on the state space S is revealed.

[0038] Each diffusion model The regularized training objective is defined as:

[0039]

[0040] in, represents the expected calculation for different offline trajectories τ0 and diffusion steps k;

[0041] It traverses each element in the h-th layer sub-target sequence; A noise term used to measure the model's predictions The difference between the conditional prediction and the real noise ∈ is minimized to make the model prediction more accurate; ω is used as a guide weight to control the interpolation ratio between conditional prediction and unconditional prediction. Its reasonable adjustment can balance the effect of the model under different prediction methods;

[0042] It is the key manifestation of structural entropy regularization; when h = 1, If it is 1, the item is activated; Shannon entropy represents the state space S. Maximizing it can encourage the model to explore the state space more balancedly, avoid over-reliance on historical trajectories, and discover potential states that are not fully reflected in the offline dataset; It represents the weighted sum of the entropy of different hierarchical structures in the encoding tree. By minimizing it, the decision hierarchy is preserved, preventing the model from deviating too much from the behavioral strategy, thereby reducing the extrapolation error.

[0043] Furthermore, for the coding tree and complete state diagram Structural entropy and Shannon entropy The following inequality is satisfied:

[0044]

[0045] in, is the total number of layers, h is the current layer index, η h is the weight, Partition U for the h-th layer state h The Shannon entropy of .

[0046] The beneficial effects of adopting this technical solution are:

[0047] This invention achieves stable and effective offline policy learning in a long-term, sparse reward environment. The workflow is as follows: first, structural information is extracted from offline trajectories. A topological graph is constructed by analyzing the similarity of state representations to obtain a tree-like partitioning of state communities. Based on this, a multi-scale diffusion hierarchy is adaptively constructed for each trajectory, enabling flexible trajectory generation at different time scales. Next, in a conditional diffusion process, the top-level model generates a sub-goal sequence based on the task reward, and the lower-level model generates the corresponding sequence based on the upper-level sub-goals. By calculating the structural information gain of the state community as a guide, the reliance on reward prediction is reduced. Finally, an exploration regularizer based on structural entropy is designed to encourage policies to cover a wide range of state spaces, alleviate the extrapolation error caused by the difference in offline and online policy distributions, and improve the model's generalization ability.

[0048] The present invention creates an adaptive hierarchical diffusion architecture: Currently, in offline reinforcement learning-based game strategy generation, the hierarchical diffusion method has the problem of structural rigidity. Existing technologies often use a preset two-layer diffusion hierarchy with fixed time division. However, the confrontation rhythm and strategy cycle of different game types vary significantly. For example, real-time strategy games require rapid decision-making in seconds, while turn-based strategy games allow long-term planning in minutes. As a result, the fixed structure model is difficult to adapt to dynamic confrontation scenarios and lacks flexibility when generating personalized game strategies. The present invention is committed to solving this problem by deeply mining the structural information in offline game trajectories, innovatively extracting topological structure relationships based on the similarity of game state characteristics, deriving tree-like partitioning for different game scenarios, and then constructing a flexible multi-scale hierarchical diffusion framework. This framework can flexibly plan strategies at multiple time scales based on the characteristics of the game type and the real-time confrontation state, significantly improving the model's adaptability to various game scenarios and providing a guarantee for accurate strategy generation in complex confrontation environments.

[0049] The present invention reduces the reliance on reward predictions: When traditional game strategies are generated based on offline reinforcement learning, model decisions are overly dependent on reward predictions of state-action sub-trajectories. However, game data is affected by factors such as layout habits and random events, and reward predictions are prone to deviations, resulting in unstable strategy decisions and prone to decision-making errors. In order to improve this situation, the present invention introduces a conditional guidance method based on the structural information gain of the game state community. By accurately quantifying the structural information gain of each game state community in the diffusion layer and using it as a conditional input without classifier guidance, a single reward signal is replaced. This method reduces the model's reliance on short-term reward predictions, improves the accuracy and stability of conditional strategy sequence generation, helps the model explore potential correlations between states, achieves a more global game decision-making perspective, and significantly improves the winning rate of confrontation.

[0050] The present invention reduces extrapolation errors and enhances exploration capabilities: The limitations of offline data sets often lead to extrapolation errors in the model during learning, which manifests as a weak ability to handle states and actions that have not appeared. For example, a game strategy model relies only on conventional battle data for training. When faced with rare scenarios such as unconventional enemy combinations, it is easy to output unreasonable strategies. To address this problem, the present invention designs a structural entropy exploration regularizer, which forces the strategy to cover a wide range of state spaces through hierarchical planning constraints: on the one hand, it maximizes the Shannon entropy of the state distribution, encouraging the model to explore potential states that are not fully represented in the offline data; on the other hand, it minimizes the hierarchical structural entropy, retaining the stability of the decision hierarchy, and avoiding decision drift caused by excessive deviation from historical strategies. This mechanism effectively alleviates the distribution differences between offline and online strategies. Even when encountering adversarial scenarios not covered by the data, the model can still generate reasonable strategies guided by structural entropy, significantly improving generalization capabilities and decision reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a framework diagram of a method for generating game confrontation strategies based on efficient hierarchical diffusion offline reinforcement learning according to the present invention;

[0052] Figure 2 This is an operational flow chart of a method for generating game confrontation strategies based on efficient hierarchical diffusion offline reinforcement learning according to the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described below with reference to the accompanying drawings.

[0054] In this embodiment, see Figure 1 and Figure 2 As shown, the present invention proposes a game confrontation strategy generation method based on efficient hierarchical diffusion offline reinforcement learning, comprising the steps of:

[0055] S10, constructing a game confrontation strategy generation model, including the following steps:

[0056] S101 extracts structural information from offline trajectories, constructs a topological graph by analyzing the similarity of state representations, and obtains a tree-like partition of state communities. Based on this, a multi-scale diffusion hierarchy is adaptively constructed for each trajectory, enabling flexible trajectory generation at different time scales.

[0057] S102, during the conditional diffusion process, the top-level model generates a sequence of game sub-goals according to the overall game goal. The lower-level model then generates a sequence of intermediate-level game sub-goals or a sequence of bottom-level game state strategies. Each sequence is conditionally generated based on the corresponding upper-level game sub-goal. The structural information gain of each game state strategy community is calculated according to the hierarchical division, and this gain is then used as a free guide for the classifier to enhance the generation of conditional sequences.

[0058] S103, establishes an exploration regularizer based on structural entropy to encourage policies to cover a wide range of state spaces, alleviate the extrapolation error caused by the difference in offline and online policy distributions, and improve the model's generalization ability;

[0059] S20, inputting the game data to be tested into the game confrontation strategy generation model, and the model outputs the game confrontation strategy.

[0060] As an optimization solution of the above embodiment, constructing a multi-scale diffusion hierarchy structure includes the following steps:

[0061] Given an offline game adversarial dataset Extract all game state elements s to form a vertex set S, and build weighted edges between each vertex and its first k neighbors according to their feature similarity, thus forming a k-nearest neighbor state graph Then the HCSE optimization algorithm is applied to derive the height of The optimal coding tree The tree minimizes Dimensional structural entropy Coding Tree express The tree community partitioning of α, where each node α corresponds to a state community at a specific granularity level The parent-child relationship between nodes reflects the inclusion relationship of the state characteristics of different game groups;

[0062] Based on the state S Community division in offline game dataset Each trajectory τ0 in is adaptively segmented hierarchically;

[0063] By The hierarchical sub-goals are taken into control to build a hierarchical multi-scale diffusion model; on each layer h>1, a length of The sub-goal sequence For each sub-goal Its related next-level sub-goal sequence recursion;

[0064] Defined as:

[0065]

[0066] in, Represents sub-goal The time scale is not predefined but is derived from The hierarchical partitioning and trajectory τ0 are inferred;

[0067] When h=1, the sub-goal The next level sub-goal sequence Equivalent to the state-action segment

[0068] Following the control-as-reasoning paradigm, a binary optimality variable O is introduced here i , used to indicate the trajectory segment Is it relative to the subgoal? is optimal;

[0069] Optimality variable O i The posterior probability of is defined by a Boltzmann distribution parameterized by the cumulative reward as follows:

[0070]

[0071] Given each offline trajectory τ0 Layer trajectory segmentation and sub-target hierarchy, performing conditional probability decomposition;

[0072] Through conditional probability decomposition, the conditional generation problem of offline trajectories is transformed into a multi-layer hierarchical diffusion process with dynamic time scales between different layers.

[0073] Unlike a rigid, single-scale, two-layer diffusion hierarchy, this invention, specifically designed for competitive gaming scenarios, extracts topological relationships between offline game state elements based on similarities in equipment performance, strategic habits, and map features. This derives a tree-like partitioning of game state communities, thereby constructing a flexible, multi-scale, hierarchical diffusion framework for each game trajectory. This framework dynamically adjusts the strategy generation scale at different time granularities based on real-time competitive demands, significantly improving the flexibility and scenario adaptability of competitive strategies.

[0074] Among them, by maximizing the upper bound of dynamic uncertainty To determine the parameter k, we need to ensure that the coding tree The tree structure has sufficient structural expression capabilities.

[0075] Among them, based on the state S Community division in offline game dataset Each trajectory τ0 in is adaptively segmented hierarchically, including:

[0076] Layer by layer Extract all tree nodes U with the same height h h The state community partition defined by the middle node divides the trajectory τ0 to ensure that each generated trajectory segment It is continuous in time and consists of states in the same community.

[0077] Each trajectory segment The formal definition of is:

[0078]

[0079] Where i represents the time index of the i-th trajectory segment in the h-th layer, and the trajectory segment The final state is designated as a subgoal variable Represents a trajectory segment The sequence length.

[0080] Among them, the conditional probability is decomposed into:

[0081]

[0082] in, It captures the cumulative reward over the entire trajectory τ0, represents the sub-goal satisfaction constraint imposed by the Dirac delta function, and Represents the h-th layer sub-goal sequence For the The prior probability distribution of the layer sub-target sequence, is the conditional probability distribution of the i-th sub-target sequence in the h-th layer, For the Conditional information of the layer sub-target sequence, is the total number of layers in the hierarchical diffusion model, representing the hierarchical depth of task decomposition, and h is the current layer index.

[0083] As an optimization solution of the above embodiment, during the conditional diffusion process:

[0084] For the diffusion model at the hth layer, each target sequence Represented as initial data Representation based on sub-goal subgoals or state-action sequences of conditions;

[0085] Then a series of noisy versions are constructed iteratively through the forward diffusion process

[0086] Noise version The calculation formula is:

[0087]

[0088] In order to reduce computational overhead and improve training stability, a shared diffusion model is used in the hth layer. Diffusion Model Use classifier free bootstrapping to jointly predict the noise term ∈~N(0,I) and estimate the posterior distribution at each diffusion step K

[0089] Posterior distribution The calculation formula is:

[0090]

[0091] in, and ∑θ h Denote the prediction noise The mean vector and covariance matrix of . Symbol Indicates no conditional input and , to enable unconditional generation. Classifier bootstrapping is applied by interpolating between conditional and unconditional predictions using the bootstrapping weights ω.

[0092] According to the conditional probability decomposition, the conditional input of the top diffusion model It is defined as an exponential function of the cumulative reward over the entire trajectory τ0; in the lower layer, the cumulative reward signal is not used, but a signal based on structural information is calculated; for each sequence Identify the coding tree The corresponding tree node α of height h in All state elements in belong to community V α ;Conditional input It is then defined as the information gain of node α;

[0093] Conditional Input The information gain of node α is then defined as:

[0094]

[0095] This gain term quantifies the gain of the given prior knowledge (i.e., the Know that single-step random state transition is at a higher level fragment In the case of Single-step random state transitions occur in lower-level fragments Additional information required within.

[0096] As an optimization solution for the above embodiment, in order to reduce excessive reliance on offline game datasets, a structural entropy-based exploration regularizer is designed here, which encourages hierarchical diffusion game strategies to maximize coverage of the game state space while avoiding game strategy extrapolation errors caused by significant deviations from conventional game strategies.

[0097] Build an exploration regularizer based on structural entropy, including:

[0098] For each trajectory τ0, the state transition (s t ,s t+1 ), and uses the lowest level diffusion model Estimated sampling probability Sampling probability instead of G s The feature similarity between the state elements in S is used to construct a new topological structure on S, that is, a complete and weighted state graph G' s ; In Figure G' s In the above example, the degree of each state s∈S is defined as its diffusion visit probability, which is obtained by marginalizing the transition probability Get; define G' s About the encoding tree The structural entropy of

[0099] G' s About the encoding tree The structural entropy of is:

[0100]

[0101] For the encoding tree and complete state diagram Structural entropy and Shannon entropy Satisfying the following inequality, we establish We obtain a variational lower bound for , and reveal its theoretical connection with the Shannon entropy of the state distribution on the state space S.

[0102] For the encoding tree and complete state diagram Structural entropy and Shannon entropy The following inequality is satisfied:

[0103]

[0104] in, is the total number of layers, h is the current layer index, η h is the weight, Partition U for the h-th layer state h The Shannon entropy of .

[0105] On the one hand, the present invention maximizes entropy To promote a more balanced exploration of the state space, thereby improving access to latent states that are underrepresented in the offline dataset and reducing over-reliance on historical trajectories during training. On the other hand, in order to preserve the The decision hierarchy in this paper minimizes the structural entropy of each layer h by appropriate weighting. This constraint prevents b Produce larger deviations and reduce extrapolation errors.

[0106] Each diffusion model The regularized training objective is defined as:

[0107]

[0108] in, It represents the expected calculation for different offline trajectories τ0 and diffusion steps K, which means that the model is comprehensively considered under a variety of different trajectories and training stages to ensure that the model performs well in various situations.

[0109] It traverses each element in the h-th layer sub-target sequence; A noise term used to measure the model's predictions The difference between the conditional prediction and the real noise ∈ is minimized to make the model prediction more accurate; ω is used as a guide weight to control the interpolation ratio between conditional prediction and unconditional prediction. Its reasonable adjustment can balance the effect of the model under different prediction methods;

[0110] It is the key manifestation of structural entropy regularization; when h = 1, If it is 1, the item is activated; Shannon entropy represents the state space S. Maximizing it can encourage the model to explore the state space more balancedly, avoid over-reliance on historical trajectories, and discover potential states that are not fully reflected in the offline dataset; It represents the weighted sum of the entropy of different hierarchical structures in the encoding tree. By minimizing it, the decision hierarchy is preserved, preventing the model from deviating too much from the behavioral strategy, thereby reducing the extrapolation error.

[0111] In actual game-playing scenarios, the weighting coefficient η requires careful adjustment. If η is too large, the model may overly focus on optimizing structural entropy, neglecting the accuracy of noise predictions. Conversely, if η is too small, the structural entropy regularization effect cannot be fully utilized, making it difficult to effectively mitigate extrapolation errors and reduce reliance on offline datasets. By continuously experimenting with different values ​​of η and evaluating them based on the characteristics of specific game types and adversarial performance indicators, we can determine the most suitable parameter settings for model training, thereby improving the model's overall performance in offline reinforcement learning tasks for game-playing adversarial games.

[0112] The operation process of the present invention is as follows Figure 2 As shown in the figure, the process revolves around data processing, model building and training, strategy generation and application, and each link is closely linked to achieve efficient offline strategy learning.

[0113] Data preparation phase: Offline game datasets are collected from various game scenarios. These datasets cover a wide range of game types and contain rich data, including game state characteristics, historical action sequences, game target outcomes, competitive strategies, and corresponding win-loss feedback. However, raw data often suffers from missing values, noise, and high dimensionality. For missing values, we use the mean, median, or machine learning algorithm predictions to fill in the missing values, depending on the data type. For numerical data such as troop strength and resource output, we use the mean, median, or machine learning algorithm predictions to fill in the missing values. For non-numeric data such as strategy types, we use the data distribution to fill in the missing values. For noise introduced by environmental interference, we use mean filtering, median filtering, or Kalman filtering, depending on the noise characteristics.

[0114] Constructing a multi-scale diffusion hierarchy: Game state elements are extracted from preprocessed game data to form a vertex set. A k-nearest neighbor state graph is constructed based on the similarity of game state features, historical strategies, and other characteristics. The parameter k is determined by maximizing the upper bound of dynamic uncertainty. The optimal encoding tree is then obtained using the HCSE optimization algorithm to achieve a tree-like partitioning of the game state community. Each game trajectory is hierarchically segmented based on the encoding tree, resulting in different levels of game process trajectory segments and staged game sub-goal sequences. A hierarchical multi-scale diffusion model is constructed, transforming the conditional generation problem of offline game trajectories into a multi-layered, hierarchical diffusion process.

[0115] Conditional Diffusion Model Training: The top-level diffusion model generates a sequence of phased game sub-goals based on the overall game objective, while the lower-level models generate a sequence of specific game strategies based on the upper-level sub-goals. During training, a shared diffusion model is used with the classifier free bootstrapping technique to jointly predict the noise term and estimate the posterior distribution. At lower levels, the structural information gain of the game state-strategy community is calculated as a conditional input, rather than relying on a single cumulative metric of game outcomes. This enhances the ability to generate conditional sequences and improves the accuracy and stability of the model's game strategy formulation.

[0116] Structural Entropy Regularization Optimization: Utilizing the lowest-level diffusion model, this approach estimates the sampling probability of game state transitions, constructs a new state graph, and calculates its structural entropy with respect to the encoding tree. By maximizing the Shannon entropy of the game state space, this approach promotes the model's exploration of potential game states and reduces over-reliance on historical trajectories. Simultaneously, it minimizes the structural entropy of each layer, maintains the game decision hierarchy, prevents significant deviations from conventional game strategies, and mitigates extrapolation errors. Incorporating structural entropy into the training objective of the diffusion model allows for optimized model training.

[0117] Strategy Generation and Application: After the model is trained through the above steps, in a competitive game scenario, the model uses input information such as the current game state and historical strategies to generate corresponding personalized game strategies through a hierarchical multi-scale diffusion model. For example, based on information such as the game situation characteristics and resource status, the model can assist in generating appropriate game strategies such as tactical choices, action sequences, and the rhythm of the confrontation.

[0118] Continuous Optimization and Improvement: During the game-playing process, feedback on the model's strategy generation results is collected to evaluate the effectiveness of the strategy. If the model's performance falls short of expectations, the data processing, model structure, and training parameters are reviewed, and targeted adjustments and optimizations are made. For example, if the model's strategy performs poorly in certain complex game-playing scenarios, further optimization of the data preprocessing method or adjustment of the parameters of the structural entropy regularization can be used to improve the model's adaptability and game-playing decision-making capabilities, ensuring that the model continues to perform stably across various game-playing scenarios.

[0119] The efficient hierarchical diffusion offline reinforcement learning technology guided by structural information proposed in this invention has made an important breakthrough in the field of offline reinforcement learning for game confrontation strategy generation. It effectively addresses the core challenges faced by existing technologies such as out-of-distribution state processing and multimodal strategy coverage, and provides new directions and solutions for strategy optimization in the field of game confrontation.

[0120] Its key technologies are as follows:

[0121] 1. Construction of multi-scale diffusion hierarchy

[0122] By analyzing the topological relationships between offline game state elements based on similarities in game characteristics and historical strategies, a k-nearest neighbor state graph is constructed. The optimal encoding tree is then derived using the HCSE optimization algorithm, which partitions the game state community into a tree-like structure. Based on this partitioning, the game trajectory is hierarchically segmented, yielding sub-game goal sequences and state-strategy segments at different levels. These are then integrated into the game decision-making reasoning framework to form a hierarchical multi-scale diffusion model. This model can flexibly generate game trajectories at different game stage scales, avoiding the limitations of the traditional two-layer diffusion hierarchy and adapting to the needs of diverse game confrontations.

[0123] 2. Conditional Diffusion Model Design

[0124] In this hierarchical framework, the top-level diffusion model generates a sequence of phased game sub-goals based on the overall game objective, while the lower-level models generate a sequence of specific game strategies based on the upper-level sub-goals. Structural information gain is calculated for each game state-strategy community and used as a classifier-free guide to enhance conditional sequence generation. The high-level model uses the cumulative results of the game process as conditional input, while the lower-level model uses structural information gain as conditional input, reducing reliance on a single game outcome metric. Furthermore, by overlaying the terminal states of the denoised game trajectory, consistency with the hierarchical sub-game objective constraints is ensured, improving model training stability and the quality of generated game strategies.

[0125] 3. Structural Entropy Regularizer

[0126] To reduce over-reliance on offline game datasets, an exploration regularizer based on structural entropy is designed. The lowest-level diffusion model is used to estimate the sampling probability of game state transitions, construct a new state graph, and calculate its structural entropy. By maximizing the Shannon entropy of the game state distribution, this promotes balanced exploration of the game state space and captures potential states that are underrepresented in offline data. Simultaneously, minimizing the structural entropy of each layer preserves the game decision hierarchy, prevents deviation from historical game strategies, and mitigates extrapolation errors. Incorporating the structural entropy regularizer into the diffusion model training objective improves the model's ability to generalize strategies in competitive games.

[0127] In summary, the framework proposed in this paper addresses the challenges of offline reinforcement learning in long-term game confrontation and sparse win-loss feedback environments. Through innovative game state structure information analysis and hierarchical diffusion design, it effectively improves game decision-making performance and generalization ability.

[0128] This framework analyzes the structural information of offline game trajectories and adaptively constructs a multi-scale diffusion hierarchy, enabling flexible game strategy generation. In the conditional diffusion model, structural information gain is used as a conditional input, reducing reliance on game outcome predictions and enhancing model stability. Furthermore, a structural entropy regularizer is introduced to encourage broad coverage of the game state space within the constraints of hierarchical game planning, reducing reliance on offline game datasets and mitigating extrapolation errors.

[0129] Experimental results show that the present invention significantly outperforms existing baseline methods in a variety of game confrontation tasks, demonstrating great potential in the field of game confrontation offline reinforcement learning, and providing new ideas and methods for game strategy optimization research and application.

[0130] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for generating game confrontation strategies based on efficient hierarchical diffusion offline reinforcement learning, characterized by: Including steps: S10, constructing a game confrontation strategy generation model, including the following steps: S101 extracts structural information from offline trajectories, constructs a topological graph by analyzing the similarity of state representations, and obtains a tree-like partition of state communities. Based on this, a multi-scale diffusion hierarchy is adaptively constructed for each trajectory, enabling flexible trajectory generation at different time scales. S102, during the conditional diffusion process, the top-level model generates a sequence of game sub-goals according to the overall game goal. The lower-level model then generates a sequence of intermediate-level game sub-goals or a sequence of bottom-level game state strategies. Each sequence is conditionally generated based on the corresponding upper-level game sub-goal. The structural information gain of each game state strategy community is calculated according to the hierarchical division, and this gain is then used as a free guide for the classifier to enhance the generation of conditional sequences. S103, establishes an exploration regularizer based on structural entropy to encourage policies to cover a wide range of state spaces, alleviate the extrapolation error caused by the difference in offline and online policy distributions, and improve the model's generalization ability; S20, inputting the game data to be tested into the game confrontation strategy generation model, and the model outputs the game confrontation strategy.

2. The method for generating game confrontation strategies based on efficient hierarchical diffusion offline reinforcement learning according to claim 1, characterized in that: Constructing a multi-scale diffusion hierarchy includes the following steps: Given an offline game adversarial dataset Extract all game state elements s to form a vertex set S, and build weighted edges between each vertex and its first k neighbors according to their feature similarity, thus forming a k-nearest neighbor state graph Then the HCSE optimization algorithm is applied to derive the height of The optimal coding tree The tree minimizes dimensional structural entropy Coding Tree express The tree community partitioning of α, where each node α corresponds to a state community at a specific granularity level The parent-child relationship between nodes reflects the inclusion relationship of the state characteristics of different game groups; Based on the state S Community division in offline game dataset Each trajectory τ0 in is adaptively segmented hierarchically; By The hierarchical sub-goals are taken into control to build a hierarchical multi-scale diffusion model; on each layer h>1, a length of The sub-goal sequence For each sub-goal Its related next-level sub-goal sequence recursion; When h=1, the sub-goal The next level sub-goal sequence Equivalent to the state-action segment Following the control-as-reasoning paradigm, a binary optimality variable O is introduced here i , used to indicate the trajectory segment Is it relative to the subgoal? is optimal; Given each offline trajectory τ0 Layer trajectory segmentation and sub-target hierarchy, performing conditional probability decomposition; Through conditional probability decomposition, the conditional generation problem of offline trajectories is transformed into a multi-layer hierarchical diffusion process with dynamic time scales between different layers.

3. The method for generating game confrontation strategies based on efficient hierarchical diffusion offline reinforcement learning according to claim 2, characterized in that: By maximizing the upper bound of dynamic uncertainty To determine the parameter k, we need to ensure that the coding tree The tree structure has sufficient structural expression capabilities.

4. The method for generating a game confrontation strategy based on efficient hierarchical diffusion offline reinforcement learning according to claim 2, characterized in that: Based on the state S Community division in offline game dataset Each trajectory τ0 in is adaptively segmented hierarchically, including: Layer by layer Extract all tree nodes U with the same height h h The state community partition defined by the middle node divides the trajectory τ0 to ensure that each generated trajectory segment It is continuous in time and consists of states in the same community.

5. The method for generating game confrontation strategies based on efficient hierarchical diffusion offline reinforcement learning according to claim 2, characterized in that: The conditional probability decomposes into: in, It captures the cumulative reward over the entire trajectory τ0, represents the sub-goal satisfaction constraint imposed by the Dirac delta function, and Represents the h-th layer sub-target sequence For the The prior probability distribution of the layer sub-target sequence, is the conditional probability distribution of the i-th sub-target sequence in the h-th layer, For the Conditional information of the layer sub-target sequence, is the total number of layers in the hierarchical diffusion model, representing the hierarchical depth of task decomposition, and h is the current layer index.

6. The method for generating game confrontation strategies based on efficient hierarchical diffusion offline reinforcement learning according to claim 1, characterized in that: In the process of conditional diffusion: For the diffusion model at the hth layer, each target sequence Represented as initial data Representation based on sub-goal subgoals or state-action sequences of conditions; Then a series of noisy versions are constructed iteratively through the forward diffusion process Use a shared diffusion model at layer h Diffusion Model Use classifier free bootstrapping to jointly predict the noise term ∈~N(0,I) and estimate the posterior distribution at each diffusion step K According to the conditional probability decomposition, the conditional input of the top diffusion model It is defined as an exponential function of the cumulative reward over the entire trajectory τ0; at the bottom layer, a signal based on structural information is calculated; for each sequence Identify the coding tree The corresponding tree node α of height h in All state elements in belong to community V α ;Conditional input It is then defined as the information gain of node α; This gain term quantifies the probability of determining the Single-step random state transitions occur in lower-level fragments Additional information required within.

7. The method for generating game confrontation strategies based on efficient hierarchical diffusion offline reinforcement learning according to claim 1, characterized in that: Build an exploration regularizer based on structural entropy, including: For each trajectory τ0, the state transition (s t ,s t+1 ), and uses the lowest level diffusion model Estimated sampling probability Sampling probability instead of G s The feature similarity between the state elements in S is used to construct a new topological structure on S, that is, a complete and weighted state graph G′ s ; In graph G′ s In the above example, the degree of each state s∈S is defined as its diffusion visit probability, which is obtained by marginalizing the transition probability Obtain; define G′ s About the encoding tree The structural entropy of For the encoding tree and complete state diagram Structural entropy and Shannon entropy Satisfying the following inequality, we establish The variational lower bound of is obtained, and the theoretical connection between it and the Shannon entropy of the state distribution on the state space S is revealed. Each diffusion model The regularized training objective is defined as: in, represents the expected calculation for different offline trajectories τ0 and diffusion steps k; It traverses each element in the h-th layer sub-target sequence; A noise term used to measure the model's predictions The difference between the conditional prediction and the real noise ∈ is minimized to make the model prediction more accurate; ω is used as a guide weight to control the interpolation ratio between conditional prediction and unconditional prediction. Its reasonable adjustment can balance the effect of the model under different prediction methods; It is the key manifestation of structural entropy regularization; when h = 1, If it is 1, the item is activated; Shannon entropy represents the state space S. Maximizing it can encourage the model to explore the state space more balancedly, avoid over-reliance on historical trajectories, and discover potential states that are not fully reflected in the offline dataset; It represents the weighted sum of the entropy of different hierarchical structures in the encoding tree. By minimizing it, the decision hierarchy is preserved, preventing the model from deviating too much from the behavior strategy, thereby reducing the extrapolation error.

8. The method for generating game confrontation strategies based on efficient hierarchical diffusion offline reinforcement learning according to claim 7, characterized in that: For the encoding tree and complete state diagram Structural entropy and Shannon entropy The following inequality is satisfied: in, is the total number of layers, h is the current layer index, η h is the weight, Partition U for the h-th layer state h The Shannon entropy of .

Images