Dynamic constraint index system construction method and system for non-transitive game scene

Abstract

The invention discloses a dynamic constraint index system construction method and system oriented to a non-transitive game scene. The method comprises the following steps: constructing a basic evaluation method set, wherein the set at least comprises an Elo algorithm, a Trueskill algorithm and an Alpha-Rank algorithm; training an evaluation model based on game simulation data, and determining weights of elements for the obtained basic evaluation method set; and fusing multiple algorithm weights based on the obtained trained model, fusing an integral result generated by a scoring function of each basic evaluation algorithm through a dynamic weight, and outputting a strategy comprehensive evaluation result. According to the method, multiple evaluation algorithms are integrated, the algorithm weight is dynamically adjusted through game simulation data, and the strategy evaluation problem in a complex game is solved. Experiments prove that the accuracy rate of the method reaches 96% and is remarkably improved compared with that of a single algorithm. The system operates based on a distributed computing platform accelerated by the GPU. According to the method, the problems of non-transitive strategy evaluation distortion, poor environmental adaptability and the like are effectively solved, and a reliable evaluation scheme is provided for related fields.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep reinforcement learning and game command confrontation, and particularly relates to a dynamic constraint index system construction method and system for non-transitive game scenarios. BACKGROUND

[0002] The background information provided in this section is merely for the purpose of enhancing the understanding of the present disclosure and does not necessarily describe prior art.

[0003] The dynamic constraint index system for non-transitive game scenarios belongs to a multi-agent game evaluation technology. It has key applications in many fields such as military simulation and game strategy analysis.

[0004] Currently, the mainstream multi-agent game evaluation technologies mainly include Elo, Trueskill, AlphaRank, etc. These technologies evaluate the agent strategies from different dimensions to promote training. The Elo algorithm is based on the assumption of normal distribution or logistic distribution, and adjusts the player score through the deviation between the actual competition result and the expected result. It is widely used in the evaluation of game ability. The TrueSkill algorithm updates the strategy skill value by considering factors such as opponent level in addition to the win-lose result, improving the accuracy of evaluation. The Alpha-Rank algorithm is proposed by DeepMind, which is based on the new solution concept in meta-game. It describes the steady-state distribution of strategy population through discrete Markov random process, and sorts and evaluates strategies according to the probability ratio.

[0005] However, the existing methods have the following defects: non-transitive strategy evaluation distortion, that is, in real game scenarios, strategies may have non-transitivity (for example, A is better than B, B is better than C, but C is better than A), and existing methods are difficult to effectively handle such cases, resulting in biased evaluation results; poor dynamic environment adaptability, that is, the evaluation index weight is fixed and cannot respond to environmental mutations in time, such as strategy update or scene change, and cannot flexibly adjust the evaluation method; single evaluation dimension, that is, mainly relying on a small number of indicators for evaluation, lacking a comprehensive evaluation system, and difficult to fully cover the multi-dimensional needs of complex game scenarios. SUMMARY

[0006] The present application relates to the field of deep reinforcement learning and game command confrontation, and particularly relates to a dynamic constraint index system construction method and system for non-transitive game scenarios.

[0007] A dynamic constraint index system construction method and system for non-transitive game scenarios, comprising the following steps:

[0008] Step 1: Construct a set of basic evaluation methods, including at least Elo, Trueskill, and Alpha-Rank algorithms.

[0009] Step 2: Train the evaluation model based on game simulation data, determine the weights of the elements for the set of basic evaluation methods obtained in step 1.

[0010] Step 3: Based on the trained model obtained in step 2, fuse the weights of multiple algorithms, and output the strategy comprehensive evaluation result by dynamically fusing the integral results generated by the scoring functions of each basic evaluation algorithm. The scoring function is the inherent scoring mechanism of the basic algorithm and will not be reconfigured.

[0011] Further, the step 2 specifically includes:

[0012] Step 2-1: For the determined game simulation data, the data contains the result data r of the game between K strategies, input the data into the set of basic evaluation methods constructed in step 1, and calculate the predicted ranking of each algorithm represents the priority prediction ranking of the i-th algorithm for the k-th strategy, i.e. the order of the k-th strategy in all K strategies based on the logic output by algorithm i; where i is the serial number of the basic evaluation algorithm, i = 1, 2,..., N, N is the total number of algorithms, k is the serial number of the strategy sample, k = 1, 2,..., K, K is the total number of strategies.

[0013] Step 2-2: Calculate the average absolute error MAE of the ranking of each algorithm:

[0014]

[0015] where i is the algorithm serial number, k is the strategy serial number, K is the total number of strategies, is the true ranking of strategy k, which is determined based on the actual game results of the strategy with all other strategies in the game simulation data, such as the strategy with the highest win rate whose true ranking is 1, is the true ranking calculated for strategy k.

[0016] Step 2-3: According to the MAE, dynamically assign the weight w of algorithm i i :

[0017]

[0018] where N is the number of evaluation algorithms. η is the temperature coefficient, which is determined based on experimental verification: this value can balance the sensitivity and stability of the weight difference of MAE, both strengthening the weight of low error algorithm and avoiding excessive concentration or dispersion of weight; its value is determined by multiple comparison experiments (η = 0.1, 0.5, 0.8, 1.0) in non-transitive game scenarios, and finally 0.5 is selected as the parameter to balance the sensitivity and stability of the weight. When η = 0.1 (too small), the influence of different MAE is weakened; when η = 0.8 (too large), it is too sensitive to MAE changes and unstable; when η = 0.5, it can use the algorithm indicators with better prediction, and prevent excessive concentration or dispersion, ensuring a certain diversity.

[0019] The initial weight of the basic evaluation method adopts the uniform distribution initialization method commonly used in the computer field, that is, for a set containing n algorithms, the initial weight of each algorithm is Avoiding initial bias, adapting to the characteristics of non-transitive scenarios; keeping consistency with the logic of dynamic weight training.

[0020] Step 2-4: Repeat steps 2-1 to 2-3 until all algorithm weights are stable.

[0021] Calculate the difference between the true value and the predicted value, and calculate the average error MAE of each algorithm i i The weight update depends on MAE i The smaller the prediction ranking error (MAE i , the larger the weight w i At this time, in each iteration, the prediction ranking of each algorithm is fused based on the current weight to form a comprehensive evaluation result; at the same time, new game data will update the true ranking, recalculate the MAE of each algorithm i , and then adjust the weight. This process makes the weight continuously tilt towards the algorithm with more reliable prediction ranking, and finally converges to a stable state.

[0022] Further, the determination condition of weight stability in step 2-4 is that in the last N iterations, the fluctuation range of each algorithm weight is less than 0.01, where N ∈ [10, 20].

[0023] Further, the game simulation data includes at least X field game results, and the standard of X is that each strategy in the game simulation has more than 3 times of interaction with any other strategy, and the data includes non-transitive strategy combination. The number can be 100 times. The core feature of transitive game is that there is a "cyclical advantage relationship" between strategies (such as A is better than B, B is better than C, but C is better than A), and this relationship needs enough game data to fully reflect: if the sample size is too small (such as <50 times), it may not cover all game situations of strategy combinations (for example, two groups of strategies have not played or have not played enough times); 100 games can ensure that each strategy interacts with other strategies more than 3 times (assuming the total number of strategies ≤10), which can ensure a certain stability. Generally speaking, according to experience, 100 times in a medium-sized game confrontation task can meet the demand for strategy diversity based on non-transitive game.

[0024] Further, step 2-1 includes preprocessing the game simulation data, including removing duplicate data.

[0025] Further, the specific steps of step 3 are:

[0026] Step 3-1: Calculate the score of strategy k based on the weight index obtained in step 2, and call each algorithm to calculate the score Score i (k). i (k) represents the score / sequence of new strategy k calculated by algorithm i.

[0027] Step 3-2: Calculate the comprehensive score FinalScore(k) according to the following formula:

[0028] FinalScore(k) = w1 Score1(k) + w2 Score2(k) +... + w N Score N (k)

[0029] Where N is the number of algorithms.

[0030] Step 3-3: Repeat steps 3-1 to 3-2 to obtain the comprehensive score of each strategy, and output the strategy ranking according to the comprehensive score FinalScore.

[0031] A dynamic constraint index system construction method for non-transitive game scenarios, and an index system constructed by the method.

[0032] The system comprises:

[0033] A basic evaluation algorithm pool: integrating Elo, Trueskill, and Alpha-Rank algorithms, supporting algorithm hot plug extension;

[0034] Data interface module: receive game result data, output structured data table, result data includes win-loss record, strategy name;

[0035] Weight calculation engine: based on MAE dynamic adjustment algorithm weight, built-in CUDA acceleration module;

[0036] Evaluation report generator: output strategy ranking, weight distribution histogram;

[0037] Data input module: used for obtaining game related data, including receiving game results, removing duplicate data, etc. Preprocessing, output structured data table.

[0038] GPU processing core module: build a set of basic evaluation methods through integrated algorithms; based on MAE dynamic weight calculation correction, built-in CUDA acceleration module realizes efficient iteration;

[0039] Dynamic evaluation output module: output strategy comprehensive evaluation results, including strategy ranking and other information.

[0040] The advantages of the method are:

[0041] (1) Solve the non-transitive evaluation distortion problem: in the non-transitive strategy data set containing the cycle advantage-disadvantage relationship (A→B→C→A), through multi-algorithm dynamic fusion, solve the ranking reversal error of single algorithm, and the evaluation accuracy is improved by 5%-8% compared with the optimal single algorithm (such as Alpha-Rank);

[0042] (2) Dynamically adapt to environmental and strategy changes: for dynamic data sets of strategy updates or environmental mutations, MAE can optimize the weight in real time, ensure that the weight fluctuation is less than 0.01, and avoid the adaptation lag problem of static weight

[0043] (3) Consider multi-dimensional evaluation requirements: in complex data sets that need to consider multiple factors (win rate, stability, etc.), integrate the dimensional advantages of different algorithms to output more comprehensive comprehensive ranking, and overcome the one-sidedness of single indicators. The final value is the two decimal places after 500 iterations, rounded to the nearest integer. Alpha-rank can solve the non-transitive strategy evaluation problem, and the present application uses the accuracy of alpha-rank combined with other algorithm results to improve the effect. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 The flowchart of the present application.

[0045] Figure 2 The use of the present application is shown in the figure. DETAILED DESCRIPTION

[0046] The technical solutions of the present application will be further described below with reference to the accompanying drawings.

[0047] The computing carrier is configured as shown below:

[0048] Hardware: x86 CPU + NVIDIA RTX 4090 graphics card (24GB video memory), supporting distributed computing;

[0049] Software: CUDA 12.2 parallel framework, achieving a weight calculation efficiency of 580 samples per second.

[0050] Referring to the accompanying Figure 1 (dynamic constraint index system method construction schematic diagram) and the accompanying Figure 2 (dynamic constraint index system method use schematic diagram), the specific implementation steps are as follows:

[0051] (I) System module

[0052] The system described in the application comprises:

[0053] Basic evaluation algorithm pool: integrated Elo, Trueskill, Alpha-Rank algorithm, supporting algorithm hot plug extension;

[0054] Data interface module: receiving game game result data (winning record, strategy name), outputting structured data table;

[0055] Weight calculation engine: based on MAE dynamic adjustment algorithm weight, built-in CUDA acceleration module;

[0056] Evaluation report generator: output strategy ranking, weight distribution histogram.

[0057] Data input module: used for obtaining game related data, including receiving game results (winning record, strategy name), removing duplicate data and other preprocessing, and outputting structured data table.

[0058] GPU processing core module: performs two major operations: (1) build a set of basic evaluation methods, integrated Elo, Trueskill, Alpha-Rank algorithm (support hot plug extension); (2) based on MAE dynamic weight calculation correction, built-in CUDA acceleration module to realize efficient iteration.

[0059] Dynamic evaluation output module: outputs the comprehensive evaluation results of the strategy, including strategy ranking and other information.

[0060] (II) Method implementation steps

[0061] Step 1: Algorithm integration and initialization

[0062] Deploy Elo, Trueskill, Alpha-Rank algorithm in the system, set the initial weight ω0 = [1 / 3, 1 / 3, 1 / 3] (corresponding to Elo, Trueskill, Alpha-Rank);

[0063] Input is the result data set of the game Where: k is the serial number of the strategy sample (k = 1, 2,..., K, K is the total number of strategies); j is the serial number of the opponent strategy (j = 1, 2,..., K, j ≠ k); r kj is the result of the game between strategy k and strategy j, where r kj = 1 means that strategy k wins, r kj = 0 means that strategy k loses.

[0064] Output is the strategy score Score i (k) of each algorithm, where i = 1, 2,..., N, N is the total number of algorithms, Score i (k) represents the comprehensive score of the i-th algorithm for the k-th strategy, which is calculated based on all game results.

[0065] Step 2: Game data processing

[0066] Step 2-1: Collect game data, including multiple strategy types, remove duplicate games to form a training set; Label the true ranking of the k-th strategy as the error calculation benchmark.

[0067] In step 2-1, the initial weight of the algorithm is set to be uniformly distributed, that is, for n basic evaluation algorithms, the initial weight This setting can avoid initial bias and ensure objective convergence of subsequent dynamic weight training based on game simulation data.

[0068] Step 2-2: Error calculation: for each algorithm i, calculate the average absolute error of the ranking:

[0069]

[0070] (K is the sample number, is the predicted ranking of algorithm i for strategy k);

[0071] Step 2-3: weight update, adjust the weight through the exponential weighting formula:

[0072]

[0073] Step 2-4, repeat steps 2-1 to 2-3 until the weights of all algorithms are stable, the termination condition is that when the weight fluctuation Δω j <0.01 in the last 10 iterations, stop training, and the final weight is ωfinal = (ω1, …, ω N )

[0074] Step 3: Policy evaluation execution

[0075] Step 3-1: Calculate the score of policy s based on the weight indicators obtained in step 2 new , call each algorithm to calculate the score Score j (k), Score j (k) represents the score of the new policy s new by algorithm j;

[0076] Step 3-2: Calculate the comprehensive score FinalScore(s new ) according to the following formula:

[0077] FinalScore(k) = ω1·Score1(k) + … + ω N ·Score N (k)

[0078] Step 3-3: Repeat steps 3-1 to 3-2 to obtain the comprehensive score of each policy, and output the policy ranking according to the comprehensive score FinalScore.

[0079] (Three) Method verification

[0080] The data set used for verification has the following typical characteristics (common characteristics of non-transitive game scenarios):

[0081] Non-transitive strategy: strategy combination containing "cyclic superiority relationship" (such as A is better than B, B is better than C, but C is better than A), single algorithm is difficult to capture this nonlinear relationship;

[0082] Dynamic: strategies will be updated during the game process (such as strategy B surpasses strategy A after optimization in the later period), environmental parameters may mutate (such as rule adjustment leading to change of win-lose logic);

[0083] Evaluation dimension diversity: strategy superiority is affected by multiple factors (such as win rate, stability, resource consumption, etc.), single index cannot comprehensively measure.

[0084] Compared with the prior art:

[0085] (1) Evaluation accuracy is improved: through 500 iterations of experiments, the accuracy of the constraint index system method in the verification set reaches 96%, which is improved by 6% compared with the single Alpha-Rank algorithm (see Table 2).

[0086] (2) Weight dynamic optimization: the model automatically weakens the low-efficiency algorithm (such as Elo weight eventually reduces to 0) and strengthens the high-efficiency algorithm (such as Alpha-Rank weight is increased from 0.35 to 0.70, see Table 1).

[0087] (3) Guarantee of calculation efficiency: based on a GPU acceleration platform, a throughput of 580 samples per second is supported, and the inference time is only 52 ms (see Table 2).

[0088] Table 1: Evaluation algorithm weight adjustment record table

[0089]

[0090] Table 2: Comparison of effect of constraint index system with original algorithm

[0091]

[0092] The application provides a dynamic constraint index system construction method and system for a non-transitive game scenario. There are many methods and approaches to realize the technical solution, and the above description is only the preferred embodiment of the application. It should be pointed out that for ordinary technical personnel in the technical field, some improvements and refinements can be made without departing from the principles of the application, and these improvements and refinements should be regarded as the protection scope of the application. The components not explicitly described in the embodiment can be realized by using existing technology.

Claims

1. A method for constructing a dynamic constraint index system for evaluation in non-transitive game scenarios, characterized in that, Includes the following steps: Step 1: Construct a set of basic evaluation methods, which includes at least Elo, Truskill, and Alpha-Rank algorithms, and initialize the algorithm weights; Step 2: Train the evaluation model based on game simulation data, and determine the weights of the elements for the set of basic evaluation methods obtained in Step 1; Step 3: Based on the trained model obtained in Step 2, the weights of multiple algorithms are fused. The integral results generated by the scoring functions of each basic evaluation algorithm are fused through dynamic weight fusion, and the comprehensive evaluation result of the strategy is output.

2. The method for constructing a dynamic constraint index system for non-transitive game scenarios according to claim 1, characterized in that, Step 2 specifically includes: Step 2-1: For the given game simulation data, which contains game result data r for K strategies, input the data into the basic evaluation method set constructed in Step 1, and calculate the predicted ranking of each algorithm. This represents the priority prediction ranking of the i-th algorithm for the k-th strategy, that is, the ranking of the k-th strategy among all K strategies based on the logic output of the i-th algorithm; where i is the index of the basic evaluation algorithm, i = 1, 2, ..., N, where N is the total number of algorithms, and k is the index of the strategy sample, k = 1, 2, ..., K, where K is the total number of strategies; Step 2-2: Calculate the mean absolute error (MAE) for each algorithm's ranking. Where i is the algorithm index, k is the strategy index, and K is the total number of strategies. The true ranking of the k-th strategy is determined based on the actual game results of this strategy against all other strategies in the game simulation data. The true ranking calculated for strategy k; Steps 2-3: Dynamically assign weights w to algorithm i based on MAE. i : (η=0.5, and ∑w i =1) Where N is the number of evaluation algorithms; Step 2-4: Repeat steps 2-1 to 2-3 until all algorithm weights are stable.

3. The method for constructing a dynamic constraint index system for non-transitive game scenarios according to claim 2, characterized in that, The criterion for determining the stability of the weights mentioned in steps 2-4 is that the fluctuation range of the weights of each algorithm is less than 0.01 in N consecutive iterations.

4. The method for constructing a dynamic constraint index system for non-transitive game scenarios according to claim 3, characterized in that, Where N∈[10,20].

5. The method for constructing a dynamic constraint index system for non-transitive game scenarios according to claim 1, characterized in that, The game simulation data includes at least X game results, where X is defined as each strategy in the game simulation having played against any other strategy more than 3 times. The data includes non-transitive strategy combinations.

6. The method for constructing a dynamic constraint index system for non-transitive game scenarios according to claim 1, characterized in that, Step 2-1 includes preprocessing the game simulation data, including removing duplicate data.

7. The method for constructing a dynamic constraint index system for non-transitive game scenarios according to claim 1, characterized in that, The specific steps of step 3 are as follows: Step 3-1: Calculate the new strategy k based on the weight index obtained in Step 2. new The score is calculated by calling various algorithms. i (k new Score i (k new ) represents the algorithm i for the new strategy k new The rating; Step 3-2: Calculate the final score (k) according to the following formula. new ): FinalScore(k new )=w1·Score1(k new )+w2·Score2(k new )+...+w N ·Score N (k new ) Where N is the number of algorithms; Step 3-3: Repeat steps 3-1 to 3-2 to obtain the comprehensive score of each strategy, and output the strategy ranking according to the comprehensive score FinalScore.

8. The method for constructing a dynamic constraint index system for non-transitive game scenarios according to claim 1, characterized in that, The initialization method in step 1 is a uniform distribution initialization method, and the initial weights of each algorithm are all... Where n is the number of algorithms.

9. A system for constructing a dynamic constraint index system for non-transitive game scenarios, characterized in that, The indicator system is constructed by applying any one of the dynamic constraint indicator system construction methods for non-transitive game scenarios in claims 1-8.

10. The system for constructing a dynamic constraint index system for non-transitive game scenarios according to claim 7, characterized in that, include: Basic evaluation algorithm pool: integrates Elo, Trueskill, and Alpha-Rank algorithms, and supports hot-swappable algorithm expansion; Data interface module: Receives game result data and outputs structured data tables. The result data includes win / loss records and strategy names. Weight calculation engine: Based on MAE, dynamically adjusts algorithm weights, with a built-in CUDA acceleration module; Evaluation report generator: Outputs strategy rankings and weight distribution histograms; Data input module: Used to acquire game-related data, including receiving game results, removing duplicate data and other preprocessing, and outputting structured data tables. GPU processing core module: It constructs a set of basic evaluation methods through integrated algorithms; it dynamically calculates and corrects weights based on MAE; and it has a built-in CUDA acceleration module to achieve efficient iteration. Dynamic evaluation output module: Outputs comprehensive evaluation results of strategies, including strategy ranking and other information.

Images