Large language model dynamic valuation method and system based on parallel game

Abstract

The invention discloses a large language model dynamic valuation method and system based on parallel game, and the method comprises the steps: firstly constructing a parallel game evaluation environment, and setting game scene parameters; and loading the to-be-evaluated large language model as an intelligent agent, and distributing roles and initial contexts. And then driving the intelligent agent to perform multi-round interactive gaming according to rules, monitoring violation in real time and recording data. And multi-dimensional evaluation results such as winning rate, influence, collaborative contribution degree and compliance are calculated based on the data. And finally, performing weighted fusion on the multi-dimensional result according to the evaluation index weight to generate a comprehensive capability estimated value of each model, thereby providing a more comprehensive and accurate basis for model evaluation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a dynamic evaluation method and system for large language models based on parallel game theory. Background Technology

[0002] With the widespread application of large language models in fields such as natural language processing, their number and application scenarios are constantly increasing, creating an urgent need for an objective and impartial evaluation method to measure the capabilities of different models. Existing evaluation methods have significant shortcomings. Most are based on static benchmark datasets, failing to consider the model's performance in multi-turn dialogues or dynamic decision-making scenarios, resulting in poor interactivity. Furthermore, the evaluation metrics are singular, ignoring dimensions such as influence and collaborative ability. Different models are evaluated independently, lacking relative comparisons. Manual evaluation is costly and yields unstable results. At the same time, the compliance of model outputs is often overlooked. Summary of the Invention

[0003] The purpose of this invention is to provide a dynamic evaluation method and system for large language models based on parallel game theory.

[0004] In a first aspect, embodiments of the present invention provide a dynamic evaluation method for large language models based on parallel game theory, comprising:

[0005] Construct a parallel game evaluation environment and set game scenario parameters for the large language model used for evaluation. The scenario parameters include at least the game type, upper limit of the number of interaction rounds, rule constraints, violation handling strategies, and evaluation index weights.

[0006] Load at least two large language models to be evaluated as agents, and assign corresponding role information and initial context to each agent so that they meet the interaction conditions of the game scenario;

[0007] In the parallel game evaluation environment, the agent is driven to play multiple rounds of interactive games according to preset rules. During the game, the illegal behavior of each agent is monitored in real time, and the interaction content, behavioral decisions, illegal events and corresponding game result values ​​are recorded. The game result values ​​are the task completion degree, win or loss judgment or profit score of each round of the game.

[0008] Based on the game process data and the recorded game result values, multi-dimensional evaluation results are calculated for each agent, including at least the win rate index, influence index, cooperation contribution index, and compliance index.

[0009] Based on the weights of the evaluation indicators, the multidimensional evaluation results are weighted and fused to generate comprehensive capability valuation results for each language model.

[0010] In one possible implementation, the game type includes at least one or more of adversarial game scenarios, cooperative game scenarios, and negotiated game scenarios. The adversarial game scenarios include debate scenarios, in which a preset adjudication algorithm evaluates the output of the agents to determine the winner of the game. The cooperative game scenarios include cooperative programming scenarios, and the negotiated game scenarios include negotiation scenarios.

[0011] In one possible implementation, the multi-round interactive game is executed in parallel, with multiple game sessions running simultaneously in the same evaluation environment to improve the evaluation efficiency of large language model valuation.

[0012] In one possible implementation, the win rate metric is calculated in the following manner:

[0013] The statistical target large language model is the number of games it wins in multiple games it participates in, and the ratio of the number of winning games to the total number of games it participates in is used as the win rate indicator.

[0014] When the game results in a draw or involves multiple participants, the win / loss result is recalculated or split according to preset rules to correct the calculated win rate index.

[0015] In one possible implementation, the influence metric is determined by the following methods:

[0016] Under the same game conditions, the game outcome value with and without the participation of the target large language model is calculated respectively, and the difference between the two is taken as the influence increment of the target large language model in the game.

[0017] The incremental influence gained from multiple game scenarios is statistically averaged, and the result of the statistical average is used as the influence index of the target large language model.

[0018] In one possible implementation, in a cooperative game scenario, the cooperative contribution index is determined based on the Shapley value calculation rule in cooperative game theory, including:

[0019] By traversing or sampling different model subsets that do not contain the target large language model, the changes in the game outcome before and after the target large language model is added to the corresponding subset are calculated, and a weighted average is performed in combination with the weighted proportion corresponding to the size of the model subset to obtain the collaborative contribution index of the target large language model.

[0020] In one possible implementation, the compliance index is determined based on the proportion of the target large language model's output content conforming to preset rule constraints during the game interaction process, and differentiated deduction weights are set according to the severity of different violations.

[0021] When a pre-defined serious violation is detected, the compliance indicator will be set to zero or the system will be deemed unqualified.

[0022] In one possible implementation, the method further includes:

[0023] When constructing the parallel game evaluation environment, the capability attainment threshold and capability overflow threshold are pre-set as sub-parameters of the game scenario parameters. The capability attainment threshold is the minimum performance standard for determining whether the agent can adapt to the difficulty of the current scenario, and the capability overflow threshold is the performance standard for determining whether the agent's capabilities are not fully utilized in the current scenario.

[0024] In the multi-round interactive game process, after each preset round of games is completed, the comprehensive performance score of the target intelligent agent is obtained by weighted summation based on the win rate index and compliance index.

[0025] If the overall performance score exceeds the capability overflow threshold for a preset number of consecutive times, the game scenario parameters are adjusted. The adjustment includes at least one of increasing the upper limit of the number of interaction rounds, increasing the complexity of rule constraints, and increasing the weight of violation handling.

[0026] If the overall performance score is lower than the capability threshold for a number of consecutive preset times, the game scenario parameters are adjusted. The adjustment includes at least one of simplifying the complexity of the game issues, reducing the upper limit of the number of interaction rounds, and relaxing the constraints of non-core rules.

[0027] The game data after scene adjustment is corrected according to a preset weighting coefficient and then included in the calculation of the multidimensional evaluation result. The difference between the evaluation data before and after scene adjustment is used as a robustness supplementary indicator and participates in the weighted fusion of the comprehensive capability valuation result together with the multidimensional evaluation result.

[0028] In one possible implementation, the multidimensional evaluation result further includes a strategy evolution capability index, the calculation process of which includes:

[0029] In the step of recording interaction content, behavioral decisions, violation events and corresponding game result values, a game strategy evolution trajectory library is constructed simultaneously for each large language model to be evaluated. After the behavioral decisions are refined into decision logic annotations, they are stored together with the interaction content, opponent response text and game result values ​​in the trajectory library.

[0030] Based on the game data of two consecutive rounds in the trajectory library, the semantic similarity and decision logic difference of the output content of the target large language model in two consecutive rounds are calculated by semantic matching algorithm. If the semantic similarity of the output of two consecutive rounds is lower than the first preset threshold, the decision logic difference is higher than the second preset ratio, and the result of the later round is better than the result of the earlier round, it is determined to be an effective strategy adjustment. The ratio of the number of effective strategy adjustments to the total number of decisions of the target large language model is used to obtain the strategy adjustment efficiency index.

[0031] Based on all game output content in the trajectory library, the topic is classified by a topic clustering model, and the strategy diversity index is obtained by counting the proportion of different topic types to the total number of output topics.

[0032] The strategy evolution capability index is obtained by weighted summation of the strategy adjustment efficiency and the strategy diversity index, wherein the strategy adjustment efficiency has a higher weight than the strategy diversity index.

[0033] In the weighted fusion step, the proportion of the strategy evolution capability indicator in the weight of the evaluation indicators is dynamically adjusted according to the game type, so that it, together with the win rate indicator, influence indicator, collaborative contribution indicator, and compliance indicator, generates the comprehensive capability valuation result. If it is detected that the target large language model repeatedly uses the same strategy for a preset number of consecutive games and the game results continue to deteriorate, the comprehensive capability valuation result is reduced by a preset proportion by adjusting the weight of the evaluation indicators.

[0034] Secondly, embodiments of the present invention provide a dynamic evaluation system for a large language model based on parallel game theory, comprising:

[0035] The module is used to build a parallel game evaluation environment, set game scenario parameters for large language models to be evaluated, and the scenario parameters include at least game type, upper limit of interaction rounds, rule constraints, violation handling strategy and evaluation index weight; at least two large language models to be evaluated are loaded as agents, and corresponding role information and initial context are assigned to each agent to meet the interaction conditions of the game scenario.

[0036] The evaluation module is used to drive the agents to engage in multiple rounds of interactive games according to preset rules in the parallel game evaluation environment. During the game, it monitors the violations of each agent in real time and records the interaction content, behavioral decisions, violation events, and corresponding game result values, where the game result values ​​are the task completion degree, win / loss determination, or gain score for each round. Based on the game process data and the recorded game result values, it calculates multi-dimensional evaluation results for each agent, including at least a win rate indicator, an influence indicator, a collaborative contribution indicator, and a compliance indicator. According to the weights of the evaluation indicators, it performs weighted fusion of the multi-dimensional evaluation results to generate a comprehensive capability valuation result for each language model.

[0037] Compared to existing technologies, the advantages of this invention include: Employing the dynamic evaluation method and system for large language models based on parallel game theory disclosed in this invention, a parallel game evaluation environment is constructed, and game scenario parameters are set. The large language model to be evaluated is loaded as an agent, and roles and initial contexts are assigned. Then, the agent is driven to engage in multiple rounds of interactive game play according to rules, with real-time monitoring of violations and data recording. Based on this data, multi-dimensional evaluation results such as win rate, influence, collaborative contribution, and compliance are calculated. Finally, the multi-dimensional results are weighted and fused according to the weights of the evaluation indicators to generate a comprehensive capability estimate for each model, providing a more comprehensive and accurate basis for model evaluation. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as limiting the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A flowchart illustrating the dynamic estimation method for large language models based on parallel game theory provided in this embodiment of the invention;

[0040] Figure 2 A schematic diagram of the overall architecture of the dynamic evaluation system for large language models based on parallel game theory provided in this embodiment of the invention;

[0041] Figure 3 A schematic diagram of the process framework for the dynamic evaluation method of large language models based on parallel game theory provided in an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of a parallel game evaluation scenario provided in an embodiment of the present invention;

[0043] Figure 5A schematic block diagram of the structure of a large language model dynamic evaluation system based on parallel game theory provided in an embodiment of the present invention;

[0044] Figure 6 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0046] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0047] In order to solve the technical problems mentioned in the background art Figure 1 This is a flowchart illustrating the dynamic estimation method for large language models based on parallel game theory provided in this embodiment. The following is a detailed description of this dynamic estimation method for large language models based on parallel game theory.

[0048] Step S201: Construct a parallel game evaluation environment and set game scenario parameters for the large language model used for evaluation. The scenario parameters include at least the game type, upper limit of the number of interaction rounds, rule constraints, violation handling strategy and evaluation index weights.

[0049] Step S202: Load at least two large language models to be evaluated as agents, and assign corresponding role information and initial context to each agent so that they meet the interaction conditions of the game scenario;

[0050] Step S203: In the parallel game evaluation environment, the agent is driven to play multiple rounds of interactive games according to preset rules. During the game, the illegal behavior of each agent is monitored in real time, and the interaction content, behavioral decisions, illegal events and corresponding game result values ​​are recorded. The game result values ​​are the task completion degree, win or loss judgment or profit score of each round of the game.

[0051] Step S204: Based on the game process data and the recorded game result values, calculate multi-dimensional evaluation results for each agent, including at least the win rate index, influence index, cooperation contribution index, and compliance index.

[0052] Step S205: Based on the weights of the evaluation indicators, the multidimensional evaluation results are weighted and fused to generate comprehensive capability valuation results for each language model.

[0053] In this embodiment of the invention, the exemplary method for dynamic evaluation of large language models based on parallel game theory is executed by a server (or central control unit) with a dedicated evaluation system deployed. The server, as the core scheduling and computational hub of the entire evaluation process, is responsible for coordinating the operation of all modules, processing data, and generating the final evaluation result. The method will now be described in detail with reference to the accompanying drawings and embodiments.

[0054] After the server starts, it first runs its internal environment configuration module. This module provides a structured human-computer interaction interface (such as a web configuration panel or API interface) for the evaluation operator to set all the game scenario parameters required for the current evaluation task. Based on the evaluation objective, the operator selects one or more scenarios from the server's pre-set game type library. For example, to evaluate the model's logical debating and persuasion abilities, the "debate scenario" is selected; to examine the model's collaborative work ability in technical tasks, the "cooperative programming scenario" is selected; and to test the model's strategic negotiation and cost-benefit balancing abilities, the "negotiation scenario" is selected.

[0055] After selecting a scenario, the server requires the operator to configure specific scenario parameters. Taking a "debate scenario" as an example, the operator enters the topic of the debate in the interface, such as "The overall impact of artificial intelligence on social employment is positive." Next, an upper limit on the number of interaction rounds is set, such as 10 rounds, to ensure that the debate does not go on indefinitely. Then, rules and constraints are defined, such as: prohibiting the use of personally offensive language, prohibiting the fabrication of false data, and limiting each round of speech to no more than 300 words. At the same time, the server allows the operator to set violation handling strategies, such as: issuing a warning and recording the first minor violation (such as using impolite language), and deducting 10 points from the compliance score of the current debate for the second similar violation; if a serious violation is detected (such as generating hate speech), the current game is immediately terminated, and the violating model is judged to have failed in this game.

[0056] Finally, and crucially, the operator needs to set the weighting coefficients for each evaluation metric in the subsequent comprehensive assessment. The server interface displays four core metrics: win rate (W), influence (I), collaborative contribution (φ), and compliance (C), and provides a weighting slider or input box. For example, in a debate assessment that emphasizes competition and persuasion, the operator might set the weighting to: win rate weighting. =0.4, Influence Weight =0.3, Collaboration Contribution Weight =0.0 (because debates are usually adversarial), compliance weight =0.3. After all parameters are configured, the server persists these parameters and initializes a brand new, isolated parallel game evaluation environment instance to prepare for loading the model agent.

[0057] Once the environment is set up, the server's agent management module begins operation. This module loads the corresponding large language models from the model repository connected to the server, based on the number of participating models specified in the environment configuration (e.g., two models are needed for a debate scenario). These models may exist as API services, locally deployed inference engines, or containerized instances. The server creates an independent agent sandbox environment for each model to be evaluated, ensuring that the models do not interfere with each other's operations.

[0058] After loading the model, the server assigns specific role information and initial context to each agent based on the selected game scenario. This assignment is accomplished by sending structured "prompts" to each agent. For example, in the "collaborative programming scenario," the task is to "collaborately develop a Python function to verify the correctness of email address formats." The server sends initialization prompts to the three loaded agents (tentatively named Model_A, Model_B, and Model_C). The prompt for Model_A might be: "You are a programmer skilled in writing regular expressions. Now you need to collaborate with two other programmers to complete a Python function that verifies email address formats. Please propose your implementation ideas." The prompt for Model_B might be: "You are a programmer who values ​​code robustness and exception handling. Your task is to collaboratively complete the email verification function; please pay attention to input boundary conditions and error handling." The prompt for Model_C might be: "You are a programmer responsible for code integration and testing. Please coordinate the code of the first two colleagues to ensure the function ultimately runs correctly." In this way, the server injects role awareness and task starting points into each agent, enabling them to enter the pre-defined game interaction state. The server records the initial state of all agents and prepares to submit them to the game engine.

[0059] Once all agents are ready, the server's game engine module is activated, driving the execution of pre-defined game logic. The engine schedules interactions between agents in parallel or serial mode, depending on the scenario type. It maintains a central state machine that records the current round, speaking order, historical dialogues, and intermediate results.

[0060] Taking a "negotiation scenario" as an example, the scenario is set as "software service procurement negotiation," with a maximum of 5 rounds of interaction. The server engine first sends a prompt to the agent playing the role of the "seller," including its role (software service provider), initial offer (1 million yuan per year), minimum acceptable price (800,000 yuan), and the goal of this round of negotiation (to negotiate a higher price). The agent generates a negotiation statement, such as: "Our software service includes 24 / 7 technical support and free upgrades; the 1 million yuan annual fee reflects its superior value." The engine records this output and uses it as part of the input context for the "buyer" agent, along with information such as the "buyer's" own role (purchaser), initial counter-offer target (600,000 yuan), and maximum budget (900,000 yuan). Next, the "buyer" agent generates a response, such as: "Considering the prices of similar products in the market, we believe 600,000 yuan is a reasonable price; the value of your additional services needs to be quantified more specifically." The engine continues recording and begins the next round of scheduling.

[0061] Throughout the multi-round interaction, the server's game engine incorporates a real-time compliance monitoring unit. This unit scans every piece of text generated by each agent, checking for violations of the rules set in step one. For example, if the rule prohibits "making promises that cannot be kept," and the "seller" agent claims in a round that it "guarantees zero software vulnerabilities," the monitoring unit will immediately identify this potential violation, mark it as "minor violation - over-promise," record it in the violation event log, and deduct from the model's compliance score at the end of the game according to a preset strategy. If an agent outputs seriously harmful content, the monitoring unit will trigger emergency measures, immediately suspending the current game.

[0062] Meanwhile, the engine determines whether the game has ended based on the scenario logic. In a negotiation scenario, the ending conditions might be: one party explicitly accepting the other's terms, both parties reaching an agreement, or failing to reach an agreement after the maximum number of rounds. When the game ends, the engine invokes a preset adjudication mechanism to calculate the outcome value of this round. For negotiations, the outcome value might be a comprehensive score: if an agreement is reached, the respective payout scores for both parties are calculated based on the closeness of the final price to their respective bottom prices; if negotiations break down, both parties receive low payout scores. The server engine packages the complete interaction content of this game, the behavioral decision sequence of each agent, all violations, and the final game outcome value into a structured data packet and sends it to the evaluation and analysis module. To improve evaluation efficiency, the server can run dozens or even hundreds of such games simultaneously, i.e., "parallel games," with the engine module responsible for the concurrent scheduling and state management of all games.

[0063] After all the pre-set matches (e.g., having Model_A and Model_B engage in 100 independent debates) have been completed, the server's evaluation and analysis module begins to perform batch analysis on the massive amount of interaction data, calculating a series of dynamic metrics for each large language model participating in the evaluation.

[0064] First, the win rate metric is calculated. The server retrieves data from the database and counts the number of debates Model_A won out of all 100 debates, assuming 65. Therefore, Model_A's win rate W_A = 65 / 100 = 0.65. For tiebreakers, for example, if the algorithm determines 5 debates to be tied, the server might, according to the rules, treat these 5 debates as each receiving 0.5 wins, thus adjusting the calculation accordingly.

[0065] Secondly, the influence index is calculated. The server uses a "leave one out" approach for analysis. Taking one debate as an example, the final overall performance score (given by the adjudication algorithm) is F(S) = 85 points (S includes Model_A and Model_B). The server estimates the possible score F(S{A}) if Model_A had not participated, and only Model_B had presented (or interacted with a baseline model). Assuming the estimated value is 50 points, then Model_A's influence increment in this round is Δ(A) = 85 - 50 = 35 points. The server repeats this process for all 100 debates, calculating 100 Δ(A) values, and then taking their arithmetic mean to obtain Model_A's influence score I_A. This score reflects the extent to which Model_A's participation typically improves the outcome of the debate.

[0066] Next, in the "collaborative programming scenario," the collaboration contribution metric is calculated. The server uses the Shapley value method from cooperative game theory. Assume there are three models in this evaluation (N={A,B,C}). The server needs to calculate the Shapley value φ(A) for Model_A. This requires evaluating the performance of all subsets S that do not contain A. For example: Subset S={} (empty set): Performance F({})=0. Subset S={B}: Performance F({B}), which is the score obtained by B alone attempting the task. Subset S={C}: Performance F({C}). Subset S={B,C}: Performance F({B,C}), which is the score of B and C collaborating. Then, for each subset S, the marginal contribution after A's inclusion is calculated: F(S∪{A})-F(S). Finally, these marginal contributions are weighted and averaged according to the Shapley value formula (considering the weights of subsets of different sizes). In practice, the server may use approximation algorithms such as Monte Carlo sampling to efficiently calculate and thus fairly quantify the average contribution φ of each model to the team's successful completion of the programming task.

[0067] Finally, compliance metrics are calculated. The server aggregates violation records for each model across all matches. Assume Model_A triggered 3 minor violation warnings and had no serious violations in a total of 500 rounds. Its number of rounds without violations is 497. Its base compliance score C_A = 497 / 500 = 0.994. The server then calculates the final compliance score C_A' = 0.994 - 3 * 0.02 = 0.934 based on preset deduction rules (e.g., deducting 0.02 points for each minor violation). If a model has one serious violation, the server may directly set its compliance metric to zero.

[0068] After obtaining the raw scores for each model across four dimensions—win rate (W), influence (I), collaborative contribution (φ), and compliance (C)—the server's evaluation and analysis module calls the weight vector set in step one. , , , For example, in this comprehensive evaluation, the weights are set as follows: =0.3, =0.3, =0.2, =0.2.

[0069] The server then applies a comprehensive capability estimation function to each model: V= W+ I+ φ+ C. Substituting the various indicators of Model_A into the calculation: V_A = 0.3 + 0.65 + 0.3I_A (assumed to be 0.7) + 0.2φ_A (assumed to be 0.8) + 0.2 + 0.934. After calculation, the overall capability estimate of Model_A is V_A ≈ 0.762.

[0070] The server repeats this calculation process for all models participating in the evaluation, ultimately generating an ordered ranking of model capabilities. Simultaneously, the evaluation and analysis module automatically generates a detailed and visual evaluation report. The report clearly lists each model's score and final overall valuation for each metric in tabular form, and may be supplemented with radar charts to visually illustrate the differences in capability dimensions between different models. For example, the report might show that Model_A has a high win rate and impact but slightly weaker compliance, while another Model_X has perfect compliance but only average collaborative contribution. This final report generated by the server provides users with comprehensive, quantitative, and comparable decision-making support, guiding the selection of large language models, performance bottleneck diagnosis, or iterative optimization.

[0071] Through the above five steps, the server fully executes the automated and parallel dynamic evaluation process of large language models, from environment construction to result output, effectively realizing the scientific evaluation of the model's comprehensive capabilities in complex interactive environments.

[0072] In this embodiment of the invention, the game type includes at least one or more of adversarial game scenarios, cooperative game scenarios, and negotiated game scenarios. The adversarial game scenario includes a debate scenario, in which a preset adjudication algorithm evaluates the output of the agent to determine the winner of the game. The cooperative game scenario includes a cooperative programming scenario, and the negotiated game scenario includes a negotiation scenario.

[0073] In this embodiment of the invention, for example, when the server performs the evaluation, it specifically constructs and drives the following types of game scenarios according to the settings of the environment configuration module:

[0074] In adversarial game scenarios, such as debates, the server sets the topic as "The advantages of remote work outweigh the disadvantages." The server loads two large language model agents as the affirmative and negative sides, respectively, and injects their respective stances and initial arguments. The game engine drives the two sides to take turns speaking in multiple rounds. After each round, the server submits the dialogue record to a built-in adjudication algorithm. This algorithm scores both sides' outputs based on dimensions such as logical consistency and persuasiveness of arguments. After all rounds are completed, the server compares the cumulative scores, declares the side with the higher score the winner, and records the result.

[0075] In collaborative game scenarios, such as cooperative programming, the server sets the task as "collaboratively writing a web crawler program." The server loads three model agents and assigns them initial contexts, prompting them to focus on URL parsing, data cleaning, and anomaly handling, respectively. The game engine coordinates the agents to exchange code snippets and suggestions for improvement in a shared session environment. The server tracks the valid code blocks submitted by each agent and the key errors fixed. At the end of the game, the server evaluates the team's overall gain based on the correct execution of the final program and provides the data foundation for subsequent calculations of the Shapley values ​​for each model.

[0076] In negotiation-based game scenarios, such as bargaining, the server assigns roles as "equipment purchaser" and "supplier," and provides them with initial quotes, target prices, and budget constraints. The game engine controls multiple rounds of bidding and counter-offers between the two parties. The server records the content of each round's proposal in real time and monitors for any fraudulent practices such as false promises. The game ends either when one party accepts the proposal and an agreement is reached, or when all rounds are exhausted and negotiations break down. Based on the closeness of the agreed-upon transaction price to each party's bottom line, the server uses a preset formula to calculate the payoff score for each party as the outcome value for this round of the game.

[0077] In this embodiment of the invention, the multi-round interactive game is executed in parallel, with multiple game sessions running simultaneously in the same evaluation environment, in order to improve the evaluation efficiency of large language model valuation.

[0078] In this embodiment of the invention, for example, the server's game engine module employs a parallel scheduling strategy to improve throughput when driving interactions. For instance, when evaluating the debating capabilities of two large language models (such as Model_Alpha and Model_Beta), the server does not have them play against each other one by one, but instead initializes and runs 50 independent debate game instances simultaneously. Each instance has the same scenario rules (such as a 5-round debate), but is assigned different debate topics (such as "Artificial intelligence should be regulated" for instance 1, and "Climate change is the most pressing global issue" for instance 2, etc.). The server uses containerization technology or asynchronous threads to make these 50 games run concurrently. The engine receives the statements generated by the agents in all instances in parallel and calls the adjudication algorithm to process them synchronously. Ultimately, the server collects 50 sets of win / loss records and interaction data in far less time than required for serial execution, which are used for subsequent calculation of win rate and influence indicators, thereby efficiently completing large-scale evaluations.

[0079] In this embodiment of the invention, the win rate metric is calculated in the following manner:

[0080] The statistical target large language model is the number of games it wins in multiple games it participates in, and the ratio of the number of winning games to the total number of games it participates in is used as the win rate indicator.

[0081] When the game results in a draw or involves multiple participants, the win / loss result is recalculated or split according to preset rules to correct the calculated win rate index.

[0082] In this embodiment of the invention, for example, when calculating the win rate metric, the server counts the number of games won by the target large language model in the multiple game scenarios it participates in. For instance, if the server lets model A participate in 100 debate scenarios and model A wins 60 of them, then the initial win rate metric for model A is 60 ÷ 100 = 0.6.

[0083] When a game ends in a draw or involves multiple parties, the server adjusts the win rate metric according to preset rules. Taking a three-party negotiation scenario as an example, with 80 games played, Model B won 20 games and 10 ended in a draw. The preset rules stipulate that each party receives 0.5 wins in a draw. Therefore, Model B's effective number of wins is 20 + 10 × 0.5 = 25, and its win rate metric is adjusted to 25 ÷ 80 = 0.3125. In this way, the server accurately calculates the win rate metric for each model, providing reliable data for subsequent comprehensive capability evaluation.

[0084] In this embodiment of the invention, the influence index is determined in the following ways:

[0085] Under the same game conditions, the game outcome value with and without the participation of the target large language model is calculated respectively, and the difference between the two is taken as the influence increment of the target large language model in the game.

[0086] The incremental influence gained from multiple game scenarios is statistically averaged, and the result of the statistical average is used as the influence index of the target large language model.

[0087] In this embodiment of the invention, for example, the server determines the influence index of the target large language model according to specific steps. Taking a collaborative programming scenario as an example, the server organizes multiple programming task matches.

[0088] In a game, the server calculates the game outcome value including the target model C. This programming task requires completing a data analysis program. After all models participate, the program is evaluated based on its completion rate, running efficiency, and other factors, resulting in a payoff score of 80 points. Next, the server simulates a scenario without model C, allowing other models to complete the same task, ultimately obtaining a payoff score of 60 points. Therefore, the incremental influence of model C in this game is 80 - 60 = 20 points. The server repeats this calculation process for multiple such games. Assuming a total of 10 games are played, the incremental influence of model C in each game is 20 points, 15 points, 22 points, etc. The server adds these 10 incremental influence values ​​together and divides by 10 to obtain a statistical average. This statistical average is the influence index of model C. In this way, the server can accurately quantify the influence of the target large language model on the game outcome.

[0089] In this embodiment of the invention, in a cooperative game scenario, the cooperative contribution index is determined based on the Shapley value calculation rule in cooperative game theory, and can be implemented through the following example.

[0090] By traversing or sampling different model subsets that do not contain the target large language model, the changes in the game outcome before and after the target large language model is added to the corresponding subset are calculated, and a weighted average is performed in combination with the weighted proportion corresponding to the size of the model subset to obtain the collaborative contribution index of the target large language model.

[0091] In this embodiment of the invention, for example, in a collaborative game scenario, when the server calculates the collaborative contribution index of the target large language model, it follows the Shapley value calculation rules. Taking a collaborative programming task of jointly developing a simple game as an example, there are three models, A, B, and C, participating in the evaluation.

[0092] First, the server iterates through or samples different subsets that do not contain the target model A. When the subset is empty, the performance score is 0 because no model is involved. When the subset contains only B, the server evaluates its performance score to 30; when the subset contains only C, the performance score is 40; and when subsets B and C work together, the performance score is 70.

[0093] Then, calculate the marginal contribution of the target model A after adding these subsets. After A is added to the subset containing only B, the performance improves to 80 points, and the marginal contribution is 80-30=50 points; after A is added to the subset containing only C, the performance improves to 90 points, and the marginal contribution is 90-40=50 points; after A is added to the subsets containing both B and C, the performance improves to 100 points, and the marginal contribution is 100-70=30 points.

[0094] Finally, the server performs a weighted average based on the weighted proportions corresponding to different subset sizes. After calculation, the collaborative contribution index of model A is obtained. Through this process, the server fairly quantifies the contribution of each model in the collaborative task.

[0095] In this embodiment of the invention, the compliance index is determined based on the proportion of the output content of the target large language model that conforms to the preset rule constraints during the game interaction process, and differentiated deduction weights are set according to the severity of different violations.

[0096] When a pre-defined serious violation is detected, the compliance indicator will be set to zero or the system will be deemed unqualified.

[0097] In this embodiment of the invention, for example, when the server determines the compliance index of the target large language model, it considers the proportion of the model's output content that conforms to preset rules, combined with the severity of the violation. Taking a debate scenario as an example, the preset rules prohibit the use of personal attacks and false arguments, with serious violations resulting in immediate disqualification and minor violations deducting 10 points. Model D participated in 20 rounds of debate, with its output content fully conforming to the rules in 18 rounds. One round used slightly impolite language and was judged as a minor violation; another round used false arguments, which constituted a serious violation. The server first calculates the basic compliance ratio, i.e., the ratio of the number of rounds without violations to the total number of rounds, 18 ÷ 20 = 0.9. However, due to the serious violation, the server directly sets Model D's compliance index to zero or judges it as unqualified. If there are no serious violations, only minor violations, the server will deduct points from the basic ratio according to the weighted deduction, thereby accurately measuring the model's compliance.

[0098] In this embodiment of the invention, the following implementation methods are also provided.

[0099] When constructing the parallel game evaluation environment, the capability attainment threshold and capability overflow threshold are pre-set as sub-parameters of the game scenario parameters. The capability attainment threshold is the minimum performance standard for determining whether the agent can adapt to the difficulty of the current scenario, and the capability overflow threshold is the performance standard for determining whether the agent's capabilities are not fully utilized in the current scenario.

[0100] In the multi-round interactive game process, after each preset round of games is completed, the comprehensive performance score of the target intelligent agent is obtained by weighted summation based on the win rate index and compliance index.

[0101] If the overall performance score exceeds the capability overflow threshold for a preset number of consecutive times, the game scenario parameters are adjusted. The adjustment includes at least one of increasing the upper limit of the number of interaction rounds, increasing the complexity of rule constraints, and increasing the weight of violation handling.

[0102] If the overall performance score is lower than the capability threshold for a number of consecutive preset times, the game scenario parameters are adjusted. The adjustment includes at least one of simplifying the complexity of the game issues, reducing the upper limit of the number of interaction rounds, and relaxing the constraints of non-core rules.

[0103] The game data after scene adjustment is corrected according to a preset weighting coefficient and then included in the calculation of the multidimensional evaluation result. The difference between the evaluation data before and after scene adjustment is used as a robustness supplementary indicator and participates in the weighted fusion of the comprehensive capability valuation result together with the multidimensional evaluation result.

[0104] In this embodiment of the invention, for example, when constructing a parallel game evaluation environment, the server pre-sets a capability attainment threshold and a capability overflow threshold as sub-parameters of the game scenario parameters. Taking the debate scenario evaluation model E as an example, the capability attainment threshold is set to a comprehensive performance score of 60 points, the capability overflow threshold is set to 80 points, the preset number of rounds is 5, and the preset number of consecutive rounds is 3.

[0105] In the multi-round interactive game, after every 5 games, the server calculates a comprehensive performance score by weighting and summing the win rate and compliance indicators of model E. Assuming the weights are 0.6 and 0.4 respectively, the win rate of model E in the first 5 games is 0.8, and the compliance indicator is 0.9. The comprehensive performance score is 0.8 × 0.6 + 0.9 × 0.4 = 0.84, or 84 points.

[0106] If model E achieves a combined score higher than 80 points for three consecutive times, the server adjusts the game scenario parameters. For example, the maximum number of interaction rounds is increased from 10 to 15; the complexity of rule constraints is increased, and the use of more ambiguous terms is prohibited; the weight of violation handling is increased, with a deduction of 20 points for the first minor violation.

[0107] If model E scores below 60 points in three consecutive comprehensive tests, the server simplifies the complexity of the game topic, reducing the debate topic from "the overall impact of artificial intelligence on social employment is positive" to "whether artificial intelligence will replace some jobs"; reducing the upper limit of the number of interaction rounds from 10 rounds to 7 rounds; and relaxing the constraints of non-core rules, no longer strictly limiting the number of words spoken in each round.

[0108] The server incorporates the adjusted game data, weighted according to a preset coefficient, into the multidimensional evaluation results. For example, if the weighted coefficient for the adjusted data is 0.7, it is proportionally merged with the previous data. Simultaneously, the difference between the evaluation data before and after the adjustment is used as a robustness supplementary indicator. For instance, if the win rate was 0.8 before adjustment and 0.75 after adjustment, the difference is 0.05. This robustness supplementary indicator, along with win rate, influence, collaborative contribution, compliance, and other multidimensional evaluation results, participates in the weighted fusion of the comprehensive capability valuation results, making the evaluation results more comprehensively and accurately reflect the model's capabilities.

[0109] In this embodiment of the invention, the multidimensional evaluation result also includes a strategy evolution capability index, and the calculation process of the strategy evolution capability index can be implemented through the following example.

[0110] In the step of recording interaction content, behavioral decisions, violation events and corresponding game result values, a game strategy evolution trajectory library is constructed simultaneously for each large language model to be evaluated. After the behavioral decisions are refined into decision logic annotations, they are stored together with the interaction content, opponent response text and game result values ​​in the trajectory library.

[0111] Based on the game data of two consecutive rounds in the trajectory library, the semantic similarity and decision logic difference of the output content of the target large language model in two consecutive rounds are calculated by semantic matching algorithm. If the semantic similarity of the output of two consecutive rounds is lower than the first preset threshold, the decision logic difference is higher than the second preset ratio, and the result of the later round is better than the result of the earlier round, it is determined to be an effective strategy adjustment. The ratio of the number of effective strategy adjustments to the total number of decisions of the target large language model is used to obtain the strategy adjustment efficiency index.

[0112] Based on all game output content in the trajectory library, the topic is classified by a topic clustering model, and the strategy diversity index is obtained by counting the proportion of different topic types to the total number of output topics.

[0113] The strategy evolution capability index is obtained by weighted summation of the strategy adjustment efficiency and the strategy diversity index, wherein the strategy adjustment efficiency has a higher weight than the strategy diversity index.

[0114] In the weighted fusion step, the proportion of the strategy evolution capability indicator in the weight of the evaluation indicators is dynamically adjusted according to the game type, so that it, together with the win rate indicator, influence indicator, collaborative contribution indicator, and compliance indicator, generates the comprehensive capability valuation result. If it is detected that the target large language model repeatedly uses the same strategy for a preset number of consecutive games and the game results continue to deteriorate, the comprehensive capability valuation result is reduced by a preset proportion by adjusting the weight of the evaluation indicators.

[0115] In this embodiment of the invention, for example, when evaluating a large language model, the server calculates the strategy evolution capability index from the multi-dimensional evaluation results. Taking the performance of model F in a debate scenario as an example.

[0116] While recording interaction content, behavioral decisions, violations, and corresponding game outcome values, the server simultaneously builds a game strategy evolution trajectory library for model F. For example, in a debate about whether artificial intelligence should be regulated, model F's behavioral decision argues for regulation from the perspective of technological risks. The server refines this decision into a decision logic annotation of "supporting regulation based on technological risks," and stores it in the trajectory library along with the interaction content, opponent's response text, and game outcome values.

[0117] Based on two consecutive rounds of game data from the trajectory database, the server analyzes the data using a semantic matching algorithm. Assume the first round, model F, focuses on technological risks, and the second round focuses on ethical considerations. If the semantic similarity is below a first preset threshold (e.g., 30%), the difference in decision logic is above a second preset proportion (e.g., 60%), and the player wins the second round but loses the first, the server determines this as a valid strategy adjustment. The server calculates the ratio of the number of valid strategy adjustments to the total number of decisions made by model F, obtaining a strategy adjustment efficiency index.

[0118] The server also categorizes the output content of all games in the trajectory database using a topic clustering model. If model F involves different topics such as technological risks, ethics, and social impact in multiple debates, the server counts the proportion of different topic types to the total number of output topics, obtaining a strategy diversity index.

[0119] The server calculates the strategy evolution capability index by weighting and summing the strategy adjustment efficiency and strategy diversity indicators, with a higher weight for strategy adjustment efficiency (e.g., 0.7 for strategy adjustment efficiency and 0.3 for strategy diversity).

[0120] In the weighted fusion step, the server dynamically adjusts the proportion of the strategy evolution capability indicator in the evaluation indicator weights based on the game type. In a debate scenario, if strategy evolution capability is relatively important, its weight can be set to 0.2. If it is detected that model F uses the same strategy for 5 consecutive rounds and the debate results continue to deteriorate, the server will adjust the evaluation indicator weights and deduct a preset percentage from the comprehensive capability valuation result, such as deducting 10%, to more accurately reflect the model's actual capability.

[0121] To more clearly describe the solutions provided in the embodiments of the present invention, a more complete implementation method is provided below. Please refer to the following reference. Figure 2 , Figure 2 A schematic diagram of the overall architecture of the dynamic valuation system for a large language model based on parallel game theory provided in an embodiment of the present invention.

[0122] The overall architecture includes the following modules: environment configuration module, agent management module, game engine module, and evaluation and analysis module. The environment configuration module sets the game type (e.g., debate, collaborative programming, negotiation), task content, maximum number of interaction rounds, rule constraints, and evaluation indicator weights. The agent management module loads the large language model to be evaluated as the game agent and assigns it roles, positions, and initial context information. The game engine module drives multiple agents to conduct multiple rounds of interaction in parallel under unified rules, recording the interaction process in real time and monitoring violations. The evaluation and analysis module calculates multi-dimensional indicators such as win rate, influence, collaborative contribution (Shapley value), and compliance based on interaction data, and generates dynamic capability scores for each model through a weighted comprehensive evaluation function, ultimately outputting a visual evaluation report.

[0123] The aforementioned environment configuration module is used to set and initialize the game scenarios and operating rules required for the evaluation, ensuring that multi-model interactions are conducted under unified, fair, and reproducible conditions. Specifically, the operator can select the game type (including but not limited to: debate scenarios; collaborative programming scenarios; negotiation scenarios) according to the evaluation objectives, and further configure rule constraints such as topic or task content, maximum number of interaction rounds, and prohibited items, while setting weight parameters for evaluation indicators such as win rate, influence, collaborative contribution, and compliance. Through the above structured parameter settings, the environment configuration module provides a standardized operating foundation for multi-round parallel games, fully stimulating the behavioral characteristics of large language models in competition, collaboration, and complex reasoning, and providing high-quality and diverse interactive data support for subsequent dynamic capability evaluation.

[0124] The aforementioned agent management module refers to loading the large language model to be evaluated as a game-theoretic agent and assigning it pre-defined roles and initial context information. Specifically, based on the selected evaluation scenario, corresponding task context is injected into each model. For example, in a debate scenario, its side (affirmative or negative) and the debate topic are specified; in a collaborative programming task, a problem description, expected output, and initial division of labor are provided; in a negotiation scenario, its role (e.g., buyer or seller), initial goals, and constraints are set. The agent management module supports the parallel operation of multiple model instances, enabling multiple independent games to be played simultaneously, significantly improving the throughput and efficiency of the evaluation.

[0125] The aforementioned game engine module refers to the activation of a parallel game engine that drives multiple model agents to conduct multi-round structured interactions based on preset scenario logic. This engine is responsible for scheduling each agent to execute dialogue generation or decision-making actions sequentially or in parallel, and embedding corresponding adjudication mechanisms or task completion logic according to specific scenario types. For example, in a debate scenario, it controls agents to speak in turn, and automatically determines the winner by calling a preset adjudication algorithm after the interaction ends; in a collaborative programming scenario, it coordinates agents to share context, collaboratively generate and debug code; in a negotiation scenario, it simulates multiple rounds of bidding and counter-bidding processes, tracking the agreement's status. Throughout the interaction process, the engine monitors in real time whether the output content of each agent violates preset rules (such as containing prohibited words, unauthorized operations, or ethical violations). Once a violation is detected, the event is immediately recorded and corresponding measures are taken, including but not limited to issuing warnings, deducting compliance scores, or forcibly terminating the current game, to ensure the standardization and security of the evaluation process.

[0126] The aforementioned evaluation and analysis module refers to the systematic processing of the collected interaction data after each game, calculating multiple dynamic evaluation indicators, including: win rate based on a statistical model of win-loss records, quantifying the model's influence on the game outcome, assessing its collaborative contribution in cooperative tasks, and detecting whether its behavior complies with preset rules and ethical norms. Specifically, the model... win rate Defined according to formula (1); its influence score Using the "leave one" method for calculation, firstly based on the... In-game model The impact value of performance differences caused by participation or non-participation is defined. (See formula (2)), and then average the results of multiple games using formula (3) to obtain ( The collaborative contribution is fairly allocated based on the Shapley value method in cooperative game theory, and its mathematical expression is shown in formula (4); compliance score The result is calculated according to formula (5), reflecting the degree to which the model adheres to the rules throughout the interaction process. The above multi-dimensional indicators together constitute the comprehensive evaluation basis of the model's capabilities, providing a quantitative basis for subsequent weighted fusion and capability value generation.

[0127] Capability score calculation: The above-mentioned multi-dimensional evaluation indicators, such as win rate, influence, collaborative contribution, and compliance, are weighted and integrated according to preset weight coefficients to generate a comprehensive capability evaluation score for each model. The specific form of the comprehensive valuation function is shown in formula (6), where the weighting coefficients are... The system can be flexibly configured according to the needs of actual application scenarios to highlight the importance of different capability dimensions. Finally, the system outputs dynamic capability scores for each participating model and automatically generates a structured evaluation report to support performance comparison, merit-disadvantage analysis, and iterative optimization decisions among models.

[0128] The aforementioned parallel game refers to an interactive adversarial or collaborative experiment in which multiple intelligent agents (models) participate simultaneously. In this invention, it refers to an evaluation method in which multiple models engage in multiple rounds of interaction (competition or cooperation) in a preset scenario to simulate real-world application situations.

[0129] The win rate mentioned above refers to the frequency or proportion of a model's wins in adversarial games, and can be specifically defined as the model's win rate. The ratio of the number of games won to the total number of games played is used to measure the model's ability to win directly in competitive scenarios.

[0130]

[0131] The aforementioned increase in influence refers to any number of... Game theory, model The impact on team benefits or task outcomes is defined as the difference in system performance between their participation and non-participation, specifically expressed as:

[0132]

[0133] in, Indicates the first The set consisting of all participating models in the game. Model subset The gains or overall score achieved in the bureau's mission. This indicator Model In the Marginal contribution in a game reflects the degree of its critical impact on the overall outcome.

[0134] The aforementioned influence score refers to the model The influence score is calculated by averaging the increase in influence over multiple games, and is expressed as follows:

[0135]

[0136] The aforementioned collaborative contribution (Shapley score) refers to the value used to evaluate the model's contribution to a collaborative task. The Shapley score method from cooperative game theory is used to fairly allocate the model's contribution. The Shapley score calculates the average marginal contribution of a model by considering all possible model combinations, reflecting the model's contribution to the overall team benefit.

[0137]

[0138] in, This represents the set of all models participating in the current evaluation. For the total number of models, For models not included any subset, Indicates a subset of the model The reward or overall score earned by collaboratively completing a designated task. This is the Shapley value. Reflects the model The average marginal contribution to the team's total revenue is fair and unique, and is suitable for contribution allocation in multi-agent collaborative scenarios.

[0139] The aforementioned compliance refers to the degree to which a model adheres to predetermined rules and ethical norms during game-theoretic interactions. Scoring is achieved by detecting whether the model's dialogue or behavior contains inappropriate content or strategies, ensuring that the model's capability assessment considers safety and reliability factors simultaneously.

[0140]

[0141] Based on actual assessment needs, different deduction weights can be set for different types of violations; for behaviors that seriously violate safety or ethical guidelines, the system can directly set the compliance score to zero or determine that the model is unqualified, thereby ensuring that the model always meets safety and compliance requirements while pursuing performance indicators such as win rate and influence.

[0142] The aforementioned comprehensive capability evaluation function refers to the evaluation and analysis module weighting and integrating the above multi-dimensional indicators according to preset weights to generate the final comprehensive capability evaluation value of the model, and its expression is as follows:

[0143]

[0144] Compared with existing technologies, this invention addresses the problems of current large language model capability assessment, such as limited testing scenarios, static indicators, lack of interactivity, and difficulty in reflecting real-world application performance. It provides a dynamic evaluation method and system based on parallel game theory. This system effectively captures strategic interactions and emergent intelligent behaviors that static evaluations cannot reveal by constructing a parallel game environment of multi-model competition and collaboration. It combines multi-dimensional dynamic indicators such as win rate, influence, collaborative contribution, and compliance to achieve a comprehensive and refined quantification of model capabilities. The introduction of a relative evaluation mechanism based on actual interaction results between models significantly improves the fairness and discriminative power of the assessment. Furthermore, the system has a clear modular architecture, flexible and configurable parameters, supports multi-scenario expansion and batch automated testing, and the evaluation process is efficient, stable, and reproducible, providing reliable and practical technical support for the selection, iterative optimization, and security governance of large language models.

[0145] Specifically, in view of the above content, the embodiments of the present invention also provide the following more specific real-time processes, please refer to them. Figure 3 and Figure 4 , Figure 3 This is a schematic diagram of the process framework for the dynamic evaluation method of a large language model based on parallel game theory provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of a parallel game evaluation scenario provided in an embodiment of the present invention.

[0146] The large language model dynamic evaluation system based on parallel game theory provided in this invention consists of multiple participating model agents and an evaluation server (or central control unit), with each module deployed within a unified evaluation framework. Specifically, the system includes core units such as an environment configuration module, an agent management module, a game engine module, and an evaluation and analysis module. These modules work collaboratively to form an automated evaluation framework integrating scenario setting, model interaction, process monitoring, and result evaluation. The environment configuration module initializes the evaluation scenario parameters; the agent management module loads the large language model to be evaluated and assigns it roles and context information; the game engine module drives the pre-defined game logic to execute interactions between models and monitors rule compliance during the process; and the evaluation and analysis module performs calculations and analysis on the collected data after the game ends, outputting the model capability evaluation results. The entire system can support parallel evaluation of multiple models in the same environment, with all data and calculations completed within a controlled environment during the evaluation process.

[0147] In the actual evaluation process, the environment configuration module first sets the specific evaluation scenario and parameters according to requirements. Operators can select the game type and configure relevant options through the human-computer interaction interface. For example, in a debate scenario, they can input the topic, set the positions of both sides, and the maximum number of debate rounds; in a collaborative programming scenario, they can provide a description of the programming task to be completed, and optionally specify the sub-task division of different models; in a negotiation scenario, they can set the initial conditions for both buyers and sellers (such as the buyer's psychological price and the seller's minimum offer) and the maximum number of negotiation rounds. Simultaneously, the environment configuration module also allows setting evaluation rules, including the definition and penalties for violations (such as prohibiting the use of sensitive words, resulting in point deductions or termination), and adjustments to the weights of evaluation indicators. After configuration, the system enters the evaluation preparation phase.

[0148] Next, the agent management module loads each large language model to be evaluated, packages it into an independent agent entity, and assigns it the necessary initial context information. Specifically, in a debate scenario, the agent management module provides each model with the debate topic and the corresponding affirmative or negative roles through prompts; in a collaborative programming scenario, it provides each model with a problem description and can suggest its areas of expertise; in a negotiation scenario, it sets parameters such as initial offer and target price for the buyer and seller models respectively. The agent management module ensures that each model agent is in the correct initial state and role setting at the start of the evaluation. Furthermore, when multiple games are required, the module can duplicate model instances or schedule idle models in parallel across multiple threads / processes to participate in different games, thereby improving the test throughput of the evaluation. Once all model agents are ready, the system enters the game interaction phase.

[0149] The game interaction is uniformly scheduled by the game engine module. Taking a debate scenario as an example, the game engine allows the two opposing models to take turns expressing their views and refuting each other according to a predetermined round order. After each round of speaking, the engine records the dialogue content and checks whether the termination conditions have been met (e.g., reaching the maximum number of rounds or neither side has new content). When the debate ends, the engine calls the built-in adjudication mechanism to evaluate the performance of both sides, thereby determining the winner. This adjudication mechanism can be a pre-trained discriminative model or a rule-based scoring algorithm acting as a "judge," scoring and comparing the persuasiveness and sufficiency of arguments to determine which side wins. For collaborative programming scenarios, the game engine allows models to freely communicate and collaborate: models can propose solutions and write code snippets, while other models can check and improve the code. The engine tracks the entire coding process, ending the game when all models have jointly completed the task or reached the maximum number of iterations. During this period, the engine will calculate the contribution of each model to the final solution (e.g., recording the number of lines of code written by each model, the number of critical errors found and fixed, etc.), providing a basis for subsequent calculation of collaborative contribution metrics. In the negotiation scenario, the game engine simulates a multi-round bidding and counter-offering process between the buyer and seller: both models take turns proposing offers or modifying conditions until one party accepts the other's conditions and an agreement is reached, or the negotiation breaks down if no agreement is reached after all rounds have been exhausted. The engine evaluates the negotiation performance of both models based on the negotiation results (such as whether an agreement is reached, and the gap between the final price and the ideal expectations of both parties).

[0150] Throughout all game sessions, the game engine module and the evaluation and analysis module work in tandem to collect the necessary process data in real time. For example, the dialogue content, decision options, and outcome determination for each round are recorded and stored. The engine's built-in compliance monitoring unit scans the model output in real time, marking prohibited words, non-compliant strategies, etc. Once a violation is detected, the system handles it according to preset strategies: minor violations result in warnings or point deductions in the final score, while serious violations can immediately terminate the current game. All this data and events are notified to the evaluation and analysis module for recording and use in the outcome analysis phase. This design ensures that all dimensions of data required for evaluation are fully captured during the interaction process.

[0151] After completing the predetermined rounds of game interaction, the system enters the results analysis and evaluation phase. The evaluation and analysis module summarizes and processes the recorded data, calculating the capability indicators for each model:

[0152] Win rate calculation: In adversarial scenarios (such as debates and competitive games), the win rate... Defined as a model The percentage of wins out of the total number of games played. When the game involves multiple participants and a tie is possible, the win rate can be adjusted based on the final ranking or the points can be evenly distributed according to the rules. The win rate, as the most intuitive indicator of relative strength, directly reflects the model's performance in a competitive environment.

[0153] Impact Analysis: The evaluation and analysis module identifies key impact points for each model in each round by reviewing the game interaction logs. For example, in a debate scenario, it can pinpoint whether a particular side's statement significantly shifts the judges' scores; in a collaborative task, it can assess how much the team's task success rate or overall performance would decrease if a certain model were removed. From a mathematical perspective, the "leave-one-out" method is used to quantify the models. Impact: Its marginal effect is measured by comparing the difference in team performance with and without the model. Let the team payoff function be... The set of all participating models is Then the model The influence value in a single game is defined as The evaluation and analysis module averages the impact values ​​in multi-game scenarios to obtain the model. Overall Influence Score Higher The results indicate that the model plays a crucial role in the interaction process, and its behavior or decisions have a significant influence on the final outcome.

[0154] Collaboration Contribution Calculation: In collaborative tasks, the evaluation and analysis module calculates the collaboration contribution based on the marginal contribution of each model to team performance using the Shapley value method. The specific formula is as described above (see formula (4)). In actual implementation, the module approximates the introduced model by analyzing the intermediate results generated by different model subset combinations during task execution. The incremental impact on team performance. For scenarios with high complexity or large-scale model sets, approximation algorithms such as Monte Carlo sampling can be used to efficiently estimate the Shapley value, thereby reasonably distributing the team's total revenue to each participating model while ensuring fairness. A higher collaborative contribution indicates that the model plays a more crucial role in the collaborative task completion process and contributes more significantly to the overall outcome.

[0155] Compliance scoring: The evaluation module calculates compliance metrics for each model based on violation logs recorded by the game engine. The basic method involves statistically analyzing the proportion of output content that conforms to preset standards across all interaction rounds, and then applying differentiated deductions based on the nature and severity of the violation. For example, if the model... If a model makes only one slightly inappropriate response in 10 interactions (recorded as a minor violation) and there are no serious violations, its compliance score can be set to 90 points (out of 100) or 0.9. If a major violation occurs (such as generating illegal, harmful, or seriously unethical content), the system can directly determine that the model's evaluation is unsatisfactory or impose a significant penalty on its overall ability score. The compliance scoring mechanism aims to ensure that while pursuing performance goals such as win rate and influence, the model always adheres to safety and ethical bottom lines, avoiding the behavior of "trading violations for performance," thereby achieving responsible artificial intelligence evaluation.

[0156] After obtaining various evaluation indicators such as win rate, influence, collaborative contribution, and compliance, the evaluation and analysis module calculates the comprehensive capability estimate of each model based on preset weighting coefficients. Weight vector It can be flexibly configured according to the focus of specific application scenarios: for example, in security-sensitive scenarios (such as finance, government affairs or medical fields), the compliance weight can be significantly increased. This strengthens the requirements for ethical and rule compliance; while in tasks that emphasize collaborative efficiency (such as multi-agent programming or joint decision-making), the weight of collaborative contribution can be appropriately increased. To highlight the value of teamwork.

[0157] After weighted fusion, the system generates a unified comprehensive capability score for each participating model. This result can be further presented visually, including but not limited to: clearly listing the specific values ​​of each model on each indicator and the total score in tabular form, or using radar charts to intuitively display the capability spectrum and strengths of different models. Such charts and structured reports help users quickly identify model strengths and weaknesses, understand capability distribution, and provide data-driven decision support for model selection, optimization, or deployment.

[0158] To verify the effectiveness of the method of this invention, multiple mainstream large-scale language models can be tested under the same evaluation scenario. Experiments show that, compared to traditional single-index rankings, the parallel game-theoretic evaluation of this invention can reveal the inverse relationship between the performance of different dimensions of the models: for example, a model with a high win rate but poor compliance will be reflected in the overall score; another model may perform poorly in a single task but excel in collaborative scenarios, scoring high in influence and contribution, thus not falling behind in the overall evaluation. This demonstrates that the multi-dimensional dynamic evaluation provided by this invention more comprehensively and fairly reflects the model's capability spectrum. Furthermore, this evaluation framework supports continuous iterative evaluation: as model versions are updated or new models are added, they can be re-evaluated at any time through the same game-theoretic process, and the relative capability values ​​of each model will be dynamically updated, providing a reliable basis for model optimization and improvement and the selection of winners.

[0159] In summary, this invention innovatively introduces parallel game interaction to achieve dynamic, relative, and multi-dimensional capability assessment of large language models. The method and system possess high innovation and feasibility, and can be widely applied to the evaluation stages of large model research, selection, and deployment, helping to discover more powerful models and ensuring their safe and reliable application.

[0160] Please refer to the following: Figure 5 , Figure 5 A dynamic estimation system 110 for a large language model based on parallel game theory, provided in an embodiment of the present invention, includes:

[0161] The construction module 110 is used to construct a parallel game evaluation environment, set game scenario parameters for large language models to be evaluated, and the scenario parameters include at least game type, upper limit of interaction rounds, rule constraints, violation handling strategy and evaluation index weight; load at least two large language models to be evaluated as agents, and assign corresponding role information and initial context to each agent so that they meet the interaction conditions of the game scenario;

[0162] The evaluation module 1102 is used to drive the agents to engage in multiple rounds of interactive games according to preset rules in the parallel game evaluation environment. During the game, it monitors the violations of each agent in real time and records the interaction content, behavioral decisions, violation events, and corresponding game result values, wherein the game result values ​​are the task completion degree, win / loss determination, or gain score for each round. Based on the game process data and the recorded game result values, it calculates multi-dimensional evaluation results for each agent, including at least a win rate indicator, an influence indicator, a collaborative contribution indicator, and a compliance indicator. According to the weights of the evaluation indicators, it performs weighted fusion of the multi-dimensional evaluation results to generate a comprehensive capability valuation result for each language model.

[0163] It should be noted that the implementation principle of the aforementioned large language model dynamic evaluation system 110 based on parallel game theory can refer to the implementation principle of the aforementioned large language model dynamic evaluation method based on parallel game theory, and will not be repeated here. It should be understood that the division of the various modules in the above device is merely a logical functional division; in actual implementation, they can be fully or partially integrated into a single physical entity, or physically separated. Furthermore, these modules can all be implemented in software through processing element calls; they can all be implemented in hardware; or some modules can be implemented by processing element calls to software, and some modules can be implemented in hardware. For example, the large language model dynamic evaluation system 110 based on parallel game theory can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and called and executed by a processing element of the above device. The implementation of other modules is similar. Furthermore, these modules can be fully or partially integrated together, or implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step or module of the above method can be completed by the integrated logic circuit in the hardware of the processor element or by instructions in the form of software.

[0164] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to implement a system-on-a-chip (SOC).

[0165] This invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned dynamic estimation system 110 for a large language model based on parallel game theory. Figure 6 As shown, Figure 6This is a structural block diagram of a computer device 100 provided in an embodiment of the present invention. The computer device 100 includes a large language model dynamic evaluation system 110 based on parallel game theory, a memory 111, a processor 112, and a communication unit 113.

[0166] To enable data transmission or interaction, the memory 111, processor 112, and communication unit 113 are electrically connected to each other directly or indirectly. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The large language model dynamic evaluation system 110 based on parallel game theory includes at least one software function module that can be stored in the memory 111 or embedded in the operating system (OS) of the computer device 100 in the form of software or firmware. The processor 112 is used to execute the large language model dynamic evaluation system 110 based on parallel game theory stored in the memory 111, such as the software function modules and computer programs included in the large language model dynamic evaluation system 110 based on parallel game theory.

[0167] This invention provides a readable storage medium, which includes a computer program. When the computer program runs, it controls the computer device where the readable storage medium is located to execute the aforementioned dynamic estimation system 110 for a large language model based on parallel game theory.

[0168] For illustrative purposes, the foregoing description has been made with reference to specific embodiments. However, the foregoing illustrative discussions are not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Numerous modifications and variations are possible in accordance with the foregoing teachings. These embodiments were chosen and described in order to best illustrate the principles of the present disclosure and its practical application, thereby enabling those skilled in the art to best utilize the disclosure and to employ various embodiments with different modifications to suit a particular intended application.

Claims

1. A large language model dynamic valuation method based on parallel game, characterized in that, The method comprises the following steps: Construct a parallel game evaluation environment, set the game scenario parameters of the large language model to be evaluated, which at least include game type, upper limit of interaction rounds, rule constraints, violation handling strategy and evaluation index weight; Load at least two large language models to be evaluated as agents, and assign corresponding role information and initial context to each agent to meet the interaction conditions of the game scenario; In the parallel game evaluation environment, drive the agents to conduct multi-round interactive game according to the preset rules, monitor the violation behavior of each agent in real time during the game process, and record the interaction content, behavior decision, violation event and corresponding game result value, wherein the game result value is the task completion degree, win or lose judgment or income score of each round of game; Based on the game process data and the recorded game result value, the multi-dimensional evaluation results of each agent are calculated, including win rate index, influence index, cooperation contribution index and compliance index; According to the evaluation index weight, the multi-dimensional evaluation results are weighted and fused to generate the comprehensive capability evaluation result corresponding to each large language model.

2. The method of claim 1, wherein, The game type at least includes one or more of the following: adversarial game scenario, cooperative game scenario and negotiation game scenario, wherein the adversarial game scenario includes debate scenario, the debate scenario is evaluated by a preset decision algorithm to determine the win or lose of the game, the cooperative game scenario includes cooperative programming scenario, and the negotiation game scenario includes negotiation scenario.

3. The method of claim 1, wherein, The multi-round interactive game is executed in a parallel manner, and multiple game rounds are run simultaneously in the same evaluation environment to improve the evaluation efficiency of large language model evaluation.

4. The method of claim 1, wherein, The win rate index is calculated by the following method: Statistically count the number of games won by the target large language model in multiple game rounds in which it participates, and take the ratio of the number of games won to the total number of game rounds in which it participates as the win rate index; When the game result is a draw or there are multiple participants, the win or lose result is calculated or divided according to the preset rules to correct the calculation result of the win rate index.

5. The method of claim 1, wherein, The influence index is determined by the following method, including: Under the same game round condition, the game result value when the target large language model is included and the game result value when the target large language model is not included are calculated respectively, and the difference between the two is taken as the influence increment of the target large language model in the game round; Statistically average the influence increments obtained in multiple game rounds, and take the statistical average result as the influence index of the target large language model.

6. The method of claim 1, wherein, In the cooperative game scenario, the cooperation contribution index is determined based on the Shapley value calculation rule in cooperative game theory, including: By traversing or sampling different model subsets not containing the target large language model, the game result change before and after the target large language model joins the corresponding subset is calculated, and the weighted average is performed combined with the weighted proportion corresponding to the size of the model subset to obtain the cooperation contribution index of the target large language model.

7. The method of claim 1, wherein, The compliance index is determined according to a proportion of content output by the target large language model in the process of game interaction conforming to preset rule constraints, and different violation behaviors correspond to different severity degrees, and a differentiated deduction weight is set; When a preset serious violation behavior is detected, the compliance index is set to zero or judged as unqualified.

8. The method of claim 1, wherein, The method further comprises: When the parallel game evaluation environment is constructed, the capability threshold and the capability overflow threshold are preset as sub-parameters of the game scene parameters, the capability threshold is a minimum performance standard for judging whether an agent is adapted to the difficulty of a current scene, and the capability overflow threshold is a performance standard for judging whether the capability of the agent is not fully exerted in the current scene; In the process of the multi-round interactive game, after each preset round of game is completed, a comprehensive performance score of the target agent is obtained based on a weighted sum of the win rate index and the compliance index; If the comprehensive performance score is higher than the capability overflow threshold for a continuous preset number of times, the game scene parameters are adjusted, and the adjustment includes at least one of increasing the upper limit of the number of interactions, increasing the complexity of rule constraints, and increasing the violation handling weight; If the comprehensive performance score is lower than the capability threshold for a continuous preset number of times, the game scene parameters are adjusted, and the adjustment includes at least one of simplifying the complexity of game issues, reducing the upper limit of the number of interactions, and relaxing non-core rule constraints; The game data after the scene adjustment is corrected according to a preset weighting coefficient and is included in the multi-dimensional evaluation result calculation, and the difference between the evaluation data before and after the scene adjustment is used as a robustness supplementary index, which is jointly used in the weighted fusion of the comprehensive capability evaluation result together with the multi-dimensional evaluation result.

9. The method of claim 1, wherein, The multi-dimensional evaluation result further includes a strategy evolution capability index, and the calculation process of the strategy evolution capability index comprises: In the step of recording the interactive content, behavior decision, violation event and corresponding game result value, a game strategy evolution track library is constructed for each large language model to be evaluated, and the behavior decision is refined into a decision logic label and stored in the track library together with the interactive content, opponent response text and game result value; Based on the track library, the semantic similarity and decision logic difference of the output content of the target large language model in two consecutive rounds are calculated by a semantic matching algorithm, if the semantic similarity of the output content in two consecutive rounds is lower than a first preset threshold, the decision logic difference is higher than a second preset proportion, and the result of the latter round is better than that of the former round, it is determined that an effective strategy adjustment is made, and the strategy adjustment efficiency index is obtained by calculating the ratio of the number of effective strategy adjustments to the total number of decisions of the target large language model; Based on all game output content in the track library, a theme clustering model is used for theme classification, and the strategy diversity index is obtained by calculating the proportion of the number of different theme types in the total number of output themes; The strategy adjustment efficiency and the strategy diversity index are weighted and summed to obtain the strategy evolution capability index, and the weight of the strategy adjustment efficiency is higher than that of the strategy diversity index. In the weighting fusion step, the proportion of the strategy evolution capability index in the evaluation index weight is dynamically adjusted according to the game type, so that it generates the comprehensive capability evaluation result together with the win rate index, the influence index, the cooperation contribution index and the compliance index. If it is detected that the target large language model repeatedly uses the same strategy for a preset number of games and the game result continues to deteriorate, the comprehensive capability evaluation result is reduced by a preset proportion by adjusting the evaluation index weight.

10. A large language model dynamic valuation system based on parallel game, characterized in that, Comprise: The construction module is used for constructing a parallel game evaluation environment, setting game scene parameters of large language models for evaluation, and the scene parameters at least include game type, upper limit of interaction rounds, rule constraint condition, violation handling strategy and evaluation index weight; at least two large language models to be evaluated are loaded as agents, and each agent is assigned corresponding role information and initial context to meet the interaction condition of the game scene; The evaluation module is used for driving the agents to perform multi-round interactive game according to the preset rule in the parallel game evaluation environment, monitoring the violation behavior of each agent in real time during the game process, and recording the interaction content, behavior decision, violation event and corresponding game result value, wherein the game result value is the task completion degree, win or lose judgment or income score of each round of game; based on the game process data and the recorded game result value, the multi-dimensional evaluation result of each agent is calculated, which at least includes win rate index, influence index, cooperation contribution index and compliance index; according to the evaluation index weight, the multi-dimensional evaluation result is weighted and fused to generate the comprehensive capability evaluation result corresponding to each large language model.

Images