Incomplete information game method based on language model and game tree CFR algorithm

Abstract

The invention discloses an incomplete information game method based on a language model and a game tree CFR algorithm. The method comprises the following steps: constructing a game match data set; selecting a loss function and a hyper-parameter, and training a large language model based on the game match data set; obtaining a current game playing state; setting a large language model; creating an initial game tree; expanding a game tree through Beam Search based on a large language model; dividing information sets through prior knowledge of a large language model and merging the information sets; calculating a leaf node value by using a large language model scoring mechanism; and calculating an equalization strategy until convergence. According to the method, powerful priori knowledge and emergence capability of a large language model are utilized, and the problem that node expansion and merging are difficult to perform in a complex system by a traditional CFR algorithm can be solved under the condition that extra system complexity is as small as possible.

Description

Technical Field

[0001] The invention belongs to the field of game technology, in particular to an incomplete information game method based on a language model and a game tree CFR algorithm. Background Art

[0002] In game theory, incomplete information games refer to games in which it is difficult for participants to obtain complete information when making decisions. This type of game is widely present in many scenarios in real life, such as poker games, negotiations, and market transactions. In these games, participants must formulate strategies based on limited information and need to consider different actions that their opponents may take. The solution to incomplete information games is to calculate its Nash equilibrium state. Nash equilibrium was proposed by John Nash and represents a strategy combination in game theory in which no single participant can obtain a better result by unilaterally changing his or her strategy. In other words, in the Nash equilibrium state, each participant's strategy is the best response to the strategies of other players. The CFR algorithm is an effective algorithm for solving Nash equilibrium in incomplete information games. The CFR algorithm gradually approaches Nash equilibrium by continuously iteratively updating the strategy to minimize the virtual regret value (that is, if the participant had adopted a different strategy in the past game, he or she might have obtained a better result). The core idea of ​​​​CFR is to re-evaluate past choices in each iteration and adjust the strategy based on new information. Although the CFR algorithm is very successful in theory and practice, it still has some limitations:

[0003] 1. Computational complexity: For very large game trees or highly branched games, the CFR algorithm requires a lot of computing resources to converge to an approximate Nash equilibrium.

[0004] 2. Memory consumption: In order to store the strategy and regret value of each information set, the CFR algorithm needs to occupy a large amount of memory space, which is a challenge for large-scale games.

[0005] 3. Convergence speed: Although CFR will eventually converge, it may converge slowly in some cases, especially when the game tree is very deep or the game state space is very large. Summary of the invention

[0006] The purpose of the present invention is to provide an incomplete information game method based on a language model and a game tree CFR algorithm in order to solve the problems existing in the above-mentioned prior art.

[0007] The technical solution to achieve the purpose of the present invention is: an incomplete information game method based on a language model and a game tree CFR algorithm, the method comprising:

[0008] Step 1, construct a game data set;

[0009] Step 2, selecting a loss function and hyperparameters, and training a large language model based on the game data set;

[0010] Step 3, obtain the current game status;

[0011] Step 4: Set up a large language model;

[0012] Step 5, create an initial game tree;

[0013] Step 6, expand the game tree through Beam Search based on the large language model;

[0014] Step 7, divide the information set by the prior knowledge of the large language model and merge the information sets;

[0015] Step 8: Use the large language model scoring mechanism to calculate the leaf node value;

[0016] Step 9: Calculate the equilibrium strategy until convergence.

[0017] Furthermore, step 1 of constructing the game data set specifically includes:

[0018] Step 1-1, classifying the game data according to preset standards;

[0019] Step 1-2, collecting various types of game data from public databases, and performing data cleaning and data sorting to form a game data set; the data cleaning includes removing duplicate items, irrelevant information or erroneous records; the data sorting is to sort the data into a unified format;

[0020] Steps 1-3, for the data in the dataset, add necessary contextual information for each decision step, and annotate the decision results of each step to form the final game dataset.

[0021] Furthermore, the step 3 of obtaining the current game status specifically includes:

[0022] Read the current information set as I, the current state in the game tree is recorded as s, and the current player is p now , the player p(s) or p(I) currently in state s or information set I, the current opponent op, and the information info(s,p) and info(I,p) that the player p can observe in state s and information set I.

[0023] Furthermore, the step 4 of setting up a large language model specifically includes:

[0024] Initialize the large language model: load the large language model pre-trained in step 2 to generate actions and evaluate state values;

[0025] Define the large language model interface: Define the interface between the large language model and the environment to enable the model to receive state descriptions, perform actions, and obtain feedback;

[0026] Initialize memory: Initialize a memory structure to store past game states, actions, and corresponding value estimates.

[0027] Furthermore, the step 5 of creating an initial game tree includes:

[0028] Step 5-1, creating the root node of the initial game tree according to the structured construction method of the game tree node;

[0029] Step 5-2, infer the opponent's hidden information: Based on the current game state and known information, the pre-trained large language model is used to infer the opponent's possible hidden information, and the opponent's possible information or state and its probability are obtained, which are expressed as (state, probability) tuples: (s1, p1), (s2, p2), (s3, p3), ...; where s i represents the ith possible state of the opponent, p i Indicates i The corresponding probability is i=1, 2, ...;

[0030] Step 5-3, infer your own information: Based on the possible information or state and its probability obtained in step 5-2, infer which of your own information will be considered important or possible by the opponent through the pre-trained large language model, and obtain your own possible information or state and its probability, expressed as a (state, probability) tuple: (s 11 , p 11 ), (s 12 , p 12 ), ...; where s ij It means that when the opponent has the i-th possible state, the opponent sees the j-th possible state of the opponent, p ij Indicates ij The corresponding probabilities are i, j = 1, 2, ...;

[0031] Step 5-4, create child nodes of the initial game tree: For all tuples obtained in steps 5-2 and 5-3, create a node for each tuple as a child node of the initial game tree, and create an edge and its corresponding transition probability for each child node.

[0032] Furthermore, the step 6 uses the Beam Search based on the large language model to expand the game tree, specifically including:

[0033] Step 6-1, for each layer of node set in the game tree, select the best k nodes through the pre-trained large language model;

[0034] Step 6-2, based on k nodes, expand the game tree through Beam Search, expressed as: T new =BeamSearch(T current , k), where T current is the current game tree, T new is the expanded game tree, and BeamSearch indicates the BeamSearch expansion;

[0035] In step 6-3, a large language model is used to evaluate each node and predict the next possible action or state and its probability.

[0036] Furthermore, the step 7 divides the information set and merges the information set by using the prior knowledge of the large language model, specifically including:

[0037] Step 7-1, using the pre-trained large language model to divide the information set according to its prior knowledge, and setting a hidden label for each game state in the information set to indicate its category;

[0038] Step 7-2, by clustering based on the state cosine similarity of the hidden layer representation of the large language model after inputting different game states to merge the game states, specifically:

[0039] The similarity between different inputs is calculated using the cosine angular distance represented by the hidden layer:

[0040]

[0041] in, represents the cosine similarity of two different input states s1 and s2 represented in the i-th hidden layer, Respectively represent the hidden layer representation of the i-th layer corresponding to s1 and s2;

[0042] If the similarity is greater than a preset threshold, the input information sets corresponding to the two inputs are merged.

[0043] Furthermore, step 9 specifically includes:

[0044] Step 9-1, Initialization

[0045] Initialize a strategy vector for each information set in the game tree, and initialize a regret vector for each information set;

[0046] Step 9-2, traverse the game tree

[0047] Traverse the game tree starting from the root node and calculate the regret value of each node;

[0048] Step 9-3, Update strategy

[0049] Update the strategy of the game tree according to the calculated regret value;

[0050] Step 9-4, convergence test

[0051] Repeat steps 9-2 and 9-3. After multiple regret value calculations and strategy updates, check whether the strategy has converged. If the regret value is lower than a preset threshold or the current predetermined number of iterations has been reached, stop the iteration.

[0052] Furthermore, calculating the regret value in step 9-2 specifically includes:

[0053] When a certain strategy σ is adopted, the action sequence h is obtained according to the strategy σ, and the virtual value v brought by h i (σ, h) is calculated as:

[0054]

[0055] Among them, π σ (h, z) represents the probability of reaching the final state z through action sequence h under the σ strategy, represents the probability of occurrence of action sequence h, u i (z) represents the value of player i’s final state z;

[0056] At this point, under a given strategy σ, the regret value of player i taking action a is calculated as follows:

[0057] r(h,a)=v i (σ I→a , h)-v i (σ, h)

[0058] Among them, r(h, a) represents the regret value of player i taking action a, and v i (σ I→a ,h) represents the virtual value of taking action a directly in the information set, position,v i (σ, h) represents the virtual value of not taking action a directly but taking action according to strategy σ normally;

[0059] The information set corresponding to the action sequence h has a regret value of r(I, a) = ∑ h∈I r(h, a), the regret value of player i taking action a in round T is

[0060]

[0061] In the formula, Represents the information set and regret value corresponding to player i taking action a in round t.

[0062] Furthermore, the formula for updating the strategy in step 9-3 is:

[0063]

[0064] In the formula, represents the probability that strategy σ directly takes action a at position I in information set after the T+1th policy iteration, represents the regret value of player i taking action a in round T, and A(I) represents the set of actions that can be taken under information set I.

[0065] Compared with the prior art, the present invention has the following significant advantages:

[0066] 1) Through the carefully designed Prompt and TreeNode data structures, the present invention enables a large language model to more effectively understand and memorize the game tree structure, thereby improving its analytical ability.

[0067] 2) The present invention utilizes the prior knowledge of the large language model to quickly and effectively merge the game tree states that are complex and difficult to automatically merge through fixed rules based on the hidden state, thereby simplifying the state space.

[0068] 3) The present invention utilizes the prior knowledge of the large language model to provide efficient value calculation for each node, thereby providing a fast and accurate initialization process for the CFR (Counterfactual Regret Minimization) algorithm and accelerating the convergence process.

[0069] 4) By combining the large language model with the CFR algorithm, the present invention can achieve fast and stable Nash equilibrium calculation in incomplete information games, thereby improving the overall performance and practicality of the algorithm.

[0070] The present invention is further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 The present invention is a flow chart of the incomplete information game method based on the language model and the game tree CFR algorithm.

[0072] Figure 2 Schematic diagram of information set merging in one embodiment.

[0073] Figure 3 A schematic diagram of leaf node calculation in one embodiment.

[0074] Figure 4 It is a schematic diagram of combination and probability prediction taking a certain game as an example in an embodiment. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0076] It should be noted that if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0077] Currently, traditional CFR algorithms face huge challenges in incomplete information games (such as Texas Hold'em). In these games, the incompleteness of information leads to extremely large search trees. Players need to make reasonable assumptions about the possible distribution of other opponents' hands. In this case, relying solely on the CFR algorithm often requires a large number of iterations to converge to the Nash equilibrium, which is often not feasible in practical applications. For example, in actual games, players may encounter some unconventional opponent behaviors, such as a player who always chooses to call instead of raising or folding. Such behavior patterns do exist in actual games, but for traditional CFR algorithms, it may take a long time to adapt and find the right response strategy.

[0078] Therefore, if we can leverage the powerful prior knowledge and emergence capabilities of the language model to quickly learn and adapt to complex opponent behaviors with limited training data, we can significantly improve the convergence speed of the CFR algorithm and ensure that the algorithm is more flexible and effective in dealing with unusual game situations.

[0079] Specifically, the present invention converts the decision-making scenarios encountered during the game into natural language descriptions, and uses a large language model to predict the most likely opponent behavior and the optimal response strategy. This method allows the incorporation of human-level intuition and experience into the algorithm, thereby better handling problems that are difficult to solve with traditional CFR algorithms.

[0080] In one embodiment, in combination Figure 1 The present invention provides an incomplete information game method based on a language model and a game tree CFR algorithm, the method comprising:

[0081] Step 1, construct a game data set;

[0082] Step 2, selecting a loss function and hyperparameters, and training a large language model based on the game data set;

[0083] Step 3, obtain the current game status;

[0084] Step 4: Set up a large language model;

[0085] Step 5, create an initial game tree;

[0086] Step 6, expand the game tree through Beam Search based on the large language model;

[0087] Step 7, divide the information set by the prior knowledge of the large language model and merge the information sets;

[0088] Step 8: Use the large language model scoring mechanism to calculate the leaf node value;

[0089] Step 9: Calculate the equilibrium strategy until convergence.

[0090] Furthermore, in one embodiment, the step 1 of constructing the game data set specifically includes:

[0091] Step 1-1, classifying the game data according to preset standards;

[0092] Here, the preset criteria include but are not limited to the complexity of the game, the level of the participants, the diversity of strategies, etc.;

[0093] Step 1-2, collect various types of game data from public databases, and perform data cleaning and data sorting to form a game data set; the data cleaning includes removing duplicates, irrelevant information or erroneous records; the data sorting is to sort the data into a unified format (for example, using JSON and CSV files to store the decisions and results of each step);

[0094] Here, high-quality data can be obtained through collection, cleaning and organization;

[0095] Steps 1-3, for the data in the dataset, add necessary contextual information for each decision step, such as the state of the game, optional actions, etc., and annotate the decision results of each step, such as the final winner, profit value, etc., to form the final game data set.

[0096] In some embodiments, the present invention designs a set of prompt templates that can be used to solve imperfect-information extensive-form games, ensuring that the model can understand the structure of the game tree and can perform reasonable thinking, action and observation steps according to the given state. Specifically, it includes the following steps: defining the game tree nodes, clarifying the format of Thought and Action, refining the role of Observation and integrating the feedback iteration mechanism, etc. Prompt template example:

[0097] Solve incomplete information expansion games with interleaved thinking, action, and observation steps. The game is represented by a game tree; the nodes of the tree are organized according to the following template: [TreeNode template]. Thinking can be reasoning about the current situation, and action can be divided into two types:

[0098] (1) Predict [node]: Give a prediction for the selected tree node. It is necessary to predict the possible future game state of the selected node.

[0099] (2) Scoring [Leaf Node]: Scoring a leaf node to show the value of the actions at that leaf node.

[0100] After each observation, provide the next step of thinking and action.

[0101] Preferably, in one embodiment, a cross entropy loss function is selected in step 2 to measure the difference between the probability distribution predicted by the model and the probability distribution of the true label. The cross entropy loss function is very effective for supervised learning process because it can effectively punish wrong predictions and reward correct predictions.

[0102] In multi-label classification problems, each sample can belong to multiple categories. In this case, the true label y of each category is ij It can be 0 or 1, where 1 means sample i belongs to category j and 0 means it does not. Therefore, for each category j, the cross entropy loss function can be defined as:

[0103] L ij =-y ij log(p ij )-(1-y ij )log(1-p ij )

[0104] For the entire dataset, we can calculate the average cross entropy loss over all samples and all classes:

[0105]

[0106] Preferably, in one embodiment, the hyperparameter combination in step 2 mainly includes:

[0107] (1) Learning rate: controls the learning speed of the model. A learning rate that is too large will cause the model to oscillate back and forth near the optimal solution and fail to converge to the optimal solution. A learning rate that is too small will cause the model to converge too slowly and may cause the model to fall into a local optimal solution, resulting in a decrease in model performance.

[0108] (2) Batch size: Controls the amount of data the model receives each time. Too large a batch size will increase the training cost, while too small a batch size will lead to unstable training direction and make the model more likely to fall into a local optimal solution.

[0109] (3) Training length: Control the total training length. Too long training length will lead to overfitting of the model, while too short training length will lead to underfitting of the model.

[0110] Furthermore, in one embodiment, the step 3 of obtaining the current game status specifically includes:

[0111] Read the current information set as I, the current state in the game tree is recorded as s, and the current player is p now , the player p(s) or p(I) currently in state s or information set I, the current opponent op, and the information info(s,p) and info(I,p) that the player p can observe in state s and information set I.

[0112] Further, in one embodiment, the step 4 of setting the large language model specifically includes:

[0113] Initialize the large language model: load the large language model pre-trained in step 2 to generate actions and evaluate state values;

[0114] Define the large language model interface: Define the interface between the large language model and the environment to enable the model to receive state descriptions, perform actions, and obtain feedback;

[0115] Initialize memory: Initialize a memory structure to store past game states, actions, and corresponding value estimates.

[0116] Further, in one embodiment, the step 5 of creating the initial game tree includes:

[0117] Step 5-1, creating the root node of the initial game tree according to the structured construction method of the game tree node;

[0118] Here, the initial game tree is created to ensure that the game tree contains enough branches to cover multiple possible game paths. The quality of game tree node construction is also crucial to the understanding of the model. Well-structured nodes can help the model better understand and process the game state.

[0119] Here, we use structured data to construct the game tree nodes. The structured example is as follows:

[0120] Node(id=1, player=1, information=[Ace of Spades, King of Spades, Queen of Spades, Jack of Spades, Ten of Spades], action=[Discard, Check], result=[(441,5), probability=0.0014], child nodes=[2,4,5])

[0121] Node(id=2, player=2, information=[Hearts 3, Hearts 4, Hearts 5, Hearts 6, Hearts 7], action=[Discard], result=[(-5, -441), probability=0.0055], child nodes=[11, 13, 25]).

[0122] Step 5-2, infer the opponent's hidden information: Based on the current game state and known information, the pre-trained large language model is used to infer the opponent's possible hidden information, and the opponent's possible information or state and its probability are obtained, which are expressed as (state, probability) tuples: (s1, p1), (s2, p2), (s3, p3), ...; where s i represents the ith possible state of the opponent, p i Indicates i The corresponding probability is i=1, 2, ...;

[0123] Step 5-3, infer your own information: Based on the possible information or state and its probability obtained in step 5-2, infer which of your own information will be considered important or possible by the opponent through the pre-trained large language model, and obtain your own possible information or state and its probability, expressed as a (state, probability) tuple: (S 11 , p 11 ), (s 12 , p 12 ), ...; where s ij It means that when the opponent has the i-th possible state, the opponent sees the j-th possible state of the opponent, p ij Indicates ij The corresponding probabilities are i, j = 1, 2, ...;

[0124] Step 5-4, create child nodes of the initial game tree: For all tuples obtained in steps 5-2 and 5-3, create a node for each tuple as a child node of the initial game tree, and create an edge and its corresponding transition probability for each child node.

[0125] Further, in one embodiment, the step 6 of expanding the game tree using Beam Search based on a large language model specifically includes:

[0126] Step 6-1, for each layer of node set in the game tree, select the best k nodes through the pre-trained large language model;

[0127] Step 6-2, based on k nodes, expand the game tree through Beam Search, expressed as: T new =BeamSearch(T current , k), where T current is the current game tree, T new is the expanded game tree, and BeamSearch indicates the BeamSearch expansion;

[0128] In step 6-3, a large language model is used to evaluate each node and predict the next possible action or state and its probability.

[0129] Here, Beam Search is a search algorithm that only retains a certain number of the best candidate solutions in each search stage to reduce the search space and improve efficiency. This method is particularly suitable for situations where a large number of candidate solutions need to be processed, such as sequence generation tasks in natural language processing. At each search stage, the algorithm selects the best k candidate solutions from the current set of candidate solutions. k is a pre-set positive integer, called (beam width), and the search continues based on these candidate solutions. This method is particularly suitable for situations where a large number of candidate solutions need to be processed, because unnecessary computational overhead can be avoided by gradually screening out the candidate solutions that are most likely to succeed.

[0130] The solution of this embodiment is adopted to perform Beam Search in combination with a large language model, and the powerful prediction ability and prior knowledge of the large model are used to guide the search process, thereby improving the search efficiency.

[0131] The solution of this embodiment, combined with the large language model and Beam Search, can not only effectively expand the game tree, but also ensure that the expansion process is efficient, because we only focus on the nodes with the most potential. This helps to maximize the depth and breadth of the search under limited computing resources.

[0132] Furthermore, in one embodiment, the step 7 divides the information set and merges the information set by using the prior knowledge of the large language model, specifically:

[0133] Let the large model merge nodes with similar states in the game tree based on its prior knowledge to simplify the game tree. In incomplete information games, there are usually a large number of possible states, which is reflected in the fact that each node in the game tree may have a large number of child nodes, that is, the state corresponding to the node has a large number of possible transition states. However, in actual game effectiveness, there are usually situations where several nodes can be merged. For example, when "the opponent holds A, K, Q, J, 10", although from the opponent's perspective, we may hold cards like "A, A, 4, 5, 6" or "A, A, 3, 4, 5", the difference between these two possibilities is small and will not affect the opponent's judgment in terms of strategy. Therefore, they can be regarded as the same state and the information sets can be merged into the same state, such as Figure 2 shown.

[0134] In this step, we use the powerful prior knowledge of the big model to avoid manually designing complex merging rules, and use the big model's understanding of the game state, that is, the "Hidden States" inside the model after the state is input into the model to determine the mergeability of the state. Specifically, the states are merged by performing threshold clustering based on the state cosine similarity of the "Hidden States" of the model after different game states are input.

[0135] This step specifically includes:

[0136] Step 7-1, using the pre-trained large language model to divide the information set according to its prior knowledge, and setting a hidden label for each game state in the information set to indicate its category;

[0137] Step 7-2, by clustering based on the state cosine similarity of the hidden layer representation of the large language model after inputting different game states to merge the game states, specifically:

[0138] The similarity between different inputs is calculated using the cosine angular distance represented by the hidden layer:

[0139]

[0140] in, represents the cosine similarity of two different input states s1 and s2 represented in the i-th hidden layer, Respectively represent the hidden layer representation of the i-th layer corresponding to s1 and s2;

[0141] If the similarity is greater than a preset threshold, the input information sets corresponding to the two inputs are merged.

[0142] By adopting the solution of this embodiment, clustering will merge similar states, reduce the branch tree of the game tree, and improve search efficiency.

[0143] Further, in one embodiment, step 8 is specifically as follows: define the value function as V(s), where s is a state in the state space. Input the state corresponding to the leaf node directly into the large language model, and let the large language model calculate the corresponding score, such as Figure 3 The value calculation is expressed as V(s) = LLM Score(s) , where V(s) is the value of the leaf node state s.

[0144] Further, in one embodiment, step 9 specifically includes:

[0145] Step 9-1, Initialization

[0146] Initialize a strategy vector for each information set in the game tree (i.e., the probability of selecting the information set, which can be initialized to the average probability of selecting all information sets, representing a random strategy), and initialize a regret vector for each information set (i.e., the loss of benefits after selecting the strategy);

[0147] Here, a good initialization is provided to reduce the number of CFR iterations and help reach the Nash equilibrium quickly;

[0148] Step 9-2, traverse the game tree

[0149] Traverse the game tree starting from the root node and calculate the regret value of each node;

[0150] Step 9-3, Update strategy

[0151] Update the strategy of the game tree according to the calculated regret value;

[0152] Step 9-4, convergence test

[0153] Repeat steps 9-2 and 9-3, after multiple regret value calculations and strategy updates, check whether the strategy has converged, and stop iterating if the regret value is lower than a preset threshold or the current predetermined number of iterations has been reached. Preferably, the average regret value of all information sets can be calculated to measure.

[0154] Here, based on the performance and feedback of the generated data, the Prompt template and model parameters are continuously adjusted and multiple rounds of iterations are performed to improve the accuracy and practicality of the generated data.

[0155] Here, in some embodiments, calculating the regret value using the CFR algorithm in step 9-2 specifically includes:

[0156] For each possible action a, calculate the regret value of the action. The regret value reflects the benefit that would be missed if action a was selected at this node instead of the existing strategy, that is, the benefit that could not be obtained. When the regret value is positive, it means that the action will have a positive benefit compared to the existing given strategy (that is, the existing strategy after initialization or iteration).

[0157] When a certain strategy σ is adopted, the action sequence h is obtained according to the strategy σ, and the virtual value v brought by h i (σ,h) is calculated as:

[0158]

[0159] Among them, π σ (h, z) represents the probability of reaching the final state z through action sequence h under the σ strategy, represents the probability of occurrence of action sequence h, u i (z) represents the value of player i’s final state z;

[0160] At this point, under a given strategy σ, the regret value of player i taking action a is calculated as follows:

[0161] r(h,a)=v i (σ I→a , h)-v i (σ, h)

[0162] Among them, r(h, a) represents the regret value of player i taking action a, and v i (σ I→a ,h) represents the virtual value of taking action a directly in the information set, position,v i (σ, h) represents the virtual value of not taking action a directly but taking action according to strategy σ normally;

[0163] The information set corresponding to the action sequence h has a regret value of r(I, a) = ∑ h∈I r(h, a), the regret value of player i taking action a in round T is

[0164]

[0165] In the formula, Represents the information set and regret value corresponding to player i taking action a in round t.

[0166] Here, in some embodiments, the formula for updating the strategy in step 9-3 is:

[0167]

[0168] In the formula, It represents the probability that the strategy σ takes action a directly in the information set position after the T+1th policy iteration. represents the regret value of player i taking action a in round T, and A(I) represents the set of actions that can be taken under information set I.

[0169] That is, if the regret value is negative, it means that the selected action will not bring positive benefits, so all actions with negative benefits are selected with equal probability; if the regret value is positive, it means that the selected action will bring positive benefits, so the strategy value of the action is redistributed according to the proportion of the positive regret value brought by the action in all positive regret values. The action with a higher regret value will get more strategy value allocation and has a higher probability of being executed.

[0170] In one embodiment, an incomplete information game system based on a language model and a game tree CFR algorithm is provided, the system comprising:

[0171] The first module is used to construct a game data set;

[0172] The second module is used to select a loss function and hyperparameters, and train a large language model based on the game data set;

[0173] The third module is used to obtain the current game status;

[0174] The fourth module is used to set up a large language model;

[0175] The fifth module is used to create an initial game tree;

[0176] The sixth module is used to expand the game tree through Beam Search based on a large language model;

[0177] The seventh module is used to divide the information set and merge the information set through the prior knowledge of the large language model;

[0178] The eighth module is used to calculate the value of leaf nodes using the large language model scoring mechanism;

[0179] The ninth module is used to calculate the equilibrium strategy until convergence.

[0180] For the specific definition of the incomplete information game system based on the language model and the game tree CFR algorithm, please refer to the definition of the incomplete information game method based on the language model and the game tree CFR algorithm in the above text, which will not be repeated here. Each module in the above-mentioned incomplete information game system based on the language model and the game tree CFR algorithm can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0181] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following is achieved:

[0182] Step 1, construct a game data set;

[0183] Step 2, selecting a loss function and hyperparameters, and training a large language model based on the game data set;

[0184] Step 3, obtain the current game status;

[0185] Step 4: Set up a large language model;

[0186] Step 5, create an initial game tree;

[0187] Step 6, expand the game tree through Beam Search based on the large language model;

[0188] Step 7, divide the information set by the prior knowledge of the large language model and merge the information sets;

[0189] Step 8: Use the large language model scoring mechanism to calculate the leaf node value;

[0190] Step 9: Calculate the equilibrium strategy until convergence.

[0191] For the specific limitations of each step, please refer to the limitations of the incomplete information game method based on the language model and game tree CFR algorithm mentioned above, which will not be repeated here.

[0192] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the computer program implements:

[0193] Step 1, construct a game data set;

[0194] Step 2, selecting a loss function and hyperparameters, and training a large language model based on the game data set;

[0195] Step 3, obtain the current game status;

[0196] Step 4: Set up a large language model;

[0197] Step 5, create an initial game tree;

[0198] Step 6, expand the game tree through Beam Search based on the large language model;

[0199] Step 7, divide the information set by the prior knowledge of the large language model and merge the information sets;

[0200] Step 8: Use the large language model scoring mechanism to calculate the leaf node value;

[0201] Step 9: Calculate the equilibrium strategy until convergence.

[0202] For the specific limitations of each step, please refer to the limitations of the incomplete information game method based on the language model and game tree CFR algorithm mentioned above, which will not be repeated here.

[0203] As a specific example, the present invention is further described in detail in one of the embodiments.

[0204] In a Texas Hold'em game, a player has a hand of K and 2, and the public cards that have been turned over are 6, 5, K, and 3. Player 1 starts to raise the bet after the first three public cards are turned over, while Player 2 is more conservative and does not raise the bet before the fourth card is turned over. However, when the fourth public card (3) appears, Player 2 also starts to raise the bet.

[0205] The first stage of constructing the game tree:

[0206] Based on the current game state, a large language model can be used to perform probability assessment and opponent behavior prediction. In this case, since Player 2 started raising after the fourth public card 3 appeared, it indicates that he may have a better hand. Therefore, it can be assumed that Player 2 has a high probability of holding a straight, such as 50%, and a 50% probability of holding a weak pair.

[0207] Next, from the perspective of Player 2, we can further infer the types of hands that Player 1 may hold and their probability distribution. Based on Player 1’s previous raising behavior, we can infer that they may hold the following two types of hands:

[0208] Holding a high hand, 30% probability;

[0209] Holding a pair with a 70% probability.

[0210] Combining the above information, we can deduce the following four possible situations and their corresponding probabilities:

[0211] Player 1 holds a high card and Player 2 holds a straight: The probability of this happening is 15% (i.e. 30%*50%).

[0212] Player 1 holds a high card and Player 2 holds a weak pair: The probability of this happening is also 15% (i.e. 30%*50%).

[0213] Player 1 holds a pair and Player 2 holds a straight: The probability of this happening is 35% (i.e. 70%*50%).

[0214] Player 1 holds a pair and Player 2 holds a weak pair: The probability of this happening is also 35% (i.e. 70%*50%).

[0215] The results are as follows Figure 4 shown.

[0216] Beam Search and Information Set Merging Phase:

[0217] Beam Search

[0218] Use Beam Search to explore the game tree and keep only the paths with the highest probability. The beam width can be set based on the computing resources and accuracy requirements. At each decision point, the most likely paths are kept and other unlikely paths are ignored.

[0219] Information set merging

[0220] When two or more nodes represent the same decision point, they are merged into one information set. When merging information sets, the probability distribution needs to be updated to reflect all possible paths.

[0221] Building a game tree

[0222] Player 1's decision node

[0223] Fold

[0224] Call

[0225] Raise

[0226] Player 2's decision node

[0227] Fold

[0228] Call

[0229] Raise

[0230] Example game tree structure

[0231] 1. Starting node: The player has a hand of K and 2, and the community cards are 6, 5, K, 3.

[0232] 2. Player 1’s first decision node:

[0233] Bet

[0234] Player 2's first decision node:

[0235] Call

[0236] The fifth community card appears

[0237] Player 1's second decision node:

[0238] Fold

[0239] Call

[0240] Raise

[0241] Player 2's second decision node:

[0242] Fold

[0243] Call

[0244] Raise

[0245] 3. Use Beam Search to filter paths:

[0246] Assuming the beam width is 3, at each decision node, the three paths with the highest probability are retained.

[0247] 4. Information set merging:

[0248] At each decision node of player one and player two, merge the decision points with the same information into the same information set.

[0249] Derive the game tree.

[0250] Probability distribution update

[0251] At each decision node, the probability distribution is updated based on the opponent's behavior and the known community cards.

[0252] in conclusion

[0253] Through the above steps, we can build a simplified game tree and use the CFR algorithm to solve it. The Nash equilibrium strategy is 100% call and 0% fold. This process needs to be continuously iterated and improved in order to more accurately predict the opponent's behavior and strategy.

[0254] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An incomplete information game method based on language model and game tree CFR algorithm, characterized in that: The method comprises: Step 1, construct a game data set; Step 2, selecting a loss function and hyperparameters, and training a large language model based on the game data set; Step 3, obtain the current game status; Step 4: Set up a large language model; Step 5, create an initial game tree; Step 6, expand the game tree through Beam Search based on the large language model; Step 7, divide the information set by the prior knowledge of the large language model and merge the information sets; Step 8: Use the large language model scoring mechanism to calculate the leaf node value; Step 9: Calculate the equilibrium strategy until convergence.

2. The incomplete information game method based on language model and game tree CFR algorithm according to claim 1 is characterized in that: Step 1 of constructing the game data set specifically includes: Step 1-1, classifying the game data according to preset standards; Step 1-2, collecting various types of game data from public databases, and performing data cleaning and data sorting to form a game data set; the data cleaning includes removing duplicate items, irrelevant information or erroneous records; the data sorting is to sort the data into a unified format; Steps 1-3, for the data in the dataset, add necessary contextual information for each decision step, and annotate the decision results of each step to form the final game dataset.

3. The incomplete information game method based on language model and game tree CFR algorithm according to claim 1 is characterized in that: Step 3 of obtaining the current game status specifically includes: Read the current information set as I, the current state in the game tree is s, and the current player is p now , the player p(s) or p(I) currently in state s or information set I, the current opponent op, and the information info(s,p) and info(I,p) that the player p can observe in state s and information set I.

4. The incomplete information game method based on language model and game tree CFR algorithm according to claim 1 is characterized in that: Step 4 describes setting up a large language model, specifically including: Initialize the large language model: load the large language model pre-trained in step 2 to generate actions and evaluate state values; Define the large language model interface: Define the interface between the large language model and the environment to enable the model to receive state descriptions, perform actions, and obtain feedback; Initialize memory: Initialize a memory structure to store past game states, actions, and corresponding value estimates.

5. The incomplete information game method based on language model and game tree CFR algorithm according to claim 1 is characterized in that: Step 5 creates an initial game tree, including: Step 5-1, creating the root node of the initial game tree according to the structured construction method of the game tree node; Step 5-2, infer the opponent's hidden information: Based on the current game state and known information, the pre-trained large language model is used to infer the opponent's possible hidden information, and the opponent's possible information or state and its probability are obtained, which are expressed as (state, probability) tuples: (s1, p1), (s2, p2), (s3, p3), ...; where s i represents the ith possible state of the opponent, p i Indicates i The corresponding probability is i=1,2,...; Step 5-3, infer your own information: Based on the possible information or state and its probability obtained in step 5-2, infer which of your own information will be considered important or possible by the opponent through the pre-trained large language model, and obtain your own possible information or state and its probability, expressed as a (state, probability) tuple: (i 11 ,p 11 ),(s 12 ,p 12 ),...; where s ij It means that when the opponent has the i-th possible state, the opponent sees the j-th possible state of the opponent, p ij Indicates ij The corresponding probabilities are i,j=1,2,...; Step 5-4, create child nodes of the initial game tree: For all tuples obtained in steps 5-2 and 5-3, create a node for each tuple as a child node of the initial game tree, and create an edge and its corresponding transition probability for each child node.

6. The incomplete information game method based on language model and game tree CFR algorithm according to claim 5 is characterized in that: Step 6 uses Beam Search based on a large language model to expand the game tree, specifically including: Step 6-1, for each layer of node set in the game tree, select the best k nodes through the pre-trained large language model; Step 6-2, based on k nodes, expand the game tree through Beam Search, expressed as: T new =BeamSearch(T current ,k), where T current is the current game tree, T new is the expanded game tree, and BeamSearch indicates the BeamSearch expansion; In step 6-3, a large language model is used to evaluate each node and predict the next possible action or state and its probability.

7. The incomplete information game method based on language model and game tree CFR algorithm according to claim 1 is characterized in that: Step 7 divides the information set and merges the information set using the prior knowledge of the large language model, specifically including: Step 7-1, using the pre-trained large language model to divide the information set according to its prior knowledge, and setting a hidden label for each game state in the information set to indicate its category; Step 7-2, by clustering based on the state cosine similarity of the hidden layer representation of the large language model after inputting different game states to merge the game states, specifically: The similarity between different inputs is calculated using the cosine angular distance represented by the hidden layer: in, represents the cosine similarity of two different input states s1 and s2 represented in the i-th hidden layer, Respectively represent the hidden layer representation of the i-th layer corresponding to s1 and s2; If the similarity is greater than a preset threshold, the input information sets corresponding to the two inputs are merged.

8. The incomplete information game method based on language model and game tree CFR algorithm according to claim 1, characterized in that: Step 9 specifically includes: Step 9-1, Initialization Initialize a strategy vector for each information set in the game tree, and initialize a regret vector for each information set; Step 9-2, traverse the game tree Traverse the game tree starting from the root node and calculate the regret value of each node; Step 9-3, Update strategy Update the strategy of the game tree according to the calculated regret value; Step 9-4, convergence test Repeat steps 9-2 and 9-3. After multiple regret value calculations and strategy updates, check whether the strategy has converged. If the regret value is lower than a preset threshold or the current predetermined number of iterations has been reached, stop the iteration.

9. The incomplete information game method based on language model and game tree CFR algorithm according to claim 8, characterized in that: The calculation of regret value in step 9-2 specifically includes: When a certain strategy σ is adopted, the action sequence h is obtained according to the strategy σ, and the virtual value v brought by h i (σ,h) is calculated as: Among them, π σ (h,z) represents the probability of reaching the final state z through action sequence h under the σ strategy, represents the probability of occurrence of action sequence h, u i (z) represents the value of player i’s final state z; At this point, under a given strategy σ, the regret value of player i taking action a is calculated as follows: r(h,a)=vi(σ I→a ,h)-v i (σ,h) Among them, r(h,a) represents the regret value of player i taking action a, and v i (σ I→a ,h) represents the virtual value of taking action a directly at position I in information set, v i (σ,h) represents the virtual value of not taking action a directly but taking action according to strategy σ normally; The regret value of the information set I corresponding to the action sequence h is r(I,a) = ∑ h∈I r(h,a), the regret value of player i taking action a in round T is In the formula, It represents the regret value of information set I corresponding to player i taking action a in round t.

10. The incomplete information game method based on language model and game tree CFR algorithm according to claim 8, characterized in that: The formula for step 9-3 updating the strategy is: In the formula, represents the probability that strategy σ directly takes action a at position I in information set after the T+1th policy iteration, represents the regret value of player i taking action a in round T, and A(I) represents the set of actions that can be taken under information set I.

Images