A solution for large-scale poker games based on reward function transformation and regret discount.
By using a perturbation-regularized extended game model based on reward function transformation and regret discount, the problems of slow convergence in the final round of large-scale poker game solving are solved, achieving fast convergence and efficient policy update, thus improving the engineering practicality and versatility of the algorithm.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUN YAT SEN UNIV
- Filing Date
- 2026-03-13
- Publication Date
- 2026-06-02
AI Technical Summary
Existing methods for solving large-scale poker games suffer from problems such as insufficient convergence in the final iteration, slow convergence speed, difficulty in parameter tuning, and poor compatibility with reward transformation frameworks, which limit their application and engineering deployment in resource-constrained environments.
A perturbation-regularized extended game model based on reward function transformation and regret discount is adopted. By initializing parameters, constructing and iteratively solving the perturbation-regularized game model, and combining optimistic prediction mechanism and regret discount technique, the strategy is updated to achieve asymptotic final-round iterative convergence and fast convergence speed.
While maintaining theoretical convergence, it significantly reduces memory overhead, simplifies parameter debugging process, improves the engineering practicality and versatility of the algorithm, achieves exponential convergence speed improvement in various game scenarios, and ensures high accuracy and direct applicability of the strategy.
Smart Images
Figure CN122124473A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the fields of artificial intelligence and game theory, specifically relating to a method for solving large-scale poker games based on reward function transformation and regret discount. Background Technology
[0002] Large-scale poker games, as a typical example of extended-form games (EFGs), possess core characteristics such as exponential growth of the state space, incomplete information, complex decision paths, and strong randomness. Efficiently solving these games has significant theoretical and practical value for the application of artificial intelligence in complex decision-making scenarios (such as financial transactions, autonomous driving decision-making, and cybersecurity defense). The core objective of solving extended-form games is to find a Nash equilibrium (NE), in which no agent can increase its payoff by unilaterally adjusting its strategy.
[0003] Currently, efficient methods for solving extended games mainly fall into two categories: First-Order Methods (FOM) and Counterfactual Regret Minimization (CFR) algorithms. Among these, CFR algorithms are more advantageous due to their ability to flexibly combine various acceleration techniques (such as regret discounting) and their ability to achieve performance exceeding human capabilities in practical applications. CFR algorithms minimize the individual regret value of each participant, ensuring that the average policy generated over all iterations converges to an approximate Nash equilibrium; this property is known as "average iterative convergence."
[0004] However, existing algorithms based on the CFR framework (including CFR+, DCFR, etc.) have the following significant drawbacks:
[0005] 1. Lack of convergence in the final iteration: Existing algorithms only possess "average iteration convergence," which means that the strategies of all historical iterations must be stored and their averages calculated to approximate the Nash equilibrium. This leads to huge memory overhead, and the strategies of the final round cannot be directly used in practical engineering deployments, limiting the application of the algorithm in resource-constrained environments.
[0006] 2. Poor compatibility with the reward transformation framework: Although the "Reward Transformation (RT)" framework was proposed to prove the final iteration convergence of CFR-type algorithms, existing regret discount methods are difficult to directly embed into this framework and lack theoretical guarantees under perturbed regularized EFGs.
[0007] 3. Parameter sensitivity and debugging difficulties: Existing algorithms are often highly sensitive to step size and initialization parameters, requiring tedious parameter adjustments for specific game scenarios.
[0008] 4. Lack of efficient predictive algorithms: Under the reward transformation framework, there is a lack of efficient algorithms that combine optimistic prediction mechanisms to further accelerate convergence.
[0009] Therefore, there is an urgent need for a method to solve large-scale poker games that can guarantee convergence in the final iteration, maintain a fast convergence speed, and be robust to parameters. Summary of the Invention
[0010] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a method for solving large-scale poker games based on reward function transformation and regret discount, thereby solving the problems of lack of final iteration convergence guarantee, slow convergence speed, and difficulty in parameter adjustment in existing large-scale poker game solutions.
[0011] To achieve the above objectives, the present invention adopts the following technical solution:
[0012] This invention provides a method for solving large-scale poker games based on reward function transformation and regret discount, applicable to large-scale poker games, and includes the following steps:
[0013] S1. Parameter Initialization: Set the number of perturbation-regularized extended game models to be built. Solving for the number of iterations in the perturbation-regularized extended game model Regularization coefficient Disturbance coefficient and reference strategy Initialize the cumulative counterfactual regret of the first perturbation-regularized game model and step length If an optimistic solution strategy is adopted, an additional discount factor is set. ;
[0014] S2. Construct a perturbation-regularized extended game model: based on current parameters and Construct the first A perturbation-regularized extended game model is constructed, and the next model is constructed only after the previous model has been solved.
[0015] S3. Iteratively solve the current perturbation regularized game model: for each internal time step For each information set in the game tree and its actions Perform the following update rules:
[0016] Calculate player strategy in sequence form loss function Calculate the counterfactual value by backtracking from the leaf nodes of the game to the root node. Calculating instantaneous counterfactual regret based on counterfactual values By utilizing instantaneous counterfactual regret and combining it with regret discounting techniques, cumulative counterfactual regret is updated. If an optimistic solution strategy is adopted, the cumulative counterfactual regrets of the predictions are updated synchronously. Then update the behavioral strategy based on accumulated counterfactual regret. Combined with the disturbance coefficient Update perturbation behavior strategy Update the perturbation sequence form policy based on the perturbation behavior policy. ;
[0017] S4. Update parameters and initialize the next round of the model: Whenever a perturbation-regularized game model is solved, update the parameters. The system initializes the initial cumulative regret of the next round model using the cumulative regret value of the last round of the previous round model, and continues to build and solve N models until the final sequence form strategy is output as the Nash equilibrium strategy of the poker game.
[0018] As a preferred technical solution, in step S1, the cumulative counterfactual regret of the first perturbation regularized game model is initialized. =0, step size The value is 1.
[0019] As a preferred technical solution, the perturbation-regularized extended game model constructed in step S2 is as follows:
[0020]
[0021] in, These represent the sequential strategies of the two players. For setting reference strategies, Let Bregman divergence be the property of the divergence. .
[0022] As a preferred technical solution, in step S3, the loss function is calculated as follows:
[0023] At the current iteration time Calculate the strategy in sequence form for each of the two players. loss function and The calculation formula is as follows:
[0024]
[0025]
[0026] in, For the profit matrix, For reference strategy, is the regularization coefficient.
[0027] As a preferred technical solution, in step S3, the counterfactual value is calculated. The formula is as follows:
[0028]
[0029] in, This indicates the players that can be reached after performing action 'a'. The set of sub-information sets.
[0030] As a preferred technical solution, in step S3, the instantaneous counterfactual regret is calculated. The formula is as follows:
[0031]
[0032] in, Information set The counterfactual value, For players in the information set The current behavioral strategy.
[0033] As a preferred technical solution, in step S3, a maximum value function is used to ensure that the cumulative counterfactual regret is non-negative, and the update formula is:
[0034]
[0035] in, As a regret discount factor, = Ensure the regret value is non-negative;
[0036] If an optimistic solution strategy is adopted, the update formula for predicting cumulative counterfactual regrets is:
[0037] .
[0038] As a preferred technical solution, in step S3, the behavior strategy is updated based on the accumulated counterfactual regret. Specifically:
[0039]
[0040] Combined with disturbance coefficient Update behavioral strategies :
[0041]
[0042] in, , For information set The number of elements in the action set.
[0043] As a preferred technical solution, in step S3, the strategy for updating the perturbation sequence form is... Specifically:
[0044]
[0045] in, In order to comply with the disturbance behavior policy At that time, the player Information set The contribution value of the probability of occurrence.
[0046] As a preferred technical solution, in step S4, referencing strategy The update method is That is, the reference strategy for the next model is updated using the sequential strategy obtained from the current model solution; regularization coefficient and disturbance coefficient A non-incremental decay strategy is adopted, eventually approaching 0, while the cumulative regret value in the last round of the previous model is used. As the initial cumulative regret value for the next model .
[0047] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0048] 1. Based on the reward transformation framework, this invention proves that the RTDCFR+ and RTPDCFR+ algorithms have asymptotic final iteration convergence in perturbation regularized extended form games. The final strategy at the time of algorithm termination can be directly used as the Nash equilibrium strategy without storing all historical iteration strategies and taking the average value. This completely solves the problem of huge memory overhead of traditional CFR algorithms and greatly improves the engineering practicality of the algorithm.
[0049] 2. By introducing an optimistic prediction mechanism and regret discounting technique, the policy update process is significantly accelerated while maintaining theoretical convergence. Compared with the RTCFR+ algorithm, this invention achieves an exponential improvement in convergence speed in large-scale games. Moreover, in various game scenarios such as LeducPoker, Goofspiel, and Battleship, the convergence speed is significantly better than mainstream algorithms such as PCFR+, Reg-CFR, and DCFR.
[0050] 3. This invention proves that the convergence of the algorithm holds true for any initialization parameters and step size settings (invariant convergence), eliminating the need for tedious parameter debugging for different large-scale poker game scenarios, greatly simplifying the engineering deployment process of the algorithm, and improving the algorithm's versatility and adaptability.
[0051] 4. This invention derives a more compact average iterative convergence regret bound than existing technologies, ensuring that, with the same number of iterations, the policy generated by this invention has a smaller error in distance from Nash equilibrium and higher solution accuracy.
[0052] 5. It solves the compatibility problem between the existing regret discount method and the reward transformation framework, and provides a rigorous theoretical guarantee for the combination of the two techniques under the perturbation regularized extended game, forming a complete solution for large-scale poker games. Attached Figure Description
[0053] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0054] Figure 1 This is a flowchart of the large-scale poker game solution method based on reward function transformation and regret discount in Embodiment 1 of the present invention;
[0055] Figure 2 This is a flowchart of the large-scale poker game solution method based on reward function transformation and regret discount in Embodiment 2 of the present invention;
[0056] Figure 3 This is a comparison of the convergence speed of the Optimistic Counterfactual Regret Minimization Algorithm (RTPDCFR+) based on reward function transformation and regret discount with mainstream algorithms in various game scenarios. Detailed Implementation
[0057] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.
[0058] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.
[0059] Explanation of relevant terms:
[0060] Extensive-Form Games (EFGs): A game model that describes a sequential decision-making process, including a set of players, historical nodes, information set, action set, and utility function.
[0061] Nash Equilibrium (NE): A strategy combination in game theory in which neither side can gain a higher payoff by unilaterally changing their strategy.
[0062] Counterfactual Regret: In the CFR algorithm, it refers to the expected additional benefit that could be obtained by taking a certain action compared to the current strategy, given a specific set of information.
[0063] Reward transformation (RT): A technical framework that transforms the original game utility function into a perturbation-regularized game problem by adding regularization and perturbation terms.
[0064] Last-Iterate Convergence: This refers to the algorithm's final output policy converging to a Nash equilibrium at the end of the iteration, rather than the average of historical policies.
[0065] Example 1
[0066] like Figure 1 As shown, this embodiment 1 provides a method for solving large-scale poker games based on reward function transformation and regret discount. This embodiment is a two-player zero-sum game, set as player 0 and player 1, and adopts the counterfactual regret minimization algorithm (RTDCFR+) based on reward function transformation and regret discount, including the following steps:
[0067] S1, Initialize parameters;
[0068] Set the number of perturbation-regularized extended game models to be built. Solving for the number of iterations in the perturbation-regularized extended game model Regularization coefficient Disturbance coefficient Reference Strategy .
[0069] Furthermore, the cumulative counterfactual regret of initializing the first perturbation-regularized game model. =0, step size The value is 1. It is worth noting that cumulative counterfactual regret is used. 0 step size Setting it to 1 is only one possible option; initializing the cumulative counterfactual regret of the first perturbation-regularized game model. and step length It can be any value.
[0070] S2. Construct a series of perturbation-regularized extended game models;
[0071] For constructing the A perturbed regularized extensive-form game model, based on the current parameters. and Construct a perturbation-regularized extended form of game theory.
[0072]
[0073] in, These represent the sequential strategies of the two players. For the set reference strategy, the Bregman divergence is represented as: In this embodiment,
[0074] After constructing a perturbation-regularized extended form game, the current game is solved first. Only after the previous perturbation-regularized extended form game has been solved can the next perturbation-regularized extended form game be constructed.
[0075] S3. Iteratively solve the current perturbation-regularized game model;
[0076] The algorithm solves the perturbation-regularized game model iteratively, with a total of [number] iterations. Wheel. For each internal time step For each information set in the game tree and its actions Execute the following update rules:
[0077] S31. Calculate the loss function: at the current iteration time... For players and players Calculate their strategies in sequence form respectively. loss function This loss function includes not only linear terms but also a regularization term. To ensure the smoothness of the strategy:
[0078]
[0079]
[0080] in, For the profit matrix, For reference strategy, is the regularization coefficient.
[0081] S32. Calculate the counterfactual value: Backtrack from the leaf nodes of the game tree to the root node, and calculate the counterfactual value for each information set. In action Counterfactual values on This value is composed of the current loss and the expected counterfactual value of the child nodes:
[0082]
[0083] in, This represents the set of sub-information sets reached after the execution of action a.
[0084] S33. Calculate instantaneous counterfactual regret: Based on the counterfactual value, calculate the current moment. In information set The instantaneous counterfactual regret :
[0085]
[0086] This step quantifies the immediate gap between the current strategy and the optimal response.
[0087] S34. Update Cumulative Counterfactual Regrets: Utilize Instantaneous Regrets and step length Update accumulated counterfactual regrets. An operation was used here. (i.e., the function that takes the maximum value) ), ensuring the regret value is non-negative:
[0088]
[0089] S35, Update Behavioral Strategies and Disruptive Behavioral Strategies: For players Based on accumulated counterfactual regret Update behavioral strategies :
[0090]
[0091] Combined with disturbance coefficient Update the perturbation behavior strategy :
[0092]
[0093] S36. Update sequence form strategy: Let... To comply with the strategy Below, information set The probability of occurrence. ,but Information set In the probability of occurrence, the player The contributions made. (Combined with player contributions) The current information set Update players Sequence-based strategies :
[0094]
[0095] S4, Update Parameters Initialize the initial cumulative regret for the next round of the model. ,like Figure 1 As shown, specifically:
[0096] S41. Update Reference Strategy Update the reference policy for the next perturbation-regularized game using the sequential policy result from the previous perturbation-regularized game solution.
[0097]
[0098] S42. Update the regularization coefficients. Disturbance coefficient In this embodiment, the update method is set as follows:
[0099]
[0100] S43. Update and initialize cumulative counterfactual regret value: for all information sets of all players. Use the cumulative regret value from the last round of the previous perturbation-regularized game. Update the initial cumulative regret value for the next perturbation-regularized game. :
[0101] .
[0102] Example 2
[0103] This embodiment 2 provides a method for solving large-scale poker games based on reward function transformation and regret discounting. It employs the Optimistic Counterfactual Regret Minimization Algorithm (RTPDCFR+) based on reward function transformation and regret discounting. Figure 2 As shown, the method includes the following steps:
[0104] S1, Initialize parameters;
[0105] Set the number of perturbation-regularized extended game models to be built. Solving for the number of iterations in the perturbation-regularized extended game model Regularization coefficient Disturbance coefficient Reference Strategy Discount coefficient ;
[0106] Furthermore, the cumulative counterfactual regret of initializing the first perturbation-regularized game model. =0, step size The value is 1.
[0107] S2. Construct a series of perturbation-regularized extended game models;
[0108] For constructing the A perturbed regularized extensive-form game model, based on the current parameters. and Construct a perturbation-regularized extended form of game theory.
[0109]
[0110] in, These represent the sequential strategies of the two players. For the set reference strategy, the Bregman divergence is represented as: In this embodiment,
[0111] After constructing a perturbation-regularized extended form game, the current game is solved first. Only after the previous perturbation-regularized extended form game has been solved can the next perturbation-regularized extended form game be constructed.
[0112] S3. Iteratively solve the current perturbation-regularized game model;
[0113] The algorithm solves the perturbation-regularized game model iteratively, with a total of [number] iterations. Wheel. For each internal time step For each information set in the game tree and its actions Execute the following update rules:
[0114] S31. Calculate the loss function at the current iteration time. For players and players Calculate their strategies in sequence form respectively. loss function This loss function includes not only linear terms but also a regularization term. To ensure the smoothness of the strategy:
[0115]
[0116]
[0117] in, For the profit matrix, For reference strategy, is the regularization coefficient.
[0118] S32. Calculate the counterfactual value: Backtrack from the leaf nodes of the game tree to the root node, and calculate the counterfactual value for each information set. In action Counterfactual values on This value is composed of the current loss and the expected counterfactual value of the child nodes:
[0119]
[0120] in, This represents the set of sub-information sets reached after the execution of action a.
[0121] S33. Calculate instantaneous counterfactual regret: Based on the counterfactual value, calculate the current moment. In information set The instantaneous counterfactual regret :
[0122]
[0123] This step quantifies the immediate gap between the current strategy and the optimal response.
[0124] S34. Update cumulative counterfactual regrets and predict cumulative counterfactual regrets: Utilize instantaneous regrets and step length Update accumulated counterfactual regret And the cumulative counterfactual regrets of the forecast Operations were used here. (i.e., the function that takes the maximum value) ), ensuring the regret value is non-negative:
[0125]
[0126]
[0127] S35, Update Behavioral Strategies and Disruptive Behavioral Strategies: For players Based on accumulated counterfactual regret Update behavioral strategies :
[0128]
[0129] Combined with disturbance coefficient Update the perturbation behavior strategy :
[0130]
[0131] S36. Update sequence form strategy: Let... To comply with the strategy Below, information set The probability of occurrence. ,but Information set In the probability of occurrence, the player The contributions made. (Combined with player contributions) The current information set Update players Sequence-based strategies :
[0132]
[0133] S4, Update Parameters Initialize the initial cumulative regret for the next round of the model. ,like Figure 1 As shown, specifically:
[0134] S41. Update Reference Strategy Update the reference policy for the next perturbation-regularized game using the sequential policy result from the previous perturbation-regularized game solution.
[0135]
[0136] S42. Update the regularization coefficients. Disturbance coefficient In this embodiment, the update method is set as follows:
[0137]
[0138] S43. Update and initialize cumulative counterfactual regret value: for all information sets of all players. Use the cumulative regret value from the last round of the previous perturbation-regularized game. Update the initial cumulative regret value for the next perturbation-regularized game. :
[0139] .
[0140] To demonstrate the effectiveness of the RTDCFR+ and RTPDCFR+ algorithms proposed in this invention, the following theoretical analysis is conducted from two dimensions: Last-Iterate Convergence and Average Iterate Convergence.
[0141] First, this invention will explain several key concepts, as follows:
[0142] Extensive-Form Games (EFGs) assume that the game model consists of a finite set of players. Historical Node Set Player mapping function Action Collection Information set and utility function Structure. To achieve Last-Iterate Convergence, this invention transforms the original game into Perturbed Regularized Extensive-Form Games. This framework introduces the following core components: one is the perturbation term, which introduces a perturbation factor. and reference strategy Build -Perturbation simplex ( -perturbedSimplex). Secondly, regularization: introducing a strongly convex regularization term. (in In this embodiment, Meanwhile, the problem of finding Nash equilibrium is equivalent to solving the following bilinear saddle point problem:
[0143]
[0144] Reward Transformation (RT): Defines the loss function after the transformation. In this embodiment, it is set as a two-player zero-sum game, then the... The loss function of the wheel can be calculated using the following formula:
[0145]
[0146]
[0147] Based on theoretical analysis, the RTDCFR+ algorithm can be considered a special case of the RTPDCFR+ algorithm. In this embodiment, it is assumed that the first... In the cycle, the predicted instantaneous counterfactual regret is... In this step, if the instantaneous counterfactual regret is set to 0, the RTPDCFR+ algorithm degenerates into the RTDCFR+ algorithm. Below, we will only provide specific explanations of the relevant properties of the RTPDCFR+ algorithm; the properties of the RTDCFR+ algorithm can be derived from the properties of the RTPDCFR+ algorithm.
[0148] The core improvement of this invention lies in overcoming the limitation of the traditional CFR algorithm, which can only guarantee "average strategy convergence," and proving that this algorithm has asymptotic final-round iterative convergence in perturbation-regularized extended games. This means that as the number of iterations increases... The final strategy output by the algorithm It itself converges to Nash equilibrium.
[0149]
[0150] Theorem 1 shows that the RTPDCFR+ algorithm can handle arbitrary initialization parameters. and step length All configurations are robust, guaranteeing convergence without the need for fine-tuning of parameters. Furthermore, RTDCFR+, as a special case of RTPDCFR+ (i.e., the version without optimistic prediction terms), naturally also possesses this convergence property.
[0151] Although this invention focuses on solving the final convergence problem, in order to compare with traditional algorithms, this invention also provides a regret bound analysis for average iterative convergence, as shown in Theorem 2.
[0152]
[0153] The theory guarantees the effectiveness of strategy combinations in the form of weighted average sequences. With rate It converges to the Nash equilibrium of the perturbation-regularized extended game. Furthermore, when initializing... At that time, the aforementioned regret bound is restored to the PDCFR+ algorithm for the original extended game ( The regret of the world.
[0154] This invention proves that the RTDCFR+ and RTPDCFR+ algorithms exhibit asymptotic final-round convergence in perturbation-regularized extended games. This means that the final policy output by the algorithm at termination directly converges to a Nash equilibrium. Compared to the traditional CFR algorithm (which requires storing historical policies and averaging), this invention significantly reduces memory storage overhead, allowing the algorithm to directly output the final policy for decision-making, greatly improving its engineering practicality.
[0155] This invention proves the asymptotic final iteration convergence of the algorithm for any initialized parameters. and step length With all settings in place, the algorithm deployment process is greatly simplified for engineering implementation, eliminating the need to adjust parameters for different poker games. and The debugging process improved the algorithm's versatility.
[0156] This invention significantly improves the convergence speed while maintaining theoretical convergence. According to... Figure 3 It is evident that the algorithms of this invention (RTDCFR+ and RTPDCFR+) exhibit significantly faster convergence speeds than existing advanced algorithms such as PCFR+, Reg-CFR, and DCFR in various game environments (including LeducPoker, Goofspiel, Battleship, etc.).
[0157] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0158] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A method for solving large-scale poker games based on reward function transformation and regret discount, applied to large-scale poker games, characterized in that: Includes the following steps: S1. Parameter Initialization: Set the number of perturbation-regularized extended game models to be built. Solving for the number of iterations in the perturbation-regularized extended game model Regularization coefficient Disturbance coefficient and reference strategy Initialize the cumulative counterfactual regret of the first perturbation-regularized game model and step length ; If an optimistic solution strategy is adopted, an additional discount factor is set. ; S2. Construct a perturbation-regularized extended game model: based on current parameters and Construct the first A perturbation-regularized extended game model is constructed, and the next model is constructed only after the previous model has been solved. S3. Iteratively solve the current perturbation regularized game model: for each internal time step For each information set in the game tree and its actions Perform the following update rules: Calculate player strategy in sequence form loss function Calculate the counterfactual value by backtracking from the leaf nodes of the game to the root node. Calculating instantaneous counterfactual regret based on counterfactual values By utilizing instantaneous counterfactual regret and combining it with regret discounting techniques, cumulative counterfactual regret is updated. ; If an optimistic solution strategy is adopted, the cumulative counterfactual regrets of the predictions will be updated synchronously. Then update the behavioral strategy based on accumulated counterfactual regret. Combined with the disturbance coefficient Update perturbation behavior strategy Update the perturbation sequence form policy based on the perturbation behavior policy. ; S4. Update parameters and initialize the next round of the model: Whenever a perturbation-regularized game model is solved, update the parameters. The system initializes the initial cumulative regret of the next round model using the cumulative regret value of the last round of the previous round model, and continues to build and solve N models until the final sequence form strategy is output as the Nash equilibrium strategy of the poker game.
2. The method for solving large-scale poker games based on reward function transformation and regret discount according to claim 1, characterized in that, In step S1, the cumulative counterfactual regret of the first perturbation-regularized game model is initialized. =0, step size The value is 1.
3. The method for solving large-scale poker games based on reward function transformation and regret discount according to claim 1, characterized in that, In step S2, the perturbation-regularized extended game model is constructed as follows: in, These represent the sequential strategies of the two players. For setting reference strategies, Let Bregman divergence be the property of the divergence. .
4. The method for solving large-scale poker games based on reward function transformation and regret discount according to claim 1, characterized in that, In step S3, the loss function is calculated as follows: At the current iteration time Calculate the strategy in sequence form for each of the two players. loss function and The calculation formula is as follows: in, For the profit matrix, For reference strategy, is the regularization coefficient.
5. The method for solving large-scale poker games based on reward function transformation and regret discount according to claim 1, characterized in that, In step S3, the counterfactual value is calculated. The formula is as follows: in, This indicates the players that can be reached after performing action 'a'. The set of sub-information sets.
6. The method for solving large-scale poker games based on reward function transformation and regret discount according to claim 1, characterized in that, In step S3, the instantaneous counterfactual regret is calculated. The formula is as follows: in, Information set The counterfactual value, For players in the information set The current behavioral strategy.
7. The method for solving large-scale poker games based on reward function transformation and regret discount according to claim 1, characterized in that, In step S3, the maximum value function is used to ensure that the cumulative counterfactual regret is non-negative, and the update formula is: in, As a regret discount factor, = Ensure the regret value is non-negative; If an optimistic solution strategy is adopted, the update formula for predicting cumulative counterfactual regrets is: 。 8. The method for solving large-scale poker games based on reward function transformation and regret discount according to claim 1, characterized in that, In step S3, the behavioral strategy is updated based on the accumulated counterfactual regret. Specifically: Combined with disturbance coefficient Update behavioral strategies : in, , For information set The number of elements in the action set.
9. The method for solving large-scale poker games based on reward function transformation and regret discount according to claim 1, characterized in that, In step S3, the perturbation sequence form strategy is updated. Specifically: in, In order to comply with the disturbance behavior policy At that time, the player Information set The contribution value of the probability of occurrence.
10. The method for solving large-scale poker games based on reward function transformation and regret discount according to claim 1, characterized in that, In step S4, refer to the strategy The update method is That is, the reference strategy for the next model is updated using the sequential strategy obtained from the current model solution; regularization coefficient and disturbance coefficient A non-incremental decay strategy is adopted, eventually approaching 0, while the cumulative regret value in the last round of the previous model is used. As the initial cumulative regret value for the next model .