Online high-performance processing method and device based on expected value estimation in incomplete information game

Abstract

The invention relates to the field of artificial intelligence, in particular to an online high-performance processing method and device based on expected value estimation in an incomplete information game. The method comprises the following steps: acquiring a to-be-solved sub game; calculating respective target expected values of a plurality of information sets in the sub-games, wherein the target expected values are used for indicating the expected benefits of the game participants for adopting target decisions under the information sets; pruning and / or abstracting the sub-games according to the target expected values of the multiple information sets; and according to the processed game tree, adopting a preset game algorithm to solve and obtain a target equalization strategy corresponding to the sub-game. According to the embodiment of the invention, the online high-performance pruning and / or abstraction technology based on expected value estimation is designed in the incomplete information game, the target balancing strategy can be generated for any sub-game scene within several seconds, and the requirement of online dynamic strategy generation is met.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to an online high-performance processing method and device based on expected value estimation in incomplete information games. Background Art

[0002] Game algorithms are widely used in situations where strategy optimization and decision making are required, such as war games, market transactions, and board games. Incomplete information games provide a powerful framework for analyzing games with hidden information and multiple participants. In the field of incomplete information games, especially in complex games such as Texas Hold'em and online futures trading, the development of algorithms is crucial. The Counterfactual Regret Minimization (CFR) algorithm and its variants, such as Discounted CFR (DCFR) and DCFR+, have become the main tools for solving such problems. These algorithms continuously optimize strategies through an iterative process to approach Nash equilibrium. However, since the computational overhead of the CFR algorithm is proportional to the size of the game tree, efficiency remains a challenge when dealing with large-scale games with extended forms of incomplete information. For example, the number of nodes in the game tree of Heads-Up No-Limit Texas Hold'em (HUNL) with 200 big blinds is about 10 165 indivual.

[0003] To improve efficiency, researchers have adopted processing methods such as pruning and abstraction. These methods make it possible to quickly solve large-scale games by reducing the number of nodes in the game tree. For example, through information abstraction, similar information sets are merged to reduce the amount of calculation; action abstraction constructs the game tree by selecting representative actions, further reducing the size of the tree.

[0004] However, current processing methods often rely on intermediate calculated values ​​during the CFR iteration process, resulting in dynamic and temporary processing, and these processing methods are complex and need to be adjusted according to CFR variants. In addition, current methods can usually only solve problems under specific initial scenarios (environments) and require a long time to calculate strategies under specific scenarios. In actual games, the initial scenarios are changing, and current methods cannot efficiently cope with such changes. In addition, when resources and time are limited, current methods have difficulty in obtaining solutions that approximate Nash equilibrium. In reality, it is necessary to obtain an approximate solution to any situation within a few seconds, and current methods are difficult to meet this requirement. Summary of the invention

[0005] In view of this, the present disclosure proposes an online high-performance processing method and device based on expected value estimation in incomplete information games.

[0006] According to one aspect of the present disclosure, there is provided an online high-performance processing method based on expected value estimation in an incomplete information game, the method comprising:

[0007] Obtain a subgame to be solved, wherein the subgame is a game subtree starting from an arbitrary node in a game tree of an incomplete information game and ending at a plurality of arbitrary nodes, wherein the subgame includes respective information sets of a plurality of game participants at a plurality of nodes, and the information sets include game state information of the game participants when making decisions;

[0008] Calculating target expected values ​​of each of the plurality of information sets in the sub-game, wherein the target expected values ​​are used to indicate the expected benefits of the game participants taking target decisions under the information sets;

[0009] Pruning and / or abstracting the subgames according to respective target expected values ​​of the plurality of information sets;

[0010] According to the processed game tree, a preset game algorithm is used to solve and obtain the target equilibrium strategy corresponding to the sub-game, and the target equilibrium strategy is used to indicate the strategy combination of the multiple game participants in an equilibrium state.

[0011] In a possible implementation, pruning and / or abstracting the sub-games according to the target expected values ​​of the respective information sets includes:

[0012] According to the target expected values ​​of each of the plurality of information sets and the target expected values ​​of each of the plurality of sub-information sets, branches corresponding to the sub-information sets are pruned, the information set comprising a plurality of sub-information sets corresponding to different decisions; and / or,

[0013] The information sets are abstracted according to respective target expected values ​​of the plurality of information sets.

[0014] In another possible implementation, pruning branches corresponding to the sub-information sets according to the target expected values ​​of the plurality of information sets and the target expected values ​​of the plurality of sub-information sets includes:

[0015] When the target expected value of the sub-information set is lower than the target expected value of the corresponding information set, the branch corresponding to the sub-information set is pruned.

[0016] In another possible implementation, abstracting the information sets according to respective target expected values ​​of the plurality of information sets includes:

[0017] When there are at least two information sets whose respective target expected values ​​satisfy the merging condition, merging the at least two information sets, wherein the abstract processing includes merging the at least two information sets;

[0018] Among them, the target expected value of the information set includes a current expected value and a future expected value, the merging condition includes that the similarity between the current expected values ​​of at least two of the information sets is higher than a first similarity threshold, and the similarity between the future expected values ​​is higher than a second similarity threshold, the current expected value is the expected value of the current information set, and the future expected value is the expected value expected to be achieved under the future information set after taking the target decision under the current information set.

[0019] In another possible implementation, the calculating the target expected value of each of the plurality of information sets in the sub-game includes:

[0020] For each of the information sets in the sub-game, an expected value is estimated through a target model to determine a target expected value of the information set, wherein the target model is a pre-trained neural network model for estimating expected values.

[0021] In another possible implementation, the target models include multiple ones, and the target expected value of the information set includes at least one of an average value, a maximum value, and a minimum value of the expected value estimates output by each of the multiple target models.

[0022] In another possible implementation, the method is applied to an agent that is one of the players in a multiplayer Texas Hold'em game;

[0023] The game participants are players;

[0024] The game status information includes hand information and table display information, wherein the table display information includes displayed community card information, the number of chips of all players at the table, the amount of the prize pool at the table, and the historical decision information of all players in this round;

[0025] Said decisions include folding, calling, and raising.

[0026] In another possible implementation, the method is applied to an agent that is one of the participants in online market transactions;

[0027] The game participants are transaction participants;

[0028] The game status information includes private information and market public information, and the market public information includes historical transaction data, current transaction volume, transaction amount and price of multiple transaction participants;

[0029] The decisions include buying, bidding, selling and listing.

[0030] According to another aspect of the present disclosure, an online high-performance processing device based on expected value estimation in an incomplete information game is provided, the device comprising:

[0031] An acquisition module is used to acquire a subgame to be solved, wherein the subgame is a game subtree starting from an arbitrary node in a game tree of an incomplete information game and ending at a plurality of arbitrary nodes, wherein the subgame includes respective information sets of a plurality of game participants at a plurality of nodes, and the information sets include game state information of the game participants when making decisions;

[0032] A calculation module, used for calculating the target expected value of each of the plurality of information sets in the sub-game, wherein the expected value is used to indicate the expected benefits of the game participants taking the target decision under the information set;

[0033] A processing module, configured to prune and / or abstract the sub-games according to respective target expected values ​​of the plurality of information sets;

[0034] A solution module is used to solve the target equilibrium strategy corresponding to the sub-game according to the processed game tree using a preset game algorithm, and the target equilibrium strategy is used to indicate the strategy combination of the multiple game participants in an equilibrium state.

[0035] In a possible implementation, the processing module is further configured to:

[0036] According to the target expected values ​​of each of the plurality of information sets and the target expected values ​​of each of the plurality of sub-information sets, branches corresponding to the sub-information sets are pruned, the information set comprising a plurality of sub-information sets corresponding to different decisions; and / or,

[0037] The information sets are abstracted according to respective target expected values ​​of the plurality of information sets.

[0038] In another possible implementation, the processing module is further configured to:

[0039] When the target expected value of the sub-information set is lower than the target expected value of the corresponding information set, the branch corresponding to the sub-information set is pruned.

[0040] In another possible implementation, the processing module is further configured to:

[0041] When there are at least two information sets whose respective target expected values ​​satisfy the merging condition, merging the at least two information sets, wherein the abstract processing includes merging the at least two information sets;

[0042] Among them, the target expected value of the information set includes a current expected value and a future expected value, the merging condition includes that the similarity between the current expected values ​​of at least two of the information sets is higher than a first similarity threshold, and the similarity between the future expected values ​​is higher than a second similarity threshold, the current expected value is the expected value of the current information set, and the future expected value is the expected value expected to be achieved under the future information set after taking the target decision under the current information set.

[0043] In another possible implementation, the computing module is further configured to:

[0044] For each of the information sets in the sub-game, an expected value is estimated through a target model to determine a target expected value of the information set, wherein the target model is a pre-trained neural network model for estimating expected values.

[0045] In another possible implementation, the target models include multiple ones, and the target expected value of the information set includes at least one of an average value, a maximum value, and a minimum value of the expected value estimates output by each of the multiple target models.

[0046] In another possible implementation, the device is applied to an agent that is one of the players in a multiplayer Texas Hold'em game;

[0047] The game participants are players;

[0048] The game status information includes hand information and table display information, wherein the table display information includes displayed community card information, the number of chips of all players at the table, the amount of the prize pool at the table, and the historical decision information of all players in this round;

[0049] Said decisions include folding, calling, and raising.

[0050] In another possible implementation, the device is applied to an intelligent agent that is one of the participants in online market transactions;

[0051] The game participants are transaction participants;

[0052] The game status information includes private information and market public information, and the market public information includes historical transaction data, current transaction volume, transaction amount and price of multiple transaction participants;

[0053] The decisions include buying, bidding, selling and listing.

[0054] According to another aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0055] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0056] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of a computing device, the processor in the computing device executes the above method.

[0057] The online high-performance processing method based on expected value estimation in an incomplete information game provided by the embodiment of the present disclosure can obtain a sub-game to be solved, where the sub-game is a game sub-tree starting from an arbitrary node in the game tree of the incomplete information game and ending at several arbitrary nodes, and the sub-game includes respective information sets of multiple game participants at multiple nodes, and the information set includes game state information of the game participants when making decisions; calculate the target expected values ​​of each of the multiple information sets in the sub-game, and the target expected values ​​are used to indicate the expected benefits of the game participants taking target decisions under the information set; prune and / or abstract the sub-game according to the respective target expected values ​​of the multiple information sets; according to the processed game tree, use a preset game algorithm to solve and obtain the target equilibrium strategy corresponding to the sub-game, and the target equilibrium strategy is used to indicate the strategy combination of multiple game participants in the equilibrium state; that is, by designing an online high-performance pruning and / or abstraction technology (named EVPA), it is possible to generate a target equilibrium strategy for any sub-game scenario within a few seconds, which meets the needs of online dynamic strategy generation and is particularly suitable for real-time decision-making environments. Moreover, this method performs pruning and abstraction before applying the CFR algorithm, which avoids the dependence of traditional pruning technology on intermediate calculated values ​​in the CFR iteration process, reduces the complexity of calculation, and improves the stability and reliability of the algorithm.

[0058] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0060] Figure 1A schematic diagram of the structure of a computing device provided by an exemplary embodiment of the present disclosure is shown.

[0061] Figure 2 A flowchart of an online high-performance processing method based on expected value estimation in an incomplete information game provided by an exemplary embodiment of the present disclosure is shown.

[0062] Figure 3 A flowchart of an online high-performance processing method based on expected value estimation in an incomplete information game provided by another exemplary embodiment of the present disclosure is shown.

[0063] Figure 4 A flowchart of an online high-performance processing method based on expected value estimation in an incomplete information game provided by another exemplary embodiment of the present disclosure is shown.

[0064] Figure 5 A schematic diagram showing test results of performance comparison of different AI strategies in the river stage of No-Limit Texas Hold'em provided by an exemplary embodiment of the present disclosure is shown.

[0065] Figure 6 The invention is a block diagram of a device for online high-performance processing based on expected value estimation in an incomplete information game according to an exemplary embodiment. DETAILED DESCRIPTION

[0066] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0067] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0068] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0069] In the field of incomplete information games, pruning and abstraction techniques can significantly reduce the number of nodes in the game tree, making it possible to quickly solve large incomplete information extended form games. Generally speaking, we need to reduce the size of the game tree to 10 7In this way, the CFR algorithm can find an approximate Nash equilibrium strategy within a few seconds. In addition to pruning and abstraction techniques, depth-limited solving (English: Depth-limited Solving), subgame solving (English: Subgame Solving) and other technologies can also reduce the scale of solving the game tree. Combining pruning and abstraction techniques with these methods and using the CFR algorithm for solving is a key technology for solving large-scale incomplete information games. The time complexity of the CFR algorithm is O(T|G|), where T is the number of iterations (usually 100 to 2000) and |G| is the size of the game tree.

[0070] In complete information games, pruning has a wide range of applications, such as Minimax Pruning and Alpha-Beta Pruning. In large-scale extended form games with incomplete information, pruning is different from traditional games because such games are solved by the CFR algorithm. Since the value and arrival probability of the information set change in each iteration of the CFR, previous methods mainly perform temporary pruning based on the intermediate calculated values ​​of the CFR. Among them, regret-based pruning can avoid traversing the path caused by any player taking an action with zero probability, best response pruning can temporarily prune poorly performing actions, and dynamic threshold pruning can prune low-probability actions.

[0071] Abstraction is divided into information abstraction and action abstraction. Information abstraction puts similar information sets into a bucket and adopts the same strategy, thereby reducing computational overhead. Action abstraction selects several representative actions from a huge action space and builds a game tree based on these representative actions, thereby greatly reducing the size of the game tree. Our method mainly improves information abstraction. Next, we will introduce some mainstream information abstraction methods. Information abstraction is mainly divided into two categories: expectation-based abstraction and potential-aware abstraction. Expectation-based abstraction methods classify information sets according to their current expectation strength, while potential-aware abstraction methods consider the performance of information sets in different future situations. Imperfect recall technology allows players to forcibly ignore certain information, thereby greatly reducing the size of the game tree. The previous leading information abstraction algorithm, potential-aware imperfect recall abstraction, combines potential-aware abstraction with imperfect recall and is the key to the success of many Texas Hold'em AIs. The main application of large-scale incomplete information extended form games is poker. Based on the CFR algorithm, many Texas Hold'em AIs with outstanding performance have been born in recent years.

[0072] However, current approaches to dealing with incomplete information games have many flaws. In terms of pruning techniques, current pruning methods often rely on intermediate calculated values ​​during the CFR iteration process, which leads to dynamic and temporary pruning. In addition, these methods are complex and require tailored adjustments based on the CFR variant used. For example, methods such as DCFR and DCFR+ have better performance than the original CFR, but previous pruning methods are very difficult to implement on these CFR variants and the results are mediocre. In addition, in the initial iteration stage, the size of the game tree remains unchanged, resulting in no reduction in memory overhead.

[0073] In terms of information abstraction, expectation-based abstractions often ignore the future evolution of the information set, resulting in poor abstraction effects; potential-aware abstractions require a large number of simulations and clustering, resulting in significant computational overhead, and it often takes several days to calculate an abstract solution for the entire game. In addition, when using subgame solving techniques, previous methods often use the same abstraction in different subgames, which may not be optimal. For example, in poker, if players all choose to check to the river, holding a bottom pair may be the best hand; but if there are multiple raises before the river, holding a bottom pair is unlikely to be the best hand. These methods also tend to focus only on the strength of the information set, without considering the blocking effect, which is particularly important in poker because the opponent cannot hold the cards we have.

[0074] In terms of Texas Hold'em AI, previous Texas Hold'em AI can often only solve a certain number of chip depths, and it takes several days to calculate the abstract and blueprint strategies under certain specific chip depths. However, in real poker games, the amount of chips of players will change, and the previous AI using information abstraction cannot deal with this situation efficiently. In addition, when computing resources and time are limited, it is difficult for previous AI to obtain an approximate Nash equilibrium solution. However, if Texas Hold'em AI is to be applied to reality, it is necessary to obtain an approximate solution for any situation within a few seconds, which requires that our pruning, abstraction and CFR solving steps must be completed within a few seconds, which is difficult for previous AI to do.

[0075] In order to solve the deficiencies in the above-mentioned technologies, the embodiments of the present disclosure design an online high-performance pruning and / or abstraction technology (named EVPA) to solve the temporary, universal and abstract huge computational overhead, fixity and blocking effect of pruning. Through EVPA technology, an approximate equilibrium solution, i.e., the target equilibrium strategy, can be obtained for any chip depth and any sub-game in poker games such as Texas Hold'em, Short Deck, and Omaha within a few seconds, making it possible to generate online dynamic strategies for large-scale games with incomplete information extension forms, which has broad application prospects and practical value.

[0076] In the disclosed embodiment, an online high-performance processing method based on expected value estimation in incomplete information games is proposed. The method is particularly suitable for online game environments that require real-time strategy generation. In order to achieve this goal, high-performance computing devices are involved, and the computing devices constitute the execution subject of the method. Please refer to Figure 1 , which shows a schematic diagram of the structure of a computing device provided by an exemplary embodiment of the present disclosure.

[0077] The computing device can be a server or a terminal. The terminal includes a mobile terminal or a fixed terminal, such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc. The server can be a server, a server cluster consisting of several servers, or a cloud computing service center.

[0078] The computing device includes a processor 10, a memory 20, and a communication interface 30. Those skilled in the art will appreciate that Figure 1 The structure shown in the figure does not constitute a limitation on the computing device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0079] The processor 10 is the control center of the computing device. It connects the various parts of the entire computing device using various interfaces and lines, and executes various functions of the computing device and processes data by running or executing software programs and / or modules stored in the memory 20, and calling data stored in the memory 20, thereby controlling the computing device as a whole. The processor 10 can be implemented by a CPU or a GPU, and can handle complex computing tasks and provide users with a smooth gaming experience.

[0080] The memory 20 can be used to store software programs and modules. The processor 10 executes various functional applications and data processing by running the software programs and modules stored in the memory 20. The memory 20 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system 21, an acquisition module 22, a calculation module 23, a processing module 24, a solution module 25 and an application 26 required for at least one function, etc.; the data storage area may store data created according to the use of the computing device, etc. The memory 20 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk. Accordingly, the memory 20 may further include a memory controller to provide the processor 10 with access to the memory 20 .

[0081] The processor 10 executes the following functions by running the acquisition module 22: obtaining a sub-game to be solved, wherein the sub-game is a game sub-tree starting from an arbitrary node in the game tree of the incomplete information game and ending at several arbitrary nodes, and the sub-game includes respective information sets of multiple game participants at multiple nodes, and the information set includes game state information of the game participants when making decisions; the processor 10 executes the following functions by running the calculation module 23: the calculation module is used to calculate the target expected values ​​of the multiple information sets in the sub-game, and the expected values ​​are used to indicate the expected benefits of the game participants taking target decisions under the information sets; the processor 10 executes the following functions by running the processing module 24: pruning and / or abstracting the sub-game according to the respective target expected values ​​of the multiple information sets; the processor 10 executes the following functions by running the solution module 25: according to the processed game tree, using a preset game algorithm to solve and obtain the target equilibrium strategy corresponding to the sub-game, and the target equilibrium strategy is used to indicate the strategy combination of the multiple game participants in the equilibrium state.

[0082] With the support of hardware devices, the online high-performance processing method based on expected value estimation in incomplete information games provided by the embodiments of the present disclosure can provide real-time strategy generation for online games, and its execution subject can be any intelligent system that needs to make strategy decisions in an incomplete information environment. Through this method, the system can make efficient strategy decisions in an incomplete information environment, thereby providing players with a richer and more challenging gaming experience. Whether on a mobile device or on a cloud computing platform, the system can ensure the smooth operation of the game and the real-time update of the strategy, meeting the high-performance requirements of modern online games.

[0083] Below, several exemplary embodiments are used to introduce the online high-performance processing method based on expected value estimation in the incomplete information game provided by the embodiments of the present disclosure.

[0084] Please refer to Figure 2 , which shows a flowchart of an online high-performance processing method based on expected value estimation in an incomplete information game provided by an exemplary embodiment of the present disclosure. The method includes the following steps.

[0085] Step 201, obtaining a sub-game to be solved, where the sub-game is a game sub-tree starting from an arbitrary node in the game tree of the incomplete information game and ending at several arbitrary nodes, and the sub-game includes respective information sets of multiple game participants at multiple nodes, and the information sets include game state information of the game participants when making decisions.

[0086] An incomplete information game is a type of game in game theory where at least one of the game players has incomplete knowledge of the state of the game, i.e. they do not know the payoff functions or action spaces of all other game players. This is in contrast to a perfect information game, which assumes that all game players have perfect knowledge of all aspects of the game.

[0087] A game tree is a graphical representation used to describe all possible paths through a game with sequential decisions. It starts with a root node, and each node represents a decision point, which represents a point at which a player needs to make a decision. Each decision point corresponds to one or more possible actions for a player in the game. Each branch represents a possible action, and the terminal node represents the outcome of the game.

[0088] A subgame is a part of the game tree, starting from the current node and ending at several arbitrary nodes, and including a subsequent game process starting from the current node. "Several" means one or more. The concept of subgame is particularly important in dynamic games because it allows analyzing the best strategy after a certain history.

[0089] Game participants are individuals or entities that participate in the game, make decisions, and receive benefits based on the outcome of the game.

[0090] The information set is a concept in game theory, which represents the set of different historical states that game participants cannot distinguish at a specific decision node due to lack of information. Within the information set, participants cannot distinguish different historical paths, so they must adopt the same strategy to deal with all situations in the information set.

[0091] The information set includes the game state information of the game participants at the time of decision-making. Game state information refers to the configuration of the game at a specific moment, including the action history of all participants, the current payoff situation, and all other relevant information. This information is crucial for game participants to make decisions.

[0092] Step 202, calculate the target expected value of each of the multiple information sets in the sub-game, where the target expected value is used to indicate the expected benefits of the game participants taking the target decision under the information set.

[0093] For each information set in the subgame, the target expected value of each information set is calculated by a preset algorithm. The target expected value is used to indicate the expected benefits of the game participants taking the target decision under the information set. It reflects the expected benefits that the game participants can obtain by taking a certain decision under a specific information set.

[0094] The target expected value is used to indicate the expected benefits of the game participants taking the target decision under the information set. The target decision refers to the action chosen by the game participants under the information set. Different decisions will affect the progress and final result of the game. The target decision can be the decision with the largest expected value among multiple decisions. The expected benefit can be the benefit value calculated based on the probabilities of various possible results after the game participants take the target decision. It is an important indicator for evaluating the quality of decisions.

[0095] It should be noted that the relevant details of the method for calculating the target expected value in the embodiment of the present disclosure can be referred to the relevant description in the following embodiments and will not be introduced here.

[0096] Step 203: prune and / or abstract the sub-games according to the target expected values ​​of the multiple information sets.

[0097] In the embodiments of the present disclosure, before using a preset game algorithm to solve the target equilibrium strategy, the sub-games may be pruned and / or abstracted based on the expected estimation.

[0098] Pruning is an optimization technique used to reduce the search space of a game tree. Pruning can improve computational efficiency by avoiding the evaluation of nodes that clearly do not affect the final decision. In a game tree, pruning usually involves estimating upper and lower bounds on the value of a node. If the value of a node is determined to be no better or worse than what is known, then the node and its subtree can be pruned.

[0099] Abstract processing is a technique for simplifying the game tree, which reduces the complexity of the game tree by merging similar nodes or states. This processing method helps to reduce the complexity of calculations, allowing the algorithm to handle larger-scale game problems in a limited time.

[0100] It should be noted that the details of pruning and / or abstract processing in the embodiments of the present disclosure can be referred to the relevant description in the following embodiments and will not be introduced here.

[0101] Step 204: Based on the processed game tree, a preset game algorithm is used to solve and obtain a target equilibrium strategy corresponding to the sub-game. The target equilibrium strategy is used to indicate a strategy combination of multiple game participants in an equilibrium state.

[0102] The preset game algorithm is an algorithm that has been determined before the game analysis is performed, and is used to solve the target equilibrium strategy corresponding to the subgame. These algorithms may include but are not limited to the CFR algorithm and its variants, which are the basic tools for solving incomplete information game problems.

[0103] The target equilibrium strategy is the strategy combination of multiple game participants in the equilibrium state in a multi-player game. The target equilibrium strategy is obtained by solving the processed game tree using a preset game algorithm, and is used to indicate the optimal strategy that each game participant should adopt in the equilibrium state. In incomplete information games, the target equilibrium strategy takes into account the incompleteness of information and strives to find the optimal strategy combination under such conditions. This strategy combination satisfies the conditions of Nash equilibrium, that is, given that the strategies of other game participants remain unchanged, a single game participant cannot obtain a higher expected return by changing his own strategy. In other words, the strategy of each game participant is the best response to the strategies of other game participants.

[0104] In some embodiments, the method is applied to an agent that is one of the players in a multiplayer Texas Hold'em game; the game participants are players; the game state information includes hand information and table display information, the table display information includes the displayed public card information, the number of chips of all players at the table, the amount of the prize pool at the table, and the historical decision information of all players in this round; the decision includes folding, calling, and raising. The target equilibrium strategy can guide the agent to make the best decision in different game states, such as deciding whether to fold, call, or raise at the river stage.

[0105] In an exemplary application scenario, the software (i.e., the intelligent agent) that executes the method can run on a game server and act as one of the players. The other players can be real users, forming a "man-versus-machine" game scenario. The chips involved in the game can be counted in the form of points, real players can obtain virtual point rewards, and players can be ranked according to their scores.

[0106] This solution is not only applicable to Texas Hold'em, but also to a variety of large-scale incomplete expansion forms of games such as short deck, Omaha, poker tournaments, mahjong, egg throwing, landlords, etc., and can also be applied to online market transactions, strategic deduction and other scenarios. In some embodiments, the method is applied to an intelligent agent that is one of the participants in online market transactions; the game participants are transaction participants; the game state information includes private information and market public information, and the market public information includes historical transaction data of multiple transaction participants, current transaction volume, transaction amount and price; decisions include buying, buying, selling and listing. This disclosed embodiment is not limited to this.

[0107] In summary, the online high-performance processing method based on expected value estimation in incomplete information games provided by the embodiments of the present disclosure significantly improves the efficiency and performance of processing large-scale incomplete information extended form games by designing an online high-performance pruning and / or abstraction technology (named EVPA). The following are the beneficial effects of this method: 1. Fast response: EVPA technology can generate target equilibrium strategies for any sub-game scenario within a few seconds, meeting the needs of online dynamic strategy generation, and is particularly suitable for real-time decision-making environments. 2. Reduced dependence: This method performs pruning and abstraction before applying the CFR algorithm, avoiding the dependence of traditional pruning technology on intermediate calculated values ​​in the CFR iteration process, reducing the complexity of calculation, and improving the stability and reliability of the algorithm. 3. Estimation of expected value of information set: By calculating the target expected value of each information set in the sub-game, a more accurate decision-making basis is provided for game participants, and the predictability and adaptability of the strategy are enhanced. 4. Pruning and abstraction technology: EVPA combines pruning technology based on expected value and abstraction technology designed for sub-games, effectively reducing the size of the game tree and reducing the consumption of memory and computing resources. 5. Generation of equilibrium strategies: Through the processed game tree, the target equilibrium strategy corresponding to the sub-game can be solved, providing a strategy combination in the equilibrium state for multiple game participants, enhancing the practicality of the strategy. 6. Improved user experience: In application scenarios that require quick decision-making, such as online games, online market transactions, etc., this solution can provide a smoother and more responsive experience and enhance user satisfaction. 7. Promote industry development: The development and application of EVPA technology is expected to promote research and application in the field of incomplete information games, especially in areas that require real-time strategy generation, such as online games, transactions, negotiations, strategic deductions, etc. Therefore, the EVPA technology provided in the embodiments of the present disclosure, through its efficient pruning and abstract processing, can obtain a target equilibrium strategy for any sub-game scenario within a few seconds, making it possible to generate online dynamic strategies for large-scale incomplete information extended form games, and has broad application prospects and practical value.

[0108] Below, the three parts involved in EVPA technology are further introduced, which are: expected value estimation of information set, pruning based on expected value, and information abstraction designed for sub-games.

[0109] The first part is the estimation of the expected value of the information set, which is to calculate the target expected value of each of the multiple information sets in the sub-game.

[0110] In order to estimate the target expected value of various information sets in incomplete information games, a framework for calculating depth-restricted subgame equilibrium strategies can be used, similar to the method used by DeepStack. It estimates the target expected value through a pre-trained neural network model. These neural network models can be multiple, and the expected value estimates output by each of them can be used to calculate the target expected value of the information set, which may include the average, maximum, or minimum of these expected value estimates.

[0111] The pre-training process of the target model involves the following steps: 1. Sampling diverse sub-games: This process starts with a random initial public state and generates a depth-restricted sub-game. This is done to ensure that the sample can cover a wide range of possible game states. 2. Using randomized action abstractions: When constructing sub-games, randomized action abstractions are used to increase the diversity of samples, thereby improving the generalization ability of the model. 3. Calculating Nash equilibrium strategies and expected values: In each sub-game, the Nash equilibrium strategy is calculated and the expected value of each information set is estimated. These expected values ​​are then used as training data. 4. Continue sampling leaf nodes: If a leaf node of a sub-game is not a terminal state, the sub-game with that leaf node as the root will continue to be solved to further enrich the training data.

[0112] In some embodiments, multiple independent neural network models are trained, each of which is used to perform expected value estimation. The following is a further description of this process: 1. Model input: Each neural network model receives a high-dimensional feature vector as input. This feature vector is used to describe the state of the information set, including all relevant game information, such as the player's chips, the state of the community cards, possible actions, etc. 2. Network architecture: These neural network models are composed of multi-layer perceptrons (MLP), which are feed-forward neural networks consisting of multiple layers, each containing multiple neurons. This architecture allows the model to learn complex nonlinear relationships and thus estimate the expected value more accurately. 3. Model output: The output of each neural network model is a scalar value representing the expected value of the corresponding information set. This value is relative to the pot size, which means that it represents the player's expected return relative to the pot under this information set. 4. Training process: These neural network models can be trained using the Adam optimizer. Adam is an adaptive learning rate optimization algorithm that can automatically adjust the learning rate to accelerate the training process and improve the performance of the model. 5. Model independence: Each model is trained independently, which means that they can capture different features and patterns, thereby providing diverse expected value estimates. This diversity helps improve the robustness and accuracy of the final estimate.

[0113] After the training is completed, the expected value estimation networks corresponding to the multiple neural network models are obtained. Each neural network model corresponds to an expected value estimation network, and each expected value estimation network can provide an expected value estimation for each information set. In order to obtain a comprehensive expected value representation, we calculated the target expected value of each information set as follows:

[0114] 1. Average expected value estimation: Calculate the average expected value estimation of all expected value estimation networks for the information set. The formula is: Where M is the number of expected value estimation networks, I p is the information set, EV i (I p ) is the expected value estimate of the information set by the i-th expected value estimation network. The value of i ranges from 1 to M, and i is a positive integer. This average value provides a measure of the central tendency and reflects the consensus of the neural network model on the expected value of the information set.

[0115] 2. Maximum expected value estimation: Calculate the expected value estimation network for the information set I p The maximum value among the expected value estimates is: EV max (I p )=max{EV1(I p ),EV2(I p ),…,EV M (Ip )}, this maximum gives the most optimistic estimate of the expected value, indicating the potential value of the information set under the most favorable circumstances.

[0116] 3. Minimum expected value estimation: Calculate the expected value estimation network for the information set I p The minimum value among the expected value estimates is: EV min (I p )=min{EV1(I p ),EV2(I p ),…,EV M (I p )}This minimum value provides a conservative estimate of the expected value, reflecting the potential value of the information set under the worst-case scenario. By calculating these three values, a comprehensive range of expected values ​​is provided for each information set, which helps to consider uncertainty and risk in the subsequent decision-making process. These values ​​will be used in the pruning and abstraction components to optimize the decision-making process and improve the effectiveness of the overall strategy.

[0117] The second part is pruning based on expected value, that is, pruning the sub-games according to the target expected values ​​of multiple information sets.

[0118] Expected value based pruning is a technique to reduce the amount of computation in incomplete information games. The core idea of ​​this method is to identify and exclude actions that have low expected values ​​relative to other actions, thereby simplifying the game tree and improving computational efficiency. That is, if there are other actions with higher expected values ​​for a certain information set, the optimal player will not choose the current action. For example, in No Limit Texas Hold'em (HUNL), folding a pair of Aces before the flop is suboptimal compared to calling or raising, because the latter has a higher expected value. The following is a further introduction to the pruning method:

[0119] 1. Computational overhead problem: Accurately calculate each information set I in the subgame pand the expected value of action a are computationally very expensive. To address this problem, EVPA uses expected value estimates from multiple expected value estimation networks and prunes them before applying the CFR algorithm. In some embodiments, branches corresponding to suboptimal sub-information sets are pruned according to the target expected values ​​of each of the multiple information sets and the target expected values ​​of each of the multiple sub-information sets, and the information set includes multiple sub-information sets, and the multiple sub-information sets correspond to different decisions. In an incomplete information game, an information set is a set of game states that are indistinguishable from the perspective of the game participants. That is, the game participants cannot determine which specific state they are in this information set. When game participants make decisions in a certain information set, they may obtain more information, which enables them to distinguish different states in the information set. These more specific information states constitute sub-information sets. A sub-information set is a subset of an information set that represents more detailed private information that the game participants have at a specific decision point. For example, in a poker game, the information set of the game participants before the flop includes all possible combinations of hole cards because they do not yet know what the public cards are. After the flop, the game participants can distinguish different game states based on the public cards they see and their own hole cards, and they are now in a sub-information set.

[0120] 2. Pruning conditions: When the target expected value of a sub-information set is lower than the target expected value of the corresponding information set, the branch corresponding to the sub-information set is pruned. The target expected value of the sub-information set can be one of the maximum value, minimum value and average value among the multiple expected value estimates of the sub-information set, and the target expected value of the information set can be one of the maximum value, minimum value and average value among the multiple expected value estimates of the information set. In some embodiments, when the maximum value among the multiple expected value estimates of the sub-information set is lower than the minimum value among the multiple expected value estimates of the corresponding information set, the branch corresponding to the sub-information set is pruned. Alternatively, when the sum of the maximum value among the multiple expected value estimates of the sub-information set and the preset value is lower than the minimum value among the multiple expected value estimates of the corresponding information set, the branch corresponding to the sub-information set is pruned. That is, when establishing a sub-game, for the information set I belonging to the current action player p, p and action a, if the following conditions are met:

[0121] EV max (I p ·a)+δ <EV min (I p )

[0122] Among them, δ is a small constant. This condition means that if an action a is in the information set I p The maximum expected value estimate under the condition of adding a small constant is still less than the current information set I p The minimum expected value estimate, then it can be considered that player p is in the information set Ip Taking action a is suboptimal.

[0123] 3. Pruning operation: In this case, it can be determined that the expected value of the sub-information set is lower than the current information set, so this branch of the sub-information set can be pruned from the game tree. This helps to reduce the number of branches that need to be further calculated, thereby reducing the computational complexity.

[0124] The computational complexity of the pruning algorithm provided by the embodiment of the present disclosure can be estimated by the following formula: O(MD|I|+M|S||I|), where M is the number of expected value estimation networks. D is the current depth of the subgame, which represents the path length from the root node to the current node. |S| is the number of common nodes in the subgame, which are nodes of the game state that can be observed by all game participants. |I| is the number of different information sets for each node in the subgame. Compared with the CFR algorithm, the computational overhead of the pruning algorithm is negligible. The CFR algorithm needs to iterate the entire game tree and calculate the strategy and counterfactual regret value of each information set, and its computational complexity is usually much higher than that of the pruning algorithm. The pruning algorithm significantly reduces the amount of calculation by reducing the number of branches that need to be calculated, making it possible to apply it in large-scale incomplete information games. This pruning method not only improves computational efficiency, but also helps to achieve faster strategy iteration and optimization in practical applications.

[0125] The third part is the information abstraction designed for sub-games, that is, abstracting sub-games according to the target expected values ​​of multiple information sets.

[0126] Information abstraction is a technique used in game theory to simplify complex game trees, especially in incomplete information games. The information abstraction method of EVPA technology achieves this by clustering the current and future expected values ​​of information sets. The following is a further introduction to the abstract processing method: 1. Core idea: If the current and future expected values ​​of two information sets are similar, then they can be considered to be strategically equivalent, so they can be merged. This merger can reduce the number of information sets that need to be considered independently, thereby simplifying the game tree. 2. Clustering process: By calculating the expected value of each information set and clustering these expected values ​​as features. Clustering algorithms (such as K-means, hierarchical clustering, etc.) can group information sets into different clusters based on the similarity of expected values. 3. Abstraction of subgames: For the information sets Ip and Ip′ on the two root nodes in the subgame, if their expected values ​​corresponding to each common node of the subgame are relatively close, then the two information sets can be merged into a bucket under this subgame. This means that in the context of the subgame, the two information sets can be regarded as a single entity. 4. Comparison of expected values: When comparing the expected values ​​of information sets, one can consider the current expected value as well as the expected values ​​that may be in the future. This may involve predictions of future game states and estimates of the expected values ​​of those states. 5. Merging information sets: By merging similar information sets, the complexity of subgames can be reduced. This merging not only reduces the number of information sets that need to be considered, but also reduces the number of strategy combinations that need to be calculated.

[0127] In some embodiments, the abstract processing is information abstract processing, and the abstract processing includes merging at least two information sets. When there are at least two information sets whose respective target expected values ​​meet the merging condition, the at least two information sets are merged; wherein the target expected value of the information set includes the current expected value and the future expected value, the current expected value is the expected value of the current information set, that is, the current expected value is the expected benefit when the target decision is adopted under the current information set, and the future expected value is the expected value expected to be achieved under the future information set after the target decision is adopted under the current information set.

[0128] The merging conditions include that the similarity between the current expected values ​​of at least two information sets is higher than the first similarity threshold, and the similarity between the future expected values ​​is higher than the second similarity threshold. The similarity between the current expected values ​​needs to be higher than the first similarity threshold, which means that if the current expected values ​​of the two information sets are close enough, they can be considered similar. The similarity between the future expected values ​​needs to be higher than the second similarity threshold, which ensures that not only the current decisions, but also the expectations for the future are similar. This merging is based on the similarity of the current and future expected values, which helps to simplify the game tree without losing too much strategy quality.

[0129] The first similarity threshold and the second similarity threshold are preset, and may be a default setting or a custom setting. The first similarity threshold and the second similarity threshold may be the same value or different values. This embodiment of the present disclosure does not limit this.

[0130] The abstraction process described in the embodiments of the present disclosure is performed on subgames, aiming to simplify the game tree through pruning and clustering, thereby reducing computational complexity. The process may include but is not limited to the following steps:

[0131] 1. Pruning root node information set:

[0132] The branches corresponding to the multiple information sets on the root node of the sub-game are pruned and judged. The multiple information sets on the root node of the sub-game are also called root node information sets, including multiple information sets on the path from the initial state of the game to the root node of the current sub-game. This step is to remove those information sets with low expected values ​​that are unlikely to be selected by the optimal strategy.

[0133] 2. Enumerate common nodes and estimate expected values:

[0134] Enumerate each common node s of the subgame and estimate if the root node information set I p The expected value EV when reaching these public nodes s avg (I p →s). That is, traverse all possible common nodes in the subgame. In Texas Hold'em, common nodes may refer to stages such as flop, turn or river, and the information of these stages is visible to all players. For each common node, estimate the expected value when the player reaches the common node if he starts from the root node information set. This expected value can be determined by a preset trained neural network model.

[0135] 3. Calculate significance and add features:

[0136] The estimated expected value is multiplied by a significance function and attached to the features of the root node information set. The significance function is a mathematical expression that adjusts the estimated expected value based on specific features of the common node and the distance from the root node to the common node. For example, the significance function is f(g(s),n), and the function g(s) contains various information about the common node s, which may include the current pot size, the number of chips a player has, the state of the public cards, etc. And n represents the distance from the root node to the node s, which may affect the significance of the common node, because the farther the distance, the smaller its impact on the initial decision may be. In the game of poker, the significance function can be defined as Where pot(s) represents the number of chips in the pot on the public node s, which reflects the potential value of the public node s. max{n,0.2} ensures that even if n is small, the function has a certain non-zero value to avoid division by zero. The features of the root node information set are those that can distinguish or classify data. The selection of these features is usually based on criteria such as information gain, information gain ratio or Gini coefficient, with the aim of maximizing classification accuracy. After multiplying the estimated expected value by the significance function and appending it to the features of the root node information set, a weighted expected value is obtained, which can be used as a basis for clustering.

[0137] 4. Clustering root node information set:

[0138] Use a clustering algorithm (such as k-means++) to cluster the root node information set of each game participant. The weighted expected value mentioned above can be used as the basis for clustering. Clustering is also called merging. Through clustering, similar information sets can be grouped together, thereby simplifying the game tree and reducing the number of information sets that need to be considered independently. For example, they are divided into K classes, where K is a positive integer greater than 1. The time complexity of the clustering process is O(TK|S||I r |), where T is the number of iterations of the k-means++ algorithm (which can be set to T = 8), K is the number of clusters, |S| is the number of nodes in the subgame, and |I r | is the number of legal information sets of the subgame at the root node. The complexity is comparable to that of the CFR algorithm, but the constant is much smaller than that of the CFR algorithm, so the running time is negligible relative to that of the CFR algorithm.

[0139] 5. Pruning clustering:

[0140] After clustering is completed, continue to prune each cluster in the subgame. For the common node s in the subgame, if each corresponding information set in cluster k does not exist in the pruned game tree, then it can be considered that this cluster k has no effect on the decision of the common node s. Therefore, starting from the common node s, the entire cluster k can be pruned, that is, no longer consider any information set in this cluster. The purpose of this is to further simplify the game tree and reduce unnecessary calculations. Because in the pruned game tree, if all the information sets of a cluster do not exist, then the strategies and possible results corresponding to the information sets in this cluster will not affect the final equilibrium strategy. By pruning these clusters, the efficiency of solving the subgame equilibrium strategy can be improved.

[0141] Through this abstraction process, the number of information sets that need to be considered can be significantly reduced, thereby reducing computational complexity. This method is particularly suitable for real-time games where the number of information sets is large and decisions need to be made quickly. Through pruning and clustering, the game tree can be effectively simplified while maintaining the quality and effectiveness of the strategy.

[0142] In a possible implementation, the flowchart of the online high-performance processing method based on expected value estimation in the incomplete information game provided by the embodiment of the present disclosure is as follows: Figure 3 As shown, the method includes the following steps.

[0143] Step 301, obtaining the sub-game to be solved.

[0144] Extract a subgame from the larger game that needs to be solved. This subgame may start from a specific state and have its own rules and structure.

[0145] Step 302, for each information set in the sub-game, an expected value is estimated through a target model to determine a target expected value of the information set, where the target model is a pre-trained neural network model for estimating expected values.

[0146] For each information set in the subgame, its expected value is estimated using a pre-trained neural network model (target model). These models are specifically designed to predict the average, maximum, or minimum expected payoff given a given information set.

[0147] Step 303: prune branches corresponding to the sub-information sets according to the target expected values ​​of each of the multiple information sets and the target expected values ​​of each of the multiple sub-information sets, and perform abstract processing on the information sets.

[0148] According to the target expected value of the information set and the target expected value of its sub-information set, the branches corresponding to several sub-information sets are pruned to remove those with lower expected values. Several information sets are abstracted and those with similar expected values ​​are merged to simplify the game tree.

[0149] Step 304: Based on the processed game tree, a preset game algorithm is used to solve and obtain the target equilibrium strategy corresponding to the sub-game.

[0150] According to the processed game tree, a preset game algorithm (such as CFR algorithm) is used to solve the sub-game and obtain the corresponding target equilibrium strategy.

[0151] It should be noted that for the relevant details of the above steps, reference can be made to the relevant description in the above embodiments, which will not be repeated here.

[0152] In another possible implementation, the flowchart of the online high-performance processing method based on expected value estimation in the incomplete information game provided by the embodiment of the present disclosure is as follows: Figure 4 As shown, the method includes the following steps.

[0153] Step 401, obtaining the sub-game to be solved.

[0154] Step 402: for each information set in the sub-game, estimate the expected value through multiple target models to determine the target expected value of the information set.

[0155] Use multiple target models to estimate the expected value of each information set in the sub-game, and calculate the average, maximum and minimum values ​​of these expected value estimates as the target expected value of the information set, that is, the target expected value of each information set includes the average, maximum and minimum values ​​of the expected value estimates output by each of the multiple target models.

[0156] Step 403, pruning the branches corresponding to the sub-information sets on the root nodes of the sub-games according to the target expected values ​​of the multiple information sets.

[0157] According to the target expected values ​​of each of the multiple information sets on the root node of the sub-game and the target expected values ​​of each of the multiple sub-information sets, the branches corresponding to the suboptimal sub-information sets on the root node of the sub-game are pruned to remove those branches with lower expected values.

[0158] Step 404, abstract processing is performed on the branches corresponding to the pruned information set.

[0159] Perform information abstraction on the pruned sub-games, merge similar information sets, and further simplify the game tree.

[0160] Step 405: Continue to prune the branches corresponding to the suboptimal clustering buckets after abstract processing.

[0161] After the abstraction process is completed, branches corresponding to the information sets in the sub-games are pruned according to the target expected values ​​of the multiple information sets in the sub-games to ensure that the game tree is simplified as much as possible. It should be noted that the embodiment of the present disclosure does not limit the execution order and execution times of pruning and abstraction.

[0162] Step 406: Based on the processed game tree, a preset game algorithm is used to solve and obtain the target equilibrium strategy corresponding to the sub-game.

[0163] After the processing is completed, the preset game algorithm is used to solve the target equilibrium strategy corresponding to the sub-game according to the processed game tree.

[0164] It should be noted that for the relevant details of the above steps, reference can be made to the relevant description in the above embodiments, which will not be repeated here.

[0165] The EVPA technology provided by the embodiments of the present disclosure achieves online high-performance pruning and abstraction on large-scale incomplete information games. Illustratively, in terms of pruning, pruning rates of 69.72% to 79.51% were achieved in various sub-games of two-player no-limit Texas Hold'em with 200 big blinds. In addition, after adopting the pruning algorithm of the embodiments of the present disclosure, the exploitability under the same computing time is only 1.4% of the original. In terms of abstraction, the technology of the embodiments of the present disclosure can generate online information abstractions for sub-games of various poker games within a few seconds, and the number of buckets of information abstraction can be adjusted according to resource and solution time constraints, thereby achieving a balance between abstract granularity and computational overhead. Experiments have shown that the use of 30 buckets of abstraction in two-player no-limit Texas Hold'em can achieve an approximate Nash equilibrium within less than 1% of the computational overhead of the baseline algorithm.

[0166] The EVPA technology provided by the embodiment of the present disclosure also performs very well in actual poker games, significantly higher than previous poker AI. 8 Under the condition of 100 information sets, EVPA achieved a win rate of 24.8±4.0 big blinds per 100 hands compared to the DeepStack replica. In the duel with the high-performance online Texas Hold'em AI: Slumbot, the technology of the disclosed embodiment achieved a win rate of 32.1±7.1 big blinds per 100 hands. These results show that the poker AI based on the technology of the disclosed embodiment is one of the most powerful poker AIs in the world.

[0167] A replica of DeepStack was used as the BASE (baseline algorithm), and on this basis, three AIs were designed using the patented technology provided by the embodiments of the present disclosure: EVPA-full (pruning only, no information abstraction), EVPA-169 (pruning and information abstraction of 169 buckets), and EVPA-30 (pruning and information abstraction of 30 buckets). Then, the DCFR algorithm was used to conduct a rich test of the average exploitability (English: Exploitability) relative to the current pot size in the river stage of 200 big blinds two-player no-limit Texas Hold'em to evaluate their performance relative to the current pot size, where exploitability is an indicator of strategy quality. The test results are as follows Figure 5 As shown in the figure, the horizontal axis is the number of information sets contacted, and the vertical axis is the exploitability of the algorithm relative to the current pot size. The EVPA algorithm significantly reduces the exploitability with the same number of information sets contacted.

[0168] In other words, EVPA-full has 1*10 8After 1×1071*10 information sets, the lowest exploitability was achieved, which was only 1.4% of the exploitability of the baseline algorithm. 7 The EVPA-30 showed the lowest exploitability when exposed to 1×1071*10 6 The lowest exploitability is achieved with 10 information sets, which is 54% less than EVPA-169. It is worth noting that in order to achieve an exploitability of no more than 0.08, the baseline algorithm requires about 1*10 8 information set contacts, while EVPA-30 requires less than 1*10 6 This shows that EVPA only requires about 1% of the number of information set contacts required by the baseline algorithm to reach an approximate Nash equilibrium. These findings highlight the significant advantages of the EVPA algorithm in improving the efficiency of incomplete information game processing, especially in real-time games that require fast decision-making. Through carefully designed pruning and information abstraction strategies, the EVPA algorithm is able to maintain or improve the performance of the strategy while significantly reducing the computational burden.

[0169] In summary, the method provided by the embodiment of the present disclosure brings a series of significant beneficial effects, which are mainly reflected in the following aspects: On the one hand, the EVPA technology provides a set of efficient pruning and abstraction methods for incomplete information extended form games through information set expected value estimation. This includes innovative information set expected value estimation methods, expected value-based pruning techniques, and abstraction techniques designed specifically for subgames. The combination of these technologies enables significantly improved computational efficiency and strategy quality when dealing with complex incomplete information games. On the other hand, the pruning and abstraction methods provided by the embodiment of the present disclosure implement AI that can be used to solve large incomplete extended form games, significantly improving the efficiency and effect of solving large incomplete extended form games, that is, through pruning technology, AI can ignore those branches with low expected values, thereby reducing unnecessary calculations and accelerating the convergence of strategies. This method is particularly suitable for games with a large number of information sets and can effectively narrow the search space. Abstract technology allows AI to process similar information sets together, which helps AI capture key features in the game instead of being disturbed by too many details. Such strategies are often more robust and can better cope with different game situations. On the other hand, this technical solution is not only applicable to Texas Hold'em, but also extends to games such as Short Deck, Omaha, Poker Tournament, Mahjong, Throwing Eggs, Landlords, and even online market transactions, strategic deductions, and other large-scale incomplete expansion forms of games. This shows that the method of solving these games based on the CFR algorithm has wide applicability and flexibility, and can effectively solve these complex game problems under limited resources and running time.

[0170] The following is an apparatus embodiment of the present disclosure. For parts not described in detail in the apparatus embodiment, reference may be made to the technical details disclosed in the above method embodiment.

[0171] The embodiment of the present disclosure provides a schematic diagram of the structure of an online high-performance processing device based on expected value estimation in an incomplete information game. The device can realize all or part of a computing device through software, hardware, or a combination of the two. The device includes: an acquisition module, a calculation module, a processing module, and a solution module.

[0172] An acquisition module is used to acquire a subgame to be solved, where the subgame is a game subtree starting from an arbitrary node in the game tree of the incomplete information game and ending at several arbitrary nodes, and the subgame includes respective information sets of multiple game participants at multiple nodes, and the information sets include game state information of the game participants when making decisions;

[0173] A calculation module is used to calculate the target expected value of each of the multiple information sets in the sub-game, and the expected value is used to indicate the expected benefits of the game participants taking the target decision under the information set;

[0174] A processing module, used for pruning and / or abstracting subgames according to respective target expected values ​​of multiple information sets;

[0175] The solving module is used to solve the target equilibrium strategy corresponding to the sub-game according to the processed game tree using a preset game algorithm. The target equilibrium strategy is used to indicate the strategy combination of multiple game participants in the equilibrium state.

[0176] In a possible implementation, the processing module is further configured to:

[0177] According to the target expected values ​​of each of the multiple information sets and the target expected values ​​of each of the multiple sub-information sets, branches corresponding to the sub-information sets are pruned, the information set includes multiple sub-information sets, and the multiple sub-information sets correspond to different decisions; and / or,

[0178] The information sets are abstracted according to their respective target expected values.

[0179] In another possible implementation, the processing module is further configured to:

[0180] When the target expected value of a sub-information set is lower than the target expected value of the corresponding information set, the branch corresponding to the sub-information set is pruned.

[0181] In another possible implementation, the processing module is further configured to:

[0182] When there are at least two information sets whose respective target expected values ​​satisfy the merging condition, merging the at least two information sets, and the abstract processing includes merging the at least two information sets;

[0183] Among them, the target expected value of the information set includes the current expected value and the future expected value, the merging condition includes that the similarity between the current expected values ​​of at least two information sets is higher than the first similarity threshold, and the similarity between the future expected values ​​is higher than the second similarity threshold, the current expected value is the expected value of the current information set, and the future expected value is the expected value expected to be achieved under the future information set after taking the target decision under the current information set.

[0184] In another possible implementation, the computing module is further configured to:

[0185] For each information set in the sub-game, the expected value is estimated through the target model to determine the target expected value of the information set. The target model is a pre-trained neural network model for estimating expected values.

[0186] In another possible implementation, the target models include multiple ones, and the target expected value of the information set includes at least one of an average value, a maximum value, and a minimum value of the expected value estimates output by each of the multiple target models.

[0187] In another possible implementation, the device is applied to an agent that is one of the players in a multiplayer Texas Hold'em game;

[0188] The participants in the game are players;

[0189] The game status information includes hand information and table display information. The table display information includes the displayed public card information, the number of chips of all players at the table, the amount of the prize pool at the table, and the historical decision information of all players in this round.

[0190] Decisions include folding, calling, and raising.

[0191] In another possible implementation, the device is applied to an intelligent agent that is one of the participants in online market transactions;

[0192] The participants in the game are the transaction participants;

[0193] Game status information includes private information and market public information. Market public information includes historical transaction data of multiple transaction participants, current transaction volume, transaction amount and price;

[0194] Decisions include buying, seeking, selling and listing.

[0195] It should be noted that the device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example to implement its functions. In practical applications, the above-mentioned functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0196] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0197] An embodiment of the present disclosure also provides a computing device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0198] The embodiment of the present disclosure further provides a non-volatile computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in the claim is implemented.

[0199] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of a computing device, the processor in the computing device executes the above method.

[0200] Figure 6 1 is a block diagram of an apparatus 1900 for online high-performance processing based on expected value estimation in an incomplete information game according to an exemplary embodiment. For example, the apparatus 1900 can be provided as a server or a terminal device. Figure 6 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0201] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , UnixTM ,Linux TM , FreeBSD TM or similar.

[0202] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.

[0203] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0204] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0205] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0206] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0207] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0208] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0209] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0210] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0211] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An online high-performance processing method based on expected value estimation in incomplete information games, characterized in that: The method comprises: Obtain a subgame to be solved, wherein the subgame is a game subtree starting from an arbitrary node in a game tree of an incomplete information game and ending at a plurality of arbitrary nodes, wherein the subgame includes respective information sets of a plurality of game participants at a plurality of nodes, and the information sets include game state information of the game participants when making decisions; Calculating target expected values ​​of each of the plurality of information sets in the sub-game, wherein the target expected values ​​are used to indicate the expected benefits of the game participants taking target decisions under the information sets; Pruning and / or abstracting the subgames according to respective target expected values ​​of the plurality of information sets; According to the processed game tree, a preset game algorithm is used to solve and obtain the target equilibrium strategy corresponding to the sub-game, and the target equilibrium strategy is used to indicate the strategy combination of the multiple game participants in an equilibrium state.

2. The method according to claim 1, characterized in that The pruning and / or abstracting of the sub-games according to the target expected values ​​of the respective information sets includes: According to the target expected values ​​of each of the plurality of information sets and the target expected values ​​of each of the plurality of sub-information sets, branches corresponding to the sub-information sets are pruned, the information set comprising a plurality of sub-information sets corresponding to different decisions; and / or, The information sets are abstracted according to respective target expected values ​​of the plurality of information sets.

3. The method according to claim 2, characterized in that The pruning of branches corresponding to the sub-information sets according to the target expected values ​​of the plurality of information sets and the target expected values ​​of the plurality of sub-information sets comprises: When the target expected value of the sub-information set is lower than the target expected value of the corresponding information set, the branch corresponding to the sub-information set is pruned.

4. The method according to claim 2, characterized in that: The abstracting of the information sets according to the target expected values ​​of the respective information sets includes: When there are at least two information sets whose respective target expected values ​​satisfy the merging condition, merging the at least two information sets, wherein the abstract processing includes merging the at least two information sets; Among them, the target expected value of the information set includes a current expected value and a future expected value, the merging condition includes that the similarity between the current expected values ​​of at least two of the information sets is higher than a first similarity threshold, and the similarity between the future expected values ​​is higher than a second similarity threshold, the current expected value is the expected value of the current information set, and the future expected value is the expected value expected to be achieved under the future information set after taking the target decision under the current information set.

5. The method according to any one of claims 1 to 4, characterized in that: The calculating the target expected value of each of the plurality of information sets in the sub-game comprises: For each of the information sets in the sub-game, an expected value is estimated through a target model to determine a target expected value of the information set, wherein the target model is a pre-trained neural network model for estimating expected values.

6. The method according to claim 5, characterized in that The target models include multiple ones, and the target expected value of the information set includes at least one of an average value, a maximum value and a minimum value of the expected value estimates output by each of the multiple target models.

7. The method according to any one of claims 1 to 4, characterized in that The method is applied to an agent that is one of the players in a multiplayer Texas Hold'em game; The game participants are players; The game status information includes hand information and table display information, wherein the table display information includes displayed community card information, the number of chips of all players at the table, the amount of the prize pool at the table, and the historical decision information of all players in this round; Said decisions include folding, calling, and raising.

8. The method according to any one of claims 1 to 4, characterized in that The method is applied to an agent that is one of the participants in an online market transaction; The game participants are transaction participants; The game status information includes private information and market public information, and the market public information includes historical transaction data, current transaction volume, transaction amount and price of multiple transaction participants; The decisions include buying, bidding, selling and listing.

9. An online high-performance processing device based on expected value estimation in incomplete information games, characterized in that: The device comprises: An acquisition module is used to acquire a subgame to be solved, wherein the subgame is a game subtree starting from an arbitrary node in a game tree of an incomplete information game and ending at a plurality of arbitrary nodes, wherein the subgame includes respective information sets of a plurality of game participants at a plurality of nodes, and the information sets include game state information of the game participants when making decisions; A calculation module, used for calculating the target expected value of each of the plurality of information sets in the sub-game, wherein the expected value is used to indicate the expected benefits of the game participants taking the target decision under the information set; A processing module, used for pruning and / or abstracting the sub-games according to the target expected values ​​of the respective information sets; A solution module is used to solve the target equilibrium strategy corresponding to the sub-game according to the processed game tree using a preset game algorithm, and the target equilibrium strategy is used to indicate the strategy combination of the multiple game participants in an equilibrium state.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images