An interactive method and device based on a game strategy model, equipment and medium

Abstract

The application discloses an interactive method and device based on a game strategy model, equipment and medium, relates to the technical field of computers, and comprises the following steps: monitoring game information of a current domino game session and determining an input feature set; in a perfect information environment where all player hands are known, simulating an independent game based on the input feature set and a current game strategy model, determining a first trained strategy model by using obtained training sample sets; completing opponent modeling self-game training in a non-perfect information environment simulation based on global state features and the first strategy model; training a second strategy model obtained by using a teacher network trained based on game session win or loss information; training a third strategy model obtained based on the input feature set and player card playing behavior types, and generating current card playing decisions for players by using a fourth trained strategy model to interact. The application solves the problem that the gap between perfect information and non-perfect information is difficult to bridge in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an interactive method and device based on a game strategy model, equipment and medium. BACKGROUND

[0002] Traditional Dou Di Zhu AI (Artificial Intelligence) training schemes mostly rely on rule-based strategies or simple supervised learning methods, which are difficult to adapt to the changing strategy game needs in dynamic environments, resulting in poor interactive experience. In recent years, reinforcement learning schemes based on self-play mechanism have gradually become mainstream, but in the incomplete information game such as Dou Di Zhu, the traditional self-play scheme still faces the following challenges:

[0003] (1) The gap between perfect information and imperfect information is difficult to bridge. In actual games, players can only know their own information and cannot obtain the opponent's hand information and other implicit information, which leads to slow convergence speed and weak generalization ability of strategy learning due to noise interference;

[0004] (2) The imbalance of action space leads to low sampling efficiency. The difference between active card playing and passive card taking in Dou Di Zhu is significant in decision space, but the traditional scheme uses a unified sampling mechanism for different types of actions, which often cannot balance exploration and stability, resulting in low strategy updating efficiency. SUMMARY

[0005] Therefore, the purpose of the present application is to provide an interactive method and device based on a game strategy model, which can solve the problem that the gap between perfect information and imperfect information is difficult to bridge and the imbalance of action space leads to low sampling efficiency in the prior art, thereby effectively realizing efficient, stable and continuous automation and reliable training of the Dou Di Zhu game strategy model, improving the adaptability to the changing strategy game needs in dynamic environments and the action sampling efficiency, and improving the experience of players in the game. The specific scheme is as follows:

[0006] In a first aspect, the present application provides an interactive method based on a game strategy model, applied to a computer device, comprising:

[0007] monitoring game information of a current Dou Di Zhu game, wherein the game information includes call card information, hand information, historical card playing actions, a current card playing action and player remaining hand information after execution of the current card playing action corresponding to each player, the each player includes a target player and other players participating in the current Dou Di Zhu game, the target player is a real player or a virtual player, and the other players include real players and / or virtual players;

[0008] encoding the game information to determine an input feature set; the input feature set includes global state features and card playing action features;

[0009] In a perfect information environment where the hand information of each player participating in the game is known, based on a Monte Carlo simulation algorithm, the input feature set and the current game strategy model, and introducing the number of times of using the target card pattern as a reward multiplier to perform independent game simulation, to complete self-game model training using the simulation obtained training sample set, and obtain the first trained strategy model; the current game strategy model includes a card playing action decision component and a next hand information prediction component;

[0010] Based on the global state features in the training sample set, a preset random dropout regularization technique and the first strategy model, the next hand information prediction is performed to complete the opponent modeling self-game training under the imperfect information environment simulation, and the second trained strategy model is obtained;

[0011] Based on the game win or lose information corresponding to the training sample set, the teacher network is trained, and the corresponding current trained teacher network and a preset game difficulty control mechanism are used to train the second strategy model, to determine the third trained strategy model when the win rate of players in different roles meets a preset training termination condition;

[0012] Based on the input feature set, the player card playing behavior type and a preset player card playing action difference sampling mechanism, the third strategy model is trained to determine the fourth trained strategy model, and the fourth strategy model is used to generate a current card playing decision for the target player, so that the target player uses the current card playing decision to interact with the other players.

[0013] Optionally, the encoding the game information to determine an input feature set; the input feature set includes global state features and card playing action features, includes:

[0014] Encoding the bidding information, the hand information and the historical card playing actions in the game information to determine global state features;

[0015] Encoding the current card playing action and the player's remaining hand information after the execution of the current card playing action in the game information to determine card playing action features;

[0016] Determining an input feature set based on the global state features and the card playing action features.

[0017] Optionally, the method further includes:

[0018] Collect different types of off-site target card pattern information, and encode the off-site target card pattern information to determine the corresponding target card pattern features.

[0019] Optionally, the Monte Carlo simulation algorithm, the input feature set and the current game strategy model are introduced, and the number of times of using the target card pattern is used as a reward multiplier to perform independent game simulation, so as to complete self-game model training by using the simulation obtained training sample set, including:

[0020] Based on the Monte Carlo simulation algorithm, the input feature set and the current game strategy model, the initial state of the game is generated to determine the initial state generation result;

[0021] Based on the current game strategy model, the entire card playing process is simulated from the player who is the landlord, and the state information, action information and target card pattern use information of each player at each step are recorded, as well as the game win or lose result; the target card pattern use information includes the number of times of using the target card pattern;

[0022] Based on the game win or lose result, the target card pattern use information and the preset reward function, the corresponding training reward information is determined;

[0023] Based on the state information of each player in the entire game, the corresponding state sequence is determined;

[0024] The training reward information is used as the supervision target corresponding to each state in the state sequence, sample construction is performed, and the training sample is determined;

[0025] The training sample is stored in a buffer pool, and it is judged whether the number of samples in the buffer pool meets a preset number condition to determine a number judgment result;

[0026] If the number judgment result is yes, the training sample set is determined based on the training sample in the buffer pool;

[0027] The training sample set and a preset loss function are used to perform self-game model training on the current game strategy model to obtain a trained first strategy model.

[0028] Optionally, the global state feature in the training sample set, a preset random drop regularization technique and the first strategy model are used to predict the hand information of the next player to complete the modeling of the opponent in the non-perfect information environment simulation self-game training, including:

[0029] For any player in the game, based on the first strategy model and the hand information of the current player and the hand information of the current player's opponent, the hand information of the next player of the current player is predicted to determine the next player hand feature prediction result;

[0030] fusing the lower hand feature prediction result and the global state feature corresponding to the hand information of the lower, to determine a fused feature;

[0031] performing lower hand information prediction based on the fused feature, a preset random dropout regularization technique and the first strategy model, to complete self-game training of an opponent modeling under imperfect information environment simulation, and obtain a trained second strategy model.

[0032] Optionally, the teacher network is trained based on the game match win / loss information corresponding to the training sample set, and the second strategy model is trained by using a corresponding current trained teacher network and a preset match difficulty control mechanism, including:

[0033] In the process of simulating the entire card playing flow based on the current game strategy model and starting from the player as the landlord, an action selection sequence corresponding to three players is obtained;

[0034] The action selection sequence is used as a supervision signal, and the hand information of the player at the beginning of the match is used as input information of the model, to train the teacher network, so as to obtain a current trained teacher network; the teacher network is used for predicting a landlord winning rate;

[0035] Based on the current trained teacher network and the game match win / loss information corresponding to the training sample set, match screening is performed to determine a match screening result; the match screening result includes a peasant strategy training sample and a landlord strategy training sample, the peasant strategy training sample is a game match with a landlord winning rate greater than a first probability, and the landlord strategy training sample is a game match with a landlord winning rate meeting a preset probability interval;

[0036] Based on the match screening result and a preset match difficulty control mechanism, the second strategy model is trained for different roles, to determine a trained third strategy model when the winning rate of the player of different roles meets a preset training termination condition.

[0037] Optionally, the third strategy model is trained based on the input feature set, a player card playing behavior type and a preset player card playing action difference sampling mechanism, including:

[0038] The third strategy model is trained under different player card playing behavior types based on the input feature set and a preset player card playing action difference sampling mechanism, so as to determine a fourth trained strategy model; wherein the player card playing behavior types include an active card playing type and a passive card receiving type; the preset player card playing action difference sampling mechanism includes a preset random sampling strategy corresponding to the active card playing type and a preset temperature control sampling strategy corresponding to the passive card receiving type.

[0039] In a second aspect, the present application provides an interactive device based on a game strategy model, applied to a computer device, comprising:

[0040] A game information monitoring module is configured to monitor game information of a current Dou Di Zhen game; the game information includes calling card information, hand card information, historical card playing actions, a current card playing action, and player remaining hand card information after the current card playing action, the players include a target player and other players participating in the current Dou Di Zhen game, the target player is a real player or a virtual player, and the other players include real players and / or virtual players;

[0041] A feature set acquisition module is configured to encode the game information to determine an input feature set; the input feature set includes global state features and card playing action features;

[0042] A first training module is configured to simulate an independent game based on a Monte Carlo simulation algorithm, the input feature set, and a current game strategy model, and introduce the number of times of using a target card type as a reward multiplier, to complete self-game model training by using a simulation obtained training sample set, and obtain a first trained strategy model; the current game strategy model includes a card playing action decision component and a next player hand card prediction component;

[0043] A second training module is configured to predict next player hand card information based on the global state features in the training sample set, a preset random discarding regularization technique, and the first strategy model, to complete self-game training of an opponent under a non-perfect information environment simulation, and obtain a second trained strategy model;

[0044] A third training module is configured to train a teacher network based on game win or lose information corresponding to the training sample set, and train the second strategy model by using a corresponding current trained teacher network and a preset game difficulty control mechanism, to determine a third trained strategy model when a player win rate of different roles meets a preset training termination condition;

[0045] A fourth training module is configured to train the third strategy model based on the input feature set, the player card playing behavior type, and a preset player card playing action difference sampling mechanism, determine a trained fourth strategy model, and generate a current card playing decision for the target player by using the fourth strategy model, so that the target player performs corresponding interaction with the other players by using the current card playing decision.

[0046] In a third aspect, the present application provides an electronic device, comprising:

[0047] a memory configured to save a computer program;

[0048] a processor configured to execute the computer program to implement the steps of the game strategy model based interaction method.

[0049] In a fourth aspect, the present application provides a computer readable storage medium configured to save a computer program, wherein the computer program is executed by a processor to implement the steps of the game strategy model based interaction method.

[0050] It can be seen that, in the present application, the game information of the current domino game is monitored by applying the computer device, the game information includes the calling card information, the hand card information, the historical card action, the current card action and the remaining hand card information of the players after the execution of the current card action, the players include the target player and other players participating in the current domino game, the target player is a real player or a virtual player, and the other players include real players and / or virtual players; the game information is encoded to determine the input feature set; the input feature set includes global state features and card action features; in the perfect information environment where the hand card information of each player participating in the game is known, the Monte Carlo simulation algorithm, the input feature set and the current game strategy model are used to simulate independent games by introducing the number of times of using the target card type as a reward multiplier, to complete the self-game model training by using the simulation obtained training sample set, and to obtain the first trained strategy model; the current game strategy model includes a card action decision component and a next hand card prediction component; the global state features in the training sample set, the preset random drop regularization technology and the first strategy model are used to predict the next hand card information, to complete the opponent modeling self-game training in the imperfect information environment simulation, and to obtain the second trained strategy model; the game win or lose information corresponding to the training sample set is used to train the teacher network, and the corresponding current trained teacher network and the preset game difficulty control mechanism are used to train the second strategy model, to determine the third trained strategy model when the win rate of the players in different roles meets the preset training termination condition; the input feature set, the player card action type and the preset player card action difference sampling mechanism are used to train the third strategy model to determine the fourth trained strategy model, and the fourth strategy model is used to generate the current card decision for the target player, so that the target player interacts with the other players by using the current card decision.That is, in the present application, the game information of the mahjong game is first collected and encoded to obtain an input feature set, then in the perfect information environment, the input feature set and the current game strategy model are used to simulate independent games based on the use frequency of the target card pattern as a reward multiplier, to train the self-game model using the simulation training sample set, and obtain the first strategy model, then the global state feature, the preset random drop regularization technology and the first strategy model are used to predict the hand information of the next player, to complete the opponent modeling self-game training in the non-perfect information environment simulation, and obtain the second strategy model, then the teacher network is trained using the game win or lose information, and the second strategy model is trained using the current trained teacher network, to determine the third strategy model when the win rate of the player in different roles meets the preset training termination condition, then the third strategy model is trained according to the player's card behavior type and the preset player card action difference sampling mechanism, and the fourth strategy model obtained after training is used to generate card decision for the target player to interact with other players. In this way, the gap between perfect information and non-perfect information in the prior art can be bridged, and the low sampling efficiency caused by unbalanced action space can be solved, so that efficient, stable, continuous automation and reliable mahjong game strategy model training are effectively realized, and the adaptability to dynamic environment and the action sampling efficiency of the strategy game demand are improved, and the experience of the players in the game is improved. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0052] Figure 1 A flowchart of an interactive method based on a game strategy model is provided for the present application.

[0053] Figure 2 A flowchart of a specific interactive method based on a game strategy model is provided for the present application.

[0054] Figure 3 A schematic diagram of the coupling and separation process of the perfect information and non-perfect information model is provided for the present application.

[0055] Figure 4 A course learning process schematic diagram is provided for the present application.

[0056] Figure 5 A structure schematic diagram of an interactive device based on a game strategy model is provided for the present application.

[0057] Figure 6 An electronic device structure diagram is provided for the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0059] The traditional self-game solution still faces the following challenges: (1) the gap between perfect information and imperfect information is difficult to bridge. In actual games, players can only know their own information and cannot obtain the hidden information such as the opponent's hand, which leads to that the strategy learning is easily disturbed by noise, the convergence speed is slow, and the generalization ability is weak; (2) the imbalance of action space leads to low sampling efficiency. In Douzhimin, the difference between active card playing and passive card taking in the decision space is significant, but the traditional solution adopts a unified sampling mechanism for different types of actions, which often cannot balance the exploratory and stability, leading to low strategy updating efficiency.

[0060] Therefore, the present application provides an interactive solution based on a game strategy model, which can solve the problems of the gap between perfect information and imperfect information being difficult to bridge and the imbalance of action space leading to low sampling efficiency in the prior art, thereby effectively realizing efficient, stable, and continuous automation and reliable training of a Douzhimin game strategy model, improving the adaptability to the changing strategy game demand in a dynamic environment and the action sampling efficiency, and improving the experience of players in the game.

[0061] Referring to Figure 1 The embodiments of the present application disclose an interactive method based on a game strategy model, applied to a computer device, comprising:

[0062] Step S11, monitoring game information of a current Douzhimin game; the game information includes calling card information, hand information, historical card playing actions, a current card playing action, and player remaining hand information after execution of the current card playing action corresponding to each player, the each player includes a target player and other players participating in the current Douzhimin game, the target player is a real player or a virtual player, and the other players include real players and / or virtual players.

[0063] In this embodiment, first, the current game of Dou Dizhu needs to be monitored to obtain the corresponding calling information, hand information, historical card action, current card action and player remaining hand information after the current card action of each player in the current game of Dou Dizhu, each player including a target player participating in the current game of Dou Dizhu and other players, the target player being a real player or a virtual player, and the other players including real players and / or virtual players. In this way, a foundation is laid for the subsequent feature construction step.

[0064] In step S12, the game information is encoded to determine an input feature set; the input feature set includes global state features and card action features.

[0065] In this embodiment, in combination with Figure 2 As shown in the figure, while obtaining the game information, the game information is encoded to obtain the input feature set, that is, the calling information, the hand information and the historical card action in the game information are encoded to determine the global state features; the current card action and the player remaining hand information after the current card action in the game information are encoded to determine the card action features; and the input feature set is determined based on the global state features and the card action features. Different kinds of off-table target card pattern information are collected, and the off-table target card pattern information is encoded to determine the corresponding target card pattern features. The off-table target card pattern information refers to the target card pattern information that has been played by a player, and the target card pattern includes a card pattern composed of four cards of the same value and a card pattern composed of a big king card and a small king card.

[0066] Specifically, regarding the construction of features, in this embodiment, a binary decision matrix based on the order of players is constructed, the first row of the matrix being the selection in the “calling landlord” stage, and the second row being the selection in the “stealing landlord” stage. Each column corresponds to a player, and the matrix element takes a value of 1 indicating selection (calling or stealing) or 0 indicating abandonment. Taking the first player calling the landlord and the three players all selecting to steal the landlord as an example, the process can be represented as the following matrix:

[0067] .

[0068] For the off-table target card pattern features, in this embodiment, a binary target card pattern feature vector is designed: each dimension corresponds to a possible target card pattern type, and if the target card pattern of this type has appeared off-table, the corresponding element is recorded as 1, and vice versa. For example, if it is known that the “four three card pattern”, “four five card pattern” and “big and small king card pattern” have appeared off-table, the target card pattern feature vector is represented as (1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1).

[0069] In addition, for the features of the player's hand, the current action, the remaining hand after the current action, and the historical action, a 54-dimensional vector coding is adopted to code the number of each card.

[0070] In step S13, in the perfect information environment where the hand information of each player participating in the game is known, based on the Monte Carlo simulation algorithm, the input feature set, and the current game strategy model, the number of uses of the target card pattern is introduced as a reward multiplier to perform independent game simulation, to complete self-game model training by using the simulation obtained training sample set, and obtain the trained first strategy model; the current game strategy model includes a card playing action decision component and a next player hand prediction component.

[0071] In this embodiment, in combination with Figure 2 As shown in the figure, after the feature construction is completed, the features are used to train the perfect information self-game model, that is, based on the Monte Carlo simulation algorithm, the input feature set, and the current game strategy model, the initial state of the game is generated to determine the initial state generation result; based on the current game strategy model, the entire card playing process is simulated from the player who is the landlord, and the state information, action information, and target card pattern use information of each player at each step, and the game win or lose result are recorded; the target card pattern use information includes the number of uses of the target card pattern; based on the game win or lose result, the target card pattern use information, and a preset reward function, the corresponding training reward information is determined; based on the state information of each player in the entire game, the corresponding state sequence is determined; the training reward information is used as the supervision target corresponding to each state in the state sequence, sample construction is performed to determine the training sample; the training sample is stored in the buffer pool, and it is determined whether the number of samples in the buffer pool meets the preset number condition to determine the number determination result; if the number determination result is yes, the training sample set is determined based on the training sample in the buffer pool; the training sample set and a preset loss function are used to train the current game strategy model to obtain the trained first strategy model.

[0072] Specifically, in combination with Figure 3As shown, in this embodiment, regarding the perfect information self-game model training process, in the perfect information environment where the hands of the three parties are known, a method based on Monte Carlo simulation is used to generate high-quality game data, and the number of target card types used is introduced as a reward multiplier to optimize the signal quality of strategy training. Under this setting, independent game simulation is performed using a strategy model, and the specific steps include: 1) initial state generation: random card distribution is used to ensure that the hands of the three parties have no overlap, and the calling process is performed to determine the landlord and the farmer, and to meet the rules of the game; 2) strategy-driven simulation: according to the current strategy model, the entire card distribution process is automatically executed from the landlord, and the state, action and target card type usage information at each step are recorded; 3) state trajectory recording: for each player, all states in the card distribution sequence are recorded , and the state-reward sample is constructed; 4) result evaluation: based on the game result and the number of target card types used, the final reward required for training is calculated; reward mechanism: composite reward based on win-lose result and target card type usage number. To enhance the sensitivity of the reward signal to the use of key target card types, the composite reward function shown below is designed:

[0073] ;

[0074] wherein, is the final training reward of the i-th player; is the win-lose result of the i-th player in the current game (win is +1, lose is -1); is the number of target card types actually used by the i-th player in the game; is the target card type sensitivity coefficient, which is used to control the amplification degree of the target card type to the reward signal.

[0075] Further, regarding the data construction and training signal generation in this process, for each player i, the state sequence of the entire game is extracted, and the composite reward is used as the supervision target for each state in the sequence to form the training sample set :

[0076] .

[0077] wherein, is the state sequence of the i-th player at time T in the entire game.

[0078] In the strategy model training phase, the mean square error loss function is used to optimize the state value function V(s), and the specific target is as follows:

[0079] .

[0080] wherein, for the i-th sample state, N is the number of samples; for the i-th player.

[0081] In step S14, the hand information of the next player is predicted based on the global state feature in the training sample set, a preset random dropout regularization technique, and the first strategy model, so as to complete the modeling of the opponent in the non-perfect information environment simulation self-game training, and obtain the second strategy model after training.

[0082] In combination Figure 3 As shown in the figure, after the perfect information self-game model training is completed, the embodiment will also perform non-perfect information opponent modeling self-game training, that is, for any player in the game, based on the first strategy model and the hand information of the current player and the hand information of the current player's next player, the hand information of the next player of the current player is predicted to determine the hand feature prediction result of the next player; the hand feature prediction result of the next player and the global state feature corresponding to the hand information of the next player are fused to determine the fused feature; the hand information of the next player is predicted based on the fused feature, a preset random dropout regularization technique, and the first strategy model, so as to complete the modeling of the opponent in the non-perfect information environment simulation self-game training, and obtain the second strategy model after training. The preset random dropout regularization technique can be a step-by-step Dropout mechanism.

[0083] It should be understood that in the process of non-perfect information opponent modeling self-game training, in the embodiment, the opponent modeling self-game aims to simulate the strategy learning process in the real non-perfect information environment. Based on the global game state feature, the hand information possibly held by the next player is estimated by a prediction module, so as to construct the feature expression under non-perfect information.

[0084] Specifically, the model is first trained using the game data under perfect information, and on this basis, a hand prediction task is introduced, the estimated hand feature is obtained by modeling the probability of the next player's hand, and then the predicted hand feature is fused with the real hand information for training the strategy network. In order to gradually transition to a completely non-perfect information environment, a step-by-step Dropout mechanism is used in the training process, that is, the proportion of dependence on the real hand (perfect information) is gradually reduced in multiple stages. In the final stage, the model outputs the strategy only based on the predicted feature, thereby having the generalization ability of modeling unknown opponents in the real game environment.

[0085] In this way, the opponent modeling self-game mechanism realizes the migration from the ideal training environment to the real game environment, and strengthens the robustness and game adaptability of the model under non-perfect information conditions.

[0086] Step S15, training of the teacher network based on the game result information corresponding to the training sample set, and training of the second strategy model using the corresponding current trained teacher network and a preset game difficulty control mechanism, to determine a third strategy model after training when the win rate of the players of different roles meets a preset training termination condition.

[0087] In this embodiment, in combination with Figure 2 As shown in FIG. 6, the course learning will also be performed, a coach network (i.e., a teacher network) is trained using the win / loss information of the training games to predict the win rate of the landowner, so as to realize the evaluation of the card strength value, that is, in the process of simulating the entire card playing process based on the current game strategy model starting from the player with the role of landowner, the action selection sequence corresponding to the three players is obtained; the action selection sequence is used as a supervision signal, and the hand information of the player at the beginning of the game is used as the input information of the model, and the training of the teacher network is performed to obtain the current trained teacher network; the teacher network is used to predict the win rate of the landowner; based on the current trained teacher network and the game result information corresponding to the training sample set, the game screening is performed to determine the game screening result; the game screening result includes the peasant strategy training sample and the landowner strategy training sample, the peasant strategy training sample is a game with a landowner win rate greater than a first probability, and the landowner strategy training sample is a game with a landowner win rate meeting a preset probability interval; based on the game screening result and a preset game difficulty control mechanism, the second strategy model is trained for the strategy of different roles respectively, to determine a third strategy model after training when the win rate of the players of different roles meets a preset training termination condition.

[0088] Specifically, in combination with Figure 4 As shown in FIG. 6, in the process of course learning, the game difficulty control mechanism is introduced to realize a robust training process of the strategy model from easy to difficult. This step includes two core modules: a coach network and a course sample scheduling strategy.

[0089] Under the condition of perfect information, a large number of games are sampled using a high-quality strategy based on MCTS (Monte Carlo Tree Search), and whether the landowner wins in the end is recorded under the initial state of each game. A coach network is trained using this as a supervision signal to predict the win rate of the landowner corresponding to any initial hand configuration as an index for measuring the difficulty of the current game. The input of the network is the initial state (including the hand of the landowner, the peasant and the three families), and the output is the estimated value of the win rate of the landowner.

[0090] In the strategy model training stage, the sample selection is performed according to the landowner win rate score provided by the coach network to realize a course training process.

[0091] (1) Farmer strategy training: priority selection of landlord win rate The game with a (optional value 60%) as the starting point of training, that is, the farmer is in a more favorable position. As the training progresses, the difficulty of the confrontation is gradually increased, and the training samples are expanded to high-pressure scenarios with a landlord win rate of ≥90% to enhance the farmer's ability to respond to unfavorable situations;

[0092] (2) Landlord strategy training: reverse operation, starting from a medium-difficulty game with a landlord win rate of about 40%, gradually transitioning to difficult samples with a win rate of only 10%, guiding the model to cope with complex and disadvantaged scenarios, thereby improving the robustness and generalization ability of the strategy.

[0093] Step S16, training the third strategy model based on the input feature set, player card behavior type, and preset player card action difference sampling mechanism, determining the fourth strategy model after training, and using the fourth strategy model to generate a current card decision for the target player, so that the target player uses the current card decision to interact with the other players accordingly.

[0094] In combination Figure 2 with the above, in the present embodiment, the difference sampling of different card behaviors is also trained, that is: based on the input feature set and the preset player card action difference sampling mechanism, the third strategy model is sampled and trained under different player card behavior types to determine the fourth strategy model after training; wherein the player card behavior type includes an active card type and a passive card type; the preset player card action difference sampling mechanism includes a preset random sampling strategy corresponding to the active card type and a preset temperature-controlled sampling strategy corresponding to the passive card type.

[0095] It should be understood that in the process of training the difference sampling of different card behaviors, since in the game of Dou Dizhu, card behaviors can be divided into two categories: active card and passive card, and there are significant differences in the degree of freedom of strategy selection and the value of game. In order to more effectively guide the strategy exploration while maintaining the stability of the sample distribution and the diversity of the strategy output in the training process, a set of difference sampling mechanism is designed, as shown below:

[0096] (1) Active card: using a random sampling strategy.

[0097] When the player is in an active card position (i.e. no upper limit, can freely choose to play cards), the strategy network faces a larger action space and game branches. In order to enhance the exploratory, the greedy ( greedy strategy) is used for action sampling: with a certain probability The action with the highest probability in the current policy output is selected, and in other cases, random sampling is performed from the available actions. This mechanism can prevent the policy from getting stuck in early local optima during the active phase, thus improving the diversity and generalization ability of the policy.

[0098] (2) Passive card receiving: adopting a temperature-controlled soft sampling strategy.

[0099] During the passive play phase (i.e., the card played must be higher than the previous player's), the range of cards played is limited. To better balance exploration and stability, a temperature sampling strategy based on Softmax is adopted. :

[0100] ;

[0101] in, The action score output by the policy network; Indicates a specific action that is currently being calculated; This represents summing over the entire set of actions; This is a temperature parameter. Higher... Values ​​can increase sampling diversity, lower values The value tends to favor greedy choices.

[0102] Furthermore, this embodiment includes steps to accelerate the aforementioned training process. In terms of system implementation, this embodiment addresses both data transmission and model parallelism, designing a highly efficient training acceleration mechanism to improve model iteration efficiency and computational resource utilization. Details are as follows:

[0103] (1) Asynchronous tensor transfer mechanism.

[0104] In the data preprocessing stage, an asynchronous non-blocking tensor transfer strategy is adopted: after the CPU (Central Processing Unit) completes data generation and sample encoding, it uses... This asynchronous method transfers data tensors to the GPU (Graphics Processing Unit), avoiding the blocking of GPU inference due to data loading and improving overall throughput. This mechanism fully utilizes PyTorch's tensor streaming mechanism (torch.cuda.Stream) and... Preprocessing options enable parallel execution of data loading and model inference, effectively reducing data communication bottlenecks between the CPU and GPU.

[0105] (2) Multi-card, multi-process distributed parallel training.

[0106] The training stage adopts a distributed training manner based on a multi-card multi-process architecture. Each training process is bound to a single GPU, and gradient synchronization and model parameter consistency maintenance are achieved through the distributed training manner of torch.nn.parallel.Distributed Data Parallel (DDP, a distributed training component provided by PyTorch). This structure has good scalability, can support multi-machine multi-card deployment, significantly speeds up model convergence, and improves training stability.

[0107] To further improve parallel efficiency, the training system adopts a main process control training logic in implementation. Each sub-process asynchronously obtains training samples through a shared queue and independently completes forward inference and gradient calculation. High-speed communication between GPUs is completed through the NCCL (NVIDIA Collective Communications Library, a high-performance communication library developed by NVIDIA) backend, supporting stable expansion of the model in large-scale data scenarios.

[0108] As can be seen, in the present application, the game information of the Mahjong game is first collected and encoded to obtain an input feature set, then in a perfect information environment, based on the input feature set and the current game strategy model, the number of uses of the target card pattern is introduced as a reward multiplier for independent game simulation, to use the simulation obtained training sample set to complete self-game model training and obtain a first strategy model, then based on the global state feature, the preset random dropout regularization technology and the first strategy model, the next hand card information prediction is performed to complete the opponent modeling self-game training in the non-perfect information environment simulation to obtain a second strategy model, then the teacher network is trained using the game win / loss information, and the second strategy model is trained using the current trained teacher network, to determine a third strategy model when the win rate of the players in different roles meets the preset training termination condition, then the third strategy model is trained according to the player's card behavior type and the preset player card action difference sampling mechanism, and the fourth strategy model obtained after training is used to generate card decision for the target player to interact with other players. In this way, the gap between perfect information and non-perfect information in the existing scheme can be bridged, and the low sampling efficiency caused by unbalanced action space can be solved, thereby effectively realizing efficient, stable, continuous automation and reliable training of the Mahjong game strategy model, and improving the adaptability to the changing strategy game demand in the dynamic environment and the action sampling efficiency, and improving the experience of the players in the game.

[0109] Referring to Figure 5 The embodiment of the present application also discloses an interactive device based on a game strategy model, applied to a computer device, comprising:

[0110] The game information monitoring module 11 is configured to monitor game information of a current game of Douzheng. The game information includes calling card information, hand card information, historical card playing actions, a current card playing action, and remaining hand card information of a player after the current card playing action, and the players include a target player and other players participating in the current game of Douzheng. The target player is a real player or a virtual player, and the other players include real players and / or virtual players.

[0111] The feature set acquisition module 12 is configured to encode the game information to determine an input feature set. The input feature set includes global state features and card playing action features.

[0112] The first training module 13 is configured to simulate an independent game based on a Monte Carlo simulation algorithm, the input feature set, and a current game strategy model, and introduce a number of times of using a target card pattern as a reward multiplier, to complete self-game model training by using a simulation obtained training sample set, and obtain a trained first strategy model. The current game strategy model includes a card playing action decision component and a next player hand card prediction component.

[0113] The second training module 14 is configured to predict next player hand card information based on the global state features in the training sample set, a preset random drop regularization technique, and the first strategy model, to complete self-game model training of an opponent under a non-perfect information environment simulation, and obtain a trained second strategy model.

[0114] The third training module 15 is configured to train a teacher network based on game win or lose information corresponding to the training sample set, and train the second strategy model by using a corresponding current trained teacher network and a preset game difficulty control mechanism, to determine a trained third strategy model when a win rate of a player in different roles meets a preset training termination condition.

[0115] The fourth training module 16 is configured to train the third strategy model based on the input feature set, a player card playing behavior type, and a preset player card playing action difference sampling mechanism, to determine a trained fourth strategy model, and generate a current card playing decision for the target player by using the fourth strategy model, so that the target player interacts with the other players by using the current card playing decision.

[0116] In some embodiments, the feature set obtaining module 12 can be specifically configured to: encode the calling information, the hand information and the historical discarding action in the game information to determine a global state feature; encode the current discarding action and the player's remaining hand information after the current discarding action to determine a discarding action feature; and determine an input feature set based on the global state feature and the discarding action feature.

[0117] In some embodiments, the interactive device based on the game strategy model can be further configured to: collect different types of off-table target card pattern information, and encode the off-table target card pattern information to determine corresponding target card pattern features.

[0118] In some embodiments, the first training module 13 can be specifically configured to: generate an initial state of a game based on a Monte Carlo simulation algorithm, the input feature set and a current game strategy model to determine an initial state generation result; simulate an entire discarding process from a player who is a land lord based on the current game strategy model, and record state information, action information and target card pattern usage information of each player at each step, as well as a game win or lose result; the target card pattern usage information includes the number of times of using a target card pattern; determine corresponding training reward information based on the game win or lose result, the target card pattern usage information and a preset reward function; determine a corresponding state sequence based on the state information of each player in the entire game; construct a sample by taking the training reward information as a supervision target corresponding to each state in the state sequence to determine a training sample; store the training sample in a buffer pool, and determine a quantity judgment result by judging whether the number of samples in the buffer pool meets a preset number condition; if the quantity judgment result is yes, determine a training sample set based on the training sample in the buffer pool; and perform self-game model training on the current game strategy model by using the training sample set and a preset loss function to obtain a first strategy model after training.

[0119] In some embodiments, the second training module 14 can be specifically configured to: for any player in a game, perform a next player's hand information prediction based on the first strategy model and the current player's hand information and the current player's opponent's hand information to determine a next player's hand feature prediction result; fuse the next player's hand feature prediction result and the global state feature corresponding to the next player's hand information to determine a fused feature; and perform next player's hand information prediction based on the fused feature, a preset random drop regularization technique and the first strategy model to complete self-game training of an opponent modeling in an imperfect information environment simulation, and obtain a second strategy model after training.

[0120] In some embodiments, the third training module 15 can be specifically configured to: obtain a sequence of action selections of three players in a process of simulating an entire card playing process starting from the player as the landowner based on the current game strategy model; take the sequence of action selections as a supervision signal, and take the hand information of the player at the beginning of the game as input information of the model to train the teacher network to obtain a current trained teacher network; the teacher network is used to predict the landowner win rate; based on the current trained teacher network and the game session win / loss information corresponding to the training sample set, session screening is performed to determine a session screening result; the session screening result includes a peasant strategy training sample and a landowner strategy training sample, the peasant strategy training sample is a game session with a landowner win rate greater than a first probability, and the landowner strategy training sample is a game session with a landowner win rate meeting a preset probability interval; based on the session screening result and a preset session difficulty control mechanism, the second strategy model is trained for different roles to determine a third trained strategy model when the win rate of the player in different roles meets a preset training termination condition.

[0121] In some embodiments, the fourth training module 16 can be specifically configured to: based on the input feature set and a preset player card playing action difference sampling mechanism, sample train the third strategy model under different player card playing behavior types to determine a fourth trained strategy model; wherein the player card playing behavior types include an active card playing type and a passive card taking type; the preset player card playing action difference sampling mechanism includes a preset random sampling strategy corresponding to the active card playing type and a preset temperature control sampling strategy corresponding to the passive card taking type.

[0122] Further, the embodiment of the present application also discloses an electronic device, Figure 6 is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the contents in the figure cannot be considered as any limitation on the use range of the present application.

[0123] Figure 6 A structural schematic diagram of an electronic device 20 provided by the embodiment of the present application. The electronic device 20 can specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. Wherein, the memory 22 is used to store a computer program, the computer program is loaded and executed by the processor 21 to realize the related steps in the interactive method based on the game strategy model disclosed in any of the preceding embodiments. In addition, the electronic device 20 in the embodiment can be specifically an electronic computer.

[0124] In this embodiment, the power supply 23 is configured to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 is configured to create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which will not be specifically limited herein; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which will not be specifically limited herein.

[0125] In addition, the memory 22 as a carrier for storing resources can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0126] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the interactive method based on the game strategy model executed by the electronic device 20 disclosed in any of the preceding embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0127] Further, the present application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the interactive method based on the game strategy model disclosed above. For the specific steps of the method, please refer to the corresponding content disclosed in the preceding embodiments, which will not be described here.

[0128] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. For the same or similar parts between each embodiment, please refer to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and please refer to the method part for the relevant part.

[0129] The skilled person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly show the interchangeability of hardware and software, the components and steps of each example have been described in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0130] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The processor can be configured to execute the software module.

[0131] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or importance, but rather are used to distinguish one element from another, and are not necessarily intended to denote the temporal or chronological sequence of the execution of the respective steps. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without further constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0132] The above detailed description of the technical solutions provided by the present application has been described in detail, and the principles and implementation modes of the present application have been described by applying specific examples; the above description of the embodiments is only for the purpose of helping to understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. An interactive method based on a game strategy model, characterized in that, Applied to computer devices, including: Monitor the game information of the current Dou Dizhu game; the game information includes each player's bidding information, hand information, historical playing actions, current playing action, and the player's remaining hand information after the current playing action is executed. Each player includes the target player participating in the current Dou Dizhu game and other players. The target player is a real player or a virtual player, and the other players include real players and / or virtual players. The game information is encoded to determine the input feature set; the input feature set includes global state features and card-playing action features. In a perfect information environment where the hand information of all players participating in the game is known, independent game simulations are conducted based on the Monte Carlo simulation algorithm, the input feature set, and the current game strategy model, with the number of times the target card type is used as a reward multiplier. The training sample set obtained from the simulation is used to complete the training of the self-game model and obtain the first strategy model after training. The current game strategy model includes a card-playing action decision component and a next player's hand prediction component. Based on the global state features in the training sample set, the preset random discard regularization technique, and the first strategy model, the next player's hand information is predicted to complete the opponent modeling self-game training under the imperfect information environment simulation, and the trained second strategy model is obtained. The teacher network is trained based on the game match win / loss information corresponding to the training sample set. The second strategy model is trained using the corresponding current post-training teacher network and the preset match difficulty control mechanism. When the win rate of players in different roles meets the preset training termination condition, the third strategy model after training is determined. The third strategy model is trained based on the input feature set, the player's card-playing behavior type, and the preset player card-playing action difference sampling mechanism to determine the trained fourth strategy model. The fourth strategy model is then used to generate the current card-playing decision for the target player, so that the target player can use the current card-playing decision to interact with other players accordingly.

2. The interaction method based on a game strategy model according to claim 1, characterized in that, The input feature set is determined by encoding based on the game information. The input feature set includes global state features and card-playing action features, including: Encoding is performed based on the bidding information, hand information, and historical card-playing actions in the game information to determine global state characteristics; Encode the current card-playing action and the player's remaining hand information after the current card-playing action is executed based on the game information to determine the characteristics of the card-playing action; The input feature set is determined based on the global state features and the card-playing action features.

3. The interaction method based on a game strategy model according to claim 2, characterized in that, Also includes: Collect different types of off-site target hand information and encode the off-site target hand information to determine the corresponding target hand characteristics.

4. The interaction method based on a game strategy model according to claim 1, characterized in that, The method involves using the Monte Carlo simulation algorithm, the input feature set, and the current game strategy model, and incorporating the number of times the target card type is used as a reward multiplier to conduct independent game simulations. This is done to train the self-play model using the training sample set obtained from the simulations. The method includes: The initial state of the game is generated based on the Monte Carlo simulation algorithm, the input feature set, and the current game strategy model, so as to determine the initial state generation result; Based on the current game strategy model, the entire card-playing process is simulated starting with the player who is the landlord, and the status information, action information, target card type usage information, and game outcome of each player are recorded at each step; the target card type usage information includes the number of times the target card type is used. Based on the game results, the target hand information, and the preset reward function, the corresponding training reward information is determined. The corresponding state sequence is determined based on the state information of each player throughout the game; The training reward information is used as the supervision target for each state in the state sequence to construct samples and determine training samples. The training samples are stored in a buffer pool, and it is determined whether the number of samples in the buffer pool meets the preset quantity condition to determine the quantity judgment result. If the quantity determination result is yes, then the training sample set is determined based on the training samples in the buffer pool; Using the training sample set and a preset loss function, the current game strategy model is trained as a self-play model to obtain the first strategy model after training.

5. The interaction method based on a game strategy model according to claim 1, characterized in that, The step of predicting the next player's hand information based on the global state features in the training sample set, a preset random discard regularization technique, and the first strategy model to complete the opponent modeling self-game training under an imperfect information environment simulation includes: For any player in the game, based on the first strategy model, the current player's hand information, and the hand information of the player before the current player, the hand information of the player after the current player is predicted to determine the prediction result of the hand characteristics of the player after the current player; The predicted hand features of the next player are fused with the global state features corresponding to the hand information of the next player to determine the fused features. Based on the fused features, the preset random discard regularization technique, and the first strategy model, the next player's hand information is predicted to complete the opponent modeling self-game training under the imperfect information environment simulation, and the trained second strategy model is obtained.

6. The interaction method based on a game strategy model according to claim 4, characterized in that, The step of training the teacher network based on the game match win / loss information corresponding to the training sample set, and training the second strategy model using the corresponding currently trained teacher network and a preset match difficulty control mechanism, includes: Based on the current game strategy model, during the process of simulating the entire card-playing process starting from the player whose role is landlord, the action selection sequence corresponding to the three players is obtained; The action selection sequence is used as a supervision signal, and the player's hand information at the start of the game is used as the input information for the model to train the teacher network, so as to obtain the current trained teacher network; the teacher network is used to predict the landlord's winning rate. Based on the game match win / loss information corresponding to the current trained teacher network and the training sample set, the game match selection results are determined; the game match selection results include peasant strategy training samples and landlord strategy training samples, the peasant strategy training samples are game matches where the landlord's win rate is greater than the first probability, and the landlord strategy training samples are game matches where the landlord's win rate meets the preset probability interval. Based on the match selection results and the preset match difficulty control mechanism, the second strategy model is trained for different roles. When the win rate of players in different roles meets the preset training termination condition, the trained third strategy model is determined.

7. The interaction method based on a game strategy model according to any one of claims 1 to 6, characterized in that, The training of the third strategy model based on the input feature set, player card-playing behavior types, and a preset player card-playing action difference sampling mechanism includes: Based on the input feature set and the preset player card-playing action difference sampling mechanism, the third strategy model is sampled and trained under different player card-playing behavior types to determine the trained fourth strategy model; wherein, the player card-playing behavior types include active card-playing type and passive card-receiving type; the preset player card-playing action difference sampling mechanism includes a preset random sampling strategy corresponding to the active card-playing type and a preset temperature control sampling strategy corresponding to the passive card-receiving type.

8. An interactive device based on a game strategy model, characterized in that, Applied to computer devices, including: The game information monitoring module is used to monitor the game information of the current Dou Dizhu game. The game information includes each player's bidding information, hand information, historical playing actions, current playing action, and the player's remaining hand information after the current playing action is executed. Each player includes the target player participating in the current Dou Dizhu game and other players. The target player is a real player or a virtual player, and the other players include real players and / or virtual players. The feature set acquisition module is used to encode the game information to determine the input feature set; the input feature set includes global state features and card-playing action features. The first training module is used to conduct independent game simulations under a perfect information environment where the hand information of all players participating in the game is known. This simulation is based on the Monte Carlo simulation algorithm, the input feature set, and the current game strategy model, and incorporates the number of times the target card type is used as a reward multiplier. The simulation results in the training sample set used to train the self-game model and obtain the trained first strategy model. The current game strategy model includes a card-playing action decision component and a next player's hand prediction component. The second training module is used to predict the next player's hand information based on the global state features in the training sample set, the preset random discard regularization technique and the first strategy model, so as to complete the opponent modeling self-game training under the imperfect information environment simulation and obtain the trained second strategy model. The third training module is used to train the teacher network based on the game match win / loss information corresponding to the training sample set, and to train the second strategy model using the corresponding current post-training teacher network and the preset match difficulty control mechanism, so as to determine the post-training third strategy model when the player win rate of different roles meets the preset training termination condition. The fourth training module is used to train the third strategy model based on the input feature set, the player's card-playing behavior type, and the preset player card-playing action difference sampling mechanism, to determine the trained fourth strategy model, and to use the fourth strategy model to generate the current card-playing decision for the target player, so that the target player can use the current card-playing decision to interact with other players accordingly.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the interactive method based on a game strategy model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the interactive method based on a game strategy model as described in any one of claims 1 to 7.

Images