Interaction method, device and equipment based on game strategy model and medium

Abstract

The invention discloses an interaction method and device based on a game strategy model, equipment and a medium, and relates to the technical field of computers, and the method comprises the steps: monitoring the game information of a current fighter game, and determining an input feature set; simulating an independent game based on the input feature set and a current game strategy model in a perfect information environment that hands and cards of players are known, and determining a trained first strategy model by using the obtained training sample set; opponent modeling self-game training under imperfect information environment simulation is completed based on the global state features and a first strategy model; training the obtained second strategy model by using a teacher network trained based on game winning and losing information; and training the obtained third strategy model based on the input feature set and the player playing behavior type, and generating a current playing decision for the player by using the trained fourth strategy model to perform interaction. The problem that in an existing scheme, the gap between perfect information and imperfect information is difficult to meet is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to an interactive method and device based on a game strategy model, equipment and a medium. BACKGROUND

[0002] Traditional Dou Di Zhu AI (Artificial Intelligence) training schemes mostly rely on rule-based strategies or simple supervised learning methods, which are difficult to adapt to the changing strategy game needs in dynamic environments, resulting in poor interactive experience. In recent years, reinforcement learning schemes based on self-play mechanisms have gradually become mainstream, but in the incomplete information game of Dou Di Zhu, traditional self-play schemes still face the following challenges:

[0003] (1) The gap between perfect information and imperfect information is difficult to bridge. In actual games, players can only know their own information and cannot obtain the opponent's hand information and other implicit information, which makes strategy learning susceptible to noise interference, slow in convergence, and weak in generalization ability;

[0004] (2) The imbalance of action space leads to low sampling efficiency. In Dou Di Zhu, the difference between active card playing and passive card taking in the decision space is significant, but the traditional scheme uses a unified sampling mechanism for different types of actions, which often cannot balance exploration and stability, resulting in low strategy update efficiency. SUMMARY

[0005] Therefore, the purpose of the present application is to provide an interactive method and device based on a game strategy model, which can solve the problem of the gap between perfect information and imperfect information being difficult to bridge and the imbalance of action space leading to low sampling efficiency in existing schemes, thereby effectively realizing efficient, stable, and continuous automation and reliable training of the Dou Di Zhu game strategy model, improving the adaptability to the changing strategy game needs in dynamic environments and the action sampling efficiency, and improving the experience of players in the game. The specific scheme is as follows: In a first aspect, the present application provides an interactive method based on a game strategy model, applied to a computer device, comprising: monitoring game information of a current Dou Di Zhu game; the game information includes call card information, hand information, historical card playing actions, current card playing actions, and player remaining hand information after the current card playing actions, the players include a target player and other players participating in the current Dou Di Zhu game, the target player is a real player or a virtual player, and the other players include real players and / or virtual players; encoding based on the game information to determine an input feature set; the input feature set includes global state features and card playing action features; In a perfect information environment where the hand information of each player participating in the game is known, based on a Monte Carlo simulation algorithm, the input feature set and the current game strategy model, and introducing the number of times of using the target card pattern as a reward multiplier to perform independent game simulation, to complete self-game model training using the simulation obtained training sample set, and obtain the first trained strategy model; the current game strategy model includes a card playing action decision component and a next hand hand information prediction component; Based on the global state features in the training sample set, a preset random dropout regularization technique and the first strategy model, the next hand hand information is predicted to complete the opponent modeling self-game training under the simulation of the imperfect information environment, and the second trained strategy model is obtained; Based on the game match win or lose information corresponding to the training sample set, the teacher network is trained, and the corresponding current trained teacher network and the preset game difficulty control mechanism are used to train the second strategy model, so as to determine the third trained strategy model when the win rate of the player in different roles meets the preset training termination condition. Based on the input feature set, the player card playing behavior type and the preset player card playing action difference sampling mechanism, the third strategy model is trained to determine the fourth trained strategy model, and the fourth strategy model is used to generate the current card playing decision for the target player, so that the target player uses the current card playing decision to interact with the other players.

[0006] Optionally, the game information is encoded to determine the input feature set; the input feature set includes global state features and card playing action features, including: The calling information, the hand information and the historical card playing action in the game information are encoded to determine the global state features; The current card playing action and the player's remaining hand information after the execution of the current card playing action in the game information are encoded to determine the card playing action features; The input feature set is determined based on the global state features and the card playing action features.

[0007] Optionally, the method further comprises: Collect different types of off-site target card pattern information, and encode the off-site target card pattern information to determine the corresponding target card pattern features.

[0008] Optionally, based on the Monte Carlo simulation algorithm, the input feature set and the current game strategy model, and introducing the number of times of using the target card pattern as a reward multiplier to perform independent game simulation, to complete self-game model training using the simulation obtained training sample set, including: generating an initial state of a game based on a Monte Carlo simulation algorithm, the set of input features, and a current game strategy model to determine an initial state generation result; based on the current game strategy model, simulating an entire card playing process starting from the player who is the land lord, and recording state information, action information, and target tile pattern usage information of each player at each step, and a game win or lose result; the target tile pattern usage information includes the number of times of using a target tile pattern; based on the game win or lose result, the target tile pattern usage information, and a preset reward function, determining corresponding training reward information; based on the state information of each player in the entire game, determining a corresponding state sequence; using the training reward information as a supervision target corresponding to each state in the state sequence, constructing a sample to determine a training sample; storing the training sample in a buffer pool, and determining a quantity determination result by judging whether the number of samples in the buffer pool meets a preset number condition; if the quantity determination result is yes, determining a training sample set based on the training sample in the buffer pool; using the training sample set and a preset loss function to perform self-game model training on the current game strategy model to obtain a first trained strategy model.

[0009] Optionally, the hand information prediction of the next player based on the global state feature in the training sample set, a preset random dropout regularization technique, and the first strategy model is performed to complete the modeling of the opponent in the self-game training under the non-perfect information environment simulation, including: for any player in the game, based on the first strategy model and the hand information of the current player and the hand information of the current player's opponent, predicting the hand information of the next player of the current player to determine a next player hand feature prediction result; fusing the next player hand feature prediction result and the global state feature corresponding to the hand information of the next player to determine a fused feature; based on the fused feature, a preset random dropout regularization technique, and the first strategy model, predicting the hand information of the next player to complete the modeling of the opponent in the self-game training under the non-perfect information environment simulation, and obtaining a second trained strategy model.

[0010] Optionally, the training of a teacher network based on the game win or lose information corresponding to the training sample set is performed, and the second strategy model is trained using the corresponding current trained teacher network and a preset game difficulty control mechanism, including: In the process of simulating the entire card playing process based on the current game strategy model, starting from the player as the landlord, the action selection sequences of the three players are obtained; The action selection sequences are used as supervision signals, and the hand information of the players at the beginning of the game is used as input information of the model to train the teacher network to obtain a current trained teacher network; the teacher network is used to predict the landlord winning rate; Based on the current trained teacher network and the game session winning and losing information corresponding to the training sample set, game session screening is performed to determine a game session screening result; the game session screening result includes a peasant strategy training sample and a landlord strategy training sample, the peasant strategy training sample is a game session with a landlord winning rate greater than a first probability, and the landlord strategy training sample is a game session with a landlord winning rate meeting a preset probability interval; Based on the game session screening result and a preset game session difficulty control mechanism, the second strategy model is trained for different roles to determine a third trained strategy model when the winning rate of the player of different roles meets a preset training termination condition.

[0011] Optionally, the training of the third strategy model based on the input feature set, player card playing behavior type, and preset player card playing action difference sampling mechanism includes: Based on the input feature set and the preset player card playing action difference sampling mechanism, the third strategy model is sampled and trained under different player card playing behavior types to determine a fourth trained strategy model; wherein the player card playing behavior type includes an active card playing type and a passive card taking type; the preset player card playing action difference sampling mechanism includes a preset random sampling strategy corresponding to the active card playing type and a preset temperature controlled sampling strategy corresponding to the passive card taking type.

[0012] In a second aspect, the present application provides an interactive device based on a game strategy model, applied to a computer device, including: A game information monitoring module is configured to monitor game information of a current Dou Di Zhu game session; the game information includes calling card information, hand information, historical card playing actions, current card playing actions, and player remaining hand information after execution of the current card playing actions of each player; the players include a target player and other players participating in the current Dou Di Zhu game session; the target player is a real player or a virtual player; the other players include real players and / or virtual players; A feature set acquisition module is configured to encode the game information to determine an input feature set; the input feature set includes global state features and card playing action features; The first training module is configured to perform independent game simulation based on a Monte Carlo simulation algorithm, the input feature set, and a current game strategy model, and introduce the number of times of using a target card pattern as a reward multiplier, to complete self-game model training by using a simulation obtained training sample set, and obtain a first trained strategy model; the current game strategy model includes a card playing action decision component and a next player hand information prediction component; The second training module is configured to perform next player hand information prediction based on the global state feature in the training sample set, a preset random drop regularization technique, and the first strategy model, to complete opponent modeling self-game training under imperfect information environment simulation, and obtain a second trained strategy model; The third training module is configured to perform teacher network training based on game match win or lose information corresponding to the training sample set, and train the second strategy model by using a corresponding current trained teacher network and a preset match difficulty control mechanism, to determine a third trained strategy model when a win rate of a player in different roles meets a preset training termination condition. The fourth training module is configured to train the third strategy model based on the input feature set, player card playing behavior types, and a preset player card playing action difference sampling mechanism, to determine a fourth trained strategy model, and generate a current card playing decision for the target player by using the fourth trained strategy model, so that the target player performs corresponding interaction with the other players by using the current card playing decision.

[0013] In a third aspect, the present application provides an electronic device, comprising: a memory configured to save a computer program; a processor configured to execute the computer program to implement steps of the above-mentioned game strategy model based interaction method.

[0014] In a fourth aspect, the present application provides a computer readable storage medium configured to save a computer program, the computer program being executed by a processor to implement steps of the above-mentioned game strategy model based interaction method.

[0015] It can be seen that, in the present application, the game information of the current domino game is monitored by applying the computer device, the game information includes the calling card information, the hand card information, the historical card action, the current card action and the remaining hand card information of the players after the execution of the current card action, the players include the target player and other players participating in the current domino game, the target player is a real player or a virtual player, and the other players include real players and / or virtual players; the game information is encoded to determine the input feature set; the input feature set includes global state features and card action features; in the perfect information environment where the hand card information of each player participating in the game is known, the Monte Carlo simulation algorithm, the input feature set and the current game strategy model are used to simulate independent games by introducing the number of times of using the target card type as a reward multiplier, to complete the self-game model training by using the simulation obtained training sample set, and to obtain the first trained strategy model; the current game strategy model includes a card action decision component and a next hand card prediction component; the global state features in the training sample set, the preset random drop regularization technology and the first strategy model are used to predict the next hand card information, to complete the opponent modeling self-game training in the imperfect information environment simulation, and to obtain the second trained strategy model; the game win or lose information corresponding to the training sample set is used to train the teacher network, and the corresponding current trained teacher network and the preset game difficulty control mechanism are used to train the second strategy model, to determine the third trained strategy model when the win rate of the players in different roles meets the preset training termination condition; the input feature set, the player card action type and the preset player card action difference sampling mechanism are used to train the third strategy model to determine the fourth trained strategy model, and the fourth strategy model is used to generate the current card decision for the target player, so that the target player interacts with the other players by using the current card decision.That is, in the present application, the game information of the mahjong game is first collected and encoded to obtain an input feature set, then in the perfect information environment, the input feature set and the current game strategy model are used to simulate independent games based on the use frequency of the target card pattern as a reward multiplier, to train the self-game model using the simulation training sample set, and obtain the first strategy model, then the global state feature, the preset random drop regularization technology and the first strategy model are used to predict the hand information of the next player, to complete the opponent modeling self-game training in the non-perfect information environment simulation, and obtain the second strategy model, then the teacher network is trained using the game win or lose information, and the second strategy model is trained using the current trained teacher network, to determine the third strategy model when the win rate of the player in different roles meets the preset training termination condition, then the third strategy model is trained according to the player's card behavior type and the preset player card action difference sampling mechanism, and the fourth strategy model obtained after training is used to generate card decision for the target player to interact with other players. In this way, the gap between perfect information and non-perfect information in the prior art can be bridged, and the low sampling efficiency caused by unbalanced action space can be solved, so that efficient, stable, continuous and reliable mahjong game strategy model training is effectively realized, and the adaptability to dynamic environment and the action sampling efficiency of the strategy game demand are improved, and the experience of the players in the game is improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0017] Figure 1 A flowchart of an interactive method based on a game strategy model is provided for the present application; Figure 2 A flowchart of a specific interactive method based on a game strategy model is provided for the present application; Figure 3 A schematic diagram of the coupling and separation process of perfect information and non-perfect information model is provided for the present application; Figure 4 A course learning process schematic diagram is provided for the present application; Figure 5 A structural schematic diagram of an interactive device based on a game strategy model is provided for the present application; Figure 6 A structural diagram of an electronic device is provided for the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be apparently and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0019] The traditional self-game-playing scheme still faces the following challenges: (1) the gap between perfect information and imperfect information is difficult to bridge. In actual games, players can only know their own information and cannot obtain the hidden information such as the hand of the opponent, which leads to that the strategy learning is easily disturbed by noise, the convergence speed is slow, and the generalization ability is weak; (2) the unbalanced action space leads to low sampling efficiency. In Douzhin, the difference between active card playing and passive card taking in the decision space is significant, however, the traditional scheme adopts a unified sampling mechanism for different types of actions, which often cannot balance the exploratory and stability, leading to low strategy updating efficiency.

[0020] Therefore, the present application provides an interactive scheme based on a game strategy model, which can solve the problems of the gap between perfect information and imperfect information being difficult to bridge and the unbalanced action space leading to low sampling efficiency in the prior art, thereby effectively realizing efficient, stable, and continuous automation and reliable training of the Douzhin game strategy model, improving the adaptability to the changing strategy game demand in the dynamic environment and the action sampling efficiency, and improving the experience of players in the game.

[0021] Referring to Figure 1 The embodiments of the present application disclose an interactive method based on a game strategy model, applied to a computer device, comprising: Step S11, monitoring game information of a current Douzhin game; the game information comprises calling card information, hand information, historical card playing actions, a current card playing action, and player remaining hand information after the current card playing action is executed, the players comprise a target player and other players participating in the current Douzhin game, the target player is a real player or a virtual player, and the other players comprise real players and / or virtual players.

[0022] In the embodiments, first, the game is monitored to obtain the calling card information, the hand information, the historical card playing actions, the current card playing action, and the player remaining hand information after the current card playing action is executed of each player in the current Douzhin game, the players comprise a target player and other players participating in the current Douzhin game, the target player is a real player or a virtual player, and the other players comprise real players and / or virtual players. In this way, a foundation is laid for the subsequent feature construction step.

[0023] Step S12, encode the game information to determine the input feature set; the input feature set includes global state features and card playing action features.

[0024] In this embodiment, the game information is acquired and encoded to obtain the input feature set, that is, the calling information, the hand information and the historical card playing actions in the game information are encoded to determine the global state features; the current card playing action and the player's remaining hand information after the current card playing action is performed in the game information are encoded to determine the card playing action features; and the global state features and the card playing action features are used to determine the input feature set. Figure 2 As shown in the figure, the game information is acquired and encoded to obtain the input feature set, that is, the calling information, the hand information and the historical card playing actions in the game information are encoded to determine the global state features; the current card playing action and the player's remaining hand information after the current card playing action is performed in the game information are encoded to determine the card playing action features; and the global state features and the card playing action features are used to determine the input feature set. Different kinds of off-table target card pattern information are collected, and the off-table target card pattern information is encoded to determine the corresponding target card pattern features. The off-table target card pattern information refers to the target card pattern information that has been played by the player, and the target card pattern includes a card pattern composed of four cards with the same value and a card pattern composed of a big king card and a small king card.

[0025] Specifically, regarding the construction of features, in this embodiment, a binary decision matrix based on the player order is constructed, the first row of the matrix is the selection in the "call the landlord" stage, and the second row is the selection in the "rob the landlord" stage. Each column corresponds to a player, and the matrix element takes a value of 1 indicating selection (call or rob), or 0 indicating abandonment. Taking the first player calling the landlord and the three players all selecting to rob the landlord as an example, the process can be represented as the following matrix: .

[0026] For the off-table target card pattern features, in this embodiment, a binary target card pattern feature vector is designed: each dimension corresponds to a possible target card pattern type, and if the target card pattern of this type has appeared off-table, the corresponding element is recorded as 1, otherwise as 0. For example, if it is known that the "four three card pattern", "four five card pattern" and "big and small king card pattern" have appeared off-table, the target card pattern feature vector is represented as (1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1).

[0027] In addition, for the player's hand, the current action, the remaining hand after the current action, the historical action and other features, a 54-dimensional vector encoding is used to encode the number of each card.

[0028] Step S13, in the perfect information environment where the hand information of each player participating in the game is known, based on the Monte Carlo simulation algorithm, the input feature set and the current game strategy model, and introducing the number of times of using the target card type as a reward multiplier, independent game simulation is performed to complete self-game model training using the training sample set obtained by simulation, and a first trained strategy model is obtained; the current game strategy model includes a card playing action decision component and a next player hand prediction component.

[0029] In this embodiment, as shown in Figure 2 After the feature construction is completed, the features are used for perfect information self-game model training, that is, based on the Monte Carlo simulation algorithm, the input feature set and the current game strategy model, the initial state of the game is generated to determine the initial state generation result; based on the current game strategy model, the entire card playing process is simulated from the player as the landlord, and the state information, action information and target card type usage information of each player at each step are recorded, as well as the game win or lose result; the target card type usage information includes the number of times of using the target card type; based on the game win or lose result, the target card type usage information and a preset reward function, the corresponding training reward information is determined; based on the state information of each player in the entire game, the corresponding state sequence is determined; the training reward information is used as the supervision target corresponding to each state in the state sequence, sample construction is performed to determine the training sample; the training sample is stored in a buffer pool, and it is determined whether the number of samples in the buffer pool meets a preset number condition to determine a number determination result; if the number determination result is yes, the training sample set is determined based on the training sample in the buffer pool; the training sample set and a preset loss function are used to perform self-game model training on the current game strategy model to obtain a first trained strategy model.

[0030] Specifically, as shown in Figure 3 In this embodiment, regarding the perfect information self-game model training process, in the perfect information environment where the hand information of three players is known, a high-quality game data is generated based on the Monte Carlo simulation method, and the number of times of using the target card type is introduced as a reward multiplier to optimize the signal quality of strategy training. Under this setting, the strategy model is used for independent game simulation, and the specific steps include: 1) initial state generation: random card distribution is adopted to ensure that the hands of three players have no overlap, the card calling process is performed, the landlord and the farmer are determined, and the rules of the game are met; 2) strategy driven simulation: according to the current strategy model, the entire card playing process is automatically executed from the landlord, and the state, action and target card type usage information at each step are recorded; 3) state trajectory recording: for each player, all states in the card playing sequence are recorded , construct state-return samples; 4), result evaluation: based on the result of the game and the number of times of using the target card type, calculate the final reward required for training; reward mechanism: composite reward based on the result of the game and the number of times of using the target card type. In order to enhance the sensitivity of the reward signal to the use of the key target card type, the composite reward function is designed as follows: ; In the formula, is the final training reward of the i-th player; is the result of the i-th player in the current game (win is +1, lose is -1); is the number of times of using the target card type by the i-th player in the game; is the target card type sensitivity coefficient, which is used to control the amplification degree of the target card type to the reward signal.

[0031] Further, regarding the data construction and training signal generation in the process, for each player i, the state sequence of the entire game is extracted , and the composite reward is used as the supervision target of each state in the sequence to form the training sample set : .

[0032] In the formula, is the state sequence of the i-th player at time T in the entire game.

[0033] In the strategy model training phase, the mean square error loss function is used to optimize the state value function V(s), and the specific goal is as follows: .

[0034] In the formula, is the i-th sample state, and N is the sample number; is the composite reward corresponding to the player i.

[0035] Step S14, based on the global state features in the training sample set, the preset random dropout regularization technology and the first strategy model, the information prediction of the next hand is performed to complete the opponent modeling self-game training under the non-perfect information environment simulation, and a trained second strategy model is obtained.

[0036] In combination with Figure 3As shown, after completing the perfect information self-game model training, the embodiment will also perform non-perfect information opponent modeling self-game training, that is, for any player in the game session, based on the first strategy model and the hand information of the current player and the hand information of the current player's upper player, the hand information of the current player's lower player is predicted to determine the lower player's hand feature prediction result; the lower player's hand feature prediction result and the global state feature corresponding to the hand information of the lower player are fused to determine the fused feature; based on the fused feature, a preset random dropout regularization technique and the first strategy model, the lower player's hand information is predicted to complete the opponent modeling self-game training under the non-perfect information environment simulation, and a trained second strategy model is obtained. The preset random dropout regularization technique can be a step-by-step Dropout mechanism.

[0037] It should be understood that in the process of non-perfect information opponent modeling self-game training, in the embodiment, the opponent modeling self-game aims to simulate the strategy learning process under the real non-perfect information environment. Based on the global game state feature, the hand information that the lower player may hold is estimated by the prediction module, so as to construct the feature expression under the non-perfect information.

[0038] Specifically, the model is first trained using the game data under perfect information, and on this basis, the hand prediction task is introduced, and the estimated hand feature is obtained by modeling the probability of the lower player's hand. Then, the predicted hand feature and the real hand information are fused for feature, which is used to train the strategy network. In order to gradually transition to a completely non-perfect information environment, a step-by-step Dropout mechanism is used in the training process, that is, the proportion of dependence on real hand (perfect information) is gradually reduced in multiple stages. In the final stage, the model outputs the strategy based on the predicted feature only, thereby having the generalization ability of modeling unknown opponents in the real game environment.

[0039] In this way, the opponent modeling self-game mechanism realizes the migration from the ideal training environment to the real game environment, and strengthens the robustness and game adaptability of the model under non-perfect information conditions.

[0040] Step S15, based on the game session win or lose information corresponding to the training sample set, the teacher network is trained, and the corresponding current trained teacher network and the preset game difficulty control mechanism are used to train the second strategy model, so as to determine the third strategy model after training when the win rate of the player in different roles satisfies the preset training termination condition.

[0041] In the embodiment, the combination of Figure 2As shown, course learning will also be carried out, using the win / loss information of the training game to train a coach network (i.e., a teacher network) to predict the winning rate of the host, so as to realize the evaluation of the card strength value, that is, in the process of simulating the entire card distribution process based on the current game strategy model starting from the player who is the host, the action selection sequence corresponding to the three players is obtained; the action selection sequence is used as a supervision signal, and the hand information of the player at the beginning of the game is used as input information of the model, and the teacher network is trained to obtain the current trained teacher network; the teacher network is used to predict the winning rate of the host; based on the current trained teacher network and the game win / loss information corresponding to the training sample set, game screening is carried out to determine the game screening result; the game screening result includes a peasant strategy training sample and a host strategy training sample, the peasant strategy training sample is a game with a host winning rate greater than a first probability, and the host strategy training sample is a game with a host winning rate meeting a preset probability interval; based on the game screening result and a preset game difficulty control mechanism, the second strategy model is trained for different roles respectively, so that when the winning rate of the player of different roles meets a preset training termination condition, a third trained strategy model is determined.

[0042] Specifically, in combination with Figure 4 As shown, in the process of course learning, a game difficulty control mechanism is introduced to realize a stable training process of the strategy model from easy to difficult. This step includes two core modules: a coach network and a course sample scheduling strategy.

[0043] Under the condition of perfect information, a large number of games are sampled using a high-quality strategy based on MCTS (Monte Carlo Tree Search), and whether the host wins in the end is recorded under the initial state of each game. A coach network is trained using this as a supervision signal to predict the host's winning rate corresponding to any initial hand configuration, which is used as an indicator to measure the difficulty of the current game. The input of the network is the initial state (including the hand of the host and the peasant), and the output is the estimated value of the host's winning rate.

[0044] In the strategy model training phase, sample selection is performed according to the host winning rate score provided by the coach network to realize a course-based training process: (1) Peasant strategy training: preferentially select games with a host winning rate > a (optional value 60%) as the starting point of training, i.e., the peasant is in a relatively favorable position. As the training progresses, the difficulty of the confrontation is gradually increased, and the training samples are expanded to high-pressure scenarios with a host winning rate of ≥ 90% to enhance the peasant's ability to respond to unfavorable situations; (2) Landlord strategy training: Start training from a medium-difficulty game with a landlord win rate of about 40%, and gradually transition to a difficult sample with a win rate of only 10%, guiding the model to cope with complex and disadvantaged scenarios, thereby improving the robustness and generalization ability of the strategy.

[0045] Step S16, based on the input feature set, player card playing behavior type and preset player card playing action difference sampling mechanism, the third strategy model is trained, the fourth strategy model after training is determined, and the fourth strategy model is used to generate the current card playing decision for the target player, so that the target player uses the current card playing decision to interact with the other players.

[0046] In combination Figure 2 In the embodiment, the difference sampling of different card playing behaviors is also trained, that is, based on the input feature set and the preset player card playing action difference sampling mechanism, the third strategy model is sampled and trained under different player card playing behavior types to determine the fourth strategy model after training; wherein the player card playing behavior type includes an active card playing type and a passive card taking type; the preset player card playing action difference sampling mechanism includes a preset random sampling strategy corresponding to the active card playing type and a preset temperature control sampling strategy corresponding to the passive card taking type.

[0047] It should be understood that in the process of training the difference sampling of different card playing behaviors, in the game of Dou Dizhu, card playing behaviors can be divided into two categories: active card playing and passive card taking, and there are significant differences in the degree of freedom of strategy selection and the value of game. In order to more effectively guide the strategy exploration while maintaining the stability of the sample distribution and the diversity of the strategy output in the training process, a set of difference sampling mechanism is designed, as shown below: (1) Active card playing: a random sampling strategy is used.

[0048] When the player is in the active card playing position (i.e. no upper limit, can freely choose to play cards), the strategy network faces a larger action space and game branches. In order to enhance the exploratory, a greedy ( greedy strategy) is used for action sampling: with a certain probability select the action with the highest probability in the current strategy output, and in other cases, randomly sample from the available actions. This mechanism can avoid the strategy falling into early local optimum in the active stage, and improve the diversity and generalization ability of the strategy.

[0049] (2) Passive card taking: a temperature-controlled soft sampling strategy is used.

[0050] During the passive play phase (i.e., the card played must be higher than the previous player's), the range of cards played is limited. To better balance exploration and stability, a temperature sampling strategy based on Softmax is adopted. : ; in, The action score output by the policy network; Indicates a specific action that is currently being calculated; This represents summing over the entire set of actions; This is a temperature parameter. Higher... Values ​​can increase sampling diversity, lower values The value tends to favor greedy choices.

[0051] Furthermore, this embodiment includes steps to accelerate the aforementioned training process. In terms of system implementation, this embodiment addresses both data transmission and model parallelism, designing a highly efficient training acceleration mechanism to improve model iteration efficiency and computational resource utilization. Details are as follows: (1) Asynchronous tensor transfer mechanism.

[0052] In the data preprocessing stage, an asynchronous non-blocking tensor transfer strategy is adopted: after the CPU (Central Processing Unit) completes data generation and sample encoding, it uses... This asynchronous method transfers data tensors to the GPU (Graphics Processing Unit), avoiding the blocking of GPU inference due to data loading and improving overall throughput. This mechanism fully utilizes PyTorch's tensor streaming mechanism (torch.cuda.Stream) and... Preprocessing options enable parallel execution of data loading and model inference, effectively reducing data communication bottlenecks between the CPU and GPU.

[0053] (2) Multi-card, multi-process distributed parallel training.

[0054] The training phase employs a distributed training approach based on a multi-GPU, multi-process architecture. Each training process is bound to a separate GPU, and gradient synchronization and model parameter consistency are maintained through the distributed training mechanism of torch.nn.parallel.Distributed Data Parallel (DDP, a distributed training component provided by PyTorch). This architecture has good scalability, supports multi-machine, multi-GPU deployment, significantly accelerates model convergence, and improves training stability.

[0055] To further improve parallel efficiency, the training system adopts a main process to control the training logic in implementation, each sub-process asynchronously obtains training samples through a shared queue, and independently completes forward inference and gradient calculation. High-speed communication between GPUs is completed through the NCCL (NVIDIA Collective Communications Library, a high-performance communication library developed by NVIDIA) backend, supporting stable expansion of the model in large-scale data scenarios.

[0056] As can be seen, in the present application, the game information of the Mahjong game is first collected and encoded to obtain an input feature set, then in a perfect information environment, based on the input feature set and the current game strategy model, the number of uses of the target card pattern is introduced as a reward multiplier for independent game simulation, to utilize the simulation obtained training sample set to complete self-game model training and obtain a first strategy model, thereafter, based on the global state feature, the preset random dropout regularization technique and the first strategy model, the next hand card information is predicted to complete the opponent modeling self-game training in the non-perfect information environment simulation to obtain a second strategy model, thereafter, the teacher network is trained using the game win / loss information, and the second strategy model is trained using the current trained teacher network, to determine a third strategy model when the win rate of the players in different roles meets the preset training termination condition, thereafter, the third strategy model is trained according to the player's card behavior type and the preset player card action difference sampling mechanism, and the fourth strategy model obtained after training is used to generate card decision for the target player to interact with other players in the game. In this way, the gap between perfect information and non-perfect information in the prior art can be bridged, and the low sampling efficiency caused by unbalanced action space can be solved, thereby effectively realizing efficient, stable, continuous automation and reliable training of the Mahjong game strategy model, and improving the adaptability to the dynamic environment and the action sampling efficiency of the changing strategy game demand, and improving the experience of the players in the game.

[0057] Referring to Figure 5 The embodiments of the present application also correspondingly disclose an interactive device based on a game strategy model, applied to a computer device, comprising: A game information monitoring module 11 is configured to monitor game information of a current Mahjong game, wherein the game information comprises calling card information, hand card information, historical card action, current card action, and player remaining hand card information after execution of the current card action of each player, the players include a target player and other players participating in the current Mahjong game, the target player is a real player or a virtual player, and the other players include real players and / or virtual players; The feature set acquisition module 12 is configured to encode the game information to determine an input feature set, wherein the input feature set comprises a global state feature and a card playing action feature. The first training module 13 is configured to, in a perfect information environment where the hand information of each player participating in the game is known, perform independent game simulation based on a Monte Carlo simulation algorithm, the input feature set, and a current game strategy model, and introduce the number of times of using a target card pattern as a reward multiplier, to complete self-game model training by using a simulation obtained training sample set, and obtain a trained first strategy model, wherein the current game strategy model comprises a card playing action decision component and a next hand information prediction component. The second training module 14 is configured to, based on the global state feature in the training sample set, a preset random drop regularization technique, and the first strategy model, predict the next hand information, to complete self-game model training under a non-perfect information environment simulation, and obtain a trained second strategy model. The third training module 15 is configured to train a teacher network based on the game win or lose information corresponding to the training sample set, and train the second strategy model by using a corresponding current trained teacher network and a preset game difficulty control mechanism, to determine a trained third strategy model when the win rate of a player in different roles meets a preset training termination condition. The fourth training module 16 is configured to train the third strategy model based on the input feature set, a player card playing behavior type, and a preset player card playing action difference sampling mechanism, to determine a trained fourth strategy model, and generate a current card playing decision for the target player by using the fourth strategy model, so that the target player performs corresponding interaction with the other players by using the current card playing decision.

[0058] In some embodiments, the feature set acquisition module 12 can be specifically configured to encode the calling information, the hand information, and the historical card playing action in the game information to determine a global state feature, encode the current card playing action and the player's remaining hand information after the current card playing action is performed in the game information to determine a card playing action feature, and determine an input feature set based on the global state feature and the card playing action feature.

[0059] In some embodiments, the interaction device based on the game strategy model can also be configured to collect different types of off-site target card pattern information, and encode the off-site target card pattern information to determine corresponding target card pattern features.

[0060] In some specific embodiments, the first training module 13 can be used to: generate the initial state of the game based on the Monte Carlo simulation algorithm, the input feature set, and the current game strategy model, to determine the initial state generation result; simulate the entire card-playing process starting from the player whose role is the landlord, based on the current game strategy model, and record the state information, action information, and target card type usage information of each player at each step, as well as the game's win / loss result; the target card type usage information includes the number of times the target card type is used; and determine the corresponding training reward based on the game's win / loss result, the target card type usage information, and a preset reward function. The process involves: 1) determining the corresponding state sequence based on the state information of each player throughout the game; 2) constructing training samples by using the training reward information as the supervision target for each state in the state sequence; 3) storing the training samples in a buffer pool and determining whether the number of samples in the buffer pool meets a preset quantity condition; 4) determining the quantity judgment result if the quantity judgment result is yes, and then determining a training sample set based on the training samples in the buffer pool; and 5) using the training sample set and a preset loss function to train a self-play model on the current game strategy model to obtain the trained first strategy model.

[0061] In some specific embodiments, the second training module 14 can be specifically used to: for any player in the game, based on the first strategy model, the current player's hand information, and the hand information of the player before the current player, predict the hand information of the player after the current player to determine the prediction result of the player after the current player's hand features; fuse the prediction result of the player after the current player's hand features with the global state features corresponding to the hand information of the player after the current player to determine the fused features; and predict the hand information of the player after the current player based on the fused features, a preset random discard regularization technique, and the first strategy model to complete the opponent modeling self-game training under the imperfect information environment simulation, and obtain the trained second strategy model.

[0062] In some embodiments, the third training module 15 can be specifically configured to: obtain a sequence of action selections of three players in a process of simulating an entire card playing process starting from the player as the landowner based on the current game strategy model; take the sequence of action selections as a supervision signal, and take the hand information of the player at the beginning of the game as input information of the model to train the teacher network to obtain a current trained teacher network; the teacher network is used to predict the landowner winning rate; based on the current trained teacher network and the game session win / loss information corresponding to the training sample set, session screening is performed to determine a session screening result; the session screening result includes a peasant strategy training sample and a landowner strategy training sample, the peasant strategy training sample is a game session with a landowner winning rate greater than a first probability, and the landowner strategy training sample is a game session with a landowner winning rate meeting a preset probability interval; based on the session screening result and a preset session difficulty control mechanism, the second strategy model is trained for different roles to determine a third trained strategy model when the winning rate of the player in different roles meets a preset training termination condition.

[0063] In some embodiments, the fourth training module 16 can be specifically configured to: based on the input feature set and a preset player card playing action difference sampling mechanism, sample train the third strategy model under different player card playing behavior types to determine a fourth trained strategy model; wherein the player card playing behavior types include an active card playing type and a passive card taking type; the preset player card playing action difference sampling mechanism includes a preset random sampling strategy corresponding to the active card playing type and a preset temperature control sampling strategy corresponding to the passive card taking type.

[0064] Further, the embodiment of the present application also discloses an electronic device, Figure 6 is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the contents in the figure cannot be considered as any limitation on the use range of the present application.

[0065] Figure 6 A structural schematic diagram of an electronic device 20 provided by the embodiment of the present application. The electronic device 20 can specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. Wherein, the memory 22 is used to store a computer program, the computer program is loaded and executed by the processor 21 to realize the related steps in the interactive method based on the game strategy model disclosed in any of the preceding embodiments. In addition, the electronic device 20 in the embodiment can be specifically an electronic computer.

[0066] In this embodiment, the power supply 23 is configured to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 is configured to create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which will not be specifically limited herein; the input and output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which will not be specifically limited herein.

[0067] In addition, the memory 22 as a carrier of resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0068] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the interactive method based on the game strategy model executed by the electronic device 20 disclosed in any of the preceding embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0069] Further, the present application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the interactive method based on the game strategy model disclosed above. For the specific steps of the method, please refer to the corresponding content disclosed in the preceding embodiments, which will not be described here.

[0070] In the specification, each embodiment is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. For the same or similar parts between the embodiments, please refer to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and please refer to the method part for the relevant part.

[0071] The skilled person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly show the interchangeability of hardware and software, the components and steps of each example have been described in the above description. Whether the functions are realized by hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0072] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The

[0073] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or importance, but rather are used to distinguish one element from another, and do not imply or require any such actual relationship or order. Moreover, the terms "include", "contain", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the phrase "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0074] The above provides a detailed description of the technical solutions of the present application. The principles and implementation modes of the present application are described by applying specific examples. The above description of the embodiments is only to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. An interactive method based on a game strategy model, characterized in that, Applied to computer devices, including: Monitor the game information of the current Dou Dizhu game; the game information includes each player's bidding information, hand information, historical playing actions, current playing action, and the player's remaining hand information after the current playing action is executed. Each player includes the target player participating in the current Dou Dizhu game and other players. The target player is a real player or a virtual player, and the other players include real players and / or virtual players. The game information is encoded to determine the input feature set; the input feature set includes global state features and card-playing action features. In a perfect information environment where the hand information of all players participating in the game is known, independent game simulations are conducted based on the Monte Carlo simulation algorithm, the input feature set, and the current game strategy model, with the number of times the target card type is used as a reward multiplier. The training sample set obtained from the simulation is used to complete the training of the self-game model and obtain the first strategy model after training. The current game strategy model includes a card-playing action decision component and a next player's hand prediction component. Based on the global state features in the training sample set, the preset random discard regularization technique, and the first strategy model, the next player's hand information is predicted to complete the opponent modeling self-game training under the imperfect information environment simulation, and the trained second strategy model is obtained. The teacher network is trained based on the game match win / loss information corresponding to the training sample set. The second strategy model is trained using the corresponding current post-training teacher network and the preset match difficulty control mechanism. When the win rate of players in different roles meets the preset training termination condition, the third strategy model after training is determined. The third strategy model is trained based on the input feature set, the player's card-playing behavior type, and the preset player card-playing action difference sampling mechanism to determine the trained fourth strategy model. The fourth strategy model is then used to generate the current card-playing decision for the target player, so that the target player can use the current card-playing decision to interact with other players accordingly.

2. The interaction method based on a game strategy model according to claim 1, characterized in that, The input feature set is determined by encoding based on the game information. The input feature set includes global state features and card-playing action features, including: Encoding is performed based on the bidding information, hand information, and historical card-playing actions in the game information to determine global state characteristics; Encode the current card-playing action and the player's remaining hand information after the current card-playing action is executed based on the game information to determine the characteristics of the card-playing action; The input feature set is determined based on the global state features and the card-playing action features.

3. The interaction method based on a game strategy model according to claim 2, characterized in that, Also includes: Collect different types of off-site target hand information and encode the off-site target hand information to determine the corresponding target hand characteristics.

4. The interaction method based on a game strategy model according to claim 1, characterized in that, The method involves using the Monte Carlo simulation algorithm, the input feature set, and the current game strategy model, and incorporating the number of times the target card type is used as a reward multiplier to conduct independent game simulations. This is done to train the self-play model using the training sample set obtained from the simulations. The method includes: The initial state of the game is generated based on the Monte Carlo simulation algorithm, the input feature set, and the current game strategy model, so as to determine the initial state generation result; Based on the current game strategy model, the entire card-playing process is simulated starting with the player who is the landlord, and the status information, action information, target card type usage information, and game outcome of each player are recorded at each step; the target card type usage information includes the number of times the target card type is used. Based on the game results, the target hand information, and the preset reward function, the corresponding training reward information is determined. The corresponding state sequence is determined based on the state information of each player throughout the game; The training reward information is used as the supervision target for each state in the state sequence to construct samples and determine training samples. The training samples are stored in a buffer pool, and it is determined whether the number of samples in the buffer pool meets the preset quantity condition to determine the quantity judgment result. If the quantity determination result is yes, then the training sample set is determined based on the training samples in the buffer pool; Using the training sample set and a preset loss function, the current game strategy model is trained as a self-play model to obtain the first strategy model after training.

5. The interaction method based on a game strategy model according to claim 1, characterized in that, The step of predicting the next player's hand information based on the global state features in the training sample set, a preset random discard regularization technique, and the first strategy model to complete the opponent modeling self-game training under an imperfect information environment simulation includes: For any player in the game, based on the first strategy model, the current player's hand information, and the hand information of the player before the current player, the hand information of the player after the current player is predicted to determine the prediction result of the hand characteristics of the player after the current player; The predicted hand features of the next player are fused with the global state features corresponding to the hand information of the next player to determine the fused features. Based on the fused features, the preset random discard regularization technique, and the first strategy model, the next player's hand information is predicted to complete the opponent modeling self-game training under the imperfect information environment simulation, and the trained second strategy model is obtained.

6. The interaction method based on a game strategy model according to claim 4, characterized in that, The step of training the teacher network based on the game match win / loss information corresponding to the training sample set, and training the second strategy model using the corresponding currently trained teacher network and a preset match difficulty control mechanism, includes: Based on the current game strategy model, during the process of simulating the entire card-playing process starting from the player whose role is landlord, the action selection sequence corresponding to the three players is obtained; The action selection sequence is used as a supervision signal, and the player's hand information at the start of the game is used as the input information for the model to train the teacher network, so as to obtain the current trained teacher network; the teacher network is used to predict the landlord's winning rate. Based on the game match win / loss information corresponding to the current trained teacher network and the training sample set, the game match selection results are determined; the game match selection results include peasant strategy training samples and landlord strategy training samples, the peasant strategy training samples are game matches where the landlord's win rate is greater than the first probability, and the landlord strategy training samples are game matches where the landlord's win rate meets the preset probability interval. Based on the match selection results and the preset match difficulty control mechanism, the second strategy model is trained for different roles. When the win rate of players in different roles meets the preset training termination condition, the trained third strategy model is determined.

7. The interaction method based on a game strategy model according to any one of claims 1 to 6, characterized in that, The training of the third strategy model based on the input feature set, player card-playing behavior types, and a preset player card-playing action difference sampling mechanism includes: Based on the input feature set and the preset player card-playing action difference sampling mechanism, the third strategy model is sampled and trained under different player card-playing behavior types to determine the trained fourth strategy model; wherein, the player card-playing behavior types include active card-playing type and passive card-receiving type; the preset player card-playing action difference sampling mechanism includes a preset random sampling strategy corresponding to the active card-playing type and a preset temperature control sampling strategy corresponding to the passive card-receiving type.

8. An interactive device based on a game strategy model, characterized in that, Applied to computer devices, including: The game information monitoring module is used to monitor the game information of the current Dou Dizhu game. The game information includes each player's bidding information, hand information, historical playing actions, current playing action, and the player's remaining hand information after the current playing action is executed. Each player includes the target player participating in the current Dou Dizhu game and other players. The target player is a real player or a virtual player, and the other players include real players and / or virtual players. The feature set acquisition module is used to encode the game information to determine the input feature set; the input feature set includes global state features and card-playing action features. The first training module is used to conduct independent game simulations under a perfect information environment where the hand information of all players participating in the game is known. This simulation is based on the Monte Carlo simulation algorithm, the input feature set, and the current game strategy model, and incorporates the number of times the target card type is used as a reward multiplier. The simulation results in the training sample set used to train the self-game model and obtain the trained first strategy model. The current game strategy model includes a card-playing action decision component and a next player's hand prediction component. The second training module is used to predict the next player's hand information based on the global state features in the training sample set, the preset random discard regularization technique and the first strategy model, so as to complete the opponent modeling self-game training under the imperfect information environment simulation and obtain the trained second strategy model. The third training module is used to train the teacher network based on the game match win / loss information corresponding to the training sample set, and to train the second strategy model using the corresponding current post-training teacher network and the preset match difficulty control mechanism, so as to determine the post-training third strategy model when the player win rate of different roles meets the preset training termination condition. The fourth training module is used to train the third strategy model based on the input feature set, the player's card-playing behavior type, and the preset player card-playing action difference sampling mechanism, to determine the trained fourth strategy model, and to use the fourth strategy model to generate the current card-playing decision for the target player, so that the target player can use the current card-playing decision to interact with the other players accordingly.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the interactive method based on a game strategy model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the interactive method based on a game strategy model as described in any one of claims 1 to 7.

Images