A virtual user card strategy generation method, device, equipment and storage medium

Abstract

The application discloses a virtual user card strategy generation method and device, equipment and a storage medium, and relates to the technical field of computers. The method obtains a first card playing strategy with high humanization through the output of a supervised learning model, and obtains a second card playing strategy with high competitive level through the output of a policy head of a reinforcement learning model. According to the outcome probability of the current card game output by the terminal state prediction head of the newly added reinforcement learning model, a fusion coefficient is obtained based on the outcome probability of the current card game, so as to dynamically adjust the influence degree of the first card playing strategy and the second card playing strategy on the final card playing strategy. Through the organic combination of the high performance of reinforcement learning and the high humanization characteristic of supervised learning, not only can the decision-making ability of the game intelligent agent in the complex game be effectively improved, but also the card playing style of the game intelligent agent can be made close to that of a real human player, and the game experience of the real player can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for generating virtual user card strategies. Background Technology

[0002] Card games typically involve complex strategies and teamwork. To enhance the player experience and meet the demand for gaming companions, game platforms often automate the processing of game data to provide virtual gaming companionship and avoid issues such as the game being unable to start due to a lack of players.

[0003] In related technologies, determining card play involves pre-setting numerous game rules and logic, which is simple but lacks strategic flexibility and results in a poor user experience. Existing technologies also utilize neural network models to achieve intelligent card play, but these also suffer from insufficient human-likeness or low competitive level. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method, apparatus, device, and storage medium for generating virtual user card strategies, capable of achieving high performance and high anthropomorphism in card game decision-making. The specific solution is as follows:

[0005] Firstly, this application discloses a method for generating virtual user card strategies, including:

[0006] After detecting the first card-playing action of the target real user, the current card game information is obtained and then input into the supervised learning model and the reinforcement learning model respectively. The reinforcement learning model is a multi-head output neural network, which includes a policy head and an end-game prediction head. The supervised learning model is trained with the goal of imitating the card-playing behavior of real users, and the reinforcement learning model is trained with the goal of maximizing the win rate.

[0007] The first playing strategy is obtained based on the output of the supervised learning model, the second playing strategy is obtained based on the output of the strategy head of the reinforcement learning model, and the probability of the current game's outcome is obtained based on the output of the end prediction head.

[0008] The fusion coefficient is determined based on the probability of the current game outcome. Based on the first playing strategy, the second playing strategy, and the fusion coefficient, a final playing strategy is generated through weighted fusion. The second playing action of the target virtual user is selected from the final playing strategy so that the first playing action can be responded to according to the second playing action.

[0009] Optionally, the training process of the supervised learning model includes:

[0010] Acquire real players' game data, and input the state vector encoding corresponding to the game data into the supervised learning model to be trained; the supervised learning model to be trained is used to output the predicted card-playing strategy based on the currently input state vector encoding.

[0011] The supervised learning model to be trained is fine-tuned based on the difference between the predicted card-playing strategy and the actual card-playing strategy until the model meets the training requirements, thus obtaining the supervised learning model.

[0012] Optionally, before inputting the state vector encoding corresponding to the game data into the supervised learning model to be trained, the method further includes:

[0013] The game state in the game data is vector-encoded according to the encoding rules; the encoding rules include hand information encoding rules, historical card play record encoding rules, current round information encoding rules, player's remaining card count encoding rules, and winner information encoding rules.

[0014] Optionally, the training process of the reinforcement learning model includes:

[0015] Acquire historical game data, input the state vector encoding corresponding to the historical game data into the reinforcement learning model to be trained, and train the end prediction head with the actual outcome of the historical game as the supervision signal.

[0016] The model parameters of the reinforcement learning model to be trained are optimized using a target loss function until the model meets the training requirements, thus obtaining the reinforcement learning model. The target loss function is constructed by weighted summation based on the first loss of the policy head and the second loss of the final prediction head.

[0017] Optionally, the current game information includes: the current hand, the number of cards remaining for each player, historical card play records, tribute card information, and each player's rating.

[0018] Optionally, determining the fusion coefficient based on the probability of the current hand's outcome includes:

[0019] Query the probability of the current game's outcome, and filter out the maximum probability from the probabilities corresponding to different outcomes;

[0020] The fusion coefficient is determined based on the maximum probability value; the larger the maximum probability value, the larger the fusion coefficient, and the greater the weight of the first playing strategy; the smaller the maximum probability value, the smaller the fusion coefficient, and the greater the weight of the second playing strategy.

[0021] Optionally, the formula for calculating the fusion coefficient is:

[0022] ;

[0023] in, The fusion coefficient is... is the maximum probability, k is the steepness coefficient, and T is the probability threshold;

[0024] Before determining the fusion coefficient based on the probability of the current hand's outcome, the process also includes:

[0025] Determine the game phase corresponding to the current hand; the game phase is divided according to the game duration.

[0026] Based on the pre-built correspondence between game stages and probability thresholds, the probability threshold corresponding to the current game is determined; the earlier the game stage, the smaller the probability threshold.

[0027] Secondly, this application discloses a virtual user card strategy generation device, comprising:

[0028] The information acquisition module is used to detect the first card-playing action of the target real user, acquire the current card game information, and input the current card game information into the supervised learning model and the reinforcement learning model respectively; the reinforcement learning model is a multi-head output neural network, including a policy head and an end-game prediction head; the supervised learning model is trained with the goal of imitating the card-playing method of real users, and the reinforcement learning model is trained with the goal of maximizing the win rate.

[0029] The strategy acquisition module is used to obtain a first playing strategy based on the output of the supervised learning model, obtain a second playing strategy based on the output of the strategy head of the reinforcement learning model, and obtain the probability of the current game's outcome based on the output of the end prediction head.

[0030] The final playing strategy generation module is used to determine the fusion coefficient based on the probability of the current game's outcome, generate a final playing strategy by weighted fusion based on the first playing strategy, the second playing strategy, and the fusion coefficient, and select the second playing action of the target virtual user from the final playing strategy so as to respond to the first playing action according to the second playing action.

[0031] Thirdly, this application discloses an electronic device, including:

[0032] Memory, used to store computer programs;

[0033] A processor is used to execute the computer program to implement the aforementioned virtual user card strategy generation method.

[0034] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned virtual user card strategy generation method.

[0035] In this application, after detecting the first card-playing action of the target real user, the current game information is obtained and input into a supervised learning model and a reinforcement learning model respectively. The reinforcement learning model is a multi-head output neural network, including a policy head and an end-game prediction head. The supervised learning model is trained with the goal of imitating the card-playing method of real users, and the reinforcement learning model is trained with the goal of maximizing the win rate. A first card-playing strategy is obtained based on the output of the supervised learning model, a second card-playing strategy is obtained based on the output of the policy head of the reinforcement learning model, and the end-game prediction head is used to obtain the probability of the current game's outcome. A fusion coefficient is determined based on the probability of the current game's outcome. Based on the first card-playing strategy, the second card-playing strategy, and the fusion coefficient, a final card-playing strategy is generated through weighted fusion. The second card-playing action of the target virtual user is selected from the final card-playing strategy so that the first card-playing action can be responded to based on the second card-playing action.

[0036] It is evident that a highly human-like first playing strategy is obtained through the output of the supervised learning model, and a highly competitive second playing strategy is obtained through the output of the strategy head of the reinforcement learning model. Based on the probability of the current game's outcome output by the newly added reinforcement learning model's end-game prediction head, a fusion coefficient is obtained to dynamically adjust the influence of the first and second playing strategies on the final playing strategy. By organically combining the high performance of reinforcement learning with the highly human-like characteristics of supervised learning, not only can the decision-making ability of the game agent in complex games be effectively improved, but its playing style can also be made closer to that of real human players, thus enhancing the gaming experience for real players. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0038] Figure 1 A flowchart of a virtual user card strategy generation method provided in this application;

[0039] Figure 2 A flowchart illustrating a specific method for generating virtual user card strategies provided in this application;

[0040] Figure 3 This application provides a schematic diagram of a specific virtual user card strategy generation system architecture;

[0041] Figure 4A schematic diagram of a virtual user card strategy generation device provided in this application;

[0042] Figure 5 This application provides a structural diagram of an electronic device. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0044] Existing technologies rely on pre-setting numerous game rules and logics to determine card play, resulting in simple implementation but rigid strategies and a poor user experience. While existing technologies utilize neural network models for intelligent card play, these also suffer from insufficient human-likeness or low competitive level. To overcome these technical problems, this application proposes a virtual user card strategy generation method that achieves high performance and high human-likeness in card game decision-making.

[0045] This application discloses a method for generating virtual user card strategies. See also... Figure 1 As shown, the method may include the following steps:

[0046] Step S11: After detecting the first card-playing action of the target real user, obtain the current card game information and input the current card game information into the supervised learning model and the reinforcement learning model respectively; the reinforcement learning model is a multi-head output neural network, including a policy head and an end-game prediction head; the supervised learning model is trained with the goal of imitating the card-playing method of real users, and the reinforcement learning model is trained with the goal of maximizing the win rate.

[0047] First, after detecting the first card-playing action of the target real user, the current game information is obtained. This information includes, but is not limited to: the current hand, the number of cards remaining for each player, historical card-playing records, tribute card information, and each player's rating. In other words, the current game state is received from the game server for subsequent analysis. This current game information is then simultaneously input into both the supervised learning model and the reinforcement learning model. The supervised learning model is trained to mimic the card-playing behavior of a real user, while the reinforcement learning model is trained to maximize the win rate. Specifically, the supervised learning model outputs a highly human-like card-playing strategy based on the input current game information; similarly, the reinforcement learning model outputs a highly competitive card-playing strategy based on the input current game information, aiming to achieve the optimal strategy for winning.

[0048] In this embodiment, the supervised learning model is a model trained using a large amount of real human player game data, and its card-playing decisions have highly human-like characteristics. Specifically, the training process of the supervised learning model includes: acquiring real player game data, inputting the state vector encoding corresponding to the game data into the supervised learning model to be trained; the supervised learning model to be trained outputs a predicted card-playing strategy based on the current input state vector encoding; fine-tuning the supervised learning model to be trained based on the difference between the predicted card-playing strategy and the actual card-playing strategy until the model meets the training requirements, thus obtaining the supervised learning model. By collecting a large amount of human player game data, especially high-level real player game data, a standard supervised learning model is trained; the input of the model is the game state (i.e., the current information of the game), and the output is the card-playing action that a human player is most likely to take in this state.

[0049] In this embodiment, the reinforcement learning model includes two output heads: a policy head and an end-game prediction head. The policy head, the traditional output head, is used to predict the Q-value or policy probability of each legal action to output high-performance decisions. The end-game prediction head, a new output head in this embodiment, is used to predict the final state of the game, i.e., the probability of all possible outcomes. Taking the game of Guandan as an example, there are six possible outcomes: double kill, double kill by the opponent, single kill, single kill by the opponent, tie, and tie. The end-game prediction head outputs the probability of each outcome. The reinforcement learning model can be trained based on deep Monte Carlo methods or similar methods; this embodiment does not limit the training.

[0050] The endgame prediction head employs a Multilayer Perceptron (MLP) as an independent branch from the backbone network, consisting of three fully connected layers, each followed by a Mish activation function. The input to the endgame prediction head, like the policy head, comes from feature representations of the backbone neural network (responsible for processing state vectors). By sharing the backbone network, the model can simultaneously learn common features helpful for policy decisions and endgame prediction, which is computationally efficient and enhances the model's understanding of the game state. The last fully connected layer of the endgame prediction head outputs a dimension N equal to the number of outcomes. The final output layer applies a Softmax function to the N-dimensional vector output by the prediction head, transforming it into a probability distribution where the sum of the N values ​​is 1, representing the predicted probabilities of the N outcomes. MLP has a simple structure, low computational overhead, and is easy to integrate into existing neural network architectures. At the same time, by sharing the backbone network with the policy head, it can utilize the learned game state features to achieve collaborative optimization of policy learning and endgame prediction. Furthermore, since endgame prediction is a classification task, the cross-entropy loss function is used to optimize this branch, resulting in a stable and effective optimization process.

[0051] Specifically, the training process of the reinforcement learning model includes: acquiring historical game data, inputting the state vector encoding corresponding to the historical game data into the reinforcement learning model to be trained, and training the end-game prediction head using the actual outcome of the historical games as a supervision signal; optimizing the model parameters of the reinforcement learning model to be trained using a target loss function until the model meets the training requirements to obtain the reinforcement learning model; the target loss function is constructed by weighted summation based on the first loss of the policy head and the second loss of the end-game prediction head.

[0052] In the reinforcement learning training framework, the neural network model is modified by adding a final prediction head in addition to the original policy head used to calculate the Q-value. This results in a definite game outcome after each self-play game; this actual final outcome is used as a supervisory signal for the final prediction head. During training, the model optimizes not only the policy head using reward signals but also the prediction head using supervisory signals, enabling it to accurately predict the final outcome based on the game's state.

[0053] Furthermore, during training, a combined loss function is chosen as the target loss function. This function is a weighted sum of the first loss (policy head loss) and the second loss (final prediction head loss); the target loss function is as follows:

[0054] ;in, The loss function for the policy head can be calculated using temporal difference error or Monte Carlo reward in reinforcement learning, with the goal of making Q-value estimation more accurate. The loss function for the final prediction head can be cross-entropy loss; and These are hyperparameters used to balance the importance of the two tasks during training; by adjusting these two weights, the learning emphasis of the model in pursuing victory and understanding the situation can be controlled.

[0055] In a specific embodiment, before inputting the state vector encoding corresponding to the game data into the supervised learning model to be trained, the method further includes: vector encoding the game state in the game data according to the encoding rules; the encoding rules include hand information encoding rules, historical card play record encoding rules, current round information encoding rules, player's remaining card count encoding rules, and winner information encoding rules.

[0056] Understandably, to enable neural networks to process game information, the discrete and complex current hand information needs to be converted into a fixed-length numerical vector (i.e., a state vector). Taking Guandan as an example, the specific encoding rules include: 1. Hand information encoding rules: Hand information is represented by a 54-dimensional vector. Since Guandan uses two decks of cards, each type of card (from A to K, plus the Jokers, a total of 13×4+2=54 types) has a maximum of 2 cards; each dimension of this vector corresponds to a type of card, and its value can be 0, 1, or 2, representing the number of that type of card in the current hand. 2. Historical card play record encoding rules: Four 54-dimensional vectors are used to record all the cards played from the start of the game to the current round and their quantities (1×54), as well as all the cards played by each other player and their quantities (3×54). 3. Current round information encoding rules: One 58-dimensional vector is used to represent the hand of the previous player (54 dimensions) and the relative seating position of the player playing the card relative to this agent (4 dimensions). 4. Remaining Card Count Encoding Rules: A 112-dimensional vector is used to represent the number of cards remaining in the hands of the four players, including the current agent. Since each player starts with 28 cards, the total number of dimensions after one-hot encoding is 112. 5. Bomb Information Encoding Rules: A 28-dimensional vector is used to represent the number of bombs played by each player. Also using one-hot encoding, since each player can have a maximum of 7 bombs, the total number of dimensions is 28. 6. Winner Information Encoding Rules: Since the ranking of subsequent players needs to be determined after the first place is decided, a 3-dimensional vector is used to indicate whether there is a winning player. If so, it indicates whether the winning player is an opponent or a teammate. Additionally, game progress-related information is encoded. 7. Level Card Information Encoding Rules: A 24-dimensional vector is used to represent the current level of both the player and the opponent. 8. Tribute Card Information Encoding Rules: Four 54-dimensional vectors are used to represent the tribute cards and return tribute cards in this round. Finally, the vectors from all the above parts are concatenated to form a complete, high-dimensional state vector, which serves as the final input to the neural network; both supervised learning and reinforcement learning models use the above state vector encoding method.

[0057] Step S12: Obtain a first playing strategy based on the output of the supervised learning model, obtain a second playing strategy based on the output of the strategy head of the reinforcement learning model, and obtain the probability of the current game's outcome based on the output of the endgame prediction head.

[0058] The first card-playing strategy, which is highly human-like, is obtained based on the output of the supervised learning model; the second card-playing strategy, which has a high level of competitiveness, is obtained based on the output of the endgame prediction head; and the probability of the current game's outcome is predicted in real time based on the output of the endgame prediction head.

[0059] This means that the supervised learning model and the reinforcement learning model use parallel prediction. The current game state is input into the pre-trained reinforcement learning model, which calculates two sets of results in parallel. The policy head outputs the Q-values ​​of each legal action. The reinforcement learning model outputs the card-playing strategy. Where 'a' represents a specific legal action, and 's' represents the state (i.e., the current hand information mentioned above), the probability distribution of the six final outcomes output by the endgame prediction head is shown. ,in The probability corresponding to a certain outcome; the policy output by the supervised learning model. The Q value is an estimate of the expected total future reward that can be obtained after performing a specific action (a) in a specific state (s); the Q value measures how "good" or "bad" it is to make a choice in the current situation.

[0060] Step S13: Determine the fusion coefficient based on the probability of the current game's outcome. Based on the first playing strategy, the second playing strategy, and the fusion coefficient, generate the final playing strategy through weighted fusion. Select the target virtual user's second playing action from the final playing strategy so as to respond to the first playing action according to the second playing action.

[0061] For example Figure 2 As shown, after obtaining the first playing strategy output by the supervised learning model, the second playing strategy output by the reinforcement learning model, and the outcome probability output by the final prediction head, strategies are fused based on the outcome probabilities. The fusion coefficient is obtained through the outcome probability of the current game to dynamically adjust the influence of the first and second playing strategies on the final playing strategy. Finally, the action with the largest action value function (Q-value) from the final playing strategies is selected as the second playing action. The execution result of the second playing action is displayed on the interactive interface to realize the interactive response of the target virtual user to the target real user's first playing action, thereby realizing a method for interaction between virtual and real users in a card game.

[0062] Understandably, in existing technologies, supervised learning methods learn from a large amount of game data from skilled human players, mimicking human decision-making patterns. These methods achieve a high degree of anthropomorphism, but their strategic level is limited by the quality and breadth of the training data, making it difficult to discover innovative strategies that surpass human experience. Reinforcement learning methods, on the other hand, learn optimal strategies through massive self-play in simulated environments, achieving very high competitive levels. However, because their training process does not rely on human data, their decision-making behavior may deviate significantly from human habits in certain situations, resulting in insufficient anthropomorphism. The technical problem addressed in this application is how to solve the anthropomorphism problem of reinforcement learning robots, enabling them to maintain a high level of competition while making their playing style closer to human players and improving the interactive experience.

[0063] This application combines reinforcement learning and supervised learning models. Within the neural network structure of the reinforcement learning model, a dedicated endgame prediction head is incorporated to predict the final outcome of the current game. This mechanism dynamically integrates card-playing decisions based on the probability of the current game's outcome. This achieves an organic combination of high performance and high human-likeness, dynamically generating the optimal final strategy for the current situation through a dynamic switching mechanism. Compared to a fixed model, this application dynamically adjusts its decision logic based on the probability of winning or losing in the game, achieving adaptive strategy switching. This makes card-playing more diverse and intelligent, closer to the thinking patterns of advanced human players who adopt different mindsets and strategies in different situations. Furthermore, the training framework has good compatibility; the introduction of the endgame prediction head can be seamlessly integrated into most existing self-game-based reinforcement learning training frameworks. Only a supervised loss term for the prediction result needs to be added to the loss function, resulting in low implementation cost and no impact on the efficiency of the original strategy learning.

[0064] In a specific embodiment, determining the fusion coefficient based on the probability of the current game's outcome includes: querying the probability of the current game's outcome, filtering out the maximum probability from the probabilities corresponding to different outcomes; determining the fusion coefficient based on the maximum probability; the larger the maximum probability, the larger the fusion coefficient, and the greater the weight of the first playing strategy; the smaller the maximum probability, the smaller the fusion coefficient, and the greater the weight of the second playing strategy.

[0065] Understandably, the maximum probability represents the certainty of the game. When the game is highly certain, a supervised learning model is invoked to make decisions and ensure human-like card-playing behavior. In a close game, a reinforcement learning model is used to pursue the optimal strategy, ensuring high-intensity competition; this achieves a balance between human-like behavior and competitiveness. The certainty of the current game outcome is judged by comparing the probabilities of the final prediction head output, and this serves as the basis for decisions combining the reinforcement learning and supervised learning models. In non-extreme win rate situations, the policy output of the reinforcement learning model is used as much as possible to pursue the highest win rate; in extreme win rate situations, the policy output of the supervised learning model is used as much as possible to ensure human-like behavior.

[0066] Specifically, determining the fusion coefficient based on the maximum probability may include: mapping the maximum probability to the interval between 0 and 1 to obtain the fusion coefficient; using the fusion coefficient as the weight of the first playing strategy, and using the difference between 1 and the fusion coefficient as the weight of the second playing strategy, and obtaining the final playing strategy through weighted summation; the larger the maximum probability, the closer the fusion coefficient is to 1, and the greater the weight of the first playing strategy; the smaller the maximum probability, the closer the fusion coefficient is to 0, and the greater the weight of the second playing strategy. That is, ensuring the fusion coefficient is in the interval between 0 and 1 facilitates subsequent weighted summation.

[0067] The steps include: determining the fusion coefficient based on the probability of the current game's outcome; generating a final playing strategy through weighted fusion based on the first playing strategy, the second playing strategy, and the fusion coefficient; and selecting the target virtual user's second playing action from the final playing strategy to respond to the first playing action.

[0068] Step S130: Probability distribution output from the final prediction head In this case, the value with the highest probability is selected as the certainty of the battle outcome. .

[0069] Step S131: Calculate the smoothing fusion coefficient .

[0070] Specifically, the Sigmoid function is used to calculate the maximum probability value. This is mapped to a fusion coefficient that varies smoothly within the interval (0, 1). :

[0071] ;

[0072] in, represents the weight of the supervised learning model in the current policy; T is the probability threshold, which serves as the center offset of the Sigmoid function; k is the steepness coefficient, used to control the intensity of the fusion.

[0073] Of course, other smoothing functions such as step functions and piecewise linear functions can also be used to calculate the fusion weights.

[0074] Step S132: Apply the policy of the reinforcement learning model Strategies for supervised learning models Through fusion coefficient By performing weighted fusion, the final card-playing strategy is obtained. :

[0075] ;

[0076] It is evident that the clearer the battle situation becomes, The larger, The closer the value is to 1, the more the final policy leans towards the policy of the supervised learning model. Conversely, strategies tend to favor reinforcement learning models. .

[0077] Step S133: Select the optimal action as the second card-playing action.

[0078] Choose the action that maximizes the Q value based on the final strategy. As a decision to play a card:

[0079] The argmax function is used to find the value within the function. The action 'a' that maximizes the Q value is taken as the action 'a'. ;

[0080] The final selected card-playing action is sent back to the game server for execution.

[0081] Before determining the fusion coefficient based on the probability of the current hand's outcome, the process further includes: determining the game stage corresponding to the current hand; the game stage is divided according to the game duration; determining the probability threshold corresponding to the current hand based on a pre-constructed correspondence between game stages and probability thresholds; wherein, the earlier the game stage, the smaller the probability threshold.

[0082] During the game, the threshold T is dynamically adjusted, specifically based on different game stages (e.g., early, mid, and late). A specific threshold adjustment rule is as follows: different thresholds are preset according to the game's progress. For the early game (e.g., remaining cards > 20), T = T_early (e.g., 0.3). This is understandable because in the early game, the situation is usually uncertain, every card is crucial, and the system's goal is to gain an advantage. At this time, the threshold can be set lower, making the system more inclined to use reinforcement learning models to pursue optimal strategies and quickly establish an advantage. For the mid-game (e.g., 5 < remaining cards ≤ 20), T = T_mid (e.g., 0.5). This is understandable because as the game enters the mid-game, the situation gradually becomes clearer, but the outcome is still uncertain. At this time, the threshold can be increased, making the AI ​​more inclined to use supervised learning models when the situation is more certain, thus maintaining human-like card-playing. For the late game (e.g., remaining cards ≤ 5), T = T_late (e.g., 0.9). Understandably, as the game progresses into its later stages, the outcome is usually very clear—either win or lose. At this point, the threshold should be significantly increased, allowing the system to rely almost entirely on a supervised learning model. This ensures that card-playing behavior during "garbage time" is highly human-like, preventing random card plays. By selecting different thresholds at different stages of the game and thus controlling strategy selection, the user experience can be effectively improved.

[0083] As can be seen from the above, after detecting the first card-playing action of the target real user in this embodiment, the current game information is obtained and input into the supervised learning model and the reinforcement learning model respectively. The reinforcement learning model is a multi-head output neural network, including a strategy head and an end-game prediction head. The supervised learning model is trained with the goal of imitating the card-playing method of the real user, and the reinforcement learning model is trained with the goal of maximizing the win rate. The first card-playing strategy is obtained according to the output of the supervised learning model, the second card-playing strategy is obtained according to the output of the strategy head of the reinforcement learning model, and the end-game prediction head is obtained according to the output of the end-game prediction head. The fusion coefficient is determined according to the end-game probability of the current game. Based on the first card-playing strategy, the second card-playing strategy and the fusion coefficient, the final card-playing strategy is generated by weighted fusion. The second card-playing action of the target virtual user is selected from the final card-playing strategy so as to respond to the first card-playing action according to the second card-playing action.

[0084] It is evident that a highly human-like first playing strategy is obtained through the output of the supervised learning model, and a highly competitive second playing strategy is obtained through the output of the strategy head of the reinforcement learning model. Based on the probability of the current game's outcome output by the newly added reinforcement learning model's end-game prediction head, a fusion coefficient is obtained to dynamically adjust the influence of the first and second playing strategies on the final playing strategy. By organically combining the high performance of reinforcement learning with the highly human-like characteristics of supervised learning, not only can the decision-making ability of the game agent in complex games be effectively improved, but its playing style can also be made closer to that of real human players, thus enhancing the gaming experience for real players.

[0085] This application aims to address the problem that pure reinforcement learning lacks human-like characteristics in card-playing strategies under specific game situations by constructing a high-performance and highly human-like AI agent for the card game "Guandan". During the training and inference process of the reinforcement learning model, an end-game probability prediction head is introduced to assess the certainty of the current game's outcome. When the game's certainty is high (e.g., one side has gained a significant advantage, and the outcome is basically certain), the system will likely call the supervised learning model for decision-making, ensuring the AI's card-playing strategy is highly human-like and avoiding the non-human-like card-playing behaviors that pure reinforcement learning models might exhibit when there are obvious advantages or disadvantages. When the game is in a stalemate (i.e., the outcome is still unclear, requiring sophisticated strategic play), the system will likely continue to use the reinforcement learning model; at this time, the reinforcement learning model will leverage its advantage of pursuing optimal strategies to achieve victory in complex and ever-changing games. This solution is applicable to various imperfect information card game games, such as Dou Dizhu, Shuang Kou, and Run Fast.

[0086] This application also discloses a specific virtual user card strategy generation system, including a supervised learning model, a reinforcement learning model, and a strategy fusion module.

[0087] For example Figure 3 As shown, the reinforcement learning model includes a policy head and an end-game prediction head. The policy head, also known as the traditional output head, is used to predict the Q-value or policy probability of each legal action to make high-performance decisions. The end-game prediction head, a new output head in this embodiment, is used to predict the final state of the game. The supervised learning model is trained using a large amount of human player game data, and its card-playing decisions have highly human-like characteristics. The policy fusion module is used to execute steps S12 and S13 above, that is, to fuse the decision results of the reinforcement learning model and the supervised learning model according to the probability based on the output of the end-game prediction head of the reinforcement learning model.

[0088] Accordingly, this application also discloses a virtual user card strategy generation device, see [link to relevant documentation]. Figure 4 As shown, the device includes:

[0089] The information acquisition module 11 is used to acquire the current game information after detecting the first card-playing action of the target real user, and input the current game information into the supervised learning model and the reinforcement learning model respectively; the reinforcement learning model is a multi-head output neural network, including a strategy head and an end-game prediction head; the supervised learning model is trained with the goal of imitating the card-playing method of the real user, and the reinforcement learning model is trained with the goal of maximizing the win rate.

[0090] The strategy acquisition module 12 is used to obtain a first playing strategy based on the output of the supervised learning model, obtain a second playing strategy based on the output of the strategy head of the reinforcement learning model, and obtain the probability of the current game's outcome based on the output of the end prediction head.

[0091] The final playing strategy generation module 13 is used to determine the fusion coefficient based on the probability of the current game outcome, generate the final playing strategy by weighted fusion based on the first playing strategy, the second playing strategy and the fusion coefficient, and select the second playing action of the target virtual user from the final playing strategy so as to respond to the first playing action according to the second playing action.

[0092] As can be seen from the above, after detecting the first card-playing action of the target real user in this embodiment, the current game information is obtained and input into the supervised learning model and the reinforcement learning model respectively. The reinforcement learning model is a multi-head output neural network, including a strategy head and an end-game prediction head. The supervised learning model is trained with the goal of imitating the card-playing method of the real user, and the reinforcement learning model is trained with the goal of maximizing the win rate. The first card-playing strategy is obtained according to the output of the supervised learning model, the second card-playing strategy is obtained according to the output of the strategy head of the reinforcement learning model, and the end-game prediction head is obtained according to the output of the end-game prediction head. The fusion coefficient is determined according to the end-game probability of the current game. Based on the first card-playing strategy, the second card-playing strategy and the fusion coefficient, the final card-playing strategy is generated by weighted fusion. The second card-playing action of the target virtual user is selected from the final card-playing strategy so as to respond to the first card-playing action according to the second card-playing action.

[0093] It is evident that a highly human-like first playing strategy is obtained through the output of the supervised learning model, and a highly competitive second playing strategy is obtained through the output of the strategy head of the reinforcement learning model. Based on the probability of the current game's outcome output by the newly added reinforcement learning model's end-game prediction head, a fusion coefficient is obtained to dynamically adjust the influence of the first and second playing strategies on the final playing strategy. By organically combining the high performance of reinforcement learning with the highly human-like characteristics of supervised learning, not only can the decision-making ability of the game agent in complex games be effectively improved, but its playing style can also be made closer to that of real human players, thus enhancing the gaming experience for real players.

[0094] In some specific embodiments, the training process of the supervised learning model includes:

[0095] Acquire real players' game data, and input the state vector encoding corresponding to the game data into the supervised learning model to be trained; the supervised learning model to be trained is used to output the predicted card-playing strategy based on the currently input state vector encoding.

[0096] The supervised learning model to be trained is fine-tuned based on the difference between the predicted card-playing strategy and the actual card-playing strategy until the model meets the training requirements, thus obtaining the supervised learning model.

[0097] In some specific embodiments, the training process of the reinforcement learning model includes:

[0098] Acquire historical game data, input the state vector encoding corresponding to the historical game data into the reinforcement learning model to be trained, and train the end prediction head with the actual outcome of the historical game as the supervision signal.

[0099] The model parameters of the reinforcement learning model to be trained are optimized using a target loss function until the model meets the training requirements, thus obtaining the reinforcement learning model. The target loss function is constructed by weighted summation based on the first loss of the policy head and the second loss of the final prediction head.

[0100] In some specific embodiments, the current game information includes: the current hand, the number of cards remaining for each player, historical card play records, tribute card information, and each player's rating.

[0101] In some specific embodiments, the final card-playing strategy generation module 13 may specifically include:

[0102] The filtering unit is used to query the probability of the current game's outcome and filter out the maximum probability from the probabilities corresponding to different outcomes.

[0103] The fusion coefficient calculation unit is used to determine the fusion coefficient based on the maximum probability value; the larger the maximum probability value, the larger the fusion coefficient, and the greater the weight of the first playing strategy; the smaller the maximum probability value, the smaller the fusion coefficient, and the greater the weight of the second playing strategy.

[0104] In some specific embodiments, the formula for calculating the fusion coefficient is as follows:

[0105] ;

[0106] in, The fusion coefficient is... Where k is the maximum probability, T is the steepness coefficient, and T is the probability threshold.

[0107] Accordingly, the virtual user card strategy generation device may specifically include:

[0108] A stage determination unit is used to determine the game stage corresponding to the current hand before determining the fusion coefficient based on the probability of the current hand's outcome; the game stage is divided according to the game duration.

[0109] The probability threshold determination unit is used to determine the probability threshold corresponding to the current game based on the pre-built correspondence between game stages and probability thresholds; the earlier the game stage, the smaller the probability threshold.

[0110] Furthermore, this application also discloses an electronic device, see [link to relevant documentation]. Figure 5 As shown, the content in the figure should not be considered as any limitation on the scope of use of this application.

[0111] Figure 5This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the virtual user card strategy generation method disclosed in any of the foregoing embodiments.

[0112] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0113] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon include operating system 221, computer program 222 and data 223 including current game information, etc. The storage method can be temporary storage or permanent storage.

[0114] The operating system 221 manages and controls the various hardware devices on the electronic device 20 and the computer program 222 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the virtual user card strategy generation method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0115] Furthermore, this application also discloses a computer storage medium storing computer-executable instructions. When the computer-executable instructions are loaded and executed by a processor, they implement the virtual user card strategy generation method steps disclosed in any of the foregoing embodiments.

[0116] Furthermore, this application also discloses a computer program product, including a computer program that, when executed by a processor, implements the virtual user card strategy generation method steps disclosed in any of the foregoing embodiments.

[0117] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0118] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0119] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0120] The above provides a detailed description of a virtual user card strategy generation method, apparatus, device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for generating virtual user card strategies, characterized in that, include: After detecting the first card-playing action of the target real user, the current game information is obtained and then input into the supervised learning model and the reinforcement learning model respectively; the reinforcement learning model is a multi-head output neural network, which includes a policy head and an end-game prediction head; The supervised learning model is trained with the goal of mimicking the card-playing style of real users, and the reinforcement learning model is trained with the goal of maximizing the win rate. The first playing strategy is obtained based on the output of the supervised learning model, the second playing strategy is obtained based on the output of the strategy head of the reinforcement learning model, and the probability of the current game's outcome is obtained based on the output of the end prediction head. The fusion coefficient is determined based on the probability of the current game's outcome. Based on the first playing strategy, the second playing strategy, and the fusion coefficient, a final playing strategy is generated through weighted fusion. The second playing action of the target virtual user is selected from the final playing strategy so as to respond to the first playing action according to the second playing action. The step of determining the fusion coefficient based on the probability of the current hand's outcome includes: Query the probability of the current game's outcome, and filter out the maximum probability from the probabilities corresponding to different outcomes; The fusion coefficient is determined based on the maximum probability value; the larger the maximum probability value, the larger the fusion coefficient, and the greater the weight of the first playing strategy; the smaller the maximum probability value, the smaller the fusion coefficient, and the greater the weight of the second playing strategy.

2. The virtual user card strategy generation method according to claim 1, characterized in that, The training process of the supervised learning model includes: Acquire real players' game data, and input the state vector encoding corresponding to the game data into the supervised learning model to be trained; the supervised learning model to be trained is used to output the predicted card-playing strategy based on the currently input state vector encoding. The supervised learning model to be trained is fine-tuned based on the difference between the predicted card-playing strategy and the actual card-playing strategy until the model meets the training requirements, thus obtaining the supervised learning model.

3. The virtual user card strategy generation method according to claim 2, characterized in that, Before inputting the state vector encoding corresponding to the game data into the supervised learning model to be trained, the method further includes: The game state in the game data is vector-encoded according to the encoding rules; the encoding rules include hand information encoding rules, historical card play record encoding rules, current round information encoding rules, player's remaining card count encoding rules, and winner information encoding rules.

4. The virtual user card strategy generation method according to claim 1, characterized in that, The training process of the reinforcement learning model includes: Acquire historical game data, input the state vector encoding corresponding to the historical game data into the reinforcement learning model to be trained, and train the end prediction head with the actual outcome of the historical game as the supervision signal. The model parameters of the reinforcement learning model to be trained are optimized using a target loss function until the model meets the training requirements, thus obtaining the reinforcement learning model. The target loss function is constructed by weighted summation based on the first loss of the policy head and the second loss of the final prediction head.

5. The virtual user card strategy generation method according to claim 1, characterized in that, The current game information includes: the current hand, the number of cards remaining for each player, historical card play records, tribute card information, and each player's rating.

6. The virtual user card strategy generation method according to claim 1, characterized in that, The formula for calculating the fusion coefficient is as follows: ； in, The fusion coefficient is... is the maximum probability, k is the steepness coefficient, and T is the probability threshold; Before determining the fusion coefficient based on the probability of the current hand's outcome, the process also includes: Determine the game phase corresponding to the current hand; the game phase is divided according to the game duration. Based on the pre-built correspondence between game stages and probability thresholds, the probability threshold corresponding to the current game is determined; the earlier the game stage, the smaller the probability threshold.

7. A virtual user card strategy generation device, characterized in that, include: The information acquisition module is used to acquire the current game information after detecting the first card-playing action of the target real user, and input the current game information into the supervised learning model and the reinforcement learning model respectively; the reinforcement learning model is a multi-head output neural network, which includes a policy head and an end-game prediction head; The supervised learning model is trained with the goal of mimicking the card-playing style of real users, and the reinforcement learning model is trained with the goal of maximizing the win rate. The strategy acquisition module is used to obtain a first playing strategy based on the output of the supervised learning model, obtain a second playing strategy based on the output of the strategy head of the reinforcement learning model, and obtain the probability of the current game's outcome based on the output of the end prediction head. The final playing strategy generation module is used to determine the fusion coefficient based on the probability of the current game's outcome, generate a final playing strategy based on the first playing strategy, the second playing strategy, and the fusion coefficient through weighted fusion, and select the second playing action of the target virtual user from the final playing strategy so as to respond to the first playing action according to the second playing action; The final card-playing strategy generation module is used to query the probability of the current game's outcome, filter out the maximum probability from the probabilities corresponding to different outcomes, and determine the fusion coefficient based on the maximum probability. The greater the maximum probability, the greater the fusion coefficient, and the greater the weight of the first playing strategy; the smaller the maximum probability, the smaller the fusion coefficient, and the greater the weight of the second playing strategy.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the virtual user card strategy generation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein the computer programs, when executed by a processor, implement the virtual user card strategy generation method as described in any one of claims 1 to 6.

Images