An intelligent decision-making method and system for mahjong, a storage medium and equipment

Abstract

The application provides an intelligent decision-making method and system for mahjong, a storage medium and equipment, the method comprising: obtaining historical scene information of mahjong, obtaining hidden information of mahjong according to the historical scene information, and obtaining forward-looking features of mahjong tiles according to the historical scene information; then performing feature coding to obtain a historical feature matrix, inputting the historical feature matrix into a reinforcement learning model for training to obtain an optimal tile decision-making model; obtaining current scene information, performing feature coding on the current scene information to obtain a current feature matrix, inputting the current feature matrix into the optimal tile decision-making model, obtaining a mahjong tile decision-making result, and making a tile decision according to the tile decision-making result. The intelligent decision-making method and system for mahjong, the storage medium and the equipment combine the optimal tile decision-making model, obtain the mahjong tile decision-making result, and make a tile decision, thereby solving the technical problem that the current situation is not accurately judged in the prior art, which affects the tile decision-making of the self.

Description

Technical Field

[0001] This invention relates to the field of intelligent decision-making technology for mahjong, and in particular to an intelligent decision-making method, system, storage medium, and device for mahjong. Background Technology

[0002] Incomplete information games refer to games where no player knows the information of the other players when making a decision. In other words, in an incomplete information game, no player can obtain information about the other players when making a decision. Mahjong is an example of an incomplete information game. Taking mahjong as an example, decisions in the game mainly include playing a tile, making melds (chow, pong, kong), and winning.

[0003] In existing technologies, Mahjong games contain a great deal of hidden information, such as the opponent's hand and the wall tiles. Due to the lack of consideration for this hidden information, the judgment of the current game situation is not accurate enough, which affects the player's own card-playing decisions. This results in a small advantage for the player to win or a low winning score, and the final score is far from the player's best possible score. Summary of the Invention

[0004] Based on this, the purpose of the present invention is to provide an intelligent decision-making method, system, storage medium and device for Mahjong, to solve the technical problem that the existing card-playing decision schemes fail to consider hidden information, resulting in inaccurate judgment of the current game situation and affecting card-playing decisions.

[0005] This is used to solve the card-playing decision-making scheme in existing technology.

[0006] This invention provides an intelligent decision-making method for playing mahjong, the method comprising:

[0007] The historical scene information of Mahjong is obtained, and the hidden information of Mahjong is obtained based on the historical scene information. The hidden information includes the opponent's hand information and the wall information. Based on the historical scene information and the search tree algorithm, the prospective features of Mahjong play are obtained. The prospective features are the search extension features within a preset step size after the current node of the search tree.

[0008] The historical scene information, the hidden information, and the prospective features are feature-encoded to obtain a historical feature matrix. The historical feature matrix is ​​then input into a reinforcement learning model for training to obtain the optimal card-playing decision model.

[0009] Obtain current situation information and perform feature encoding on the current situation information to obtain a current feature matrix. Input the current feature matrix into the optimal play decision model to obtain the mahjong play decision result. Make a play decision based on the play decision result.

[0010] The aforementioned intelligent decision-making method for Mahjong, by acquiring hidden information about the game, avoids situations where inaccurate judgment of hidden information affects the player's card-playing decisions, thereby improving the win rate and final score in the Mahjong game. Furthermore, by combining the acquisition of prospective features of Mahjong card-playing, a historical feature matrix is ​​obtained by feature encoding historical scene information, hidden information, and prospective features. This historical feature matrix is ​​then used to train a reinforcement learning model to obtain an optimal card-playing decision model, ensuring that each card-playing decision during the Mahjong game is the optimal one. When current scene information is acquired, the optimal card-playing decision model is combined to obtain the Mahjong card-playing decision result and make the card-playing decision, enabling the player to ultimately obtain the best score. This solves the technical problem in existing card-playing decision schemes where inaccurate judgment of the current game situation due to the lack of consideration for hidden information affects card-playing decisions.

[0011] In addition, the intelligent decision-making method for mahjong according to the present invention may also have the following additional technical features:

[0012] Furthermore, the step of inputting the historical feature matrix into the reinforcement learning model for training to obtain the optimal card-playing decision model includes:

[0013] The historical feature matrix is ​​input into a reinforcement learning model, which includes an information extraction module. The information extraction module extracts and understands the historical feature matrix and outputs the understood information.

[0014] The reinforcement learning model acquires the understood information, then configures a training strategy and performs hierarchical training.

[0015] The policy entropy and divergence terms of the model training are obtained based on the training results.

[0016] The reinforcement learning model is dynamically adjusted and optimized based on the policy entropy term and the divergence term to obtain the optimal card-playing decision model.

[0017] Furthermore, the dynamic adjustment formula for the strategy entropy term is:

[0018] γ←γ+c1(Ent targ -Ent θ )

[0019] Where γ is the weighting coefficient of the policy entropy in the objective function, Ent targ It is the target value of the policy entropy, Ent θ c1 is the current policy entropy value, and c1 is the coefficient for dynamically adjusting the policy entropy.

[0020] Furthermore, the dynamic adjustment formula for the divergence term is:

[0021] δ←δ+c2(KL targ -KL θ )

[0022] Where δ is the weighting coefficient of the divergence in the objective function, KL targ It is the target value of the divergence, KL θ c1 is the current divergence value, and c2 is the coefficient for dynamically adjusting the divergence.

[0023] Furthermore, in the step of adjusting and optimizing the reinforcement learning model based on the policy entropy term and the divergence term to obtain the optimal card-playing decision model, the optimization function is:

[0024] J(θ)=αL C +βL V +γEnt θ +δKL θ

[0025] Where αL C This is the loss function of the actor network, where α is the correlation coefficient of the loss function, and βL... V γEnt is the loss function of the critic network, β is the correlation coefficient of this loss function, and γEnt is the loss function of the critic network. θ It is the policy entropy of the current updated game trajectory, γ is the correlation coefficient of the policy entropy, and δKL θ δ represents the divergence term of the currently updated game trajectory, where δ is the correlation coefficient of the divergence term.

[0026] Furthermore, the step of inputting the historical feature matrix into the reinforcement learning model for model training to obtain the optimal card-playing decision model includes:

[0027] The loss functions of the actor network and the critic network are obtained by matrix calculation using historical features.

[0028] The loss values ​​of the actor network and the critic network are obtained based on the loss function of the actor network and the loss function of the critic network. The loss value of the actor network includes the probability of playing each card, and the loss value of the critic network includes the predicted score under the current card.

[0029] The convergence value of the reinforcement learning model is calculated by combining the loss value of the actor network, the loss value of the critic network, the policy entropy value of the current updated game trajectory, and the divergence term value of the current updated game trajectory.

[0030] Based on the calculated convergence value, the optimal card-playing decision model is obtained.

[0031] Another aspect of the present invention provides an intelligent decision-making system for Mahjong, the system comprising:

[0032] The acquisition module is used to acquire historical scene information of mahjong, acquire hidden information of mahjong based on the historical scene information, the hidden information includes the opponent's hand information and the wall of tiles information, and acquire the prospective features of mahjong play based on the historical scene information combined with the search tree algorithm.

[0033] The training module is used to encode the historical scene information, the hidden information, and the prospective features to obtain the historical feature matrix after feature encoding. The historical feature matrix is ​​then input into the reinforcement learning model for training to obtain the optimal card-playing decision model.

[0034] The decision module is used to acquire current situation information, encode the current situation information to obtain a current feature matrix, input the current feature matrix into the optimal play decision model to obtain the mahjong play decision result, and make a play decision based on the play decision result.

[0035] The aforementioned intelligent decision-making system for mahjong, by acquiring hidden information about the game, avoids situations where inaccurate judgment of hidden information affects the player's card-playing decisions, thereby improving the win rate and final score in the mahjong game. Furthermore, by combining the acquisition of prospective features of mahjong card-playing, a historical feature matrix is ​​obtained by feature encoding historical scene information, hidden information, and prospective features. This historical feature matrix is ​​then used to train a reinforcement learning model to obtain an optimal card-playing decision model, ensuring that each card-playing decision during the mahjong game is the optimal one. When current scene information is acquired, the optimal card-playing decision model is combined to obtain the mahjong card-playing decision result and make the card-playing decision, enabling the player to ultimately obtain the best score. This solves the technical problem in existing card-playing decision schemes that fail to consider hidden information, leading to inaccurate judgment of the current game situation and affecting card-playing decisions.

[0036] In another aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intelligent decision-making method for mahjong as described above.

[0037] In another aspect, the present invention provides a data processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the intelligent decision-making method for mahjong as described above. Attached Figure Description

[0038] Figure 1 This is a flowchart of the steps of the intelligent decision-making method for Mahjong in the first embodiment of the present invention;

[0039] Figure 2 This is a flowchart of the steps of the intelligent decision-making method for mahjong in the second embodiment of the present invention;

[0040] Figure 3 This is a flowchart of step S202 in the second embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of a private hand in the second embodiment of the present invention;

[0042] Figure 5 This is a diagram showing the private hand card encoding in the second embodiment of the present invention;

[0043] Figure 6 This is a schematic diagram of the Dummy encoding in the second embodiment of the present invention;

[0044] Figure 7 The standard PPO algorithm is used to implement the convergence of Chinese four-player Mahjong;

[0045] Figure 8 The PPO algorithm in this application is used to implement the convergence of Chinese four-player Mahjong;

[0046] Figure 9 This is a system block diagram of an intelligent decision-making method for Mahjong according to the third embodiment of the present invention.

[0047] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0048] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0050] Data processing of the data involved in the decision-making model is a prerequisite for building the decision-making model. Taking Mahjong as an example, the Mahjong decision-making model first obtains some data information in the current Mahjong game (including hand, concealed hand, opponents' concealed hand, discarded hand, dealer's position, etc.), and then makes reasonable decision-making actions by processing this data information in order to win more points in the Mahjong game.

[0051] Incomplete information games refer to games in which no player knows the information of other players when making a decision. In other words, in the process of an incomplete information game, no player can obtain information about other players when making a decision.

[0052] The Policy-Based Learning (PPO) algorithm is a policy-based reinforcement learning algorithm. Reinforcement learning has two main components: the agent and the environment. The agent is the entity itself that learns through reinforcement learning; it's a model that automatically acquires valuable information through learning. The environment is the world that the agent depends on, operates in, and interacts with. In each interaction between the agent and the environment, the agent observes its current state, or a portion of its state, within the environment and uses this information to determine its appropriate action. Simultaneously, during each interaction, the agent receives feedback from the environment, indicating how good or bad its current performance is. The agent's learning objective is to obtain the maximum cumulative reward.

[0053] Reinforcement learning is a method that enables an agent to learn how to take actions to achieve its goals through continuous information interaction with the environment.

[0054] The intelligent decision-making method for Mahjong in this application establishes an optimal play decision model by acquiring historical scene information, hidden information, and prospective features. This solves the problems caused by information gaps, huge information sets, and uncertainty in multi-player decision-making in incomplete information games like Mahjong. Specifically, the intelligent decision-making method for Mahjong in this application not only incorporates basic features but also hidden information and prospective features. By incorporating hidden information and prospective features, the model can converge faster and more efficiently, thus making play decisions based on the convergence results. Furthermore, the intelligent decision-making method in this application classifies the above features and uses different feature codes for different categories of features. This ensures that the model fully acquires the information of each feature, guaranteeing the accuracy of the information. A hierarchical reinforcement learning strategy is used to address the problem of difficulty in obtaining positive samples and the small number of positive samples in the early stages of model training. This ensures that reinforcement learning has a sufficient number of positive samples to drive model updates and improvements, guaranteeing the optimal play decision results of the optimal decision model.

[0055] Example 1

[0056] Please see Figure 1 The image shows an intelligent decision-making method for Mahjong according to a first embodiment of the present invention, the method comprising steps S101 to S103:

[0057] S101. Obtain historical scene information of Mahjong, and obtain hidden information of Mahjong based on historical scene information. Hidden information includes the opponent's hand information and the wall of tiles information. Obtain the prospective features of Mahjong play based on historical scene information combined with search tree algorithm.

[0058] In the above steps, visible information about mahjong from a human perspective is obtained, including discarded tiles, discarded tiles, and the dealer, etc. Then, a search tree is used to search for possible situations within the next three moves after a tile is played, thus providing some forward-looking feature information. This yields forward-looking features of mahjong play, which are search extension features within a preset step size after the current node of the search tree. For example, the possible discard and draw situations within the next three moves after a tile is played are used as forward-looking features. Afterward, hidden information about mahjong is obtained, including information about competitors' hands and the wall of tiles. Hidden information is also added in the early stages of training.

[0059] Before feature encoding, the above information is classified and an innovative feature encoding method is adopted to finally obtain a three-dimensional feature matrix, which consists of 0 and 1.

[0060] S102. Encode the historical scene information, hidden information and forward features to obtain the encoded historical feature matrix. Input the historical feature matrix into the reinforcement learning model for training to obtain the best card-playing decision model.

[0061] This method incorporates hidden information, including the opponent's hand and the cards in the wall, to facilitate understanding of the opponent and the cards in the wall. This allows the model to converge faster and more efficiently, increasing the probability of winning. A lookahead feature search component is added in conjunction with the search tree algorithm. The lookahead features mainly include recommended card-playing results obtained through the search tree, including the card type and scoring pattern of the recommended card-playing action.

[0062] A hierarchical reinforcement learning strategy is employed to address the challenges of obtaining and maintaining sufficient positive samples in the early stages of model training. This ensures that reinforcement learning has enough positive samples to drive model updates and improvements, leading to better convergence of the objective function, more efficient model decision-making, and ultimately, a more effective card-playing model. As a concrete example, the number of positive samples can be understood as the number of rounds won by the model after making its decisions. By continuously updating the model through a hierarchical reinforcement learning strategy, a sufficient number of positive samples are obtained, making the model's decisions more aligned with the requirements and resulting in a more effective card-playing model.

[0063] By incorporating basic features and encoding scene information, along with hidden and prospective features, the model converges faster and more efficiently. The added hidden information is then gradually removed later in the training process. Furthermore, the features are categorized, with different feature encodings for different categories, ensuring the model fully captures the information from each feature and guarantees its accuracy. A hierarchical reinforcement learning strategy is employed to address the difficulty in obtaining and the limited number of positive samples in the early stages of training. This ensures sufficient positive samples to drive model updates and improvements, resulting in a high-performing card-playing model. This addresses the problems caused by information gaps, massive information sets, and uncertainty in multi-player decision-making in incomplete information games like Mahjong.

[0064] S103. Obtain the current situation information and perform feature encoding on the current situation information to obtain the current feature matrix. Input the current feature matrix into the optimal card-playing decision model to obtain the mahjong card-playing decision result. Make a card-playing decision based on the card-playing decision result.

[0065] In summary, the intelligent decision-making method for mahjong in the above embodiments of the present invention improves the win rate and final score in mahjong games by acquiring hidden information about the game and avoiding situations where inaccurate judgment of hidden information affects the player's card-playing decisions. Furthermore, by combining the acquisition of prospective features of mahjong card-playing, a historical feature matrix is ​​obtained by feature encoding historical scene information, hidden information, and prospective features. The historical feature matrix is ​​then used to train a reinforcement learning model to obtain an optimal card-playing decision model, ensuring that each card-playing decision in the mahjong game is the optimal decision. When the current scene information is acquired, the optimal card-playing decision model is combined to obtain the mahjong card-playing decision result and make a card-playing decision, enabling the player to ultimately obtain the best score. This solves the technical problem in the prior art where the card-playing decision scheme fails to consider hidden information, leading to inaccurate judgment of the current game situation and affecting the card-playing decision.

[0066] Example 2

[0067] Please refer to Figure 2 The figure shows an intelligent decision-making method for mahjong according to a second embodiment of the present invention, the method comprising steps S201 to S203:

[0068] S201. Obtain historical scene information of Mahjong, and obtain hidden information of Mahjong based on historical scene information. Hidden information includes the opponent's hand information and the wall of tiles information. Obtain the prospective features of Mahjong play based on historical scene information combined with search tree algorithm.

[0069] Taking Chinese four-player mahjong as an example, specifically, the scene information in incomplete information machine game games like Chinese four-player mahjong is first classified into card set features, sequence features, integer features, and look-ahead features, and different encoding formats are adopted for different parts of the features.

[0070] Specifically, the card-playing actions in the card set features, sequence features, and forward-looking features are encoded using matrix coding. The coding format is as follows: w represents Wan (characters), s represents Tiao (bamboo), t represents Tong (circles), and D, N, X, B, Z, F, and Ba represent East, South, West, North, Red Dragon, Green Dragon, and White Dragon respectively. Figure 4 Taking private hand cards as an example, for Figure 4 The private hand cards in the game are encoded, and the encoded result is as follows: Figure 5 As shown, by Figure 4 It can be deduced that the private hand contains one "two of characters" and two "four of characters". Therefore, in Figure 5 In the diagram, the first row below "2w" is "1", and the first and second rows below "4w" are both "1", indicating that there is 1 "2w" and 2 "4w".

[0071] S202. Encode the historical scene information, hidden information and forward features to obtain the encoded historical feature matrix. Input the historical feature matrix into the reinforcement learning model for training to obtain the best card-playing decision model.

[0072] Furthermore, the encoding format for card types and hand patterns in integer features and lookahead features uses dummy encoding, which allows all values ​​to be set to 0, but only entire rows can be set to 1 when representing specific features. For example, when representing the number of a specific card as 4, its encoded feature is as follows: Figure 6 As shown.

[0073] Encoding scene information allows the model to better recognize it, enabling its application in reinforcement learning training. Furthermore, classifying these features and using different feature codes for different categories ensures that the model fully captures the information from each feature, guaranteeing its accuracy.

[0074] As a concrete example, such as Figure 3 As shown, step S202 may further include steps S2021 to S2024:

[0075] S2021. Input the historical feature matrix into the reinforcement learning model. The reinforcement learning model includes an information extraction module. The information extraction module extracts and understands the historical feature matrix and outputs the understood information.

[0076] The encoded feature matrix is ​​input into the deep learning information extraction module in the reinforcement learning model. This module is responsible for extracting and understanding the meaning of 0 and 1 at different positions in the input feature matrix, so that the reinforcement learning model can fully grasp the meaning and thus affect the final decision-making effect.

[0077] S2022, The reinforcement learning model acquires the understood information, then configures a training strategy and performs hierarchical training.

[0078] A hierarchical reinforcement learning strategy is adopted to adjust the difficulty of the opponent according to the training progress, which solves the problem of difficulty in obtaining positive samples and the small number of positive samples in the early stage of model training, so that reinforcement learning has enough positive samples to drive the model to update and improve.

[0079] Specifically, the training opponents were categorized into three random card-playing models, two random card-playing models plus one simple rule-based model, and finally, three human-level average card-playing models. In essence, the hierarchical reinforcement learning strategy stratifies the difficulty of the opponents: the three random card-playing models represent the lowest difficulty level, the two random card-playing models plus one simple rule-based model represent a relatively higher difficulty level, and the three human-level average card-playing models represent the highest difficulty level. Experiments have demonstrated that this hierarchical reinforcement learning strategy enables the reinforcement learning algorithm to converge effectively to a good level.

[0080] S2023. Obtain the policy entropy and divergence terms for model training based on the training results.

[0081] The optimization function is the objective function of a novel approximate policy optimization algorithm proposed in this method. It combines the loss functions of the Actor network and the Critic network, and can update these two networks simultaneously. It also dynamically adjusts the policy entropy (EntPolicy) and Approxkl divergence. Experiments show that it can enable the model to converge faster and with better performance.

[0082] Specifically, based on the scene information, the policy entropy of the current updated game trajectory and the divergence term of the current updated game trajectory in the objective function of the model are obtained respectively. The policy entropy of the current updated game trajectory includes the current policy entropy value, and the divergence term of the current updated game trajectory includes the current divergence value. The current policy entropy value and the current divergence value are adjusted respectively so that the current policy entropy value converges to the target policy entropy value and the current divergence value converges to the target divergence value.

[0083] S2024. Dynamically adjust and optimize the reinforcement learning model based on the policy entropy and divergence terms to obtain the optimal card-playing decision model.

[0084] First, a policy entropy term is added to the objective function. This term helps the PPO algorithm model balance the choice between exploration and exploitation during training. Second, an Approximation divergence term is added to the objective function. This prevents the model performance from gradually diverging as the Approximation divergence term becomes increasingly large. Finally, target values ​​are set for both the policy entropy and the Approximation divergence term—target policy entropy and target divergence values, respectively. During training, the parameters of these two newly added terms can be dynamically adjusted, allowing them to dynamically converge to their respective target values.

[0085] Specifically, the dynamic adjustment formula for the policy entropy term is:

[0086] γ←γ+c1(Ent targ -Ent θ )

[0087] Where γ is the weighting coefficient of the policy entropy in the objective function, typically initialized to 0.01, Ent targ It is the target value of the policy entropy, Ent θ c1 is the current policy entropy value, and c1 is the coefficient for dynamically adjusting the policy entropy, which is usually a very small positive number.

[0088] The above formula dynamically adjusts the policy entropy (EntPolicy) term, allowing it to dynamically approach the target policy entropy value during training. Including this term allows the PPO algorithm model to balance the choice between exploration and exploitation during training.

[0089] Policy entropy (EntPolicy) reflects the certainty of which card the decision model will play. The smaller the policy entropy, the greater the certainty of playing a particular card, which is more conducive to model convergence. The Approximation divergence term reflects the magnitude of the difference between two updates. During training, dynamic adjustments to the divergence term control the difference in updates, ensuring that updates proceed at a reasonable pace. For a converged model, the magnitude of the difference between two updates should be close to zero. These two terms are optimizations to the standard PPO algorithm.

[0090] Furthermore, the dynamic adjustment formula for the divergence term is:

[0091] δ←δ+c2(KL targ -KL θ )

[0092] Where δ is the weighting coefficient of the divergence in the objective function, typically initialized to 0.005, KL targ It is the target value of the divergence, KL θc1 is the current divergence value, and c2 is the coefficient for dynamically adjusting the divergence, which is usually a very small positive number.

[0093] The above formula dynamically adjusts the Approxkl divergence term, allowing it to dynamically approach the target divergence value during training. Adjusting the Approxkl divergence controls the magnitude of the difference between two consecutive model updates. In this application, by dynamically adjusting the coefficients during training, the model is optimized towards the target direction.

[0094] The divergence term represents the difference between two updates of the decision-making model, such as the process of gradually improving decision-making ability from very poor to very strong. From a macroscopic perspective, this process is iterative and updates in a positive direction. Therefore, the divergence term can be represented as the difference in decision-making ability between two consecutive updates. Because the divergence term represents the difference between two updates, it does not have an initial value.

[0095] Specifically, the optimization function is:

[0096] J(θ)=αL C +βL V +γEnt θ +δKL θ

[0097] Where αL C This is the loss function of the actor network, where α is the correlation coefficient of the loss function, usually 1, and βL... V γEnt is the loss function of the critic network, β is the correlation coefficient of this loss function, usually set to 0.5. θ It is the policy entropy of the current updated game trajectory, γ is the correlation coefficient of the policy entropy, which is a very small positive number, δKL θ δ represents the divergence term of the currently updated game trajectory, and δ is the correlation coefficient of the divergence term, which is also a very small positive number.

[0098] The loss functions of the actor network and the critic network are obtained by matrix calculation using historical features. The loss values ​​of the actor network and the critic network are then calculated based on their respective loss functions. The actor network loss value includes the probability of playing each card, and the critic network loss value includes the predicted score for the current hand. The convergence value of the reinforcement learning model is calculated by combining the actor network loss value, the critic network loss value, the policy entropy value of the currently updated game trajectory, and the divergence term value of the currently updated game trajectory. Based on the calculated convergence value, the optimal card-playing decision model is obtained.

[0099] The inclusion of forward-looking features and hidden information during feature encoding allows for more comprehensive information acquisition. This includes not only hidden information such as the wall of cards and opponents' hands, but also potential future information, thereby reducing uncertainty. In the improved PPO algorithm's objective function, this is reflected in reducing γEnt. θ Ent in θ This reduces the strategy entropy in the current situation. θ The reduction in means the equation is closer to 0, thus reducing training time and allowing the model to converge faster. In decision-making systems, this translates to shorter training times and, in more reasonable decisions during games, higher average scores.

[0100] In this application, the model obtains different current policy entropy and divergence values ​​in each training iteration. By setting target policy entropy and target divergence values ​​and adjusting γ and δ through continuous optimization iterations, γ and δ that meet the usage requirements are obtained, thereby adjusting the current policy entropy and current divergence values ​​so that the current policy entropy and current divergence values ​​converge to the target policy entropy and target divergence values, respectively. The converged current policy entropy, current divergence values, and corresponding γ and δ are then applied to the optimization function to obtain the convergence value of the objective function at this time.

[0101] Taking adjusting the current policy entropy as an example, if the current policy entropy value is too large compared to the target policy entropy value, decrease γ. This causes the current policy entropy value in the optimization function to converge to the target policy entropy value. The current policy entropy value and γ are then applied to the optimization function to obtain the convergence value of the objective function, making the convergence value close to a preset threshold. The principle for adjusting the current divergence value is the same; refer to this section for further understanding.

[0102] If γ is relatively large, it means that the policy entropy is more important. Therefore, during optimization, it is necessary to increase the weight of the policy entropy in the model output.

[0103] If the policy entropy value is large, it means that the probability of playing each card in the model can be interpreted as the same, causing the model to be uncertain about which card to play. In this case, the model is not performing well and is not conducive to card-playing decisions. If the policy entropy value is large, we reduce γ so that the policy entropy value output by the model will decrease in the next optimization, and then gradually decrease until the policy entropy value reaches the target policy entropy value. Setting a target policy entropy value is to ensure that the policy entropy value output by the model in each iteration is roughly the same as the target policy entropy value, thus facilitating the model's card-playing decisions.

[0104] Similarly, the divergence term is added to ensure that the update magnitude of subsequent updates is not too large compared to previous updates. If the magnitude is too large, it indicates a significant difference in playing styles, which is detrimental to making moves. The divergence term ensures that the style and moves of subsequent moves are consistent with those of previous moves, thus facilitating the model's move decisions. In this application, the model's playing rules are driven by the divergence term and policy entropy, and the move decisions are then made by the objective function.

[0105] S203. Obtain the current situation information and perform feature encoding on the current situation information to obtain the current feature matrix. Input the current feature matrix into the optimal card-playing decision model to obtain the mahjong card-playing decision result. Make a card-playing decision based on the card-playing decision result.

[0106] In the early stages, the optimal card-playing decision model is obtained through model training. When a card-playing decision needs to be made during a mahjong game, the optimal card-playing decision model obtains the current situation information of the current game state, performs feature encoding based on the current situation information to obtain the current feature matrix, and then inputs the current feature matrix into the optimal card-playing decision model. The optimal card-playing decision model extracts and identifies the current feature matrix, and then obtains the optimal mahjong card-playing decision result.

[0107] like Figure 7 and Figure 8 As can be seen, the optimal card-playing decision model obtained by this method converges faster and achieves higher performance after convergence. Compared with the standard PPO algorithm and standard basic feature information, this scheme has look-ahead features, incorporates hidden information in the early stage, and uses a new optimization function. Through experiments comparing the proposed method with the standard PPO algorithm to implement a decision model for a four-player mahjong game with incomplete information, using the same hyperparameters, the experimental results are shown in the figure below. `ep_reward_mean` represents the average reward per game in each batch_size, which is a direct standard for measuring algorithm performance. Figure 7 For the standard PPO algorithm, Figure 8 The PPO algorithm after applying this method is shown. Experimental results demonstrate that this method converges faster and achieves higher performance after convergence. Therefore, the intelligent decision-making method for Mahjong in this application can better determine card plays, solving the technical problem in existing card-playing decision schemes that fail to consider hidden information, leading to inaccurate judgments of the current game situation and affecting card-playing decisions.

[0108] In summary, the intelligent decision-making method for mahjong in the above embodiments of the present invention improves the win rate and final score in mahjong games by acquiring hidden information about the game and avoiding situations where inaccurate judgment of hidden information affects the player's card-playing decisions. Furthermore, by combining the acquisition of prospective features of mahjong card-playing, a historical feature matrix is ​​obtained by feature encoding historical scene information, hidden information, and prospective features. The historical feature matrix is ​​then used to train a reinforcement learning model to obtain an optimal card-playing decision model, ensuring that each card-playing decision in the mahjong game is the optimal decision. When the current scene information is acquired, the optimal card-playing decision model is combined to obtain the mahjong card-playing decision result and make a card-playing decision, enabling the player to ultimately obtain the best score. This solves the technical problem in the prior art where the card-playing decision scheme fails to consider hidden information, leading to inaccurate judgment of the current game situation and affecting the card-playing decision.

[0109] Example 3

[0110] Please see Figure 9 The figure shows an intelligent decision-making system for mahjong according to a third embodiment of the present invention. The system includes...

[0111] The acquisition module is used to acquire historical scene information of mahjong, acquire hidden information of mahjong based on the historical scene information, the hidden information includes the opponent's hand information and the wall of tiles information, and acquire prospective features of mahjong play based on the historical scene information and the search tree algorithm, the prospective features being the search extension features within a preset step size after the current node of the search tree.

[0112] The training module is used to encode the historical scene information, the hidden information, and the prospective features to obtain the historical feature matrix after feature encoding. The historical feature matrix is ​​then input into the reinforcement learning model for training to obtain the optimal card-playing decision model.

[0113] The decision module is used to acquire current situation information, encode the current situation information to obtain a current feature matrix, input the current feature matrix into the optimal play decision model to obtain the mahjong play decision result, and make a play decision based on the play decision result.

[0114] In summary, the intelligent decision-making system for mahjong in the above embodiments of the present invention improves the win rate and final score in mahjong games by acquiring hidden information about the game and avoiding situations where inaccurate judgment of hidden information affects the player's card-playing decisions. Furthermore, by combining the acquisition of prospective features of mahjong card-playing, a historical feature matrix is ​​obtained by feature encoding historical scene information, hidden information, and prospective features. The historical feature matrix is ​​then used to train a reinforcement learning model to obtain an optimal card-playing decision model, ensuring that each card-playing decision in the mahjong game is the optimal decision. When the current scene information is acquired, the optimal card-playing decision model is combined to obtain the mahjong card-playing decision result and make a card-playing decision, enabling the player to ultimately obtain the best score. This solves the technical problem in the prior art where the card-playing decision scheme fails to consider hidden information, leading to inaccurate judgment of the current game situation and affecting the card-playing decision.

[0115] Furthermore, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods described above.

[0116] Furthermore, embodiments of the present invention also propose a data processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the methods described above.

[0117] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0118] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0119] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0120] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0121] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. An intelligent decision-making method for Mahjong, characterized in that, The method includes: The historical scene information of Mahjong is obtained, and the hidden information of Mahjong is obtained based on the historical scene information. The hidden information includes the opponent's hand information and the wall information. Based on the historical scene information and the search tree algorithm, the prospective features of Mahjong play are obtained. The prospective features are the search extension features within a preset step size after the current node of the search tree. The historical scene information, the hidden information, and the prospective features are feature-encoded to obtain a historical feature matrix. The historical feature matrix is ​​then input into a reinforcement learning model for training to obtain the optimal card-playing decision model. The current situation information is obtained and the current situation information is feature-encoded to obtain the current feature matrix. The current feature matrix is ​​input into the optimal play decision model to obtain the mahjong play decision result. The play decision is made based on the play decision result. The step of inputting the historical feature matrix into a reinforcement learning model for training to obtain the optimal card-playing decision model includes: The historical feature matrix is ​​input into a reinforcement learning model, which includes an information extraction module. The information extraction module extracts and understands the historical feature matrix and outputs the understood information. The reinforcement learning model acquires the understood information, then configures a training strategy and performs hierarchical training. The policy entropy and divergence terms of the model training are obtained based on the training results. The reinforcement learning model is dynamically adjusted and optimized based on the strategy entropy term and the divergence term to obtain the optimal card-playing decision model; The dynamic adjustment formula for the strategy entropy term is: γ←γ+c1(Ent targ -Ent θ ) Where γ is the weighting coefficient of the policy entropy in the objective function, Ent targ It is the target value of the policy entropy, Ent θ c1 is the current policy entropy value, and c1 is the coefficient for dynamically adjusting the policy entropy. The dynamic adjustment formula for the divergence term is: δ←δ+c2(KL targ -KL θ ) Where δ is the weighting coefficient of the divergence in the objective function, KL targ It is the target value of the divergence, KL θ c1 is the current divergence value, and c2 is the coefficient for dynamically adjusting the divergence.

2. The intelligent decision-making method for Mahjong according to claim 1, characterized in that, In the step of adjusting and optimizing the reinforcement learning model based on the policy entropy term and the divergence term to obtain the optimal card-playing decision model, the optimization function is: J(θ)=αL C +βL V +γEnt θ +δKL θ Where αL C This is the loss function of the actor network, where α is the correlation coefficient of the loss function, and βL... V γEnt is the loss function of the critic network, β is the correlation coefficient of this loss function, and γEnt is the loss function of the critic network. θ It is the policy entropy of the current updated game trajectory, γ is the correlation coefficient of the policy entropy, and δKL θ δ represents the divergence term of the currently updated game trajectory, where δ is the correlation coefficient of the divergence term.

3. The intelligent decision-making method for Mahjong according to claim 1, characterized in that, The step of inputting the historical feature matrix into the reinforcement learning model for model training to obtain the optimal card-playing decision model includes: The loss functions of the actor network and the critic network are obtained by matrix calculation using historical features. The loss values ​​of the actor network and the critic network are obtained based on the loss function of the actor network and the loss function of the critic network. The loss value of the actor network includes the probability of playing each card, and the loss value of the critic network includes the predicted score under the current card. The convergence value of the reinforcement learning model is calculated by combining the loss value of the actor network, the loss value of the critic network, the policy entropy value of the current updated game trajectory, and the divergence term value of the current updated game trajectory. Based on the calculated convergence value, the optimal card-playing decision model is obtained.

4. An intelligent decision-making system for Mahjong, characterized in that, The system includes The acquisition module is used to acquire historical scene information of mahjong, acquire hidden information of mahjong based on the historical scene information, the hidden information includes the opponent's hand information and the wall of tiles information, and acquire prospective features of mahjong play based on the historical scene information and the search tree algorithm, the prospective features being the search extension features within a preset step size after the current node of the search tree. The training module is used to encode the historical scene information, the hidden information, and the prospective features to obtain the historical feature matrix after feature encoding. The historical feature matrix is ​​then input into the reinforcement learning model for training to obtain the optimal card-playing decision model. The decision module is used to acquire current situation information, encode the current situation information to obtain a current feature matrix, input the current feature matrix into the optimal play decision model to obtain the mahjong play decision result, and make a play decision based on the play decision result; The step of inputting the historical feature matrix into a reinforcement learning model for training to obtain the optimal card-playing decision model includes: The historical feature matrix is ​​input into a reinforcement learning model, which includes an information extraction module. The information extraction module extracts and understands the historical feature matrix and outputs the understood information. The reinforcement learning model acquires the understood information, configures a training strategy, and performs hierarchical training. Based on the training results, the strategy entropy and divergence terms of the model training are obtained. The reinforcement learning model is dynamically adjusted and optimized based on the strategy entropy and divergence terms to obtain the optimal card-playing decision model. The dynamic adjustment formula for the strategy entropy term is: γ←γ+c1(Ent targ -Ent θ ) Where γ is the weighting coefficient of the policy entropy in the objective function, Ent targ It is the target value of the policy entropy, Ent θ c1 is the current policy entropy value, and c1 is the coefficient for dynamically adjusting the policy entropy. The dynamic adjustment formula for the divergence term is: δ←δ+c2(KL targ -KL θ ) Where δ is the weighting coefficient of the divergence in the objective function, KL targ It is the target value of the divergence, KL θ c1 is the current divergence value, and c2 is the coefficient for dynamically adjusting the divergence.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the intelligent decision-making method for Mahjong as described in any one of claims 1-3.

6. A data processing apparatus, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the intelligent decision-making method for Mahjong as described in any one of claims 1-3.

Images