A decision-making method and system guided by phased rewards.

Abstract

This invention discloses a decision-making system guided by phased rewards. The system includes: a memory, a processor, and a computer module stored in the memory. The computer module provides a computer program that runs on the processor. The computer module includes: a feature encoding module for acquiring current game information and converting the features included in the current game information into a high-dimensional feature vector and a phased feature subset; a game prediction module for receiving the phased feature subset and determining the final ranking probability distribution based on the phased feature subset; a reward calculation module for acquiring the corresponding phased reward signal based on the final ranking probability distribution; and a decision network module for receiving the high-dimensional feature vector and the phased reward signal, and determining the final decision action based on the high-dimensional feature vector and the phased reward signal. This invention solves the problem of low decision quality in existing decision-making methods.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a decision-making method and system oriented towards phased rewards. Background Technology

[0002] Mahjong is a typical game of incomplete information, with a wide following in Asia. From a game theory perspective, mahjong possesses extremely complex characteristics: the game state space is enormous, theoretically reaching 10^120 possible states; there is a large amount of hidden information, including other players' hands and the remaining tiles in the wall; it requires simultaneous consideration of offensive strategies (aiming for a winning hand) and defensive strategies (avoiding discarding tiles that would allow the opponent to win); and it involves complex risk assessments and benefit trade-offs. These characteristics make mahjong one of the major challenges in the field of artificial intelligence research.

[0003] With AlphaGo's success in Go, deep reinforcement learning technology has received widespread attention. However, unlike Go, a game of complete information, Mahjong's incomplete information nature presents additional challenges for AI design. Particularly within the reinforcement learning framework, designing reasonable reward signals for intermediate decision-making steps has become a core problem. Mahjong games typically only have a clear score at the end of each round, and there is no immediate feedback for each decision made during the game. This reward sparsity severely impacts the training efficiency and final performance of reinforcement learning algorithms.

[0004] Currently, there are three main technical approaches to solving the decision-making problem in Mahjong AI. The first is a rule-based approach, which uses a large number of manually designed if-then rules to guide decision-making. The advantage of this approach is its strong interpretability, but its disadvantages include the inability to cover all game situations, poor performance in complex scenarios, and an inability to adapt to different opponents' playing styles. The second is a supervised learning approach, which learns from the game records of many high-level players to mimic human decision-making. This approach can achieve good performance, but it heavily relies on the quality and quantity of training data and struggles to surpass the highest level achieved in the training data. The third is a reinforcement learning approach, which continuously optimizes strategies through self-play. Theoretically, this can achieve superhuman performance, but it faces the challenge of designing effective rewards.

[0005] In terms of feature representation, existing methods typically only encode basic game state information, such as hand cards, discards, and secondary runs, lacking a deep understanding of the situation and prediction of future trends. This forces neural networks to learn complex game rules from scratch, resulting in low decision-making quality. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a decision-making method and system oriented towards phased rewards, aiming to solve the problem of low decision-making quality in existing decision-making methods.

[0007] The embodiments of the present invention are implemented as follows:

[0008] A decision-making system guided by phased rewards, the system comprising:

[0009] A memory, a processor, and a computer module stored in the memory, the computer module being used to provide a computer program that runs on the processor;

[0010] The computer module includes:

[0011] The feature encoding module is used to acquire the current hand information and convert the features included in the current hand information into high-dimensional feature vectors and staged feature subsets.

[0012] The game prediction module is used to receive a phased feature subset and determine the final ranking probability distribution based on the phased feature subset;

[0013] The reward calculation module is used to obtain the corresponding staged reward signal based on the final ranking probability distribution.

[0014] The decision network module is used to receive high-dimensional feature vectors and staged reward signals, and determine the final decision action based on the high-dimensional feature vectors and staged reward signals.

[0015] Among them, the high-dimensional feature vector includes at least basic features, river features, secondary reveal information, expected value features, and special rule features. The basic features include at least hand cards, score, and ranking. The river features include at least the features of one's own river cards and other players' river cards, as well as the time decay weight. The secondary reveal information includes at least the secondary reveal card groups of all players. The expected value features include at least the analysis of the expected value of the cards and the calculation of the expected value of the effective cards. The special rule features include at least the special rule state.

[0016] Furthermore, in the aforementioned decision-making system oriented towards phased rewards, the expected value includes the probability of drawing a winning hand, the probability of winning, and the expected score.

[0017] The probability of reaching a ready hand (or "ready hand") represents the likelihood of achieving that state in future rounds.

[0018]

[0019] The probability of winning a hand represents the likelihood of successfully winning in future rounds:

[0020]

[0021] Expected score represents the score expected in future rounds:

[0022] ;

[0023] in, Indicates the current round. This represents all possible single cards in the remaining undrawn tiles. Indicates that the player touched " in the current round "The probability of this card..."

[0024] It means that you touched " in the current round "After that, the next round" The probability of reaching a ready hand state; This indicates the probability of drawing a winning tile in the current round. This indicates the probability of drawing a non-winning tile in the current round. This means that if you draw a non-winning tile in the current round, the next round... The probability of successfully winning a hand; For the current round, touch " The probability of " If you touch " "The score you get for successfully winning the hand is..." For indicator functions, if "is touched" "If you can win the hand later..." ,otherwise , For the current round, touch " "If you don't win the hand, the next round..." The expected score.

[0025] Furthermore, in the aforementioned decision-making system oriented towards phased rewards, the game prediction module includes:

[0026] Stage Feature Integration Unit: Used to generate feature vectors for each stage based on the staged feature subsets, and then arrange them in the order of stage occurrence to obtain a stage feature sequence;

[0027] Temporal modeling unit: used to process stage feature sequences and capture the influence of the previous stage on the next stage;

[0028] Ranking distribution unit: used to map the temporal features processed by the temporal modeling unit into the final ranking probability distribution through a fully connected layer.

[0029] Furthermore, in the aforementioned decision-making system guided by phased rewards, the reward calculation module is specifically used for:

[0030] Calculate the expected ranking for the current stage based on the ranking probability distribution and the ranking point vector:

[0031] ;

[0032] By comparing the expected ranking in the current stage with the expected ranking in the previous stage, the reward value is obtained:

[0033] ;

[0034] in, For the ranking point vector, For ranking probability, if Give a positive reward; if Negative rewards will be given.

[0035] Furthermore, in the aforementioned decision-making system oriented towards phased rewards, the decision network module is specifically used for:

[0036] The high-dimensional feature vector from the feature encoding module is received, and 1024-dimensional deep semantic features are extracted through a residual network and a channel attention mechanism.

[0037] Input 1024-dimensional deep semantic features into DDQN to separate state value (V) and action advantage (A). State value V is the overall value of the current hand state, and action advantage A is the advantage of each action relative to other actions.

[0038] Through formula Calculate the Q value for each action (the mean of the action advantage) to avoid action advantage bias;

[0039] Apply an action mask to the Q-value, and then use an ε-greedy strategy to select the action with the highest Q-value to determine the final decision action;

[0040] Among them, the phased reward signal provides value feedback during the training process of the decision network, which is used to optimize the network's prediction accuracy of action value.

[0041] Furthermore, in the aforementioned decision-making system oriented towards phased rewards, the process of converting the features included in the current game information into a high-dimensional feature vector and a phased feature subset includes:

[0042] Real-time scanning to see if key events are triggered in the current game. The initial state of the entire game is taken as stage 1. Each time a key event is triggered, the stage number is automatically incremented by 1. The number of key events triggered in the current game is recorded, the current stage number is determined, and numerical encoding is performed.

[0043] Read all the card river data of the current game, count the cumulative number of all cards played in the card river, and use the cumulative number as the feature value of the number of cards played;

[0044] The status of each player is monitored in real time, and the status is divided into 3 categories and encoded in binary / multi-base. The encoding results of each player are arranged in order of player to form the status feature items of each player and their hand.

[0045] Read the real-time scores of each player in the current stage, calculate the score difference for each player, and arrange the score differences of all players in order to form a score change feature item;

[0046] The current stage number, the feature value of the number of cards played, the feature items of each player's winning hand status, and the feature items of score changes are integrated into a structured feature vector in a fixed order to obtain a staged feature subset.

[0047] Furthermore, in the aforementioned decision-making system oriented towards phased rewards, the system further includes:

[0048] The training module is used to train the decision network module using a hybrid offline-online training mode. In the offline stage, high-quality historical game data is used, and the CQL algorithm is applied to prevent Q-value overestimation. In the online stage, the strategy is continuously optimized and dynamically adjusted through self-play.

[0049] Another object of the present invention is to provide a decision-making method guided by phased rewards, implemented using the aforementioned decision-making system guided by phased rewards, the method comprising:

[0050] Obtain the current hand information and convert the features included in the current hand information into high-dimensional feature vectors and staged feature subsets;

[0051] Receive a phased feature subset and determine the final ranking probability distribution based on the phased feature subset;

[0052] Obtain the corresponding staged reward signal based on the final ranking probability distribution;

[0053] It receives high-dimensional feature vectors and staged reward signals, and determines the final decision action based on the high-dimensional feature vectors and staged reward signals.

[0054] This invention acquires current game information and converts the features included in the current game information into high-dimensional feature vectors and staged feature subsets; receives the staged feature subsets and determines the final ranking probability distribution based on the staged feature subsets; obtains the corresponding staged reward signal based on the final ranking probability distribution; receives the high-dimensional feature vectors and staged reward signals, and determines the final decision action based on the high-dimensional feature vectors and staged reward signals; in addition to basic hand, river, and secondary revealed information, it innovatively introduces expected value features, temporal decay features, and special rule features. The feature encoding adopts a hierarchical design, using differentiated encoding strategies for different types of features: the river information introduces temporal decay weights, reflecting the importance of recently discarded cards; for some special rules of Mahjong, an extension interface is retained to introduce dedicated feature dimensions, ensuring that the rule characteristics are fully expressed, thus solving the problem of low decision quality in existing decision-making methods. Attached Figure Description

[0055] Figure 1 This is a structural block diagram of a decision-making system guided by phased rewards in the third embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the feature extraction network structure in the decision network module of the decision system guided by phased rewards in the first embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of the DDQN network structure in the decision network module of the decision-making system guided by phased rewards in the first embodiment of the present invention.

[0058] Figure 4 This is a schematic diagram of the overall decision-making process of the decision-making system guided by phased rewards in the first embodiment of the present invention.

[0059] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0060] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0061] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0063] First Embodiment

[0064] Please see Figure 1 The figure shows a decision-making system guided by phased rewards proposed in the first embodiment of the present invention. The system includes:

[0065] A memory, a processor, and a computer module stored in the memory, the computer module being used to provide a computer program that runs on the processor;

[0066] The computer module includes:

[0067] The feature encoding module 100 is used to acquire the current hand information and convert the features included in the current hand information into a high-dimensional feature vector and a staged feature subset.

[0068] The game prediction module 200 is used to receive a phased feature subset and determine the final ranking probability distribution based on the phased feature subset;

[0069] The reward calculation module 300 is used to obtain the corresponding staged reward signal based on the final ranking probability distribution.

[0070] The decision network module 400 is used to receive high-dimensional feature vectors and staged reward signals, and determine the final decision action based on the high-dimensional feature vectors and staged reward signals.

[0071] The high-dimensional feature vector includes at least basic features, river features, secondary information, expected value features, and special rule features. The basic features include at least hand cards, score, and ranking.

[0072] The features of the river card include at least the features of the player's own river card and the other players' river cards, as well as the time decay weight. The secondary reveal information includes at least the secondary reveal card groups of all players. The expected value features include at least the analysis of the hand and the expected value calculation of the effective cards. The special rule features include at least the special rule status.

[0073] Specifically, the feature encoding module is responsible for converting the game state into a high-dimensional feature vector, which not only contains currently visible information but also provides forward-looking information through expectation value calculation. The high-dimensional feature vector includes:

[0074] Basic features (16 dimensions): hand, score, ranking, etc.;

[0075] River characteristics (259 dimensions): characteristics of one's own river and other players' river, as well as time decay weights;

[0076] Secondary Deck Information (80 Dimensions): All players' secondary decks;

[0077] Expected value features (100 dimensions): Analysis of listening patterns and calculation of the expected value of effective cards;

[0078] Special rule features (22 dimensions): Special rule states specific to regional Mahjong;

[0079] Understandably, in addition to basic hand, discarded tile, and secondary discard information, it innovatively introduces expected value features, temporal decay features, and special rule features. Feature encoding adopts a hierarchical design, employing differentiated encoding strategies for different types of features: temporal decay weights are introduced into discarded tile information to reflect the importance of recently discarded tiles; expected values ​​use multiple normalization methods to adapt to different numerical ranges; and for some special rules of Mahjong, extended interfaces are retained to introduce dedicated feature dimensions, ensuring that the characteristics of the rules are fully expressed.

[0080] Specifically, the calculation of expected value mainly includes three key indicators;

[0081] Probability of drawing a winning hand, probability of winning, and expected score;

[0082] The probability of reaching a ready hand (or "ready hand") represents the likelihood of achieving that state in future rounds.

[0083]

[0084] The probability of winning a hand represents the likelihood of successfully winning in future rounds:

[0085]

[0086] Expected score represents the score expected in future rounds:

[0087] ;

[0088] in, Indicates the current round. This represents all possible single cards in the remaining undrawn tiles. Indicates that the player touched " in the current round "The probability of this card..."

[0089] It means that you touched " in the current round "After that, the next round" The probability of reaching a ready hand state; This indicates the probability of drawing a winning tile in the current round. This indicates the probability of drawing a non-winning tile in the current round. This means that if you draw a non-winning tile in the current round, the next round... The probability of successfully winning a hand; For the current round, touch " The probability of " If you touch " "The score you get for successfully winning the hand is..." For indicator functions, if "is touched" "If you can win the hand later..." ,otherwise , For the current round, touch " "If you don't win the hand, the next round..." The expected score.

[0090] For example, the three forward-looking indicators (scalar values) need to undergo three steps: multi-dimensional expansion, normalization, and structured assembly, to finally form a 100-dimensional feature. The specific process is as follows:

[0091] Extended probability of winning: Calculates the probability of winning in the next 1 / 3 / 5 / 10 rounds (4 sub-indicators), covering short-term and long-term offensive potential;

[0092] Extended winning probability: Calculates the probability of winning by self-draw, the probability of winning by another player discarding a winning tile, and the winning probability corresponding to different number of points (such as 1 point, 3 points, yakuman, a total of 5 sub-indicators), distinguishing the winning method and value;

[0093] Expected score expansion: Calculate "expected score for self-draw", "loss score for discarding and being won (negative expected score)", "expected score for different stages (such as early / late stage)", and "expected score for making a big hand (high score)" (such as 10 sub-indicators), covering both gains and risks.

[0094] The expanded sub-indicators have large differences in their numerical ranges and need to be normalized to the [0,1] or [-1,1] range: Probability-based sub-indicators (probability of winning / drawing): directly retain the original values ​​(already between 0 and 1); Score-based sub-indicators (expected score): use "historical maximum / minimum value normalization", the formula is as follows: The maximum / minimum scores in the offline training data are used; negative indicators (such as loss from discarding a winning tile) are processed into positive features through "sign reversal + normalization" (e.g., loss of -500 points → converted to 0.2, loss of -1000 points → converted to 0.1, the smaller the value, the higher the risk). All normalized sub-indicators are concatenated in a fixed order of "related to waiting for a winning hand → related to winning the hand → related to expected score" to form a 100-dimensional "expected value feature vector".

[0095] The game prediction module includes:

[0096] Stage Feature Integration Unit: Used to generate feature vectors for each stage based on the staged feature subsets, and then arrange them in the order of stage occurrence to obtain a stage feature sequence;

[0097] Temporal modeling unit: used to process stage feature sequences and capture the influence of the previous stage on the next stage;

[0098] Ranking distribution unit: used to map the temporal features processed by the temporal modeling unit into the final ranking probability distribution through a fully connected layer.

[0099] Specifically, addressing the "temporal" and "incomplete" nature of Mahjong games, a "dynamic phased approach + GRU network" is used to predict the final ranking. Phase transitions are triggered by "key events" (such as a player revealing a concealed hand, reaching a winning hand, or discarding a dangerous tile), dividing the entire game into discrete phases to avoid the complexity of a "global single model." The feature encoding module receives a subset of phased features and generates a feature vector for each phase (including phase number, number of tiles played, each player's winning hand status, and score changes). For example, the system scans in real-time whether a key event has been triggered in the current game. Starting with the initial state of the game as phase 1, each trigger of a key event automatically increments the phase number by 1, recording the number of key events triggered in the current game to determine the current phase. The system performs numerical encoding on each player's hand; it reads all the current hand data, counts the cumulative number of all played cards, and uses this cumulative number as a feature value of the number of played cards; it monitors the status of each player in real time, classifies the status into 3 categories and encodes them in binary / multi-base, arranges the encoding results of each player in order, and forms the feature items of each player's hand status; it reads the real-time scores of each player in the current stage, calculates the score difference of each player, arranges the score differences of all players in order, and forms the score change feature items; it integrates the current stage number, the feature value of the number of played cards, the feature items of each player's hand status, and the score change feature items into a structured feature vector in a fixed order to obtain a staged feature subset.

[0100] Furthermore, the reward calculation module is specifically used for:

[0101] Calculate the expected ranking for the current stage based on the ranking probability distribution and the ranking point vector:

[0102] ;

[0103] By comparing the expected ranking in the current stage with the expected ranking in the previous stage, the reward value is obtained:

[0104] ;

[0105] in, For the ranking point vector, For ranking probability, if Give a positive reward; if Negative rewards will be given.

[0106] This system breaks away from the traditional reward logic based on changes in ranking, instead calculating quantitative rewards for intermediate decisions based on expected ranking changes. The phased reward signals provide value feedback during the training of the decision network, optimizing its prediction accuracy of action value. The decision network employs a hybrid offline-online training strategy. In the offline phase, high-quality historical game data is used, and the CQL algorithm is applied to prevent Q-value overestimation. In the online phase, continuous optimization and dynamic strategy adjustments are achieved through self-play. During online self-play, the system records a four-tuple of "state-action-reward-next state," which is stored in an "experience replay pool" to avoid temporal correlation of training samples. During offline training, a batch of samples (e.g., 32 / 64 four-tuples) is randomly sampled from the experience replay pool to update the decision network parameters. The core of the decision network is "predicting the Q-value of each action (i.e., the long-term value of the action)," and the reward value is the key input for calculating the "target Q-value," specifically implemented using the temporal difference (TD) algorithm in conjunction with the CQL algorithm to prevent Q-value overestimation.

[0107] Furthermore, such as Figures 2 to 4 As shown, the decision network module is specifically used for:

[0108] The high-dimensional feature vector from the feature encoding module is received, and 1024-dimensional deep semantic features are extracted through a residual network and a channel attention mechanism.

[0109] Input 1024-dimensional deep semantic features into DDQN to separate state value (V) and action advantage (A). State value V is the overall value of the current hand state, and action advantage A is the advantage of each action relative to other actions.

[0110] Through formula Calculate the Q value for each action (the mean of the action advantage) to avoid action advantage bias;

[0111] An action mask is applied to the Q value, and then the action with the highest Q value is selected through an ε-greedy strategy to determine the final decision action, such as eating, ponging, konging, or winning.

[0112] This invention acquires current game information and converts the features included in the current game information into high-dimensional feature vectors and staged feature subsets; receives the staged feature subsets and determines the final ranking probability distribution based on the staged feature subsets; obtains the corresponding staged reward signal based on the final ranking probability distribution; receives the high-dimensional feature vectors and staged reward signals, and determines the final decision action based on the high-dimensional feature vectors and staged reward signals; in addition to basic hand, river, and secondary revealed information, it innovatively introduces expected value features, temporal decay features, and special rule features. The feature encoding adopts a hierarchical design, using differentiated encoding strategies for different types of features: the river information introduces temporal decay weights, reflecting the importance of recently discarded cards; for some special rules of Mahjong, an extension interface is retained to introduce dedicated feature dimensions, ensuring that the rule characteristics are fully expressed, thus solving the problem of low decision quality in existing decision-making methods.

[0113] In addition, the embodiments of the present invention have at least the following beneficial effects:

[0114] 1. The expected value calculation module uses a recursive algorithm to calculate three core indicators: the probability of drawing a winning hand, the probability of winning, and the expected score, providing a quantitative assessment of future value for decision-making. The system constructs a table of effective hand acquisition probabilities and an ineffective hand probability table, and calculates the expected returns of different paths using dynamic programming. This forward-looking calculation allows AI to assess the long-term impact of current decisions, rather than focusing solely on immediate gains.

[0115] 2. The GRP module divides the game into dynamic phases, each recording key information such as the number of cards played, the state of a winning hand, and score changes. It processes sequence features through a GRU network to predict the final ranking distribution and calculates rewards based on expected ranking changes. This design overcomes the limitations of traditional head-to-head counting methods, accurately assessing defensive value and strategic adjustments.

[0116] 3. The system uses the Monte Carlo method to process the full game reward and combines it with the conservative Q-learning algorithm for offline training, which effectively solves the core technical problems of reward sparsity and incomplete information in the Mahjong game.

[0117] Second Embodiment

[0118] This invention also proposes a decision-making method guided by phased rewards, implemented using the phased reward-oriented decision-making system in the first embodiment. The method includes:

[0119] Obtain the current hand information and convert the features included in the current hand information into high-dimensional feature vectors and staged feature subsets;

[0120] Receive a phased feature subset and determine the final ranking probability distribution based on the phased feature subset;

[0121] Obtain the corresponding staged reward signal based on the final ranking probability distribution;

[0122] It receives high-dimensional feature vectors and staged reward signals, and determines the final decision action based on the high-dimensional feature vectors and staged reward signals.

[0123] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequential list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0125] More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable storage media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0126] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0127] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0128] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A decision-making system guided by phased rewards, characterized in that, The system includes: A memory, a processor, and a computer module stored in the memory, the computer module being used to provide a computer program that runs on the processor; The computer module includes: The feature encoding module is used to acquire the current hand information and convert the features included in the current hand information into high-dimensional feature vectors and staged feature subsets. The game prediction module is used to receive a phased feature subset and determine the final ranking probability distribution based on the phased feature subset; The reward calculation module is used to obtain the corresponding staged reward signal based on the final ranking probability distribution. The decision network module is used to receive high-dimensional feature vectors and staged reward signals, and determine the final decision action based on the high-dimensional feature vectors and staged reward signals. Among them, the high-dimensional feature vector includes at least basic features, river features, secondary information, expected value features and special rule features. The basic features include at least hand cards, score and ranking. The river features include at least the features of one's own river cards and other players' river cards as well as the time decay weight. The secondary information includes at least the secondary card groups of all players. The expected value features include at least the analysis of the expected value of the cards and the calculation of the expected value of the effective cards. The special rule features include at least the special rule state. The expected value includes the probability of drawing a winning hand, the probability of winning, and the expected score; The probability of reaching a ready hand (or "ready hand") represents the likelihood of achieving that state in future rounds. The probability of winning a hand represents the likelihood of successfully winning in future rounds: Expected score represents the score expected in future rounds: ； in, Indicates the current round. Represents all possible single cards in the deck. Indicates that the player touched " in the current round "The probability of this card..." Indicates touching " in the current round "After that, the next round" The probability of reaching a ready hand state; This indicates the probability of drawing a winning tile in the current round. This indicates the probability of drawing a non-winning tile in the current round. This means that if you draw a non-winning tile in the current round, the next round... The probability of successfully winning a hand; For the current round, touch " The probability of " If you touch it "The score you get for successfully winning the hand is..." For indicator functions, if "is touched" "If you can win the hand later..." ,otherwise , For the current round, touch " "If you don't win the hand, the next round..." Expected score; The game prediction module includes: Stage Feature Integration Unit: Used to generate feature vectors for each stage based on the staged feature subsets, and then arrange them in the order of stage occurrence to obtain a stage feature sequence; Temporal modeling unit: used to process stage feature sequences and capture the influence of the previous stage on the next stage; Ranking distribution unit: used to map the temporal features processed by the temporal modeling unit into the final ranking probability distribution through a fully connected layer; The reward calculation module is specifically used for: Calculate the expected ranking for the current stage based on the ranking probability distribution and the ranking point vector: ； By comparing the expected ranking in the current stage with the expected ranking in the previous stage, the reward value is obtained: ； in, For the ranking point vector, For ranking probability, if Give a positive reward; if Give negative rewards; The decision network module is specifically used for: The high-dimensional feature vector from the feature encoding module is received, and 1024-dimensional deep semantic features are extracted through a residual network and a channel attention mechanism. Input 1024-dimensional deep semantic features into DDQN and separate state value V and action advantage A. State value V is the overall value of the current hand state, and action advantage A is the advantage of each action relative to other actions. Through formula To calculate the Q value for each action as the mean of action advantage, we avoid action advantage bias. Apply an action mask to the Q-value, and then use an ε-greedy strategy to select the action with the highest Q-value to determine the final decision action; Among them, the phased reward signal provides value feedback during the training process of the decision network, which is used to optimize the network's prediction accuracy of action value.

2. The decision-making system guided by phased rewards according to claim 1, characterized in that, The process of converting the features included in the current game information into a high-dimensional feature vector and a staged feature subset includes: Real-time scanning to see if key events are triggered in the current game. The initial state of the entire game is taken as stage 1. Each time a key event is triggered, the stage number is automatically incremented by 1. The number of key events triggered in the current game is recorded, the current stage number is determined, and numerical encoding is performed. Read all the card river data of the current game, count the cumulative number of all cards played in the card river, and use the cumulative number as the feature value of the number of cards played; The status of each player is monitored in real time, and the status is divided into 3 categories and encoded in binary / multi-base. The encoding results of each player are arranged in order of player to form the status feature items of each player and their hand. Read the real-time scores of each player in the current stage, calculate the score difference for each player, and arrange the score differences of all players in order to form a score change feature item; The current stage number, the feature value of the number of cards played, the feature items of each player's winning hand status, and the feature items of score changes are integrated into a structured feature vector in a fixed order to obtain a staged feature subset.

3. The decision-making system guided by phased rewards according to claim 1, characterized in that, The system also includes: The training module is used to train the decision network module using a hybrid offline-online training mode. In the offline stage, high-quality historical game data is used, and the CQL algorithm is applied to prevent Q-value overestimation. In the online stage, the strategy is continuously optimized and dynamically adjusted through self-play.

4. A decision-making method guided by phased rewards, characterized in that, The method, implemented using any one of claims 1 to 3, is a staged reward-oriented decision-making system, comprising: Obtain the current hand information and convert the features included in the current hand information into high-dimensional feature vectors and staged feature subsets; Receive a phased feature subset and determine the final ranking probability distribution based on the phased feature subset; Obtain the corresponding staged reward signal based on the final ranking probability distribution; It receives high-dimensional feature vectors and staged reward signals, and determines the final decision action based on the high-dimensional feature vectors and staged reward signals.

Images