A chessboard reinforcement learning method and system based on Monte Carlo sampling and a medium

Abstract

The application provides a chess and card reinforcement learning method and system based on Monte Carlo sampling and a medium, comprising: obtaining a current state of a player; determining a plurality of predicted actions of the player in the current state; for each of the plurality of predicted actions: inputting the current state and the predicted action into a chess and card model, wherein the chess and card model is trained based on training samples obtained through Monte Carlo sampling; and taking the predicted action corresponding to the maximum first score as the current action of the player, so that the output of the model can be changed into one dimension, and the result converges.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, in particular to a chess reinforcement learning method based on Monte Carlo sampling, a system and a medium. BACKGROUND

[0002] For the current popular deep reinforcement learning methods, such as ppo, ac and dqn, etc., the probability of each action of the player or the value of each action is obtained by fitting the game strategy through a neural network. However, in a chess game, there are many actions that the player can perform, such as four-person mahjong, which involves various card types, each card type has multiple types, and different card types have many different combinations. Therefore, it is difficult to calculate the value of each action of the player through the existing neural network.

[0003] Therefore, the present application provides a chess reinforcement learning method based on Monte Carlo sampling, a system and a medium, so that the player can more accurately obtain the value of each operable action in the case of multiple operable actions. SUMMARY

[0004] The purpose of the present application is to provide a chess reinforcement learning method, which comprises: obtaining the current state of the player; determining a plurality of predicted actions of the player in the current state; for each of the plurality of predicted actions: inputting the current state and the predicted action into a chess model, and the model outputs a first score corresponding to the predicted action; wherein the chess model is trained by training samples obtained based on Monte Carlo sampling; and taking the predicted action corresponding to the maximum first score as the current action of the player.

[0005] Further, the chess model is obtained by: obtaining sample game information; based on the win or loss of the player in the sample game information, labeling the score of the sample game information; inputting the labeled sample game information into an initial chess model for training to obtain a trained chess model.

[0006] Further, the obtaining of the sample game information comprises: obtaining a plurality of sample current states of the player in each game; for each of the plurality of sample current states: based on the labeled score, respectively assigning values to a plurality of sample predicted actions and a sample current action corresponding to the sample current state; and taking the plurality of sample current states in each game, the value of the sample predicted action corresponding to each sample current state and the value of the sample current action as the sample game information.

[0007] Further, the obtained trained chess model comprises: inputting the sample game information into the initial chess model, constructing a loss function based on the output of the initial chess model and a label; wherein the loss function is:

[0008]

[0009] wherein N is the total number of the sample current states in each game; y i is the output of the initial chess model, y' i is a label value; iteratively updating the parameters of the initial chess model based on the loss function to obtain the chess model.

[0010] Further, the current state at least comprises the cards currently owned by the player, the cards played by other players, the cards of the pung and / or the cards that the player can currently win; the obtaining of the current state of the player comprises: inputting the current game information into a neural network model, and the model outputs a first feature vector of the cards currently owned by the player, the cards played by other players, the cards of the pung and / or the cards that the player can currently win.

[0011] Further, the current state further comprises the card types left by the player and the card types left by other players; the inputting of the current state and the predicted action into the chess model, and the model outputting a first score corresponding to the predicted action comprises: fully connecting the first feature vector, the card types left by the player and the card types left by other players to obtain a second feature vector; inputting the second feature vector and the predicted action into the chess model, and the model outputting a first score of the predicted action.

[0012] Further, the neural network model at least comprises two 9*3*4 convolution layers and one 9*4*1 convolution layer; wherein the 9*3*4 convolution layers are respectively used for extracting the features of the cards currently owned by the player and the features of the cards played by other players; and the 9*4*1 convolution layer is used for extracting the features of the pung and winning cards.

[0013] The purpose of the present application is to provide a chess reinforcement learning system based on Monte Carlo sampling, comprising: a current state acquisition module for acquiring the current state of the player; a predicted action determination module for determining a plurality of predicted actions of the player in the current state; a first score determination module for each of the plurality of predicted actions: inputting the current state and the predicted action into a chess model, and the model outputting a first score corresponding to the predicted action; a current action determination module for taking the predicted action corresponding to the maximum first score as the current action of the player; a model training module for obtaining a chess model; wherein the chess model is trained based on the training samples obtained based on Monte Carlo sampling; and a sample acquisition module for acquiring sample game information.

[0014] Further, the chess model is obtained by: obtaining a plurality of sample game information; labeling scores of the sample game information based on wins and losses of players in the sample game information; inputting the labeled sample game information into an initial chess model to train the initial chess model and obtain a trained chess model.

[0015] The purpose of the present application is to provide a computer-readable storage medium, which stores computer instructions, and when the computer instructions are executed by a processor, the corresponding operations of the chess-based reinforcement learning method described in any of the above are implemented.

[0016] The technical scheme of the embodiment of the present application has at least the following advantages and beneficial effects:

[0017] Some embodiments in the specification input the predicted action and the current state into the model together to obtain a first score to evaluate the pros and cons of the predicted action; so that the output of the model can be changed to 1 dimension, facilitating the convergence of the results. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 An exemplary flowchart of a chess reinforcement learning method based on Monte Carlo sampling is provided for some embodiments of the present application;

[0019] Figure 2 An exemplary schematic diagram for obtaining a chess model is provided for some embodiments of the present application;

[0020] Figure 3 An exemplary flowchart for obtaining sample game information is provided for some embodiments of the present application;

[0021] Figure 4 An exemplary module diagram of a chess reinforcement learning system based on Monte Carlo sampling is provided for some embodiments of the present application. DETAILED DESCRIPTION

[0022] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0023] Figure 1 An exemplary flowchart of a chess reinforcement learning method based on Monte Carlo sampling is provided for some embodiments of the present application. In some embodiments, flow 100 can be executed by system 400. As shown in Figure 1 flow 100 can include the following steps:

[0024] At step 110, the current state of the player is obtained. In some embodiments, step 110 can be performed by the current state obtaining module 410.

[0025] The current state can be the state when the player is playing a card. Taking playing mahjong as an example, the current state can be the state when it is the player’s turn to play a card. The current state can also be the state when the player is playing a card of a type that can be played.

[0026] In some embodiments, the current state at least includes one or more of the following: the cards currently held by the player, the cards played by other players, the cards that can be played, and the cards that can be played by the player. The current state of the player can be obtained in various ways, for example, by image recognition technology.

[0027] In some embodiments, the current game information of the player can be input into a neural network model, and the model outputs a first feature vector of the cards currently held by the player, the cards played by other players, the cards that can be played, and / or the cards that can be played by the player. The current game information can refer to data carrying game information when the player is playing a card. Taking playing mahjong as an example, the current game information can include a photo of the mahjong table, a photo of the cards held by the player, and game data. The current game information can be obtained in various ways, for example, for offline, it can be obtained by a camera; for online, it can be directly extracted from the game data. For example, different convolution layers are used to process the current game information to obtain the first feature vector.

[0028] In some embodiments, since mahjong is divided into three types of cards, each type of card corresponds to nine types of numbers, and each type of number corresponds to four cards, a 9*3*4 convolution layer can be used to extract the features of the cards currently held by the player; similarly, a 9*3*4 convolution layer can be used to extract the features of the cards played by other players; since the cards that can be played by the player and the cards that can be played by other players are all of the same type, a 9*4*1 convolution layer can be used to extract the features of the cards that can be played. By concatenating the features of the cards currently held by the player, the features of the cards played by other players, and the features of the cards that can be played, the first feature vector can be obtained. The features of the cards currently held by the player, the features of the cards played by other players, and the features of the cards that can be played at least include the number of cards and the type of cards.

[0029] In some embodiments, the current state can further include one or more of a player's remaining tile pattern, other players' remaining tile patterns, and the like. The remaining tile pattern is a tile pattern that a player needs to keep. Taking playing mahjong as an example, each player has at most two tile patterns of two colors, and the color that is not played is the remaining tile pattern of the player. In some embodiments, the player's remaining tile pattern can be determined by identifying the tiles that the player has. For example, according to the number of colors of the tiles that the player has, the color with the largest number and the second largest number are determined as the tile patterns of the player. In some embodiments, the remaining tile patterns of other players can be determined by identifying the tiles that are not played by other players. For example, the color that is not played by other players is determined as the remaining tile pattern of other players.

[0030] At step 120, a plurality of predicted actions of the player in the current state is determined. In some embodiments, step 120 can be performed by the predicted action determination module 420.

[0031] The predicted action can refer to an action that can be taken by the player in the current game state. The system can determine a plurality of predicted actions based on the tiles that the player currently has and the game rules. Taking playing mahjong as an example, the tiles that the player currently has include one tile, two tiles, and three tiles, and the predicted actions can include the actions of playing one tile, playing two tiles, playing three tiles, or not playing a tile.

[0032] At step 130, for each of the plurality of predicted actions: the current state and the predicted action are input into the board game model, and the model outputs a first score corresponding to the predicted action. In some embodiments, step 130 can be performed by the first score determination module 430.

[0033] The first score can be used to represent the value of the player making a certain predicted action in the current state. The higher the value, the higher the probability that the player wins the game by making the current action, and vice versa.

[0034] The board game model can be obtained by training an initial board game model. For more information about obtaining the board game model, see Figure 2 and related descriptions thereof.

[0035] In some embodiments, the first score determination module 430 can fully connect the first feature vector, the player's remaining tile pattern, and the other players' remaining tile patterns to obtain a second feature vector. The second feature vector and the predicted action are input into the board game model, and the model outputs the first score of the predicted action. In some embodiments, the board game model can be trained based on training samples obtained by Monte Carlo sampling. For example, the board game model can count the total number of times that the player makes the current action and the number of times that the player wins in the current state in the training samples to determine the first score. For more information about the training samples, see Figure 3 and related descriptions thereof.

[0036] Step 140: The predicted action corresponding to the highest first score is taken as the player's current action. In some embodiments, step 140 can be performed by the current action determination module 440.

[0037] The current action is the action the player takes in their current state. After the player takes the current action, their current state changes.

[0038] Figure 2 This is an exemplary schematic diagram illustrating the acquisition of a chess and card game model according to some embodiments of the present invention. In some embodiments, process 200 may be executed by model training module 450. Figure 2 As shown in the diagram, process 200 includes the following:

[0039] Acquire multiple sample game information. Sample game information can refer to training samples used to train the card game model. Sample game information can include information about a player's game in a specific match. For example, a sample game information set may include one or more of the following: multiple current states of the player, the predicted action for each current state, the current action for each current state, and the player's win or loss in that game. In some embodiments, sample game information is obtained based on Monte Carlo sampling. For example, game information containing the player's current state is extracted from the collected game information as sample game information. For more information on acquiring sample game information, see [link to relevant documentation]. Figure 3 And its related descriptions.

[0040] Based on the players' wins and losses in the sample game information, the score of the sample game information is labeled. For example, the score of a player's loss in a sample game information can be determined as -1, and the score of a player's win in a sample game information can be determined as 1. Taking mahjong as an example, the score of a player's winning hand can be determined as 1; the score of not giving another player a winning hand can be determined as 0; and the score of giving another player a winning hand can be determined as -1.

[0041] The labeled sample game information is input into the initial chess and card game model for training, resulting in a trained chess and card game model.

[0042] In some embodiments, the model training module 450 can input sample game information into an initial card game model and construct a loss function based on the output and labels of the initial card game model; wherein the loss function is:

[0043]

[0044] Where N is the total number of current states of the samples in each game; y i y′ is the output of the initial chess and card model. i This is the tag value.

[0045] The parameters of the initial chess model are iteratively updated based on the loss function, and a chess model is obtained. When the loss function meets a preset condition, the model training is completed, and a trained chess model is obtained. The preset condition can include loss function convergence, iteration number reaching a threshold, etc. The training method includes but is not limited to gradient descent method, etc.

[0046] Figure 3 An example flowchart for obtaining sample game information is provided for some embodiments of the application. In some embodiments, flowchart 300 can be performed by sample obtaining module 460. As shown in Figure 3 Flowchart 300 includes the following steps:

[0047] Step 310: Obtain a plurality of sample current states of the player in each game. The sample current state can be the current state in each game. For example, the sample obtaining module can obtain a plurality of sample current states of the player in the order of the player's card playing.

[0048] In some embodiments, the information of a game can be obtained by repeatedly performing steps 110-140 until the game ends. Of course, the information of a game can also be obtained by other feasible ways.

[0049] Step 320: For each of the plurality of sample current states: based on the label score, respectively assign values to the plurality of sample predicted actions and the sample current action corresponding to the sample current state. The sample predicted action can be the predicted action in each game. The sample current action can be the current action in each game. For example, when the player wins this game, the sample current action corresponding to each sample current state can be assigned a value of 1, and the other sample predicted actions can be assigned a value of 0; when the player loses this game, the sample current action corresponding to each sample current state can be assigned a value of -1, and the other sample predicted actions can be assigned a value of 0.

[0050] Step 330: Take the plurality of sample current states in each game, the value of the sample predicted action corresponding to each sample current state, and the value of the sample current action as sample game information. For example, the sample obtaining module can extract the sample current state, the value of the sample predicted action corresponding to the sample current state, and the value of the sample current action as a sample feature vector, then determine a sample feature sequence of the current game from the sample feature vectors of the plurality of current states, and take the sample feature sequence as the sample game information.

[0051] In some embodiments, the model training module can further input the sample feature vector in the sample game information and the sample current action corresponding to the sample feature vector into an initial chess model, and the initial chess model outputs a first score of the sample current action, construct a loss function based on the output of the initial chess model and the value of the sample current action in the sample feature vector, and update the parameters of the initial chess model based on the loss function to obtain the chess model. For more information about the loss function, see Figure 2 . Similarly, the input sample current action can also be replaced by a sample predicted action.

[0052] Figure 4 An exemplary module diagram of a chess reinforcement learning system based on Monte Carlo sampling is provided for some embodiments of the present application. As shown in Figure 4 , the system 400 can include a current state obtaining module 410, a predicted action determining module 420, a first score determining module 430, a current action determining module 440, a model training module 450, and a sample obtaining module 460.

[0053] The current state obtaining module 410 is configured to obtain the current state of the player. For more information about the current state module 410, see Figure 1 .

[0054] The predicted action determining module 420 is configured to determine a plurality of predicted actions of the player in the current state. For more information about the predicted action determining module 420, see Figure 1 .

[0055] The first score determining module 430 is configured to, for each of the plurality of predicted actions: input the current state and the predicted action into a chess model, and the model outputs a first score corresponding to the predicted action. For more information about the first score determining module 430, see Figure 1 .

[0056] The current action determining module 440 is configured to determine the predicted action corresponding to the maximum first score as the current action of the player. For more information about the current action determining module 440, see Figure 1 .

[0057] The model training module 450 is configured to obtain a chess model, wherein the chess model is trained based on training samples obtained by Monte Carlo sampling. For more information about the model training module 450, see Figure 2 .

[0058] The sample obtaining module 460 is configured to obtain sample game information. For more information about the sample obtaining module 460, see Figure 3 .

[0059] The computer readable storage medium stores computer instructions, and the computer instructions are executed by a processor to realize the operation corresponding to the chess-based reinforcement learning method.

[0060] The above merely describes the preferred embodiments of the present application, but is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for board game reinforcement learning based on Monte Carlo sampling, characterized in that, The method comprises the following steps: obtaining a current state of a player; determining a plurality of predicted actions of the player in the current state; for each of the plurality of predicted actions: inputting the current state and the predicted action into a chess model, and the model outputs a first score corresponding to the predicted action; wherein the chess model is trained based on training samples obtained by Monte Carlo sampling; taking the predicted action corresponding to the maximum first score as the current action of the player; the chess model is obtained by the following method: obtaining a plurality of sample game information; based on the win or loss of the player in the sample game information, labeling the score of the sample game information; the score of the sample game information in which the player loses is determined as -1, and the score of the sample game information in which the player wins is determined as 1; inputting the labeled sample game information into an initial chess model to obtain a trained chess model; obtaining sample game information, comprising: obtaining a plurality of sample current states of the player in each game; for each of the plurality of sample current states: based on the labeled score, respectively assigning values to a plurality of sample predicted actions and sample current actions corresponding to the sample current state; the sample predicted action is a predicted action in each game; the sample current action is a current action in each game; when the player wins in this game, the sample current action corresponding to each sample current state is assigned a value of 1, and the other sample predicted actions are assigned a value of 0; the plurality of sample current states in each game, the values of the sample predicted actions corresponding to each sample current state, and the values of the sample current actions are taken as the sample game information.

2. The Monte Carlo sampling based backgammon reinforcement learning method according to claim 1, wherein, the trained chess model comprises: inputting the sample game information into the initial chess model, and constructing a loss function based on the output of the initial chess model and the label; wherein the loss function is: where N is the total number of the sample current states in each game; y i is the output of the initial chessboard model, y' i is the label value; iteratively updating the parameters of the initial chess model based on the loss function to obtain the chess model.

3. The Monte Carlo sampling based backgammon reinforcement learning method according to claim 1, wherein, the current state at least includes the cards currently owned by the player, the cards played by other players, the cards that are hit or blocked, and / or the cards that the player can currently win; the current state of the player comprises: inputting the current game information into a neural network model, and the model outputs a first feature vector of the cards currently owned by the player, the cards played by other players, the cards that are hit or blocked, and / or the cards that the player can currently win.

4. The chess reinforcement learning method based on Monte Carlo sampling according to claim 3, wherein the current state further includes the card types left by the player and the card types left by other players; inputting the current state and the predicted action into the chess model, and the model outputs a first score corresponding to the predicted action, comprising: fully connecting the first feature vector, the card types left by the player, and the card types left by other players to obtain a second feature vector; inputting the second feature vector and the predicted action into the chess model, and the model outputs a first score of the predicted action.

5. The Monte Carlo sampling based backgammon reinforcement learning method according to claim 3, wherein, The neural network model at least comprises two convolutional layers of 9*3*4 and a convolutional layer of 9*4*1; wherein the convolutional layers of 9*3*4 are respectively used for extracting features of the cards currently held by the player and features of the cards played by other players; and the convolutional layer of 9*4*1 is used for extracting features of the card of pung, gang and hu.

6. A Monte Carlo sampling based backgammon reinforcement learning system using the Monte Carlo sampling based backgammon reinforcement learning method according to any one of claims 1-5. Comprise, a current state acquisition module, configured to acquire a current state of a player; a predicted action determination module, configured to determine a plurality of predicted actions of the player in the current state; a first score determination module, configured to, for each of the plurality of predicted actions: input the current state and the predicted action into a chess and card model, and output a first score corresponding to the predicted action by the model; a current action determination module, configured to determine a predicted action corresponding to the maximum first score as a current action of the player; a model training module, configured to acquire the chess and card model; wherein the chess and card model is obtained by training based on training samples obtained by Monte Carlo sampling; a sample acquisition module, configured to acquire sample game information; the chess and card model is obtained in the following manner: acquire a plurality of sample game information; based on the win or loss of the player in the sample game information, label the score of the sample game information; determine the score of the sample game information in which the player loses as -1, and determine the score of the sample game information in which the player wins as 1; input the labeled sample game information into an initial chess and card model to obtain a trained chess and card model.

7. A computer-readable storage medium, characterized in that, The storage medium stores computer instructions, and the computer instructions are executed by the processor to implement the operations corresponding to the chess and card reinforcement learning method based on Monte Carlo sampling in any one of claims 1 to 5.

Images