Game model construction method and device, storage medium and electronic equipment

Abstract

The invention relates to a method and a device for constructing a game model, a storage medium and electronic equipment. The method comprises the following steps: acquiring target game information for an incomplete information target game from an expert system; preprocessing the target game information to obtain a game data set; segmenting the game data into a plurality of game data subsets according to a segmentation strategy, and obtaining a generative pre-training first network model based on the maximum game data subset; retraining the two generative pre-training first network models with different hyper-parameters based on the second large game data subset, and obtaining a generative pre-training second network model from the two retrained generative pre-training first network models; and retraining the two generative pre-training second network models with different hyper-parameters in sequence based on other game data subsets according to the proportion from large to small, and obtaining a game model from the two finally trained generative pre-training network models. And the model precision can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of decision intelligence research, and in particular to a method, device, storage medium and electronic device for constructing a game model. Background Art

[0002] Board games have the characteristics of varied combinations and require players to cooperate. From the perspective of game theory, they are a kind of incomplete information game problem. Incomplete information game means that the participants are not fully aware of the state of the game, there is hidden information and uncertainty, and the strategies, profit functions and other information of other players cannot be fully and accurately grasped.

[0003] Deep reinforcement learning algorithms can be used to solve incomplete information game problems. Among them, the Generative Pretrained Transformer (GPT) network model is the largest language model with the strongest feature representation and generalization reasoning capabilities. Its core mechanism is the Transformer architecture and reinforcement learning based on human feedback. The Transformer architecture is a complex neural network architecture, and reinforcement learning based on human feedback is to improve the performance of the neural network model by evaluating the neural network model generated based on the neural network architecture.

[0004] In the related technology, a method of using deep reinforcement learning method to improve the policy performance of neural network model by evaluating the value of state and the Q value of state-action pair exists errors in the evaluation of the value of state and the Q value of state-action pair. At the same time, the deep reinforcement learning method uses ∈-greedy algorithm for action exploration and starts the exploration from random strategy. Random exploration will lead to insufficient and unreasonable exploration of action space, resulting in low accuracy and low performance of the generated neural network model. Summary of the invention

[0005] In view of this, the present invention provides a method, device, storage medium and electronic device for constructing a game model.

[0006] Specifically, the present invention is achieved through the following technical solutions:

[0007] According to a first aspect of the present invention, a method for constructing a game model is provided. The method for constructing a game model comprises:

[0008] Obtaining target game information for playing the target game with incomplete information from a pre-set expert system;

[0009] Preprocessing the target game information to obtain a game data set;

[0010] Divide the game data set into multiple game data subsets according to a preset segmentation strategy, and obtain a generative pre-trained first network model based on the game data subset with the largest segmentation ratio;

[0011] Based on the subset of game data with the second largest split ratio, two generative pre-trained first network models with different hyperparameters are retrained, and a generative pre-trained second network model is obtained from the two retrained generative pre-trained first network models;

[0012] According to the order of the split ratio from large to small, based on other subsets of the game data, the two generative pre-trained second network models with different hyperparameters are retrained in turn, and the game model is obtained from the two generative pre-trained network models that have been trained last.

[0013] Optionally, the preprocessing of the target game information to obtain a game data set includes:

[0014] Deleting incomplete information in the target game information, and, according to a preset filtering strategy for the target game with incomplete information, filtering out information that meets the filtering strategy to obtain an initial game data set;

[0015] Extracting the initial game data subsets of each round from the initial game data set according to the time sequence of the game, arranging the initial game data subsets according to the data arrangement strategy set for the incomplete information target game, and obtaining a round data sequence;

[0016] For each round of the round data sequence, the data corresponding to the participants who did not participate in the round are set to 0 to obtain the intermediate data sequence;

[0017] Based on the game object attributes and the number of game objects included in the data arrangement strategy, matrix expansion is performed on each data in the intermediate data sequence to obtain the game data set.

[0018] Optionally, the obtaining of a generative pre-trained first network model based on the game data subset with the largest segmentation ratio includes:

[0019] Extracting the game data subset with the largest split ratio from multiple game data subsets to obtain the game data subset with the largest ratio;

[0020] Randomly setting model parameters of two generative pre-trained initial network models, setting hyperparameters according to a preset strategy, and inputting the maximum proportion game data subsets into the generative pre-trained initial network models respectively;

[0021] The input sequence is obtained by positionally encoding each element in the maximum proportion of the game data subset input into the generative pre-trained initial network model using triangular position encoding, and then simultaneously input into the Q network, K network and V network through a single multi-head self-attention mechanism block of the generative pre-trained initial network model;

[0022] Calculate the attention weight of each element based on the Query value, Key value, and Value value of each element in the input sequence;

[0023] Input the calculated attention weights of each element into the first dropout layer, the first fully connected network layer, and the second dropout layer of the generative pre-trained initial network model in sequence to obtain an element processing sequence, and perform a residual connection operation with the input sequence to obtain a first residual connection sequence;

[0024] Perform layer normalization on the residual connection sequence to obtain a normalized sequence;

[0025] Inputting the normalized sequence into the second fully connected network layer of the generative pre-trained initial network model, performing a residual connection operation on the output of the second fully connected network layer and the normalized sequence to obtain a second residual connection sequence;

[0026] Input the second residual connection sequence into the third dropout layer of the generative pre-trained initial network model to obtain a predicted sequence;

[0027] According to the target prediction decision information in the prediction sequence and the corresponding target actual decision information in the input sequence, gradient backpropagation is performed on the generative pre-trained initial network model to update the model parameters of the generative pre-trained initial network model;

[0028] From the two generative pre-trained initial network models that perform gradient back-propagation update, the generative pre-trained initial network model with higher decision-making ability is selected as the generative pre-trained first network model according to the test environment.

[0029] Optionally, the GPT network hyperparameters include: batch size, learning rate, dropout rate, number of attention heads, and number of multi-head self-attention blocks.

[0030] Optionally, the two generative pre-trained initial network models have different hyperparameters.

[0031] Optionally, the target game information includes: the cooperative relationship between the participants in the incomplete information target game, the total game resources of the participants, and the game resources of the participants in each round of the incomplete information target game.

[0032] Optionally, the method further comprises:

[0033] Obtain the cooperative relationship of each participant in the current round of the incomplete information target game, the total game resources of the participants, and the game resources of each participant except the participant itself, input the game model, obtain the game resources of the participant itself in the current round, and play the game based on the obtained game resources.

[0034] The method for constructing a game model in the present technical solution is to obtain the target game information for the incomplete information target game from the expert system; pre-process the target game information to obtain a game data set; divide it into multiple game data subsets according to the segmentation strategy, and obtain the generative pre-trained first network model based on the largest game data subset; retrain the two generative pre-trained first network models with different hyperparameters based on the second largest game data subset, and obtain the generative pre-trained second network model from the two retrained generative pre-trained first network models; retrain the two generative pre-trained second network models with different hyperparameters based on other game data subsets in order from large to small proportions, and obtain the game model from the two generative pre-trained network models that have been trained last. In this way, model construction based on professional game feedback behavior can improve the accuracy of the constructed model. Furthermore, by using a tiered model training method, the time required for model training can be effectively reduced and the training performance of the model can be improved.

[0035] According to a second aspect of the present invention, there is provided a device for constructing a game model, the device for constructing a game model comprising:

[0036] A game information acquisition module, used to acquire target game information for playing a target game with incomplete information from a pre-set expert system;

[0037] An information preprocessing module, used to preprocess the target game information to obtain a game data set;

[0038] An initial training module, used to segment the game data set into multiple game data subsets according to a preset segmentation strategy, and obtain a generative pre-trained first network model based on the game data subset with the largest segmentation ratio;

[0039] A retraining module is used to retrain two generative pre-trained first network models with different hyperparameters based on a subset of game data with a second largest segmentation ratio, and obtain a generative pre-trained second network model from the two retrained generative pre-trained first network models;

[0040] The game model determination module is used to retrain two generative pre-trained second network models with different hyperparameters in order of segmentation ratio from large to small based on other game data subsets, and obtain the game model from the two generative pre-trained network models that have been trained last.

[0041] According to a third aspect of the present invention, there is provided a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for constructing a game model in any possible implementation of the first aspect.

[0042] According to a fourth aspect of the present invention, there is provided an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for constructing a game model in any possible implementation of the first aspect are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 A flowchart of a method for constructing a game model provided by an embodiment of the present invention;

[0046] Figure 2 A flowchart of step S102 in a method for constructing a game model provided by an embodiment of the present invention;

[0047] Figure 3 A flowchart of step S103 in a method for constructing a game model provided by an embodiment of the present invention;

[0048] Figure 4 Another schematic diagram of the process of a method for constructing a game model provided by an embodiment of the present invention;

[0049] Figure 5 Another schematic diagram of the process of a method for constructing a game model provided by an embodiment of the present invention;

[0050] Figure 6 A schematic diagram of a device for constructing a game model provided by an embodiment of the present invention;

[0051] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] See also Figure 1 The embodiment of the present invention provides a method for constructing a game model, which may include the following steps:

[0054] S101, obtaining target game information for playing a target game with incomplete information from a preset expert system;

[0055] In this embodiment, for incomplete information games, the game information of each expert in the incomplete information game stored in the expert system is obtained. As an optional embodiment, the target game information includes but is not limited to: the cooperative relationship of each participant in the incomplete information target game, the total game resources of the participants, and the game resources of the participants in each round of the incomplete information target game.

[0056] In this embodiment, the expert is a person who has in-depth research and practical experience in incomplete information games. Taking the game of throwing eggs as an example, the complete game information of each game of high-level human players can be collected through multiple channels, that is, the target game information includes: the cooperative relationship of each player (the cooperative relationship of each participant), the hand cards of each player (the total game resources of the participant), and the cards played by each player in each round (the game resources of each round of the participant in the incomplete information target game). By collecting the complete game information of each game of the throwing egg game by high-level human players, the original data set of the throwing egg game is obtained.

[0057] S102, preprocessing the target game information to obtain a game data set;

[0058] In this embodiment, the collected original data set of the throwing egg data is preprocessed to obtain the throwing egg game data set.

[0059] Figure 2 A flowchart of step S102 in a method for constructing a game model provided by an embodiment of the present invention. Figure 2 As shown, in this embodiment, as an optional embodiment, the target game information is preprocessed to obtain a game data set, including:

[0060] S201, deleting incomplete information in the target game information, and, according to a preset filtering strategy for the target game with incomplete information, filtering out information that meets the filtering strategy to obtain an initial game data set;

[0061] In this embodiment, erroneous, missing, unavailable, and low-quality information in the target game information is deleted.

[0062] S202, extracting initial game data subsets of each round from the initial game data set according to the time sequence of the game, and arranging the initial game data subsets according to the data arrangement strategy set for the incomplete information target game to obtain a round data sequence;

[0063] In this embodiment, the incomplete information target game is taken as an example of the Pai Egg game, and the data arrangement strategy arranges the original data of a round of the Pai Egg game, including but not limited to: (next player plays, opponent plays, previous player plays, card counter, own player's hand, play). That is, for the Pai Egg game, the Pai Egg game information (playing card data) of each round in each game is arranged in the form of (next player plays, opponent plays, previous player plays, card counter, own player's hand, play), and the playing card data of each round is arranged in the order of time, and the rounds in the order of sequence are spliced ​​to obtain the round data sequence.

[0064] S203, for each round of the round data sequence, set the data at the corresponding position of the participant who did not participate in the round to 0, to obtain an intermediate data sequence;

[0065] In this embodiment, taking the egg-throwing game as an example, specifically, the first round of the egg-throwing game is performed in the form of (the next player plays cards, the opponent plays cards, the previous player plays cards, the card counter, the player's hand, plays cards) for the players who have not played cards. The opening round of the egg-throwing game and the subsequent card-throwing rounds are performed in the form of (the next player plays cards, the opponent plays cards, the previous player plays cards, the card counter, the player's hand, plays cards) for the players who have not played cards. The last round of the egg-throwing game is performed in the form of (the next player plays cards, the opponent plays cards, the previous player plays cards, the card counter, the player's hand, plays cards) for the players who have not played cards.

[0066] S204. Based on the game object attributes and the number of game objects included in the data arrangement strategy, perform matrix expansion on each data in the intermediate data sequence to obtain the game data set.

[0067] In this embodiment, taking the game of throwing eggs as an example, the game objects included in the data arrangement strategy include: the next player plays, the opponent plays, the previous player plays, the card counter, the player's hand, and the play. That is, the number of game objects is the number of data included in the initial game data subset of each round, which is 6 in this embodiment. The game object attribute is that the game object has four suits, and each suit includes 14 cards. Therefore, for each game object, the game object attribute is a 4*14 matrix, in which the rows represent the suits and the columns represent the sizes of the playing cards.

[0068] In the present embodiment, each data in the intermediate data sequence, for example, the next player plays cards, is stored in a 4*14 matrix form, and each row represents the different suits of playing cards, specifically expressed as: spades, hearts, diamonds, and clubs. Each column represents the size of playing cards, specifically expressed as: A, 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, trump. The size of each value of the matrix is ​​the number of playing cards of the suit represented by the column where the value is located and the number of playing cards of the suit represented by the row, and the value range is (0, 1, 2). There are no four suits for trump cards, so the first line representing the trump card is the number of big trump cards, the second line is the number of small trump cards, and the third and fourth lines are always 0.

[0069] In this embodiment, as an optional embodiment, the matrix corresponding to each element of the intermediate data sequence is flattened into a one-dimensional tensor, and the last two elements of the tensor that are always 0 are deleted to facilitate neural network learning.

[0070] S103, dividing the game data set into multiple game data subsets according to a preset segmentation strategy, and obtaining a generative pre-trained first network model based on the game data subset with the largest segmentation ratio;

[0071] In this embodiment, a GPT network model is constructed as a behavior cloning model. The GPT network model is composed of a plurality of stacked multi-head self-attention mechanism blocks. In this embodiment, since the dimensions of each element of the game data set (throwing egg game data set) are the same, the game data set in sequence form can be directly used as the input sequence of the multi-head self-attention mechanism block.

[0072] In this embodiment, the Throwing Egg game data set (game data set) is segmented, for example, the Throwing Egg game data set is segmented according to 50%, 30%, 10%, and 10%. As another optional embodiment, the Throwing Egg game data set can also be segmented according to other proportions, and the game data subset with the highest segmentation ratio is used as the first training set. When segmenting, it is necessary to ensure that the amount of data of the game data subset with the highest segmentation ratio is sufficient, so that it can be ensured that the generative pre-training model trained by the neural network has the game playing ability after being trained with the game data subset, that is, it has a certain Throwing Egg game ability.

[0073] Figure 3A flowchart of step S103 in a method for constructing a game model provided by an embodiment of the present invention. Figure 3 As shown, in this embodiment, as an optional embodiment, based on the game data subset with the largest segmentation ratio, a generative pre-trained first network model is obtained, including:

[0074] S301, extracting a game data subset with the largest split ratio from multiple game data subsets to obtain a game data subset with the largest ratio;

[0075] S302, randomly setting model parameters of two generative pre-trained initial network models, setting hyperparameters according to a preset strategy, and inputting the maximum proportion game data subsets into the generative pre-trained initial network models respectively;

[0076] In this embodiment, the GPT network hyperparameters of the GPT network model are set, and two GPT network models with the same structure are trained using the game data subset with the highest segmentation ratio. During the training process, only the gradient back propagation is performed on the prediction head to update the parameters. Among them, the two GPT network models with the same structure have different GPT network hyperparameters.

[0077] In this embodiment, as an optional embodiment, the GPT network hyperparameters include but are not limited to: batch size, learning rate, dropout rate, number of attention heads, number of multi-head self-attention blocks, etc.

[0078] S303, position encoding each element in the maximum proportion game data subset input to the generative pre-trained initial network model using triangular position encoding to obtain an input sequence, and simultaneously input the Q network, K network and V network through a single multi-head self-attention mechanism block of the generative pre-trained initial network model;

[0079] S304, calculating the attention weight of each element according to the Query value, Key value and Value value of each element in the input sequence;

[0080] In this embodiment, as an optional embodiment, the formula for calculating the attention weight of an element is as follows:

[0081]

[0082] Among them, Attention(Q,K,V) is the attention weight of the element, Q, K, V represent the Query value, Key value and Value value of the element respectively, d k Indicates the dimension of each element. In this embodiment, the dimension is 4x14=56. 4 represents 4 suits, 14 represents 14 card sizes, and a 4*14 tensor can represent all card types.

[0083] In this embodiment, the input sequence is a six-tuple arranged in chronological order (next player plays, opponent plays, previous player plays, card counter, own player's hand, play). Each element of the input sequence is simultaneously sent to three networks with the same network structure, Q network, K network, and V network, to calculate the QKV value of each element, and calculate the attention weight of the current element and the previous element according to the preset attention weight formula, where Q comes from the Q value of the current element, and K and V come from the target element for which the attention weight needs to be calculated.

[0084] In this embodiment, as another optional embodiment, the attention weight can also be calculated using the following formula:

[0085]

[0086] Where D is the input dimension of the qkv network, that is, 4*14 / n, where n is the number of attention heads.

[0087] S305, inputting the calculated attention weights of each element into the first dropout layer, the first fully connected network layer, and the second dropout layer of the generative pre-trained initial network model in sequence to obtain an element processing sequence, and performing a residual connection operation with the input sequence to obtain a first residual connection sequence;

[0088] In this embodiment, the obtained attention weights of each element are input into the first drop layer for processing, the result output by the first drop layer is input into the first fully connected network layer for processing, the result output by the first fully connected network layer is input into the second drop layer for processing, and the result output by the second drop layer is residually connected with the initial input (input sequence), that is, the output of the last drop layer is added to the input sequence of the multi-head self-attention mechanism block.

[0089] In this embodiment, as an optional embodiment, the structure of the first drop layer is the same as that of the second drop layer, including but not limited to: using the standard drop layer of pytorch.

[0090] S306, performing a layer normalization operation on the residual connection sequence to obtain a normalized sequence;

[0091] In this embodiment, the result after residual connection is input to perform layer normalization operation, that is, using the following formula:

[0092]

[0093] Among them, x i is the output of the second dropout layer, i.e., the first residual connection sequence, μ is the mean of the output of the second dropout layer, σ is the standard deviation of the output of the second dropout layer, γ is the scaling parameter, and β is the offset parameter.

[0094] S307, inputting the normalized sequence into the second fully connected network layer of the generative pre-trained initial network model, performing a residual connection operation on the output of the second fully connected network layer and the normalized sequence to obtain a second residual connection sequence;

[0095] In this embodiment, the result of the layer normalization operation is input into a second fully connected network layer whose output dimension is the same as the input dimension of the multi-head self-attention mechanism block, and then the second fully connected network layer is residually connected, that is, the output of the second fully connected network layer and the normalized sequence of the input second fully connected network layer are added.

[0096] S308, inputting the second residual connection sequence into the third discard layer of the generative pre-trained initial network model to obtain a predicted sequence;

[0097] In this embodiment, the result of the residual connection (the second residual connection sequence) is input into the third discard layer to obtain the output of the multi-head self-attention block.

[0098] In this embodiment, a behavior cloning GPT model is built according to the GPT network structure. The input sequence is positionally encoded using triangular position encoding. The input sequence of the behavior cloning GPT model is the sequence of (next player plays, opponent plays, previous player plays, card counter, own player’s hand, play) in each round, and the QKV network input dimension is 4*14-2=54.

[0099] In this embodiment, as an optional embodiment, the prediction sequence includes playing cards, and other parameters are not predicted.

[0100] S309, performing gradient back propagation on the generative pre-trained initial network model according to the target prediction decision information in the prediction sequence and the corresponding target actual decision information in the input sequence to update the model parameters of the generative pre-trained initial network model;

[0101] In this embodiment, the game data subset of the previous round is used as the input of the generative pre-trained initial network model to obtain the output of the generative pre-trained initial network model, that is, the cards played in the game data subset of the current round predicted by the generative pre-trained initial network model. For example, the first round of the Pai Gow game data is input into the pre-trained initial network model, and the pre-trained initial network model predicts the cards played in the next round of Pai Gow game data (next player plays, opponent plays, previous player plays, card recorder, own player's hand, playing cards), and based on the predicted playing cards and the actual playing cards of the next round, the model parameters are updated according to the gradient back propagation.

[0102] In this embodiment, the game data subset with the largest segmentation ratio is used to train two GPT initial network models with the same structure respectively. During the training process, the card prediction head is subjected to gradient back propagation to update the model parameters.

[0103] In this embodiment, the generative pre-trained initial network model only predicts the card head.

[0104] S310. From the two generative pre-trained initial network models that are updated with gradient back-propagation, select the generative pre-trained initial network model with higher decision-making ability as the generative pre-trained first network model according to the test environment.

[0105] In this embodiment, after training with the first part of the data set (the subset of game data with the highest segmentation ratio), the two GPT network models have different network parameters due to different dropout layers and initialization hyperparameters.

[0106] In this embodiment, two GPT network models are tested in a game of throwing eggs. During the test, the input of the GPT network model, i.e., the next player's card, the opponent's card, the previous player's card, and the hand of the family are given by the environment, and the card counter is a full card matrix with all elements of 2 except the last two elements of the trump card column being 0, minus the next player's card, the opponent's card, the previous player's card, and the hand of the family in each round, and real-time calculation is performed in each round. The card is generated by the GPT initial network model. After the two GPT initial network models have played a preset number of throwing eggs, the GPT network model with better performance is selected by humans or based on a pre-set selection strategy as the training model for the next stage.

[0107] S104, based on the game data subset with the second largest segmentation ratio, retraining the two generative pre-trained first network models with different hyperparameters, and obtaining a generative pre-trained second network model from the two retrained generative pre-trained first network models;

[0108] In this embodiment, since the neural network used for training the generative pre-trained network model has a random dropout layer and different hyperparameters, the models with the same structure are trained using the same data set, and the obtained neural network parameters (model parameters) are different, so that the performance of the two trained generative pre-trained network models is also different.

[0109] In this embodiment, as an optional embodiment, two models are trained at each stage of training, and then the model with better performance is selected in the test environment for the next step of training. In this way, the data set can be more effectively utilized to train a model with better performance.

[0110] S105. In order of the segmentation ratio from large to small, based on other subsets of game data, two generative pre-trained second network models with different hyperparameters are retrained in turn, and the game model is obtained from the two generative pre-trained network models that have been trained last.

[0111] In this embodiment, two GPT network models with the same structure are trained respectively, and the two trained GPT network models with the same structure are tested against each other to obtain a GPT network training model. For example, if the other game data subsets also include a 10% first game data subset and an 8% second game data subset, then two generative pre-trained second network models with different hyperparameters are retrained based on the first game data subset, and from the two retrained generative pre-trained second network models, a generative pre-trained second network model with higher decision-making ability is selected according to the test environment to obtain a generative pre-trained third network model, and two generative pre-trained third network models with different hyperparameters are retrained based on the second game data subset, and from the two retrained generative pre-trained third network models, a generative pre-trained third network model with higher decision-making ability is selected according to the test environment to obtain a game model.

[0112] In this embodiment, the two GPT network models are trained using the second part of the data set by copying the selected better model. Due to the presence of the dropout layer, the two GPT network model parameters are different. Similarly, the two trained GPT network models are tested against each other to select a better model. This process is repeated until all data sets are trained.

[0113] In the present invention, the egg-throwing data set is divided into multiple data sets according to 50%, 30%, 10%, and 10%, and the training is carried out in sequence. In this way, by dividing into multiple data sets, a better model can be selected and its performance can be improved. After each part of the data set is used for training, the two models are selected and the better model parameters are used for the next stage of training.

[0114] In this embodiment, the amount of the first part of the data set should be as large as possible, so that the model can be trained from a random model to a model with the ability to play the game of throwing eggs, so that it can be optimized in a test environment. The size of the subsequent data sets can be reduced step by step, and the model can be further trained on the basis of the previous model to improve the model's ability to play the game of throwing eggs. The purpose of using a small-scale data set for training in the later stage of training is similar to the fine-tuning process of the language model, that is, through the training of a small-scale data set, a model with certain basic capabilities can be better qualified for downstream tasks. The purpose here is to further improve the model's understanding and decision-making ability of the game of throwing eggs.

[0115] Figure 4 Another schematic diagram of the process of a method for constructing a game model provided by an embodiment of the present invention. Figure 4As shown, in this embodiment, the input sequence X (B, T, N * D) is input into a single multi-head self-attention mechanism block, and the single multi-head self-attention mechanism block inputs the input sequence into the Q network (q_net), the K network (k_net) and the V network (v_net), and outputs q, k, and v respectively. The attention coefficient is calculated based on q and k, and then the attention weight is calculated based on the attention coefficient and v. The attention weights of each element calculated are sequentially input into the first dropout layer (dropout), the first fully connected network layer (project) and the second dropout layer (dropout) of the generative pre-trained initial network model to obtain an element processing sequence, and a residual connection operation is performed with the input sequence to obtain a first residual connection sequence; the first residual connection sequence is input into the normalization layer (layernorm) for layer normalization operation to obtain a normalized sequence, the normalized sequence is input into the second fully connected network layer, and the output of the second fully connected network layer is residually connected with the normalized sequence to obtain a second residual connection sequence; the second residual connection sequence is input into the third dropout layer (dropout layer) to obtain a predicted sequence X' (B, T, N * D).

[0116] Figure 5 Another schematic diagram of the process of a method for constructing a game model provided by an embodiment of the present invention. Figure 5 As shown, the input sequence X(B, T, N*D) is (next player plays, opponent plays, previous player plays, card counter, own player's hand, play). After the input sequence is triangularly position encoded, it is sent to the GPT / decoder structure. Then, the linear prediction head is used to obtain the prediction sequence, among which the next player plays, opponent plays, previous player plays, card counter, and play are not predicted and updated, and the own player's hand is predicted to update the playing parameters.

[0117] The method for constructing a game model proposed in this embodiment constructs a neural network model based on the human feedback behavior of the transformer, imitates the strategies of high-level human players through the neural network, and realizes high-level decision-making for incomplete information game problems using the neural network in the game model.

[0118] In this embodiment, as an optional embodiment, the method further includes:

[0119] Obtain the cooperative relationship of each participant in the current round of the incomplete information target game, the total game resources of the participants, and the game resources of each participant except the participant itself, input the game model, obtain the game resources of the participant itself in the current round, and play the game based on the obtained game resources.

[0120] In this embodiment, as another optional embodiment, the method further includes:

[0121] If it is determined that the game based on the obtained game resources fails, obtaining the game resources of the participant in each round of the incomplete information target game;

[0122] The game resources of each round are changed according to a preset expert system, and the game model is retrained according to the changed game resources of each round.

[0123] Based on the same inventive concept, Figure 6 As shown, an embodiment of the present invention further provides a device for constructing a game model, the device comprising:

[0124] The game information acquisition module 601 is used to acquire target game information for the incomplete information target game from a preset expert system;

[0125] In this embodiment, as an optional embodiment, the target game information includes but is not limited to: the cooperative relationship of each participant in the incomplete information target game, the total game resources of the participants, and the game resources of the participants in each round of the incomplete information target game.

[0126] An information preprocessing module 602 is used to preprocess the target game information to obtain a game data set;

[0127] In this embodiment, as an optional embodiment, the information preprocessing module 602 includes:

[0128] A filtering unit, configured to delete incomplete information in the target game information, and, according to a preset filtering strategy for the target game with incomplete information, filter out information that complies with the filtering strategy to obtain an initial game data set;

[0129] A data arrangement unit is used to extract the initial game data subsets of each round from the initial game data set according to the time sequence of the game, and arrange the initial game data subsets according to the data arrangement strategy set for the incomplete information target game to obtain a round data sequence;

[0130] A data completion unit is used to set the data corresponding to the position of the participant who did not participate in the round to 0 for each round of the round data sequence, so as to obtain an intermediate data sequence;

[0131] The data expansion unit is used to perform matrix expansion on each data in the intermediate data sequence based on the game object attributes and the number of game objects included in the data arrangement strategy to obtain the game data set.

[0132] An initial training module 603 is used to divide the game data set into multiple game data subsets according to a preset segmentation strategy, and obtain a generative pre-trained first network model based on the game data subset with the largest segmentation ratio;

[0133] In this embodiment, as an optional embodiment, the initial training module 603 is specifically used for:

[0134] Extracting the game data subset with the largest split ratio from multiple game data subsets to obtain the game data subset with the largest ratio;

[0135] Randomly setting model parameters of two generative pre-trained initial network models, setting hyperparameters according to a preset strategy, and inputting the maximum proportion game data subsets into the generative pre-trained initial network models respectively;

[0136] The input sequence is obtained by positionally encoding each element in the maximum proportion of the game data subset input into the generative pre-trained initial network model using triangular position encoding, and then simultaneously input into the Q network, K network and V network through a single multi-head self-attention mechanism block of the generative pre-trained initial network model;

[0137] Calculate the attention weight of each element based on the Query value, Key value, and Value value of each element in the input sequence;

[0138] Input the calculated attention weights of each element into the first dropout layer, the first fully connected network layer, and the second dropout layer of the generative pre-trained initial network model in sequence to obtain an element processing sequence, and perform a residual connection operation with the input sequence to obtain a first residual connection sequence;

[0139] Perform layer normalization on the residual connection sequence to obtain a normalized sequence;

[0140] Inputting the normalized sequence into the second fully connected network layer of the generative pre-trained initial network model, performing a residual connection operation on the output of the second fully connected network layer and the normalized sequence to obtain a second residual connection sequence;

[0141] Input the second residual connection sequence into the third dropout layer of the generative pre-trained initial network model to obtain a predicted sequence;

[0142] Based on the target prediction decision information in the prediction sequence and the corresponding target actual decision information in the input sequence, gradient backpropagation is performed on the generative pre-trained initial network model to update the model parameters of the generative pre-trained initial network model;

[0143] From the two generative pre-trained initial network models that perform gradient back-propagation update, the generative pre-trained initial network model with higher decision-making ability is selected as the generative pre-trained first network model according to the test environment.

[0144] In this embodiment, as an optional embodiment, the GPT network hyperparameters include but are not limited to: batch size, learning rate, dropout rate, number of attention heads, and number of multi-head self-attention blocks.

[0145] In this embodiment, as an optional embodiment, the hyperparameters of the two generative pre-trained initial network models are different.

[0146] A retraining module 604 is used to retrain two generative pre-trained first network models with different hyperparameters based on the game data subset with the second largest segmentation ratio, and obtain a generative pre-trained second network model from the two retrained generative pre-trained first network models;

[0147] The game model determination module 605 is used to retrain the two generative pre-trained second network models with different hyperparameters in order of segmentation ratio from large to small based on other game data subsets, and obtain the game model from the two generative pre-trained network models that have been trained last.

[0148] In this embodiment, as an optional embodiment, the device further includes:

[0149] The game module (not shown in the figure) is used to obtain the cooperative relationship of each participant in the current round of the incomplete information target game, the total game resources of the participants, and the game resources of each participant except itself, input the game model, obtain the game resources of its own participant in the current round, and play the game based on the obtained game resources.

[0150] In this embodiment, as another optional embodiment, the device further includes:

[0151] A correction module, for obtaining the gaming resources of the own participant in each round of the incomplete information target game if it is determined that the gaming according to the acquired gaming resources fails;

[0152] The game resources of each round are changed according to a preset expert system, and the game model is retrained according to the changed game resources of each round.

[0153] Based on the same inventive concept, an embodiment of the present invention further provides a storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the method for constructing a game model in any possible implementation manner described above are implemented.

[0154] Alternatively, the storage medium may be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0155] Based on the same inventive concept, see Figure 7 The embodiment of the present invention further provides an electronic device, including a memory 101 (such as a non-volatile memory), a processor 102, and a computer program stored in the memory 101 and executable on the processor 102. When the processor 102 executes the program, the steps of the method for constructing a game model in any possible implementation manner described above can be equivalent to the device for constructing a game model as described above. Of course, the processor can also be used to process other data or operations. The electronic device can be a PC, a server, a terminal, and other devices.

[0156] like Figure 7 As shown, the electronic device may also generally include: a memory 103, a network interface 104, and an internal bus 105. In addition to these components, other hardware may also be included, which will not be described in detail.

[0157] It should be pointed out that the above-mentioned device for constructing a game model can be implemented through software. As a device in a logical sense, it is formed by the processor 102 of the electronic device in which it is located reading the computer program instructions stored in the non-volatile memory into the memory 103 for execution.

[0158] The embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0159] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform the corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.

[0160] Computers suitable for executing computer programs include, for example, general and / or special microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, the computer does not necessarily have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0161] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0162] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although the features may work as above in certain combinations and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0163] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0164] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0165] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0166] The above is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features applied herein.

Claims

1. A method for constructing a game model, It is characterized in that include: Obtaining target game information for playing the target game with incomplete information from a pre-set expert system; Preprocessing the target game information to obtain a game data set; Divide the game data set into multiple game data subsets according to a preset segmentation strategy, and obtain a generative pre-trained first network model based on the game data subset with the largest segmentation ratio; Based on the subset of game data with the second largest split ratio, two generative pre-trained first network models with different hyperparameters are retrained, and a generative pre-trained second network model is obtained from the two retrained generative pre-trained first network models; According to the order of the split ratio from large to small, based on other subsets of the game data, the two generative pre-trained second network models with different hyperparameters are retrained in turn, and the game model is obtained from the two generative pre-trained network models that have been trained last.

2. The method for constructing a game model according to claim 1, It is characterized in that The preprocessing of the target game information to obtain a game data set includes: Deleting incomplete information in the target game information, and, according to a preset filtering strategy for the target game with incomplete information, filtering out information that meets the filtering strategy to obtain an initial game data set; Extracting the initial game data subsets of each round from the initial game data set according to the time sequence of the game, arranging the initial game data subsets according to the data arrangement strategy set for the incomplete information target game, and obtaining a round data sequence; For each round of the round data sequence, the data corresponding to the participants who did not participate in the round are set to 0 to obtain the intermediate data sequence; Based on the game object attributes and the number of game objects included in the data arrangement strategy, matrix expansion is performed on each data in the intermediate data sequence to obtain the game data set.

3. The method for constructing a game model according to claim 1, It is characterized in that The obtaining of a generative pre-trained first network model based on the game data subset with the largest segmentation ratio comprises: Extracting the game data subset with the largest split ratio from multiple game data subsets to obtain the game data subset with the largest ratio; Randomly setting model parameters of two generative pre-trained initial network models, setting hyperparameters according to a preset strategy, and inputting the maximum proportion game data subsets into the generative pre-trained initial network models respectively; The input sequence is obtained by positionally encoding each element in the maximum proportion of the game data subset input into the generative pre-trained initial network model using triangular position encoding, and then simultaneously input into the Q network, K network and V network through a single multi-head self-attention mechanism block of the generative pre-trained initial network model; Calculate the attention weight of each element based on the Query value, Key value, and Value value of each element in the input sequence; Input the calculated attention weights of each element into the first dropout layer, the first fully connected network layer, and the second dropout layer of the generative pre-trained initial network model in sequence to obtain an element processing sequence, and perform a residual connection operation with the input sequence to obtain a first residual connection sequence; Perform layer normalization on the residual connection sequence to obtain a normalized sequence; Inputting the normalized sequence into the second fully connected network layer of the generative pre-trained initial network model, performing a residual connection operation on the output of the second fully connected network layer and the normalized sequence to obtain a second residual connection sequence; Input the second residual connection sequence into the third dropout layer of the generative pre-trained initial network model to obtain a predicted sequence; Based on the target prediction decision information in the prediction sequence and the corresponding target actual decision information in the input sequence, gradient backpropagation is performed on the generative pre-trained initial network model to update the model parameters of the generative pre-trained initial network model; From the two generative pre-trained initial network models that perform gradient back-propagation update, the generative pre-trained initial network model with higher decision-making ability is selected as the generative pre-trained first network model according to the test environment.

4. The method for constructing a game model according to claim 3, It is characterized in that The GPT network hyperparameters include: batch size, learning rate, dropout rate, number of attention heads, and number of multi-head self-attention blocks.

5. The method for constructing a game model according to claim 3, It is characterized in that The two generative pre-trained initial network models have different hyperparameters.

6. The method for constructing a game model according to any one of claims 1 to 5, It is characterized in that The target game information includes: the cooperative relationship of each participant in the incomplete information target game, the total game resources of the participants, and the game resources of the participants in each round of the incomplete information target game.

7. The method for constructing a game model according to claim 6, It is characterized in that The method further comprises: Obtain the cooperative relationship of each participant in the current round of the incomplete information target game, the total game resources of the participants, and the game resources of each participant except the participant itself, input the game model, obtain the game resources of the participant itself in the current round, and play the game based on the obtained game resources.

8. A device for constructing a game model, It is characterized in that The device for constructing the game model comprises: A game information acquisition module, used to acquire target game information for playing a target game with incomplete information from a pre-set expert system; An information preprocessing module, used to preprocess the target game information to obtain a game data set; An initial training module, used to segment the game data set into multiple game data subsets according to a preset segmentation strategy, and obtain a generative pre-trained first network model based on the game data subset with the largest segmentation ratio; A retraining module is used to retrain two generative pre-trained first network models with different hyperparameters based on a subset of game data with a second largest segmentation ratio, and obtain a generative pre-trained second network model from the two retrained generative pre-trained first network models; The game model determination module is used to retrain two generative pre-trained second network models with different hyperparameters in order of segmentation ratio from large to small based on other game data subsets, and obtain the game model from the two generative pre-trained network models that have been trained last.

9. A storage medium, It is characterized in that The storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the steps of the method for constructing a game model as described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the steps of the method for constructing a game model according to any one of claims 1 to 7 are implemented.

Images