Method, system, computer program product and storage medium for generating a learned model

Abstract

A method capable of generating a learned model for predicting an action to be selected by a user in any game state is provided. The method of one embodiment of the invention is for generating a learned model for predicting an action to be selected by a user in a game that progresses and updates a game state in accordance with the action selected by the user, the method including the steps of generating text data, i.e., game state texts and action texts, expressed in a prescribed form, from data of game states and actions included in historical record data related to the game, and generating learning data including pairs of game state texts and action texts corresponding to pairs of one game state and an action selected in the one game state; and generating the learned model based on the generated learning data.

Description

Technical Field

[0001] The present invention relates to a method for generating a learned model for predicting the action a user will choose, a method for determining the action predicted to be chosen by the user, etc. Background Technology

[0002] In recent years, there has been an increase in the number of players who can participate in online games via the internet. These games are implemented through game systems that allow communication between mobile devices and game operators' servers, enabling players operating mobile devices to engage in battles with other players.

[0003] Online games include those that progress and update game state information representing the game state in response to actions selected by the user. Examples of such games include card games known as digital collectible card games (DCCGs), where various actions are performed based on combinations of game media such as cards and characters.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent No. 6438612

[0007] Non-patent literature

[0008] Non-patent document 1: Jacob Devlin and Ming-Wei Chang and Kenton Lee andKristina Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," arXiv:1810.04805, 2018

[0009] Non-patent document 2: Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17). CurranAssociates Inc., Red Hook, NY, USA, 6000-6010 Summary of the Invention

[0010] The problem the invention aims to solve

[0011] In online games, the goal is to achieve AI that utilizes historical game data (replay logs) as data for machine learning to predict the actions a user will choose (execute) in any game state, thus reproducing near-human behavior. For example, Patent Document 1 discloses a technique for inferring actions that are more likely to be performed by the user. On the other hand, neural network technology called a transformer (transformer neural network technology) (non-patent documents 1 and 2) is effective in learning causal and sequential relationships, such as in turn-based battle games, but it is difficult to use for learning from historical game data.

[0012] This invention was made to solve such a problem, and its main objective is to provide a method for generating a learned model that can use neural network technology capable of natural language processing to predict the actions a user will choose in any game state.

[0013] Solution for solving the problem

[0014] One embodiment of the present invention provides a method for generating a learned model, the learned model being used to predict the action a user will choose in a game that progresses and updates the game state in accordance with an action selected by a user, the method comprising the following steps:

[0015] Based on game-related historical data containing game state and action data, text data in a prescribed format, namely game state text and action text, is generated. Learning data is also generated, containing pairs of game state text and action text corresponding to a game state and the action selected in that game state.

[0016] The learned model is generated based on the generated learning data.

[0017] Furthermore, in one embodiment of the present invention,

[0018] The steps for generating the learning data include the following: generating multiple game state texts with different arrangements of multiple element texts contained in the game state text as game state texts corresponding to a game state, and generating learning data containing each game state text in the generated multiple game state texts and the action text corresponding to the action selected in the game state.

[0019] Furthermore, in one embodiment of the present invention,

[0020] The steps for generating the learned model include the following: generating the learned model by having a pre-learned natural language learning model, which has learned grammatical structures and relationships between articles related to natural language, learn the generated learning data.

[0021] Furthermore, in one embodiment of the present invention,

[0022] The step of generating the learning data includes the following steps: generating learning data containing a first pair and a second pair, wherein the first pair is a pair of game state text and action text corresponding to a game state and the action selected in that game state, generated based on the game state and action data contained in the historical data; and the second pair is a pair of game state text and action text corresponding to an action text randomly selected from the actions available to the user in that game state and not included in the first pair.

[0023] The steps for generating the learned model include the following: learning the first pair as correct data and learning the second pair as incorrect data to generate the learned model.

[0024] Furthermore, in one embodiment of the present invention,

[0025] The steps for generating the learning data include the following: generating game state text and action text using grammar, syntax, and lexical representations suitable for machine transformation into distributed representations, based on pre-created rule-based systems, game state data, and action data.

[0026] One embodiment of the present invention provides a method for determining an action predicted to be chosen by the user in a game that progresses and updates the game state in accordance with actions selected by the user, the method comprising the following steps:

[0027] Determines the multiple actions a user can choose in the game state that is being predicted;

[0028] For each of the determined actions, pairs of game state text and action text are generated based on the pairs of game state data and action data; and

[0029] Using the generated pairs of game state texts and action texts, along with the learned model described above, the action predicted to be chosen by the user is determined.

[0030] The program of one embodiment of the present invention is characterized in that it causes a computer to execute the steps of the above-described method.

[0031] Additionally, a system according to one embodiment of the present invention is used to generate a learned model, which is used to predict the action a user will choose in a game that progresses and updates the game state in accordance with the action selected by the user. The system performs the following processing:

[0032] Based on game-related historical data containing game state and action data, text data in a prescribed format, namely game state text and action text, is generated. Learning data is also generated, containing pairs of game state text and action text corresponding to a game state and the action selected in that game state.

[0033] The learned model is generated based on the generated learning data.

[0034] Additionally, in one embodiment of the present invention, a system is used to determine an action predicted to be chosen by the user in a game where the game state progresses and updates accordingly with actions selected by the user. The system performs the following processing:

[0035] Determines the multiple actions a user can choose in the game state that is being predicted;

[0036] For each of the determined actions, pairs of game state text and action text are generated based on the pairs of game state data and action data; and

[0037] Using the generated pairs of game state texts and action texts, along with the learned model described above, the action predicted to be chosen by the user is determined.

[0038] The effects of the invention

[0039] According to the present invention, neural network technology capable of natural language processing can be used to generate a fully learned model for predicting the action a user will choose in any game state. Attached Figure Description

[0040] Figure 1 This is a block diagram illustrating the hardware structure of a learning device according to one embodiment of the present invention.

[0041] Figure 2 This is a functional block diagram of a learning device according to one embodiment of the present invention.

[0042] Figure 3 This is an example of a game screen of the game in this embodiment displayed on the display of a user's terminal device.

[0043] Figure 4 This is an example of a game state.

[0044] Figure 5 This is a diagram showing a summary of how the learning device generates pairs of game status and action descriptions based on the playback log.

[0045] Figure 6 This is a flowchart illustrating the generation process of a learned model by a learning device according to one embodiment of the present invention.

[0046] Figure 7 This is a block diagram illustrating the hardware structure of a decision device according to one embodiment of the present invention.

[0047] Figure 8 This is a functional block diagram of a determining device according to one embodiment of the present invention.

[0048] Figure 9 This is a flowchart illustrating the decision-making process of a decision-making device according to an embodiment of the present invention, in which the decision is predicted to be the action that the user will choose. Detailed Implementation

[0049] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. One embodiment of the present invention includes a learning device 10 for generating a learned model for predicting the action a user will choose in a game where the game state progresses and updates in accordance with actions selected by the user (player). Another embodiment of the present invention includes a decision device 50 for determining the action predicted to be chosen by the user in a game where the game state progresses and updates in accordance with actions selected by the user. For example, the game targeted by the learning device 10 and decision device 50 is a game where, when a user selects an action in a certain game state, the selected action (attack, event, etc.) is executed and the game state is updated; for example, a card battle game.

[0050] The learning device 10 is an example of a system configured to generate a learned model, comprising one or more devices. In the following embodiments, for ease of explanation, it will be described as a single device. The system for generating the learned model can also refer to the learning device 10. The same applies to the decision device 50. Furthermore, in this embodiment, determining the game state or action can refer to data that determines the game state or action.

[0051] The card battle game described in this embodiment (the game of this embodiment) is provided by a game server, which is configured to include one or more server devices, similar to a typical online game. The game server stores the game program, which is an application for playing the game, and connects to the terminal devices of each user playing the game via a network. While each user is executing the game application installed on their terminal device, the terminal device communicates with the game server, and the game server provides game services via the network. At this time, the game server stores historical data related to the game (e.g., replay logs). However, the game server is not limited to the structure described above as long as it can access the replay logs.

[0052] In this embodiment, the game progresses by having a user select cards from a deck of multiple cards and place them in the game arena 43. Various events are then executed in accordance with the combination of cards and classes. Alternatively, this embodiment is a battle game where the user operating the user terminal device and other users operating other user terminal devices each select cards from their decks and place them in the game arena 43 to engage in combat. In this embodiment, each card 41 has card definition information including parameters such as card ID, card type, life points, attack power, and attributes; each class has class definition information.

[0053] Figure 3This is an example of a game screen of this embodiment displayed on the display of a user's terminal device. The game screen shows a game screen 40 of card battles between the user and other users. Game screen 40 shows a first card set 42a as the user's hand and a first card set 42b as the other user's hand. The first card set 42a and the first card set 42b include cards 41 associated with characters, items, or spells. The game is configured such that the user cannot see the cards 41 of the other user's first card set 42b. Game screen 40 also shows a second card set 44a as the user's deck and a second card set 44b as the other user's hand. Furthermore, the user or other users may not be actual players, but may be operated by a computer such as a game AI.

[0054] Each user's holding deck consists of a first deck 42 (42a or 42b) which is the user's hand, and a second deck 44 (44a or 44b) which is the user's card pile; this is generally referred to as a card deck. Whether the cards 41 held by a user are included in the first deck 42 or the second deck 44 depends on the progress of the game. The first deck 42 is the deck that the user can choose and place in the game field 43, while the second deck 44 is the deck that the user cannot choose. A holding deck consists of multiple cards 41, but depending on the progress of the game, a holding deck may sometimes consist of only one card 41. Furthermore, each user's card deck can consist of cards 41 of all different types, or it can consist of some cards 41 of the same type. In addition, the types of cards 41 constituting a user's card deck may be different from the types of cards 41 constituting other users' card decks. Alternatively, each user's holding deck may consist only of the first deck 42.

[0055] Game screen 40 displays the character 45a selected by the current user and the characters 45b selected by other users. The character selected by the user differs from the character associated with the card and is used to determine the class representing the type of card deck held. In this embodiment, the game configuration is such that the cards 41 held by the user vary according to class. In one example, the game configuration in this embodiment is such that the types of cards that can form each user's card deck vary according to class. However, the game in this embodiment may also not include classes. In this case, the game in this embodiment may not be limited to classes as described above, and game screen 40 may be set not to display the character 45a selected by the current user and the characters 45b selected by other users.

[0056] This embodiment of the game is a card battle game consisting of multiple rounds. In one example, the game is structured such that in each round, the user or other users can attack the opponent's card 41 or character 45 by performing actions such as selecting their own card 41, or using their own card 41 to produce a specified effect or trigger an event. In one example, the game is structured such that, for example, when the user selects card 41 and attacks, the user can select the opponent's card 41 or character 45 as the target of the attack. In one example, the game is structured such that, when the user selects card 41 and attacks, the target of the attack is automatically selected based on the card. In one example, the game is structured such that, in response to user actions on a card or character on the game screen 40, parameters such as the life points and attack power of other cards or characters are changed. In one example, the game is structured such that, when a specified condition is met in the game state, the card 41 corresponding to that condition is removed from the game field or moved to the user's or other user's card deck. For example, the replay log can be configured to comprehensively include a historical record of the information described above.

[0057] Furthermore, the card 41 (card set) can be a medium such as a character or item (medium set), and the holding card set can be a holding medium set consisting of multiple mediums held by the user. For example, if the medium set consists of mediums of characters and items, the game screen 40 will display the character or item itself as card 41.

[0058] Figure 1 This is a block diagram illustrating the hardware structure of a learning device 10 according to one embodiment of the present invention. The learning device 10 includes a processor 11, an input device 12, a display device 13, a storage device 14, and a communication device 15. These components are connected via a bus 16. Furthermore, interfaces are provided between the bus 16 and each component as needed. The learning device 10 has a structure similar to that of a general server, PC, etc.

[0059] The processor 11 controls the overall operation of the learning device 10. The processor 11 is, for example, a CPU. The processor 11 performs various processes by reading and executing programs and data stored in the storage device 14. The processor 11 can also be composed of multiple processors.

[0060] Input device 12 is a user interface for accepting input from the user to the learning device 10, such as a touch panel, touch pad, keyboard, mouse, or button. Display device 13 is a display that shows application screens, etc., to the user of the learning device 10 under the control of processor 11.

[0061] Storage device 14 includes a main storage device and an auxiliary storage device. The main storage device is, for example, a semiconductor memory such as RAM. RAM is a volatile storage medium capable of high-speed information reading and writing, and is used as the storage area and working area when the processor 11 processes information. The main storage device may also include ROM as a dedicated non-volatile storage medium for reading. The auxiliary storage device stores various programs and data used by the processor 11 when executing each program. As long as the auxiliary storage device can store information, it can be any non-volatile storage device or non-volatile memory, and it can also be a removable storage device.

[0062] The communication device 15 exchanges data with user terminals or other computers such as servers via a network, for example, a wireless LAN module. The communication device 15 can be configured as a Bluetooth module or other wireless communication device or module, or as an Ethernet module, USB interface or other wired communication device or module.

[0063] The learning device 10 is configured to retrieve a replay log as historical data related to the game from a game server. The replay log can contain data for each individual match or for each pre-determined unit. The replay log includes data on the game state and actions. For example, for each match, the replay log contains data on the game state and actions arranged chronologically. In one example, the replay log contains information for each round and for each user regarding the card 41 or character 45 selected by each user and the associated attacks. In another example, the replay log contains information for each round and for each user regarding the card 41 or character 45 selected by each user and the associated prescribed effects or events.

[0064] In this embodiment, the game state at least represents information that a user can visually recognize or perceive through playing the game, such as through game operations or displays on the game screen. The game state data includes data on the cards 41 placed in the game arena 43. Each piece of game state data corresponds to the game state at a specific moment in the game's progression. The game state data can include information about the cards 41 of the user's first card set 42a (or the card set they hold), and can also include information about the cards 41 of other users' first card sets 42b (or the card sets they hold).

[0065] In this embodiment, an action is an action performed by the user in a certain game state that can change that game state. For example, an action is an attack by a card 41 or a character 45 on other cards 41 or characters 45, or a predetermined effect or event triggered by a card 41 or a character 45. For example, an action is performed by the user selecting card 41. The data for each action corresponds to the action selected by the user in each game state. In one example, the action data includes data indicating that in a game state, the user has selected a card 41 to attack and the card 41 that is the target of the attack. In another example, the action data includes data indicating that in a game state, the user has selected a card 41 to use.

[0066] In one example, the replay log is defined by columns of game state data and data of actions performed by the user in that game state, where the game state data is text data representing a tree structure of the state of the game arena 43. In one example, the replay log is a sequence containing pairs of initial game state and initial action, and pairs of game state and next action as a result of the action, terminating with the final game state that ultimately determines the winner, which can be represented by equation (1).

[0067]

[0068] Here, State i Represents the i-th game state, Action i State represents the i-th action executed. e It indicates the final game state, such as victory, defeat, draw, or invalid match.

[0069] In one example, State i It is the set of cards 41 placed in the game field 43 and cards 41 held by the user, which can be represented by equation (2).

[0070]

[0071] Here,

[0072]

[0073] These are the cards from the 0th to the nathth placed on Player 1's (first turn) side in the game field 43.

[0074]

[0075] These are the 0th to nth cards placed on Player 2's (going second) side in the game field 43.

[0076]

[0077] These are cards numbered 0 through nc from player 1's (first to go) hand.

[0078]

[0079] These are the cards from the 0th to the ndth in Player 2's (going second) hand. For example, if Player 1 has 1 card placed in Game Field 43, then the card placed in Game Field 43 by Player 1 is State. i Only have

[0080]

[0081] Given that Player 1 has 0 cards placed in Game Field 43, the State is the card placed in Game Field 43. i This includes data indicating that there are no cards. The same applies to player 2's cards placed in the game arena 43, cards placed in the hand, etc. Additionally, it can also be set to State. i Includes cards 41 placed in the game arena 43, but excludes cards 41 held by the user. Alternatively, it can be set to State. i Includes information other than card 41.

[0082] Each card i It can be expressed by equation (3).

[0083]

[0084] Here, "name" is text data representing the card's name, and "explanation" is text data describing the card's abilities and skills.

[0085] Figure 2 This is a functional block diagram of a learning device 10 according to one embodiment of the present invention. The learning device 10 includes a learning data generation unit 21 and a learning unit 22. In this embodiment, these functions are implemented by the processor 11 executing a program stored in the storage device 14 or received via the communication device 15. Since various functions are implemented by reading programs in this way, a part or all of a function may be possessed by other parts. However, these functions may also be implemented in hardware by constructing circuits or the like to implement a part or all of each function.

[0086] The learning data generation unit 21 transforms the game state data and action data contained in the replay log into controlled natural language data expressed in a prescribed form, namely, game state description text and action description text. This is how the game state description text and action description text are created. In this embodiment, the learning data generation unit 21 uses a pre-created rule-based system to generate the game state description text and action description text based on the game state data and action data. In this embodiment, the controlled natural language expressed in a prescribed form is generally referred to as CNL (Controlled Natural Language), a natural language whose grammar and vocabulary are controlled to meet prescribed requirements. CNL is expressed, for example, in English. In this case, CNL is expressed in English with restrictions such as not containing relative pronouns. The learning data generation unit 21 generates learning data (training data) containing pairs of the generated (transformed) game state description text and action description text. The controlled natural language (CNL) data expressed in a prescribed form is, for example, text data expressed in a prescribed form using grammar, syntax, and vocabulary suitable for machine transformation into distributed representation. Furthermore, in this embodiment, generating learning data and other data can refer to the entire process of creating that data.

[0087] Figure 4 This is an example of a game state. For simplicity, Figure 4 The game state shown is when only two cards are placed in the game area 43 on player 1's side. Figure 4 In the game state shown, the two cards 41 placed on the game field 43 by player 1 are the "Twinblade Mage" card and the "Mechabook Sorcerer" card. In one example, the game state data contained in the replay log is the following text data.

[0088]

[0089] In this case, the learning data generation unit 21 transforms the above-mentioned game state data into the following game state description text (CNL).

[0090]

[0091] The learning data generation department 21 supplements the underlined words, commas, etc., to generate a sentence for each card. Each sentence contains, for example, words indicating the location where the card is placed, such as "on the player 1 side," words indicating attributes, such as "with" and "evolved," and commas indicating word breaks, etc.

[0092] When the game state data is text data recorded in a predetermined manner, the learning data generation unit 21 can use known rule-based system techniques to supplement the text data with prescribed words, commas, periods, etc., to transform the game state data into CNL. The rule-based system used for this transformation is pre-created, and the learning device 10 communicates with this rule-based system via the communication device 15, thereby enabling the transformation of the game state data into CNL. Furthermore, the learning device 10 may also possess this rule-based system.

[0093] The transformation from action data to action description text is the same as the transformation from game state data to game state description text. In one example, the action data contained in the replay log is the following text data.

[0094]

[0095] The learning data generation unit 21 transforms the above-mentioned action data into the following action description text (CNL).

[0096]

[0097] The learning data generation department 21 supplements the underlined words, etc., to generate a sentence for each action. For example, the sentence above indicates that player 1's "Figher" attacked "Fairy Champion".

[0098] In one example, the transformation of the learning data generation unit 21 into the game state description is implemented using the encode function shown in equation (4).

[0099]

[0100] The `encode` function is as follows: it accepts the data of the i-th game state, `State`. i Using this State i The explanation attributes of each card in the system, as shown in Equation (3), and the rule-based system, are used to process the accepted State. i Transformed into controlled natural language data State_T in a prescribed form i The learning data generation unit 21 generates action description texts (Action_T). i The transformation of ) can also be achieved by a function that has the same functionality as the encode function shown in equation (4).

[0101] As shown in equation (1), the replay log has any k-th State. k and Actionk Paired data structures (e.g., State0 and Action0 are paired, and State1 and Action1 are paired). In other words, the replay log contains data about a game state other than the final game state (State). k ) and the data of the action selected in that game state (Action) k Paired data structures. The learning data generation department 21 pairs of data for a game state (State). k ) and the data of the action selected in that game state (Action) k The transformation is performed to generate a game state description (State_T) that contains a pair of game states and the actions selected in that game state. k ) and Action Description (Action_T) k () the learning data of the correct.

[0102] Since most game state data contains multiple elements (data from multiple cards), the following implementation will be described assuming that the game state data contains data from multiple cards. The learning data generation unit 21 generates data based on the game state data (State). k Generate (transform) the game state description text (State_T) k The text contains multiple sentences. In this embodiment, each sentence in the game state description text corresponding to a game state corresponds to each element (card data) in the game state data. As the data corresponding to a game state (State) k The corresponding game state description text (State_T) k The learning data generation unit 21 generates multiple game state descriptions by shuffling the order of multiple sentences contained in the game state description. In this way, as data related to a game state (State) k The learning data generation unit 21 generates multiple game state descriptions (multiple game state descriptions) with different sentence orders within the corresponding game state description. The generated multiple game state descriptions can also include game state descriptions with the original sentence order.

[0103] The learning data generation unit 21 generates text data pairs of game state descriptions and action descriptions corresponding to the actions selected in the game state that forms the basis of the game state description, as described above, and generates learning data containing this generated text data. The action descriptions generated here are based on the game state (State) that forms the basis of the game state description. k Data of the selected action under ) k The generated action description text (Action_T) k In the case of generating pairs of game state descriptions and action descriptions corresponding to a game state, the action descriptions paired with each of the generated game state descriptions are the same action description.

[0104] If with State k The corresponding game status description includes N k If there are N sentences, then the arrangement of these sentences is N. k !type. In one example, the learning data generation unit 21 generates m game state descriptions with different sentence orders to serve as the basis for the State. k The corresponding game state description text (State_T) k There are m game state descriptions containing the same sentences but arranged differently. m is set to 2 ≤ m ≤ N. k Any integer of the order !. m can be set according to State. k For different values ​​of k, it is also possible to set a value such that 2≤m≤N is satisfied for any k. k The value of ! is fixed. However, m can also be set to 1.

[0105] Figure 5 This is a diagram showing an overview of how the learning device 10 generates pairs of game state descriptions and action descriptions based on the playback log. The learning data generation unit 21 generates m modes of game state descriptions as game state descriptions (State_T0) corresponding to State0.

[0106]

[0107] These are m game state descriptions generated as game state descriptions corresponding to State0, but with different sentence orders. The learning data generation unit 21 generates pairs of each generated game state description and an action description (Action_T0) generated based on the action data Action0 selected in the game state of State0.

[0108] Similarly, the learning data generation department 21 generates...

[0109]

[0110] These m game state descriptions are used as the game state descriptions corresponding to State1. The learning data generation unit 21 generates pairs of each generated game state description and the action description (Action_T1) generated based on the action data Action1 selected in the game state of State1.

[0111] Learning data generation unit 21 targets data other than the final game state (State) e For all game states except those mentioned above, m game state description texts are generated as corresponding game state description texts to the game state data. Pairs (text data) of the generated m game state description texts and corresponding action description texts are also generated. The learning data generation unit 21 generates pairs of game state description texts and action description texts as described above, and generates learning data containing these pairs (text data). However, the learning data generation unit 21 may also be configured to generate game state description texts corresponding to the game state data only for a portion of the game state data, and generate m game state description texts and corresponding action description texts.

[0112] In one example, the learning data generation unit 21 uses the shuffle function shown in equation (5) to shuffle the order of multiple sentences contained in the game state description.

[0113]

[0114] The shuffle function accepts the i-th game state statement, State_T. i And generate a shuffled State_T i The State_T of m patterns obtained from the sequence of elements within the context i In this implementation, the shuffle function generates a shuffled State_T. i The State_T of m patterns obtained by arranging the sentences within the text i .

[0115] Furthermore, when the game status description contains only one sentence, the learning device 10 can be configured to generate text data that consists only of the game status description and the action description.

[0116] The learning unit 22 generates a completed learning model based on the learning data generated by the learning data generation unit 21, for example, by performing machine learning using the learning data. In this embodiment, the learning unit 22 generates a completed learning model by having a pre-learned natural language learning model, which has pre-learned grammatical structures and relationships between articles related to natural language, learn learning data (training data) containing pairs of game state description texts and action description texts.

[0117] The natural language learning model is stored in a different device than the learning device 10. The learning device 10 communicates with this other device via the communication device 15, thereby enabling the natural language learning model to learn and obtain the learned model from the other device. However, the learning device 10 may also store the natural language learning model in the storage device 14.

[0118] Natural Language Processing (NLP) learning models are learning models generated by pre-learning a large number of natural language articles using the learning of grammatical structures and the learning of relationships between articles. The learning of grammatical structures refers to, for example, learning the structure of the sentence "My dog ​​is hairy" by learning three modes: (1) word masking "My dog ​​is [MASK]", (2) random word substitution "My dog ​​is apple", and (3) no word operation "My dog ​​is hairy". The learning of relationships between articles refers to, for example, when there are pairs (groups) of two consecutive sentences as learning objects, creating half of the original pairs of two sentences (correct pairs) and half of the randomly selected pairs of sentences (incorrect pairs), and learning whether there is a relationship between the sentences as a binary classification problem.

[0119] In one example, the pre-learned natural language model is a learned model called BERT provided by Google. The learning unit 22 communicates with the BERT system via the communication device 15, enabling BERT to learn from the learning data and obtain the generated learned model. In this case, the learning unit 22 uses the natural language data of the game state description and action description as learning data to fine-tune the pre-learned natural language model, thereby generating a learned model. Fine-tuning refers to re-learning the pre-learned natural language model to reweight the parameters. Therefore, in this case, the learning unit 22 generates a new learning model by re-learning the already learned pre-learned natural language model using the game state description and action description, resulting in a fine-tuned natural language model. In this embodiment, as described above, generating a learned model includes fine-tuning or reweighting the pre-learned and generated learned model to obtain the learned model.

[0120] In this embodiment, the learning unit 22 enables the natural language learning model to learn the relationships between articles. The processing of the learning data generation unit 21 in this embodiment will be further explained in conjunction with this.

[0121] As described above, the learning data generation unit 21 generates a first pair of game state description text and action description text corresponding to a pair of game state data and the data of the action selected in that game state, based on the game state data and action data contained in the replay log. In addition, the learning data generation unit 21 also generates a second pair of game state description text and action description text corresponding to a pair of game state data and the data of an action randomly selected from the actions available to the user in that game state and not included in the first pair. In this way, the learning data generation unit 21 generates the second pair in a different manner than the first and second pairs of action description text paired with the same game state description text. The learning data generation unit 21 generates learning data containing both the first and second pairs. In one example, the learning data generation unit 21 generates the first and second pairs for all game state data contained in the replay log acquired by the learning device 10, and generates learning data containing them.

[0122] As an example, the learning data generation unit 21 generates a State containing data as a game state. N The corresponding game state description text (State_T) N Processing of learning data in the case of ), the learning data generation unit 21 processes the State contained in the playback log. N and as in StateN The selected action data of the Action N To generate the corresponding game state description text (State_T) N ) and Action Description (Action_T) N The first pair. The learning data generation unit 21, based on the State contained in the replay log. N and from in State N Randomly selected from the available actions, excluding Action. N Data on actions other than those mentioned above is used to generate corresponding game state statements (State_T). N ) and Action Description (Action_T' N ) of the second pair.

[0123] As described above, the learning data generation unit 21 generates m game state descriptions for each mode to form a single game state description (State_T). N Therefore, m first pairs are generated for each game state description. Similarly, the learning data generation unit 21 generates m second pairs. For example, the first pair can be represented by equation (6).

[0124]

[0125] For example, the second pair can be represented by equation (7).

[0126]

[0127] In this way, the learning data generation unit 21 generates learning data containing the first pair and the second pair.

[0128] Learning Unit 22 sets the first pair of data as correct data, such as assigning "IsNext", and learns it, based on the pre-learned natural language model. It sets the second pair of data as incorrect data, such as assigning "NotNext", and learns it.

[0129] In one example, the learning unit 22 uses the `learn` function to enable the pre-learned model to learn from the training data. The `learn` function uses the first and second pairs of game state and action descriptions shown in equations (6) and (7) to fine-tune a pre-learned natural language model such as BERT. As a result of this fine-tuning, a pre-learned model (neural network model) is generated. Here, learning refers to updating the weights of the layers constituting the neural network by applying deep learning techniques.

[0130] Next, use Figure 6The flowchart shown illustrates the generation process of the learned model by the learning device 10 according to one embodiment of the present invention.

[0131] In step 101, the learning data generation unit 21 generates game state description text and action description text based on the game state data and action data contained in the replay log, and generates learning data containing pairs of game state description text and action description text corresponding to a game state and the action selected in that game state.

[0132] In step 102, the learning unit 22 generates a learned model based on the learning data generated by the learning data generation unit 21.

[0133] Figure 7 This is a block diagram illustrating the hardware structure of a decision device 50 according to one embodiment of the present invention. The decision device 50 includes a processor 51, an input device 52, a display device 53, a storage device 54, and a communication device 55. These components are connected via a bus 56. Furthermore, interfaces are provided between the bus 56 and each component as needed. The decision device 50 has a structure similar to that of a typical server, PC, etc.

[0134] Processor 51 controls the overall operation of decision-making device 50. Processor 51 is, for example, a CPU. Processor 51 performs various processes by reading and executing programs and data stored in storage device 54. Processor 51 can also be composed of multiple processors.

[0135] Input device 52 is a user interface for accepting input from the user to decision device 50, such as a touch panel, touch pad, keyboard, mouse, or button. Display device 53 is a monitor that displays application screens, etc., to the user of decision device 50 according to the control of processor 51.

[0136] Storage device 54 includes a main storage device and an auxiliary storage device. The main storage device is, for example, a semiconductor memory such as RAM. RAM is a volatile storage medium capable of high-speed information reading and writing, and is used as the storage area and working area when the processor 51 processes information. The main storage device may also include ROM as a dedicated non-volatile storage medium for reading. The auxiliary storage device stores various programs and data used by the processor 51 when executing each program. As long as the auxiliary storage device can store information, it can be any non-volatile storage device or non-volatile memory, and it can also be a removable storage device.

[0137] The communication device 55 exchanges data with user terminals or other computers such as servers via a network, for example, a wireless LAN module. The communication device 55 can be configured as a Bluetooth module or other wireless communication device or module, or as an Ethernet module, USB interface or other wired communication device or module.

[0138] Figure 8 This is a functional block diagram of a decision device 50 according to one embodiment of the present invention. The decision device 50 includes an inference data generation unit 61 and a decision unit 62. In this embodiment, these functions are implemented by the processor 11 executing a program stored in the storage device 54 or received via the communication device 55. Since various functions are implemented by reading programs in this way, a part or all of a part (function) may also be possessed by other parts. However, these functions may also be implemented in hardware by configuring circuits or the like to implement part or all of each function. In one example, the decision device 50 receives data on the game state as the prediction target from a game system such as a game AI, performs inference using a learned model generated by the learning device 10, and sends action data to the game system.

[0139] The inference data generation unit 61 generates inference data that is input to the learned model generated by the learning device 10, becoming the object of inference. The inference data generation unit 61 determines the actions the user can choose in the game state that is the object of prediction. Typically, the user can choose multiple actions. In one example, the inference data generation unit 61 determines the actions the user can choose based on the game state that is the object of prediction, such as based on the cards 41 placed in the game area 43 or the cards 41 in the user's hand. In other examples, the inference data generation unit 61 receives data on the game state that is the object of prediction from a game system such as a game AI, and receives the actions that the user can choose, determining the received actions as the actions the user can choose. In other examples, the actions the user can choose in a certain game state are predetermined by the game program, and the inference data generation unit 61 determines the actions the user can choose for each game state according to that game program.

[0140] In one example, the data generation unit 61 infers that receives game state data in the same data format as the replay log, and determines action data in the same data format as the replay log.

[0141] In each determined action, the inference data generation unit 61 generates pairs of game state descriptions and action descriptions based on pairs of game state data and action data. When predicting the action a user will choose in a game state that is the prediction target, the game state descriptions generated for each determined action and paired with the action descriptions are the same game state description. In one example, the inference data generation unit 61 uses the same rule-based system as the learning data generation unit 21 to generate pairs of game state descriptions and action descriptions based on pairs of game state data and action data. In this case, for example, the decision device 50 communicates with this rule-based system via the communication device 15, thereby enabling the transformation of game state data and action data into game state descriptions and action descriptions as CNL. Furthermore, the decision device 50 may also include this rule-based system.

[0142] The decision unit 62 uses pairs of game state descriptions and action descriptions generated by the inference data generation unit 61 and the learned model generated by the learning device 10 to determine the action predicted to be chosen by the user. For example, the data for the game state as the prediction target is State. α Furthermore, the data corresponding to the actions that the user can choose in this game state are respectively...

[0143]

[0144] This section explains the situation regarding game state data. α The corresponding game state description text is State_T α The action descriptions corresponding to the action data are as follows:

[0145] .

[0146] Inference is used to generate State_T using data generation unit 61. α and

[0147]

[0148] Each of the pairs in the list.

[0149] The decision unit 62 inputs the pairs generated by the inference data generation unit 61 into the learned model generated by the learning device 10, and calculates a score indicating whether an action is feasible for the user. Based on the calculated score, the decision unit 62 determines the action corresponding to an action description. In one example, the decision unit 62 determines the action corresponding to the action description of the pair with the highest score and sends information related to the determined action to the game system, which receives data on the game state as the prediction target.

[0150] In one example, the learned model generated by the learning device 10 is equipped with the infer function shown in equation (8).

[0151]

[0152] The infer function receives the game state description (State_T) corresponding to the game state being predicted from the decision unit 62. α ), and a list of action descriptions corresponding to the actions the user can choose in this game state.

[0153] .

[0154] The `infer` function assigns a score of 0 to 1 to each action description (or action), indicating whether it is the next action to be taken, and outputs a pair of action descriptions (or actions) and scores. For example, with this score, 0 indicates the least desirable action, and 1 indicates the most desirable action.

[0155] In one example, decision unit 62 uses the `select` function to select the action predicted the user will choose. The `select` function determines the action description or its corresponding action based on the action description and score pairs output by the `infer` function. The `select` function is configured to select the action corresponding to the action description of the pair with the highest score. However, the `select` function can also be configured to select the action corresponding to the action description of the second-highest score pair, the third-highest score pair, and so on.

[0156] Next, use Figure 9 The flowchart shown illustrates the decision processing of a decision device 50 in one embodiment of the present invention, which determines the action predicted to be chosen by the user.

[0157] In step 201, the inference data generation unit 61 determines the actions that the user can choose in the game state that is the subject of prediction.

[0158] In step 202, the inference data generation unit 61 transforms the pairs of game state data and action data into CNL to generate pairs of game state description text and action description text for each action determined in step 201.

[0159] In step 203, the decision unit 62 uses the pairs of game state description text and action description text generated in step 202, as well as the learning completion model generated by the learning device 10, to determine the action predicted to be chosen by the user.

[0160] Next, the main effects of the learning device 10 and the decision device 50 according to the embodiments of the present invention will be explained.

[0161] In this embodiment, the learning device 10 transforms pairs of game state and action data contained in the replay logs stored on the game server into pairs of game state description text and action description text as CNL (Content-Based Language), and generates learning data containing the transformed text data. The learning device 10 generates a first pair and a second pair, and generates learning data containing the first and second pairs. The first pair is a pair of game state description text and action description text generated based on the replay logs, and the second pair is obtained by pairing the game state description text with an action description text that is different from the action description text in the first pair and is randomly selected from the actions that the user can choose in the game state corresponding to the same game state description text as the first pair. The first pair in the learning data contains multiple patterns of game state description text obtained by shuffling the order of sentences contained in the game state description text for each game state, and contains pairs of game state description text and action description text for each game state. The second pair included in the learning data also contains the same game state description text as the first pair for each game state, and for each game state, it contains a pair of game state description text and action description text (action description text different from the first pair). The learning device 10 generates a learned model by having a pre-learned natural language model learn the generated learning data.

[0162] In this embodiment, the decision device 50 receives game state data, which is the subject of prediction, from a game system such as a game AI, and determines multiple actions that the user can choose under the predicted game state. For each determined action, the decision device 50 transforms the pair of game state data and action data into a pair of game state description text and action description text. Using the transformed pairs and the learned model generated by the learning device 10, the decision device 50 determines the action predicted that the user will choose.

[0163] In this embodiment, as a learning phase, the replay logs of non-natural language data stored on the game server are converted into natural language data and used as input to learn using a converter neural network technique capable of natural language processing, thereby generating a learned model. Until now, replay logs have not been converted into natural language data in a manner as in this embodiment. In this embodiment, a converter neural network-based natural language processing technique is used as an example of a distributed representation model with high contextual representation capabilities, enabling learning from contextual replay logs (such as card game battle records). Furthermore, the distributed representation of words represents the co-occurrence relationships between words in sentences and paragraphs as vectors, which can be applied to a wide range of tasks such as article summarization, translation, and dialogue. Moreover, as in this embodiment, by learning the relationship between the game state and actions at each moment as a next sentence prediction, strategic thinking can be obtained through converter neural network-based natural language processing techniques. Furthermore, instead of converting the replay logs into natural language, the same effect can be achieved by transforming them into text data in a form suitable for transforming the replay log machine into a distributed representation.

[0164] In addition, in this embodiment, when converting the replay logs into natural language, a natural language with certain limitations, such as CNL, is used to transform them into articles with low ambiguity, thereby generating more appropriate learning data.

[0165] Furthermore, in this embodiment, when generating the first pair of game state descriptions and action descriptions, multiple patterns are generated by randomly rearranging the sentences contained in the game state description. Since the game state description is a sentence used to describe the current game state, its order is meaningless. On the other hand, natural language processing techniques based on converter neural networks learn the combination rules of words and word sequences, enabling direct learning of dialogue interactions (actions) based on specific grammar (rules) and specific context (game state) of card games. By shuffling the sentences in the game state description, the relationships between the sentences in the game state description, i.e., the elements of the game state, and the action description (action) can be learned as a distributed representation, independent of their position in the game state description. Moreover, in this embodiment, since the card description is also explained in natural language along with the card name, even new cards can be autonomously positioned.

[0166] In this embodiment, as an inference phase, by converting game state data, etc., into natural language (CNL) and inputting it into the fully learned model (transformer neural network model), inference that effectively utilizes the representational capabilities of the distributed representation model can be achieved. For example, when making the AI ​​play a game, the decision device 50 can input the game state and the set of actions that can be taken in that game state into the fully learned model, and select the next action based on the result and input it into the game. In this case, the action determined by the decision device 50 is an action performed by the AI ​​considering the actions predicted by the fully learned model as the actions the user would choose. Furthermore, for example, when making the AI ​​play a game, the decision device 50 can be configured to select the second to third highest-scoring actions, actions near the median value, rather than the highest-scoring action. This allows for adjustment of the AI's strength.

[0167] Furthermore, the learning method of this embodiment can be widely applied to turn-based battle games, expanding AI that mimics human gaming tendencies to various types. Additionally, the method of generating a fully learned model using fine-tuning, as an example of this embodiment, is suitable for games with continuously increasing replay logs and is well-suited for long-term gameplay. Moreover, the fully learned model generated in this embodiment explains card descriptions along with card names in natural language, enabling highly accurate inferences even for newly released cards. Furthermore, the method for generating the fully learned model in this embodiment does not rely on specific converter neural network techniques or fine-tuning methods, and can utilize any natural language learning system based on a converter neural network corresponding to the learning of the next sentence prediction. Therefore, it is possible to switch natural language learning systems when more accurate neural network-based systems become available, or depending on the support of external libraries.

[0168] Unless otherwise specified, the above-described effects are the same in other implementation methods and embodiments.

[0169] As embodiments of the present invention, it can be configured as an apparatus or system comprising only the learning device 10, or as an apparatus or system comprising both the learning device 10 and the decision device 50. In other embodiments of the present invention, it can be configured as a method or program for implementing the functions of the embodiments of the present invention described above, the information processing method shown in the flowchart, or a computer-readable storage medium storing the program. Alternatively, in other embodiments of the present invention, it can be configured as a server capable of supplying the program to a computer. Furthermore, in other embodiments, it can be configured as a system or virtual machine for implementing the functions of the embodiments of the present invention described above, the information processing method shown in the flowchart, or a virtual machine.

[0170] In embodiments of the present invention, the game state description text and action description text generated by the learning data generation unit 21 based on the game state data and action data are examples of game state text and action text, respectively, which are text data expressed in a prescribed form. Similarly, the game state description text and action description text generated by the inference data generation unit 61 based on the game state data and action data are also examples of game state text and action text, respectively, which are text data expressed in a prescribed form. The text data expressed in a prescribed form is text data that is readable by both machines and humans, for example, text data expressed in a form suitable for machine transformation into a distributed representation. The game state text corresponding to a game state includes multiple element texts, each element text corresponding to each element included in the game state, such as the data of each card included in the game state. An element text can be a sentence, a clause, or a phrase. The sentences included in the game state description text are examples of element texts included in the game state text. In embodiments of the present invention, it is also possible to configure each phrase included in the game state description text to correspond to each element included in the game state.

[0171] In an embodiment of the present invention, the natural language pre-learning model learned by the learning unit 22 using training data is an example of a deep learning model aimed at learning data arranged in sequence.

[0172] In embodiments of the present invention, CNL can also be set to a language other than English, such as Japanese.

[0173] In one variation, the learning device 10 does not use natural language to pre-learn the model, i.e., it does not perform fine-tuning, but uses the learning data generated by the learning device 10 to construct (generate) the learned model.

[0174] In one variation, the decision device 50 is configured to store the learned model generated by the learning device 10 into the storage device 54 and perform inference and decision processing without communication.

[0175] In a variation, each card i It contains only the name, not the explanation. In this variation, as long as the card itself (name) can be transformed into a word, the semantic distance relationship between cards can also be learned. In this case, for example, the encode function accepts the data State of the i-th game state. i and use the accepted State i The names of each card within the state and a rule-based system are used to define the state. iTransformed into controlled natural language data State_T in a prescribed form i .

[0176] In the processes or actions described above, as long as no contradiction arises such as using data that should not be available in a certain step, the processes or actions can be freely modified. Furthermore, the embodiments described above are illustrative of the invention, and the invention is not limited to these embodiments. The invention can be implemented in various ways without departing from its spirit.

[0177] Explanation of reference numerals in the attached figures

[0178] 10: Learning device; 11: Processor; 12: Input device; 13: Display device; 14: Storage device; 15: Communication device; 16: Bus; 21: Learning data generation unit; 22: Learning unit; 40: Game screen; 41: Cards; 42: First card set; 43: Game field; 44: Second card set; 45: Character; 50: Decision device; 51: Processor; 52: Input device; 53: Display device; 54: Storage device; 55: Communication device; 56: Bus; 61: Deduction data generation unit; 62: Decision unit.

Claims

1. A method for generating a learned model, the learned model being used to predict a user's next action in a game that progresses and updates the game state in accordance with actions selected by a user, the method comprising the steps of: Based on the game state and action data contained in the game-related historical data, controllable natural language data, namely game state text and action text, is generated in a prescribed form, and learning data containing pairs of game state text and action text corresponding to a game state and the action selected in that game state is generated. as well as A fully learned model is generated by enabling a neural network model capable of learning rules for combining words or word sequences to learn the generated learning data. The historical data has a data structure consisting of pairs of data for a game state other than the final game state and data for the actions selected in that game state.

2. The method according to claim 1, wherein, The steps for generating the learning data include the following: generating multiple game state texts with different arrangements of multiple element texts contained in the game state text as game state texts corresponding to a game state, and generating learning data containing each game state text in the generated multiple game state texts and the action text corresponding to the action selected in the game state.

3. The method according to claim 1 or 2, wherein, The steps for generating the learned model include the following: generating the learned model by having a pre-learned natural language learning model, which has learned grammatical structures and relationships between articles related to natural language, learn the generated learning data.

4. The method according to claim 1 or 2, wherein, The step of generating the learning data includes the following steps: generating learning data containing a first pair and a second pair, wherein the first pair is a pair of game state text and action text corresponding to a game state and the action selected in that game state, generated based on the game state and action data contained in the historical data; and the second pair is a pair of game state text and action text corresponding to an action text randomly selected from the actions available to the user in that game state and not included in the first pair. The steps for generating the learned model include the following: learning the first pair as correct data and learning the second pair as incorrect data to generate the learned model.

5. The method according to claim 1 or 2, wherein, The steps for generating the learning data include the following: generating game state text and action text using grammar, syntax, and lexical representations suitable for machine transformation into distributed representations, based on pre-created rule-based systems, game state data, and action data.

6. A method for determining an action predicted to be chosen by the user in a game that progresses and updates the game state in accordance with actions selected by the user, the method comprising the steps of: Determines the multiple actions a user can choose in the game state that is being predicted; For each of the determined actions, pairs of game state text and action text are generated based on the pairs of game state data and action data; and Using the generated pairs of game state texts and action texts, and the learned model according to any one of claims 1 to 5, the action predicted to be chosen by the user is determined.

7. A computer program product comprising a program that causes a computer to perform the steps of the method according to any one of claims 1 to 6.

8. A system for generating a learned model, the learned model being used to predict the action a user will choose in a game that progresses and updates the game state in accordance with actions selected by a user, the system performing the following processing: Based on game-related historical data containing game state and action data, controlled natural language data, namely game state text and action text, is generated in a prescribed format. Learning data is also generated, containing pairs of game state text and action text corresponding to a game state and the action selected in that game state. A fully learned model is generated by enabling a neural network model capable of learning rules for combining words or word sequences to learn the generated learning data. in, The historical data has a data structure consisting of pairs of data for a game state other than the final game state and data for the actions selected in that game state.

9. A system for determining an action predicted to be chosen by the user in a game that progresses and updates the game state in accordance with actions selected by the user, the system performing the following processing: Determines the multiple actions a user can choose in the game state that is being predicted; For each of the determined actions, pairs of game state text and action text are generated based on the pairs of game state data and action data; and Using the generated pairs of game state texts and action texts, and the learned model according to claim 8, the action predicted to be chosen by the user is determined.

10. A computer-readable storage medium storing a program that causes a computer to perform the steps of the method according to any one of claims 1 to 6.

Images