Information processing method, information processing device, and program

Abstract

Provided are an information processing method and the like that can provide an appropriate game experience to each user.　The information processing method acquires status information representing a play status of a game played by a user, inputs, then, when the status information is input, the acquired status information is input to a first model (51) that has finished learning so as to output act information representing an act to have in the game, to thereby output the act information, and calculates reward on the basis of the act information and the status information after performing the act represented by the act information. A computer then executes a processing of updating the first model (51) for each of the user on the basis of the reward.

Description

[Technical field]

[0001] The present invention relates to an information processing method, an information processing device, and a program. [Background technology]

[0002] With the advancement of artificial intelligence technology, various systems that apply artificial intelligence to electronic games have been proposed. For example, Patent Document 1 discloses an information processing device that uses a reinforcement learning technique to make an artificial intelligence agent that plays an electronic game learn a strategy regarding the use of objects in the game. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-197592 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the invention disclosed in Patent Document 1 does not perform agent learning for each user who plays the game.

[0005] In one aspect, an object of the present invention is to provide an information processing method etc. that can provide a gaming experience that is suited to each user. [Means for solving the problem]

[0006] An information processing method according to one aspect includes acquiring state information representing a play state of a game played by a user, inputting the acquired state information into a first model that has been trained to output action information representing an action to be taken in the game when the state information is input, and outputting the action information, calculating a reward based on the action information and the state information after the action represented by the action information has been performed, and updating the first model for each user based on the reward. Then, difference information before and after the update of each of the first models updated for each of the users is acquired, and the first models are updated based on the difference information.The processing is carried out by a computer. Effect of the Invention

[0007] In one aspect, a gaming experience suited to each user can be provided. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram showing an example of the configuration of a game system. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of a federated server. [Diagram 3] FIG. 2 is a block diagram showing an example of the configuration of an intermediate server. [Figure 4] FIG. 2 is a block diagram showing a configuration example of a terminal. [Diagram 5] FIG. 11 is an explanatory diagram showing an example of the record layout of a user DB, a play log DB, a dialogue log DB, an opening sentence table, a conversion table, and a scenario table. [Figure 6] FIG. 1 is a functional block diagram showing an example of the configuration of a game system. [Figure 7] An explanatory diagram regarding the battle mode or the cooperative mode. [Figure 8] FIG. 11 is an explanatory diagram regarding an escort mode. [Figure 9] FIG. 11 is an explanatory diagram regarding a free talk mode. [Figure 10] 11 is a flowchart showing a procedure for generating a first model. [Figure 11] 13 is a flowchart showing a procedure for generating a second model. [Figure 12] 13 is a flowchart showing a process performed by an intermediate server. [Figure 13] 13 is a flowchart showing the processing steps of a subroutine in a battle mode or a cooperative mode. [Figure 14] 13 is a flowchart showing a processing procedure of a subroutine in an escort mode. [Figure 15] 13 is a flowchart showing a processing procedure of a subroutine in a free talk mode. [Figure 16] 13 is a flowchart showing a procedure for updating a first model in the federated server. [Figure 17] FIG. 11 is a functional block diagram showing a configuration example of a game system according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention will now be described in detail with reference to the drawings showing embodiments thereof. (Embodiment 1) 1 is an explanatory diagram showing a configuration example of a game system. In this embodiment, a game system is described in which a machine learning model (a first model 51 and a second model 52 described later) that has been trained for each user is used to control the actions of a second player who competes against or cooperates with the user in a game, or to give advice regarding the game.

[0010] This system provides an artificial intelligence agent function that controls the actions of a second player who competes against or cooperates with the user in a game, and a chatbot function that allows a conversation with a predetermined character regarding the game, etc. This system uses state information that indicates the play state of the game played by the user as input, and uses a first model 51 (e.g., a deep reinforcement learning model) that has been trained to output action information that indicates the action to be taken in the game to infer the action to be selected as the second player and control the action of the second player (see FIG. 7). In addition, this system uses a second model 52 (e.g., a deep learning model) that has been trained to generate an output sentence following an arbitrary input sentence, and generates text indicating a strategy for the game, etc., based on the action information output from the first model 51, and outputs it as a speech sentence of the character who is the conversation partner (see FIG. 8). In addition, this system is provided with a free talk mode in which the user can have an arbitrary conversation with the character regardless of the game.

[0011] The game may be, for example, an action game, a fighting game, etc., but the type of game is not particularly limited. As described later, in this system, by inputting the game screen of the game being played by the user to the first model 51 as state information, it is possible to target any type of game.

[0012] The game system includes a federated server 1, intermediate servers 2, 2, 2 ..., terminals 3, 3, 3, and game machines 4, 4, 4 .... Each device is communicatively connected to a network such as the Internet.

[0013] The federated server 1 is an information processing device capable of various information processing and information transmission and reception, and is a server computer that performs integrated management of this system. Note that a device equivalent to the federated server 1 may be a personal computer or the like, rather than a server computer. The federated server 1 functions as a generating device that performs machine learning to generate a first model 51 and a second model 52. The federated server 1 transmits data of the first model 51 and the second model 52 to the intermediate server 2, and causes processing using each model to be performed.

[0014] The intermediate server 2 is an information processing device capable of various information processing and sending and receiving information, and is a server computer located between the federated server 1 and the users. Note that the device equivalent to the intermediate server 2 may not be a server computer, but may be a personal computer or the like. As will be described later, the intermediate server 2 uses each model acquired from the federated server 1 to control the actions of the second player and generate text for the chatbot. For example, multiple intermediate servers 2 are prepared according to the nationality, region, etc. of the users.

[0015] In this embodiment, the federated server 1 and the intermediate server 2 are described as separate computers, but the federated server 1 and the intermediate server 2 may be integrated into a single computer.

[0016] The terminal 3 is an information processing terminal used by each user, and may be, for example, a smartphone, a tablet terminal, a personal computer, or the like. The terminal 3 transmits information about the game being played by the user on the game machine 4 to the intermediate server 2. The terminal 3 also obtains information for controlling the actions of the second player in the game from the intermediate server 2, and outputs the information to the game machine 4. The terminal 3 also displays a character object corresponding to the user's conversation partner (chatbot), and displays text generated by the intermediate server 2 as the character's dialogue, or outputs the text as audio.

[0017] Game machine 4 is, for example, a home game machine, and is a terminal device that executes processes related to the game played by the user. For example, game machine 4 is paired with terminal 3, and transfers information about the game being played to terminal 3. Terminal 3 transmits the information transferred from game machine 4 to intermediate server 2.

[0018] In this embodiment, the terminal 3 and the game machine 4 are described as separate devices, but the distinction between the two is for convenience, and may refer to, for example, an application game installed in the terminal 3. Furthermore, the game machine 4 is not limited to a home game machine, and may be, for example, an arcade game machine or other game machine.

[0019] As described above, this system provides a function for controlling the actions of the second player or a chatbot function using a machine learning model. In this embodiment, the model is updated (relearned) for each user, thereby improving the actions of the second player and the character's speech, etc., to suit each user. Specifically, the intermediate server 2 updates the first model 51 for each user by having the first model 51 learn the game play log of each user (actions selected when fighting or cooperating with a user, and the game state that changed based on those actions). In this way, an artificial intelligence agent that takes into account the characteristics of each user is constructed.

[0020] 2 is a block diagram showing an example of the configuration of the federated server 1. The federated server 1 includes a control unit 11, a main memory unit 12, a communication unit 13, and an auxiliary memory unit . The control unit 11 is an arithmetic processing device such as one or more CPUs (Central Processing Units), MPUs (Micro-Processing Units), GPUs (Graphics Processing Units), etc., and performs various information processing, control processing, etc. by reading and executing a program P1 stored in the auxiliary storage unit 14. The main storage unit 12 is a temporary storage area such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), a flash memory, etc., and temporarily stores data required for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing processing related to communication, and transmits and receives information to and from the outside.

[0021] The auxiliary storage unit 14 is a non-volatile storage area such as a large-capacity memory or a hard disk, and stores the program P1 and other data necessary for the control unit 11 to execute processing. The auxiliary storage unit 14 also stores a first model 51 and a second model 52. The first model 51 is a machine learning model constructed by deep reinforcement learning, and is a model that receives state information representing the play state of a game played by a user as input and outputs action information indicating an action to be taken in the game. The second model 52 is a machine learning model constructed by deep learning, and is a model that receives an arbitrary input sentence as input and generates an output sentence following the input sentence. The first model 51 and the second model 52 are expected to be used as software modules constituting artificial intelligence software.

[0022] The auxiliary storage unit 14 may be an external storage device connected to the federated server 1. The federated server 1 may be a multi-computer consisting of a plurality of computers, or may be a virtual machine virtually constructed by software.

[0023] In the present embodiment, the federated server 1 is not limited to the above configuration, and may include, for example, an input unit for accepting operation input, a display unit for displaying images, etc. The federated server 1 may also include a reading unit for reading portable storage medium 1a such as a CD (Compact Disk)-ROM or a DVD (Digital Versatile Disc)-ROM, and may read and execute the program P1 from the portable storage medium 1a. Alternatively, the federated server 1 may read the program P1 from a semiconductor memory 1b.

[0024] 3 is a block diagram showing an example of the configuration of the intermediate server 2. The intermediate server 2 includes a control unit 21, a main memory unit 22, a communication unit 23, and an auxiliary memory unit 24. The control unit 21 is an arithmetic processing device such as a CPU, and performs various information processing, control processing, etc. by reading and executing a program P2 stored in the auxiliary storage unit 24. The main storage unit 22 is a temporary storage area such as a RAM, and temporarily stores data necessary for the control unit 21 to execute arithmetic processing. The communication unit 23 is a communication module for performing processing related to communication, and transmits and receives information to and from the outside.

[0025] The auxiliary storage unit 24 is a non-volatile storage area such as a large capacity memory or a hard disk, and stores a program P2 and other data necessary for the control unit 21 to execute processing. The auxiliary storage unit 24 also stores a first model 51, a second model 52, a user DB 241, a play log DB 242, a dialogue log DB 243, an opening sentence table 244, a conversion table 245, and a scenario table 246.

[0026] The user DB 241 is a database that stores information about each user who uses the system. The play log DB 242 is a database that stores game play logs (status information and a history of behavioral information output based on the status information) of each user. The dialogue log DB 243 is a database that stores a dialogue history between a user and a character. The opening sentence table 244 is a table that stores text (opening sentences) corresponding to behavioral information output by the first model 51. The conversion table 245 is a table for converting a part of the text (e.g., endings) that is output as a spoken sentence by a character. The scenario table 246 is a table that stores a scenario (output sentence) to be developed according to the content of an input sentence input by a user.

[0027] The auxiliary storage unit 24 may be an external storage device connected to the intermediate server 2. The intermediate server 2 may be a multi-computer consisting of multiple computers, or may be a virtual machine virtually constructed by software.

[0028] The intermediate server 2 may also include a reading unit that reads a portable storage medium 2a such as a CD-ROM, and may read and execute the program P2 from the portable storage medium 2a, or the intermediate server 2 may read the program P2 from the semiconductor memory 2b.

[0029] 4 is a block diagram showing an example of the configuration of the terminal 3. The terminal 3 includes a control unit 31, a main memory unit 32, a communication unit 33, a display unit 34, an input unit 35, a voice output unit 36, and an auxiliary memory unit 37. The control unit 31 is an arithmetic processing device such as a CPU, and performs various information processing, control processing, etc. by reading and executing the program P3 stored in the auxiliary storage unit 37. The main storage unit 32 is a temporary storage area such as a RAM, and temporarily stores data necessary for the control unit 31 to execute arithmetic processing. The communication unit 33 is a communication module for performing processing related to communication, and transmits and receives information to and from the outside. The display unit 34 is a display screen such as a liquid crystal display, and displays images. The input unit 35 is an operation interface such as a mechanical key, and accepts operation input from the user. The audio output unit 36 ​​is a speaker that outputs audio, and outputs audio provided by the control unit 31. The auxiliary storage unit 37 is a non-volatile storage area such as a hard disk or a large-capacity memory, and stores the program P3 and other data necessary for the control unit 31 to execute processing.

[0030] The terminal 3 may include a reading unit for reading a portable storage medium 3a such as a CD-ROM, and may read and execute the program P3 from the portable storage medium 3a. Alternatively, the terminal 3 may read the program P3 from the semiconductor memory 3b.

[0031] FIG. 5 is an explanatory diagram showing an example of the record layout of the user DB 241, the play log DB 242, the dialogue log DB 243, the opening sentence table 244, the conversion table 245, and the scenario table 246. As shown in FIG. The user DB 241 includes a user ID column, a name column, a user attribute column, a play attribute column, a character column, and a friend column. The user ID column stores a user ID for identifying each user. The name column, the user attribute column, the play attribute column, the character column, and the friend column store, in association with the user ID, the user's name, user attribute (e.g., age, sex, place of residence, hobbies, etc.), play attribute, the type of character set as a conversation partner, and the user ID of another user registered as a friend. The play attribute column stores, for example, the user's play type in the game (for example, information representing a play style, such as "vanguard type" in a shooting game).

[0032] The play log DB 242 includes a date column, a user column, a game column, a category column, and a play log column. The date column stores the date on which the user played the game. The user column, the game column, the category column, and the play log column store, in association with the date, the user ID of the user who played the game, the type of game played (e.g., game title), the game category (e.g., whether it was an action game or a fighting game), and the play log. The play log column stores state information (e.g., game screen) of the game being played by the user, and a history of action information (e.g., operation commands) output from the first model 51 based on the state information.

[0033] The dialogue log DB 243 includes a date and time column, a user column, an input sentence column, and an output sentence column. The date and time column stores the date and time when a dialogue with a character took place in a free talk mode described below. The user column, the input sentence column, and the output sentence column store, in association with the date and time, the user ID of the user who conducted the dialogue, the input sentence input by the user, and the output sentence output as a response to the input sentence.

[0034] The opening sentence table 244 stores a game sequence, an action sequence, and an opening sentence sequence. The game sequence stores the type of game (game title). The action sequence and the opening sentence sequence each store, in association with the type of game, action information that can be output from the first model 51, and text (opening sentence) corresponding to the action information. As described below, the intermediate server 2 selects an opening sentence (input sentence) corresponding to the action information output from the first model 51 from the opening sentence table 244, and inputs the selected opening sentence into the second model 52 to generate a subsequent sentence (output sentence), thereby completing the text consisting of the opening sentence and the subsequent sentence, and outputs it as the character's dialogue.

[0035] The conversion table 245 includes a character string, an emotion string, an intimacy string, a pre-conversion string, and a post-conversion string. The character string stores the type of character. The emotion string, intimacy string, pre-conversion string, and post-conversion string each store, in association with the type of character, the user's emotion (e.g., positive or negative), the intimacy between the user and the character, the ending of the output sentence before conversion, and the ending after conversion. As described below, the intermediate server 2 converts the ending of the output sentence that is output as the character's dialogue based on the user's emotion, the intimacy between the user and the character, etc.

[0036] The scenario table 246 includes a keyword string and an output sentence string. The keyword string stores the keyword of the input sentence that should be the trigger for starting the scenario. The output sentence string stores a template of the output sentence according to the scenario in association with the keyword. As will be described later, when the input sentence entered by the user includes a keyword, the intermediate server 2 generates the output sentence based on a rule, rather than using the second model 52.

[0037] 6 is a functional block diagram showing an example of the configuration of a game system. In FIG. 6, functional blocks of a federated server 1 and an intermediate server 2 are shown.

[0038] Before describing the functional configuration of this system, the functional modes provided in this system will be described. This system provides a battle mode, a joint battle mode, an escort mode, and a free talk mode (see Figs. 7 to 9).

[0039] The battle mode is a mode in which an artificial intelligence agent controls an opponent (second player) who competes against the user. The cooperative mode is a mode in which an artificial intelligence agent controls an allied player (second player) when the user cooperates with another player, such as in a game in which multiple teams fight against each other. In this system, a first model 51 is prepared that has learned the actions that the agent should take in the game by deep reinforcement learning. In the battle mode and cooperative mode, the first model 51 is used to select the actions of the second player and control the movement.

[0040] The escort mode is a mode in which the user receives advice such as strategies from a character regarding the game he is playing. In this system, in addition to the first model 51, a second model 52 is prepared for generating text to be output as dialogue for the chatbot. In the escort mode, text such as strategies is generated using the second model 52 based on the action to be taken in the game selected by the first model 51, and is output as dialogue from the character.

[0041] In addition, a free talk mode is provided in which the user can freely converse with the character. The mode to be used is selected at the terminal 3, and the intermediate server 2 executes the process of the selected mode.

[0042] Next, the functional configuration of this system will be described. The control unit 11 of the federated server 1 executes the program P1 to function as a first model generation unit 101 and a second model generation unit 102. The first model generation unit 101 performs machine learning to learn a game play log and generates a first model 51. The first model 51 is a model that receives state information indicating the play state of a game played by a user as input and outputs action information indicating an action to be taken in the game. Specifically, the first model 51 is a deep reinforcement learning model such as DQN (Deep Q-network).

[0043] Reinforcement learning is a technique that teaches an artificial intelligence agent to select the optimal action a to achieve a goal when in state s in a certain environment. In reinforcement learning, if the agent selects action a in state s at the current time t, the agent is given a reward r (reword) according to the transition (change) to state s at the next time t + Δt, and the action value function Q(s, a) is updated so that the sum of rewards r is maximized. The agent selects action a with a high action value function Q(s, a). In a deep reinforcement learning model, the calculation of the action value function Q(s, a) corresponds to the calculation in a neural network.

[0044] The federated server 1 performs deep reinforcement learning and generates a deep reinforcement learning model as a first model 51 that outputs the next action a (action information) to be taken when the game being played by the user is in a certain state s (state information).

[0045] The state information input to the first model 51 may be any data that correctly represents the play state, but in this embodiment, the game screen displayed on the display of the game machine 4 is used as the state information. As described below, the intermediate server 2 uses a known remote control means to acquire the game screen displayed on the display of the game machine 4 via the terminal 3, and inputs it to the first model 51. The first model 51 accepts the game screen as state information, and outputs behavior information. In this way, by inputting appearance information in the form of the game screen, rather than internal parameters specific to each type of game (software), it becomes possible to infer behavior information for any title of game.

[0046] In addition to the game screen, the status information may include data such as the type of game (game title) and category that the user is playing. By accepting data such as the type and category of the game as input, the first model 51 can output behavioral information suitable for each title or genre.

[0047] In addition, in this embodiment, a game screen is mainly used as the status information, but instead of or in addition to the game screen, internal parameters specific to each game, operation commands input by the user to the controller of the game machine 4, etc. may be used.

[0048] The behavioral information output by the first model 51 may be any data that correctly represents the behavior to be taken in the game, but in this embodiment, the first model 51 outputs, as the behavioral information, an operation command to be input to the controller of the game machine 4. That is, the first model 51 infers the next operation command to be input according to the current play state of the game.

[0049] For example, the federated server 1, like the process executed by the intermediate server 2 described below, remotely accesses the game machine 4 to acquire a game screen, inputs it to the first model 51, and outputs behavioral information. The federated server 1 transmits the behavioral information (operation commands) output from the first model 51 to the game machine 4, and controls the actions of the second player. The federated server 1 then calculates a reward based on the game screen that has changed as a result of the action of the second player, i.e., based on state information, and updates the action value function. In this way, the federated server 1 generates the first model 51.

[0050] As described below, the alliance server 1 may generate a first model 51 for the battle mode or the cooperative mode and a first model 51 for the escort mode separately.

[0051] The second model generation unit 102 performs machine learning to learn predetermined training data, and generates the second model 52. The second model 52 is a model that, when an arbitrary input sentence is input, generates an output sentence following the input sentence. Specifically, the second model 52 is a neural network generated by deep learning, and is a Transformer model such as GPT-2 or BERT (Bidirectional Encoder Representations from Transformers).

[0052] Transformer is a type of encoder-decoder model, and is equipped with an encoder that accepts input sentences and a decoder that generates (outputs) output sentences. The encoder and decoder include a module that calculates a score called Attention, and calculates which words in a sentence should be focused on. The Transformer understands the structure of the input sentence while focusing on the words in the sentence, and generates an output sentence that follows the input sentence.

[0053] The federated server 1 generates the second model 52 using training data in which a correct output sentence is associated with an input sentence for training. The training data may be a text created manually, or may be a dialogue text that a user actually has with a chatbot through this system. The federated server 1 inputs the training input sentence into the second model 52 and generates an output sentence. The federated server 1 compares the output sentence generated by the second model 52 with the correct output sentence, and optimizes parameters such as the weight between neurons so that the two are similar. In this way, the federated server 1 generates the second model 52, which takes an arbitrary input sentence as input and generates an output sentence following the input sentence.

[0054] As described later, the federated server 1 may generate the second model 52 for escort and the second model 52 for free talk separately.

[0055] The federated server 1 transmits the generated data of the first model 51 and the second model 52 to the intermediate server 2, which installs them. The intermediate server 2 uses the first model 51 and the second model 52 acquired from the federated server 1 to perform the following processes.

[0056] The control unit 21 of the intermediate server 2 executes the program P2, and thereby functions as a state information acquisition unit 201, a behavior information output unit 202, a second player control unit 203, a first model update unit 204, a text generation unit 205, a character control unit 206, and a text input unit 207.

[0057] The state information acquisition unit 201 acquires game state information from the terminal 3. As described above, the state information is a game screen or the like displayed on the display of the game machine 4. The state information acquisition unit 201 acquires the game screen from the game machine 4 via the terminal 3 using a known remote control means, and transfers it to the behavior information output unit 202.

[0058] In this embodiment, the intermediate server 2 transmits and receives information via the terminal 3, but it goes without saying that the intermediate server 2 may transmit and receive information directly to and from the game machine 4.

[0059] The behavior information output unit 202 inputs the state information acquired by the state information acquisition unit 201 to the first model 51 and outputs the behavior information. As described above, the behavior information is information indicating an action to be taken in the game, for example, an operation command of the game. The behavior information output unit 202 passes the behavior information acquired from the first model 51 to the second player control unit 203.

[0060] In this case, the behavior information output unit 202 selects the first model 51 of the user corresponding to the acquisition source of the state information from among the first models 51, 51, 51 ... that have learned the game play logs for each user, and inputs the state information. As described later, the first model update unit 204 learns the game play logs (behavior information output by the first model 51 and state information after the second player acts based on the behavior information) for each user and updates the first model 51 individually. The behavior information output unit 202 selects the first model 51 corresponding to the user, inputs the state information, and outputs the behavior information.

[0061] The second player control unit 203 controls the actions of the second player who competes against or cooperates with the user when a battle mode or a cooperative mode is selected in the terminal 3. Specifically, the second player control unit 203 outputs the action information acquired from the action information output unit 202 to the game machine 4 via the terminal 3, and controls the action of the second player in the game.

[0062] Fig. 7 is an explanatory diagram relating to the battle mode or the joint battle mode. Fig. 7 conceptually illustrates a state in which a first player 1P operated by a user battles or joins forces with a second player 2P whose actions are controlled by the intermediate server 2.

[0063] As described above, the intermediate server 2 acquires state information (game screen) from the game machine 4 via the terminal 3 and inputs it to the first model 51. In this case, the intermediate server 2 inputs the state information to the first model 51 that corresponds to the user from which the state information was acquired, out of the first models 51, 51, 51 ... that are updated for each user, and outputs the behavior information. The intermediate server 2 outputs the behavior information to the game machine 4 via the terminal 3 and controls the actions of the second player 2P.

[0064] For example, the terminal 3 outputs the action information acquired from the intermediate server 2 to the game machine 4 as operation commands for the 2P controller. The game machine 4 causes the first player 1P and the second player 2P to act based on the operation commands of the 1P controller operated by the user and the operation commands acquired from the terminal 3.

[0065] In the battle mode, the intermediate server 2 acquires state information from the game device 4 of the user, inputs the state information into the first model 51 corresponding to that user, outputs behavior information, and controls the actions of the second player 2P as the opponent of the user (first player 1P). In the joint battle mode, the intermediate server 2 acquires state information from the game device 4 of the first user and / or the second user, inputs the state information into the first models 51, 51 corresponding to the first user and the second user, respectively, outputs behavior information, and controls the actions of the second player 2P corresponding to the first user and the second player 2P corresponding to the second user.

[0066] In the above, the action of the second player 2P is controlled using the first model 51 corresponding to the user himself / herself, but the first model 51 of another user may be used instead of the user himself / herself. For example, as shown in the upper left of FIG. 7, the terminal 3 displays a screen showing a list of multiple users including the user himself / herself, and accepts a selection input to select one of the users from the screen. For example, the terminal 3 displays a list of other users registered as the user's friends in the user DB 241, and accepts a selection input. The intermediate server 2 selects the first model 51 of the selected user, inputs the state information, and outputs the action information. This makes it possible to reproduce the second player 2P simulating the play of a friend or the like, rather than the user himself / herself.

[0067] Furthermore, the intermediate server 2 may operate multiple second players 2P, 2P, 2P... in the game, simulating the play of each user, using first models 51, 51, 51... corresponding to each of the multiple users. That is, the intermediate server 2 inputs state information to the multiple first models 51, 51, 51... corresponding to each user, acquires action information from each first model 51, and outputs it to the game machine 4. This allows the user to fight against or cooperate with multiple agents.

[0068] Furthermore, for example, the intermediate server 2 may output behavioral information based on not only the state information of the game but also the attribute information of the user playing the game. The attribute information may include user attributes such as age and gender, and play attributes (play style) when the user plays the game. For example, the intermediate server 2 reads out the user attributes and play attributes from the user DB 241, inputs them to the first model 51 together with the state information, and outputs the behavioral information. Note that the play attributes may be those registered in the user DB 241 by the user himself / herself as his / her own play style, or may be automatically determined by the intermediate server 2 from the game screen by image recognition. By using the user's attribute information as input to the first model 51, more appropriate behavioral information can be inferred.

[0069] Returning to Fig. 6, the explanation will be continued. The first model update unit 204 performs re-learning to update the first model 51 for each user based on state information after the action of the second player. For example, the intermediate server 2 sequentially stores a history of state information s at each time point t and action information a output based on state information s as a play log in the play log DB 242. After the game on the game machine 4 ends or at predetermined intervals (for example, once a day), the intermediate server 2 reads out the play log of each user from the play log DB 242 and updates the first model 51 for each user.

[0070] That is, the first model updating unit 204 calculates the reward r to be given to the agent based on the action information a output at time t and the state information s at the next time t+Δt that has changed based on the output action information a. The first model updating unit 204 updates the action value function Q(s, a) based on the calculated reward r. The first model updating unit 204 calculates the reward r for each time t and updates the action value function Q(s, a). The intermediate server 2 performs this process based on the play log of each user, and updates the first model 51 for each user.

[0071] The federated server 1 also updates the first model 51. Specifically, while the intermediate server 2 updates the first model 51 for each user, the federated server 1 re-learns by integrating the play logs of all users, and updates the first model 51.

[0072] For example, the federated server 1 performs federated learning to update the first model 51 based on the update result of each first model 51 updated for each user. That is, the federated server 1 acquires difference information before and after the update of the first models 51, 51, 51 corresponding to each user from each intermediate server 2, and updates the first model 51 based on the difference information of each first model 51. The difference information is a difference value before and after the update of the parameter of the first model 51, for example, a difference value of the weight between neurons. The federated server 1 averages the difference values ​​of the weight before and after the update in each first model 51, and updates the weight of the first model 51 based on the averaged difference value. For example, the federated server 1 periodically acquires difference information from each intermediate server 2, 2, 2, . . . to update the first model 51, and transmits the data of the first model 51 after the update to the intermediate server 2. By using the difference information as data for re-learning (update), the first model 51 can be updated without using personal information of each user.

[0073] In the above, federated learning is given as a method for relearning the play logs of each user in an integrated manner, but the present embodiment is not limited to this. For example, the federated server 1 may collect the play logs (data linking state information and behavior information) of each user and provide them to the first model 51 to update the first model 51. In this way, the relearning process executed by the federated server 1 is not limited to federated learning.

[0074] Next, a process equivalent to the chatbot function (escort mode or free talk mode) will be described. The text generation unit 205 inputs an input sentence to the second model 52, and generates an output sentence following the input sentence. When the escort mode is selected in the terminal 3, the text generation unit 205 acquires behavioral information from the behavioral information output unit 202, inputs an input sentence corresponding to the behavioral information to the second model 52, and generates text showing advice such as strategies. The text generation unit 205 passes the generated text to the character control unit 206.

[0075] The character control unit 206 controls the movement of a predetermined character (avatar) corresponding to the user's conversation partner, and outputs the text generated by the text generation unit 205 as the dialogue of the character to the terminal 3. The terminal 3 displays the character on the screen, displays the text in the form of a speech bubble or the like, and / or outputs the text as voice.

[0076] Fig. 8 is an explanatory diagram relating to the escort mode, which conceptually illustrates a state in which a character C gives advice on game strategies and the like to a user while the user is playing the game.

[0077] In the escort mode, the intermediate server 2 first inputs game state information to the first model 51 and outputs behavioral information indicating the actions the user should take in the game. The intermediate server 2 then inputs an input sentence corresponding to the behavioral information to the second model 52. The input sentence is the opening sentence of the text that should ultimately be output to the terminal 3. The intermediate server 2 converts the behavioral information output by the first model 51 into an opening sentence, and inputs the opening sentence to the second model 52 to generate the subsequent output sentence, thereby completing the text that should ultimately be output.

[0078] The method of converting the behavioral information into an opening sentence is not particularly limited, but for example, the intermediate server 2 converts the behavioral information into an opening sentence by referring to an opening sentence table 244 that associates behavioral information with opening sentences for each game. The intermediate server 2 acquires the title of the game as one piece of state information, selects an opening sentence from the opening sentence table 244 that corresponds to the title and to the behavioral information output by the first model 51, and inputs it to the second model 52.

[0079] The above process is an example, and the present embodiment is not limited to this. For example, the federated server 1 and the intermediate server 2 may prepare a first model 51 for an escort mode that takes state information as input and outputs an opening sentence as action information, in addition to the first model 51 for the battle mode or the cooperative mode, and may be able to generate an opening sentence directly from the state information. For example, the federated server 1 constructs a first model 51 that takes a game screen and a game title as input and generates an opening sentence (text indicating an action that the user should take), and installs it in the intermediate server 2. In the case of the escort mode, the intermediate server 2 inputs state information to the first model 51 to generate (output) an opening sentence, which is then input to the second model 52. In this way, the first model 51 may be able to generate an opening sentence directly.

[0080] The intermediate server 2 outputs text consisting of the opening sentence input to the second model 52 and the output sentence generated by the second model 52 to the terminal 3. The terminal 3 displays the text as a spoken sentence of the character C and / or outputs it by voice. Note that the intermediate server 2 may output the text to the game machine 4 instead of the terminal 3, and may superimpose the text on the game screen. This allows the user to play the game while receiving advice on strategies and the like.

[0081] Returning to Fig. 6, the explanation will be continued. In this embodiment, the first model update unit 204 updates the first model 51 even in the escort mode. That is, when the first model update unit 204 outputs text based on the behavior information a at time t, the first model update unit 204 calculates the reward r based on the behavior information a that is the source of the output text and the state information s at the next time t+Δt, and updates the action value function Q(s, a). This makes it possible to update the first model 51 so that appropriate advice can be given to each user.

[0082] The text input unit 207 accepts an input sentence input by a user when a free talk mode is selected in the terminal 3. The text input unit 207 passes the input sentence to the text generation unit 205.

[0083] In the free talk mode, the text generation unit 205 generates an output sentence to be output as a response from the character to an input sentence acquired from the text input unit 207. Specifically, the text generation unit 205 generates the output sentence while selecting the second model 52 or a rule base according to the content of the input sentence.

[0084] Fig. 9 is an explanatory diagram relating to the free talk mode, which conceptually illustrates a process flow for generating an output sentence in the free talk mode.

[0085] As described above, the intermediate server 2 accepts an input sentence from the terminal 3. The intermediate server 2 first analyzes the input sentence and determines whether an output sentence should be generated using the second model 52 or whether the output sentence should be generated based on a rule.

[0086] Specifically, the intermediate server 2 refers to the scenario table 246 to determine whether or not a specific keyword is included in the input sentence. Fig. 9 illustrates how processing branches when two types of input sentences, "Good morning" and "I'll do my best!", are received. When "Good morning" is input, the input sentence is stored as a keyword in the scenario table 246, so the intermediate server 2 generates an output sentence based on the rules. On the other hand, "I'll do my best!" does not contain a keyword, so the intermediate server 2 generates an output sentence using the second model 52.

[0087] When an input sentence contains a keyword, the intermediate server 2 refers to the scenario table 246 and selects an output sentence template corresponding to the keyword. The intermediate server 2 generates an output sentence by synthesizing the selected output sentence template with user information related to the user who input the input sentence. For example, the intermediate server 2 acquires information stored as user attributes in the user DB 241 as user information and synthesizes it into the template.

[0088] Alternatively, the intermediate server 2 may acquire user information through an external API, as shown in Fig. 9. For example, the intermediate server 2 acquires the schedule for the day by referring to a scheduler that manages the user's schedule, and synthesizes it into a template. Note that the scheduler is an example of an external API, and the user information acquired from the external API is not limited to the user's schedule.

[0089] The intermediate server 2 outputs the output sentences generated based on the scenario table 246 to the terminal 3, and causes them to be displayed as the character's dialogue or to be output as voice.

[0090] On the other hand, if the input sentence does not contain a keyword, the intermediate server 2 inputs the input sentence to the second model 52 and generates an output sentence for the input sentence.

[0091] In addition to the second model 52 for escort mode, the federated server 1 and the intermediate server 2 may prepare a second model 52 for free talk that receives an input sentence by a user and generates an output sentence corresponding to a response from a character. In this case, the federated server 1 learns training data that associates a correct output sentence (response sentence) with an input sentence by a training user, and generates the second model 52. In the case of free talk mode, the intermediate server 2 inputs an input sentence to the model and generates an output sentence. This makes it possible to generate appropriate sentences separately for escort mode and free talk mode.

[0092] The intermediate server 2 converts a part of the generated output sentence according to the character set by the user for the conversation partner. Specifically, the intermediate server 2 converts the endings of the output sentence according to the character by referring to the conversion table 245. The intermediate server 2 performs a morphological analysis of the output sentence to identify the endings, and converts them into endings according to the character set by the user.

[0093] In this case, the intermediate server 2 may estimate the user's emotion from the input sentence and change the endings according to the estimated emotion. The emotion estimation method is not particularly limited, but for example, a rule-based estimation may be performed by referring to a table that associates words (keywords) that may be included in the input sentence with estimated emotion values. Alternatively, the intermediate server 2 may use a machine learning model that inputs the input sentence and outputs an estimated emotion value. The intermediate server 2 estimates the emotion from the input sentence as an emotion type such as "positive" or "negative" or a numerical value. The intermediate server 2 converts the endings of the output sentence by referring to a conversion table 245 that associates pre-conversion endings with converted endings according to the user's emotion.

[0094] Further, the intermediate server 2 may estimate the intimacy between the user and the character from the input sentence in addition to the user's emotion, and change the endings according to the estimated intimacy. For example, the intermediate server 2 analyzes past input sentences stored in the dialogue log DB 243 to estimate the intimacy between the user and the character. As in the emotion estimation, the intimacy estimation method may be a rule-based estimation by referring to a table that associates words that may be included in the input sentence with the intimacy, or a machine learning model that inputs the input sentence and outputs the intimacy may be used. The intermediate server 2 converts the endings of the output sentence by referring to a conversion table 245 that associates the converted endings according to the intimacy with the endings before conversion.

[0095] In this embodiment, the emotion and intimacy are used only for changing the endings of the output sentences, but parts other than the endings (for example, interjections, auxiliary verbs, etc.) may be changed according to the emotion and intimacy. In addition to the output sentences generated using the second model 52, parts of the output sentences generated on a rule basis may also be changed according to the emotion and intimacy.

[0096] The intermediate server 2 outputs the output sentence after the ending conversion to the terminal 3, and displays or outputs as voice the dialogue of the character. In this way, by selectively using output sentence generation based on the rule base and output sentence generation using the second model 52, it is possible to handle more complex dialogue. In addition, by converting the endings of the generated output sentence, it is possible to edit it to an expression that reflects the personality of the character.

[0097] 10 is a flowchart showing the procedure of the process of generating first model 51. The process content when federated server 1 generates first model 51 by machine learning will be described with reference to FIG. The control unit 11 of the federated server 1 acquires state information that indicates the play state of the game (step S11). The control unit 11 inputs the acquired state information to the first model 51, and outputs action information that indicates an action to be taken in the game (step S12).

[0098] The control unit 11 controls the action of the second player in the game based on the action information output in step S12 (step S13). The control unit 11 calculates a reward based on the action information output in step S12 and state information after the action of the second player (step S14). The control unit 11 updates the first model 51 (action value function) based on the calculated reward (step S15).

[0099] The control unit 11 judges whether or not to end the learning of the first model 51 (step S16). For example, the control unit 11 judges whether or not the learning process of steps S11 to S15 has been executed a predetermined number of times, or whether or not the action value function has sufficiently converged. If it is judged that the learning should not be ended (S16: NO), the control unit 11 returns the process to step S11. If it is judged that the learning should be ended (S16: YES), the control unit 11 transmits the data of the first model 51 to the intermediate server 2 (step S17), and ends the series of processes.

[0100] Fig. 11 is a flowchart showing the procedure of the process of generating second model 52. The process content when federated server 1 generates second model 52 by machine learning will be described with reference to Fig. 11. Control unit 11 of federated server 1 acquires training data for generating second model 52 (step S31). The training data includes a training input sentence and a correct output sentence following the input sentence.

[0101] The control unit 11 generates a second model 52 that generates an output sentence following an input sentence based on the training data when the input sentence is input (step S32). The control unit 11 inputs the training input sentence to the second model 52 to generate an output sentence, and compares it with the correct output sentence. The control unit 11 optimizes the parameters (weights between neurons, etc.) of the second model 52 so that the two are close to each other, and generates the second model 52. The control unit 11 transmits data of the generated second model 52 to the intermediate server 2 (step S33), and ends the series of processes.

[0102] Fig. 12 is a flowchart showing the process steps executed by the intermediate server 2. In Fig. 12, the process contents executed by the intermediate server 2 will be explained. The control unit 21 of the intermediate server 2 determines whether or not the battle mode or the cooperative mode has been selected at the terminal 3 (step S51). If it is determined that the battle mode or the cooperative mode has been selected (S51: YES), the control unit 21 executes a subroutine related to the battle mode or the cooperative mode (step S52).

[0103] After executing the process of step S52, or if step S51 is NO, the control unit 21 determines whether or not the escort mode has been selected in the terminal 3 (step S53). If it is determined that the escort mode has been selected (S53: YES), the control unit 21 executes a subroutine related to the escort mode (step S54).

[0104] After executing the process of step S54, or if step S53 is NO, the control unit 21 judges whether or not the free talk mode has been selected in the terminal 3 (step S55). If it is judged that the free talk mode has been selected (S55: YES), the control unit 21 executes a subroutine related to the free talk mode (step S56). After executing the process of step S56, or if step S55 is NO, the control unit 21 ends the series of processes.

[0105] 13 is a flowchart showing the processing procedure of a subroutine in the battle mode or the joint battle mode. The processing contents of the subroutine in step S52 will be described with reference to FIG. The control unit 21 of the intermediate server 2 outputs a list of multiple users to the terminal 3, and receives a selection input from the terminal 3 to select one of the users (step S71). The control unit 21 acquires state information of the game being played by the user from the terminal 3 (step S72). The control unit 21 inputs the state information acquired in step S72 to the first model 51 corresponding to the user selected in step S71, and outputs behavior information (step S73). The control unit 21 controls the second player in the game based on the behavior information output from the first model 51 (step S74).

[0106] The control unit 21 judges whether the game being played by the user has ended (step S75). If it is judged that the game has not ended (S75: NO), the control unit 21 returns the process to step S72. If it is judged that the game has ended (S75: YES), the control unit 21 stores a history of the state information acquired at each time point and the action information output based on the state information as a play log in the play log DB 242 (step S76). The control unit 21 updates the first model 51 by referring to the play log (step S77). That is, the control unit 21 calculates a reward based on the action information output at each time point and the state information after the second player operates based on the action information, and updates the action value function based on the reward. The control unit 21 returns the subroutine.

[0107] 14 is a flowchart showing the procedure of the subroutine of the escort mode. The process of the subroutine of step S54 will be described with reference to FIG. The control unit 21 of the intermediate server 2 acquires state information of the game being played by the user (step S91). The control unit 21 inputs the acquired state information into the first model 51, and outputs behavior information (step S92).

[0108] The control unit 21 inputs the behavioral information output from the first model 51 to the second model 52, and generates text for assisting the progress of the game (step S93). For example, the control unit 21 converts the behavioral information into an opening sentence by referring to an opening sentence table 244 that associates behavioral information and opening sentences that can be output from the first model 51. The control unit 21 inputs the converted opening sentence as an input sentence to the second model 52, and generates an output sentence following the opening sentence. The control unit 21 outputs text consisting of the opening sentence and the output sentence to the terminal 3 (step S94).

[0109] The control unit 21 judges whether the game being played by the user has ended (step S95). If it is judged that the game has not ended (S95: NO), the control unit 21 returns the process to step S91. If it is judged that the game has ended (S95: YES), the control unit 21 stores the play log in the play log DB 242 (step S96). The control unit 21 also updates the first model 51 based on the play log (step S97). That is, the control unit 21 calculates a reward based on the behavior information that was the source of the text output at each time point and the state information after the text was output, and updates the action value function based on the reward. The control unit 21 returns the subroutine.

[0110] 15 is a flowchart showing the procedure of the subroutine of the free talk mode. The process of the subroutine in step S56 will be described with reference to FIG. The control unit 21 of the intermediate server 2 accepts an input of an input sentence from the terminal 3 (step S111). The control unit 21 refers to the scenario table 246 that stores keywords, and determines whether the input sentence includes a keyword (step S112).

[0111] If it is determined that the keyword is included (S112: YES), the control unit 21 refers to the scenario table 246 and selects a template of an output statement corresponding to the keyword (step S113). The control unit 21 acquires user information from the user DB 241 or an external API and synthesizes the user information into the template (step S114).

[0112] When it is determined that the keyword is not included (S112: NO), the control unit 21 inputs the input sentence to the second model 52 to generate an output sentence (step S115). The control unit 21 also estimates the user's emotion based on the input sentence accepted in step S111 (step S116). The control unit 21 also estimates the intimacy between the user and the character based on the past input sentences stored in the dialogue log DB 243 (step S117). The control unit 21 refers to the conversion table 245 and converts a part (ending) of the output sentence according to the character set as the user's dialogue partner, the emotion estimated in step S116, the intimacy estimated in step S117, etc. (step S118).

[0113] The control unit 21 outputs to the terminal 3 the output statement into which the user information has been synthesized in step S114, or the output statement converted in step S118 (step 119), and returns to the subroutine.

[0114] 16 is a flowchart showing the procedure of the update process of first model 51 in federated server 1. Re-learning process in which federated server 1 updates first model 51 will be described with reference to FIG. The control unit 11 of the federated server 1 acquires difference information before and after the update of each first model 51 updated for each user from the intermediate servers 2, 2, 2, ... (step S131). The difference information is a difference value before and after the update of the parameters of the first model 51, for example, a difference value of the weight between neurons. The control unit 11 updates the first model 51 based on the difference information related to each first model 51 (step S132). That is, the control unit 11 averages the difference value of the weight of each first model 51, and updates the weight based on the averaged difference value. The control unit 11 transmits data of the updated first model 51 to the intermediate server 2 (step S133), and ends the series of processes.

[0115] In this embodiment, only the first model 51 is updated, but the second model 52 may also be re-learned and updated for each user.

[0116] As described above, according to the first embodiment, the first model 51 is updated for each user based on the play log of each user (behavior information and state information after the behavior indicated by the behavior information is performed), thereby constructing an agent suitable for each user. This makes it possible to provide a game experience suitable for each user.

[0117] Furthermore, according to the first embodiment, by using the game screen as the state information, it is possible to infer behavioral information for any type (title) of game.

[0118] Furthermore, according to the first embodiment, the actions of a second player who competes against or cooperates with a user can be optimized according to the play of the user.

[0119] Moreover, according to the first embodiment, advice (text) such as strategies can be presented based on the behavioral information output from the first model 51. In particular, in this embodiment, by using the second model 52, the text can be suitably generated.

[0120] Moreover, according to the first embodiment, the second model 52 can also be used for free talk. In particular, in the present embodiment, the output sentence can be modified in an appropriate form, such as by partially converting the output sentence according to the character, or by changing the output sentence according to the emotion or intimacy.

[0121] (Embodiment 2) In the first embodiment, a configuration in which the intermediate server 2 mainly performs the processing has been described. In the present embodiment, a configuration in which the terminal 3 mainly performs the processing will be described. Note that the same reference numerals are used to denote content that overlaps with the first embodiment, and the description thereof will be omitted.

[0122] 17 is a functional block diagram showing an example of the configuration of a game system according to embodiment 2. In this embodiment, data of a first model 51 and a second model 52 are installed in a terminal 3. In this embodiment, the terminal 3 performs a series of processes using the first model 51 and the second model 52.

[0123] The control unit 31 of the terminal 3 executes the program P3 to function as a state information acquisition unit 201, a behavior information output unit 202, a second player control unit 203, a text generation unit 205, a character control unit 206, and a text input unit 207. Since the functions of each module are the same as those in the first embodiment, the same reference symbols are used in this embodiment and the description thereof is omitted.

[0124] The terminal 3 performs an inference process using the first model 51 and the second model 52. That is, the terminal 3 inputs state information acquired from the game machine 4 to the first model 51 and outputs behavior information. When the battle mode or the joint battle mode is selected, the terminal 3 controls the second player based on the behavior information. When the escort mode is selected, the terminal 3 inputs an input sentence (opening sentence) according to the behavior information to the second model 52 to generate an output sentence, generates text consisting of the input sentence and the output sentence, and displays and / or outputs as a dialogue sentence of the character. When the free talk mode is selected, the terminal 3 accepts an input sentence from the user, inputs the input sentence to the second model 52, and generates an output sentence.

[0125] As described above, in this embodiment, a series of inference processes is executed in the edge device (terminal 3). Meanwhile, the update (relearning) process of the first model 51 is executed by the intermediate server 2, as in the first embodiment. That is, the intermediate server 2 acquires state information at each point in time and action information based on the state information from the terminal 3, and stores the state information as a play log in the play log DB 242. The intermediate server 2 calculates a reward based on the play log, and updates the action value function. The intermediate server 2 transmits the updated data of the first model 51 to the terminal 3.

[0126] In this embodiment, the intermediate server 2 updates (relearns) the first model 51. However, the update process for the first model 51 may be executed by the terminal 3.

[0127] As described above, the inference processing using the first model 51 and the second model 52 may be executed by the terminal 3. Note that, instead of executing the inference processing by only one of the intermediate server 2 and the terminal 3, the processing may be distributed, such that a part of the inference processing (e.g., the processing of the first model 51) is executed by the terminal 3 and a part of the inference processing (e.g., the processing of the second model 52) is executed by the intermediate server 2. Also, the edge device that executes the processing may be the game machine 4 instead of the terminal 3.

[0128] Other than the above, this embodiment is the same as the first embodiment, so a detailed description of the flowchart and other details will be omitted.

[0129] As described above, according to the second embodiment, a series of inference processes can also be executed by an edge device.

[0130] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0131] 1. Federated Server 11 Control section 12 Main memory 13. Communications Department 14 Auxiliary storage P1 Program 2. Intermediate server (information processing device) 21 Control section 22 Main memory 23 Communications Department 24 Auxiliary storage P2 Program 241 User DB 242 Playlog DB 243 Dialogue Log DB 244 Heading Table 245 Conversion Table 246 Scenario Table 3. Terminal 31 Control Unit 32 Main memory 33 Communications Department 34 Display section 35 Input section 36 Audio output section 37 Auxiliary storage P3 Program

Claims

1. Acquire state information representing a play state of a game played by a user; inputting the acquired state information into a first model that has been trained to output action information indicating an action to be taken in the game when the state information is input, and outputting the action information; Calculating a reward based on the behavior information and the state information after performing the behavior indicated by the behavior information; updating the first model for each user based on the reward; Acquire difference information before and after the update of each of the first models updated for each user; The first model is updated based on the difference information. An information processing method in which processing is performed by a computer.

2. acquiring a play screen of the game as the state information; The play screen is input to the first model, and the action information is output. The information processing method according to claim 1 .

3. Controlling an action of a second player in the game based on the action information; Calculating the reward based on the action information and the state information after the second player has performed an action.

3. The information processing method according to claim 1 or 2.

4. generating and outputting text for assisting the progress of the game based on the behavioral information; Calculating the reward based on the behavior information and the state information after outputting the text. The information processing method according to any one of claims 1 to 3.

5. inputting an input sentence according to the behavioral information into a second model that has been trained to generate an output sentence following the input sentence when the input sentence is input, and generating the output sentence; outputting the text consisting of the input sentence and the output sentence The information processing method according to claim 4.

6. accepting input of an input sentence from the user; inputting the input sentence into the second model to generate the output sentence; converting a part of the output sentence in accordance with a type of character set as a conversation partner of the user; The converted output sentence is output as an utterance sentence of the character. The information processing method according to claim 5.

7. determining whether the input sentence contains a predetermined keyword; If it is determined that the keyword is not included, the input sentence is input to the second model to generate the output sentence; If it is determined that the keyword is included, the output sentence is generated by referring to a table in which the keyword and the output sentence are associated with each other. The information processing method according to claim 6.

8. Estimating an emotion or a degree of intimacy of the user based on the input sentence; The output sentence is changed according to the estimated emotion or intimacy.

8. The information processing method according to claim 6 or 7.

9. Accepting a selection input for selecting one of the users from the plurality of users; The state information is input to the first model corresponding to the selected user, and the behavior information is output. The information processing method according to any one of claims 1 to 8.

10. A first acquisition unit that acquires state information representing a play state of a game played by a user; an output unit that inputs the acquired state information into a first model that has been trained to output action information indicating an action to be taken in the game when the state information is input, and outputs the action information; a calculation unit that calculates a reward based on the behavior information and the state information after the behavior indicated by the behavior information is performed; A first update unit that updates the first model for each user based on the reward; A second acquisition unit that acquires difference information before and after the update of each of the first models updated for each user; a second update unit that updates the first model based on the difference information; An information processing device comprising:

11. Acquire state information representing a play state of a game played by a user; inputting the acquired state information into a first model that has been trained to output action information indicating an action to be taken in the game when the state information is input, and outputting the action information; Calculating a reward based on the behavior information and the state information after performing the behavior indicated by the behavior information; updating the first model for each user based on the reward; Acquire difference information before and after the update of each of the first models updated for each user; The first model is updated based on the difference information. A program that causes a computer to carry out processing.

Images