Information processing method, program, and information processing system

Abstract

The avatar can act in accordance with the situation of the screen.SOLUTION: An information processing method according to an embodiment includes a determination step of monitoring a screen using a first learning model and determining a change in the screen, a decision step of deciding a motion of an avatar using a second learning model according to the change in the screen determined in the determination step, and an instruction step of instructing the avatar to perform the motion decided in the decision step.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing method, a program, and an information processing system. [Background technology]

[0002] Patent Document 1 discloses an object control system that stands next to a user and controls the actions of a robot in accordance with the user's emotions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6814089 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above prior art, a robot is controlled to empathize with the user's emotions by being next to the user. However, emotions are estimated by taking pictures of the user's facial expressions with a camera or based on the user's speech, and emotions are not estimated by sharing the screen that the user is viewing.

[0005] In consideration of the above, an object of the present invention is to provide an information processing method, a program, and an information processing system that allow an avatar to act in accordance with the situation on the screen. [Means for solving the problem]

[0006] According to one embodiment of the information processing method, the method includes a determination step of monitoring a screen using a first learning model and determining a change in the screen, a decision step of determining an action of an avatar using a second learning model in accordance with the change in the screen determined in the determination step, and an instruction step of instructing the avatar to perform the action determined in the determination step.

[0007] According to one embodiment of the program, an information processing device is caused to execute an information processing method including a determination step of monitoring a screen using a first learning model and determining changes in the screen, a decision step of determining an action of an avatar using a second learning model in accordance with the changes in the screen determined in the determination step, and an instruction step of instructing the avatar to perform the action determined in the determination step.

[0008] According to one embodiment, the information processing system is an information processing system equipped with an information processing device, and performs a determination step of monitoring a screen using a first learning model and determining changes in the screen, a decision step of determining an avatar's behavior using a second learning model in accordance with the changes in the screen determined in the determination step, and an instruction step of instructing the avatar to perform the behavior determined in the determination step. [Effects of the Invention]

[0009] According to one embodiment, the avatar can act in accordance with the situation on the screen. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a server according to an embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a user terminal according to the embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of a server according to an embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of a functional configuration of a user terminal according to the embodiment. [Figure 6] 10 is a flowchart illustrating an example of a process for extracting an M&A partner, which is executed by the information processing system according to the embodiment. [Figure 7] FIG. 1 is a diagram showing an outline of a case where an avatar according to an embodiment moves in accordance with the situation on the screen. DETAILED DESCRIPTION OF THE INVENTION

[0011] An embodiment of an information processing system according to the present invention will be described below with reference to Figures 1 to 5. In each drawing, the same or equivalent components and parts are denoted by the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0012] (System Overview) First, an overview of the information processing system 10 according to this embodiment will be described. The information processing system 10 according to this embodiment is a system that causes an avatar to act in accordance with the situation on the screen. In this embodiment, the information processing system 10 is a system that causes an avatar 3 to move and speak in accordance with the situation on the screen of the user terminal 2. Specifically, the situation on the screen can be classified into a plurality of types, for example, negative, positive, and neutral, and the facial expression, words, attitude, behavior, etc. of the avatar 3 can be changed according to the classification.

[0013] (System Configuration) Fig. 1 is a diagram showing an example of the configuration of an information processing system 10 according to this embodiment. As shown in Fig. 1, the information processing system 10 according to this embodiment includes a server device 1 and a user terminal 2, which are connected to each other so as to be able to communicate with each other via a network N. The network N is, for example, a wired local area network (LAN), a wireless LAN, the Internet, a public line network, a mobile data communication network, or a combination thereof. In the example of Fig. 1, the information processing system 10 includes one server device 1 and one user terminal 2, but may include multiple of each.

[0014] The server device 1 is an example of an information processing device that monitors the screen of the user terminal 2 and issues operational instructions to the avatar 3 in response to changes in the screen. The server device 1 may be a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a microcomputer, or a combination of these. The specific configuration and operation of the server device 1 will be described later.

[0015] The user terminal 2 is an example of an information processing device that performs operations for inputting and displaying various information. The user terminal 2 may be a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a microcomputer, a wearable device, or a combination of these. In this embodiment, an information processing device that can display a screen and an avatar of a monitoring target is used as an example.

[0016] (Hardware configuration - Server) 2 is a block diagram showing the hardware configuration of the server device 1. The server device 1 includes a processor 101, a memory 102, a storage 103, and a communication I / F 104, which are communicably connected to each other via a bus B.

[0017] The processor 101 controls each component of the server device 1 and realizes the functions of the server device 1 by loading various programs stored in the storage 103 into the memory 102 and executing them. The programs executed by the processor 101 include, but are not limited to, an OS (Operating System) and various programs described below. Execution of these programs by the processor 101 realizes part of the state visualization method according to this embodiment. The processor 101 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), or a combination thereof.

[0018] The memory 102 is, for example, a read-only memory (ROM), a random access memory (RAM), or a combination thereof. The ROM is, for example, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a combination thereof. The RAM is, for example, a dynamic random access memory (DRAM), a static random access memory (SRAM), a magnetoresistive random access memory (MRAM), or a combination thereof.

[0019] The storage 103 stores an OS, various programs (described later), and various data. The storage 103 is, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), a storage class memory (SCM), or a combination of these.

[0020] The communication I / F 104 is an interface for connecting the server device 1 to external devices including the user terminal 2 via the network N and controlling communication. The communication I / F 104 is, for example, an adapter compliant with Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), Ethernet (registered trademark), or optical communication (e.g., Fibre Channel), but is not limited to these.

[0021] (Hardware configuration - user terminal) 3 is a block diagram showing the hardware configuration of the user terminal 2. The user terminal 2 includes a processor 201, a memory 202, a storage 203, a communication I / F 204, an input / output I / F 205, an input device 206, and an output device 207, which are communicably connected to each other via a bus B. The communication I / F 204 is an interface for connecting the user terminal 2 to external devices including the server device 1 via a network N and for controlling communication. The communication I / F 204 is, for example, an adapter compliant with Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), Ethernet (registered trademark), or optical communication (e.g., Fibre Channel), but is not limited to these.

[0022] The input / output I / F 205 is an interface for connecting an input device 206 and an output device 207 to the user terminal 2. The input device 206 is, for example, a mouse, a keyboard, a touch panel, a microphone, a scanner, a camera, various sensors, operation buttons, or a combination of these. The output device 207 as a user interface is, for example, a display, a projector, a printer, a speaker, a vibrator, or a combination of these. In this embodiment, as an example, the output device 207 and the input device 206 are an integrally configured touch panel display.

[0023] In this embodiment, the program may be written to the memory 202 or the storage 203 during the manufacturing stage of the server device 1, or may be provided to the server device 1 via the network N. Alternatively, the program may be provided to the server device 1 via a non-transitory computer-readable recording medium such as a disk medium (not shown).

[0024] (Functional configuration - Server) Next, the functional configuration of the server device 1 will be described. Fig. 4 is a diagram showing an example of the functional configuration of the server device 1. When executing various programs, the server device 1 realizes various functions using the above-mentioned hardware resources. The server device 1 has a communication unit 11, a storage unit 12, and a control unit 13 as functional configurations realized by the server device 1. Each functional configuration is realized by the processor 101 reading and executing a program stored in the memory 102 or storage 103.

[0025] The storage unit 12 also stores, for each character, character prompts 121 describing the characteristics and speech patterns of the character, and situation prompts 122 describing the relationship between the set character and the user and how the character should interact with the user. The character prompts 121 are prompts that indicate the characteristics of the character, and include information such as the character's movements, speech patterns, catchphrases, facial expressions, and personality. The situation prompts 122 are prompts that include information such as the relationship between the character and the user and how the character should interact with the user. For example, when a user selects a desired character from multiple characters as an avatar to be displayed on multiple user terminals 2, the character prompts 121 and situation prompts 122 are stored in advance for the multiple characters.

[0026] The control unit 13 includes a receiving unit 131 that receives the screen of the user terminal 2, and a first learning model 132 that monitors the screen of the user terminal 2 and determines the status of the screen. The control unit 13 also includes a second learning model 133 that determines the character's behavior, and an instruction unit 143 that instructs the character to display the determined character behavior on the user terminal 2. The first learning model 132 monitors the screen of the user terminal 2, determines a plurality of classifications (e.g., positive, negative, neutral) according to changes in the screen, and generates a prompt to be input to the second learning model 133 according to the determination result. The second learning model 133 determines the character's behavior from the character prompt 121, situation prompt 122, and the prompt generated by the first learning model 132.

[0027] In this embodiment, two learning models are used to determine the behavior of an avatar in response to changes in the screen of the user terminal 2: a first learning model 132 for monitoring the screen of the user terminal 2, and a second learning model 133 for determining the behavior of the character. However, this is not limited to this, and the type is not limited as long as a model equivalent to the above two learning models can be used. Details of these will be described later.

[0028] (Functional configuration - user terminal) Next, the functional configuration of the user terminal 2 will be described. Fig. 5 is a diagram showing an example of the functional configuration of the user terminal 2. When executing various programs, the user terminal 2 realizes various functions using the above-mentioned hardware resources. The user terminal 2 has, as functional components realized by the user terminal 2, a communication unit 21, a storage unit 22 in which a program 221 is stored, and a control unit 23. Each functional component is realized by the processor 201 reading and executing the program 221 stored in the memory 202 or the storage 203. The control unit 23 includes an information acquisition control unit 231 that acquires information sent from the server device 1, and a display unit 232 that displays the acquired information on the output device 207.

[0029] Next, the flow of processing in the information processing system according to this embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of processing executed by the information processing system 10 according to this embodiment to determine the behavior of an avatar depending on the screen status. Specifically, the processing will be described in which the screen of the user terminal 2 is monitored and the avatar displayed on the screen behaves depending on the screen status.

[0030] First, in step S101, the receiving unit 131 monitors the screen of the user terminal 2. Specifically, the user terminal 2 transmits screen information of the user terminal 2 to the server device 1, and the receiving unit 131 receives the screen of the user terminal 2. The first learning model 132 then monitors this screen. Furthermore, the screen information may be, for example, a display of a specific application on the screen, and may be multiple applications without being limited to a single application. In other words, the screen information may be, for example, a screen displaying content such as game footage or a movie, or a screen displaying an application such as a spreadsheet or word processing software, or a combination of these. The first learning model 132 may monitor not only a specific screen but also multiple screens; for example, the user may select the screen to monitor.

[0031] Next, in step S102, the first learning model 132 determines whether a change has occurred in the screen being monitored. If no change has occurred in the screen (NO), the first learning model 132 continues monitoring. On the other hand, if a change has occurred in the screen (YES), the first learning model 132 determines what type of change in the screen should be classified as, and generates a prompt according to that classification.

[0032] Here, a case where the types of screen changes are classified into three types, positive, negative, and neutral, in this embodiment will be described. While monitoring the screen, the first learning model 132 determines which of these three types the screen falls into. That is, if classified as neutral, it is determined that there is no screen change, and if classified as positive or negative, it is determined that there is a screen change. Note that, although the classification into three types, positive, negative, and neutral, is used in this embodiment, this is not limiting. For example, the screen change may be determined based on the cursor movement status, the input status of a formula in a spreadsheet software, or the page creation status in software for creating presentation materials. In other words, the multiple classifications may be changed depending on the type of screen to be monitored. The classification may be set according to the screen to be monitored, or may be automatically set from the monitored screen.

[0033] Next, in step S103, the first learning model 132 generates a prompt according to the screen situation determined in step S102. Next, in step S104, the second learning model 133 determines the character's behavior. Specifically, the character's behavior is determined from the prompt generated by the first learning model 132 in step S103, the character prompt 121, and the situation prompt 122. The character in question is a character previously set by the user, and the character prompt 121 and situation prompt 122 of the selected character are used. Then, in step S105, the instruction unit 134 instructs the user terminal 2 on the character's behavior determined in step S104, and the avatar character displayed on the screen of the user terminal 2 moves in accordance with the instruction. Here, the character's behavior includes the character's whole-body movements, facial expressions, speech, etc.

[0034] Next, referring to FIG. 7, the behavior of an avatar according to the screen situation will be described. FIG. 7 is a diagram illustrating an overview of a case where an avatar behaves according to the screen situation. A monitoring screen im01, a character selection screen im02, and an avatar im03 are displayed on a screen A1 displayed on the user terminal 2. The monitoring screen im01 is a screen monitored by the first learning model 132. The character selection screen im02 is a screen on which the user selects a desired character. The avatar im03 is a character selected by the user on the character selection screen im02. In this embodiment, since the person on the monitoring screen im01 is smiling, the first learning model 132 can determine that this screen is positive, and the first learning model 132 generates a positive prompt. In addition, a character prompt 121 and a situation prompt 122 are determined from the character selected on the character selection screen im02, and the second learning model 133 determines the behavior of the avatar im03 based on the generated prompt and the prompt corresponding to the character. The avatar im03 behaves based on the determined behavior.

[0035] As described above, according to this embodiment, the avatar can act in accordance with the situation on the screen. In other words, by using the first learning model 132 that monitors the screen of the user terminal 2 and the second learning model 133 that determines the character's behavior, the avatar's behavior can be changed according to the situation on the screen. Furthermore, the screen to be monitored is not limited to content, etc., and can be the screen of an application running on the information processing device. By appropriately changing the classification determined by the first learning model 132 depending on the application, a prompt can be generated with a classification appropriate for each application, and the avatar can perform an action according to the application.

[0036] In this embodiment, the server device 1 monitors the screen of the user terminal 2 and determines the character's actions. However, without being limited to this, for example, the screen may be monitored and the character's actions may be determined on the user terminal 2 without processing on the server device 1 via a network. That is, a program that performs the processing of the server device 1 is stored in the user terminal 2, and the user terminal 2 has the functions of the server device 1. In this case, the user terminal 2 stores character prompts and situation prompts for each avatar character, and is equipped with a first learning model, a second learning model, and an instruction unit. This allows the avatar to act according to the situation on the screen without processing on the server device 1.

[0037] Furthermore, in this embodiment, the avatar acts on the screen of the user terminal 2 according to the screen status, but for example, the avatar may move from the screen of the user terminal 2 to the screen of another terminal. That is, if the user terminal 2 is a desktop PC and the other terminal is a smartphone, the avatar may move from the desktop PC to the smartphone screen according to the screen status. Specifically, when the screen monitored by the desktop PC is displayed on the entire display device, the display of the avatar may hide part of the screen. In this case, the avatar can move to the smartphone and act within the smartphone.

[0038] As described above, according to this embodiment, by using two types of learning models, the first learning model and the second learning model, the avatar can act in accordance with the situation on the screen. For example, depending on the situation of the game or movie content, the avatar can behave, move, and speak in a way that empathizes with the user. That is, in a positive situation, the avatar can behave or speak in a positive way, while in a negative situation, the avatar can behave or speak in a supportive way. For example, depending on the character, the avatar may behave or speak in a contrarian way. Furthermore, in the case of spreadsheet software, word processing software, presentation software, or the like, the avatar may monitor the screen situation, such as input operations or cursor movements, and cheer the user on, provide input support, give advice, or assist with operations.

[0039] Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications, improvements, etc. within the scope of achieving the object of the present invention are included in the present invention.

[0040] Furthermore, for example, the above-described series of processes can be executed by hardware or software. In other words, the functional configuration is merely an example and is not particularly limited. That is, it is sufficient that the information processing system has the function of being able to execute the above-described series of processes as a whole, and there is no particular limit to the type of functional block used to realize this function. Furthermore, the location of the functional block is also not particularly limited and may be arbitrary. For example, a functional block of a server may be transferred to a user terminal, etc. Conversely, a functional block of a user terminal may be transferred to a server, etc. Furthermore, one functional block may be configured as a single piece of hardware, a single piece of software, or a combination thereof.

[0041] Furthermore, for example, when a series of processes is executed by software, the programs constituting the software are installed onto a computer or the like from a network or a recording medium. The computer may be a computer incorporated into dedicated hardware. Furthermore, the computer may be a computer capable of executing various functions by installing various programs thereon, such as a server, a general-purpose smartphone, or a personal computer.

[0042] Furthermore, for example, the recording medium containing such a program may be configured not only as a removable medium (not shown) that is distributed separately from the device main body in order to provide the program to the user, but also as a recording medium that is provided to the user in a state that is pre-installed in the device main body.

[0043] In this specification, the steps describing the program recorded on the recording medium include not only processes that are performed in chronological order, but also processes that are not necessarily performed in chronological order but are performed in parallel or individually. Also, in this specification, the term "system" means an overall device composed of multiple devices or multiple means, etc. [Explanation of symbols]

[0044] 10 Information Processing Systems 1 server 2 user terminals 131 Receiver 132 First Learning Model 133 Second Learning Model 134 instruction section

Claims

1. An information processing method executed by an information processing device, a determination step of monitoring a screen using a first learning model and determining a change in the screen; a determining step of determining an action of an avatar using a second learning model in response to the change in the screen determined in the determining step; an instruction step of instructing the avatar to perform the action determined in the determination step; An information processing method including:

2. The first learning model generates a prompt for input to the second learning model in response to the change in the screen determined in the determining step; The learning model determines the action of the avatar using the generated prompt as an input. The information processing method according to claim 1 .

3. The screen changes include multiple classifications, In the determination step, it is determined whether the image on the screen corresponds to one of the plurality of classifications, In the determining step, the action of the avatar is determined using the second learning model in accordance with the determined classification. The information processing method according to claim 2 .

4. The second learning model determines the behavior of the avatar using as input the prompt generated by the first learning model, a character prompt indicating a characteristic of the character of the avatar stored in advance, and a situation prompt indicating a relationship between the character and the user.

4. The information processing method according to claim 2 or 3.

5. In the information processing device, a determination step of monitoring a screen using a first learning model and determining a change in the screen; a determining step of determining an action of an avatar using a second learning model in response to the change in the screen determined in the determining step; an instruction step of instructing the avatar to perform the action determined in the determination step; A program for executing an information processing method including the steps of:

6. An information processing system including an information processing device, a determination step of monitoring a screen using a first learning model and determining a change in the screen; a determining step of determining an action of an avatar using a second learning model in response to the change in the screen determined in the determining step; an instruction step of instructing the avatar to perform the action determined in the determination step; An information processing system that performs the following:

Images