Information processing methods, information processing systems, and programs

Abstract

To provide information processing methods, information processing systems, and programs that enable more natural conversations with users. [Solution] The information processing method involves a server device causing a large-scale language model to generate utterances by a virtual character in response to the content of utterances made by the user of the client device. If the user has not yet initiated the next utterance, the server device causes the virtual character to output the utterance generated by the large-scale language model.

Description

Technical Field

[0001] The present invention relates to an information processing method, an information processing system, and a program.

Background Art

[0002] Patent Document 1 discloses an agent system that interacts with a user.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is room for improvement in enabling more natural conversation with a user.

Means for Solving the Problems

[0005] According to one aspect of the present invention, an information processing method executed by an information processing system is provided. This information processing method causes a large language model to generate the content of a statement by a virtual character according to the content of a statement by a user. And when the next statement by the user has not started, it causes the virtual character to output the content of the statement generated by the large language model.

Brief Description of the Drawings

[0006] [Figure 1] FIG. 1 is a diagram showing an example of the system configuration of an information processing system. [Figure 2] FIG. 2 is a diagram showing an example of the hardware configuration of a server device. [Figure 3] FIG. 3 is a diagram showing an example of the hardware configuration of a client device. [Figure 4]Figure 4 is a sequence diagram (part 1) showing an example of information processing in an information processing system. [Figure 5] Figure 5 shows an example of a user interface in a client device. [Figure 6] Figure 6 is a sequence diagram (part 2) showing an example of information processing in an information processing system. [Modes for carrying out the invention]

[0007] Embodiments of the present invention will be described below with reference to the drawings. The various features shown in the embodiments below (including modified examples; the same applies hereinafter) can be combined with each other.

[0008] <Embodiment 1> 1. System Configuration Diagram Figure 1 shows an example of the system configuration of the information processing system 1000. As shown in Figure 1, the information processing system 1000 includes a server device 100, a client device 110, and a server device 120 as its system configuration. The server device 100 is an example of a computer. The server device 100, the client device 110, and the server device 120 are connected to each other via a network 150. The network 150 may include a WAN (Wide Area Network), a LAN (Local Area Network), and the Internet, or both. The network 150 is configured to allow communication between devices connected to the network 150 via wired and / or wireless connections. The information processing system 1000 is a system that provides so-called SaaS (Software as a Service) functionality.

[0009] The server device 100 performs information processing as described in Embodiment 1, specifically speech processing by an AI avatar, in response to requests from the client device 110, etc. The details of the processing performed by the server device 100 will be explained later using sequence diagrams, etc. Note that the information processing system 1000 may consist of one server device or multiple server devices. If the information processing system 1000 consists of multiple server devices, the functions of the server device 100 are provided as a so-called distributed system.

[0010] The client device 110 is a device operated by an operator and utilizes the functions provided by the server device 100. As will be described later, the client device 110 will be described assuming that a predetermined web browser (hereinafter simply referred to as a web browser) is installed. In Figure 1, a smartphone is shown as an example of the client device 110, but the client device 110 is not limited to a smartphone and may be a PC (Personal Computer) and / or a tablet terminal device. Any device that can input operator information and output the results of processing in the server device 100 is acceptable. In Figure 1, only one client device 110 is shown as an example included in the information processing system 1000, but multiple client devices 110 may be included in the information processing system 1000.

[0011] Server device 120 is a device that provides services for large-scale language models. A large-scale language model is a computer language model composed of an artificial neural network with numerous parameters, and is a language model trained using self-supervised or semi-supervised learning with vast amounts of unlabeled text. Furthermore, a large-scale language model can also be described as an advanced artificial intelligence model that can learn from large amounts of text data and generate, translate, and answer questions using natural language used by humans. The large-scale language model may also be implemented on the server device 100.

[0012] Here, the information processing system described in the claims may consist of multiple devices or of a single device. If the information processing system described in the claims consists of a single device, an example of such a device is, for example, a server device 100. If the information processing system described in the claims consists of multiple devices, examples of the multiple devices are, for example, a distributed system that provides the functions of server device 100, or server device 100 and server device 120.

[0013] 2. Hardware Configuration (1) Hardware configuration of server device 100 Figure 2 shows an example of the hardware configuration of server device 100. As shown in Figure 2, the server device 100 includes, as a hardware configuration, a control unit 210, a storage unit 220, a communication unit 230, and an internal bus 240. The control unit 210, the storage unit 220, and the communication unit 230 are electrically connected via the internal bus 240.

[0014] The control unit 210 is a CPU (Central Processing Unit) or the like, and controls the entire server device 100.

[0015] The storage unit 220 is any one of an HDD (Hard Disk Drive), a ROM (Read Only Memory), a RAM (Random Access Memory), a SSD (Solid State Drive), or any combination thereof, and stores a program and data (for example, text information received from the client device 110, the speech content of the AI (Artificial Intelligence) avatar output from the large language model, data for displaying the AI avatar, etc.) that the control unit 210 uses when executing processing based on the program. The storage unit 220 is an example of a storage medium. In the specification, it is described that the data used when the control unit 210 executes processing based on the program is stored in the storage unit 220, but it may also be stored in the storage unit of another device that can communicate with the server device 100. That is, the data may be stored in the storage unit of any device as long as the control unit 210 can reference or acquire it. By the control unit 210 executing processing based on the program stored in the storage unit 220, the functions of the server device 100 and the processing of the server device 100 in the sequence diagrams shown in FIGS. 4 and 6 described later are realized.

[0016] The communication unit 230 connects the server device 100 to the network 150 and controls communication with other devices.

[0017] The server device 100 may include a plurality of each hardware configuration shown in FIG. 2. For example, the server device 100 may have a plurality of control units. The hardware configuration of the server device 120 is also the same as the hardware configuration of the server device 100.

[0018] (2) Hardware Configuration of the Client Device 110 FIG. 3 is a diagram showing an example of the hardware configuration of the client device 110. As shown in FIG. 3, as a hardware configuration, the client device 110 includes a control unit 310, a storage unit 320, an imaging unit 330, an input / output unit 340, a microphone 350, a speaker 360, a communication unit 370, and an internal bus 380. The control unit 310, the storage unit 320, the imaging unit 330, the input / output unit 340, the microphone 350, the speaker 360, and the communication unit 370 are electrically connected via the internal bus 380.

[0019] The control unit 310 is a CPU or the like and controls the entire client device 110.

[0020] The storage unit 320 is any one of an HDD, a ROM, a RAM, an SSD, or the like, or any combination thereof, and stores a program and data used when the control unit 310 executes processing based on the program. The storage unit 320 is an example of a storage medium. When the control unit 310 executes processing based on the program stored in the storage unit 320, the functions of the client device 110 and the processing of the client device 110 in the sequence diagrams shown in FIGS. 4 and 6 described later are realized.

[0021] The imaging unit 330 is a camera and images a subject. An example of the subject is an operator operating the client device 110. Note that the operator is also referred to as a user. The user is a user of the information processing system 1000 or a user who uses the functions of the server device 100.

[0022] The input / output unit 340 is a touch panel display, and inputs information based on a selection operation and / or an input operation by an operator, or displays input information via a screen by the operator, the result of processing by the control unit 310, and the like.

[0023] The microphone 350 is a device that converts sound into an electrical signal. The microphone 350 converts the sound related to the speech of the user into an electrical signal. Speaker 360 is a device that converts electrical signals into sound. Speaker 360 converts the content of speech generated by a large-scale language model into sound and outputs it.

[0024] The communications unit 370 connects the client device 110 to the network 150 and manages communication with other devices. The client device 110 may include multiple hardware configurations as shown in Figure 3. For example, the client device 110 may have multiple control units.

[0025] 3. Information Processing (1) Overview of the process The control unit 210 causes the large-scale language model to generate the content of the virtual character's utterance in response to the user's utterance. If the user has not yet initiated the next utterance, the control unit 210 causes the virtual character to output the content of the utterance generated by the large-scale language model. Here, an example of a virtual character is an AI avatar. For the sake of simplicity, the specification uses an AI avatar as an example.

[0026] By performing this process, the AI ​​avatar will be able to have more natural conversations with the user.

[0027] (2) Details of the process Figure 4 is a sequence diagram (part 1) showing an example of information processing in the information processing system 1000. In sequence SQ401, the web browser of the client device 110 inputs audio data related to the user's utterance via the microphone 350. Here, the web browser inputs one sentence of utterance from the user as a single unit of audio data. Here, a sentence is the smallest unit of communication that has a single coherent meaning. In sequence SQ402, the web browser of the client device 110 converts the audio data into text information. The web browser converts one sentence of audio data into a single block of text information. The processing in sequence SQ402 is an example of the process by which the web browser of the client device 110 converts the audio data of a user's utterance into text information.

[0028] In sequence SQ403, the web browser of the client device 110 sends text information to the server device 100. The control unit 210 of the server device 100 receives the content of the user's utterance as text information from the client device 110.

[0029] In sequence SQ404, the control unit 210 generates a prompt based on text information received from the client device 110. A prompt is a sentence used to instruct or ask a question to the AI. For example, a prompt may include text information of a sentence spoken by the user. The prompt may also include an instruction to generate speech content for the AI ​​avatar corresponding to the content of the sentence spoken. The processing of sequence SQ404 is an example of the process in which the control unit 210 generates a prompt containing text information of the content of a sentence uttered by the user. In sequence SQ405, the control unit 210 inputs the generated prompt to the large-scale language model. Sequence SQ405, or sequences SQ404 and SQ405, is an example of a process in which the control unit 210 causes a large-scale language model to generate speech content from an AI avatar corresponding to the content of each sentence uttered by the user.

[0030] In sequence SQ406, the control unit 210 receives the speech content of the AI ​​avatar output from the large-scale language model. The control unit 210 temporarily stores the received speech content in a predetermined memory area such as the memory unit 220. In sequence SQ407, the control unit 210 determines whether the user has started their next utterance. For example, if the control unit 210 has not received text information of the user's next utterance from the client device 110 at the time it receives the utterance content of the AI ​​avatar output from the large-scale language model, it determines that the user has not started their next utterance. On the other hand, if the control unit 210 has received text information of the user's next utterance from the client device 110 at the time it receives the utterance content of the AI ​​avatar output from the large-scale language model, it determines that the user has started their next utterance.

[0031] If the control unit 210 determines that the user has not yet initiated the next utterance, in sequence SQ408, it sends output data to the client device 110 to cause the AI ​​avatar to output the utterance as audio data. The output data includes the utterance of the AI ​​avatar output from the large-scale language model, display data related to the shape of the AI ​​avatar's mouth and facial expressions corresponding to the utterance of the AI ​​avatar, and a control signal instructing the microphone 350 to be turned OFF. When the control unit 210 receives information from the client device 110 indicating that the output of the utterance has ended, it sends a control signal to the client device 110 instructing the microphone 350 to be turned ON. This process is an example of a process in which the control unit 210 controls the input of user voice data to the client device 110 to turn OFF while the content of the utterance generated by the large-scale language model is being output to the AI ​​avatar. Furthermore, the processing of sequence SQ408 is an example of the process by which the control unit 210 outputs the content of the utterance generated by the large-scale language model as speech data.

[0032] Upon receiving output data from the server device 100, the client device 110 displays the AI ​​avatar on the input / output unit 340 and outputs the spoken content as audio data from the speaker 360. Figure 5 shows an example of a user interface in the client device 110. The AI ​​avatar 610 is displayed on screen 600. Based on the displayed data, the AI ​​avatar 610 changes the shape of its mouth and its facial expression according to the content of its speech. In addition, the content of the speech is output as audio data (data 620 in Figure 6) from the microphone 350. The content of the speech may also be output as text data along with the audio data.

[0033] When the user speaks in response to the AI ​​avatar 610's speech, the processing from sequence SQ401 is repeated.

[0034] Figure 6 is a sequence diagram (part 2) showing an example of information processing in the information processing system 1000. Figure 6 is intended to illustrate a case in which the determination of sequence SQ407 indicates that the next utterance by the user has begun.

[0035] In sequence SQ501, the web browser of the client device 110 receives audio data related to the next utterance from the user via the microphone 350. In sequence SQ502, the web browser of client device 110 converts the audio data into text information. In sequence SQ503, the web browser of the client device 110 sends text information to the server device 100.

[0036] In sequence SQ407, the control unit 210 determines whether the user has started their next utterance. In the example in Figure 6, at the same time that the AI ​​avatar's utterance output from the large-scale language model is received, the control unit 210 receives text information of the user's next utterance from the client device 110, and therefore determines that the user has started their next utterance.

[0037] If it is determined that the user has started the next utterance, in sequence SQ504, the control unit 210 deletes the utterance content that was received from the large-scale language model in sequence SQ406 and temporarily stored in a predetermined memory area such as the memory unit 220. Then, in sequence SQ505, the control unit 210 generates a prompt based on the text information received from the client device 110 in sequence SQ403 and the text information received from the client device 110 in sequence SQ503. The prompt includes the text information received from the client device 110 in sequence SQ403 and the text information received from the client device 110 in sequence SQ503. The prompt further includes an instruction requesting the generation of speech content for the AI ​​avatar in accordance with the two pieces of text information.

[0038] In sequence SQ506, the control unit 210 inputs the generated prompt into the large-scale language model. The processing of sequences SQ505 and SQ506 is an example of a process in which, if the user starts the next utterance before the content of the utterance generated by the large-scale language model is output to the AI ​​avatar, the control unit 210 causes the large-scale language model to generate the content of the utterance to be delivered by the AI ​​avatar according to the content of the user's utterance and the content of the next utterance. In this way, if the user continues speaking, the AI ​​avatar's speech can be generated based on the content of the previous and subsequent utterances. Therefore, meaningful conversations can be established.

[0039] In sequence SQ507, the control unit 210 receives the speech content of the AI ​​avatar output from the large-scale language model. The control unit 210 determines whether the user has started the next utterance. If it determines that the user has not started the next utterance, the control unit 210 executes the processing of sequence SQ408. If it determines that the user has started the next utterance, the control unit 210 executes the processing of sequence SQ504 and later.

[0040] According to the processing of Embodiment 1, the AI ​​avatar will be able to have more natural conversations with the user.

[0041] (Variation 1) Modification 1 of Embodiment 1 will now be described. Modification 1 is included in Embodiment 1 and is not different from Embodiment 1. Configurations and processes not described in Modification 1 are the same as in Embodiment 1.

[0042] In the modified example 1, the control unit 210, when certain conditions are met, causes the AI ​​avatar to output the content of the utterance generated by the large-scale language model, even if the user has already started speaking again. For example, the control unit 210 determines that a predetermined condition is met if the content of the utterance generated by the large-scale language model contains a predetermined sentence or word indicating urgency. Furthermore, if a time limit is set that the process must be completed by a certain time and minute, the control unit 210 determines that a predetermined condition is met if the set time is reached while the content of the utterance generated by the large-scale language model is being output to the AI ​​avatar. Furthermore, the control unit 210 determines that a predetermined condition is met if it determines, based on the user's image captured by the imaging unit 330, that the user is experiencing a predetermined emotion, such as being irritated, angry, sad, or crying.

[0043] According to Modification 1, if certain conditions are met, the content of the utterance generated by the large-scale language model can be output to the AI ​​avatar even if the user has already started speaking again.

[0044] (Modification 2) Modification 2 of Embodiment 1 will now be described. Modification 2 is included in Embodiment 1 and is not different from Embodiment 1. Configurations and processes not described in Modification 2 are the same as in Embodiment 1.

[0045] In Modification 2, the control unit 210 evaluates the user's role-playing based on the content of the user's utterances. For example, the control unit 210 may evaluate the user's role-playing based on the content of the user's utterances and the content of the AI ​​avatar's utterances in response to the user's utterances. For example, the control unit 210 inputs the content of the user's utterances and the content of the AI ​​avatar's utterances in response to the user's utterances into a trained model. The trained model is a trained model that has been trained using the content of the utterances between the user and the AI ​​avatar as input data and evaluation values ​​indicating the evaluation of the user's role-playing as output data. The control unit 210 may evaluate the user's role-playing based on the evaluation values ​​output from the trained model.

[0046] Another example of user evaluation regarding role-playing is that the control unit 210 may evaluate the user's role-playing based on how many predetermined point values ​​(words) or sentences are included in the user's utterances with the AI ​​avatar. The key words or sentences may differ depending on the type of role-playing. Furthermore, the key words or sentences may be appropriately selected from designated documents (scripts related to the role-playing) depending on the type of role-playing selected via the screen or other means.

[0047] In Modification 2, the control unit 210 may further evaluate the user's role-playing based on the user's facial expressions captured by the imaging unit 330. For example, the control unit 210 may evaluate the user's role-playing based on how long the user smiled during conversation with the AI ​​avatar, the number of smiles, and / or the duration for which a smile was detected. For example, the control unit 210 may increase the value indicating the evaluation of the user's role-playing the more times the user smiled and / or the longer the duration for which a smile was detected.

[0048] According to Modification 2, it is possible to evaluate the user's role-playing by using conversations with an AI avatar.

[0049] (Variation 3) Modification 3 of Embodiment 1 will now be described. Modification 3 is included in Embodiment 1 and is not different from Embodiment 1. Configurations and processes not described in Modification 3 are the same as in Embodiment 1.

[0050] Some or all of the processing performed by the server device 100 may be performed by the client device 110. In such a configuration, the large-scale language model may be implemented on a server device 120 different from the client device 110, or it may be implemented on the client device 110. If the client device 110 performs all the processing that the server device 100 would normally perform, the control unit 310 converts the user's utterance audio data into text information. Then, based on the text information, the control unit 310 causes the large-scale language model to generate utterances from a virtual character that correspond to the content of the user's utterances. If the user has not yet started their next utterance, the control unit 310 causes the virtual character to output the utterance generated by the large-scale language model.

[0051] The effects of Embodiment 1 described above can also be achieved with the configuration of Modification 3.

[0052] <Note> This embodiment includes the following disclosures.

[0053] (Note 1) An information processing method performed by an information processing system, Based on the content of the user's utterance, a large-scale language model generates the content of utterances by a virtual character. If the user has not yet initiated the next utterance, the content of the utterance generated by the large-scale language model is output to the virtual character. Information processing methods.

[0054] (Note 2) The information processing method described in Appendix 1, For each sentence uttered by the user, the large-scale language model generates the content of the utterance by the virtual character corresponding to the content of each sentence uttered. Information processing methods.

[0055] (Note 3) The information processing method described in Appendix 1 or Appendix 2, The content of the user's utterance is received as text information from the client device. Information processing methods.

[0056] (Note 4) The information processing method described in Appendix 3, The client device converts the voice data of the user's speech into text information. Information processing methods.

[0057] (Note 5) The information processing method described in Appendix 3, A prompt is generated that includes text information of the content of a sentence uttered by the user. The prompt further includes an instruction requesting the generation of speech content for the virtual character corresponding to the content of the sentence utterance, The prompt is input to the large-scale language model. Information processing methods.

[0058] (Note 6) An information processing method described in any one of the appendices 1 to 5, If certain conditions are met, even if the user has started the next utterance, the content of the utterance generated by the large-scale language model will be output to the virtual character. Information processing methods.

[0059] (Note 7) An information processing method described in any one of the appendices 1 to 6, Based on the content of the user's utterances, an evaluation of the user's role-playing is performed. Information processing methods.

[0060] (Note 8) An information processing method described in any one of the appendices 1 to 7, If the user initiates the next utterance before the content of the utterance generated by the large-scale language model is output to the virtual character, the large-scale language model is instructed to generate the content of the virtual character's utterance in accordance with the content of the user's previous utterance and the content of the next utterance. Information processing methods.

[0061] (Note 9) The information processing method described in Appendix 4, While the content of the utterance generated by the large-scale language model is being output to the virtual character, the input of the user's voice data to the client device is controlled to be OFF. Information processing methods.

[0062] (Note 10) An information processing method described in any one of the appendices 1 to 9, The aforementioned virtual character is an AI (Artificial Intelligence) avatar. Information processing methods.

[0063] (Note 11) An information processing method described in any one of the appendices 1 to 10, The content of the utterance generated by the large-scale language model is output as audio data. Information processing methods.

[0064] (Note 12) An information processing system, Having at least one control unit, The control unit, Based on the content of the user's utterance, a large-scale language model generates the content of utterances by a virtual character. If the user has not yet initiated the next utterance, the content of the utterance generated by the large-scale language model is output to the virtual character. Information processing system.

[0065] (Note 13) An information processing device, Having at least one control unit, The control unit, Converts the audio data of the user's speech into text information. Based on the aforementioned text information, a large-scale language model generates the content of speech by a virtual character corresponding to the content of speech by the user. If the user has not yet initiated the next utterance, the content of the utterance generated by the large-scale language model is output to the virtual character. Information processing device.

[0066] (Note 14) It is a program, On the computer, A program for executing the information processing method described in any one of the appendices 1 through 11.

[0067] Although embodiments have been described above, these are presented as examples and are not intended to limit the scope of the invention. Novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. The embodiments are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of symbols]

[0068] 100: Server device 110: Client device 150: Network 210: Control Unit 220: Storage section 230: Communications Department 1000: Information Processing System

Claims

1. An information processing method performed by an information processing system, Based on the content of the user's utterance, a large-scale language model generates the content of utterances by a virtual character. If the user has not yet initiated the next utterance, the content of the utterance generated by the large-scale language model is output to the virtual character. Information processing methods.

2. The information processing method according to claim 1, For each sentence uttered by the user, the large-scale language model generates the content of the utterance by the virtual character corresponding to the content of each sentence uttered. Information processing methods.

3. The information processing method according to claim 1, The content of the user's utterance is received as text information from the client device. Information processing methods.

4. The information processing method according to claim 3, The client device converts the voice data of the user's speech into text information. Information processing methods.

5. The information processing method according to claim 3, A prompt is generated that includes text information of the content of a sentence uttered by the user. The prompt further includes an instruction requesting the generation of speech content for the virtual character corresponding to the content of the sentence utterance, The prompt is input to the large-scale language model. Information processing methods.

6. The information processing method according to claim 1, If certain conditions are met, even if the user has started the next utterance, the content of the utterance generated by the large-scale language model will be output to the virtual character. Information processing methods.

7. The information processing method according to claim 1, Based on the content of the user's utterances, an evaluation of the user's role-playing is performed. Information processing methods.

8. The information processing method according to claim 1, If the user initiates the next utterance before the content of the utterance generated by the large-scale language model is output to the virtual character, the large-scale language model is instructed to generate the content of the virtual character's utterance in accordance with the content of the user's previous utterance and the content of the next utterance. Information processing methods.

9. The information processing method according to claim 4, While the content of the utterance generated by the large-scale language model is being output to the virtual character, the input of the user's voice data to the client device is controlled to be OFF. Information processing methods.

10. The information processing method according to claim 1, The aforementioned virtual character is an AI (Artificial Intelligence) avatar. Information processing methods.

11. The information processing method according to claim 1, The content of the utterance generated by the large-scale language model is output as audio data. Information processing methods.

12. An information processing system, Having at least one control unit, The control unit, Based on the content of the user's utterance, a large-scale language model generates the content of utterances by a virtual character. If the user has not yet initiated the next utterance, the content of the utterance generated by the large-scale language model is output to the virtual character. Information processing system.

13. An information processing device, Having at least one control unit, The control unit, Converts the audio data of the user's speech into text information. Based on the aforementioned text information, a large-scale language model generates the content of speech by a virtual character corresponding to the content of speech by the user. If the user has not yet initiated the next utterance, the content of the utterance generated by the large-scale language model is output to the virtual character. Information processing device.

14. It is a program, On the computer, A program for executing the information processing method described in any one of claims 1 to 11.

Images