Information processing method, program and information processing system

Abstract

To converse with an avatar more naturally.SOLUTION: An information processing method is executed by an information processing device, and includes: a storing step for vectorizing a document of a conversation and associatively storing the document of the conversation and the vector of the document; an acquiring step for acquiring a voice of a user; a converting step for performing text conversion and vector conversion on the acquired voice; an extracting step for extracting a vector of the document corresponding to the vector obtained by the conversion; a generating step for generating an answer to the voice acquired based upon the document of the conversation corresponding to the extracted vector of the document; and an uttering step for uttering the generated answer.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing method, a program, and an information processing system. [Background technology]

[0002] Patent Document 1 discloses a training system that incorporates communication AI and is intended to support the education of professionals who require conversation skills. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-14188 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above prior art requires processes such as converting the user's speech into text, performing language processing on the converted text to generate a response, and then converting the generated response into speech. These processes require a long time for the avatar to respond after the user speaks, making it difficult to have a natural conversation. Furthermore, since the time it takes for the avatar to speak varies depending on the complexity of the user's speech, a more natural conversation is required, using filler words or other appropriate phrases for the avatar to respond in response to the time.

[0005] In consideration of the above, an object of the present invention is to provide an information processing method, a program, and an information processing system that enable a user to have a more natural conversation with an avatar. [Means for solving the problem]

[0006] According to one embodiment of an information processing method, the information processing method is executed by an information processing device, and includes a storage step of vectorizing sentences in a conversation and storing the sentences of the conversation in correspondence with the vectors of the sentences; an acquisition step of acquiring a user's voice; a conversion step of converting the acquired voice into text and vector converting it; an extraction step of extracting vectors of the sentences corresponding to the converted vectors; a generation step of generating an answer to the acquired voice based on the sentences of the conversation that correspond to the extracted vectors of the sentences; and a speaking step of speaking the generated answer.

[0007] According to one embodiment of the program, an information processing device is caused to execute an information processing method, the information processing method including a storage step of vectorizing sentences in a conversation and storing the sentences of the conversation in correspondence with the vectors of the sentences; an acquisition step of acquiring a user's voice; a conversion step of converting the acquired voice into text and vector converting it; an extraction step of extracting vectors of the sentences corresponding to the converted vectors; a generation step of generating an answer to the acquired voice based on the sentences of the conversation corresponding to the extracted vectors of the sentences; and a speech step of speaking the generated answer.

[0008] According to one embodiment of the information processing system, the information processing system is executed by an information processing device, and the information processing method is executed by the information processing device, and includes a storage step of vectorizing sentences in a conversation and storing the sentences of the conversation in correspondence with the vectors of the sentences; an acquisition step of acquiring a user's voice; a conversion step of converting the acquired voice into text and vector converting it; an extraction step of extracting vectors of the sentences corresponding to the converted vectors; a generation step of generating an answer to the acquired voice based on the sentences of the conversation that correspond to the extracted vectors of the sentences; and a speaking step of speaking the generated answer. [Effects of the Invention]

[0009] According to one embodiment, it is possible to reduce the time it takes for an avatar to respond after a user speaks, allowing for a more natural conversation with the avatar. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a server according to an embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a user terminal according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of a server according to an embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of a functional configuration of a user terminal according to an embodiment. [Figure 6] 10 is a flowchart illustrating an example of a process executed by the information processing system according to the embodiment until an avatar speaks. [Figure 7] 10 is a flowchart illustrating an example of a process performed by the information processing system according to the embodiment, in which an avatar speaks when a new answer is generated. [Figure 8] 1 is a diagram showing an overview of a role-playing system according to an embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0011] An embodiment of an information processing system according to the present invention will be described below with reference to Figures 1 to 5. In each drawing, the same or equivalent components and parts are denoted by the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0012] (System Overview) First, an overview of an information processing system 10 according to this embodiment will be described. The information processing system 10 according to this embodiment is a system that provides a role-playing environment and supports human resource training for occupations that require conversation skills. In this embodiment, the information processing system 10 is a system that can shorten the time between when a user speaks and when the avatar responds in role-playing using an avatar. This allows for more natural conversations with the avatar.

[0013] (System Configuration) Fig. 1 is a diagram showing an example of the configuration of an information processing system 10 according to this embodiment. As shown in Fig. 1, the information processing system 10 according to this embodiment includes a server device 1 and a user terminal 2, which are connected to each other so as to be able to communicate with each other via a network N. The network N is, for example, a wired local area network (LAN), a wireless LAN, the Internet, a public line network, a mobile data communication network, or a combination thereof. In the example of Fig. 1, the information processing system 10 includes one server device 1 and one user terminal 2, but may include multiple of each.

[0014] The server device 1 is an example of an information processing device that provides a role-playing environment, generates answers from the user's voice input at the user terminal 2, and causes an avatar to speak the generated answers. The server device 1 may be a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a microcomputer, or a combination of these. The specific configuration and operation of the server device 1 will be described later.

[0015] The user terminal 2 is an example of an information processing device that performs operations for inputting and displaying various types of information. The user terminal 2 may be a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a microcomputer, a wearable device, or a combination of these. In this embodiment, an information processing device equipped with a voice input function and a photographing function is used as an example, but other functions are not limited as long as the voice input function is provided.

[0016] (Hardware configuration - Server) 2 is a block diagram showing the hardware configuration of the server device 1. The server device 1 includes a processor 101, a memory 102, a storage 103, and a communication I / F 104, which are communicably connected to each other via a bus B.

[0017] The processor 101 controls each component of the server device 1 and realizes the functions of the server device 1 by loading various programs stored in the storage 103 into the memory 102 and executing them. The programs executed by the processor 101 include, but are not limited to, an OS (Operating System) and various programs described below. Execution of these programs by the processor 101 realizes part of the state visualization method according to this embodiment. The processor 101 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), or a combination thereof.

[0018] The memory 102 is, for example, a read-only memory (ROM), a random access memory (RAM), or a combination thereof. The ROM is, for example, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a combination thereof. The RAM is, for example, a dynamic random access memory (DRAM), a static random access memory (SRAM), a magnetoresistive random access memory (MRAM), or a combination thereof.

[0019] The storage 103 stores an OS, various programs (described later), and various data. The storage 103 is, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), a storage class memory (SCM), or a combination of these.

[0020] The communication I / F 104 is an interface for connecting the server device 1 to external devices including the user terminal 2 via the network N and controlling communication. The communication I / F 104 is, for example, an adapter compliant with Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), Ethernet (registered trademark), or optical communication (e.g., Fibre Channel), but is not limited to these.

[0021] (Hardware configuration - user terminal) 3 is a block diagram showing the hardware configuration of the user terminal 2. The user terminal 2 includes a processor 201, a memory 202, a storage 203, a communication I / F 204, an input / output I / F 205, an input device 206, and an output device 207, which are communicatively connected to each other via a bus B. The communication I / F 204 is an interface for connecting the user terminal 2 to external devices including the server device 1 via a network N and for controlling communication. The communication I / F 204 is, for example, an adapter compliant with Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), Ethernet (registered trademark), or optical communication (e.g., Fibre Channel), but is not limited to these.

[0022] The input / output I / F 205 is an interface for connecting an input device 206 and an output device 207 to the user terminal 2. The input device 206 is, for example, a mouse, a keyboard, a touch panel, a microphone, a scanner, a camera, various sensors, operation buttons, or a combination of these. The output device 207 as a user interface is, for example, a display, a projector, a printer, a speaker, a vibrator, or a combination of these. In this embodiment, as an example, the output device 207 and the input device 206 are an integrally configured touch panel display.

[0023] In this embodiment, the program may be written to the memory 202 or the storage 203 during the manufacturing stage of the server device 1, or may be provided to the server device 1 via the network N. Alternatively, the program may be provided to the server device 1 via a non-transitory computer-readable recording medium such as a disk medium (not shown).

[0024] (Functional Configuration - Server) Next, the functional configuration of the server device 1 will be described. FIG. 4 is a diagram illustrating an example of the functional configuration of the server device 1. When executing various programs, the server device 1 realizes various functions using the above-mentioned hardware resources. The server device 1 has a communication unit 11, a storage unit 12, and a control unit 13 as functional components realized by the server device 1. Each functional component is realized by the processor 101 reading and executing a program stored in the memory 102 or the storage 103. The storage unit 12 also vectorizes conversation sentences that can be used in role-playing games and stores conversation sentences 121 and corresponding sentence vectors 122. The storage unit 12 also stores time adjustment sentences 123 that are used by an avatar to speak when a predetermined time is required to generate an answer from a user's voice. The conversation sentences 121 are pairs of exchanges in a conversation, and the storage unit 122 also stores the speech at the beginning of each pair of answers. The conversation sentences to be stored may be set for each theme, for example, ice-breaking, closing, product explanation, and other various themes may be set, and conversation sentences 121 that correspond to the theme may be stored together with the theme. Furthermore, the time adjustment sentences 123 are sentences that can buy time, such as "Please wait a moment" or "I'll think about it for a moment," and are stored in multiple patterns. Furthermore, the time adjustment sentences for adjusting the time are not stored in the memory unit 12, but are dynamically generated.

[0025] The control unit 13 includes an acquisition unit 131 that acquires voice sent from the user terminal 2, a conversion unit 132 that converts the voice acquired by the acquisition unit 131 into text and vectorizes it, and an extraction unit 133 that extracts a sentence vector 122 that is close to (corresponding to) the vector converted by the conversion unit 132. The control unit 13 also includes a generation unit 134 that generates a corresponding conversation sentence from the sentence vector 122 extracted by the extraction unit 133, and a speaking unit 135 that causes an avatar to speak the generated answer. These will be described in detail later. The generation unit 134 generates an answer using, for example, LLM (Large Language Models).

[0026] (Functional configuration - User terminal) Next, the functional configuration of the user terminal 2 will be described. Figure 5 is a diagram showing an example of the functional configuration of the user terminal 2. When executing various programs, the user terminal 2 uses the above-mentioned hardware resources to realize various functions. The user terminal 2 has, as functional components realized by the user terminal 2, a communication unit 21, a storage unit 22 in which a program 221 is stored, and a control unit 23. Each functional component is realized by the processor 201 reading and executing the program 221 stored in the memory 202 or the storage 203. The control unit 23 includes an information acquisition control unit 231 that acquires information sent from the server device 1, and a display unit 232 that displays the acquired information on the output device 207.

[0027] Next, the flow of processing in the information processing system according to this embodiment will be described with reference to Figs. 6 to 8. Fig. 6 is a flowchart showing an example of the process executed by the information processing system 10 according to this embodiment until an avatar speaks. Fig. 7 is a flowchart showing an example of the process executed by the information processing system 10 according to this embodiment until an avatar speaks when a new answer is generated. Fig. 8 is a diagram showing an overview of the role-playing system according to this embodiment.

[0028] First, referring to FIG. 6, a process from when a user speaks until an avatar speaks will be described. Specifically, a process from when a user speaks until an avatar responds in a role-playing game using an avatar will be described. In step S101, the acquisition unit 131 acquires the user's speech input to the user terminal 2. Next, in step S102, the conversion unit 132 converts the user's speech acquired in step S101 into text. Next, in step S103, the conversion unit 132 vectorizes the converted text. Here, when converting speech into text, any software capable of converting speech into text, such as speech recognition software, may be used, and the software is not limited to a specific one. Furthermore, the vectorization of text may be, for example, vectorization of natural language processing, but is not limited to this, and other methods may also be used.

[0029] Next, in step S104, the extraction unit 133 determines a theme stored in the storage unit 12 from the vector converted in step S103. Next, in step S105, the extraction unit 133 extracts a sentence vector 122 that is close to (corresponding to) the vector converted in step S103 for the theme determined in step S104. Next, in step S106, the speech unit 135 causes speech of the beginning part of an answer in the conversation sentence 121 that is paired with the sentence vector 122 in the conversation sentence 121 corresponding to the extracted sentence vector 122. Next, in step S107, while the beginning part of the answer is being spoken in step S106, the generation unit 134 generates the rest of the answer to the user's voice. At this time, the generation unit 134 generates an answer using, for example, LLM (Large Language Models), but is not limited to this and another method may be used. Next, in step S108, speaking unit 135 causes the avatar to speak the remaining sentences of the answer generated in step S107.

[0030] In this embodiment, conversation themes and conversational exchanges (e.g., questions and answers) are stored as pairs, and sentence vectors obtained by vectorizing the conversation sentences are also stored in advance. This reduces the time it takes to generate a response to a user's voice. By storing the speech of the initial sentence of the stored answer sentence together with the speech of the initial sentence, the answer can be immediately uttered in response to the user's voice. By uttering the initial sentence first, the time between the user's speech and the avatar's speech can be reduced. Furthermore, by storing themes and the conversations corresponding to the themes in advance, when extracting the text vectors of the user's voice, the corresponding vectors are extracted from the theme the user is currently speaking, thereby reducing the time it takes to extract the text vectors. In other words, conventional systems require processes such as converting the user's voice into text, performing language processing on the converted text to generate a response, and then converting the generated response into speech. In contrast, this embodiment reduces the time it takes from the user's speech to the avatar's response, allowing for a more natural conversation with the avatar.

[0031] Next, a case where the sentence vector 122 extracted in step S105 of Fig. 6 is not within a predetermined range from the vector of the user's voice will be described using the flowchart of Fig. 7. The processing from step S101 to step S105 of Fig. 7 is the same as the processing from step S101 to step S104 of Fig. 6, and therefore a description thereof will be omitted. After step S105, in step S109, the extraction unit 133 determines whether the sentence vector 122 extracted in step S105 is within a predetermined range from the vector of the user's voice. If it is within the predetermined range (YES), the processing is the same as steps S106 to S108 of Fig. 6, and therefore a description thereof will be omitted.

[0032] On the other hand, if it is not within the predetermined range (NO), the process proceeds to step S110, where the utterance unit 135 utters the time adjustment sentence 123. Then, in step S111, the generation unit 134 generates a new answer to the voice. At this time, for example, the answer is generated based on the text converted in step S102 using an LLM (large-scale language model). If the extracted sentence vector 122 is not within the predetermined range, it may not be appropriate as an answer to the user's voice, so a new answer needs to be generated. Note that since the theme of the conversation has been determined in step S104, an answer that is in line with the theme can be generated for the user's voice. When generating a new answer, in step S106, the utterance unit 135 utters the beginning part first, even if the new answer is still being generated. Then, in step S107, the generation unit 134 generates the remaining answer.

[0033] In this way, even if the sentence vector 122 extracted in step S105 is not within a predetermined range from the user's voice vector, an appropriate new answer can be generated and uttered by the avatar without an inappropriate (off-center) answer. Even if it takes time to generate a new answer, a more natural conversation is possible by storing the time adjustment sentence 123 in advance and having the avatar utter the sentence during that time. In other words, if it takes time to generate a new answer, silence continues, making it difficult for the user to tell whether the role-playing is progressing properly. Therefore, by preparing the time adjustment sentence 123 in advance, even if it takes time to generate a new answer, the user can utter a sentence that can buy time, and understand that the role-playing is not stopping midway. Furthermore, the user can be made to not notice that it is taking time to generate a new answer, allowing for a more natural conversation.

[0034] Next, an example of a screen of the role-playing system according to this embodiment is shown in FIG. 8. The screen of the role-playing system includes areas A1 and A2. Area A1 contains an avatar screen im01, a user screen im02, and a progress screen im03. Area A2 contains a retry button B1, a pause button B2, and a complete button B3. When role-playing begins, the screen shown in FIG. 8 appears, and a conversation with the avatar displayed on avatar screen im01 begins. During the conversation, the user himself is displayed on user screen im02. Progress screen im03 may display progress information, such as the type of conversation with the avatar, as well as a real-time evaluation of the conversation. The type of conversation may be determined by converting the voice of the conversation between the user and the avatar into text and extracting vectors similar to the vectors obtained by vector-converting the converted text from the text vectors 122 stored in storage unit 12.

[0035] Furthermore, if the user wishes to start the conversation with the avatar over again, the user can do so by pressing the retry button B1, and if the user wishes to pause the conversation, the user can do so by pressing the pause button B2. When the role-playing is complete, the user presses the complete button B3. Although not shown, after the role-playing is completed by pressing the complete button B3, a screen may be presented that displays the entire role-playing, including an evaluation of the role-playing as a whole, showing necessary skills, conversation omissions, good points, and areas for improvement.

[0036] In this manner, in this embodiment, by storing the conversation theme and the conversational approach in advance, and also storing the vector of the conversational sentence in advance, the time from when the user speaks until the avatar responds in role-playing using an avatar can be shortened. Also, by storing the audio of the first sentence of the stored conversational response sentence, it becomes possible to respond immediately to the user's voice. Furthermore, by storing the time adjustment sentence 123, even if it takes time to generate a response, the avatar's speech does not stop unnaturally, and a more natural conversation with the avatar can be carried out.

[0037] As described above, according to this embodiment, in role-playing using an avatar, it is possible to reduce the time it takes for the avatar to respond after the user speaks.

[0038] Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications, improvements, etc. within the scope of achieving the object of the present invention are included in the present invention.

[0039] Furthermore, for example, the above-described series of processes can be executed by hardware or software. In other words, the functional configuration is merely an example and is not particularly limited. That is, it is sufficient that the information processing system has the function of being able to execute the above-described series of processes as a whole, and there is no particular limit to the type of functional block used to realize this function. Furthermore, the location of the functional block is also not particularly limited and may be arbitrary. For example, a functional block of a server may be transferred to a user terminal, etc. Conversely, a functional block of a user terminal may be transferred to a server, etc. Furthermore, one functional block may be configured as a single piece of hardware, a single piece of software, or a combination thereof.

[0040] Furthermore, for example, when a series of processes is executed by software, the programs constituting the software are installed onto a computer or the like from a network or a recording medium. The computer may be a computer incorporated into dedicated hardware. Furthermore, the computer may be a computer capable of executing various functions by installing various programs thereon, such as a server, a general-purpose smartphone, or a personal computer.

[0041] Furthermore, for example, the recording medium containing such a program may be configured not only as a removable medium (not shown) that is distributed separately from the device main body in order to provide the program to the user, but also as a recording medium that is provided to the user in a state that is pre-installed in the device main body.

[0042] In this specification, the steps of describing a program to be recorded on a recording medium include not only processes that are performed chronologically in accordance with the order, but also processes that are not necessarily performed chronologically but are performed in parallel or individually. In addition, in this specification, the term "system" refers to an overall device that is made up of a plurality of devices, a plurality of means, etc. [Explanation of symbols]

[0043] 10 Information Processing Systems 1 server 2. User terminal 131 Acquisition Department 132 conversion unit 133 Extraction part 134 generation part 135 speech unit

Claims

1. An information processing method executed by an information processing device, a storage step of vectorizing sentences in a conversation and storing the sentences of the conversation and the vectors of the sentences in correspondence with each other; an acquisition step of acquiring a user's voice; a conversion step of converting the acquired speech into text and vectors; an extraction step of extracting vectors of the sentence corresponding to the transformed vectors; a generation step of generating a response to the acquired voice based on a sentence of the conversation corresponding to the extracted sentence vector; a speaking step of speaking the generated answer; An information processing method, including:

2. In the storing step, the sentences of the conversation are stored according to a theme; In the extraction step, the theme is determined from the converted vector, and the vector of the sentence is extracted from the determined theme. The information processing method according to claim 1 .

3. In the storing step, a voice of a beginning part of a sentence of a response in the conversation is further stored, In the speaking step, while the speech of the beginning part is being spoken, in the generating step, the remaining sentence of the answer is generated. The information processing method according to claim 1 .

4. In the extraction step, if the vector of the extracted sentence is not within a predetermined range from the converted vector, a new response to the speech is generated using natural language processing. The information processing method according to any one of claims 1 to 3.

5. The storing step further stores a sentence for adjusting the time, In the utterance step, when the new answer is generated, the sentence for adjusting the time is uttered. The information processing method according to claim 4.

6. When the new answer is generated in the generating step, the converted vector and the vector of the new answer are associated with each other and further stored in the storing step. The information processing method according to claim 4.

7. In the information processing device, An information processing method executed by an information processing device, a storage step of vectorizing sentences in a conversation and storing the sentences of the conversation and the vectors of the sentences in correspondence with each other; an acquisition step of acquiring a user's voice; a conversion step of converting the acquired speech into text and vectors; an extraction step of extracting vectors of the sentence corresponding to the transformed vectors; a generation step of generating a response to the acquired voice based on a sentence of the conversation corresponding to the extracted sentence vector; a speaking step of speaking the generated answer; A program for executing an information processing method, including:

8. An information processing system executed by an information processing device, An information processing method executed by an information processing device, a storage step of vectorizing sentences in a conversation and storing the sentences of the conversation and the vectors of the sentences in correspondence with each other; an acquisition step of acquiring a user's voice; a conversion step of converting the acquired speech into text and vectors; an extraction step of extracting vectors of the sentence corresponding to the transformed vectors; a generation step of generating a response to the acquired voice based on a sentence of the conversation corresponding to the extracted sentence vector; a speaking step of speaking the generated answer; An information processing system that performs the following:

Images