Information processing equipment, methods, programs and systems

Abstract

The goal is to reduce the unnatural intonation of AI avatars and achieve natural conversations similar to those between humans. [Solution] The program causes the processor to perform the following steps: input the user's utterance into a generating AI and output a response sentence to the utterance; analyze the outputted response sentence character by character and divide the response sentence at each punctuation mark that is determined; when the response sentence is divided from the first character to the first punctuation mark, convert that range into a first response voice data as the first unit; present the first response voice data to the user; when the conversion to the first response voice data is complete, convert the character from the last character of the first unit to the last punctuation mark that was divided before the completion of the first conversion into a second response voice data as the second unit; and present the second response voice data to the user.

Description

Technical Field

[0004] , , ,

[0005] ,

[0001] The present disclosure relates to an information processing apparatus, method, program, and system.

Background Art

[0002] In Patent Document 1, the information processing apparatus executes control to acquire a first input input from a user by a control unit of the information processing apparatus, output a first response by sound information by the control unit, and output a second response to be displayed by the control unit based on a language model and the first input. When the language model unit receives an input sentence based on the dialogue input sentence, the language model unit executes inference processing. The language model unit outputs a generated sentence based on the input sentence. The control unit of the information processing apparatus executes generated sentence splitting processing. In the generated sentence splitting processing, the control unit of the information processing apparatus splits the generated sentence based on punctuation marks of the generated sentence, not limited to but as an example, and generates a first generated sentence and a second generated sentence. The information processing apparatus returns a response without waiting for the completion of the inference processing in the language model, thereby smoothly advancing the dialogue and enabling the user to comfortably conduct the dialogue.

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, there is a description of splitting a generated sentence based on punctuation marks and outputting a first response sentence and a second response sentence. When the text separated based on punctuation marks is individually vocalized, when the AI avatar speaks continuously, the intonation frequently changes, and the user receives an unnatural impression of the way the AI avatar speaks.

[0005] An object of the present disclosure is to reduce the unnatural intonation of an AI avatar and realize a natural conversation like a conversation between humans.

Means for Solving the Problems

[0006] The program is executed by a processor that performs the following steps: inputs the user's utterance into a generating AI and has the generating AI output a response sentence to the utterance; analyzes the response sentence output by the generating AI character by character from the first character, and each time a punctuation mark is determined in the response sentence, divides the response sentence according to that punctuation mark; when the part of the response sentence from the first character to the first punctuation mark is divided, converts the part from the first character to the first punctuation mark into the first response audio data as the first unit; presents the first response audio data to the user; and when the conversion to the first response audio data is complete, converts the part of the response sentence from the character following the end of the first unit up to the last punctuation mark that was divided before the completion of the conversion to the first response audio data into the second response audio data as the second unit; and presents the second response audio data to the user. [Effects of the Invention]

[0007] According to this disclosure, it is possible to reduce the unnatural intonation of AI avatars and achieve natural conversations that resemble those between humans. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example of the overall configuration of System 1. [Figure 2] This is a block diagram showing an example of the functional configuration of terminal device 10. [Figure 3] This block shows a functional configuration example for server 20. [Figure 4] This diagram shows the data structure of User Table 2021. [Figure 5] This flowchart illustrates an example of interaction between a user and an AI avatar in System 1. [Figure 6] Figure 5 is a flowchart detailing the steps of step S1004 (presenting a response). [Figure 7]This is an example of a screen display when a user interacts with an avatar on terminal device 10. [Figure 8] A block diagram showing the basic hardware configuration of Computer 90. [Modes for carrying out the invention]

[0009] The embodiments of this disclosure will be described below with reference to the drawings. In all the drawings illustrating the embodiments, common components are denoted by the same reference numerals, and repeated explanations are omitted. The following embodiments are not intended to unduly limit the content of this disclosure as described in the claims. Not all components shown in the embodiments are necessarily essential components of this disclosure. Also, each drawing is a schematic diagram and is not necessarily a strict illustration.

[0010] Furthermore, in the following description, "processor" refers to one or more processors. At least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may be another type of processor such as a GPU (Graphics Processing Unit). At least one processor may be single-core or multi-core.

[0011] Furthermore, at least one processor may be a broad-sense processor, such as a hardware circuit that performs some or all of the processing (e.g., an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)).

[0012] Furthermore, in the following explanation, we may use expressions such as "xxx table" to describe information that yields an output for a given input. This information can be data with any structure, or it can be a learning model such as a neural network that generates an output for a given input. Therefore, "xxx table" can be referred to as "xxx information."

[0013] In the following description, the configuration of each table is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table.

[0014] In the following description, the "program" may be used as the subject to describe the process. However, since the program performs the defined process by being executed by a processor while appropriately using a storage unit and / or an interface unit, etc., the subject of the process may be the processor (or a device such as a controller having the processor).

[0015] The program may be installed in a device such as a computer, or may be, for example, in a program distribution server or a computer-readable (e.g., non-temporary) recording medium. Also, in the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0016] In the following description, an identification number is used as identification information for various objects, but other types of identification information (e.g., an identifier including letters or symbols) may be adopted.

[0017] [[ID=e17]]In the following description, when describing elements of the same type without distinction, reference signs (or common signs among the reference signs) are used, and when describing elements of the same type by distinction, the identification numbers (or reference signs) of the elements may be used.

[0018] In the following description, the control lines and information lines indicate those considered necessary for the description, and not necessarily all control lines and information lines on the product are shown. All components may be interconnected.

[0019] Each information processing device is composed of a computer having an arithmetic unit and a storage unit. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. Regarding each of the server 20 and the terminal device 10, descriptions overlapping with the basic hardware configuration of the computer and the basic functional configuration of the computer described later will be omitted.

[0020] <Overview> In the system according to this embodiment, in a voice dialogue with an AI avatar, the text output from the generation AI is segmented into predetermined blocks (for example, punctuation unit). The text of the first block in a predetermined utterance is output to the AI avatar by performing voice conversion on only the text. The text of subsequent blocks is collectively voice-converted and output to the AI avatar by collecting the blocks accumulated while the voice of the previous unit is being output from the AI avatar. Thereby, while reducing the waiting time for the AI avatar to start speaking, the unnatural intonation generated by voice conversion for each segmented block is reduced. Therefore, a natural tempo and smooth conversation similar to a conversation between humans are realized.

[0021] In this embodiment, the continuous user's utterance and the AI's response are collectively referred to as one dialogue. This embodiment mainly takes an example where the dialogue is an employment interview, but all dialogues are assumed in this embodiment. One dialogue is, for example, 10 minutes of casual conversation, 30 minutes of business negotiation, 60 minutes of media interview, and in addition, interview, interview, hearing, sales, telephone reception, primary consultation window for professional services, role-play (for example, customer service practice). The avatar is, for example, an interface that has a human appearance and voice and can interact with the members based on the output data from the generation AI model. The avatar is, for example, an icon of the appearance of an office worker in the office.

[0022] <1. Configuration diagram of the entire system> Figure 1 is a block diagram showing an example of the overall configuration of System 1. As shown in Figure 1, System 1 includes, for example, a terminal device 10, a server 20, and a generation AI system 30. The terminal device 10, the server 20, and the generation AI system 30 are connected by communication via, for example, a network 80.

[0023] In Figure 1, for the sake of illustration simplicity, an example is shown where System 1 includes two terminal devices 10. However, it is not limited to this; in reality, it may include one terminal device 10 for use by multiple users, or it may include three or more terminal devices 10. Similarly, the server 20 and the generation AI system 30 may each consist of a single device, or they may be configured as a collection of multiple devices. For example, the multiple functions required to realize the server 20 can be distributed to one or more physical servers depending on the processing capacity of each hardware. Although an example is shown where System 1 includes one generation AI system 30, the number of generation AI systems 30 included in System 1 is not limited to one. There may be two or more generation AI systems 30 included in System 1. Also, although Figure 1 shows an example where the generation AI system 30 is independent of the server 20, the server 20 may include the functions of the generation AI system 30, that is, it may store the generation AI.

[0024] Terminal device 10 is an information processing device used by users of the conversational service provided by server 20. In this embodiment, the user may be rephrased as an applicant for a job interview conducted through the conversational service. Terminal device 10 can be implemented as, for example, a mobile device such as a smartphone or tablet, a stationary PC (Personal Computer), or a laptop PC. Terminal device 10 can be implemented as a computer equipped with hardware such as a processor 19, memory 15, storage 16, output device 14, communication interface 12, and input device 13.

[0025] Server 20 is an information processing device used by a service provider that provides conversational services (hereinafter referred to as "service provider"). In this embodiment, the service provider is described as an employer in a job interview conducted through the conversational service, but the service provider and the employer may be different (in this case, the employer is the administrator of the conversational service and allows users to interact on the conversational service). Server 20 is implemented by a computer equipped with hardware such as a processor 29, memory 25, storage 26, communication interface 22, and input / output interface 23.

[0026] The generative AI system 30 is, for example, a system on which generative AI (generative artificial intelligence) is constructed. The generative AI system 30 functions, for example, as a system for interacting with a user.

[0027] The generative AI built into the generative AI system 30 is, for example, a Large Language Model (LLM). A Large Language Model is a natural language model designed to perform multiple tasks of natural language processing. A Large Language Model is an example of a trained model, trained using a large number of parameters (e.g., billions to hundreds of billions) and high-level computing resources. A natural language model refers to a computer program or algorithm designed to perform tasks of natural language processing. For example, in natural language processing, processes such as morphological analysis, syntactic analysis, information extraction, and text generation are performed, enabling a computer to analyze human language (i.e., natural language) and perform predetermined processing. A Large Language Model receives a prompt (instruction) as input and generates output based on the text, image, etc., of the prompt. The prompt can be defined in natural language.

[0028] Examples of large-scale language models include the GPT series (Generative Pre-trained Transformer) developed by OPEN AI, BERT (Bidirectional Encoder Representations from Transformers) developed by Google, StableLM developed by StabilityAI, and Llama2, Palm2®, and LamDA2® developed by Meta. Large-scale language models tend to have high training costs because they are trained using a very large number of parameters and computational resources. In this embodiment, the training cost is reduced by using a publicly available pre-trained model by the generative AI system 30. In this embodiment, for example, a definition is passed to GPT, allowing it to understand the context and then generate information.

[0029] A prompt is primarily a query (including text, strings, images, videos, audio, etc.) input to a generating AI. Users of a generating AI can instruct it to process information by inputting prompts. Users can input prompts to the generating AI to produce the desired output. Prompts do not necessarily have to be strings; they can also be images, videos, audio, etc. For example, gestures and voice commands from the user can also serve as prompts.

[0030] Prompts can be entered through input operations on the user's terminal. Document files, images, videos, audio, etc., can also be uploaded and used as prompts. Users can create prompts by combining instructions entered on their terminal with other instructions, data, etc. In this case, a prompt may contain one or more instructions, data, etc. Alternatively, a prompt may contain only one or more instructions, or only one or more data. A prompt can be created by including other instructions, data, etc., as part of a predetermined set of instructions, data, etc. A prompt can be created by inserting other instructions, data, etc., as part of a predetermined set of instructions, data, etc. A prompt can be created by combining predetermined instructions, data, etc., with other instructions, data, etc. A prompt can be created by combining predetermined instructions, data, etc., with other instructions, data, etc. A prompt can be created by adding other instructions, data, etc., to predetermined instructions, data, etc.

[0031] In this disclosure, expressions such as "include," "insert," "combine," "combine," and "add" used when creating prompts from predetermined instructions, data, etc., are used as terms that refer to the same information processing. In other words, they can be treated as the same information processing in that they create the same prompt based on one or more instructions, data, etc. For example, the term "include" includes information processing such as "insert," "combine," "combine," and "add." Similarly, "insert," "combine," "combine," and "add" also include information processing such as "include," "insert," "combine," "combine," and "add," respectively.

[0032] <2. Configuration of terminal equipment> Figure 2 is a block diagram showing an example of the functional configuration of the terminal device 10. As shown in Figure 2, the terminal device 10 includes a communication unit 120, an input device 13, an output device 14, an audio processing unit 17, a microphone 171, a speaker 172, a camera 160, a location information sensor 150, a storage unit 180, and a control unit 190. Each block included in the terminal device 10 is electrically connected, for example, by a bus.

[0033] The communication unit 120 performs processing such as modulation and demodulation processing for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on the signal generated by the control unit 190 and transmits it to an external source (for example, the server 20). The communication unit 120 performs reception processing on the signal received from an external source and outputs it to the control unit 190.

[0034] The input device 13 is a device for a user operating the terminal device 10 to input instructions or information. The input device 13 can be implemented, for example, by a touch-sensitive device 131 on which instructions are input by touching the operating surface. If the terminal device 10 is a PC, the input device 13 may be implemented by a reader, keyboard, mouse, etc. The input device 13 converts the instructions input by the user into electrical signals and outputs the electrical signals to the control unit 190. The input device 13 may also include, for example, a receiving port that accepts electrical signals input from an external input device.

[0035] The output device 14 is a device for presenting information to the user operating the terminal device 10. The output device 14 is implemented, for example, by a display 141. The display 141 displays data according to the control of the control unit 190. The display 141 is implemented, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0036] The audio processing unit 17 performs, for example, digital-to-analog conversion processing of the audio signal. The audio processing unit 17 converts the signal received from the microphone 171 into a digital signal and provides the converted signal to the control unit 190. The audio processing unit 17 also provides the audio signal to the speaker 172. The audio processing unit 17 is implemented, for example, by an audio processing processor. The microphone 171 receives an audio input and provides the audio signal corresponding to that audio input to the audio processing unit 17. The speaker 172 converts the audio signal received from the audio processing unit 17 into audio and outputs the audio to the outside of the terminal device 10.

[0037] Camera 160 is a device that receives light using a photodetector and outputs it as a shooting signal.

[0038] The location information sensor 150 is a sensor that detects the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. A GPS module is a receiving device used in a satellite positioning system. In a satellite positioning system, signals are received from at least three or four satellites, and the current position of the terminal device 10, which is equipped with a GPS module, is detected based on the received signals. The location information sensor 150 may also detect the current position of the terminal device 10 from the position of the wireless base station to which the terminal device 10 is connected.

[0039] The storage unit 180 is implemented by, for example, memory 15 and storage 16, and stores data and programs used by the terminal device 10. The storage unit 180 stores, for example, user information 181.

[0040] User information 181 includes, for example, information about the user who uses the terminal device 10. User information includes, for example, the user's name, age, address, date of birth, contact information, etc.

[0041] The control unit 190 is realized when the processor 19 reads a program stored in the memory unit 180 and executes instructions contained in the program. The control unit 190 controls the operation of the terminal device 10. By operating according to the program, the control unit 190 performs the functions of an operation reception unit 191, a transmission / reception unit 192, and a presentation control unit 193.

[0042] The operation reception unit 191 processes instructions or information input from the input device 13. Specifically, for example, the operation reception unit 191 receives instructions or information input from a touch-sensitive device 131 or the like.

[0043] Furthermore, the operation reception unit 191 receives voice instructions input from the microphone 171. Specifically, for example, the operation reception unit 191 receives voice signals input from the microphone 171 and converted into digital signals by the voice processing unit 17. The operation reception unit 191 obtains instructions from the user by, for example, analyzing the received voice signals and extracting predetermined nouns.

[0044] The transmitting / receiving unit 192 performs processing to enable the terminal device 10 to send and receive data with an external device such as the server 20 in accordance with a communication protocol. Specifically, for example, the transmitting / receiving unit 192 sends information input by the user or instructions from the user to the server 20. The transmitting / receiving unit 192 also receives information provided by the server 20.

[0045] The presentation control unit 193 controls the output device 14 in order to present information provided by the server 20 to the user. Specifically, for example, the presentation control unit 193 displays the information transmitted from the server 20 on the display 141. The presentation control unit 193 also outputs the information transmitted from the server 20 through the speaker 172.

[0046] <3. Functional Configuration of the Server> Figure 3 is a block diagram showing an example of the functional configuration of server 20. As shown in Figure 3, server 20 functions as a communication unit 201, a storage unit 202, and a control unit 203.

[0047] The communications unit 201 performs processing to enable the server 20 to communicate with external devices.

[0048] The storage unit 202 includes, for example, a user table 2021, etc. The tables stored in the storage unit 202 are not limited to these.

[0049] User Table 2021 is a table that stores information related to users.

[0050] The control unit 203 operates according to a program, thereby performing the function of controlling all processing related to the dialogue service.

[0051] Specifically, for example, the control unit 203 controls the transmission and reception of information with an external device. The control unit 203 controls the transmission and reception of information related to the interaction with an external device via the communication unit 201.

[0052] Specifically, for example, the control unit 203 controls the presentation of information related to the dialogue. The control unit 203 controls the speech and actions of the avatar displayed on the display 141, for example, via the presentation control unit 193.

[0053] Specifically, for example, the control unit 203 controls the generation of response sentences. The control unit 203 causes the generation AI system 30 to output a response sentence to the user's statement.

[0054] Specifically, for example, the control unit 203 divides the response sentence into units according to predetermined rules. For example, the control unit 203 divides the response sentence into units each time it determines whether there are punctuation marks in the response sentence.

[0055] Specifically, for example, the control unit 203 converts the data format to another format. For example, the control unit 203 converts text data to audio data. For example, the control unit 203 converts audio data to text data.

[0056] The control unit 203 is implemented by the processor 29 reading a program stored in the memory unit 202 and executing instructions contained in the program. The program includes applications such as web browser applications. The program includes programming languages ​​such as JavaScript® that are executed on the web browser application stored in the terminal device 10. The control unit 203 may implement the functions according to this embodiment by cooperating with external systems or services using an API (Application Programming Interface) as needed during the process of executing the program. That is, the control unit 203 may, for example, call a program stored in an external system using an API.

[0057] <4. Data Structure> This section describes the data structure of the tables stored by server 20. Note that the described data structure is an example and does not exclude data not listed. Furthermore, even data listed in the same table may be stored in separate memory areas within the storage unit 202. Each table may have columns other than those described in that table. Each table may not have any of the columns described in that table.

[0058] Figure 4 shows the data structure of User Table 2021. As shown in Figure 4, User Table 2021 has columns such as Name, Date of Birth, Email Address, etc., with User ID as the key.

[0059] The User ID is a column that stores an identifier to uniquely identify a user.

[0060] The "Name" column stores the user's name.

[0061] The date of birth column stores the user's date of birth.

[0062] The email address column stores the user's email address.

[0063] <5. Operation> An example of the operation flow in System 1 is described below.

[0064] <5.1. Operation 1>

[0065] Figure 5 is a flowchart illustrating an example of interaction between a user and an AI avatar in System 1.

[0066] In step S1001, the server 20 accepts a request to start a dialogue. Specifically, the user accesses the dialogue platform (interactor screen) via a web browser or a dedicated dialogue application on the terminal device 10 after performing a predetermined operation. The predetermined operation is, for example, logging in. The transmitting / receiving unit 192 sends a selected dialogue start request to the server 20 via the communication unit 120. The control unit 203 receives the dialogue start request, for example, via the communication unit 201. The control unit 203 inputs a prompt, for example, containing an instruction message to start the dialogue, to the generating AI system 30 via the communication unit 201. The instruction message is, for example, "Conduct a 30-minute job interview with the user. Investigate the user's aptitude, abilities, personality, etc."

[0067] In step S1002, the server 20 inputs the user's utterance to the generation AI system 30. Specifically, for example, the operation reception unit 191 receives the user's utterance as voice input via the input device 13. The transmission / reception unit 192 transmits the voice data of the utterance to the server 20 via, for example, the communication unit 120. The control unit 203 receives the voice data of the utterance via, for example, the communication unit 201. The control unit 203 converts the voice data of the utterance into text data. The control unit 203 transmits the text data of the utterance to the generation AI system 30 via, for example, the communication unit 201.

[0068] In step S1003, the server 20 causes the generating AI system 30 to output a response to the user's statement. Specifically, the generating AI system 30 generates a response that is in line with the interlocutor's statement. For example, the generating AI system 30 outputs responses as text data that express interest in the user's statement, praise, summary, rephrasing, questions, nods, fullers, dialogue moderation, etc. An example of an outputted response might be, "That's an interesting experience. I imagine it was a challenging experience, but please tell me what you learned from it." The generating AI system 30 outputs the response one character at a time. The generating AI system 30 sends the outputted response to the server 20 as it is generated.

[0069] In step S1004, the server 20 presents the outputted response to the user. Details of step S1004 will be described later with reference to Figure 6.

[0070] Even if step S1003 is not completed, that is, even if the generating AI system 30 is outputting a response statement, the server 20 may start step S1004, that is, present the outputted response statement at any time.

[0071] During the dialogue, steps S1002 to S1004 are repeated. The process ends when the time set for the dialogue has elapsed. The process may also end when the theme set for the dialogue and the discussion on that theme are completed. The process may also end when the questions set for the dialogue and the answers to those questions are completed. Before the process ends, for example, the control unit 203 informs the user, through the AI ​​avatar, that the dialogue will soon end.

[0072] Figure 6 is a flowchart showing the detailed procedure for step S1004 (presenting a response) in Figure 5.

[0073] In step S1004A, the server 20 determines the punctuation in the response sentence output in step S1003. Specifically, the control unit 203 analyzes the output response sentence character by character from the first character and determines the punctuation in the response sentence. For example, the control unit 203 analyzes the output response sentence character by character from the first character to the last character and determines the punctuation in the response sentence up to the last punctuation. For example, the control unit 203 determines the period in "That's an interesting experience.", the comma in "I imagine that experience was also difficult," and the period in "Please tell me what you learned from it."

[0074] In step S1004B, the server 20 divides the response sentence at each punctuation mark whenever it determines that there is punctuation in the response sentence. Specifically, the control unit 203 divides the response sentence from the first character to the first punctuation mark (hereinafter referred to as the first division). For example, the control unit 203 divides "That sounds like an interesting experience." as the first division. For example, the control unit 203 stores the first division in memory 25. Specifically, the control unit 203 divides the response sentence from the character following the end of the first division to the first punctuation mark (up to the second punctuation mark in the entire response sentence) (hereinafter referred to as the second division). For example, the control unit 203 divides "I imagine that experience was also difficult," as the second division. For example, the control unit 203 stores the second division in memory 25. Next, specifically, the control unit 203 divides the response sentence from the character following the end of the second division to the first punctuation mark (up to the third punctuation mark in the entire response sentence) (hereinafter referred to as the third division). For example, the control unit 203 divides "Please tell me what you learned from that." as the third division. For example, the control unit 203 stores the third division in memory 25. Specifically, similarly, the control unit 203 divides the response sentence to the end. That is, the control unit 203 divides the response sentence sequentially from the first character to the last character.

[0075] In step S1004C, when the server 20 separates the response sentence from the first character to the first punctuation mark (i.e., the first segment), it converts the first segment into first response voice data as the first unit. Specifically, the control unit 203 converts the first segment, as text data, into first response voice data as the first unit at the time the first segment is formed. For example, the control unit 203 converts the first segment "That's an interesting experience." into first response voice data 1 that reads out "That's an interesting experience." as the first unit. The control unit 203 uses speech synthesis to convert the voice data into voice data that matches the avatar's tone, quality, and volume. The conversion from text data to voice data may be performed by, for example, the voice processing unit 17.

[0076] In step S1004D, once the conversion to the first response voice data is complete, the server 20 presents the first response voice data to the user. Specifically, the control unit 203 transmits the first response voice data and the text data of the voice data to the terminal device 10 via the communication unit 201 at the time the data is converted to the first response voice data as a first unit. The control unit 203 controls the speech and actions of the avatar displayed on the display 141 via the presentation control unit 193. The control unit 203 plays the first response voice data in a way that causes the avatar to speak via the presentation control unit 193 and controls the avatar's actions in accordance with the speech. The control unit 203 may, for example, control the presentation control unit 193 to display the text data of the voice data on the display 141.

[0077] In step S1004E, the server 20 converts the text data accumulated during the conversion to the first response voice data into voice data. Specifically, for example, once the conversion to the first response voice data is complete, the control unit 203 converts the text data from the character following the end of the first unit in the response sentence up to the last punctuation mark that separated the response sentence before the completion of the conversion to the first response voice data into the second response voice data as the second unit. In other words, for example, once the conversion to the first response voice data is complete, the control unit 203 converts the text data accumulated up to the start of step S1004E, from the character following the end of the first unit up to the last punctuation mark that separated the response sentence before the completion of the conversion to the first response voice data into the second response voice data as the second unit. For example, if the last delimiter before the conversion to the first response voice data is completed is the third delimiter, "Please tell me what you learned from that," the control unit 203 converts the second delimiter, "I imagine that experience was difficult," and the third delimiter, "Please tell me what you learned from that," into a second unit, which seamlessly reads aloud the second response voice data, "I imagine that experience was difficult, but please tell me what you learned from that." (If there are further delimiters before the conversion to the first response voice data is completed, the first character of the second delimiter to the end of the last delimiter (i.e., the punctuation of that delimiter) is converted into the second response voice data as the second unit). The conversion from text data to voice data may be performed, for example, by the voice processing unit 17.

[0078] In step S1004F, the server 20 presents the second response voice data to the user. Specifically, the control unit 203 transmits the second response voice data and the text data of the voice data to the terminal device 10 via the communication unit 201. The control unit 203 controls the speech and actions of the avatar displayed on the display 141, for example, via the presentation control unit 193. The control unit 203 plays the second response voice data in a way that causes the avatar to speak, via the presentation control unit 193, and controls the avatar's actions in accordance with the speech. The control unit 203 may, for example, control the presentation control unit 193 to display the text data of the voice data on the display 141.

[0079] <6. Screen Example> An example of the screen of the display 141 of the terminal device 10 in this disclosure is described below.

[0080] Figure 7 shows an example screen when a user interacts with an avatar on the terminal device 10. This screen is a user interface for interaction between the user and the generating AI, for example, to receive statements from the user and to present responses from the generating AI to the user. The presentation control unit 193 controls, for example, the speech and actions of the avatar displayed on the display 141, and causes the avatar to speak voice data. Note that this screen example is just one example, and various screen configurations and screen contents can be adopted. For example, only the avatar 3011, which will be described later, may be displayed on the display 141.

[0081] The avatar display area 3001 is the area where the image or animation of the avatar 3011 is displayed. The avatar 3011 is, for example, an interface that has the appearance and voice of a human and speaks and acts under the control of the control unit 203. When the avatar 3011 speaks, the movements of the avatar 3011's mouth and other features may be expressed in sync. The avatar 3011's speech is realized, for example, by sound output from the speaker 172, but it may also be realized by displaying the content of the speech as text near the display area of ​​the avatar 3011. In this case, the avatar 3011 may function only as an icon without speaking. The text 3012 is text indicating the ongoing question topic. The button 3013 is a button for the user to voluntarily withdraw from the dialogue.

[0082] The speech display area 3002 is an area where speech entered by the user via voice or text is displayed in text format.

[0083] The response display area 3003 is an auxiliary display area for the user to recognize the response from the avatar 3011. Since the avatar 3011 responds by voice, the text of the response may not be displayed in this area, or if text data is sent from the server 20 along with the voice data, that text may be displayed.

[0084] The input means display area 3004 is an area that displays means for the user to input a statement to the avatar 3011. In this example screen, for example, a microphone icon 3021 for starting voice input and a field 3022 for supplementary text input are displayed in this area. For example, it is assumed that when the operation reception unit 191 receives a tap of the microphone icon 3021 by the user, voice input from the microphone 171 will be started to be received.

[0085] The send button 3005 is a button used to send a statement (mainly voice input or text input) entered by the user via the input means display area 3004 to the server 20.

[0086] The dialogue log display area 3006 is an area where the history of a series of interactions between the user and the avatar 3011 (specifically, the generated AI) is displayed in chronological order. The user can, for example, scroll through and review past conversations.

[0087] <7.Summary> As described above, in the above embodiment, the server 20 inputs the user's utterance into the generating AI and causes the generating AI to output a response sentence to the utterance. The server 20 analyzes the response sentence output by the generating AI character by character from the first character, and each time it determines a punctuation mark in the response sentence, it divides the response sentence at that punctuation mark. When the server 20 divides the response sentence from the first character to the first punctuation mark, it converts that section from the first character to the first punctuation mark into first response voice data as the first unit. The server 20 presents the first response voice data to the user. Once the conversion to first response voice data is complete, the server 20 converts the section from the character following the end of the first unit in the response sentence up to the last punctuation mark that was divided before the completion of the conversion to first response voice data into second response voice data as the second unit. The server 20 presents the second response voice data to the user. This reduces the waiting time before the AI ​​avatar starts speaking and mitigates unnatural intonation caused by voice conversion for each divided text. Therefore, it achieves a natural tempo and smooth conversation, similar to a conversation between humans.

[0088] <8. Variation> A modified example of the above embodiment will be described.

[0089] <8.1. Variation 1> In the above embodiment, the control unit 203 divides the response sentence into units based on punctuation. However, the control unit 203 may also divide the response sentence into units based on exclamation marks ("!"). Specifically, the control unit 203 analyzes the response sentence character by character from the first character, and divides the response sentence into units each time an exclamation mark is determined in the response sentence. For example, if the response sentence is "That must have been tough! But it was a wonderful experience!", the control unit 203 divides the part up to the first exclamation mark, "That must have been tough!", converts it into first response voice data, and presents it to the user. At the same time, the control unit 203 stores the remaining part, "But it was a wonderful experience!", which was stored before the conversion to the first response voice data was completed, as a second unit, converts it into second response voice data, and presents it to the user.

[0090] <8.2. Modification 2> In the above embodiment, the control unit 203 divides the response sentence into units based on punctuation. However, the control unit 203 may also divide the response sentence into units based on question marks ("?"). Specifically, the control unit 203 analyzes the response sentence character by character from the first character, and divides the response sentence into units each time it determines that there is a question mark in the response sentence. For example, if the response sentence is "When and where was that experience? What did you learn from it?", the control unit 203 divides the part up to the first question mark, "When and where was that experience?", converts this into first response voice data as a first unit, and presents it to the user. At the same time, the control unit 203 stores the remaining part, "What did you learn from it?", which was stored before the conversion to first response voice data was completed, as a second unit, converts it into second response voice data, and presents it to the user.

[0091] <8.3. Modification 3> In the above embodiment, the control unit 203 converts the first unit into first response voice data and the second unit into second response voice data, and presents them to the user. If the control unit 203 determines that the second unit has a predetermined number of characters or more (for example, 50 characters or more), in step S1004D, the control unit 203 may, via the presentation control unit 193, have the avatar speak the first response voice data at a predetermined slow speed (for example, 0.8 times the normal playback speed). This ensures that time is available to complete the generation process of the second response voice data, which takes time to convert, while the first response voice data is being presented. Therefore, seamless and natural conversation can be achieved without interruption in the avatar's speech between the presentation of the first response voice data and the presentation of the second response voice data.

[0092] <8.4. Modification 4> In the above embodiment, the control unit 203 converts the first unit into first response voice data and the second unit into second response voice data, and presents them to the user. If the control unit 203 determines that the second unit has a predetermined number of characters or more (for example, 50 characters or more), the control unit 203 divides the second unit into second response voice data and third response voice data based on the number of characters and converts them sequentially. Specifically, the control unit 203 converts the first character of the second segment of the second unit up to the last character of the segment corresponding to the 50th character (i.e., punctuation) into second response voice data. Specifically, the control unit 203 converts the segment following the last segment converted into second response voice data up to the last character of the second unit (i.e., punctuation) into third response voice data. For example, if the second unit consists of divisions 2 through 8, and the 50th character of the second unit is in the middle of division 4, the control unit 203 converts the portion from division 2 to the end of division 4 into second response voice data, and the portion from division 5 to the end of division 8 into third response voice data. Once the control unit 203 has finished converting to second response voice data, it presents the second response voice data to the user. Once the control unit 203 has finished converting to third response voice data, it presents the third response voice data to the user. In this way, by sequentially converting and presenting long subsequent units as independent voice data, the time required to generate the second response voice data can be reduced, and seamless and natural conversation can be achieved without interruption in the avatar's speech between the presentation of the first response voice data and the presentation of the second response voice data.

[0093] <8.5. Modification Example 5> In the above embodiment, the control unit 203 converted the text data accumulated during the conversion to the first response voice data into voice data. However, the timing of the conversion of the accumulated text data into voice data may be different. The server 20 converts the text data accumulated during the transmission of the first response voice data to the terminal device 10 into voice data. Specifically, for example, once the transmission of the first response voice data to the terminal device 10 is complete, the control unit 203 converts the text data from the character following the end of the first unit in the response sentence up to the last punctuation mark that separated the response sentence before the completion of transmission of the first response voice data to the terminal device 10 into the second response voice data as the second unit. In other words, for example, once the transmission of the first response voice data is complete, the control unit 203 converts the text data accumulated up to the start of step S1004E, from the character following the end of the first unit as text data up to the completion of transmission of the first response voice data into the second response voice data as the second unit. For example, if the third segment, "Please tell us what you learned from that experience," is the last segment added before the first response voice data is transmitted, the control unit 203 converts the second segment, "I imagine that experience was difficult," and the third segment, "Please tell us what you learned from that experience," into a second unit, which seamlessly reads out "I imagine that experience was difficult, but please tell us what you learned from that experience." (If the first response voice data is further segmented before transmission is completed, the control unit 203 converts everything from the first character of the second segment to the end of the last segment (i.e., the punctuation mark of that segment) into a second unit, which is the second response voice data.)

[0094] <9. Basic Computer Hardware Configuration> Figure 8 is a block diagram showing the basic hardware configuration of computer 90. Computer 90 includes at least a processor 901, main memory 902, auxiliary storage 903, and a communication interface IF991. These are electrically connected to each other by a communication bus 921.

[0095] The processor 901 is hardware for executing the instruction set written in a program. The processor 901 consists of an arithmetic unit, registers, peripheral circuits, etc.

[0096] Main memory 902 is used to temporarily store programs and data processed by programs, etc. For example, it is a volatile memory such as DRAM (Dynamic Random Access Memory).

[0097] Auxiliary storage device 903 refers to a storage device for saving data and programs. Examples include flash memory, HDD (Hard Disc Drive), magneto-optical disk, CD-ROM, DVD-ROM, and semiconductor memory.

[0098] The IF991 communication interface is an interface for inputting and outputting signals for communication with other computers via a network using wired or wireless communication standards. A network consists of various mobile communication systems, such as the internet, LANs, and wireless base stations. For example, a network includes 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks that can connect to the internet via designated access points (e.g., Wi-Fi®). When connecting wirelessly, communication protocols include, for example, Z-Wave®, ZigBee®, and Bluetooth®. When connecting via a wired connection, the network also includes connections made directly via USB (Universal Serial Bus) cables, etc.

[0099] Furthermore, by distributing all or part of each hardware configuration across multiple computers 90 and connecting them to each other via a network, a computer 90 can be virtually realized. Thus, the concept of computer 90 includes not only a computer 90 housed in a single enclosure or case, but also a virtualized computer system.

[0100] <Basic Functional Configuration of Computer 90> The functional configuration of the computer realized by the basic hardware configuration of computer 90 is described below. The computer comprises at least one functional unit: a control unit, a memory unit, and a communication unit.

[0101] Furthermore, the functional units of computer 90 can also be realized by distributing all or part of each functional unit across multiple computers 90 interconnected via a network. The concept of computer 90 includes not only a single computer 90 but also a virtualized computer system.

[0102] The control unit is realized when the processor 901 reads various programs stored in the auxiliary storage device 903, loads them into the main memory device 902, and executes processing according to those programs. The control unit can realize various functional units that perform information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0103] The memory unit is implemented by the main memory 902 and the auxiliary memory 903. The memory unit stores data, various programs, and various databases. The processor 901 can also reserve memory areas corresponding to the memory unit in the main memory 902 or the auxiliary memory 903 according to the program. The control unit can also cause the processor 901 to perform operations such as adding, updating, and deleting data stored in the memory unit according to the various programs.

[0104] A database, specifically a relational database, is used to manage and link together tabular data sets called masters, which are structurally defined by rows and columns. In a database, tables are called tables, masters are called masters, the columns of tables are called columns, and the rows of tables are called records. In a relational database, relationships can be established and linked between tables and masters. Typically, each table and master has a primary key column to uniquely identify records, but setting a primary key column is not mandatory. The control unit can instruct the processor 901 to add, delete, or update records in specific tables and masters stored in the memory unit, according to various programs. Furthermore, by storing data, various programs, and various databases in the memory unit, the information processing device and information processing system related to this disclosure can be considered to have been manufactured.

[0105] Furthermore, the databases and masters in this disclosure may include any data structures (lists, dictionaries, associative arrays, objects, etc.) in which information is structurally defined. Data structures also include data that can be considered as data structures by combining data with functions, classes, methods, etc., written in any programming language.

[0106] The communication unit is implemented by the communication IF991. The communication unit provides the functionality to communicate with other computers 90 via the network. The communication unit can receive information transmitted from other computers 90 and input it to the control unit. The control unit can cause the processor 901 to perform information processing on the received information according to various programs. The communication unit can also transmit information output from the control unit to other computers 90.

[0107] Furthermore, each of the above-mentioned configurations, functions, processing units, processing means, etc., may be implemented in hardware, either partially or entirely, by designing them as integrated circuits, for example. The present invention can also be implemented by software program code that realizes the functions of the embodiment. In this case, a storage medium on which the program code is recorded is provided to a computer, and the processor of that computer reads the program code stored in the storage medium. In this case, the program code read from the storage medium itself realizes the functions of the embodiment described above, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media used to supply such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, and the like.

[0108] Furthermore, the program code that implements the functions described in this embodiment can be implemented in a wide range of programming or scripting languages, such as assembler, C / C++, Perl, Shell, PHP, and Java (registered trademark).

[0109] Furthermore, the program code for the software that implements the functions of the embodiment may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the computer's processor may read and execute the program code stored in the storage means or storage medium.

[0110] The functions realized by the components described herein may be implemented in a circuit or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to realize the functions described herein. A processor is considered to be a circuit or processing circuitry, including transistors and other circuits. A processor may be a programmed processor that executes a program stored in memory. In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein. If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0111] While several embodiments of this disclosure have been described above, these embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications are permitted without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0112] (Note) The details described in each of the above embodiments are noted below. (Note 1) A program for operating a computer comprising a processor and memory, wherein the program is configured to operate the processor, The steps include inputting the user's statement into a generation AI and causing the generation AI to output a response to the statement, The steps include: analyzing the response sentence output from the generating AI character by character from the first character, and each time a punctuation mark in the response sentence is determined, dividing the response sentence at that punctuation mark; The steps include: separating the first character from the first punctuation mark in the aforementioned response sentence, converting the portion from the first character to the first punctuation mark into a first unit of first response audio data; The steps include presenting the first response voice data to the user, Once the conversion to the first response audio data is complete, the following steps are taken: the characters following the end of the first unit in the response sentence, up to the last punctuation mark that separated the response sentence before the completion of the conversion to the first response audio data, are converted into a second unit of second response audio data; The steps include presenting the second response voice data to the user, A program that executes the command. (Note 2) The program described in (Appendix 1), wherein in the delimiting step, the generating AI analyzes the response sentence character by character from the first character, and each time an exclamation mark in the response sentence is determined, the response sentence is delimited according to that exclamation mark. (Note 3) The program described in (Appendix 1) or (Appendix 2), wherein in the delimiting step, the program analyzes the response sentence from the generating AI character by character starting from the first character, and each time a question mark is determined in the response sentence, the program delimits the response sentence according to the question mark. (Note 4) If the second unit has a predetermined number of characters or more, the program, in the step of presenting the first response voice data, plays the first response voice data at a predetermined low speed, as described in any of (Appendix 1) to (Appendix 3). (Note 5) If the second unit has a predetermined number of characters or more, the program, as described in any of (Appendix 1) to (Appendix 4), divides the second unit into second response voice data and third response voice data based on the number of characters and converts them in order, presents the second response voice data to the user, and presents the third response voice data to the user. (Note 6) A method to be performed on a computer comprising a processor and memory, wherein the processor performs all steps performed in any of the inventions described in (Appendix 1) to (Appendix 5). (Appendix 7) An information processing apparatus comprising a control unit and a storage unit, wherein the control unit performs all steps performed in any of the inventions described in (Appendix 1) to (Appendix 5). (Note 8) A system comprising means for performing all steps performed in any of the inventions described in (Appendix 1) to (Appendix 5). [Explanation of Symbols]

[0113] 1... System 10…Terminal device 12…Communication IF 13…Input device 14…Output device 15…Memory 16…Storage 19… Processor 20... Server 22...Communication IF 23…Input / Output Interface 25…Memory 2 hours… storage 29… Processor 80…Network

Claims

1. A program for operating a computer comprising a processor and memory, wherein the program is configured to operate the processor, The steps include inputting the user's statement into a generation AI and causing the generation AI to output a response to the statement, The steps include: analyzing the response sentence output from the generating AI character by character from the first character, and each time a punctuation mark is determined in the response sentence, dividing the response sentence at that punctuation mark; The steps include: separating the first character from the first punctuation mark in the response sentence, converting the portion from the first character to the first punctuation mark into a first unit of first response audio data; The steps include presenting the first response voice data to the user, Once the conversion to the first response audio data is complete, the following steps are taken: the characters following the end of the first unit in the response sentence, up to the last punctuation mark that separated the response sentence before the completion of the conversion to the first response audio data, are converted into a second unit of second response audio data; The steps include presenting the second response voice data to the user, A program that executes the command.

2. The program according to claim 1, wherein in the delimiting step, the program analyzes the response sentence output from the generating AI character by character from the first character, and each time an exclamation mark in the response sentence is determined, the program delimits the response sentence for each exclamation mark.

3. The program according to claim 1, wherein in the delimiting step, the program analyzes the response sentence output from the generating AI character by character from the first character, and each time a question mark in the response sentence is determined, the program delimits the response sentence according to the question mark.

4. The program according to claim 1, wherein if the second unit has a predetermined number of characters or more, in the step of presenting the first response voice data, the first response voice data is played back at a predetermined low speed.

5. The program according to claim 1, wherein if the second unit has a predetermined number of characters or more, the program divides the second unit into second response voice data and third response voice data based on the number of characters and converts them in order, presents the second response voice data to the user, and presents the third response voice data to the user.

6. A method to be performed on a computer comprising a processor and memory, wherein the processor performs all steps performed in any of the inventions according to claims 1 to 5.

7. An information processing apparatus comprising a control unit and a storage unit, wherein the control unit performs all steps performed in the invention according to any one of claims 1 to 5.

8. A system comprising means for performing all steps performed in the invention according to any one of claims 1 to 5.

Images