Information processing equipment, methods, programs and systems

Abstract

The purpose of this disclosure is to maintain natural communication even when a human interrupts an AI conversation and takes over the conversation. [Solution] A program for operating a computer comprising a processor and memory, the program causing the processor to perform the following steps: receiving first audio data relating to a user's utterance from a terminal device; determining whether or not first audio data relating to a utterance different from the preceding utterance has been received between the time a generating AI starts inferring a response to the first audio data and the time a second audio data converted from text data relating to the response generated by the inference is sent to the terminal device; and controlling the system so that the second audio data is not output to the user if first audio data relating to a utterance different from the preceding utterance has been received between the time the inference is started and the time the second audio data is sent.

Description

Technical Field

[0001] This disclosure relates to an information processing apparatus, method, program, and system.

Background Art

[0002] In Patent Document 1, a conversation support system is described in which, based on the profile information of a service registrant in advance, an AI listens to the surrounding conversations at a gathering such as a drinking party and interrupts the conversation at an appropriate timing to provide topics that make it easier for that person to talk. In Patent Document 1, the AI analyzes the surrounding conversations in real time at a gathering such as a drinking party and interrupts the conversation at an appropriate timing to make it easier for the registrant to talk.

[0003] Specifically, in Patent Document 1, the interrupting unit interrupts the conversation at an appropriate timing based on the analyzed information. For example, the interrupting unit interrupts when the conversation has continued for a certain period of time or when a specific keyword appears. Thereby, the interrupting unit interrupts the conversation at an appropriate timing while keeping the flow of the conversation smooth.

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In Patent Document 1, the processing when a human interrupts an AI during conversation is not mentioned. In this case, there may also be a request for the human to interrupt the AI during conversation and resume the conversation. Therefore, when the conversation is interrupted by the AI, the human may feel stressed.

[0006] An object of this disclosure is to maintain natural communication even when a human interrupts an AI during conversation and resumes the conversation.

Means for Solving the Problems

[0007] A program for operating a computer comprising a processor and memory, the program causing the processor to perform the following steps: receiving first audio data relating to a user's utterance from a terminal device; determining whether or not first audio data relating to a utterance different from the preceding utterance has been received between the time a generating AI starts inferring a response to the first audio data and the time a second audio data converted from text data relating to the response generated by the inference is sent to the terminal device; and controlling the system so that the second audio data is not output to the user if first audio data relating to a utterance different from the preceding utterance has been received between the time the inference is started and the time the second audio data is sent. [Effects of the Invention]

[0008] According to this disclosure, even if a human interrupts an AI during a conversation and takes over the conversation, natural communication can be maintained. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing the overall configuration of System 1. [Figure 2] This is a block diagram showing an example of the functional configuration of the terminal device 10. [Figure 3] This block shows a functional configuration example for server 20. [Figure 4] This diagram shows the data structure of a table. [Figure 5] This diagram shows the data structure of a table. [Figure 6] This diagram shows an example of the operation flow in System 1. [Figure 7] This diagram shows an example of the operation flow in System 1. [Figure 8] This figure shows an example of the screen in this disclosure. [Figure 9] This is a block diagram showing the basic hardware configuration of Computer 90. [Modes for carrying out the invention]

[0010] The embodiments of this disclosure will be described below with reference to the drawings. In all the drawings illustrating the embodiments, common components are denoted by the same reference numerals, and repeated explanations are omitted. The following embodiments are not intended to unduly limit the content of this disclosure as described in the claims. Not all components shown in the embodiments are necessarily essential components of this disclosure. Also, each drawing is a schematic diagram and is not necessarily a strict illustration.

[0011] Furthermore, in the following description, "processor" refers to one or more processors. At least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may be another type of processor such as a GPU (Graphics Processing Unit). At least one processor may be single-core or multi-core.

[0012] Furthermore, at least one processor may be a broad-sense processor, such as a hardware circuit that performs some or all of the processing (e.g., an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)).

[0013] Furthermore, in the following explanation, we may use expressions such as "xxx table" to describe information that yields an output for a given input. This information can be data with any structure, or it can be a learning model such as a neural network that generates an output for a given input. Therefore, "xxx table" can be referred to as "xxx information."

[0014] Furthermore, in the following explanation, the structure of each table is just an example; one table may be divided into two or more tables, or all or part of two or more tables may be a single table.

[0015] Also, in the following description, although the "program" may be used as the subject to describe the processing, the program is executed by a processor to perform the defined processing while appropriately using a storage unit and / or an interface unit, etc. Therefore, the subject of the processing may be a processor (or a device such as a controller having the processor).

[0016] The program may be installed in a device such as a computer, or may be, for example, in a program distribution server or a computer-readable (e.g., non-temporary) recording medium. Also, in the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0017] Also, in the following description, an identification number is used as identification information for various objects, but other types of identification information (e.g., an identifier including letters or symbols) may be adopted.

[0018] Also, in the following description, when explaining without distinguishing between elements of the same type, reference signs (or common signs among the reference signs) are used, and when explaining while distinguishing between elements of the same type, the identification numbers (or reference signs) of the elements may be used.

[0019] Also, in the following description, the control lines and information lines indicate those considered necessary for the description, and not necessarily all the control lines and information lines on the product are shown. All the components may be interconnected.

[0020] Each information processing device is constituted by a computer having an arithmetic device and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. For each of the server 20 and the terminal device 10, descriptions overlapping with the basic hardware configuration of the computer and the basic functional configuration of the computer to be described later are omitted.

[0021] <Overview> The system according to this embodiment provides an AI avatar conversation service in which a user and an AI avatar converse. In the AI ​​avatar conversation service, the system receives audio data related to the user's statements from a terminal device. The system according to this embodiment determines whether it has received audio data related to a statement different from the previously received statement, from the start of inference by the response generation AI until the transmission of the audio data related to the response converted from the text data generated by the inference to the terminal device. If the system according to this embodiment receives audio data related to a statement different from the previously received statement, from the start of inference by the generation AI until the transmission of the audio data based on the inference, it controls the system not to output the audio data based on the inference to the user.

[0022] In this embodiment, a series of user statements and AI avatar responses are collectively referred to as a single conversation. Any type of conversation can constitute a single conversation. Examples of a single conversation include a 10-minute casual chat, a 30-minute business negotiation, a 60-minute job interview, a 90-minute media interview, and other forms of interaction such as face-to-face meetings, interviews, hearings, telephone reception, initial consultations for professional services, and role-playing (e.g., customer service practice). In this embodiment, we also consider an example of a face-to-face meeting in which the user converses with a generating AI model via an avatar. The avatar is, for example, an interface that has the appearance and voice of a human and can interact with the user based on output data from the generating AI model. The avatar can take the form of an icon representing any human appearance.

[0023] <1. System Configuration Diagram> Figure 1 is a block diagram showing an example of the overall configuration of System 1. As shown in Figure 1, System 1 includes, for example, a terminal device 10, a server 20, and a generation AI system 30. The terminal device 10, the server 20, and the generation AI system 30 are connected by communication via, for example, a network 80.

[0024] Figure 1 shows an example where System 1 includes two terminal devices 10, but the number of terminal devices 10 included in System 1 is not limited to two. System 1 may include one terminal device 10, or it may include three or more terminal devices 10.

[0025] Figure 1 shows an example where System 1 includes one Generative AI System 30, but the number of Generative AI Systems 30 included in System 1 is not limited to one. System 1 may include two or more Generative AI Systems 30. Also, Figure 1 shows an example where the Generative AI System 30 is independent of the Server 20, but the Server 20 may include the functions of the Generative AI System 30. In other words, the Server 20 may store the Generative AI.

[0026] In this embodiment, a collection of multiple devices may be treated as a single server. The method of allocating the multiple functions required to implement the server 20 according to this embodiment to one or more hardware can be appropriately determined in view of the processing capacity of each hardware and / or the specifications required for the server 20.

[0027] Terminal device 10 is an information processing device used by users who utilize the conversation service provided by server 20. Terminal device 10 provides applicants with a user interface for engaging in conversational dialogue with the generating AI provided by the generating AI system 30. Terminal device 10 can be implemented as, for example, a stationary PC (Personal Computer), a laptop PC, a head-mounted display, etc. Alternatively, terminal device 10 may be a portable computer such as a smartphone or tablet device.

[0028] The terminal device 10 comprises a communication interface (IF) 12, an input device 13, an output device 14, a memory 15, storage 16, and a processor 19. The communication interface 12 is an interface for inputting and outputting signals so that the terminal device 10 can communicate with devices in system 1, such as a server 20. The input device 13 is a device for receiving input operations from the user (e.g., a touch panel, touchpad, pointing device such as a mouse, keyboard, etc.). The output device 14 is a device for presenting information to the user (display, speaker, etc.). The memory 15 is for temporarily storing programs and data processed by programs, etc., and is a volatile memory such as DRAM (Dynamic Random Access Memory). The storage 16 is for saving data, and is a flash memory, HDD (Hard Disk Drive), etc. The processor 19 is hardware for executing the instruction set written in the program, and is composed of an arithmetic unit, registers, peripheral circuits, etc.

[0029] Server 20 is an information processing device used by a service provider that provides conversation services (hereinafter referred to as "service provider"). Server 20 is an information processing device implemented by, for example, a computer connected to network 80.

[0030] The server 20 comprises a communication interface 22, an input / output interface 23, a memory 25, storage 26, and a processor 29. The communication interface 22 is an interface for inputting and outputting signals so that the server 20 can communicate with devices in system 1, such as a terminal device 10. The input / output interface 23 functions as an interface to an input device for receiving input operations from a user and an output device for outputting information to the user. The memory 25 is for temporarily storing programs and data processed by programs, etc., and is a volatile memory such as DRAM. The storage 26 is for saving data, and is a flash memory or HDD, for example. The processor 29 is hardware for executing the instruction set written in the program, and is composed of an arithmetic unit, registers, peripheral circuits, etc.

[0031] Each information processing device consists of a computer equipped with an arithmetic unit and a memory device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by said hardware configuration will be described later. For each of the terminal device 10 and the server 20, explanations that overlap with the basic hardware configuration and basic functional configuration of the computer described later will be omitted.

[0032] The generative AI system 30 is, for example, a system on which generative AI (generative artificial intelligence) is constructed. The generative AI system 30 functions as a system for facilitating conversations between a user and the generative AI. That is, the AI ​​system 30 also plays a role in generating the speech content of the AI ​​avatar, such as by posing statements to the user or responding to statements from the user.

[0033] The generative AI built into the generative AI system 30 is, for example, a Large Language Model (LLM). A Large Language Model is a natural language model designed to perform multiple tasks of natural language processing. A Large Language Model is an example of a trained model, trained using a large number of parameters (e.g., billions to hundreds of billions) and high-level computing resources. A natural language model refers to a computer program or algorithm designed to perform tasks of natural language processing. For example, in natural language processing, processes such as morphological analysis, syntactic analysis, information extraction, and text generation are performed, enabling a computer to analyze human language (i.e., natural language) and perform predetermined processing. A Large Language Model generates output based on the text, image, etc., of a prompt (instruction) that is input. The prompt can be defined in natural language.

[0034] Examples of large-scale language models include the GPT series (Generative Pre-trained Transformer) developed by OPEN AI, BERT (Bidirectional Encoder Representations from Transformers) developed by Google, StableLM developed by StableAI, and Llama2, Palm2®, and LamDA2® developed by Meta. Large-scale language models tend to have high training costs because they are trained using a very large number of parameters and computational resources. In this embodiment, the training cost is reduced by using a publicly available pre-trained model by the generative AI system 30. In this embodiment, for example, a definition is passed to GPT, allowing it to understand the context and then generate information.

[0035] A prompt is primarily a query (including text, strings, images, videos, audio, etc.) input to a generating AI. Users of a generating AI can instruct it to process information by inputting prompts. Users can input prompts to the generating AI to produce the desired output. Prompts do not necessarily have to be strings; they can also be images, videos, audio, etc. For example, gestures and voice commands from the user can also serve as prompts.

[0036] Prompts can be entered through input operations on the user's terminal. Document files, images, videos, audio, etc., can also be uploaded and used as prompts. Users can create prompts by combining instructions entered on their terminal with other instructions, data, etc. In this case, a prompt may contain one or more instructions, data, etc. Alternatively, a prompt may contain only one or more instructions, or only one or more data. A prompt can be created by including other instructions, data, etc., as part of a predetermined set of instructions, data, etc. A prompt can be created by inserting other instructions, data, etc., as part of a predetermined set of instructions, data, etc. A prompt can be created by combining predetermined instructions, data, etc., with other instructions, data, etc. A prompt can be created by combining predetermined instructions, data, etc., with other instructions, data, etc. A prompt can be created by adding other instructions, data, etc., to predetermined instructions, data, etc.

[0037] In this disclosure, expressions such as "include," "insert," "combine," "combine," and "add" used when creating prompts from predetermined instructions, data, etc., are used as terms that refer to the same information processing. In other words, they can be treated as the same information processing in that the same prompt is created based on one or more instructions, data, etc. For example, the term "include" includes information processing such as "insert," "combine," "combine," and "add." Similarly, "insert," "combine," "combine," and "add" also include information processing such as "include," "insert," "combine," "combine," and "add," respectively.

[0038] <2. Configuration of terminal equipment> Figure 2 is a block diagram showing an example of the functional configuration of the terminal device 10. As shown in Figure 2, the terminal device 10 includes a communication unit 120, an input device 13, an output device 14, an audio processing unit 17, a microphone 171, a speaker 172, a camera 160, a location information sensor 150, a storage unit 180, and a control unit 190. Each block included in the terminal device 10 is electrically connected, for example, by a bus.

[0039] The communication unit 120 performs processing such as modulation and demodulation processing for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on the signal generated by the control unit 190 and transmits it to an external source (for example, the server 20). The communication unit 120 performs reception processing on the signal received from an external source and outputs it to the control unit 190.

[0040] The input device 13 is a device for a user operating the terminal device 10 to input instructions or information. The input device 13 can be implemented, for example, by a touch-sensitive device 131 on which instructions are input by touching the operating surface. If the terminal device 10 is a PC, the input device 13 may be implemented by a reader, keyboard, mouse, etc. The input device 13 converts the instructions input by the user into electrical signals and outputs the electrical signals to the control unit 190. The input device 13 may also include, for example, a receiving port that accepts electrical signals input from an external input device.

[0041] The output device 14 is a device for presenting information to the user operating the terminal device 10. The output device 14 is implemented, for example, by a display 141. The display 141 displays data according to the control of the control unit 190. The display 141 is implemented, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0042] The audio processing unit 17 performs, for example, digital-to-analog conversion processing of the audio signal. The audio processing unit 17 converts the signal received from the microphone 171 into a digital signal and provides the converted signal to the control unit 190. The audio processing unit 17 also provides the audio signal to the speaker 172. The audio processing unit 17 is implemented, for example, by an audio processing processor. The microphone 171 receives an audio input and provides the audio signal corresponding to that audio input to the audio processing unit 17. The speaker 172 converts the audio signal received from the audio processing unit 17 into audio and outputs the audio to the outside of the terminal device 10.

[0043] Camera 160 is a device that receives light using a photodetector and outputs it as a shooting signal.

[0044] The location information sensor 150 is a sensor that detects the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. A GPS module is a receiving device used in a satellite positioning system. In a satellite positioning system, signals are received from at least three or four satellites, and the current position of the terminal device 10, which is equipped with a GPS module, is detected based on the received signals. The location information sensor 150 may also detect the current position of the terminal device 10 from the position of the wireless base station to which the terminal device 10 is connected.

[0045] The storage unit 180 is implemented by, for example, memory 15 and storage 16, and stores data and programs used by the terminal device 10. The storage unit 180 stores, for example, user information 181.

[0046] User information 181 includes, for example, information about the user who uses the terminal device 10. User information includes, for example, the user's name, age, address, date of birth, contact information, etc.

[0047] The control unit 190 is realized when the processor 19 reads a program stored in the memory unit 180 and executes instructions contained in the program. The control unit 190 controls the operation of the terminal device 10. By operating according to the program, the control unit 190 performs the functions of an operation reception unit 191, a transmission / reception unit 192, a presentation control unit 193, and a conversation control unit 194.

[0048] Specifically, the control unit 190 determines whether or not it has received a statement from the user at a predetermined timing. Furthermore, if the control unit 190 has received a statement from the user at a predetermined timing, it controls the system so that the voice data of the AI ​​avatar's response is not output to the user.

[0049] The operation reception unit 191 processes instructions or information input from the input device 13. Specifically, for example, the operation reception unit 191 receives instructions or information input from a touch-sensitive device 131 or the like.

[0050] Furthermore, the operation reception unit 191 receives voice instructions input from the microphone 171. Specifically, for example, the operation reception unit 191 receives voice signals input from the microphone 171 and converted into digital signals by the voice processing unit 17. The operation reception unit 191 obtains instructions from the user by, for example, analyzing the received voice signals and extracting predetermined nouns.

[0051] The transmitting / receiving unit 192 performs processing to enable the terminal device 10 to send and receive data with an external device such as the server 20 in accordance with a communication protocol. Specifically, for example, the transmitting / receiving unit 192 sends information input by the user or instructions from the user to the server 20. The transmitting / receiving unit 192 also receives information provided by the server 20.

[0052] The presentation control unit 193 controls the output device 14 to present information provided by the server 20 to the user. Specifically, for example, the presentation control unit 193 displays the information transmitted from the server 20 on the display 141. As an example, the presentation control unit 193 displays an AI avatar in the browser of the terminal device 10. The presentation control unit 193 may also display the AI ​​avatar in a conversation-specific application of the terminal device 10. The presentation control unit 193 also outputs the information transmitted from the server 20 through the speaker 172.

[0053] The conversation control unit 194 controls the conversation between the user and the avatar. Specifically, the control unit 190 determines whether or not it has received a statement from the user at a predetermined time. More specifically, if the control unit 190 has received a statement from the user at a predetermined time, it controls the system so that the voice data of the AI ​​avatar's response is not output to the user.

[0054] <3. Functional Configuration of the Server> Figure 3 shows an example of the functional configuration of server 20. As shown in Figure 3, server 20 functions as a communication unit 201, a storage unit 202, and a control unit 203.

[0055] The communications unit 201 performs processing to enable the server 20 to communicate with external devices.

[0056] The memory unit 202 includes, for example, a user table 2021 and a conversation table 2022, etc. The tables stored in the memory unit 202 are not limited to these.

[0057] User Table 2021 is a table that stores information related to users.

[0058] Conversation Table 2022 is a table that stores information related to conversations between the user and the generating AI.

[0059] The control unit 203 functions as a module that controls all aspects of the conversation service processing by operating according to the program.

[0060] Specifically, the control unit 203 controls the conversation between the generating AI and the user. The control unit 203 controls the speech and actions of the avatar displayed on the display 141, for example, via the presentation control unit 193. The avatar's speech and actions are directly controlled by the presentation control unit 193. Specifically, the control unit 203 determines whether or not it has received audio data of the user's utterance at a predetermined timing. Furthermore, if the control unit 203 has received audio data of the user's utterance at a predetermined timing, it controls the system so that the avatar's response audio data is not output to the terminal device 10.

[0061] The control unit 203 is implemented by the processor 29 reading a program stored in the memory unit 202 and executing instructions contained in the program. The program includes applications such as web browser applications. The program includes programming languages ​​such as JavaScript® that are executed on the web browser application stored in the terminal device 10. The control unit 203 may implement the functions according to this embodiment by cooperating with external systems or services using an API (Application Programming Interface) as needed during the process of executing the program. That is, the control unit 203 may, for example, call a program stored in an external system using an API.

[0062] <4. Data Structure> This section describes the data structure of the tables stored by server 20. Note that the described data structure is an example and does not exclude data not listed. Furthermore, even data listed in the same table may be stored in separate memory areas within the storage unit 202. Each table may have columns other than those described in that table. Each table may not have any of the columns described in that table.

[0063] Figure 4 shows the data structure of User Table 2021. As shown in Figure 4, User Table 2021 has columns such as Name, Date of Birth, Email Address, etc., with User ID as the key.

[0064] The User ID is a column that stores an identifier to uniquely identify a user.

[0065] The "Name" column stores the user's name.

[0066] The date of birth column stores the user's date of birth.

[0067] The email address column stores the user's email address.

[0068] Figure 5 shows the data structure of the conversation table 2022. As shown in Figure 5, the conversation table 2022 has columns such as conversation ID, date and time, status, and message / response content, with user ID as the key. An example of a single conversation is, for example, a single daily work report.

[0069] The User ID column stores the User ID (same as in Figure 4) of the user who conversed with the generated AI.

[0070] The conversation ID is a column that stores an identifier to uniquely identify each conversation. The control unit 203 assigns one conversation ID to each conversation.

[0071] The date and time column stores the date and time the conversation took place.

[0072] The status system stores the status for each statement / response within a single conversation. The first status is the status of the user's statement. The second status is the status of the avatar's (generated AI) response.

[0073] The control unit 203 switches the status each time the speaker changes during a conversation. For example, when the user finishes speaking in the first status, the control unit 203 determines that the first status has ended and switches from the first status to the second status. The control unit 203 transmits to the terminal device 10 that it has switched to the second status. Also, for example, when the terminal device 10 has finished outputting the avatar's voice data in the second status, it transmits to the server 20 that it has finished outputting the voice data. When the control unit 203 receives from the terminal device 10 that it has finished outputting the voice data, it switches from the second status to the first status.

[0074] The "Statement / Response Content" column stores the user's statements and the AI's responses. For example, the Statement / Response Content stores the user's statements and the AI's responses as text turn by turn, following the flow of an actual conversation. Each statement and response may have a timestamp indicating the time when the statement and response occurred. The Statement / Response Content may store, for example, the audio of the conversation, the video of the conversation, or both. The server 20 receives, for example, information related to the user's statements and information related to the responses output from the terminal device 10 (information related to the avatar's statements) from the terminal device 10 and stores them in the Statement / Response Content. Responses that stopped being output midway through audio output may include a mark (for example, / ) to indicate that they stopped being output midway.

[0075] <5. Operation> An example of the operation flow in System 1 is described.

[0076] <5.1. Operation 1>

[0077] Figure 6 is a flowchart illustrating an example of a conversation between a user and a generated AI in System 1. Steps S1001 through S1007 are repeated during a single conversation.

[0078] First, the user accesses a conversational platform in a face-to-face format, where they converse with a generated AI model via an avatar, through a predetermined operation, for example, using a web browser or app on a terminal device 10. The predetermined operation is, for example, logging in. The transmitting / receiving unit 192 sends a conversation start request to the server 20 via the communication unit 120. The control unit 203 receives the conversation start request, for example, via the communication unit 201. The control unit 203 then instructs the generated AI system 30 to start the conversation, for example, via the communication unit 201. The instruction is, for example, "Please have a casual conversation with the user for 10 minutes. Please respond to the user's statements."

[0079] In step S1001, the terminal device 10 receives a statement from the user. The statement can be an affirmative sentence, a negative sentence, a question, a command, a greeting, etc. For example, the statement could be, "I want to start exercising." Specifically, for example, the operation reception unit 191 receives the user's statement input as first voice data via the input device 13. In this embodiment, the first voice data refers to, for example, the voice data of the user's statement received by the terminal device 10.

[0080] In step S1002, the server 20 receives the first voice data of the user's utterance. Specifically, the transmitting / receiving unit 192 transmits the first voice data to the server 20, for example, via the communication unit 120. The control unit 203 receives the first voice data, for example, via the communication unit 201.

[0081] In step S1003, the generating AI system 30 infers a response to the utterance. Specifically, the control unit 203 switches from the first status to the second status when the user finishes uttering. The control unit 203 converts the first audio data related to the user's utterance into text data. The control unit 203 transmits the text data of the utterance to the generating AI system 30, for example, via the communication unit 201. The generating AI system 30 outputs text data as a response to the utterance, for example, an answer to the user's question, a follow-up to the user's utterance, advice to the user, encouragement to the user, a greeting, a fuller, etc. The text data is, for example, "If you're going to exercise, I recommend jogging." For example, the generating AI system 30 transmits the text data to the server 20. The control unit 203 receives the text data, for example, via the communication unit 201.

[0082] In step S1004, the server 20 converts the text data into second audio data. Specifically, the control unit 203 converts the received text data into second audio data, for example. In this embodiment, the second audio data is, for example, the response that the avatar will speak. of This refers to audio data.

[0083] In step S1005, the server 20 transmits the second voice data to the terminal device 10. Specifically, the control unit 203 transmits the second voice data to the terminal device 10, for example, via the communication unit 201. The terminal device 10 receives the second voice data. The control unit 190 receives the second voice data, for example, via the transmitting / receiving unit 192.

[0084] Step S1006 is when the terminal device 10 prepares the output of the second audio data received from the server 20. Specifically, the control unit 190 performs, for example, a decoding process of the received compressed second audio data. The control unit 190 also performs, for example, a buffering process of the received compressed second audio data for smooth playback.

[0085] In step S1007, the terminal device 10 outputs the second voice data to the user. Specifically, the presentation control unit 193 controls, for example, the speech and actions of the avatar displayed on the display 141, causing the avatar to speak the second voice data. Once the terminal device 10 has finished outputting the avatar's voice data, it sends a message to the server 20 indicating that it has finished outputting the voice data. When the control unit 203 receives a message from the terminal device 10 indicating that it has finished outputting the voice data, it switches the second status to the first status.

[0086] Furthermore, after step S1007, the server 20 stores the text data related to the outputted second audio data. Specifically, the terminal device 10 sends information to the server 20 indicating that the second audio data has been output to the user. The server 20 stores the text data of the second audio data output from the terminal device 10 in the conversation table 2022. The control unit 203 adds a record to the conversation table 2022 and stores the information related to the text data of the second audio data output from the terminal device 10 in each column as necessary.

[0087] <5.2. Operation 2>

[0088] Figure 7 is a flowchart illustrating an example of a conversation between a user and a generated AI in System 1.

[0089] First, the conversation is in the second status (one of the steps S1002 to S1007 in Figure 6).

[0090] In step S1101, the terminal device 10 receives a statement different from the preceding statement in Figure 6 (hereinafter referred to as the "different statement"). The different statement may be, for example, a statement about something that was forgotten in the previous statement, or a statement that interrupts the avatar's statement. Specifically, the different statement may be, for example, "Indoor exercise is good." Specifically, for example, the operation reception unit 191 receives the different statement from the user as first voice data via the input device 13.

[0091] In step S1102, the server 20 receives first audio data relating to a different statement. Specifically, the transmitting / receiving unit 192 transmits the first audio data to the server 20, for example, via the communication unit 120. The control unit 203 receives the first audio data, for example, via the communication unit 201. Upon receiving the first audio data, the control unit 203 switches from the second status to the first status.

[0092] In step S1103A, the terminal device 10 determines whether it has received a different statement between the time it receives the second audio data and the time it starts outputting the second audio data (for example, while it is performing the processing in step S1006). Specifically, for example, the control unit 190 determines whether it has received a different statement while preparing to output the second audio data. If the determination is yes, the process proceeds to S1103B. If the determination is no, the process proceeds to S1103C.

[0093] In step S1103B, the terminal device 10 controls the system so that the second audio data is not output to the user. Specifically, the control unit 190 sets the output intensity of the second audio data to zero. As a result, the volume of the second audio data becomes zero, and the second audio does not reach the user. For example, the volume of the second audio data, "Jogging is recommended," becomes zero from the first character. The presentation control unit 193 may, for example, control the avatar's movements in synchronization with setting the volume to zero. For example, the avatar stops the movement of its mouth when speaking.

[0094] In step S1103C, the terminal device 10 determines whether it has received a different statement while outputting the second audio data (i.e., while performing the processing in step S1007). Specifically, the control unit 190 determines whether it has received a different statement while outputting the second audio data. If the determination is yes, the process proceeds to S1103D. If the determination is no, the process proceeds to S1104A.

[0095] In step S1103D, the terminal device 10 controls itself so that the second audio data is not output to the user after it receives a different statement. Specifically, the control unit 190 sets the output intensity of the second audio data to zero after it receives the different statement. As a result, the volume of the second audio data after it receives the different statement becomes zero, and the second audio after it receives the different statement does not reach the user. For example, if the second audio data "Jogging is recommended" has been output up to "Jogging is", the volume of the remaining "Recommended" will be set to zero. The presentation control unit 193 may, for example, control the avatar's movements in synchronization with setting the volume to zero. For example, the avatar may stop the movement of its mouth when speaking.

[0096] In step S1104A, the server 20 determines whether it has received first audio data relating to a different statement while the response generation AI is inferring a response (i.e., during step S1003). Specifically, the control unit 203 determines whether it has received first audio data while the generation AI is inferring a response. If the determination is yes, the process proceeds to S1104B. If the determination is no, the process proceeds to S1104C.

[0097] In step S1104B, the server 20 stops the inference performed by the response generation AI. Specifically, the control unit 203 stops the inference performed by the response generation AI.

[0098] In step S1104C, the server 20 determines whether it has received first audio data relating to a different statement while converting to second audio data (i.e., during step S1004). Specifically, the control unit 203 determines whether it has received first audio data while converting text data to second audio data. If the determination is yes, the process proceeds to S1104D. If the determination is no, the process proceeds to S1104E.

[0099] In step S1104D, the server 20 cancels the conversion to the second audio data. Specifically, the control unit 203 cancels the conversion to the second audio data.

[0100] In step S1104E, the server 20 determines whether it has received first audio data relating to a different statement between the time of conversion to second audio data and the completion of transmission of the second audio data to the terminal device 10 (i.e., before the completion of step S1005). Specifically, the control unit 203 determines whether it has received first audio data between the time of conversion to second audio data and before transmission of the second audio data to the terminal device 10. If the determination is yes, the process proceeds to S1104F. If the determination is no, the process proceeds to step S1003, and the first audio data is processed according to the flow in Figure 6.

[0101] In step S1104F, the server 20 stops transmitting the second voice data to the terminal device 10. Specifically, the control unit 203 stops transmitting the second voice data to the terminal device 10.

[0102] Furthermore, after step S1103C, the server 20 stores the text data related to the outputted second audio data. Specifically, the terminal device 10 sends information to the server 20 indicating that the second audio data has been output to the user (information that distinguishes between the second audio data output from the terminal device 10 and the second audio data that was not output). The server 20 stores the text data of the second audio data output from the terminal device 10 in the conversation table 2022. The control unit 203 adds a record to the conversation table 2022 and stores the information related to the text data of the second audio data output from the terminal device 10 in each column as necessary. If the ratio of the text data of the second audio data output from the terminal device 10 to the text data related to the response meets a predetermined requirement (for example, 60% or more in terms of the number of characters), the server 20 stores the text data of the second audio data output from the terminal device 10 in the conversation table 2022. If the ratio is less than a predetermined value, the server 20 does not store the text data of the second voice data output from the terminal device 10 in the conversation table 2022.

[0103] <6. Screen example> An example of the screen of the display 141 of the terminal device 10 in this disclosure is described below.

[0104] Figure 8 shows an example screen when a user converses with a generated AI via an avatar on the terminal device 10 (interview format). This screen is a user interface for conversation between the user and the generated AI, for example, to receive statements from the user and to present the user with responses from the generated AI. The presentation control unit 193 controls, for example, the speech and actions of the avatar displayed on the display 141, and causes the avatar to speak voice data. Note that this screen example is merely an example, and various screen configurations and screen contents can be adopted. For example, only the avatar 3011 described later may be displayed on the display 141.

[0105] The avatar display area 3001 is an area where an image or animation of the avatar 3011 is displayed. The avatar 3011 is, for example, an interface that has the appearance and voice of a human and speaks and acts under the control of the control unit 203. When the avatar 3011 speaks, the movements of the avatar 3011's mouth and other features may be expressed in sync. The avatar 3011's speech is realized, for example, by sound output from the speaker 172, but it may also be realized by displaying the content of the speech as text near the display area of ​​the avatar 3011. In this case, the avatar 3011 may function only as an icon without speaking.

[0106] The speech display area 3002 is an area where speech entered by the user via voice or text is displayed in text format.

[0107] The response display area 3003 is an auxiliary display area for the user to recognize the response from the avatar 3011. Since the avatar 3011 responds by voice, the text of the response may not be displayed in this area, or if text data is sent from the server 20 along with the voice data, that text may be displayed.

[0108] The input means display area 3004 is an area that displays means for the user to input a statement to the avatar 3011. In this example screen, for example, a microphone icon 3014 for starting voice input and a field 3024 for supplementary text input are displayed in this area. For example, it is assumed that when the operation reception unit 191 receives a press operation of the microphone icon 3014 by the user, voice input from the microphone 171 will be started to be received.

[0109] The send button 3005 is a button used to send a statement (mainly voice input or text input) entered by the user via the input means display area 3004 to the server 20.

[0110] The conversation log display area 3006 is an area where the history of a series of interactions between the user and the avatar 3011 (specifically, the generated AI) is displayed in chronological order. The user can, for example, scroll through and review past conversation content.

[0111] <7.Summary> As described above, in the above embodiment, the server 20 receives first audio data relating to the user's utterance from the terminal device 10. From the time the server 20 starts generating a response to the first audio data until it sends the second audio data relating to the response, which has been converted from the text data relating to the response generated by inference, to the terminal device 10, the server 20 determines whether it has received first audio data relating to a different utterance than the preceding utterance. If the server 20 receives first audio data relating to a different utterance between the start of inference and the transmission of the second audio data, it controls the system so that the second audio data is not output to the user. Specific examples of this control include stopping the inference of the response, stopping the conversion to the second audio data, and stopping the transmission of the second audio data. Furthermore, if the terminal device 10 receives a different utterance between the time of receiving the second audio data and the time of outputting the second audio data, it also controls the system so that the second audio data is not output to the user. A specific example of this control is zeroing out the output intensity of the second audio data. As a result, when the AI ​​avatar is about to respond, or when the AI ​​avatar is responding, if the user speaks, the AI ​​avatar will refrain from responding. Therefore, even if a human interrupts an AI during a conversation and takes over, natural communication can be maintained.

[0112] Furthermore, as described above, in the above embodiment, the server 20 stores the text related to the second audio data output from the terminal device in the conversation history. In the storage step, if the ratio of the text related to the audio output from the terminal device to the text related to the response satisfies a predetermined requirement, the server 20 stores the text related to the second audio data output from the terminal device in the conversation history. If the ratio does not satisfy the predetermined requirement, the server 20 does not store the text related to the second audio data output from the terminal device in the conversation history. As a result, only the avatar's responses that the user actually heard are stored, and if the avatar's responses that the user actually heard are too few compared to the text, the lines are not stored. In addition, the generating AI system 30 can perform subsequent processing in the conversation (subsequent inference, conversation summarization, conversation evaluation, user evaluation, user analysis, etc.) by referring to the avatar's lines that the user heard and not referring to the avatar's lines that the user did not hear, enabling natural subsequent processing.

[0113] <8. Variation> A modified example of the above embodiment will be described.

[0114] <8.1. Variation 1> If the first audio data for a different statement is less than or equal to a predetermined number of characters (for example, 6 characters or less), the server 20 does not need to prevent the output of the second audio data to the user. Specifically, even if the server 20 receives first audio data relating to a different statement while inference is in progress, the control unit 203 does not abort the inference in step S1104B. Also specifically, even if the server 20 receives first audio data relating to a different statement while speech conversion is in progress, the control unit 203 does not abort the conversion in step S1104D. Also specifically, even if the server 20 receives first audio data relating to a different statement before the server 20 has finished transmitting the second audio data, the control unit 203 does not abort the transmission in step S1104F.

[0115] Furthermore, if the first audio data is less than or equal to a predetermined number of characters (for example, 6 characters or less), the terminal device 10 does not need to prevent the output of the second audio data to the user. Specifically, even if the control unit 203 receives a different statement while preparing to output the second audio data, the control unit 190 does not reduce the output intensity of the second audio data to zero.

[0116] This allows for natural conversation to be maintained when the first voice data consists of short exclamations such as "Oh," interjections, etc., which do not require the avatar to interrupt the second voice data's output.

[0117] <8.2. Modification Example 2> In the above embodiment, the first status is the status of the user making a statement, and the second status is the status of the avatar (generated AI) responding.

[0118] While the avatar is responding, it is in the second status, but in step S1102, when the server 20 receives first audio data relating to a different statement from the terminal device 10, the control unit 203 switches from the second status to the first status. In this case, the server 20 may indicate the switch to the terminal device 10. Specifically, the control unit 203 transmits information relating to the switch to the terminal device 10, for example. The notification control unit 193 suggests the switch, for example, in the screen example in Figure 7. The notification control unit 193 displays information suggesting the switch, for example, "It's your turn," "The turn has changed," etc. Alternatively, the notification control unit 193 may also suggest the switch by hiding the text "It's the avatar's turn."

[0119] This makes the switch more easily communicated to the user, allowing them to easily understand whether it is the user's or the avatar's turn.

[0120] <8.3. Modification 3> In the above embodiment, the terminal device 10 was described as a stationary PC or the like. However, the terminal device 10 may also be a robot modeled after a human or character. A robot terminal device 10 may not have, for example, an input device 13, an output device 14, an operation reception unit 191, or a presentation control unit 193. Specifically, the robot has the basic configuration of a computer, such as a communication I / F 12, memory 15, storage 16, and a processor 19. When the robot plays the role of an avatar, instead of an avatar displayed on a screen, the robot presents information to the user through its own physical movements and speech. Therefore, the screen example in Figure 8 is unnecessary, and the presentation control unit 193 becomes a functional block that directly controls the robot's movements and speech. The user can input voice as speech using a microphone built into the robot. The operation reception unit 191 is dedicated to processing voice input. In addition, the robot's own speaker plays the role of speaker 172 and outputs the response from the generating AI system 30 as voice. In this case, a display as output device 14 is not necessarily required. Thus, unlike conventional stationary PCs and smartphones, the robotic terminal device 10 has its physical form and function closely linked because the robot itself performs the function of an avatar. This makes it possible to provide a more intuitive and natural communication experience in which the user interacts with the generated AI through the physical presence of the robot, rather than through an avatar.

[0121] <9. Basic Computer Hardware Configuration> Figure 9 is a block diagram showing the basic hardware configuration of computer 90. Computer 90 includes at least a processor 901, main memory 902, auxiliary storage 903, and a communication interface 991. These are electrically connected to each other by a communication bus 921.

[0122] The processor 901 is hardware for executing the instruction set written in a program. The processor 901 consists of an arithmetic unit, registers, peripheral circuits, etc.

[0123] Main memory 902 is used to temporarily store programs and data processed by programs, etc. For example, it is a volatile memory such as DRAM (Dynamic Random Access Memory).

[0124] Auxiliary storage device 903 refers to a storage device for saving data and programs. Examples include flash memory, HDD (Hard Disc Drive), magneto-optical disk, CD-ROM, DVD-ROM, and semiconductor memory.

[0125] The IF991 communication interface is an interface for inputting and outputting signals for communication with other computers via a network using wired or wireless communication standards. A network consists of various mobile communication systems, such as the internet, LANs, and wireless base stations. For example, a network includes 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks that can connect to the internet via designated access points (e.g., Wi-Fi®). When connecting wirelessly, communication protocols include, for example, Z-Wave®, ZigBee®, and Bluetooth®. When connecting via a wired connection, the network also includes connections made directly via USB (Universal Serial Bus) cables, etc.

[0126] Furthermore, by distributing all or part of each hardware configuration across multiple computers 90 and connecting them to each other via a network, a computer 90 can be virtually realized. Thus, the concept of computer 90 includes not only a computer 90 housed in a single enclosure or case, but also a virtualized computer system.

[0127] <Basic Functional Configuration of Computer 90> The functional configuration of the computer realized by the basic hardware configuration of computer 90 is described below. The computer comprises at least one functional unit: a control unit, a memory unit, and a communication unit.

[0128] Furthermore, the functional units of computer 90 can also be realized by distributing all or part of each functional unit across multiple computers 90 interconnected via a network. The concept of computer 90 includes not only a single computer 90 but also a virtualized computer system.

[0129] The control unit is realized when the processor 901 reads various programs stored in the auxiliary storage device 903, loads them into the main memory device 902, and executes processing according to those programs. The control unit can realize various functional units that perform information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0130] The memory unit is implemented by the main memory 902 and the auxiliary memory 903. The memory unit stores data, various programs, and various databases. The processor 901 can also reserve memory areas corresponding to the memory unit in the main memory 902 or the auxiliary memory 903 according to the program. The control unit can also cause the processor 901 to perform operations such as adding, updating, and deleting data stored in the memory unit according to the various programs.

[0131] A database, specifically a relational database, is used to manage and link together tabular data sets called masters, which are structurally defined by rows and columns. In a database, tables are called tables, masters are called masters, the columns of tables are called columns, and the rows of tables are called records. In a relational database, relationships can be established and linked between tables and masters. Typically, each table and master has a primary key column to uniquely identify records, but setting a primary key column is not mandatory. The control unit can instruct the processor 901 to add, delete, or update records in specific tables and masters stored in the memory unit, according to various programs. Furthermore, by storing data, various programs, and various databases in the memory unit, the information processing device and information processing system related to this disclosure can be considered to have been manufactured.

[0132] Furthermore, the databases and masters in this disclosure may include any data structures (lists, dictionaries, associative arrays, objects, etc.) in which information is structurally defined. Data structures also include data that can be considered as data structures by combining data with functions, classes, methods, etc., written in any programming language.

[0133] The communication unit is implemented by the communication IF991. The communication unit provides the functionality to communicate with other computers 90 via the network. The communication unit can receive information transmitted from other computers 90 and input it to the control unit. The control unit can cause the processor 901 to perform information processing on the received information according to various programs. The communication unit can also transmit information output from the control unit to other computers 90.

[0134] Furthermore, each of the above-mentioned configurations, functions, processing units, processing means, etc., may be implemented in hardware, either partially or entirely, by designing them as integrated circuits, for example. The present invention can also be implemented by software program code that realizes the functions of the embodiment. In this case, a storage medium on which the program code is recorded is provided to a computer, and the processor of that computer reads the program code stored in the storage medium. In this case, the program code read from the storage medium itself realizes the functions of the embodiment described above, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media used to supply such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, and the like.

[0135] Furthermore, the program code that implements the functions described in this embodiment can be implemented in a wide range of programming or scripting languages, such as assembler, C / C++, Perl, Shell, PHP, and Java (registered trademark).

[0136] Furthermore, the program code for the software that implements the functions of the embodiment may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the computer's processor may read and execute the program code stored in the storage means or storage medium.

[0137] The functions realized by the components described herein may be implemented in a circuit or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to realize the functions described herein. A processor is considered to be a circuit or processing circuitry, including transistors and other circuits. A processor may be a programmed processor that executes a program stored in memory. In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein. If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0138] While several embodiments of this disclosure have been described above, these embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications are permitted without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0139] (Note) The details described in each of the above embodiments are noted below.

[0140] (Note 1) A program for operating a computer comprising a processor and memory, wherein the program is configured to operate the processor, The steps include receiving first audio data relating to the user's speech from the terminal device, From the time the generating AI is instructed to infer a response to the first audio data, until the second audio data converted from the text data relating to the response generated by the inference is transmitted to the terminal device, the step of determining whether or not the first audio data relating to a statement different from the preceding statement has been received, If, between the start of the inference and the transmission of the second audio data, the first audio data relating to a statement different from the preceding statement is received, the second audio data is controlled so as not to be output to the user. A program that executes the command. (Note 2) In the aforementioned determination step, if it is determined during the inference that the first audio data relating to a statement different from the preceding statement has been received, The program described in (Appendix 1) for canceling the inference in the control step. (Note 3) In the aforementioned determination step, if it is determined that the first audio data relating to a statement different from the preceding statement has been received during the conversion of the text data relating to the response to the second audio data, A program according to any one of (Appendix 1) to (Appendix 2) that cancels the conversion in the control step. (Note 4) In the determination step described above, if it is determined that the first audio data relating to a statement different from the preceding statement has been received between the time the text data relating to the response is converted to the second audio data and the transmission of the second audio data to the terminal device is completed, A program according to any of (Appendix 1) to (Appendix 3) that cancels the transmission in the control step. (Note 5) In the aforementioned judgment step, if it is determined that a statement different from the preceding statement does not meet the prescribed requirements, A program according to any one of (Appendix 1) to (Appendix 4) that, in the control step, does not interfere with the output of the second audio data to the user. (Note 6) In the second status to which the generating AI responds, if it receives the first audio data relating to a statement different from the preceding statement from the terminal device, it switches the second status to the first status spoken by the user, and presents the switch to the terminal device. A program described in any of (Appendix 1) to (Appendix 5) that causes the processor to execute the above. (Note 7) A program according to any one of (Appendix 1) to (Appendix 6), which causes the processor to perform the step of storing the text relating to the second voice data output from the terminal device in the conversation history. (Note 8) The program as described in (Appendix 7), wherein in the storage step, if the ratio of text relating to the voice output from the terminal device to the text relating to the response satisfies a predetermined requirement, the text relating to the second voice data output from the terminal device is stored in the conversation history; if the ratio does not satisfy the predetermined requirement, the text relating to the second voice data output from the terminal device is not stored in the conversation history. (Note 9) A method to be performed on a computer comprising a processor and memory, wherein the processor performs all steps performed in any of the inventions described in (Appendix 1) to (Appendix 8). (Appendix 10) An information processing apparatus comprising a processor and memory, wherein the processor performs all steps performed in any of the inventions described in (Appendix 1) to (Appendix 8). (Note 11) A system comprising means for performing all steps performed in any of the inventions described in (Appendix 1) to (Appendix 8). (Note 12) A system comprising a terminal device and a server that is communicatively connected to the terminal device, The aforementioned server, Means for receiving first audio data relating to the user's first statement from the terminal device, From the time the AI ​​generates the response to the user's preceding statement prior to the first statement, until the transmission to the terminal device of the second audio data relating to the response, which has been converted from the original text data relating to the response generated by the inference, there is a means for determining whether or not the first audio data has been received. Equipped with, The aforementioned terminal device is A means for receiving the user's comments, means for receiving the second audio data from the server, A means for determining whether a statement different from the preceding statement has been received between the time of receiving the second audio data and the time of outputting the second audio data, If a statement different from the preceding statement is received between the time of the reception of the second audio data and the time of output of the second audio data, means for controlling that the second audio data is not output to the user. Equipped with, system. (Note 13) If the terminal device's determination means determines that it has received a statement different from the preceding statement between the time it receives the second audio data and the time it outputs the second audio data to the user, The means for controlling the terminal device controls the system so that the second voice data is not output to the user, as described in (Appendix 12). (Note 14) If the means used by the terminal device to make a determination receives a statement different from the preceding statement while the second voice data is being output to the user, The means for controlling the terminal device is a system according to any one of (Appendix 12) to (Appendix 13) that controls the second voice data so that it is not output to the user. (Note 15) If the terminal device's means of judgment determines that a statement different from a preceding statement does not meet the prescribed requirements, The means for controlling the terminal device is a system according to any one of (Appendix 12) to (Appendix 14), which does not interfere with the output of the second voice data to the user. [Explanation of Symbols]

[0141] 1... System 10…Terminal device 12…Communication IF 13…Input device 14…Output device 15…Memory 16…Storage 19… Processor 20... Server 22...Communication IF 23…Input / Output Interface 25…Memory 2 hours… storage 29… Processor 80…Network

Claims

1. A program for operating a computer comprising a processor and memory, wherein the program is configured to operate the processor, The steps include receiving first audio data relating to the user's speech from the terminal device, The process involves determining whether or not the first audio data relating to a statement different from the preceding statement has been received while the generating AI is inferring a response to the first audio data, If, while the generating AI is inferring a response to the first audio data, it determines that it has received first audio data relating to a statement different from the preceding statement, it terminates the inference; The steps include determining whether or not first audio data relating to a statement different from the preceding statement has been received while the text data relating to the response generated by the aforementioned inference is being converted into second audio data, If, while converting the text data relating to the response generated by the above inference into second audio data, it is determined that first audio data relating to a statement different from the preceding statement has been received, the conversion is stopped. The steps include determining whether or not first audio data relating to a statement different from the preceding statement has been received between the time of conversion to the second audio data and the completion of transmission of the second audio data to the terminal device, If it is determined that first audio data relating to a statement different from the preceding statement has been received between the time of conversion to the second audio data and the completion of transmission of the second audio data to the terminal device, the transmission is canceled. A program that executes the command.

2. If the first audio data of a statement different from the preceding statement is determined to be less than or equal to the specified number of characters, In the step of terminating the aforementioned reasoning, if the reasoning is not terminated, In the step of canceling the conversion, the conversion is not canceled. The program according to claim 1, wherein the transmission is not canceled in the step of canceling the transmission.

3. In the second status to which the generating AI responds, if it receives the first audio data relating to a statement different from the preceding statement from the terminal device, it switches the second status to the first status spoken by the user and presents the switch to the terminal device. The program according to claim 1, which causes the processor to execute the following.

4. The program according to claim 1, which causes the processor to perform the step of storing the text relating to the second voice data output from the terminal device in the conversation history.

5. The program according to claim 4, wherein in the storage step, if the ratio in terms of the number of characters of the text relating to the voice output from the terminal device to the text relating to the response satisfies a predetermined requirement, the text relating to the second voice data output from the terminal device is stored in the conversation history, and if the ratio does not satisfy the predetermined requirement, the text relating to the second voice data output from the terminal device is not stored in the conversation history.

6. A method to be performed on a computer comprising a processor and memory, wherein the processor performs all steps performed in any of the inventions according to claims 1 to 5.

7. An information processing apparatus comprising a processor and memory, wherein the processor performs all steps performed in any of the inventions according to claims 1 to 5.

8. A system comprising means for performing all steps performed in the invention according to any one of claims 1 to 5.

Images