Information processing equipment, methods, programs and systems

Abstract

The purpose of this disclosure is to maintain natural communication even when a human interrupts an AI conversation and takes over the conversation. [Solution] A program for operating a computer comprising a processor and memory, the program causing the processor to perform the following steps: receiving first audio data relating to a user's utterance from a terminal device; determining whether or not first audio data relating to a utterance different from the preceding utterance has been received between the time a generating AI starts inferring a response to the first audio data and the time a second audio data converted from text data relating to the response generated by the inference is sent to the terminal device; and if first audio data relating to a utterance different from the preceding utterance has been received between the time the inference is started and the time the second audio data is sent, ceasing the response operation; and invalidating the text data relating to the ceasing response.

Description

Technical Field

[0004] ,

[0006] , , , , , , , , , ,

[0005] ,

[0001] The present disclosure relates to an information processing apparatus, method, program, and system.

Background Art

[0002] In Patent Document 1, a conversation support system is described in which, based on the profile information of a service registrant in advance, AI listens to the surrounding conversation at a gathering such as a drinking party and interrupts the conversation at an appropriate timing to provide topics that make it easier for the person to talk. In Patent Document 1, AI analyzes the surrounding conversation in real time at a gathering such as a drinking party and interrupts the conversation at an appropriate timing to make it easier for the registrant to talk.

[0003] Specifically, in Patent Document 1, the interrupting unit interrupts the conversation at an appropriate timing based on the analyzed information. For example, the interrupting unit interrupts when the conversation has continued for a certain period of time or when a specific keyword appears. Thereby, the interrupting unit interrupts the conversation at an appropriate timing while keeping the flow of the conversation smooth.

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In Patent Document 1, the processing when a human interrupts AI during a conversation is not mentioned. In this case, there may be a request for the human to interrupt the AI during the conversation and take back the conversation. Therefore, when the conversation is interrupted by AI, the human may feel stressed.

[0006] An object of the present disclosure is to maintain natural communication even when a human interrupts AI during a conversation and takes back the conversation.

Means for Solving the Problems

[0007] A program for operating a computer comprising a processor and memory, the program causing the processor to perform the following steps: receiving first audio data relating to a user's utterance from a terminal device; determining whether or not first audio data relating to a utterance different from the preceding utterance has been received between the time a generating AI starts inferring a response to the first audio data and the time a second audio data converted from text data relating to the response generated by the inference is sent to the terminal device; and if first audio data relating to a utterance different from the preceding utterance has been received between the time the inference is started and the time the second audio data is sent, ceasing the response operation; and invalidating the text data relating to the ceasing response. [Effects of the Invention]

[0008] According to this disclosure, even if a human interrupts an AI during a conversation and takes over the conversation, natural communication can be maintained. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing the overall configuration of System 1. [Figure 2] This is a block diagram showing an example of the functional configuration of the terminal device 10. [Figure 3] This block shows a functional configuration example for server 20. [Figure 4] This diagram shows the data structure of a table. [Figure 5] This diagram shows the data structure of a table. [Figure 6] This diagram shows an example of the operation flow in System 1. [Figure 7] This diagram shows an example of the operation flow in System 1. [Figure 8] This figure shows an example of the screen in this disclosure. [Figure 9] This is a block diagram showing the basic hardware configuration of Computer 90. [Modes for carrying out the invention]

[0010] The embodiments of this disclosure will be described below with reference to the drawings. In all the drawings illustrating the embodiments, common components are denoted by the same reference numerals, and repeated explanations are omitted. The following embodiments are not intended to unduly limit the content of this disclosure as described in the claims. Not all components shown in the embodiments are necessarily essential components of this disclosure. Also, each drawing is a schematic diagram and is not necessarily a strict illustration.

[0011] Furthermore, in the following description, "processor" refers to one or more processors. At least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may be another type of processor such as a GPU (Graphics Processing Unit). At least one processor may be single-core or multi-core.

[0012] Furthermore, at least one processor may be a broad-sense processor, such as a hardware circuit that performs some or all of the processing (e.g., an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)).

[0013] Furthermore, in the following explanation, we may use expressions such as "xxx table" to describe information that yields an output for a given input. This information can be data with any structure, or it can be a learning model such as a neural network that generates an output for a given input. Therefore, "xxx table" can be referred to as "xxx information."

[0014] Furthermore, in the following explanation, the structure of each table is just an example; one table may be divided into two or more tables, or all or part of two or more tables may be a single table.

[0015] Also, in the following description, there may be cases where processing is described with "program" as the subject. However, since the program performs the defined processing by being executed by a processor, appropriately using a storage unit and / or an interface unit, etc., the subject of the processing may be the processor (or a device such as a controller having that processor).

[0016] The program may be installed in a device such as a computer, or may be, for example, in a program distribution server or a computer-readable (e.g., non-temporary) recording medium. Also, in the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0017] Also, in the following description, an identification number is used as identification information for various objects, but other types of identification information (e.g., an identifier including letters or symbols) may be adopted.

[0018] Also, in the following description, when describing elements of the same type without distinction, reference signs (or common signs among the reference signs) are used, and when describing elements of the same type while distinguishing them, the identification numbers (or reference signs) of the elements may be used.

[0019] Also, in the following description, the control lines and information lines indicate those considered necessary for the description, and not necessarily all control lines and information lines on the product are shown. All components may be interconnected.

[0020] <Overview> The system according to this embodiment provides an AI avatar conversation service in which a user and an AI avatar converse. The system according to this embodiment has a terminal device receive voice data related to the user's utterance, and receive first voice data related to the user's utterance from the terminal device. The system according to this embodiment determines whether or not it has received first voice data related to an utterance different from a preceding utterance from the time of starting, by a generation AI, the inference of a response to the first voice data until the transmission of second voice data converted from text data related to the response generated by the inference to the terminal device. When the system according to this embodiment has received first voice data related to an utterance different from a preceding utterance from the time of starting the inference until the transmission of the second voice data, the system aborts the operation related to the response. The system according to this embodiment invalidates the text data related to the aborted response.

[0021] In this embodiment, one conversation is defined as including the user's utterances and the AI avatar's responses that are continuously carried out. Any type of conversation can be one conversation. One conversation can be, for example, 10 minutes of casual conversation,  30 minutes of business negotiation, 60 minutes of job interview, 90 minutes of media interview, and other conversations, interviews, face-to-face talks, hearings, telephone reception, primary consultation window for specialized services, role-playing (e.g., customer service practice). Also, in this embodiment, an example of an interview format in which a user converses with a generation AI model via an avatar is taken up. The avatar is, for example, an interface that has a human appearance and voice and can interact with the user based on output data from the generation AI model. The avatar can take, for example, an icon of any human appearance.

[0022] <1. Overall Configuration Diagram of the System> FIG. 1 is a block diagram showing an example of the overall configuration of system 1. As shown in FIG. 1, system 1 includes, for example, terminal device 10, server 20, and generation AI system 30. Terminal device 10, server 20, and generation AI system 30 are communicatively connected via, for example, network 80.

[0023] Figure 1 shows an example where System 1 includes two terminal devices 10, but the number of terminal devices 10 included in System 1 is not limited to two. System 1 may include one terminal device 10, or it may include three or more terminal devices 10.

[0024] Figure 1 shows an example where System 1 includes one Generative AI System 30, but the number of Generative AI Systems 30 included in System 1 is not limited to one. System 1 may include two or more Generative AI Systems 30. Also, Figure 1 shows an example where the Generative AI System 30 is independent of the Server 20, but the Server 20 may include the functions of the Generative AI System 30. In other words, the Server 20 may store the Generative AI.

[0025] In this embodiment, a collection of multiple devices may be treated as a single server. The method of allocating the multiple functions required to implement the server 20 according to this embodiment to one or more hardware can be appropriately determined in view of the processing capacity of each hardware and / or the specifications required for the server 20.

[0026] Terminal device 10 is an information processing device used by users who utilize the conversation service provided by server 20. Terminal device 10 provides users with a user interface for engaging in conversational dialogue with the generating AI provided by the generating AI system 30. Terminal device 10 can be implemented as, for example, a stationary PC (Personal Computer), a laptop PC, a head-mounted display, etc. Alternatively, terminal device 10 may be a portable computer such as a smartphone or tablet device.

[0027] The terminal device 10 comprises a communication interface (IF) 12, an input device 13, an output device 14, a memory 15, storage 16, and a processor 19. The communication interface 12 is an interface for inputting and outputting signals so that the terminal device 10 can communicate with devices in system 1, such as a server 20. The input device 13 is a device for receiving input operations from the user (e.g., a touch panel, touchpad, pointing device such as a mouse, keyboard, etc.). The output device 14 is a device for presenting information to the user (display, speaker, etc.). The memory 15 is for temporarily storing programs and data processed by programs, etc., and is a volatile memory such as DRAM (Dynamic Random Access Memory). The storage 16 is for saving data, and is a flash memory, HDD (Hard Disk Drive), etc. The processor 19 is hardware for executing the instruction set written in the program, and is composed of an arithmetic unit, registers, peripheral circuits, etc.

[0028] Server 20 is an information processing device used by a service provider that provides conversation services (hereinafter referred to as "service provider"). Server 20 is an information processing device implemented by, for example, a computer connected to network 80.

[0029] The server 20 comprises a communication interface 22, an input / output interface 23, a memory 25, storage 26, and a processor 29. The communication interface 22 is an interface for inputting and outputting signals so that the server 20 can communicate with devices in system 1, such as a terminal device 10. The input / output interface 23 functions as an interface to an input device for receiving input operations from a user and an output device for outputting information to the user. The memory 25 is for temporarily storing programs and data processed by programs, etc., and is a volatile memory such as DRAM. The storage 26 is for saving data, and is a flash memory or HDD, for example. The processor 29 is hardware for executing the instruction set written in the program, and is composed of an arithmetic unit, registers, peripheral circuits, etc.

[0030] Each information processing device consists of a computer equipped with an arithmetic unit and a memory device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by said hardware configuration will be described later. For each of the terminal device 10 and the server 20, explanations that overlap with the basic hardware configuration and basic functional configuration of the computer described later will be omitted.

[0031] The generative AI system 30 is, for example, a system on which generative AI (generative artificial intelligence) is constructed. The generative AI system 30 functions as a system for facilitating conversations between a user and the generative AI. That is, the generative AI system 30 also plays a role in generating the speech content of the AI ​​avatar, such as by posing statements to the user or responding to statements from the user.

[0032] The generative AI built into the generative AI system 30 is, for example, a Large Language Model (LLM). A Large Language Model is a natural language model designed to perform multiple tasks of natural language processing. A Large Language Model is an example of a trained model, trained using a large number of parameters (e.g., billions to hundreds of billions) and high-level computing resources. A natural language model refers to a computer program or algorithm designed to perform tasks of natural language processing. For example, in natural language processing, processes such as morphological analysis, syntactic analysis, information extraction, and text generation are performed, enabling a computer to analyze human language (i.e., natural language) and perform predetermined processing. A Large Language Model generates output based on the text, image, etc., of a prompt (instruction) that is input. The prompt can be defined in natural language.

[0033] Examples of large-scale language models include the GPT® series (Generative Pre-trained Transformer) developed by OPEN AI, BERT (Bidirectional Encoder Representations from Transformers) developed by Google, StableLM developed by StableAI, and Llama2, Palm2®, and LaMDA2® developed by Meta. Large-scale language models tend to have high training costs because they are trained using a very large number of parameters and computational resources. In this embodiment, the training cost is reduced by using publicly available pre-trained models by the generative AI system 30. In this embodiment, for example, a definition is passed to GPT®, allowing it to understand the context and then generate information.

[0034] A prompt is primarily a query (including text, strings, images, videos, audio, etc.) input to a generating AI. Users of a generating AI can instruct it to process information by inputting prompts. Users can input prompts to the generating AI to produce the desired output. Prompts do not necessarily have to be strings; they can also be images, videos, audio, etc. For example, gestures and voice commands from the user can also serve as prompts.

[0035] Prompts can be entered through input operations on the user's terminal. Document files, images, videos, audio, etc., can also be uploaded and used as prompts. Users can create prompts by combining instructions entered on their terminal with other instructions, data, etc. In this case, a prompt may contain one or more instructions, data, etc. Alternatively, a prompt may contain only one or more instructions, or only one or more data. A prompt can be created by including other instructions, data, etc., as part of a predetermined set of instructions, data, etc. A prompt can be created by inserting other instructions, data, etc., as part of a predetermined set of instructions, data, etc. A prompt can be created by combining predetermined instructions, data, etc., with other instructions, data, etc. A prompt can be created by combining predetermined instructions, data, etc., with other instructions, data, etc. A prompt can be created by adding other instructions, data, etc., to predetermined instructions, data, etc.

[0036] In this disclosure, expressions such as "include," "insert," "combine," "combine," and "add" used when creating prompts from predetermined instructions, data, etc., are used as terms that refer to the same information processing. In other words, they can be treated as the same information processing in that the same prompt is created based on one or more instructions, data, etc. For example, the term "include" includes information processing such as "insert," "combine," "combine," and "add." Similarly, "insert," "combine," "combine," and "add" also include information processing such as "include," "insert," "combine," "combine," and "add," respectively.

[0037] <2. Configuration of terminal equipment> Figure 2 is a block diagram showing an example of the functional configuration of the terminal device 10. As shown in Figure 2, the terminal device 10 includes a communication unit 120, an input device 13, an output device 14, an audio processing unit 17, a microphone 171, a speaker 172, a camera 160, a location information sensor 150, a storage unit 180, and a control unit 190. Each block included in the terminal device 10 is electrically connected, for example, by a bus.

[0038] The communication unit 120 performs processing such as modulation and demodulation processing for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on the signal generated by the control unit 190 and transmits it to an external source (for example, the server 20). The communication unit 120 performs reception processing on the signal received from an external source and outputs it to the control unit 190.

[0039] The input device 13 is a device for a user operating the terminal device 10 to input instructions or information. The input device 13 can be implemented, for example, by a touch-sensitive device 131 on which instructions are input by touching the operating surface. If the terminal device 10 is a PC, the input device 13 may be implemented by a reader, keyboard, mouse, etc. The input device 13 converts the instructions input by the user into electrical signals and outputs the electrical signals to the control unit 190. The input device 13 may also include, for example, a receiving port that accepts electrical signals input from an external input device.

[0040] The output device 14 is a device for presenting information to the user operating the terminal device 10. The output device 14 is implemented, for example, by a display 141. The display 141 displays data according to the control of the control unit 190. The display 141 is implemented, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0041] The audio processing unit 17 performs, for example, digital-to-analog conversion processing of the audio signal. The audio processing unit 17 converts the signal received from the microphone 171 into a digital signal and provides the converted signal to the control unit 190. The audio processing unit 17 also provides the audio signal to the speaker 172. The audio processing unit 17 is implemented, for example, by an audio processing processor. The microphone 171 receives an audio input and provides the audio signal corresponding to that audio input to the audio processing unit 17. The speaker 172 converts the audio signal received from the audio processing unit 17 into audio and outputs the audio to the outside of the terminal device 10.

[0042] Camera 160 is a device that receives light using a photodetector and outputs it as a shooting signal.

[0043] The location information sensor 150 is a sensor that detects the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. A GPS module is a receiving device used in a satellite positioning system. In a satellite positioning system, signals are received from at least three or four satellites, and the current position of the terminal device 10, which is equipped with a GPS module, is detected based on the received signals. The location information sensor 150 may also detect the current position of the terminal device 10 from the position of the wireless base station to which the terminal device 10 is connected.

[0044] The storage unit 180 is implemented by, for example, memory 15 and storage 16, and stores data and programs used by the terminal device 10. The storage unit 180 stores, for example, user information 181.

[0045] User information 181 includes, for example, information about the user who uses the terminal device 10. User information includes, for example, the user's name, age, address, date of birth, contact information, etc.

[0046] The control unit 190 is realized when the processor 19 reads a program stored in the memory unit 180 and executes instructions contained in the program. The control unit 190 controls the operation of the terminal device 10. By operating according to the program, the control unit 190 performs the functions of an operation reception unit 191, a transmission / reception unit 192, a presentation control unit 193, and a conversation control unit 194.

[0047] Specifically, the control unit 190 determines whether or not it has received a statement from the user at a predetermined timing. Furthermore, if the control unit 190 has received a statement from the user at a predetermined timing, it controls the system so as not to output the AI ​​avatar's response audio data to the user.

[0048] The operation reception unit 191 processes instructions or information input from the input device 13. Specifically, for example, the operation reception unit 191 receives instructions or information input from a touch-sensitive device 131 or the like.

[0049] Furthermore, the operation reception unit 191 receives voice instructions input from the microphone 171. Specifically, for example, the operation reception unit 191 receives voice signals input from the microphone 171 and converted into digital signals by the voice processing unit 17. The operation reception unit 191 obtains instructions from the user by, for example, analyzing the received voice signals and extracting predetermined nouns.

[0050] The transmitting / receiving unit 192 performs processing to enable the terminal device 10 to send and receive data with an external device such as the server 20 in accordance with a communication protocol. Specifically, for example, the transmitting / receiving unit 192 sends information input by the user or instructions from the user to the server 20. The transmitting / receiving unit 192 also receives information provided by the server 20.

[0051] The presentation control unit 193 controls the output device 14 to present information provided by the server 20 to the user. Specifically, for example, the presentation control unit 193 displays the information transmitted from the server 20 on the display 141. As an example, the presentation control unit 193 displays an AI avatar in the browser of the terminal device 10. The presentation control unit 193 may also display the AI ​​avatar in a conversation-specific application of the terminal device 10. The presentation control unit 193 also outputs the information transmitted from the server 20 through the speaker 172.

[0052] The conversation control unit 194 controls the conversation between the user and the avatar. Specifically, the control unit 190 determines whether or not it has received a statement from the user at a predetermined time. More specifically, if the control unit 190 has received a statement from the user at a predetermined time, it controls the system so as not to output the AI ​​avatar's response audio data to the user.

[0053] <3. Functional Configuration of the Server> Figure 3 shows an example of the functional configuration of server 20. As shown in Figure 3, server 20 functions as a communication unit 201, a storage unit 202, and a control unit 203.

[0054] The communications unit 201 performs processing to enable the server 20 to communicate with external devices.

[0055] The memory unit 202 includes, for example, a user table 2021 and a conversation table 2022, etc. The tables stored in the memory unit 202 are not limited to these.

[0056] User Table 2021 is a table that stores information related to users.

[0057] Conversation Table 2022 is a table that stores information related to conversations between the user and the generating AI.

[0058] The control unit 203 functions as a module that controls all aspects of the conversation service processing by operating according to the program.

[0059] Specifically, the control unit 203 controls the conversation between the generating AI and the user. The control unit 203 controls the speech and actions of the avatar displayed on the display 141, for example, via the presentation control unit 193. The avatar's speech and actions are directly controlled by the presentation control unit 193. Specifically, the control unit 203 determines whether or not it has received audio data of the user's speech at a predetermined timing. Furthermore, if the control unit 203 has received audio data of the user's speech at a predetermined timing, it controls the system so as not to output audio data of the avatar's response to the terminal device 10.

[0060] The control unit 203 is implemented by the processor 29 reading a program stored in the memory unit 202 and executing instructions contained in the program. The program includes applications such as web browser applications. The program includes programming languages ​​such as JavaScript® that are executed on the web browser application stored in the terminal device 10. The control unit 203 may implement the functions according to this embodiment by cooperating with external systems or services using an API (Application Programming Interface) as needed during the process of executing the program. That is, the control unit 203 may, for example, call a program stored in an external system using an API.

[0061] <4. Data Structure> This section describes the data structure of the tables stored by server 20. Note that the described data structure is an example and does not exclude data not listed. Furthermore, even data listed in the same table may be stored in separate memory areas within the storage unit 202. Each table may have columns other than those described in that table. Each table may not have any of the columns described in that table.

[0062] Figure 4 shows the data structure of User Table 2021. As shown in Figure 4, User Table 2021 has columns such as Name, Date of Birth, Email Address, etc., with User ID as the key.

[0063] The User ID is a column that stores an identifier to uniquely identify a user.

[0064] The "Name" column stores the user's name.

[0065] The date of birth column stores the user's date of birth.

[0066] The email address column stores the user's email address.

[0067] Figure 5 shows the data structure of the conversation table 2022. As shown in Figure 5, the conversation table 2022 has columns such as conversation ID, date and time, status, and message / response content, with user ID as the key. An example of a single conversation is, for example, a single daily work report.

[0068] The User ID column stores the User ID (same as in Figure 4) of the user who conversed with the generated AI.

[0069] The conversation ID is a column that stores an identifier to uniquely identify each conversation. The control unit 203 assigns one conversation ID to each conversation.

[0070] The date and time column stores the date and time the conversation took place.

[0071] The status system stores the status for each statement / response within a single conversation. The first status is the status of the user's statement. The second status is the status of the avatar's (generated AI) response.

[0072] The control unit 203 switches the status each time the speaker changes during a conversation. For example, when the user finishes speaking in the first status, the control unit 203 determines that the first status has ended and switches from the first status to the second status. The control unit 203 transmits to the terminal device 10 that it has switched to the second status. Also, for example, when the terminal device 10 has finished outputting the avatar's voice data in the second status, it transmits to the server 20 that it has finished outputting the voice data. When the control unit 203 receives from the terminal device 10 that it has finished outputting the voice data, it switches from the second status to the first status.

[0073] The "Statement / Response Content" column stores the user's statements and the AI's responses. For example, the Statement / Response Content stores the user's statements and the AI's responses as text turn by turn, following the flow of an actual conversation. Each statement and response may have a timestamp indicating the time when the statement and response occurred. The Statement / Response Content may store, for example, the audio of the conversation, the video of the conversation, or both. The server 20 receives, for example, information related to the user's statements and information related to the responses output from the terminal device 10 (information related to the avatar's statements) from the terminal device 10 and stores them in the Statement / Response Content.

[0074] <5. Operation> An example of the operation flow in System 1 is described below.

[0075] <5.1. Operation 1>

[0076] Figure 6 is a flowchart illustrating an example of a conversation between a user and a generated AI in System 1. Steps S1001 through S1007 are repeated during a single conversation.

[0077] First, the user accesses a conversational platform in a face-to-face format, where they converse with a generated AI model via an avatar, through a predetermined operation, for example, using a web browser or app on a terminal device 10. The predetermined operation is, for example, logging in. The transmitting / receiving unit 192 sends a conversation start request to the server 20 via the communication unit 120. The control unit 203 receives the conversation start request, for example, via the communication unit 201. The control unit 203 then instructs the generated AI system 30 to start the conversation, for example, via the communication unit 201. The instruction is, for example, "Please have a casual conversation with the user for 10 minutes. Please respond to the user's statements."

[0078] In step S1001, the terminal device 10 receives a statement from the user. The statement can be an affirmative sentence, a negative sentence, a question, a command, a greeting, etc. For example, the statement could be, "I want to start exercising." Specifically, for example, the operation reception unit 191 receives the user's statement input as first voice data via the input device 13. In this embodiment, the first voice data refers to, for example, the voice data of the user's statement received by the terminal device 10.

[0079] In step S1002, the server 20 receives the first voice data of the user's utterance. Specifically, the transmitting / receiving unit 192 transmits the first voice data to the server 20, for example, via the communication unit 120. The control unit 203 receives the first voice data, for example, via the communication unit 201.

[0080] In step S1003, the generating AI system 30 infers a response to the utterance. Specifically, the control unit 203 switches from the first status to the second status when the user finishes uttering, for example. The control unit 203 converts the first audio data relating to the user's utterance into text data. The control unit 203 stores the text data relating to the first audio data in the conversation table 2022. For example, the control unit 203 adds a record to the conversation table 2022 and stores the information relating to the text data relating to the first audio data in each column as needed. The control unit 203 transmits the text data of the utterance to the generating AI system 30 via the communication unit 201, for example.

[0081] The generating AI system 30 outputs text data in response to a statement, such as answers to the user's questions, further explanations of the user's statements, advice to the user, encouragement to the user, greetings, fillers, etc. For example, the text data might be, "If you're going to exercise, I recommend jogging." For example, the generating AI system 30 sends the text data to the server 20. The control unit 203 receives the text data, for example, via the communication unit 201. In addition, in order to maintain the context of the entire conversation, the generating AI system 30 refers to the conversation history at the time, that is, the text data related to the user's statements (first voice data) and the avatar's statements (second voice data) up to that point in the conversation (excluding invalidated text data), before outputting the text data. For example, if the generating AI system 30 refers to a statement made by the avatar up to that point in the conversation whose text data has not been invalidated (i.e., the text data remains), it outputs the text data "As I said earlier" to indicate that it has already been mentioned in the conversation. For example, if the generating AI system 30 refers to a statement made by the avatar in the conversation whose text data has been invalidated, it will not output "As I said earlier" as text data, but will instead output text data that makes it appear as if it is being mentioned for the first time in that conversation.

[0082] In step S1004, the server 20 converts the text data into second audio data. Specifically, the control unit 203 converts the received text data into second audio data, for example. In this embodiment, the second audio data refers to, for example, the audio data of the response that the avatar will speak.

[0083] In step S1005, the server 20 transmits the second voice data to the terminal device 10. Specifically, the control unit 203 transmits the second voice data to the terminal device 10, for example, via the communication unit 201. The terminal device 10 receives the second voice data. The control unit 190 receives the second voice data, for example, via the transmitting / receiving unit 192.

[0084] In step S1006, the terminal device 10 prepares the output of the second audio data received from the server 20. Specifically, the control unit 190 performs, for example, a decoding process of the received compressed second audio data. The control unit 190 also performs, for example, a buffering process of the received compressed second audio data for smooth playback.

[0085] In step S1007, the terminal device 10 outputs the second voice data to the user. Specifically, the presentation control unit 193 controls, for example, the speech and actions of the avatar displayed on the display 141, causing the avatar to speak the second voice data. Once the terminal device 10 has finished outputting the avatar's voice data, it sends a message to the server 20 indicating that it has finished outputting the voice data. When the control unit 203 receives a message from the terminal device 10 indicating that it has finished outputting the voice data, it switches the second status to the first status.

[0086] Furthermore, after step S1007, the server 20 stores the text data related to the outputted second audio data. Specifically, the terminal device 10 sends information to the server 20 indicating that the second audio data has been output to the user. The server 20 stores the text data of the second audio data output from the terminal device 10 in the conversation table 2022. The control unit 203, for example, adds a record to the conversation table 2022 and stores the information related to the text data of the second audio data output from the terminal device 10 in each column as necessary.

[0087] <5.2. Operation 2>

[0088] Figure 7 is a flowchart illustrating an example of a conversation between a user and a generated AI in System 1.

[0089] First, the conversation is in the second status (one of the steps S1002 to S1007 in Figure 6).

[0090] In step S1101, the terminal device 10 receives a statement different from the preceding statement in Figure 6 (hereinafter referred to as a different statement). A different statement is, for example, a statement about something that was forgotten in the previous statement, a statement that interrupts the avatar's statement, etc. Specifically, a different statement is, for example, "Indoor exercise is good." Specifically, for example, the operation reception unit 191 receives the different statement from the user as first voice data via the input device 13.

[0091] In step S1102, the server 20 receives first audio data relating to a different statement. Specifically, the transmitting / receiving unit 192 transmits the first audio data to the server 20, for example, via the communication unit 120. The control unit 203 receives the first audio data, for example, via the communication unit 201. Upon receiving the first audio data, the control unit 203 switches from the second status to the first status.

[0092] In step S1103A, the terminal device 10 determines whether it has received a different statement between the time it receives the second audio data and the time it starts outputting the second audio data (for example, while it is performing the processing in step S1006). Specifically, for example, the control unit 190 determines whether it has received a different statement while preparing to output the second audio data. If the determination is yes, the process proceeds to S1103B. If the determination is no, the process proceeds to S1103C.

[0093] In step S1103B, the terminal device 10 is controlled so as not to output the second audio data to the user. Specifically, the control unit 190 sets the output intensity of the second audio data to zero. As a result, the volume of the second audio data becomes zero, and the second audio does not reach the user. For example, the volume of the second audio data, "Jogging is recommended," becomes zero from the first character. The presentation control unit 193 may, for example, control the avatar's movements in synchronization with setting the volume to zero. For example, the avatar stops the mouth movements that would normally produce speech.

[0094] In step S1103C, the terminal device 10 determines whether it has received a different statement while outputting the second audio data (i.e., while performing the processing in step S1007). Specifically, the control unit 190 determines whether it has received a different statement while outputting the second audio data. If the determination is yes, the process proceeds to S1103D. If the determination is no, the process proceeds to S1104A.

[0095] In step S1103D, the terminal device 10 controls itself so as not to output the second audio data to the user after receiving a different statement. Specifically, the control unit 190 zeros out the output intensity of the second audio data after receiving the different statement. As a result, the volume of the second audio data after receiving the different statement becomes zero, and the second audio after receiving the different statement does not reach the user. For example, if the second audio data "Jogging is recommended" has been output up to "Jogging is", the volume of the remaining "recommended" will be zero. The presentation control unit 193 may, for example, control the avatar's movements in synchronization with setting the volume to zero. For example, the avatar may stop the movement of its mouth when speaking.

[0096] In step S1104A, the server 20 determines whether it has received first audio data relating to a different statement while the response generation AI is inferring a response (i.e., during step S1003). Specifically, the control unit 203 determines whether it has received first audio data while the generation AI is inferring a response. If the determination is yes, the process proceeds to S1104B. If the determination is no, the process proceeds to S1104C.

[0097] In step S1104B, the server 20 stops the response-related operations. Specifically, the control unit 203 stops the response generation AI's inference.

[0098] In step S1104C, the server 20 determines whether it has received first audio data relating to a different statement while converting to second audio data (i.e., during step S1004). Specifically, the control unit 203 determines whether it has received first audio data while converting text data to second audio data. If the determination is yes, the process proceeds to S1104D. If the determination is no, the process proceeds to S1104E.

[0099] In step S1104D, the server 20 cancels the response operation. Specifically, the control unit 203 cancels the conversion to the second audio data.

[0100] In step S1104E, the server 20 determines whether it has received first audio data relating to a different statement between the time of conversion to second audio data and the completion of transmission of the second audio data to the terminal device 10 (i.e., before the completion of step S1005). Specifically, the control unit 203 determines whether it has received first audio data between the time of conversion to second audio data and before transmission of the second audio data to the terminal device 10. If the determination is yes, the process proceeds to S1104F. If the determination is no, the process proceeds to step S1003, and the first audio data is processed according to the flow in Figure 6.

[0101] In step S1104F, the server 20 stops the response operation. Specifically, the control unit 203 stops transmitting the second voice data to the terminal device 10.

[0102] Furthermore, after the server 20 cancels an operation related to the response (inference, conversion, or transmission) in step S1104B, S1104D, or S1104F, the server 20 invalidates the text data related to the second voice data whose response was canceled. Specifically, the control unit 203 discards the text data related to the canceled response that was being generated by the generation AI system 30 or the text data related to the canceled response that has already been generated. In other words, the control unit 203 does not store the text data related to the canceled response in the conversation table 2022. The control unit 203 may store the text data related to the canceled response in the conversation table 2022 with an invalidation flag so that it is not referenced as conversation history. If the text data related to the canceled response is already stored in the conversation table 2022, the control unit 203 deletes the text data related to the canceled response from the conversation table 2022, or adds an invalidation flag to the text data related to the canceled response so that it is not referenced as conversation history.

[0103] Furthermore, after the terminal device 10 controls the system in step S1103B or S1103D to prevent outputting the second voice data to the user, the server 20 invalidates the text data related to the controlled second voice data. Specifically, the control unit 190 sends a message to the server 20 indicating that it has controlled the system to prevent outputting the second voice data to the user, and the control unit 203 receives this message. The control unit 203 discards the text data related to the controlled response that has already been generated by the generation AI system 30. In other words, the control unit 203 does not store the text data related to the controlled response in the conversation table 2022. The control unit 203 may store the text data related to the controlled response in the conversation table 2022 with an invalidation flag to prevent it from being referenced as conversation history. If the text data related to the controlled response is already stored in the conversation table 2022, the control unit 203 deletes the text data related to the controlled response from the conversation table 2022, or adds an invalidation flag to the text data related to the controlled response to prevent it from being referenced as conversation history.

[0104] <6. Screen example> An example of the screen of the display 141 of the terminal device 10 in this disclosure is described below.

[0105] Figure 8 shows an example screen when a user converses with a generated AI via an avatar on the terminal device 10 (interview format). This screen is a user interface for conversation between the user and the generated AI, for example, to receive statements from the user and to present the user with responses from the generated AI. The presentation control unit 193 controls, for example, the speech and actions of the avatar displayed on the display 141, and causes the avatar to speak voice data. Note that this screen example is merely an example, and various screen configurations and screen contents can be adopted. For example, only the avatar 3011 described later may be displayed on the display 141.

[0106] The avatar display area 3001 is an area where an image or animation of the avatar 3011 is displayed. The avatar 3011 is, for example, an interface that has the appearance and voice of a human and speaks and acts under the control of the control unit 203. When the avatar 3011 speaks, the movements of the avatar 3011's mouth and other features may be expressed in sync. The avatar 3011's speech is realized, for example, by sound output from the speaker 172, but it may also be realized by displaying the content of the speech as text near the display area of ​​the avatar 3011. In this case, the avatar 3011 may function only as an icon without speaking.

[0107] The speech display area 3002 is an area where speech entered by the user via voice or text is displayed in text format.

[0108] The response display area 3003 is an auxiliary display area for the user to recognize the response from the avatar 3011. Since the avatar 3011 responds by voice, the text of the response may not be displayed in this area, or if text data is sent from the server 20 along with the voice data, that text may be displayed.

[0109] The input means display area 3004 is an area that displays means for the user to input a statement to the avatar 3011. In this example screen, for example, a microphone icon 3014 for starting voice input and a field 3024 for supplementary text input are displayed in this area. For example, it is assumed that when the operation reception unit 191 receives a press operation of the microphone icon 3014 by the user, voice input from the microphone 171 will be started to be received.

[0110] The send button 3005 is a button used to send a statement (mainly voice input or text input) entered by the user via the input means display area 3004 to the server 20.

[0111] The conversation log display area 3006 is an area where the history of a series of interactions between the user and the avatar 3011 (specifically, the generated AI) is displayed in chronological order. The user can, for example, scroll through and review past conversation content.

[0112] <7.Summary> As described above, in the above embodiment, the server 20 receives first audio data relating to the user's utterance from the terminal device 10. From the time the server 20 starts generating a response to the first audio data until it sends the second audio data relating to the response, which has been converted from the text data relating to the response generated by inference, to the terminal device 10, the server 20 determines whether it has received first audio data relating to a different utterance than the preceding utterance. If the server 20 receives first audio data relating to a different utterance between the start of inference and the transmission of the second audio data, it cancels the response operation. The server 20 invalidates the text data relating to the canceled response. Furthermore, if the terminal device 10 receives a different utterance between the time of receiving the second audio data and during the output of the second audio data, it controls the terminal device 10 not to output the second audio data to the user. The server 20 receives the control from the terminal device 10 and invalidates the text data relating to the controlled response.

[0113] As a result, when the AI ​​avatar is about to respond, or is responding, if the user speaks, the AI ​​avatar will refrain from responding. Therefore, even if the user interrupts the AI ​​in the middle of a conversation and takes back the conversation, natural communication can be maintained. Furthermore, only the avatar's responses that the user actually heard are remembered and referenced as conversation history. Therefore, the generating AI system 30 can preserve the context of the entire conversation in subsequent processing of the conversation (subsequent inference and response text data output, conversation summarization, conversation evaluation, user evaluation, user analysis, etc.), and can provide the user with a conversational experience that feels natural.

[0114] <8. Variation> A modified example of the above embodiment will be described.

[0115] <8.1. Variation 1> Server 20 determines whether the first audio data for a different statement is less than or equal to a predetermined number of characters (for example, 6 characters or less), and if it determines that the different statement is less than or equal to the predetermined number of characters, Server 20 does not have to stop the response operation. Specifically, even if Server 20 receives first audio data for a different statement while inference is in progress, in step S1104B, the control unit 203 does not stop inference. Also specifically, even if Server 20 receives first audio data for a different statement while speech conversion is in progress, in step S1104D, the control unit 203 does not stop conversion. Also specifically, even if Server 20 receives first audio data for a different statement before the transmission of the second audio data is completed, in step S1104F, the control unit 203 does not stop transmission.

[0116] Furthermore, the terminal device 10 determines whether the first audio data of a different statement is less than or equal to a predetermined number of characters (for example, 6 characters or less), and if it determines that the different statement is less than or equal to the predetermined number of characters, the terminal device 10 does not have to prevent the output of the second audio data to the user. Specifically, even if the control unit 190 receives a different statement while preparing to output the second audio data, in step S1103B, the control unit 190 does not zero out the output intensity of the second audio data. Also, specifically, even if the control unit 190 receives a different statement while outputting the second audio data, in step S1103D, the control unit 190 does not zero out the output intensity of the second audio data.

[0117] This allows for natural conversation to be maintained without interruption when the first voice data consists of short fillers ("Oh," "Really," etc.) that do not require the avatar to interrupt the second voice data's output.

[0118] <8.2. Modification 2> In the above embodiment, the first status is the status of the user making a statement, and the second status is the status of the avatar (generated AI) responding.

[0119] While the avatar is responding, it is in the second status, but in step S1102, when the server 20 receives first audio data relating to a different statement from the terminal device 10, the control unit 203 switches from the second status to the first status. In this case, the server 20 may indicate the switch to the terminal device 10. Specifically, the control unit 203 transmits information relating to the switch to the terminal device 10, for example. The notification control unit 193 suggests the switch, for example, in the screen example in Figure 8. The notification control unit 193 displays information suggesting the switch, for example, "It's your turn," "The turn has changed," etc. Alternatively, the notification control unit 193 may also suggest the switch by hiding the text "It's the avatar's turn."

[0120] This makes the switch more easily communicated to the user, allowing them to easily understand whether it is the user's or the avatar's turn.

[0121] <8.3. Modification 3> In the above embodiment, the terminal device 10 was described as a stationary PC or the like. However, the terminal device 10 may also be a robot modeled after a human or character. A robot terminal device 10 may not have, for example, an input device 13, an output device 14, an operation reception unit 191, or a presentation control unit 193. Specifically, the robot has the basic configuration of a computer, such as a communication I / F 12, memory 15, storage 16, and a processor 19. When the robot plays the role of an avatar, instead of an avatar displayed on a screen, the robot presents information to the user through its own physical movements and speech. Therefore, the screen example in Figure 8 is unnecessary, and the presentation control unit 193 becomes a functional block that directly controls the robot's movements and speech. The user can input voice as speech using a microphone built into the robot. The operation reception unit 191 is dedicated to processing voice input. In addition, the robot's own speaker plays the role of speaker 172 and outputs the response from the generating AI system 30 as voice. In this case, a display as output device 14 is not necessarily required. Thus, unlike conventional stationary PCs and smartphones, the robotic terminal device 10 has its physical form and function closely linked because the robot itself performs the function of an avatar. This makes it possible to provide a more intuitive and natural communication experience in which the user interacts with the generated AI through the physical presence of the robot, rather than through an avatar.

[0122] <8.4. Modification 4> In the above embodiment, an example was described in which, after the terminal device 10 controls the system in step S1103B or S1103D not to output the second voice data to the user, the server 20 uniformly invalidates the text data related to the second voice data for which the response was controlled. However, the server 20 may also divide its function into a part that invalidates the text data related to the second voice data for which the response was controlled and a part that stores the text data.

[0123] Specifically, the control unit 190 identifies the range of text corresponding to the audio actually output from the speaker 172 (hereinafter referred to as the output segment) before the output of the second audio data is stopped. Specifically, the control unit 190 identifies which words or phrases of the entire response text were presented to the user, based on the playback position (timestamp or data index) of the second audio data at the time the output intensity was reduced to zero. The transmitting / receiving unit 192 transmits information regarding the identified output segment to the server 20.

[0124] Based on the information about the output segments received from the terminal device 10, the control unit 203 stores the actually outputted segments in the utterance / response content column of the conversation table 2022. Alternatively, the control unit 203 invalidates the unoutput range, i.e., discards the unoutput range, or stores the entire response text and then assigns an "unoutputted flag" to the unoutput range before storing it.

[0125] This allows the generating AI system 30 to accurately refer to the context of how far the avatar has spoken in subsequent conversations. For example, if the avatar speaks up to the part of the second voice data "When exercising, it is recommended to wear clothes that are easy to take off and put on" that says "When exercising, wear clothes that are easy to take off," and the terminal device accepts an interruption from the user, the generating AI system 30 can recognize that the outputted segment is cut off in the middle of a sentence and generate text data of a natural continuation that makes the user aware of resuming the interrupted topic, such as "Regarding the clothes you were about to say when exercising..." in the next response.

[0126] <9. Basic Computer Hardware Configuration> Figure 9 is a block diagram showing the basic hardware configuration of computer 90. Computer 90 includes at least a processor 901, main memory 902, auxiliary memory 903, and a communication interface 991. These are electrically connected to each other by a communication bus 921.

[0127] The processor 901 is hardware for executing the instruction set written in a program. The processor 901 consists of an arithmetic unit, registers, peripheral circuits, etc.

[0128] Main memory 902 is used to temporarily store programs and data processed by programs, etc. For example, it is a volatile memory such as DRAM (Dynamic Random Access Memory).

[0129] Auxiliary storage device 903 refers to a storage device for saving data and programs. Examples include flash memory, HDD (Hard Disc Drive), magneto-optical disk, CD-ROM, DVD-ROM, and semiconductor memory.

[0130] The IF991 communication interface is an interface for inputting and outputting signals for communication with other computers via a network using wired or wireless communication standards. A network consists of various mobile communication systems, such as the internet, LANs, and wireless base stations. For example, a network includes 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks that can connect to the internet via designated access points (e.g., Wi-Fi®). When connecting wirelessly, communication protocols include, for example, Z-Wave®, ZigBee®, and Bluetooth®. When connecting via a wired connection, the network also includes connections made directly via USB (Universal Serial Bus) cables, etc.

[0131] Furthermore, by distributing all or part of each hardware configuration across multiple computers 90 and connecting them to each other via a network, a computer 90 can be virtually realized. Thus, the concept of computer 90 includes not only a computer 90 housed in a single enclosure or case, but also a virtualized computer system.

[0132] <Basic Functional Configuration of Computer 90> The functional configuration of the computer realized by the basic hardware configuration of computer 90 is described below. The computer comprises at least one functional unit: a control unit, a memory unit, and a communication unit.

[0133] Furthermore, the functional units of computer 90 can also be realized by distributing all or part of each functional unit across multiple computers 90 interconnected via a network. The concept of computer 90 includes not only a single computer 90 but also a virtualized computer system.

[0134] The control unit is realized when the processor 901 reads various programs stored in the auxiliary storage device 903, loads them into the main memory device 902, and executes processing according to those programs. The control unit can realize various functional units that perform information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0135] The memory unit is implemented by the main memory 902 and the auxiliary memory 903. The memory unit stores data, various programs, and various databases. The processor 901 can also reserve memory areas corresponding to the memory unit in the main memory 902 or the auxiliary memory 903 according to the program. The control unit can also cause the processor 901 to perform operations such as adding, updating, and deleting data stored in the memory unit according to the various programs.

[0136] A database, specifically a relational database, is used to manage and link together tabular data sets called masters, which are structurally defined by rows and columns. In a database, tables are called tables, masters are called masters, the columns of tables are called columns, and the rows of tables are called records. In a relational database, relationships can be established and linked between tables and masters. Typically, each table and master has a primary key column to uniquely identify records, but setting a primary key column is not mandatory. The control unit can instruct the processor 901 to add, delete, or update records in specific tables and masters stored in the memory unit, according to various programs. Furthermore, by storing data, various programs, and various databases in the memory unit, the information processing device and information processing system related to this disclosure can be considered to have been manufactured.

[0137] Furthermore, the databases and masters in this disclosure may include any data structures (lists, dictionaries, associative arrays, objects, etc.) in which information is structurally defined. Data structures also include data that can be considered as data structures by combining data with functions, classes, methods, etc., written in any programming language.

[0138] The communication unit is implemented by the communication IF991. The communication unit provides the functionality to communicate with other computers 90 via the network. The communication unit can receive information transmitted from other computers 90 and input it to the control unit. The control unit can cause the processor 901 to perform information processing on the received information according to various programs. The communication unit can also transmit information output from the control unit to other computers 90.

[0139] Furthermore, each of the above-mentioned configurations, functions, processing units, processing means, etc., may be implemented in hardware, either partially or entirely, by designing them as integrated circuits, for example. The present invention can also be implemented by software program code that realizes the functions of the embodiment. In this case, a storage medium on which the program code is recorded is provided to a computer, and the processor of that computer reads the program code stored in the storage medium. In this case, the program code read from the storage medium itself realizes the functions of the embodiment described above, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media used to supply such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, and the like.

[0140] Furthermore, the program code that implements the functions described in this embodiment can be implemented in a wide range of programming or scripting languages, such as assembler, C / C++, Perl, Shell, PHP, and Java (registered trademark).

[0141] Furthermore, the program code for the software that implements the functions of the embodiment may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the computer's processor may read and execute the program code stored in the storage means or storage medium.

[0142] The functions realized by the components described herein may be implemented in a circuit or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to realize the functions described herein. A processor is considered to be a circuit or processing circuitry, including transistors and other circuits. A processor may be a programmed processor that executes a program stored in memory. In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein. If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0143] While several embodiments of this disclosure have been described above, these embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications are permitted without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0144] (Note) The details described in each of the above embodiments are noted below.

[0145] (Note 1) A program for operating a computer comprising a processor and memory, wherein the program is configured to operate the processor, The steps include receiving first audio data relating to the user's speech from the terminal device, From the time the generating AI is instructed to infer a response to the first audio data, until the second audio data converted from the text data relating to the response generated by the inference is transmitted to the terminal device, the step of determining whether or not the first audio data relating to a statement different from the preceding statement has been received, If, between the start of the aforementioned inference and the transmission of the second audio data, first audio data relating to a statement different from the preceding statement is received, the operation relating to the response is terminated. The steps include invalidating the text data related to the second audio data from which the aforementioned response was canceled, A program that executes the command. (Note 2) In the aforementioned determination step, it is determined whether a statement different from the preceding statement is below a predetermined number of characters. The program described in (Appendix 1) does not terminate the operation related to the response in the termination step if it is determined that the statement, which is different from the preceding statement, is less than or equal to a predetermined number of characters. (Note 3) In the second status to which the generating AI responds, if it receives the first audio data relating to a statement different from the preceding statement from the terminal device, it switches the second status to the first status spoken by the user, and presents the switch to the terminal device. A program described in any of (Appendix 1) to (Appendix 2) that causes the processor to execute the above. (Note 4) If, between the commencement of the inference and the transmission of the second audio data, the first audio data relating to a statement different from the preceding statement is not received, and the terminal device receives a notification that it has controlled the response so as not to output the second audio data to the user, the text data relating to the second audio data for which the response was controlled is invalidated. A program described in any of (Appendix 1) to (Appendix 3) that causes the processor to execute the above. (Note 5) The steps include receiving information from the terminal device regarding the output segment of the second audio data that corresponds to the outputted audio, The steps include storing text data relating to the output segment from the second audio data based on the information regarding the output segment, The steps include invalidating the range of text data related to the second audio data other than the output segment, A program described in any of (Appendix 1) to (Appendix 4) that causes the processor to execute the above. (Note 6) A method to be performed on a computer comprising a processor and memory, wherein the processor performs all steps performed in any of the inventions described in (Appendix 1) to (Appendix 5). (Note 7) An information processing apparatus comprising a control unit and a storage unit, wherein the control unit performs all steps performed in any of the inventions described in (Appendix 1) to (Appendix 5). (Note 8) A system comprising means for performing all steps performed in any of the inventions described in (Appendix 1) to (Appendix 5). (Note 9) A system comprising a terminal device and a server that is communicatively connected to the terminal device, The aforementioned server, Means for receiving first audio data relating to the user's first statement from the terminal device, Means for causing a generating AI to infer a response to a preceding statement made by the user prior to the first statement, and for transmitting a second audio data relating to the response, converted from the text data relating to the response generated by the inference, to the terminal device. Means for receiving from the terminal device that the terminal device has controlled its response so as not to output the second audio data to the user. Means for invalidating the text data relating to the second voice data whose response is controlled Equipped with, The aforementioned terminal device is A means for receiving the user's comments, means for receiving the second audio data from the server, A means for determining whether a statement different from the preceding statement has been received between the time of receiving the second audio data and the time of outputting the second audio data, If a statement different from the preceding statement is received between the time of receiving the second audio data and the time of outputting the second audio data, means for controlling the output of the second audio data to the user. Means for sending a message to the server indicating that the response has been controlled so as not to output the second audio data to the user. Equipped with, system. (Note 10) The means for determining whether the terminal device has a different statement from the preceding statement and whether it has a predetermined number of characters or less, and if the statement has a different statement from the preceding statement and is determined to have a predetermined number of characters or less, the system does not interfere with the output of the second voice data to the user, as described in (Appendix 9). (Note 11) The aforementioned terminal device is Means for identifying the output segment, which is the range of the second audio data that corresponds to the outputted audio, Means for transmitting information regarding the identified output segment to the server, Equipped with, The aforementioned server, Means for receiving information regarding the identified output segment from the terminal device, A means for storing text data relating to the output segment from the second audio data, based on the information of the identified output segment, Means for invalidating the range of text data related to the second audio data other than the output segment, Equipped with, The system described in any of the following (Appendix 9) to (Appendix 10). [Explanation of Symbols]

[0146] 1... System 10…Terminal device 12…Communication IF 13…Input device 14…Output device 15…Memory 16…Storage 19… Processor 20... Server 22...Communication IF 23…Input / Output Interface 25…Memory 2 hours… storage 29… Processor 80…Network

Claims

1. A program for operating a computer comprising a processor and memory, wherein the program is provided to the processor, The steps include receiving first audio data relating to the user's speech from the terminal device, From the time the generating AI is instructed to begin inferring a response to the first audio data, until the second audio data converted from the text data relating to the response generated by the inference is transmitted to the terminal device, the step of determining whether or not the first audio data relating to a statement different from the preceding statement has been received, If, between the start of the inference and the transmission of the second audio data, the first audio data relating to a statement different from the preceding statement is received, the operation relating to the response is terminated. The steps include: excluding the text data relating to the second audio data from reference in the conversation history, A program that executes the command.

2. In the aforementioned determination step, it is determined whether a statement different from the preceding statement is below a predetermined number of characters. The program according to claim 1, wherein if it is determined that a statement different from the preceding statement is less than or equal to a predetermined number of characters, the program does not cancel the operation related to the response in the step of canceling.

3. In the second status to which the generating AI responds, if it receives the first audio data relating to a statement different from the preceding statement from the terminal device, it switches the second status to the first status spoken by the user and presents the switch to the terminal device. The program according to claim 1, which causes the processor to execute the following.

4. If, between the initiation of the aforementioned inference and the transmission of the second audio data, the terminal device does not receive the first audio data relating to a statement different from the preceding statement, and the terminal device has controlled the response so as not to output the second audio data to the user, the text data relating to the second audio data for which the response was controlled is excluded from the reference targets in the conversation history. The program according to claim 1, which causes the processor to execute the following.

5. The steps include receiving information from the terminal device regarding the output segment of the second audio data, which corresponds to the outputted audio; The steps include storing text data relating to the output segment from the second audio data based on the information regarding the output segment, The steps include: excluding the portion of the text data related to the second audio data other than the output segment from the reference target in the conversation history; The program according to claim 4, which causes the processor to execute the following.

6. A method to be performed on a computer comprising a processor and memory, wherein the processor performs all steps performed in the invention according to any one of claims 1 to 5.

7. An information processing apparatus comprising a control unit and a storage unit, wherein the control unit performs all steps performed in the invention according to any one of claims 1 to 5.

8. A system comprising means for performing all steps performed in the invention according to any one of claims 1 to 5.

9. A system comprising a terminal device and a server that is communicatively connected to the terminal device, The aforementioned server, Means for receiving first audio data relating to the user's first statement from the terminal device, Means for causing a generating AI to infer a response to a preceding statement made by the user prior to the first statement, and for transmitting a second voice data relating to the response, converted from the text data relating to the response generated by the inference, to the terminal device. Means for receiving from the terminal device that the terminal device has controlled its response so as not to output the second audio data to the user. The system includes means for excluding text data relating to the second voice data whose response has been controlled from reference in the conversation history, The aforementioned terminal device is A means for receiving the user's comments, means for receiving the second audio data from the server, A means for determining whether a statement different from the preceding statement has been received between the time of receiving the second audio data and the time of outputting the second audio data, If a statement different from the preceding statement is received between the time of receiving the second audio data and the time of outputting the second audio data, means for controlling the output of the second audio data to the user. The system includes means for sending a message to the server indicating that the response has been controlled so as not to output the second audio data to the user, system.

10. The system according to claim 9, wherein the means for determining whether the statement different from the preceding statement is less than or equal to a predetermined number of characters, and if the statement different from the preceding statement is less than or equal to the predetermined number of characters, the output of the second voice data to the user is not obstructed.

11. The aforementioned terminal device is Means for identifying the output segment, which is the range of the second audio data that corresponds to the outputted audio, The system includes means for transmitting information regarding the identified output segment to the server, The aforementioned server, Means for receiving information regarding the identified output segment from the terminal device, A means for storing text data relating to the output segment from the second audio data, based on the information of the identified output segment, The system includes means for excluding from the range of text data related to the second audio data, other than the output segment, from the reference target in the conversation history, The system according to claim 9.

Images