Programs, information processing devices, methods, and systems

Abstract

In interactions between computers and users, we aim to achieve a natural dialogue rhythm that closely resembles human-to-human conversation. [Solution] A program for operating a computer equipped with a processor, which causes the processor to perform the following steps: acquire text data indicating the content of a user's utterance; input the acquired text data into a trained model and cause the trained model to output confidence level information as a multi-level numerical value indicating the degree of likelihood that the user will make an utterance following the utterance corresponding to the text data, based on the context of the dialogue between the computer and the user at the time the text data was acquired; and determine a waiting time until the computer starts responding based on the outputted confidence level information, and start the computer's response after the determined waiting time has elapsed.

Description

Technical Field

[0001] The present disclosure relates to a program, an information processing apparatus, a method, and a system.

Background Art

[0002] As control for turn-taking in a voice dialogue system, techniques for uniformly determining a silent time based on voice section detection and estimating subsequent tokens using a language model are known. For example, Patent Document 1 discloses a technique in which a system determines the preparation and start timing of the next utterance using a probability distribution based on a language model and an estimated value of syntactic completion.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the uniform silent time determination, even when the user is silent while thinking, the AI model responds, so the user's speech is interrupted and smooth dialogue is hindered. Also, in a binary determination such as the disclosed technique of Patent Document 1 for whether to end or continue, excessive waiting and unnatural immediate responses occur due to a slight determination error, and smooth dialogue is similarly hindered.

[0005] An object of the present disclosure is to realize a natural dialogue rhythm close to that between humans in the dialogue between a computer and a user.

Means for Solving the Problems

[0006] To solve the aforementioned problems, a program according to one aspect of the present disclosure is a program for operating a computer equipped with a processor, which causes the processor to perform the following steps: acquire text data indicating the content of a user's utterance; input the acquired text data into a trained model and cause the trained model to output confidence level information as a multi-level numerical value indicating the degree of likelihood that the user will make an utterance following the utterance corresponding to the text data, based on the context of the dialogue between the computer and the user at the time the text data was acquired; and determine a waiting time until the computer starts responding based on the outputted confidence level information, and cause the computer to start responding after the determined waiting time has elapsed. [Effects of the Invention]

[0007] According to this disclosure, it is possible to achieve a natural dialogue rhythm in the interaction between a computer and a user that is similar to that of a conversation between humans. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the overall system configuration. [Figure 2] This is a block diagram showing an example of a user terminal's hardware configuration. [Figure 3] This is a block diagram showing the functional components implemented by the control unit of the user terminal. [Figure 4] Block diagram showing an example of a server hardware configuration. [Figure 5] This is a block diagram showing the functional parts implemented by the server's control unit. [Figure 6] This diagram shows the data structure of the standby control table categorized by confidence level. [Figure 7] This diagram shows the data structure of the dialogue context retention table. [Figure 8] This diagram shows the data structure of the user-specific silence characteristics table. [Figure 9]This flowchart shows an example of server operation. [Figure 10] This is a schematic diagram showing an example of a dialogue screen. [Figure 11] This is a schematic diagram showing an example of the status display screen. [Modes for carrying out the invention]

[0009] The embodiments of this disclosure will be described below with reference to the drawings. In all the drawings illustrating the embodiments, common components are denoted by the same reference numerals, and repeated explanations are omitted. The following embodiments are not intended to unduly limit the content of this disclosure as described in the claims. Not all components shown in the embodiments are necessarily essential components of this disclosure. Also, each drawing is a schematic diagram and is not necessarily a strict illustration.

[0010] Furthermore, in the following description, "processor" refers to one or more processors. A processor may be expressed, for example, as processing circuitry. At least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may be other types of processors such as a GPU (Graphics Processing Unit). At least one processor may be single-core or multi-core. Also, at least one processor may be a general-purpose processor or a purpose-specific processor.

[0011] Furthermore, at least one processor may be a broad-sense processor, such as a hardware circuit that performs some or all of the processing (e.g., an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit)).

[0012] In the following description, the expression "xxx table" may be used to describe information from which an output can be obtained for an input. This information may be data of any structure or a learning model such as a neural network that generates an output for an input. Therefore, "xxx table" can be referred to as "xxx information".

[0013] In the following description, the configuration of each table is an example. One table may be divided into two or more tables, or all or part of two or more tables may be one table.

[0014] The program may be pre-installed in the information processing apparatus shown below. For example, it may be on a recording medium (e.g., non-temporary) readable by the information processing apparatus, and this program may be installed in the information processing apparatus. Also, the program may be transmitted from a program distribution server to the information processing apparatus and installed. In the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0015] In the following description, identification information for various objects is used. The identification information may be any information indicating a predetermined object, and the specific data is not limited to the embodiments. The identification information may be an identification number or an identifier including letters or symbols.

[0016] 〔Overview〕 The system according to this embodiment is a voice dialogue system that conducts interviews, counseling, etc. using an AI avatar or the like. In this system, the end of the user's speech is highly judged from the context, and the waiting time until starting the response is dynamically controlled. Specifically, the system according to this embodiment converts the user's speech content into text in real time, and inputs the text data of the speech content and the context of the dialogue in the dialogue session at the time of obtaining the text data into a learned model. Then, confidence information indicating the degree of the possibility that the user will continue to speak is output as, for example, a numerical value in a plurality of levels from 0 to 100. The system according to this embodiment carefully determines whether to respond immediately, wait for a few seconds, or stop the response process and continue waiting based on this confidence information. Thereby, it is possible to prevent an unpleasant experience that the system interrupts the speech while the user is simply thinking and speaking, and to realize a smooth and natural communication experience.

[0017] 〔Embodiment〕 <Overall Configuration of the System> FIG. 1 is a block diagram showing the overall configuration of the system 1 according to this embodiment. As shown in FIG. 1, the system 1 includes a user terminal 10 operated by a user, a server 20 that executes various information processes, a generation AI system 30 equipped with a learned model, and an administrator terminal 40 operated by an administrator of the system 1. These devices are communicably connected to each other via a network 80.

[0018] The network 80 is a wired or wireless communication network. The network 80 can include, for example, the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), a VPN (Virtual Private Network), and the like.

[0019] The user terminal 10 is an information processing device that functions as an interface in the interaction between the computer and the user (hereinafter sometimes simply referred to as "interaction"). The user terminal 10 has the function of acquiring the user's utterances and sending them to the server 20, and presenting the response information received from the server 20 to the user.

[0020] Herein, the term "computer" as used herein is not limited to a single information processing device equipped with a processor. The term "computer" as used herein is a concept that includes the entire system consisting of multiple information processing devices distributed across a network. In this embodiment, the entire system 1, in which the user terminal 10, server 20, and generation AI system 30 cooperate via the network 80 to perform a series of processes, corresponds to an example of a "computer" as used herein. In another embodiment, if the server 20 independently performs all of the speech recognition function, the trained model execution function, and the standby control function, the server 20 itself corresponds to a "computer" as used herein.

[0021] Server 20 is an information processing device that plays a central role in information processing in this embodiment. Server 20 has the function of cooperating with the generation AI system 30 based on the user's speech content received from the user terminal 10 and controlling the appropriate response timing according to the context of the dialogue.

[0022] The Generative AI System 30 is an information processing device that uses a trained model to analyze the context of a dialogue, outputs information indicating the likelihood of the user continuing their utterance, and generates a response to the user. In this embodiment, the Generative AI System 30 is shown as a separate device from the Server 20, but the Server 20 may also be configured to include the functions of the Generative AI System 30. In this case, the Server 20 executes the trained model and outputs confidence level information and generates the response text.

[0023] The administrator terminal 40 is an information processing device used by the system administrator of System 1. The administrator terminal 40 has the function of monitoring the operating status of System 1 in real time, particularly confidence level information and the status of standby time determination. Note that the administrator terminal 40 is not an essential component of System 1.

[0024] <User terminal hardware configuration> Figure 2 is a block diagram showing an example of the hardware configuration of a user terminal 10. The user terminal 10 is an information processing device used by the user for interaction. Specific examples of the user terminal 10 include smartphones, tablet devices, personal computers (PCs), smart speakers, or wearable devices.

[0025] As shown in Figure 2, the user terminal 10 comprises a control unit 101, a storage unit 102, a communication unit 103, an input unit 104, and an output unit 105. These units are interconnected via signal lines such as buses.

[0026] The control unit 101 is composed of a processor such as a CPU or GPU, and a main memory such as RAM (Random Access Memory). The control unit 101 comprehensively controls the operation of the entire user terminal 10 by reading various programs stored in the storage unit 102 into the main memory and executing them.

[0027] The storage unit 102 is an auxiliary storage device such as ROM (Read Only Memory), flash memory, or HDD (Hard Disk Drive). The storage unit 102 stores the operating system (OS) program, application programs for realizing the interactive functions according to this embodiment, and various data.

[0028] The communication unit 103 consists of a network interface card (NIC), etc., and transmits and receives data with other devices such as the server 20 via the network 80. An example of a communication protocol used by the communication unit 103 is TCP / IP.

[0029] The input unit 104 is a device for receiving various inputs from the user. The input unit 104 includes a microphone 141 for acquiring the user's spoken voice. The input unit 104 may also include a pointing device such as a touch panel, keyboard, or mouse. Furthermore, the input unit 104 may be configured to include a camera (not shown) for acquiring the user's visual information. The camera is used to capture images of the user's facial expressions, gaze, lip movements, etc.

[0030] The output unit 105 is a device for outputting various information to the user. The output unit 105 includes a display 151 for displaying a graphical user interface (GUI) such as an interactive screen, and a speaker 152 for outputting response audio from the server 20.

[0031] <User terminal functional configuration> Figure 3 is a block diagram showing an example of a functional configuration realized by the control unit 101 of the user terminal 10 executing a program. Functionally, the control unit 101 operates as an operation reception unit 111, a transmission / reception unit 112, and a presentation control unit 113.

[0032] The operation reception unit 111 has the function of receiving operation input from the user via the input unit 104. The operation reception unit 111 receives, for example, commands to start and end dialogues, and commands to change various settings.

[0033] The transmitting / receiving unit 112 has the function of sending and receiving various data to and from the server 20 via the communication unit 103. Specifically, the transmitting / receiving unit 112 transmits the user's voice data acquired by the microphone 141 to the server 20 in streaming format or file format. The transmitting / receiving unit 112 also receives response text data, response voice data, control signals, etc., transmitted from the server 20.

[0034] The presentation control unit 113 has the function of controlling the output unit 105 and presenting information to the user. Specifically, the presentation control unit 113 displays the response text data and dialogue history received from the server 20 on the dialogue screen on the display 151. The presentation control unit 113 also causes the response audio data received from the server 20 to be output from the speaker 152.

[0035] <Server Hardware Configuration> Figure 4 is a block diagram showing an example of the hardware configuration of server 20. Server 20 can be implemented in any form, such as one or more physical servers, or virtual servers provided in a cloud computing environment.

[0036] As shown in Figure 4, the server 20 comprises a control unit 201, a storage unit 202, a communication unit 203, and an input / output IF 204. These units are interconnected via signal lines such as buses.

[0037] The control unit 201 is composed of a processor such as a CPU and a main memory such as RAM. The control unit 201 comprehensively controls the operation of the entire server 20 by reading the information processing program according to this embodiment, which is stored in the storage unit 202, into the main memory and executing it, thereby realizing the various functional units described later.

[0038] The storage unit 202 is an auxiliary storage device such as an HDD or SSD (Solid State Drive). The storage unit 202 stores not only the program to be executed by the control unit 201, but also various data used in the processing process in a table format. In this embodiment, the storage unit 202 stores a confidence level-based standby control table 221, an interaction context retention table 222, and a user-specific silence characteristic table 223.

[0039] The communication unit 203 is a network interface that transmits and receives data between the user terminal 10, the generation AI system 30, the administrator terminal 40, and other devices via the network 80.

[0040] Input / Output IF204 is a connection interface for input / output devices (not shown) such as keyboards, mice, and displays. Input / Output IF204 is used for maintenance and management of server 20.

[0041] <Server Functional Configuration> Figure 5 is a block diagram showing an example of a functional configuration realized by the control unit 201 of the server 20 executing a program. Functionally, the control unit 201 operates as a speech recognition unit 211, a context management unit 212, a confidence level information acquisition unit 213, a standby control unit 214, and a response generation unit 215.

[0042] The speech recognition unit 211 has the function of analyzing the speech data related to the user's utterance received from the user terminal 10 and converting it into text data. The speech recognition unit 211 may perform processing using an external speech recognition API service, or it may perform processing using a speech recognition engine implemented in the server 20. The speech recognition unit 211 outputs the utterance up to that point as text data each time the user's utterance is interrupted. Here, "the user's utterance is interrupted" refers to, for example, the detection of a silent period of a predetermined time or longer.

[0043] The context management unit 212 has the function of managing the context of the dialogue. Specifically, the context management unit 212 records the text data generated by the speech recognition unit 211 and the response text generated by the generation AI system 30 (specifically, the trained model) in chronological order in the dialogue context retention table 222, along with information such as the speaker and timestamp. In addition, when the context management unit 212 requests the generation AI system 30 to output confidence level information, it has the function of reading the most recent dialogue history from the dialogue context retention table 222 and formatting it as input data for the generation AI system 30. An example of the most recent dialogue history is the question from the generation AI system 30 immediately before and the user's answers to that question up to the present.

[0044] Furthermore, the context management unit 212 has the function of comparing multiple requirements included in a question from the computer (the entire system 1 or server 20) with answer elements extracted from the user's response and identifying unanswered elements. The context management unit 212 provides information on these unanswered elements to the generating AI system 30 as part of the dialogue context. This provision improves the accuracy of the calculation of confidence level information. It should be noted that it is not essential for the context management unit 212 to have the function of identifying unanswered elements; for example, the generating AI system 30 may have this function (see the third modification example below).

[0045] The confidence level information acquisition unit 213 has the function of acquiring confidence level information from the generation AI system 30. The confidence level information is a multi-level numerical value that indicates the degree to which the user is likely to make further utterances following the current utterance. In other words, the confidence level information is an indicator of whether the user's utterance is semantically complete or whether there is a possibility of further supplementary explanation. For example, the confidence level information is output as an integer value from 0 to 100. Here, the higher the number, the higher the probability that the user's utterance is complete, and the lower the number, the higher the probability that the user will continue uttering. The confidence level information acquisition unit 213 transmits the dialogue context prepared by the context management unit 212 to the generation AI system 30 and receives the confidence level information as a response from the generation AI system 30.

[0046] Preferably, the confidence level information acquisition unit 213 has the function of acquiring confidence level information in advance from the generation AI system 30 before it completes the generation of the response text to be output as a computer response. This function can be achieved, for example, by using a prompt in the request to the generation AI system 30 that causes the confidence level information to be output in a specific field in JSON format before the response text. This allows the server 20 to quickly start controlling the waiting time without waiting for the completion of the response text generation.

[0047] Furthermore, the confidence level information acquisition unit 213 may have a function to analyze the user's visual information acquired from the user terminal 10's camera, etc., and correct the confidence level information output from the generating AI system 30 based on the analysis results. The analysis of visual information is performed, for example, using an image recognition model that detects lip activity. If the confidence level information acquisition unit 213 determines that the user's mouth is moving or about to move, it estimates that the user still intends to continue speaking and corrects the confidence level information obtained from the generating AI system 30 by lowering it. This correction enables a more accurate determination of the likelihood of continuing to speak based on multimodal information.

[0048] The standby control unit 214 has the function of determining the waiting time until the computer starts responding based on the confidence level information acquired by the confidence level information acquisition unit 213, and managing that waiting time. Specifically, the standby control unit 214 compares the received numerical value of the confidence level information with the rules defined in the confidence level-specific standby control table 221, and determines the length of the waiting time to be applied.

[0049] There are several possible algorithms for determining the waiting time. The first variation is a step function algorithm. In this algorithm, the waiting control unit 214 selects a discrete time length based on multiple thresholds defined in the confidence level-based waiting control table 221. The second variation is a linear mapping algorithm. In this algorithm, the waiting control unit 214 calculates the time linearly using a mathematical formula. An example of such a formula is "Waiting time = Maximum waiting time × (1 - Confidence level / 100)". The third variation is a nonlinear (exponential decay) algorithm. In this algorithm, the waiting control unit 214 exponentially increases the waiting time in the low confidence level region.

[0050] Furthermore, the standby control unit 214 has the function of analyzing the temporal characteristics of the user's speech patterns and adjusting the standby time determination logic for each user. Specifically, the standby control unit 214 calculates the average silence time and average speech rate for each user from past dialogue history and stores them in the user-specific silence characteristics table 223. Then, when determining the standby time, the standby control unit 214 reads the corresponding user's standby coefficient and maximum standby time from the user-specific silence characteristics table 223 and applies them as parameters to the standby time determination algorithm described above. This enables more natural control of pauses that are tailored to the conversational pace of each individual user.

[0051] The standby control unit 214 starts a timer based on the determined standby time and monitors the passage of time. Furthermore, the standby control unit 214 has a function to instruct the response generation unit 215 to interrupt the response generation process if text data corresponding to a new utterance is acquired from the user terminal 10 during standby time. This function prevents the system from interrupting the response if the user reconsiders and continues speaking.

[0052] The response generation unit 215 has the function of managing responses to the user and sending them to the user terminal 10 at the appropriate time. Normally, when the response generation unit 215 detects that the timer of the standby control unit 214 has expired, it acquires the text data of the response generated by the generation AI system 30, converts it into voice data, for example, and sends it to the user terminal 10.

[0053] Furthermore, the response generation unit 215 has a function to perform follow-up based on confidence level information. Specifically, if the user does not make any further statements after the waiting time has elapsed, and the confidence level information used to determine the waiting time is below a predetermined threshold, the response generation unit 215 generates or selects a follow-up response to prompt the user to provide further answers, either in place of or along with the output of the main answer. The follow-up response may be a standard phrase such as, "Could you tell me a little more about the reason?", or it may be dynamically generated by the generation AI system 30 according to the context of the dialogue.

[0054] <Configuration of the Generative AI System> The generative AI system 30 comprises one or more trained models and provides access to said one or more trained models. This disclosure describes, as an example, the case in which the generative AI system 30 provides access to a Large Language Model (LLM), which is one of the trained models. The Large Language Model is an example of a generative AI model designed to perform multiple tasks of natural language processing, and is a model trained using a large number of parameters and high-level computational resources. The Large Language Model generates output based on text data or image data of a prompt when a prompt is input.

[0055] Examples of large-scale language models include the GPT® series (Generative Pre-Trained Transformer) developed by OPEN AI, StableLM developed by Stability AI, Llama2 developed by Meta, and Palm2® or LamDA2® developed by Google. Note that the trained model is not limited to large-scale language models; it may also include, for example, image generation models, speech synthesis models, and multimodal generative AI models capable of recognizing both image and audio data.

[0056] In this specification, "prompt" refers to the entirety of input data that is input to a large-scale language model, including instructional information that instructs the large-scale language model to process information. A prompt may include, for example, text data, strings, still image data, video data, or audio data. The user inputs a prompt through an input operation via the user terminal 10, or by uploading a document file, still image data, video data, or audio data.

[0057] There are multiple implementation variations of the trained model used in this embodiment. The first variation is the large-scale language model described above. When using this large-scale language model, the server 20 generates a prompt that includes natural language instructions, such as "output an integer from 0 to 100 indicating the confidence level that the user's utterance is complete," in addition to the context information of the dialogue, and sends it to the large-scale language model. The large-scale language model interprets the instructions and context contained in the prompt and outputs confidence level information based on advanced inference.

[0058] The second variation is a small-scale model fine-tuned specifically for this task. This model is based on models such as RNN (Recurrent Neural Network) or BERT (Bidirectional Encoder Representations from Transformers) based on the Transformer architecture. This task-specific model is trained to take dialogue text as input and directly output a numerical value indicating the likelihood of utterance completion. Because task-specific models are smaller in size compared to large-scale language models, they can determine confidence information with lower latency.

[0059] The prompts that server 20 sends to the generating AI system 30 typically include system prompts and user prompts. The system prompt defines the role, behavior, and output format of the generating AI model. For example, it might say, "You are an excellent interviewer. Proceed with the conversation with the user naturally. Your responses must be output as objects in JSON format with 'confidence' and 'response' keys." The user prompt includes the actual conversation history and the current user's utterance text.

[0060] When generating a prompt, the server 20 instructs the generating AI system 30 to structure and output confidence information and response text. For example, the server 20 specifies the output format as JSON and includes instructions in the prompt such as, "Store the confidence level of the user's utterance as an integer value from 0 to 100 in the 'confidence' key. Store the response text to the user in the 'response' key." This allows the server 20 to mechanically parse the response from the generating AI system 30 and easily separate and obtain the confidence information and response text.

[0061] The trained model of the generative AI system 30 receives the dialogue context as input from the server 20. Based on the input context, the trained model outputs at least two pieces of information. The first piece of information is confidence level information, indicating the degree to which the user is likely to make an utterance following the utterance corresponding to the text data. The second piece of information is the text data of the actual response used as a response to the user, i.e., the response text.

[0062] When analyzing confidence information, the trained model comprehensively assesses factors such as the presence or absence of fillers like "um" in the user's utterance, the clarity of the utterance's ending, grammatical completeness, and the comprehensiveness of the answer to the AI's question. For example, if the user's utterance is incomplete or only answers some of the multiple questions asked, the trained model will output a low confidence score, specifically a number indicating a high probability that the utterance will continue.

[0063] The generation AI system 30 has a function that, in response to a request from the server 20, returns confidence level information calculated in the initial stage of inference to the server 20 without waiting for the completion of the response text generation. This function can improve the responsiveness of the entire system 1.

[0064] <Administrator terminal configuration> The administrator terminal 40 is an information processing device having the same hardware configuration as the user terminal 10. Specific examples of the administrator terminal 40 include a PC or a tablet device. The control unit of the administrator terminal 40 executes a monitoring application stored in its memory unit, thereby enabling it to receive various operational data from the server 20 in real time and display it on the display.

[0065] <Data structure> Referring to Figures 6 to 8, the data structures of the various databases stored by server 20 will be explained. The databases shown in Figures 6 to 8 refer to relational databases, which are used to manage data sets called tabular tables, which are structurally defined by rows and columns, in relation to one another.

[0066] In a database, tables are called tables, the columns of a table are called columns, and the rows of a table are called records. In a relational database, relationships can be established and linked between tables. Typically, each table that makes up a database has a primary key column to uniquely identify a record, but setting a primary key for a column is not mandatory. The control unit 201 can cause the processor to add, delete, or update records in a specific table stored in the storage unit 202 according to various programs.

[0067] Please note that Figures 6 through 8 are merely examples and do not exclude any data not shown. Furthermore, even data recorded in the same database may be stored in separate memory areas within the memory unit 202.

[0068] Figure 6 shows the data structure of the confidence level-based standby control table 221. The confidence level-based standby control table 221 is a table that the standby control unit 214 refers to when determining the standby time and manages the correspondence between confidence level information and standby time, etc. The confidence level-based standby control table 221 includes, for example, columns for rule ID, confidence level lower limit, confidence level upper limit, standby time, and execution action.

[0069] The "Rule ID" column stores an identifier to uniquely identify each rule of the wait control. The "Confidence Lower Limit" and "Confidence Upper Limit" columns store numerical values ​​that define the range of the confidence score output from the trained model. An example of a confidence score is a number from 0 to 100. The "Wait Time" column stores the length of time the system should wait before starting to respond within the range of the corresponding confidence score. An example of a time length is in milliseconds or seconds. The "Action to Execute" column stores the specific system action to be executed within the range of the corresponding confidence score. Examples of system actions include immediate response, waiting, and response pause. For example, a record with a confidence lower limit of "0" and a confidence upper limit of "0" will store "infinite" as the wait time and "response pause (wait)" as the action to execute. Similarly, a record with a confidence lower limit of "91" and a confidence upper limit of "100" will store "0 milliseconds" as the wait time and "immediate response" as the action to execute. This allows the waiting time to be appropriately selected from multiple time levels, corresponding to the confidence level information output as a multi-level numerical value.

[0070] Figure 7 shows the data structure of the dialogue context retention table 222. The dialogue context retention table 222 is updated by the context management unit 212 and is a table that manages the dialogue history and contextual meaning for each dialogue session. The dialogue context retention table 222 includes, for example, columns for log ID, session ID, timestamp, speaker, utterance content, request, and extracted entity.

[0071] The "Log ID" column stores an identifier to uniquely identify each utterance in the dialogue. The "Session ID" column stores an identifier to uniquely identify the dialogue session. The "Timestamp" column stores the date and time the utterance was made. The "Speaker" column stores information indicating whether the speaker is the user or the Generative AI System 30. The "Utterance" column stores the speech-recognized text data of the user or the response text generated by the system. The "Requirements" column stores the requirements contained in the questions the system asked the user. Examples of requirements include "adaptability" and "the reason." The "Extracted Entities" column stores structured data extracted from the user's responses. Examples of structured data include keywords, intentions, and emotions. This information is referenced as the context of the dialogue when the Generative AI System 30 calculates its confidence level.

[0072] Figure 8 shows the data structure of the user-specific silence characteristics table 223. The user-specific silence characteristics table 223 is a table that manages the temporal characteristics related to the speech patterns of each user. The user-specific silence characteristics table 223 includes, for example, columns for user ID, average silence time, average speech rate, waiting coefficient, and maximum waiting time.

[0073] The "User ID" column stores an identifier to uniquely identify the user. The "Average Silence Time" column stores the average silent time between turns for the user, statistically calculated from past dialogue history. Silence time corresponds to the length of pauses. The "Average Speech Rate" column stores the average speech rate of the user, such as the number of characters spoken per unit time. The "Waiting Coefficient" column stores a correction coefficient in the waiting time determination algorithm, calculated based on these temporal characteristics. The "Maximum Waiting Time" column stores the maximum acceptable waiting time for the user. By using these parameters, it is possible to dynamically adjust waiting times to match the individual user's pace.

[0074] <Example of operation> Figure 9 is a flowchart illustrating an example of the operation of server 20. The processing in this example is mainly achieved by the control unit 201 of server 20 executing a program stored in the storage unit 202.

[0075] As a prerequisite, when a user speaks into the microphone 141 installed in the user terminal 10, the input unit 104 of the user terminal 10 acquires voice data corresponding to the user's speech. The transmitting / receiving unit 112 of the user terminal 10 transmits the acquired voice data to the server 20 via the network 80 through the communication unit 103.

[0076] First, in step S101, the speech recognition unit 211 of the server 20 analyzes the speech data received from the user terminal 10 in real time and converts it into text data representing the user's speech. This text data acquisition process is performed sequentially while the user is speaking. The speech recognition unit 211 can detect pauses in the user's speech, i.e., silent periods of a predetermined time or longer, using techniques such as voice activity detection. Note that the acquisition of text data is not limited to speech recognition. For example, the server 20 may receive text data that the user directly inputs using an input device such as the keyboard of the user terminal 10.

[0077] When the speech recognition unit 211 detects a pause in the user's speech, the confidence level information acquisition unit 213 of the server 20 prepares to send the text data acquired up to that point to the generation AI system 30. At this time, the context management unit 212 refers to the dialogue context retention table 222 of the storage unit 202 and reads the dialogue context of the dialogue session at the time the text data was acquired. The dialogue context includes at least the content of the question that the generation AI system 30 asked immediately before and the entire content of the answer that the user spoke up to the point immediately before calculating the confidence level information in response to that question.

[0078] Next, in step S102, the confidence level information acquisition unit 213 of the server 20 inputs the acquired text data and the context of the read dialogue to the generative AI system 30 equipped with a trained model. Based on the input text data and the context of the dialogue, the generative AI system 30 calculates confidence level information as a multi-level numerical value.

[0079] In this embodiment, the trained model installed in the generation AI system 30 is a large-scale language model. The confidence level information acquisition unit 213 causes the generation AI system 30 to output confidence level information by sending a prompt such as, "The previous question was 'X', and the current user's answer is 'Y'. Please output an integer value from 0 to 100 indicating the confidence level that this user will continue speaking. No other text is needed."

[0080] Specifically, if the text data representing the user's utterance includes fillers or hesitations, such as "Um, my strength is...", the generating AI system 30 determines that there is a very high probability that the utterance will continue. As a result, the generating AI system 30 outputs a very low number as confidence information, such as 0 or 10. Also, consider the case where the user's response to the question from the generating AI system 30, "Please tell me your strengths and weaknesses," is only the text data "My strength is that I am adaptable." In this case, the generating AI system 30 considers that the weakness, which is the context of the dialogue, has not been answered and determines that there is a high probability that the utterance will continue. As a result, the generating AI system 30 outputs a relatively low number, such as 30. On the other hand, consider the case where the user's response completely answers the question and concludes with a sentence-ending particle, such as "My strength is that I am adaptable, and my weakness is that I am a little anxious. That's all." In this case, the generating AI system 30 determines that there is a very high probability that the utterance has been completed, and outputs a very high number as confidence information, such as 95 or 100.

[0081] Next, in step S103, the standby control unit 214 of the server 20 determines the waiting time until the server 20 starts responding, based on the confidence level information acquired by the confidence level information acquisition unit 213 from the generation AI system 30. In this determination process, the standby control unit 214 refers to the confidence level-specific standby control table 221 stored in the storage unit 202. The confidence level-specific standby control table 221 stores in advance the range of numerical values ​​of the confidence level information and the waiting time corresponding to that range.

[0082] Next, in step S104, the standby control unit 214 starts a timer for the determined standby time and enters a standby state. During this standby time, the response generation unit 215 of the server 20 does not instruct, for example, the generation AI system 30 to generate a response, but waits for additional utterances from the user.

[0083] Finally, in step S105, when the timer of the standby control unit 214 expires and the waiting time has elapsed, the response generation unit 215 of the server 20 instructs the generation AI system 30 to generate the final response. The response generation unit 215 acquires the text data of the final response generated by the generation AI system 30 and converts the text data into audio data using a speech synthesis engine (not shown). Then, the communication unit 203 of the server 20 transmits the generated audio data of the response to the user terminal 10. In the user terminal 10, the transmitting / receiving unit 112 receives the audio data of the response and outputs it as audio from the speaker 152 of the output unit 105. This initiates the computer's response. The presentation control unit 113 of the user terminal 10 may also display the received text data of the response on the display 151.

[0084] It is not mandatory for the response generation unit 215 to instruct the generation of the main response after the waiting time has elapsed. For example, in step S102, the response generation unit 215 may instruct the generation AI system 30 to generate the main response each time it detects a pause in the voice. In this case, the confidence level information acquisition unit 213 may be synchronized with the instruction by the response generation unit 215 and instructed to generate confidence level information prior to the generation of the main response. However, in step S102, the instruction to generate the main response and the instruction to generate confidence level information may be given at different times, and the order of the instructions does not matter. Furthermore, it is not mandatory for the confidence level information to be generated prior to the generation of the main response; the generation of the main response and the generation of confidence level information may be performed in parallel, or the generation of the main response may be performed prior to the generation of confidence level information. After the waiting time has elapsed, it is sufficient that the communication unit 203 is controlled to transmit the generated voice data of the response to the user terminal 10.

[0085] <Summary> As described above, in this embodiment, the large-scale language model installed in the generation AI system 30 outputs confidence level information, which indicates the degree of likelihood of the user continuing to speak, as a multi-level numerical value. The standby control unit 214 then determines the waiting time until the computer starts responding based on the outputted confidence level information, and the response generation unit 215 starts the computer's response after the determined waiting time has elapsed. In this way, by capturing the likelihood of continuing to speak with a multi-level indicator and flexibly adjusting the time until response according to that degree, it is possible to avoid situations where the system starts responding and interrupts the user's speech even when the user is silently thinking, or conversely, situations where an unnaturally long silence occurs. As a result, it becomes possible to achieve a natural and smooth dialogue rhythm that is close to human-to-human conversation.

[0086] The aforementioned effects are unique and cannot be obtained from the technology disclosed in Patent Document 1 (Japanese Patent Publication No. 2025-003232). This is because the technology disclosed in Patent Document 1 employs a so-called "frame-synchronous threshold determination method," in which the speech timing estimater calculates the likelihood of whether the user's right to speak has ended for each predetermined analysis frame, and immediately starts system speech when it is determined that the likelihood exceeds a predetermined threshold (frame). This method is an approach that constantly monitors when a response should be started for each analysis frame.

[0087] In contrast, System 1 according to this embodiment uniquely determines the waiting time before initiating a response based on confidence level information, which is a multi-level numerical value output from a large-scale language model. System 1 according to this embodiment then starts a timer at the determined waiting time and initiates a response after the timer has expired. In other words, System 1 according to this embodiment employs a so-called "timer-type waiting control method," which fundamentally differs from the technology disclosed in Patent Document 1 in its control approach. To put it another way, this embodiment is clearly distinguished from the technology disclosed in Patent Document 1 in that it does not monitor and decide when to speak every frame, but rather calculates how long to wait from the confidence level information and sets a timer accordingly.

[0088] [Variation] <First variation: Multiple setting of waiting time> As the first modification, a specific example of the waiting time determination process in step S103 of Figure 9 will be described. In this modification, the waiting time is selected from a predetermined set of multiple time lengths, corresponding to the confidence level information output as a multi-stage numerical value. Here, "multiple time settings" refers to a set of time settings that allow the waiting time before a response begins to change in stages depending on the situation. For example, discrete time settings such as 0 milliseconds, 2000 milliseconds, 3000 milliseconds, and 5000 milliseconds are defined in advance.

[0089] The standby control unit 214 of server 20 determines the waiting time using the confidence level-based standby control table 221, which stores the rule ID, the range defined by the lower and upper limits of the confidence level, the waiting time, and the action to be executed in association with each other. Specifically, the standby control unit 214 determines which range in the confidence level-based standby control table 221 the numerical value of the confidence level information output from the generating AI system 30 falls into. Then, the standby control unit 214 determines the waiting time associated with the determined range as the waiting time until the response starts.

[0090] For example, as shown in rule ID "R005" in the table in Figure 6, if the confidence level information is in the range of 91 to 100, the standby control unit 214 determines the standby time to be 0 milliseconds. In this case, the response generation unit 215 of the server 20 immediately starts the response generation process without waiting. This setting corresponds to a situation where it is confident that the user's utterance has been completed.

[0091] For example, as shown in rule ID "R004" in the table in Figure 6, if the confidence level information is in the range of 81 to 90, the standby control unit 214 determines the standby time to be 2000 milliseconds. This setting corresponds to a situation where the user's utterance is almost complete, but there is a slight possibility of further clarification.

[0092] For example, as shown in rule ID "R003" in the table in Figure 6, if the confidence level information is in the range of 51 to 80, the standby control unit 214 determines the standby time to be 3000 milliseconds. Also, as shown in rule ID "R002", if the confidence level information is in the range of 1 to 50, the standby control unit 214 determines the standby time to be 5000 milliseconds. The set standby time is intended to provide sufficient thinking time, corresponding to situations where the user is likely to be thinking of the next word.

[0093] Furthermore, as shown in rule ID "R001" in the table in Figure 6, if the confidence level information is 0, the standby control unit 214 sets the standby time to infinity, i.e., stops response processing by the response generation unit 215. This setting corresponds to situations where the user is merely hesitating, saying something like "um," but it is determined that they have a clear intention to continue speaking. In this case, the server 20 holds the response until the user makes the next utterance.

[0094] In this way, by setting the waiting time in multiple stages according to the confidence level information, the response timing can be finely adjusted according to the likelihood of the user continuing to speak, thereby achieving a natural rhythm similar to human conversation. The method for determining the waiting time is not limited to such a step function setting. For example, a method may be used to continuously calculate the waiting time using a linear or nonlinear (such as exponential decay) calculation formula for the confidence level information.

[0095] <Second variation: Advance output of confidence level information> As a second modification, we will describe the processing when confidence level information is output in advance. The processing related to this modification is intended to quickly start controlling the waiting time even when it takes time for the generation AI system 30 to generate the response text.

[0096] When the confidence information acquisition unit 213 of the server 20 requests the generation AI system 30 to generate a response, it instructs the system to output only the confidence information first, before completing the generation process of the response text itself. This instruction can be achieved, for example, by using a prompt in the API request to the generation AI system 30 that enables streaming output and specifies that the confidence information should be included at the beginning of the output.

[0097] Upon receiving the aforementioned request, the generative AI system 30 first calculates confidence level information based on the input text data and the context of the dialogue. Then, the generative AI system 30 includes the calculated value in the first field of the data, for example, in JSON format, and begins streaming transmission to the server 20.

[0098] When the confidence level information acquisition unit 213 begins receiving streaming data from the generation AI system 30, it immediately extracts the confidence level information contained at the beginning of the data. The confidence level information acquisition unit 213 then passes the extracted confidence level information to the standby control unit 214. Based on the confidence level information received in advance, the standby control unit 214 immediately determines the standby time and starts the timer.

[0099] Meanwhile, the response generation unit 215 of the server 20 continues to receive streaming data from the generation AI system 30 in parallel with the standby control unit 214 starting standby processing, and sequentially receives the data for the subsequent response text body.

[0100] After the waiting period has elapsed, the response generation unit 215 starts the process of sending a response to the user terminal 10 using the response text received up to the point of the waiting period's expiration. In this way, by executing the time-consuming response text generation process and the waiting time management, which requires precise control down to the millisecond, asynchronously and in parallel, it becomes possible to minimize the delay between the end of the user's utterance and the system starting to respond, while ensuring an appropriate interval.

[0101] <Third variation: Decrease in confidence based on unanswered questions> As a third variation, we will describe a process for adjusting confidence level information based on the requirements contained in the context of the dialogue. This variation is particularly effective in task-oriented dialogues such as interviews or hearings.

[0102] In this modified example, the context of the dialogue may include multiple requirements contained in a question from the computer (the entire System 1, or Server 20). Here, "multiple requirements" refers to multiple items that the user is asked to answer, contained within a single question. For example, the question, "Tell me your strengths and weaknesses," contains two requirements: "strengths" and "weaknesses."

[0103] For example, in the example in Figure 7, the content of the utterance spoken by the generating AI system 30 (speaker: AI) in log ID "L001" is stored in the "utterance content" cell. The context of this utterance includes two requirements: "whether or not it is adaptable" and "the reason why".

[0104] Suppose the user answers this question with "Yes, I am confident in my adaptability" (log ID "L002"). When the speech recognition unit 211 of server 20 acquires this text data and the confidence level information acquisition unit 213 requests the generating AI system 30 to output confidence level information, the generating AI system 30 compares the multiple requirements included in the context of the dialogue with the user's response elements included in the user's response text. Here, "user's response elements" refers to the specific content extracted from the user's response that corresponds to the requirements.

[0105] In this comparison process, the generating AI system 30 detects that the user's response addresses the requirement of "adaptability," but fails to address the requirement of "the reason." This detection is achieved by the logical reasoning function within the generating AI system 30, or by entity extraction using an external natural language understanding module. In the example in Figure 7, structured data is generated as extracted entities: "Adaptability: 'Yes'" and "Reason: 'Not answered'." Note that the generating AI system 30 does not necessarily have to perform this comparison process; for example, the context management unit 212 may perform it.

[0106] The generating AI system 30 logically determines that if there are unanswered elements in the requirements, the user is likely to continue speaking about those unanswered elements. As a result, the generating AI system 30 intentionally outputs a lower confidence score compared to the case where there are no unanswered elements, i.e., when all requirements have been answered. For example, even if an utterance would normally be rated at 70 points, it will output a low score of 30 points because there are unanswered items.

[0107] The standby control unit 214 of the server 20 will set a longer waiting time based on the lower calculated confidence level information. This prevents the generating AI system 30 from interrupting the user while they are thinking about and answering multiple questions one by one, and allows the system to wait for the user to finish answering all the questions.

[0108] <Fourth variation: Follow-up based on confidence level> As a fourth variation, we will describe a process that performs follow-up to prompt the user to provide further answers. "Follow-up" refers to a supplementary response that prompts for additional information when the user's answer is deemed insufficient. The process related to this variation is intended to facilitate the conversation when the user is lost in thought or unsure of what to say.

[0109] When the waiting time determined by the server 20's standby control unit 214 has elapsed, the server 20's response generation unit 215 first checks whether any new text data corresponding to additional utterances from the user has been acquired during the waiting time.

[0110] If there are no further utterances, and the confidence level information previously acquired by the confidence level information acquisition unit 213 of the server 20 is below a predetermined threshold (for example, 50 points or less), the control unit 201 determines that the user intended to continue speaking, but the utterance was interrupted for some reason.

[0111] In this case, the response generation unit 215 performs a follow-up prompting the user to provide further answers, either in place of or at the same time as the start of the main response, which is a normal response. Here, "performing a follow-up" refers to the process by which the response generation unit 215 generates a special response text to prompt the user to provide further answers and sends it to the user terminal 10.

[0112] Specific examples of follow-up responses include "Could you tell me a little more about that?", "Is there anything else?", or responses that specifically point out unanswered items. For example, suppose there are unanswered items as mentioned above, and the user says, "Yes, I am confident in my adaptability," but then remains silent without giving a reason. In this case, the confidence score is calculated to be low at 30 points, and a waiting time (e.g., 5000 milliseconds) is set. If the user does not make any further statements after this waiting time has elapsed, the response generation unit 215 generates a follow-up response such as "Could you tell me a little more about the reason?" and sends it to the user terminal 10 (see log ID "L003" in Figure 7).

[0113] Such follow-up responses allow the user to recall what to say next and resume the conversation. The processing described in this modified version plays an important role in the effective elicitation of responses from applicants by the generating AI system 30 acting as an interviewer, particularly in scenarios such as online interviews. The threshold for performing the follow-up may be adjustable from an administrator terminal 40 or the like, depending on the purpose or situation of the conversation.

[0114] <Fifth variation: Learning user characteristics and personalization> As a fifth variation, we will describe a personalization process that optimizes response timing for each user.

[0115] The standby control unit 214 of the server 20 further performs a step of analyzing the temporal characteristics of the user's speech patterns based on the user's past dialogue history. Here, "temporal characteristics" refers to statistical values ​​regarding the duration of each audible and silent interval in the user's speech. This analysis process may be performed as a batch process after the end of a dialogue session, or it may be performed in real time during the dialogue. The dialogue history is stored, for example, in a dialogue context retention table 222 linked to the user ID.

[0116] Specific examples of temporal characteristics include the average silence time between speech turns (length of pauses), the average speech rate (e.g., number of characters per minute), and the tempo of speech. The standby control unit 214 calculates these statistical values ​​for each user.

[0117] The standby control unit 214 adjusts parameters that define the correspondence between confidence level information and standby time for each user, based on the analyzed temporal characteristics. Here, "parameters" in this modified example refer to coefficients or thresholds used in the calculation formula for calculating standby time. The adjusted parameters are stored, for example, in the user-specific silence characteristics table 223, associated with the user ID. The user-specific silence characteristics table 223 stores, for example, the average silence time for each user, the average speech rate, and the standby coefficient or maximum standby time calculated based on these.

[0118] For example, in the example in Figure 8, user ID "U002" is analyzed to have a long average silence time of 3.2 seconds and a relatively slow speaking pace. Therefore, a waiting coefficient of 1.4 and a maximum waiting time of 7 seconds are set as parameters. On the other hand, user ID "U003" is analyzed to have a short average silence time of 0.8 seconds and a fast speaking pace, so a waiting coefficient of 0.8 and a maximum waiting time of 4 seconds are set as parameters.

[0119] In determining the waiting time, the standby control unit 214 reads parameters corresponding to the user ID of the current user from the user-specific silence characteristics table 223. Then, the standby control unit 214 adjusts the waiting time by multiplying the base waiting time obtained from the confidence level-based standby control table 221 by the read waiting coefficient. The standby control unit 214 also imposes a limit so that the adjusted waiting time does not exceed the maximum waiting time.

[0120] By determining the waiting time using these adjusted parameters, it becomes possible to create a more personalized and natural conversational experience that matches the individual user's conversational rhythm, such as providing longer pauses for users who speak slowly and shorter pauses for users who speak quickly.

[0121] <Sixth variation: Confidence level adjustment based on visual information> As a sixth variation, we will describe a multimodal process that improves the accuracy of confidence level information by utilizing the user's visual information. Here, "visual information" refers to image or video data that captures the user's physical state or actions.

[0122] In this modified version, the user terminal 10 is equipped with a camera (not shown) in addition to the microphone 141. During the conversation, the input unit 104 of the user terminal 10 acquires the user's voice data and, at the same time, acquires video information including the user's face and upper body via the camera, i.e., visual information. The acquired visual information is streamed to the server 20 along with the voice data.

[0123] Furthermore, in this modified version, the control unit 201 of the server 20 also functions as an image analysis unit (not shown) that analyzes visual information. The image analysis unit extracts nonverbal features related to the user's speech intent from the received visual information. Specific examples of features to be analyzed include lip movements (lip activity), gaze direction, blinking frequency, gestures such as nodding or head shaking, and changes in facial expression such as eyebrow movements and the raising and lowering of the corners of the mouth.

[0124] The confidence level information acquisition unit 213 of the server 20 acquires confidence level information output in text format from the generating AI system 30, and then performs a process to correct the confidence level information based on the analysis results from the image analysis unit.

[0125] A concrete example of this correction process is weighted addition. For example, consider a case where the text data determines that the utterance is complete, and the generating AI system 30 outputs a high confidence score of 90 points. However, if the image analysis unit outputs an analysis result such as "the lips are still moving" or "the gaze is directed upward, indicating that the speaker is trying to remember something," the confidence score information acquisition unit 213 will, based on this analysis result, i.e., the visual evidence, subtract 30 points from the original confidence score, resulting in a corrected confidence score of 60 points.

[0126] Conversely, consider the case where the text data is cut off midway, and the generating AI system outputs a low confidence score of 30 points, but the image analysis unit outputs an analysis result such as "the user is nodding deeply" or "the user is looking at us with a satisfied expression." In this case, the confidence score information acquisition unit 213 determines that the user has given a signal that they have finished speaking, and adds, for example, 40 points to the original confidence score, making the corrected confidence score 70 points.

[0127] The standby control unit 214 of the server 20 determines the waiting time based on the final confidence level information corrected in this way. This multimodal approach makes it possible to capture the user's nonverbal intentions that cannot be fully grasped by text information alone, and to estimate the likelihood of the utterance continuing with greater accuracy.

[0128] <Variation 7: Interrupt control during standby> As the seventh modification, we will describe interrupt control when the user resumes speaking while waiting. This process is designed to ensure that the system does not interrupt the user's speech.

[0129] While the standby control unit 214 of server 20 starts the timer and enters a standby state, the speech recognition unit 211 of server 20 continues to asynchronously monitor the audio stream from the user terminal 10.

[0130] If the user changes their mind and resumes speaking, new voice data is sent from the user terminal 10, and new text data is acquired by the voice recognition unit 211, the standby control unit 214 detects this as an interrupt event. Upon detecting this interrupt event, the standby control unit 214 interrupts or resets the currently running standby timer. This allows the system to proceed to the next process without waiting for the standby time to expire.

[0131] Furthermore, if the response generation unit 215 has already started the response generation process, it will interrupt the process. Here, "interrupting the response generation process" refers to a series of actions that discard the response being generated and stop sending it to the user. Specifically, the response generation unit 215 sends a signal to cancel the API request to the generation AI system 30. Another specific example is the response generation unit 215 discarding the response text reception process.

[0132] Subsequently, the server 20's processing returns to the beginning of step S101 or S102 in Figure 9. For example, the context management unit 212 of the server 20 updates the dialogue context by concatenating the newly acquired text data with the previously spoken content. Then, the confidence level information acquisition unit 213 of the server 20 uses the updated latest text data and dialogue context to request the generation AI system 30 to output confidence level information again.

[0133] This series of dynamic interrupt and cancellation processes ensures that even after System 1 enters a standby state, if the user continues to speak, System 1's response generation will not interrupt the user's speech. This allows the user to think and speak freely at their own pace without worrying about System 1's responses, resulting in a smoother and less stressful dialogue.

[0134] [Screen example] Figure 10 is a schematic diagram showing an example of a dialogue screen 1000 displayed on the display 151 of the user terminal 10. The dialogue screen 1000 is an interface for the user to interact with the system and consists of an avatar display area 1001, a dialogue log display area 1002, and a status display area 1003. The avatar display area 1001 is an area that displays the image or animation of the AI ​​avatar that will be the conversation partner. During the waiting time determined by the standby control area 214, the presentation control unit 113 causes the AI ​​avatar displayed in the avatar display area 1001 to perform specific actions, such as a "nodding" animation that indicates listening to the user's words, or a "thinking pose." This allows the user to intuitively understand that the system 1 is intentionally pausing. The dialogue log display area 1002 is an area that displays the text data of the user's utterances and the history of responses from the system 1 in chronological order. The status display area 1003 is an area that displays system 1 status information, such as the current step of the interaction, whether a response is being generated or waiting, in text or icons.

[0135] Figure 11 is a schematic diagram showing an example of a status display screen 1100 displayed on the administrator terminal 40. The status display screen 1100 is used to monitor and adjust the internal state of system 1 and the behavior of the trained model, and consists of a confidence score display area 1101 and a waiting timer display area 1102. The confidence score display area 1101 is an area that visualizes and displays the "confidence score" output from the generating AI system 30 in real time, and is displayed in numerical and gauge meter format. The waiting timer display area 1102 is an area that visualizes and displays the countdown of the "waiting time" dynamically determined based on the calculated confidence information. This allows the administrator to visually understand what level of confidence system 1 has calculated in response to the user's utterance and how many seconds it will wait, which can be used to adjust the parameters of the confidence level-based waiting control table 221, etc.

[0136] 〔supplementary explanation〕 In the embodiments described above, the cases in which the units and means are implemented by a processor were explained, but the invention is not limited to this. The units and means may be any hardware known to perform the operation.

[0137] Furthermore, although the above-described embodiment explained an example in which the server 20 provides each function, the configuration is not limited to this. Some or all of the functions provided by the server 20 in this embodiment may be provided by the user terminal 10 instead of the server 20.

[0138] While several embodiments of this disclosure have been described above, these embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. For example, a configuration or process in one embodiment may be combined with a configuration or process in another embodiment, or a modification of one embodiment may be applied to another embodiment. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0139] [Note] The details described in the above embodiments and each of the modifications are noted below.

[0140] <Note 1> A program for operating a computer equipped with a processor, the program causing the processor to perform the following steps: acquire text data indicating the content of a user's utterance; input the acquired text data into a trained model and cause the trained model to output confidence level information as a multi-level numerical value indicating the degree of likelihood that the user will make an utterance following the utterance corresponding to the text data, based on the context of the dialogue between the computer and the user at the time the text data was acquired; and determine a waiting time before the computer starts responding based on the outputted confidence level information, and start the computer's response after the determined waiting time has elapsed.

[0141] <Note 2> The program described in Appendix 1, wherein the waiting time is selected from a predetermined set of time lengths corresponding to the confidence level information output as a multi-stage numerical value.

[0142] <Note 3> The program according to Appendix 1 or Appendix 2, wherein, in the step of outputting, the program causes the trained model to output the confidence level information in advance before completing the generation of the response text to be output as the computer's response.

[0143] <Note 4> The context of the dialogue includes a plurality of requests included in a question from the computer, and in the output step, the plurality of requests are compared with the user's response elements included in the acquired text data, and if there are unanswered elements for the requests, the trained model is made to output lower confidence information compared to the case where there are no unanswered elements, as described in any of the appendices 1 to 3.

[0144] <Note 5> The program described in any of Appendix 1 to Appendix 4, which causes the processor to perform a follow-up step to prompt the user for a further response, instead of, or together with, the start of the computer's response, if, after the elapsed time determined, no new text data corresponding to additional utterances from the user has been acquired and the output confidence level information is below a predetermined threshold.

[0145] <Note 6> A program according to any one of Appendix 1 to Appendix 5, which causes the processor to further perform the steps of: analyzing the temporal characteristics of the user's speech patterns based on the user's past dialogue history; adjusting parameters that define the correspondence between the confidence level information and the waiting time based on the analyzed temporal characteristics; and determining the waiting time using the adjusted parameters in the step of starting the process.

[0146] <Note 7> A program according to any one of the appendices 1 to 6, which causes the processor to further perform the steps of acquiring the user's visual information, analyzing the acquired visual information, correcting the confidence level information output from the trained model based on the analysis results of the visual information, and in the step of starting, determining the waiting time based on the corrected confidence level information.

[0147] <Note 8> A program according to any one of Appendix 1 to Appendix 7, which causes the processor to further perform a step of interrupting the processing for the computer's response if new text data corresponding to a new utterance from the user is acquired while waiting for the waiting time to elapse.

[0148] <Note 9> An information processing device comprising a processor, wherein the processor executes all steps in any of the programs described in Appendix 1 to Appendix 7.

[0149] <Note 10> A method to be performed on a computer equipped with a processor, wherein the processor performs all steps in any of the programs described in Appendix 1 to Appendix 8.

[0150] <Note 11> A system comprising one or more processors that execute all steps in any of the programs described in Appendix 1 to Appendix 8. [Explanation of Symbols]

[0151] 1... System 10…User terminal 101... Control Unit 102...Storage section 103... Communications Department 104...Input section 105...Output section 111... Operation reception unit 112... Transmitter / Receiver 113…Display Control Unit 141... Mike 151…Display 152...Speaker 20... Server 201... Control Unit 202...Storage section 203... Communications Department 204… Input / Output Interface 211... Voice recognition unit 212...Context Management Department 213…Confidence level information acquisition unit 214... Standby Control Unit 215...Response generation unit 221... Standby control table by confidence level 222...Dialogue Context Retention Table 223... User-specific silence characteristics table 30…Generating AI system 40…Administrator terminal 80…Network 1000... Dialogue screen 1001...Avatar display area 1002...Dialogue log display area 1003...Status display area 1100...Status display screen 1101... Confidence score display area 1102... Standby timer display area

Claims

1. A program for operating a computer equipped with a processor, The aforementioned processor, The steps include obtaining text data that represents the user's utterance, The steps include: inputting the acquired text data and the context of the dialogue between the computer and the user at the time the text data was acquired into a trained model; and causing the trained model to output confidence level information, expressed as a multi-level numerical value, indicating the degree of likelihood that the user will make an utterance following the utterance corresponding to the text data, based on the acquired text data and the context of the dialogue; A program that performs the steps of: determining a waiting time before the computer starts responding based on the output confidence level information; and starting the computer's response after the determined waiting time has elapsed.

2. The program according to claim 1, wherein the waiting time is selected from a predetermined number of time lengths corresponding to the confidence level information output as a numerical value in multiple stages.

3. The program according to claim 1, wherein, in the step of outputting, the program causes the trained model to output the confidence level information in advance before completing the generation of the response text to be output as the computer's response.

4. The context of the aforementioned dialogue includes multiple requirements included in the question from the computer, The program according to claim 1, wherein in the step of outputting, the plurality of requirements are compared with the user's response elements included in the acquired text data, and if there are unanswered elements for the requirements, the program causes the trained model to output lower confidence information compared to the case where there are no unanswered elements.

5. The program according to claim 1, further causing the processor to perform a follow-up step, instead of starting a computer response, or together with the start of such response, if, after the elapsed time determined, no new text data corresponding to additional utterances from the user is acquired and the output confidence level information is below a predetermined threshold, the computer will perform a follow-up step prompting the user to provide further answers.

6. The aforementioned processor, The steps include: analyzing the temporal characteristics of the user's speech patterns based on the user's past dialogue history; The process further involves adjusting the parameters that define the correspondence between the confidence level information and the waiting time based on the analyzed temporal characteristics, The program according to claim 1, wherein in the step of initiating the program, the waiting time is determined using the adjusted parameters.

7. The aforementioned processor, The steps include obtaining the user's visual information, Further steps include analyzing the acquired visual information and correcting the confidence level information output from the trained model based on the analysis results of the visual information, The program according to claim 1, wherein in the step of initiating the program, the waiting time is determined based on the corrected confidence information.

8. The program according to claim 1, further comprising the step of causing the processor to interrupt the processing for the computer's response if new text data corresponding to a new utterance from the user is acquired while waiting for the waiting time to elapse.

9. An information processing device comprising a processor, wherein the processor executes all steps in any one of claims 1 to 8.

10. A method to be performed on a computer having a processor, wherein the processor performs all steps of a program according to any one of claims 1 to 8.

11. A system comprising one or more of the processors that perform all steps in the program according to any one of claims 1 to 8.

Images