Inquiry response system, inquiry response method, and program

Abstract

To improve the accuracy of responses to users who make inquiries about a predetermined service. [Solution] An inquiry answering system (1) includes an inquiry information acquisition unit. The inquiry information acquisition unit (103) acquires inquiry information related to an inquiry from a user for a predetermined service. The classification information acquisition unit (102) acquires classification information related to a classification related to the inquiry, which is a classification predetermined for the service. The model answer acquisition unit (103) inputs the inquiry information and the classification information into a large-scale language model, and acquires a model answer related to the classification generated by the large-scale language model and a confidence level of the classification. An output control unit (104) controls output of the model answer based on the confidence level.

Description

[Technical Field]

[0001] The present disclosure relates to an inquiry answering system, an inquiry answering method, and a program. [Background technology]

[0002] Conventionally, there is known a technology for supporting responses to inquiries received from users in a predetermined service. For example, Patent Document 1 describes an intermediary device that transmits one or more topics as topic information to an artificial intelligence, and, on the condition that the artificial intelligence notifies the user based on the topic information that a conversation with the user is a designated conversation that includes one or more topics, selects a promotion target product related to the designated conversation from one or more products, and executes an insertion process for inserting promotion information for promoting the selected promotion target product to the user into the conversation.

[0003] For example, Patent Document 2 describes an FAQ support device that includes a bot that stores multiple queries anticipated from a client operated by a user and the wording of answers related to those queries in advance in an FAQ data file as FAQ (Frequently Asked Questions) information, and that, when a client makes a query, performs a search process to present the client with candidate queries narrowed down by selecting categories or tags related to the FAQ information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7445108 [Patent Document 2] Japanese Patent Application Publication No. 2019-185191 Summary of the Invention [Problem to be solved by the invention]

[0005] However, while the intermediary device of Patent Document 1 can insert advertisements into conversations between users and AI, the content of the user's utterances is merely casual chatter with the AI, so the technology of Patent Document 1 cannot accurately respond to inquiries from users for a given service. The FAQ support device of Patent Document 2 can only present FAQ information according to predetermined categories or tags, so it cannot respond to unknown inquiries. For this reason, conventional technologies have sometimes been unable to sufficiently improve the accuracy of responses to users.

[0006] One of the objectives of the present disclosure is to improve the accuracy of responses to users who make inquiries about a given service. [Means for solving the problem]

[0007] The inquiry answering system according to the present disclosure includes an inquiry information acquisition unit that acquires inquiry information related to an inquiry from a user in a specified service; a classification information acquisition unit that acquires classification information related to a classification related to the inquiry, the classification being predetermined in the service; a model answer acquisition unit that inputs the inquiry information and the classification information into a large-scale language model and acquires a model answer related to the classification generated by the large-scale language model and a confidence level of the classification; and an output control unit that controls output of the model answer based on the confidence level. [Effects of the Invention]

[0008] The present disclosure can improve the accuracy of responses to users who make inquiries about a given service. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of a hardware configuration of an inquiry response system. [Figure 2] FIG. 10 is a diagram illustrating an example of a process executed when a user makes an inquiry by telephone. [Figure 3]FIG. 10 is a diagram illustrating an example of a process executed when a user makes an inquiry by telephone. [Figure 4] FIG. 2 is a diagram illustrating an example of functions realized by the inquiry response system. [Figure 5] FIG. 10 is a diagram illustrating an example of a directive. [Figure 6] FIG. 10 is a diagram illustrating an example of processing executed in the inquiry response system. [Figure 7] FIG. 10 is a diagram illustrating an example of a function realized in a modified example. [Figure 8] FIG. 13 is a diagram illustrating an example of input and output of a large-scale language model in Modification 3. [Figure 9] FIG. 10 is a diagram illustrating an example of an FAQ database. [Figure 10] FIG. 13 is a diagram showing an example of input and output of a large-scale language model in Modification 4. DETAILED DESCRIPTION OF THE INVENTION

[0010] [1. Hardware configuration of inquiry response system] An example of an embodiment of an inquiry answering system, an inquiry answering method, and a program according to the present disclosure will be described. FIG. 1 is a diagram showing an example of the hardware configuration of an inquiry answering system. For example, the inquiry answering system 1 includes a server 10 and a user terminal 20. Each of the server 10 and the user terminal 20 is connected to a network N such as a public communication line, the Internet, or a LAN. Note that there may be multiple servers 10 and multiple user terminals 20 at least.

[0011] The server 10 is a server computer for a predetermined service. The predetermined service is a service that the user uses or is considering using. In this embodiment, an example is given in which an insurance service that provides insurance products to the user corresponds to the predetermined service. Therefore, the phrase "insurance service" can be read as "predetermined service." The predetermined service is not limited to an insurance service. The predetermined service may be any service. The predetermined service may be an e-commerce service, a communication service, a financial service, a travel reservation service, a video distribution service, a public service, an administrative service, or other services.

[0012] For example, the server 10 includes a control unit 11, a storage unit 12, and a communication unit 13. The control unit 11 includes at least one processor. The storage unit 12 includes at least one of a volatile memory such as a RAM and a non-volatile memory such as a flash memory. The communication unit 13 includes at least one of a communication interface for wired communication and a communication interface for wireless communication. The server 10 may be connected to a device that accepts telephone inquiries from users and be able to acquire voice information indicating the contents of the telephone calls. The server 10 may itself accept telephone inquiries from users using an IP telephone function.

[0013] The user terminal 20 is a user's computer. For example, the user terminal 20 is a smartphone, a mobile phone not classified as a smartphone, a telephone not classified as a mobile phone, a tablet, a personal computer, or a wearable terminal. The user terminal 20 includes a control unit 21, a memory unit 22, a communication unit 23, an operation unit 24, and a display unit 25. The hardware configurations of the control unit 21, the memory unit 22, and the communication unit 23 may be similar to those of the control unit 11, the memory unit 12, and the communication unit 13, respectively. The operation unit 24 is an input device such as a touch panel or a mouse. The display unit 25 is a display such as a liquid crystal or organic electroluminescence (EL) display.

[0014] The program stored in the storage units 12, 22 may be supplied to the server 10 or the user terminal 20 via the network N. Also, at least one of a reading unit (e.g., a memory card slot) that reads a computer-readable information storage medium and an input / output unit (e.g., a USB port) that inputs and outputs data to and from an external device may be included in the server 10 or the user terminal 20. For example, a program stored in an information storage medium may be supplied to the server 10 or the user terminal 20 via at least one of the reading unit and the input / output unit.

[0015] Furthermore, the inquiry answering system 1 may include at least one computer. The computers included in the inquiry answering system 1 are not limited to the example of FIG. 1. For example, the inquiry answering system 1 may include only the server 10. In this case, the user terminal 20 exists outside the inquiry answering system 1. The inquiry answering system 1 may include the server 10 and other computers not shown in FIG. 1 (for example, an operator's computer, which will be described later).

[0016] [2. Overview of the inquiry response system] In this embodiment, an example is taken of a case where a user makes an inquiry about an insurance service by calling a telephone number for the insurance service from the user terminal 20. When a call is received from the user at the telephone number for the inquiry, an automated voice response is provided. The automated voice response may be provided by the server 10 or by a device connected to the server 10.

[0017] For example, the user is asked by an automated voice, "What can I do for you?" The user then responds to the initial question posed by the automated voice by speaking a query. The server 10 acquires speech information representing the user's speech. The server 10 transcribes the speech information and converts the user's speech into text. Hereinafter, the transcribed text will be referred to as "inquiry information." The server 10 analyzes the inquiry information using a large-scale language model and transfers the user's call to an appropriate hotline according to the content of the user's inquiry. The server 10 may also analyze the inquiry information using a large-scale language model and provide corresponding guidance, send a message indicating the guidance to the user, or send a link to a web page indicating the guidance to the user. The server 10 may also analyze the inquiry information using a large-scale language model, and analyze the inquiry information or respond to the inquiry using the same large-scale language model or another large-scale language model.

[0018] 2 and 3 are diagrams showing an example of processing executed when a user makes an inquiry by telephone. In the example of FIG. 2, the user makes an inquiry such as, "I am a subscriber to your company's services and would like to complete the procedure for changing my address" in response to the first question posed by the automated voice. The server 10 inputs inquiry information indicating the text of the inquiry made by the user and classification information indicating a prepared classification of the inquiry into a large-scale language model. Details of the large-scale language model will be described later.

[0019] In this embodiment, an example is given in which the classification information indicates a call center contact point as an inquiry classification. For example, the classification information includes text indicating a call center contact point as a classification name, such as a contact point for inquiries about "1. Procedures for changing addresses, etc.", a contact point for inquiries about "2. Confirmation of contract details," and a contact point for inquiries about "3. Plan changes." The large-scale language model estimates an appropriate contact point for a user inquiry based on the inquiry information and classification information input thereto. The classification information may include text as supplemental information explaining or supplementing the classification name. Note that the classification information may include "Other." "Other" includes, for example, specific address change procedures. For example, "phone number change procedures," which are not included in the name of "address change procedures," may be linked to the classification name as supplemental information in the form of a table or the like, rather than as a classification name, thereby enabling accurate classification. Furthermore, for example, conditions for specific address change cases may be linked to the classification name as supplemental information in the form of a table or the like, so that procedures that cannot be handled at the "address change procedures" counter can be routed to the "Other" procedures. This allows classification of inquiries that are difficult to guess from the classification name alone.

[0020] In this embodiment, the large-scale language model calculates a confidence level for each of the multiple contact points indicated by the classification information. The confidence level is an index indicating how confident the large-scale language model is. The confidence level can also be referred to as the accuracy of the estimation by the large-scale language model. The confidence level may also be called likelihood, reliability, probability, or score. For example, the confidence level is expressed by a numerical value. The higher the confidence level, the more confident the large-scale language model is in estimating the contact points. The lower the confidence level, the less confident the large-scale language model is in estimating the contact points. The confidence level may be expressed by letters or symbols instead of numerical values.

[0021] For example, assuming that the confidence level is expressed as a value between 0 and 1, in the example of FIG. 2, the confidence level of the contact point "1. Procedures for address change, etc." estimated by the large-scale language model is 0.93, which indicates that the contact point "1. Procedures for address change, etc." is highly likely to be appropriate as a contact point for handling user inquiries. In this embodiment, a confidence level is calculated for each of multiple contact points. In the example of FIG. 2, the highest confidence level among the contact points is 0.93. Because the confidence level is high, the contact point "1. Procedures for address change, etc." estimated by the large-scale language model is highly likely to be appropriate. As shown in FIG. 2, the large-scale language model generates a response such as, "Your inquiry is about an address change, right? I will connect you to the contact point for procedures for address change, etc." The confidence level may be 1.0 if the relevant classification can be narrowed down to one, or 0.8 if there are two or more relevant classifications and the classifications have been narrowed down to one or less but it is difficult to determine whether they match. The confidence level may also be defined when inputting the large-scale language model. In this embodiment, by quantifying the confidence level, it is possible to facilitate cooperation between programs inside and outside the inquiry response system. Note that the large-scale language model may output in a specific format such as JSON, thereby simultaneously obtaining outputs for two or more different purposes, including the answer sentence and the confidence level, thereby facilitating cooperation between programs inside and outside the inquiry response system.

[0022] Hereinafter, the answer provided by the large-scale language model will be referred to as the model answer. The server 10 converts the text indicating the model answer into speech and outputs the speech of the model answer as the content of the automated voice utterance. If the large-scale language model has a voice reading function, the server 10 can simply output the speech generated by the large-scale language model as is. The method of outputting speech to the user may also be the same for the follow-up questions described below. The server 10 transfers the user's call to the operator at counter "1. Procedures for address change, etc." estimated by the large-scale language model. The user speaks with the operator at counter "1. Procedures for address change, etc."

[0023] On the other hand, if the user's inquiry is abstract, the confidence level of each contact point may be low. In the example of Figure 3, the user responds to the first question posed by the automated voice by asking, "I'm a subscriber to your company's services. I have a question for you." The inquiry in Figure 3 is more abstract than the inquiry in Figure 2 because it does not specify what the user specifically wants to ask. In this case, even if the large-scale language model predicts a contact point, the confidence level will be low. In the example of Figure 3, the highest confidence level among the confidence levels for each contact point is 0.25. Because the confidence level is low, there is a high probability that the contact point predicted by the large-scale language model is inappropriate. If the user's call is transferred to a contact point with low confidence level, the call may be transferred to an operator at an inappropriate contact point.

[0024] Therefore, when the confidence level is low, the large-scale language model generates a follow-up question to increase the confidence level. In the example of FIG. 3, the large-scale language model generates a follow-up question such as, "What would you like to know?" In this embodiment, when the confidence level is low, a model answer is not generated, but some model answer may be generated separately from the follow-up question. Also, in FIG. 3, the model answer and the follow-up question are distinguished from each other, but the follow-up question can also be considered an answer from the large-scale language model and therefore a type of model answer. For example, the user utters an answer to the follow-up question. Note that the follow-up question in this embodiment may be a question for clarifying an ambiguous point regarding the content of the user's inquiry, or a question for clarifying the classification of the content of the user's inquiry.

[0025] Hereinafter, a user's answer will be referred to as a "user answer." A text transcribed user answer will be referred to as "user answer information." The server 10 inputs the user answer information into the large-scale language model. The large-scale language model again estimates the appropriate contact point and calculates the confidence level based on the user answer information. In the example of FIG. 3, the user answer indicates more specific content than the initial inquiry, such as "I would like to know about the procedure for changing my address." In this case, the confidence level will be higher.

[0026] For example, suppose that the confidence level for the contact point "1. Procedures for address change, etc." is 0.95 due to the follow-up question generated by the large-scale language model. In this case, since the confidence level is sufficiently high, the large-scale language model generates a model answer such as, "Your business is about address change, right? I will connect you to the contact point for address change, etc." The server 10 outputs the model answer as a voice and transfers the call, as in the case of FIG. 2. If the confidence level is not sufficiently high even after the first follow-up question, a second follow-up question may be generated. An upper limit on the number of follow-up questions may be set. If the confidence level is not sufficiently high even after the upper limit is reached, the server 10 may transfer the call to a contact point that accepts general inquiries.

[0027] As described above, the inquiry answering system 1 inputs inquiry information and classification information into a large-scale language model, and obtains a model answer and confidence level output from the large-scale language model. If the confidence level is high, the inquiry answering system 1 outputs the model answer and transfers the call to the help desk estimated by the large-scale language model. If the confidence level is low, the inquiry answering system 1 causes the large-scale language model to generate additional questions to increase the confidence level. This allows the inquiry answering system 1 to improve the accuracy of answers to users. Details of the inquiry answering system 1 will be described below.

[0028] [3. Functions realized by the inquiry response system] FIG. 4 is a diagram showing an example of functions realized by the inquiry answering system 1. FIG. 4 shows the functions of the server 10 among the functions realized by the inquiry answering system 1. For example, the server 10 includes a data storage unit 100, an inquiry information acquisition unit 101, a classification information acquisition unit 102, a model answer acquisition unit 103, an output control unit 104, and a user answer information acquisition unit 105. The data storage unit 100 is realized by the storage unit 12. The inquiry information acquisition unit 101, the classification information acquisition unit 102, the model answer acquisition unit 103, the output control unit 104, and the user answer information acquisition unit 105 are realized by the control unit 11.

[0029] [3-1. Data storage section] The data storage unit 100 stores various data necessary for responding to inquiries from users. For example, the data storage unit 100 stores classification information. The data storage unit 100 may store default instruction sentences to be input to the large-scale language model. The instruction sentences indicate tasks to be executed by the large-scale language model. The instruction sentences can also be considered default prompts prepared by the inquiry answering system 1. The classification information may be included in the instruction sentences. For example, the classification information may be indicated as part of the instruction sentences. The classification information may be data separate from the instruction sentences.

[0030] FIG. 5 is a diagram illustrating an example of an instruction. In this embodiment, the instruction includes a sentence indicating that a model answer should be generated and a confidence score should be calculated based on the inquiry information and classification information. For example, the instruction may be a sentence such as, "You are an AI that estimates an appropriate contact point to handle an inquiry from a user. Please estimate an appropriate contact point based on the inquiry information and classification information input to you, output a model answer, and calculate a confidence score." The instruction may be any sentence that instructs the large-scale language model to generate a model answer and calculate a confidence score, and is not limited to the above example. The instruction may include a sentence that instructs the large-scale language model to generate a model answer and calculate a confidence score.

[0031] The instruction may include a sentence indicating that a follow-up question should be generated if the confidence level is below a threshold. For example, the instruction may be a sentence such as, "If the highest confidence level is below a threshold, generate a follow-up question to increase the confidence level." The confidence level threshold may also be included in the instruction. In this embodiment, the threshold is 0.9 as an example, but the threshold may be any value. For example, the threshold may be a value obtained by multiplying the upper limit of the confidence level by a predetermined percentage (e.g., 90%). The instruction for generating a follow-up question may be any sentence that instructs the large-scale language model to generate a follow-up question, and is not limited to the above example. The instruction for generating a follow-up question may include a sentence instructing the large-scale language model to generate a follow-up question.

[0032] In this embodiment, an example is taken of a case where an external system linked to the inquiry answering system 1 manages a large-scale language model. Therefore, actual data of the large-scale language model is stored in the external system. The server 10 uses the large-scale language model in the external system via the network N. When the inquiry answering system 1 manages the large-scale language model, the data storage unit 100 may store actual data of the large-scale language model. The server 10 may use the large-scale language model stored in the data storage unit 100.

[0033] A large-scale language model is a model trained on a huge amount of text. The large-scale language model includes a program that processes input text and parameters referenced by the program. When training text is trained in the large-scale language model, the parameters are adjusted. The large-scale language model calculates an embedding (feature vector) of the input text based on the parameters and produces output according to the embedding. The large-scale language model may also divide the input text into units called tokens and calculate the embedding of each token. The large-scale language model produces output according to the sequence of the embedding of each token.

[0034] The large-scale language model may be a known model. For example, the large-scale language model may be a Generative Pre-trained Transformer (GPT), Bidirectional Encoder Representations from Transformers (BERT), Amazon Bedrock (registered trademark), Language Model for Dialogue Applications (LaMDA), a neural network, Claude (registered trademark), or other models. The large-scale language model may be a model called generative AI. The large-scale language model may be fine-tuned for the query answering system 1, or may be a general-purpose model that is not fine-tuned specifically for the query answering system 1.

[0035] Furthermore, the data stored in the data storage unit 100 is not limited to the above examples. The data storage unit 100 may store a program for converting between speech and text. The data storage unit 100 may store information on a confidence threshold. If a program separate from the large-scale language model calculates the confidence, the data storage unit 100 may store the other program that calculates the confidence. If the large-scale language model is not managed by an external system, the data storage unit 100 may store the large-scale language model. The data storage unit 100 may store information for forwarding a user's call to a counter (e.g., the extension number of each counter). The data storage unit 100 may store information indicating an answer to be output when the confidence is low.

[0036] [3-2. Inquiry Information Acquisition Section] The inquiry information acquisition unit 101 acquires inquiry information related to an inquiry from a user regarding insurance services. In this embodiment, the user makes an inquiry by voice, so the inquiry information acquisition unit 101 acquires inquiry information indicating the text by converting voice information indicating the user's voice into text. The inquiry information acquisition unit 101 may acquire the voice information from the user terminal 20, or may acquire the voice information from a device that accepts calls from the user terminal 20.

[0037] The transcription of the speech information may be performed by a computer other than the server 10. In this case, the query information acquisition unit 101 may acquire, from the other computer, query information indicating text into which speech has been converted. If the large-scale language model can process speech information, the query information acquisition unit 101 may acquire the speech information as is as the query information. That is, the query information may indicate text or speech.

[0038] Furthermore, an inquiry from a user may be made by a method other than a telephone. For example, an inquiry from a user may be made by chatting in which a text message is input. In this case, the inquiry information acquisition unit 101 acquires inquiry information indicating the message input in the chat. When an inquiry from a user is made by chatting in which a voice message is input, the inquiry information acquisition unit 101 may acquire the inquiry information by a method similar to that of a telephone.

[0039] For example, an inquiry from a user may be made by email, SMS (Short Message Service), SNS (Social Networking Service), a messaging app, an inquiry form on a website, a search form on a website, or other methods. The inquiry information acquisition unit 101 may acquire inquiry information indicating an inquiry from a user made by these methods.

[0040] [3-3. Classification information acquisition part] The classification information acquisition unit 102 acquires classification information relating to classifications of inquiries that are predetermined in insurance services. The classification indicated by the classification information is a classification of inquiries that has been assumed in advance. The classification can also be referred to as the subject matter, type, genre, or category of the inquiry. The classification information indicates at least one classification. The classification information may indicate only one classification or may indicate multiple classifications.

[0041] In this embodiment, the classification indicated by the classification information is a classification related to the counter that handles inquiries. For example, a call center for insurance services has multiple counters. That is, the call center has multiple counters that are divided according to the content of the inquiry so that each of the multiple counters can handle a specific inquiry. Each counter corresponds to a classification. The classification indicated by the classification information is not limited to a counter. Other examples of classifications will be described in the modified example below.

[0042] In this embodiment, since the instruction sentences stored in the data storage unit 100 include classification information, the classification information acquisition unit 102 acquires the classification information by acquiring the instruction sentences from the data storage unit 100. The classification information may be data separate from the instruction sentences. In this case, the data storage unit 100 stores the classification information as data separate from the instruction sentences. The classification information acquisition unit 102 acquires the classification information, which is data separate from the instruction sentences, from the data storage unit 100. The classification information may be stored in a computer other than the server 10 or in an information storage medium. In this case, the classification information acquisition unit 102 may acquire the classification information from the other computer or information storage medium.

[0043] For example, the classification information acquisition unit 102 may search for classification information to be input to the large-scale language model from a database storing various types of classification information in the form of RAG (Retrieval Augmented Generation), may search for classification information from the database in the form of semantic search, or may search for classification information from the database using embedded expressions related to text included in the input to the large-scale language model. The database associates classification information with a search index. The classification information acquisition unit 102 may acquire classification information by searching the database based on query information. A specific example of acquiring classification information by search will be described in a modified example below.

[0044] [3-4. Model response acquisition section] The model answer acquisition unit 103 inputs the inquiry information and the classification information into the large-scale language model, and acquires a model answer related to the classification generated by the large-scale language model and the confidence level of the classification. In this embodiment, the classification indicated by the classification information is a classification related to the counter that responds to the inquiry, so the model answer acquisition unit 103 acquires, as the model answer, an answer related to the counter that corresponds to the classification to which the inquiry belongs.

[0045] In this embodiment, since the large-scale language model is managed by an external system, the model response acquisition unit 103 inputs the query information and classification information to the large-scale language model by transmitting the query information and classification information to the external system. The model response acquisition unit 103 inputs the query information and classification information as prompts for the large-scale language model. When actual data of the large-scale language model is stored in the data storage unit 100, the model response acquisition unit 103 may input the query information and classification information to the large-scale language model stored in the data storage unit 100.

[0046] The processing performed by the large-scale language model may be known processing. For example, when query information and classification information are input, the large-scale language model calculates embedded representations (feature vectors) of the query information and classification information based on parameters adjusted through prior learning. The large-scale language model outputs a model answer as an output corresponding to the embedded representation. The large-scale language model may divide the query information and classification information into units called tokens and calculate embedded representations for each token. In this case, the large-scale language model predicts the continuation as necessary based on the sequence of the embedded representations of the tokens and outputs a model answer.

[0047] In this embodiment, the case where the large-scale language model calculates the confidence level is taken as an example, but the confidence level may be calculated by another program other than the large-scale language model. In this case, the other program may calculate the confidence level based on the output or internal processing information of the large-scale language model. The other program may be stored in an external system or in the data storage unit 100.

[0048] The method for calculating the confidence may be a known method. For example, when generating tokens, the large-scale language model may calculate a probability distribution for tokens that are likely to appear next. In this case, the probability distribution indicates how confident the large-scale language model is about the likelihood of each token appearing. Such a probability distribution may be used as the confidence. As another example, the large-scale language model may calculate a score for the product. The score indicates how appropriate the product is for the context of the prompt. Such a score may be used as the confidence. A library (e.g., the Transformers library) used in a programming language such as Python may be used to calculate the confidence.

[0049] For example, the model response acquisition unit 103 acquires a model response and a confidence factor from an external system that manages a large-scale language model. When a large-scale language model is stored in the data storage unit 100, the model response acquisition unit 103 may acquire a model response and a confidence factor output by the large-scale language model stored in the data storage unit 100. When another program that calculates a confidence factor is stored in the data storage unit 100, the model response acquisition unit 103 may acquire a confidence factor calculated by the other program stored in the data storage unit 100.

[0050] [3-5. Output control section] The output control unit 104 controls the output of the model answer based on the confidence level. Controlling the output of the model answer means controlling whether or not to output the model answer. The confidence level is used as a condition for whether or not to output the model answer. The output of the model answer is output to the user. Output to the user includes not only audio output but also data output (transmission of data). For example, the output control unit 104 may output the model answer to the user terminal 20 by audio, or may output the model answer by text.

[0051] For example, the output control unit 104 determines whether the confidence level is equal to or greater than a threshold, and controls the output of the model answer based on the result of the determination. The threshold is assumed to be stored in the data storage unit 100. The output control unit 104 outputs the model answer to the user when it is determined that the confidence level is equal to or greater than the threshold. That is, the output control unit 104 outputs the model answer to the user on the condition that it is determined that the confidence level is equal to or greater than the threshold. The output control unit 104 does not output the model answer to the user when it is determined that the confidence level is less than the threshold.

[0052] In addition, when the confidence level of each of multiple contact points is calculated, there may be multiple contact points with confidence levels above a threshold. In this case, the model answer indicates the contact point with the highest confidence level. The instruction statement may include a sentence instructing the contact point with the highest confidence level to be included in the model answer. The output control unit 104 outputs a model answer indicating the contact point with the highest confidence level. In this embodiment, an example is given of a case where the output control unit 104 outputs a model answer that does not include the details of contact points with the second highest confidence levels or higher, but the output control unit 104 may also output a model answer that includes the details of contact points with the second highest confidence levels or higher.

[0053] For example, when the confidence level is less than a threshold, the output control unit 104 acquires a follow-up question for the user generated by the large-scale language model and outputs the follow-up question to the user. In this embodiment, an instruction statement is provided indicating that a follow-up question should be generated when the confidence level is less than the threshold. Therefore, the large-scale language model generates a follow-up question when the confidence level is less than the threshold. The output control unit 104 acquires the follow-up question generated by the large-scale language model. The output control unit 104 may output the follow-up question by voice or by text, similar to when outputting the model answer.

[0054] In this embodiment, the output control unit 104 acquires a follow-up question based on an instruction statement indicating that a follow-up question should be generated to increase the confidence level when the confidence level is less than a threshold. Increasing the confidence level means increasing the confidence level above the previously acquired confidence level. For example, the instruction statement includes a sentence indicating an increase in the confidence level, such as "When the confidence level is less than a threshold, please generate a follow-up question that will increase the confidence level." The sentence indicating an increase in the confidence level may be other wording that means an increase in the confidence level.

[0055] If no instruction statement is provided to generate a follow-up question when the confidence level is below the threshold, the output control unit 104 may input an instruction statement to the large-scale language model to generate a follow-up question when the confidence level is below the threshold, causing the large-scale language model to generate a follow-up question. That is, the acquisition of the model answer and confidence level and the acquisition of the follow-up question may be performed separately, and the large-scale language model may be processed in two separate steps.

[0056] Furthermore, if the confidence level of the model output for the initial inquiry is less than a threshold, the output control unit 104 may transfer the user's call to a general-purpose help desk without outputting any follow-up questions. In this case, the output control unit 104 does not include a function for outputting follow-up questions. The server 10 does not include the user answer information acquisition unit 105. A configuration in which the output control unit 104 does not include a function for outputting follow-up questions and the server 10 does not include the user answer information acquisition unit 105 is also within the scope of the present disclosure. Even in such a configuration, the inquiry answering system 1 can prevent a model answer with a low confidence level from being output by controlling the model answer based on the confidence level, thereby improving the accuracy of the answers to the user.

[0057] The output control unit 104 may also determine a response, such as a follow-up question, from predetermined standard phrases to prevent an unexpected response from being provided to the user. In this case, the data storage unit 100 stores standard phrase information indicating predetermined standard phrases. The output control unit 104 may select a standard phrase for the large-scale language model. The response sentence output to the user may be the same as the standard phrase, or the content of a portion of the standard phrase may be changed by the large-scale language model. An identifier such as a number may be assigned to the response sentence to prevent the large-scale language model from erroneously replacing characters in the process of selecting a response sentence. Supplementary information may also be associated with each response sentence to make the generation of a response by the large-scale language model more appropriate.

[0058] [3-6. User response information acquisition section] The user answer information acquisition unit 105 acquires user answer information related to the user's answer to the follow-up question. If no follow-up question is asked, the processing of the user answer information acquisition unit 105 is not executed. The user answer information is input from the user after the initial inquiry. Similar to the inquiry information acquisition unit 101, the user answer information acquisition unit 105 converts voice information representing the user's voice into text, and acquires user answer information representing the text.

[0059] Note that, similarly to the inquiry information acquisition unit 101, the voice information may be acquired from the user terminal 20, the voice information may be acquired from a device that accepts a call from the user terminal 20, and the transcription of the voice information may be performed by a computer other than the server 10. Also similarly to the inquiry information acquisition unit 101, the user response may be made by a method other than by telephone, and the voice information may be acquired as user response information if the large-scale language model corresponds to the voice.

[0060] In this embodiment, if the already acquired confidence level is less than a threshold, the model answer acquisition unit 103 inputs the user answer information into the large-scale language model and acquires the model answer and confidence level again. The model answer acquisition unit 103 may input not only the user answer information but also the inquiry information and classification information into the large-scale language model. An instruction statement for acquiring the model answer and confidence level again may be stored in the data storage unit 100. The model answer acquisition unit 103 may input the user answer information together with the instruction statement into the large-scale language model.

[0061] In this embodiment, the model answer acquisition unit 103 inputs the user answer information into the large-scale language model by transmitting the user answer information to an external system that manages the large-scale language model, and again acquires the model answer and confidence level. When actual data of the large-scale language model is stored in the data storage unit 100, the model answer acquisition unit 103 inputs the user answer information into the large-scale language model stored in the data storage unit 100, and again acquires the model answer and confidence level. Although the information input into the large-scale language model is different from that when the initial model answer was generated, the process itself executed by the large-scale language model may be the same as when the initial model answer was generated.

[0062] In this embodiment, when the already acquired confidence is less than a threshold, the output control unit 104 controls the output of the reacquired model answer based on the reacquired confidence. The output control unit 104 determines whether the reacquired confidence is equal to or greater than the reacquired confidence threshold, and controls the output of the model answer based on the result of the determination. The output control unit 104 outputs the reacquired model answer to the user on the condition that it is determined to be equal to or greater than the reacquired confidence threshold. When the reacquired confidence is less than the threshold, the output control unit 104 does not output the reacquired model answer to the user.

[0063] In the present embodiment, the inquiry answering system 1 repeatedly acquires and outputs follow-up questions, acquires user answer information, and acquires a model answer and a confidence level until the confidence level reaches or exceeds a threshold. If the confidence level does not reach or exceed the threshold even after the acquisition and output of follow-up questions, acquisition of user answer information, and acquisition of a model answer and a confidence level have been repeated a predetermined number of times, the output control unit 104 outputs a prepared answer for the insurance service to the user. The output of the prepared answer includes not only audio output but also data output (data transmission). For example, the output control unit 104 may output the prepared answer to the user terminal 20 as audio or as text. Furthermore, in the inquiry answering system 1 of the present embodiment, the large-scale language model may generate a summary of the inquiry information and include the summary in the output or answer by the large-scale language model.

[0064] The predetermined number of times is the upper limit of the number of repetitions. Information indicating the predetermined number of times is assumed to be stored in the data storage unit 100. Information indicating the prepared answers is also assumed to be stored in the data storage unit 100. The output control unit 104 counts the number of repetitions for a certain user and determines whether the number of repetitions has reached the predetermined number. If the output control unit 104 determines that the number of repetitions has not reached the predetermined number, it executes the above-mentioned process to perform the next repetition. If the output control unit 104 determines that the number of repetitions has reached the predetermined number but the confidence level has not yet reached or exceeded the threshold, it outputs a prepared answer to the user.

[0065] [4. Processing performed by the inquiry response system] FIG. 6 is a diagram showing an example of processing executed by the inquiry answering system 1. Of the processing executed by the inquiry answering system 1, FIG. 6 mainly shows processing executed by the server 10. The processing of FIG. 6 is executed by the control unit 11 executing a program stored in the storage unit 12. The steps of FIG. 6 are an example of an inquiry answering method. The processing of FIG. 6 is executed every time an inquiry call from a user is received at a telephone number for an inquiry about insurance services.

[0066] As shown in Fig. 6, the server 10 acquires query information by converting the voice of a query from a user into text (S1). The server 10 acquires classification information (S2). If classification information is included in a command statement as shown in Fig. 5, the server 10 acquires the classification information by acquiring the command statement in S2. The server 10 inputs the query information and classification information into the large-scale language model by transmitting the query information and classification information to an external system that manages the large-scale language model (S3).

[0067] The server 10 acquires a model answer and a confidence level from an external system that manages a large-scale language model (S4). If the confidence level is less than a threshold in S4, it is assumed that a follow-up question has also been generated by the large-scale language model. The server 10 determines whether the confidence level is equal to or greater than a threshold (S5). If it is determined in S5 that the confidence level is less than the threshold (S5: N), it determines whether the number of repetitions of the processes of S7 to S11 has reached a predetermined number (S6). Information indicating the number of repetitions is assumed to be stored in the storage unit 12.

[0068] If it is determined in S6 that the number of repetitions has not reached the predetermined number (S6: N), the server 10 increments the number of repetitions and acquires a follow-up question generated by the large-scale language model (S7). The follow-up question is assumed to have been sent from an external system in step S4. The follow-up question does not have to be generated together with the model answer and the confidence factor. In this case, in S7, the server 10 instructs the large-scale language model to generate a follow-up question. The server 10 acquires the follow-up question generated by the large-scale language model.

[0069] The server 10 outputs a follow-up question to the user (S8). The server 10 acquires user answer information by converting the voice of the user's answer to the follow-up question into text (S9). The server 10 inputs the user answer information into the large-scale language model by transmitting the user answer information to an external system that manages the large-scale language model (S10). The server 10 acquires a model answer and a confidence factor from the external system that manages the large-scale language model (S11) and proceeds to the process of S5. The processes of S6 to S11 are repeated until the confidence factor becomes equal to or exceeds a threshold value or until the number of repetitions reaches a predetermined number.

[0070] If it is determined in S5 that the confidence level is equal to or greater than the threshold (S5: Y), the server 10 outputs a model answer to the user (S12). The server 10 transfers the call to the service center with the highest confidence level (S13), and the process ends. The user then speaks to an operator at the service center to which the call has been transferred.

[0071] If it is determined in S6 that the number of repetitions has reached a predetermined number (S6: Y), the server 10 outputs a predetermined response to the user (S14). The server 10 transfers the call to a service center that accepts general inquiries (S15), and the process ends. The user then speaks with an operator at the service center to which the call has been transferred.

[0072] [5. Summary of embodiments] The inquiry answering system 1 of this embodiment acquires inquiry information. The inquiry answering system 1 acquires classification information. The inquiry answering system 1 inputs the inquiry information and classification information into a large-scale language model and acquires a model answer and a confidence level. The inquiry answering system 1 controls the output of the model answer based on the confidence level. As a result, the inquiry answering system 1 does not unconditionally output a model answer, but controls the output of the model answer based on the confidence level of the category to which the user's inquiry belongs, thereby improving the accuracy of the answer to the user. For example, when determining whether to output a model answer based on the confidence level, the inquiry answering system 1 can prevent a model answer from being output despite a low confidence level. When outputting a model answer with a high confidence level, the inquiry answering system 1 can output a model answer with high estimation accuracy using the large-scale language model.

[0073] Furthermore, the inquiry answering system 1 outputs a model answer to the user when the confidence level is equal to or greater than a threshold. When the confidence level is less than the threshold, the inquiry answering system 1 acquires a follow-up question for the user generated by the large-scale language model and outputs the follow-up question to the user. The inquiry answering system 1 acquires user answer information. The inquiry answering system 1 inputs the user answer information into the large-scale language model and acquires a model answer and confidence level again. The inquiry answering system 1 controls the output of the re-acquired model answer based on the re-acquired confidence level. This allows the inquiry answering system 1 to output a model answer when the confidence level is equal to or greater than a threshold, thereby improving the accuracy of the answer to the user. Even if the confidence level is less than the threshold, the inquiry answering system 1 has the opportunity to increase the confidence level by asking a follow-up question, thereby improving the accuracy of the answer to the user.

[0074] Furthermore, when the confidence level is below a threshold, the inquiry answering system 1 acquires a follow-up question based on a directive that indicates that a follow-up question should be generated to increase the confidence level. This makes it easier for the inquiry answering system 1 to increase the confidence level by asking a follow-up question, making it easier to obtain a more accurate model answer and improving the accuracy of the answer to the user.

[0075] Furthermore, inquiry answering system 1 repeats the acquisition and output of follow-up questions, acquisition of user answer information, and acquisition of model answers and confidence levels until the confidence level reaches or exceeds a threshold. If the inquiry answering system 1 repeats the generation and output of follow-up questions, acquisition of user answer information, and acquisition of model answers and confidence levels a predetermined number of times but the confidence level does not reach or exceed the threshold, it outputs an answer prepared in advance for the insurance service to the user. This allows inquiry answering system 1 to prevent the user from being asked follow-up questions excessively many times, thereby improving user convenience.

[0076] Furthermore, the classification indicated by the classification information is a classification relating to the contact point that responds to the inquiry. The inquiry answering system 1 acquires, as a model answer, an answer relating to the contact point that corresponds to the classification to which the inquiry belongs. This allows the inquiry answering system 1 to output a more appropriate contact point as a model answer. The inquiry answering system 1 can guide the user to a contact point that is appropriate for the user's inquiry.

[0077] [6. Modifications] The present disclosure is not limited to the above-described embodiments, and may be modified as appropriate without departing from the spirit of the present disclosure.

[0078] 7 is a diagram showing an example of functions realized in the modified example. For example, the server 10 includes a count determination unit 106, a consideration matter output unit 107, an input determination unit 108, an output restriction unit 109, a time determination unit 110, and an other answer output unit 111. Each of the count determination unit 106, the consideration matter output unit 107, the input determination unit 108, the output restriction unit 109, the time determination unit 110, and the other answer output unit 111 is realized by the control unit 11.

[0079] [6-1. Variation 1] For example, in the embodiment, the predetermined number of times corresponding to the upper limit of repetitions is a predetermined number. Depending on the confidence level, even if the number of repetitions reaches the predetermined number, the confidence level may exceed the threshold if a few more follow-up questions are output to the user. For example, suppose the confidence level threshold is 0.9, the predetermined number of times is three, and the confidence level changes between 0.12, 0.45, and 0.89. In the example embodiment, if the confidence level does not exceed 0.9 after the third follow-up question, a predetermined answer is output and the call is forwarded to a general-purpose help desk. However, if the confidence level changes in this way, the server 10 may be able to guide the user to the appropriate help desk with one more follow-up question. Therefore, in Modification 1, the predetermined number of times is determined according to the confidence level.

[0080] The inquiry answering system 1 of the first modification includes a count determination unit 106. The count determination unit 106 determines the predetermined count based on the confidence level. If the predetermined count is predetermined, changing the predetermined count corresponds to determining the predetermined count. If the predetermined count is not predetermined, generating the predetermined count corresponds to determining the predetermined count. The output control unit 104 controls the repetition of obtaining the model answer and the confidence level, etc., based on the predetermined count determined by the count determination unit 106.

[0081] For example, the count determination unit 106 may determine the predetermined number of times based on the difference obtained by subtracting the confidence level from the threshold value. The confidence level is the current confidence level. For example, the confidence level may be the confidence level at the time when a predetermined number of times has been reached, or may be the confidence level obtained up to that point. The count determination unit 106 may determine the predetermined number of times when the number of repetitions has reached a predetermined number, or may determine the predetermined number of times before the number of repetitions has reached the predetermined number (for example, when a model answer to the first query has been generated).

[0082] The predetermined difference is a value corresponding to a threshold value used to determine the predetermined number of times. Information indicating the predetermined difference is stored in the data storage unit 100. When the number of times determination unit 106 determines that the difference obtained by subtracting the confidence level from the threshold is equal to or greater than the predetermined difference, it means that the current confidence level is far from the threshold, and therefore, a follow-up question may not be very effective. When the difference obtained by subtracting the confidence level from the threshold is less than the predetermined difference, it means that the current confidence level is close to the threshold, and therefore, a follow-up question may be effective.

[0083] Therefore, the number of times determination unit 106 may determine the predetermined number of times so that the smaller the difference obtained by subtracting the confidence level from the threshold, the greater the predetermined number of times. That is, the number of times determination unit 106 may determine the predetermined number of times so that the larger the difference obtained by subtracting the confidence level from the threshold, the fewer the predetermined number of times. In other words, the number of times determination unit 106 may determine the predetermined number of times so that the predetermined number of times is greater when the difference obtained by subtracting the confidence level from the threshold is less than the predetermined difference than when the difference is equal to or greater than the predetermined difference. That is, the number of times determination unit 106 may determine the predetermined number of times so that the predetermined number of times is less when the difference obtained by subtracting the confidence level from the threshold is equal to or greater than the predetermined difference than when the difference is less than the predetermined difference.

[0084] Note that relational data indicating the relationship between the difference obtained by subtracting the confidence level from the threshold and the predetermined number of times is stored in the data storage unit 100. The count determination unit 106 may determine the predetermined number of times based on the confidence level and the relational data. The count determination unit 106 calculates the difference obtained by subtracting the confidence level from the threshold and determines the predetermined number of times so that the predetermined number of times is the predetermined number associated with the difference in the relational data. The method of determining the predetermined number by the count determination unit 106 is not limited to the above example. For example, the count determination unit 106 may determine the predetermined number of times based on a change in the confidence level. The count determination unit 106 may determine the predetermined number of times so that the predetermined number of times increases when the increase in the confidence level is equal to or greater than a predetermined amount. Note that the count determination unit 106 may increase the predetermined number of times when the confidence level exceeds another threshold, and may decrease the predetermined number of times when the confidence level falls below another threshold. In this way, for example, even if the absolute value of the confidence level of an early answer is low, the count determination unit 106 can increase the predetermined number of times when the absolute value of the confidence level of a subsequent answer is high.

[0085] The count determination unit 106 may determine the predetermined number of times based on the rate of change of the confidence levels. Each time an answer is generated using the large-scale language model, the count determination unit 106 may acquire the maximum value among the confidence levels, and may increase the predetermined number of times if the rate of change of the maximum value exceeds the predetermined rate of change, or may decrease the predetermined number of times if the rate of change of the maximum value falls below the predetermined rate of change. Here, the count determination unit 106 may determine the predetermined number of times based on the rate of change of the maximum value for answers generated within a unit time, rather than the rate of change of the consecutive maximum values. Alternatively, the count determination unit 106 may determine the predetermined number of times based on a statistical value including an average value of each confidence level, instead of the maximum value. Alternatively, the count determination unit 106 may determine the predetermined number of times based on the rate of change of the confidence levels for common classifications, instead of the maximum value. Alternatively, the count determination unit 106 may determine the predetermined number of times based on the amount of change in the confidence levels, instead of the rate of change in the confidence levels. For example, if the initial value of the predetermined number of times is 3, and the maximum value of the confidence level changes to 0.2, 0.2, and 0.6 and the rate of change from 0.2 to 0.6 exceeds the predetermined rate of change, the number of times determination unit 106 may increase the predetermined number of times. Also, if the initial value of the predetermined number of times is set to 5, and the maximum value of the confidence level changes to 0.2, 0.2, and 0.2 and the rate of change of the maximum value continues to be below the predetermined rate of change, the number of times determination unit 106 may decrease the predetermined number of times.

[0086] The count determination unit 106 may change the predetermined number of times based on the user's emotions estimated by inputting text, audio information, or video information corresponding to the inquiry information into a known model (such as a language model, a voice recognition model, or an image recognition model). Specifically, when the known model estimates the user's emotions as negative in a binary classification of positive and negative, the count determination unit 106 may reduce the predetermined number of times to reduce the number of interactions (conversations) with the user. The count determination unit 106 may change the predetermined number of times based on the number of characters required for the inquiry information from the user or the response generated by the large-scale language model, or may change the predetermined number of times based on the time required for inputting or generating the inquiry information from the user or the response generated by the large-scale language model. Specifically, when the number of characters required for the inquiry information is below the predetermined number, the predetermined number of times may be increased to allow an increase in the number of interactions with the user. The predetermined number of times may be the upper limit (allowable number) of repetitions per unit time. The count determination unit 106 may also determine the predetermined number of times based on whether the number of interactions within a unit time exceeds, falls below, or is maintained at a predetermined value. The count determination unit 106 may also determine the predetermined number of times based on whether the frequency of interactions within a unit time exceeds, falls below, or is maintained at a predetermined value. The count determination unit 106 may also determine the predetermined number of times based on whether the speed (pace) of interactions within a unit time exceeds, falls below, or is maintained at a predetermined value. For example, if more than three interactions occur within one minute and the number, frequency, or speed (pace) of interactions within a unit time exceeds a predetermined value, the count determination unit 106 may increase the predetermined number of times. Here, "maintaining a predetermined value" refers to the difference between a certain value related to interactions within a unit time and a predetermined value being less than another predetermined value.

[0087] The inquiry answering system 1 of Variation 1 determines the predetermined number of times based on the confidence level. This allows the inquiry answering system 1 to determine the number of times to repeat processes such as obtaining a model answer based on the confidence level, thereby improving the accuracy of answers to the user. For example, when the confidence level is close to reaching a threshold, the inquiry answering system 1 can increase the predetermined number of times to give the user an opportunity to obtain a model answer whose confidence level is equal to or greater than the threshold. Furthermore, instead of or in addition to the confidence level, the inquiry answering system 1 of Variation 1 may determine the predetermined number of times based on the estimated user's emotions, the number of characters required for the user's inquiry information or the answer generated by the large-scale language model, the time required for inputting or generating the user's inquiry information or the answer generated by the large-scale language model, and / or some other value required for interaction within a unit time. This allows the inquiry answering system 1 to determine the number of times to repeat processes such as obtaining a model answer based on whether the interaction with the user is progressing smoothly with the user's stress reduced.

[0088] [6-2. Variation 2] For example, the information input to the large-scale language model is not limited to the examples in the embodiment. In the second modification, a case where follow-up questions are repeatedly asked until the confidence level reaches or exceeds a threshold, as in the embodiment or the first modification, is taken as an example. In this case, the user makes an initial inquiry and answers to the follow-up questions asked after the initial inquiry, and the history of these may be input to the large-scale language model.

[0089] The model answer acquisition unit 103 of Modification 2 further inputs the history related to the inquiry information into the large-scale language model and again acquires a model answer and a confidence factor. The history related to the inquiry information may include only the inquiry information, or may include not only the inquiry information but also user answer information indicating user answers to follow-up questions asked after the initial inquiry. If multiple follow-up questions are asked, the history related to the inquiry information may include user answer information to each of the multiple follow-up questions.

[0090] For example, if a first follow-up question is asked, the history related to the inquiry information includes only inquiry information indicating the first inquiry. In this case, the model answer acquisition unit 103 inputs the inquiry information into the large-scale language model together with user answer information indicating the user's answer to the first follow-up question. Although the information input into the large-scale language model differs from that in the embodiment, the process up to the large-scale language model producing an output according to the input may be the same as in the embodiment.

[0091] For example, if a second or subsequent follow-up question is asked, the history of inquiry information includes inquiry information indicating the first inquiry and user answer information regarding user answers to the previous follow-up questions. In this case, the model answer acquisition unit 103 inputs the inquiry information and the user answer information regarding user answers to the previous follow-up questions, along with the user answer information indicating the user answer to the latest follow-up question, into the large-scale language model. In this case, the information input to the large-scale language model is different from that in the embodiment, but the processing up to the large-scale language model producing an output in response to the input may be the same as in the embodiment.

[0092] The inquiry answering system 1 of Variation 2 further inputs the history related to the inquiry information into the large-scale language model and again obtains a model answer and a confidence factor. This allows the inquiry answering system 1 to obtain a model answer and a confidence factor based on the history of input by the user, making it possible to output a model answer that is in line with the context of the interaction with the user.

[0093] [6-3. Variation 3] For example, as in the embodiment, when a large-scale language model estimates an appropriate contact point, communicating to the contact point the considerations that the contact point should take into account may make subsequent inquiry response work smoother. Since the large-scale language model has previous user inquiries and user responses input thereto, it may be able to estimate the considerations that the contact point should take into account. Note that the large-scale language model may estimate another appropriate large-scale language model instead of an appropriate contact point. Another appropriate large-scale language model refers to another large-scale language model that can generate a response equivalent to a response by an appropriate contact point. Furthermore, the large-scale language model may estimate another appropriate input to the large-scale language model instead of an appropriate contact point.

[0094] Therefore, an example will be given in which the model response acquisition unit 103 of Variation 3 further acquires considerations that should be taken into account by the customer service center and that are generated by a large-scale language model. The considerations are information provided to the customer service center. The considerations are text that indicates the content that the customer service center should consider. The considerations are not limited to text, and may be other information such as images. The considerations may be information in any format that can be generated by a large-scale language model.

[0095] FIG. 8 is a diagram showing an example of input and output of a large-scale language model in Modification 3. As shown in the upper part of FIG. 8, the instruction sentence input to the large-scale language model includes a sentence indicating that the counter estimated by the large-scale language model should generate considerations to be taken into account. For example, the instruction sentence includes a sentence such as "If there is a counter whose confidence level is equal to or greater than a threshold, please also generate considerations to be taken into account by that counter." The instruction sentence may be any sentence that instructs the generation of considerations, and is not limited to the above example. The instruction sentence may include a statement that instructs the generation of considerations. A manual for the counter operator to refer to may be input to the large-scale language model along with the instruction sentence.

[0096] For example, the model answer acquisition unit 103 inputs the inquiry information and classification information together with the instruction sentence into the large-scale language model. When an additional question is asked, the model answer acquisition unit 103 may also input the additional question into the large-scale language model. The large-scale language model calculates these embedded expressions based on the learned parameters and performs output according to the embedded expressions. In variant example 3, the output also includes considerations for the counter. The model answer acquisition unit 103 acquires considerations for the counter output from the large-scale language model. The large-scale language model may not output considerations for the counter when the confidence level is below a threshold, and may output considerations for the counter when the confidence level is equal to or greater than the threshold. The large-scale language model may output considerations for the counter regardless of the confidence level.

[0097] The inquiry answering system 1 of the third modification includes a consideration matter output unit 107. The consideration matter output unit 107 outputs the consideration matters to the counter. The output to the counter includes not only audio output but also data output (data transmission). For example, the consideration matter output unit 107 may output the consideration matters to the counter terminal by audio or by text. The counter terminal may be a terminal operated by an operator, or may be a terminal operated by someone other than the operator. As shown in the lower part of Figure 8, the consideration matter output unit 107 may cause the counter terminal to display a screen indicating the consideration matters. The consideration matter output unit 107 causes the counter terminal to display the screen by transmitting data indicating the consideration matters to the counter terminal.

[0098] The inquiry answering system 1 of Variation 3 further acquires considerations that should be taken into account by the contact point and that are generated by the large-scale language model. The inquiry answering system 1 outputs the considerations to the contact point. This allows the inquiry answering system 1 to output useful information to the contact point, thereby supporting the work of the contact point. The inquiry answering system 1 of Variation 3 may also transmit the considerations to another large-scale language model instead of the contact point. Specifically, the inquiry answering system 1 of Variation 3 may include the considerations in input to another large-scale language model as the contact point estimated by the large-scale language model. Note that the response from the large-scale language model may include a standard phrase such as "We will hand over to the next operator." The inquiry answering system 1 of Variation 3 may also reflect the considerations in another input to the large-scale language model instead of the contact point.

[0099] [6-4. Variation 4] For example, in the embodiment, an example is given of a case where a contact point for responding to inquiries from users corresponds to a category. The category indicated by the category information is not limited to the example in the embodiment. In the fourth modification, an example is given of a case where a category is a category of FAQs (Frequently Asked Questions) in insurance services. FAQs are combinations of questions frequently received in insurance services and answers to those questions. In the fourth modification, an example is given of a case where each combination corresponds to a category, but multiple combinations may belong to one category. The data storage unit 100 in the fourth modification stores an FAQ database in which multiple combinations are stored.

[0100] FIG. 9 is a diagram showing an example of an FAQ database. As shown in FIG. 9, the FAQ database associates a search index with FAQ information, which is actual data of the FAQ. The index is information referenced in a search in Variation 5, which will be described later. For example, the index is a search keyword or a vector indicating the content of the FAQ. Variation 4 takes an example in which no particular search is performed, so the FAQ database does not need to store an index. The FAQ information may be in any data format. For example, the FAQ information may be in CSV format, text format, markup language format, document format, image format, video format, or other format. The FAQ database is created by an administrator of the insurance service. The classification information acquisition unit 102 in Variation 4 acquires the actual data of each FAQ stored in the FAQ database as classification information.

[0101] FIG. 10 is a diagram showing an example of input and output of a large-scale language model in Modification 4. The model answer acquisition unit 103 in Modification 4 acquires, as a model answer, an answer in an FAQ corresponding to the classification to which the inquiry belongs. As shown in FIG. 10, the instruction statement input to the large-scale language model includes a sentence indicating that an appropriate classification is to be estimated as the classification of the FAQ indicated by the classification information. For example, the instruction statement may be a sentence such as, "You are an AI that estimates an appropriate FAQ to respond to an inquiry from a user. Based on the inquiry information and classification information input to you, estimate an appropriate FAQ, output a model answer, and calculate a confidence level." The instruction statement may be any sentence that instructs the large-scale language model to estimate an appropriate FAQ and calculate a confidence level, and is not limited to the above example. The instruction statement may include a wording that instructs the large-scale language model to estimate an appropriate FAQ and calculate a confidence level.

[0102] The large-scale language model of Variation 4 calculates an embedded representation of the instruction sentence and the inquiry information and classification information input thereto based on the trained parameters. The large-scale language model predicts the continuation as necessary based on the calculated embedded representation, and outputs a model answer indicating an FAQ that is estimated to be appropriate. If each FAQ corresponds to a classification, the large-scale language model may output a confidence level for each FAQ. The model answer acquisition unit 103 acquires the model answer output from the large-scale language model.

[0103] As shown in FIG. 10 , the model answer indicates an FAQ that the large-scale language model has estimated to be appropriate. The model answer may indicate the content of the FAQ as is, or may indicate the content of the FAQ edited by the large-scale language model. If the confidence level is below a threshold, the output control unit 104 may output an additional question to increase the confidence level, as in the embodiment. If the number of repetitions reaches a predetermined number, the output control unit 104 may forward a call from the user to an operator or forward an inquiry from the user via a tool such as chat to an operator, so that the call can be handled manually.

[0104] The inquiry answering system 1 of the fourth modification acquires, as a model answer, an answer in an FAQ according to the category to which the inquiry belongs. This allows the inquiry answering system 1 to output a more appropriate FAQ as a model answer. The inquiry answering system 1 can output information to the user to solve the user's inquiry.

[0105] [6-5. Variation 5] For example, in Modification 4, if a large number of FAQs are stored in the FAQ database, there is a possibility that the large number of FAQs will be input to the large-scale language model as classification information. In this case, the large-scale language model will not be able to recognize the large number of FAQs, and the estimation accuracy of the large-scale language model may decrease. Therefore, the classification information acquisition unit 102 of Modification 5 acquires classification information by searching the FAQ database that stores FAQ information related to the FAQ, based on the inquiry information. The processing described in Modification 5 is a processing that utilizes a technique called RAG.

[0106] For example, the classification information acquisition unit 102 searches the FAQ database based on a predetermined search algorithm using the inquiry information as a query. The predetermined search algorithm may be the same as an algorithm used in a known search engine. The classification information acquisition unit 102 acquires at least one FAQ found in the search as classification information. For example, when the degree of match between the query and the index is calculated as a score, the classification information acquisition unit 102 may acquire all FAQs with scores equal to or higher than a threshold as classification information, or may acquire FAQs with high scores up to a predetermined ranking as classification information. Note that the inquiry answering system 1 of Variation 5 may generate a follow-up question, acquire a user's answer to the follow-up question, and then search the FAQ database, so that the inquiry information can be treated as a query. Here, the inquiry answering system 1 may generate follow-up questions and acquire user answers to the follow-up questions based on the classification information input to the large-scale language model until the text included in the inquiry information corresponds to a category included in the classification information. Furthermore, the inquiry answering system 1 may generate a follow-up question and acquire a user's answer to the follow-up question in order to clarify whether the text included in the inquiry information corresponds to any of multiple classifications included in the classification information that may be confused. Note that the inquiry answering system 1 of Variation 5 may correct, convert, or format the inquiry information as a query based at least on a conversation history including the user's answer to the follow-up question generated by the large-scale language model so that the query is appropriate as a search term for a semantic search of an FAQ database. In this case, the inquiry information may be corrected, converted, or formatted as a query based on, for example, a dedicated large-scale language model and its input.

[0107] The model answer acquisition unit 103 of the fifth modification inputs, as classification information, FAQs acquired by a search by the classification information acquisition unit 102 from among FAQs stored in the FAQ database, to the large-scale language model. This modification differs from the fourth modification in that the classification information input to the large-scale language model is the FAQ acquired by a search by the classification information acquisition unit 102, but the processing executed by the large-scale language model and the acquisition of output from the large-scale language model by the model answer acquisition unit 103 are the same as those in the fourth modification.

[0108] The inquiry answering system 1 of Variation 5 acquires classification information by searching an FAQ database that stores FAQ information related to FAQs, based on inquiry information. As a result, the inquiry answering system 1 inputs more appropriate classification information into the large-scale language model, thereby improving the estimation accuracy of the large-scale language model. The inquiry answering system 1 can output to the user information appropriate to the user's inquiry. Furthermore, by appropriately narrowing down the classification information, the inquiry answering system 1 of Variation 5 can improve the quality of responses to users in voice calls where it is difficult to list the classification information. Note that, when appropriately narrowing down the classification information, the inquiry answering system 1 of Variation 5 may perform text processing to generate a summary of the output to the user that is easy for the user to recognize, and include the summary in the output to the user.

[0109] [6-6. Variation 6] For example, the user may make a new input while the model answer and the confidence factor are being acquired, or while the model answer is being output. In the case of a telephone call as in the embodiment, the new input is an utterance by the user. In the case of a message exchange such as a chat, the new input is a message entered by the user. If the processing on the inquiry answering system 1 side continues despite the user making a new input, the user may be confused. Therefore, in Variation 6, an example is given of a case where the acquisition of the model answer and the confidence factor, or the output of the model answer, is temporarily suspended when the user makes a new input.

[0110] The inquiry answering system 1 of the sixth modification includes an input determination unit 108 and an output restriction unit 109. The input determination unit 108 determines whether new input from the user has been received while the model answer and confidence level are being acquired or while the model answer is being output. The period from when the model answer and confidence level are being acquired by the model answer processing unit corresponds to the period from when the model answer processing unit starts processing until when the processing ends, corresponding to the period from when the model answer processing unit starts processing until when the processing ends, corresponding to the period from when the model answer is being output.

[0111] For example, in the case of a telephone call as in the embodiment, the input determination unit 108 determines whether a new utterance from the user has been accepted while the model answer and the confidence factor are being acquired or while the model answer is being output. The input determination unit 108 acquires voice information indicating the content of the call with the user, and determines whether a new utterance from the user has been accepted based on the voice information. In the case of an exchange of messages such as a chat, the input determination unit 108 determines whether a new message input from the user has been accepted while the model answer and the confidence factor are being acquired or while the model answer is being output. The input determination unit 108 acquires message input information indicating a message input by the user from the user terminal 20, and determines whether a new message input from the user has been accepted based on the message input information.

[0112] The output limiting unit 109 of the sixth modification limits the output of a model answer when it is determined that a new input has been received during acquisition or output. Limiting the output of a model answer means stopping the processing for outputting the model answer. For example, the output limiting unit 109 stops the processing of the model answer acquisition unit 103 when a model answer and a confidence factor are being acquired. The output limiting unit 109 stops the processing of the output control unit 104 when a model answer is being output. Stopping these processes may be achieved by program code that indicates the stopping of a process currently being executed.

[0113] For example, when a new input is made by the user, the model response acquisition unit 103 acquires new input information indicating the new input. The new input information differs from the inquiry information in that it is a new input rather than an initial inquiry, but is otherwise similar to the inquiry information. The model response acquisition unit 103 may cause the large-scale language model to estimate whether the new input information and the content of the conversation up to that point share a similar topic. The large-scale language model may output a score indicating whether they share a similar topic.

[0114] For example, the output control unit 104 may control the output to the user based on the score. For example, when the score is equal to or greater than a threshold, the output control unit 104 may control the output to the user so that the conversation continues taking into account the interrupted conversation, such as by saying, "As for your question, I will continue answering." When the score is less than the threshold, the output control unit 104 may control the output to the user so that the output corresponds to the new input rather than the interrupted conversation, such as by saying, "As for your question, I will continue answering with the current content." In this case, the output control unit 104 may resume the original output when the output corresponding to the new input is completed.

[0115] The inquiry answering system 1 of Variation 6 determines whether new input has been received from the user while acquiring a model answer and a confidence factor, or while outputting a model answer. If it is determined that new input has been received during acquisition or output, the inquiry answering system 1 restricts the output of the model answer. This prevents the inquiry answering system 1 from continuing processing despite the user having made a new input, which can confuse the user.

[0116] [6-7. Variation 7] For example, depending on the content of a user's inquiry, it may take time to process a large-scale language model. In such a case, if no response is given to the user, the user may become confused. Therefore, in the seventh modification, the time required to give a response to the user (for example, a model answer or a follow-up question) continues, and if the time required exceeds a threshold, a response such as "We are looking into it, so please wait a moment" is output.

[0117] The inquiry answering system 1 of the seventh modification includes a time determination unit 110 and an other answer output unit 111. The time determination unit 110 determines whether or not it takes time to acquire a model answer and a confidence factor. For example, the time determination unit 110 measures the time since the model answer processing unit started processing as the required time. The time measurement method may be measurement using a code adopted in a known programming language. The time determination unit 110 determines whether or not the required time has reached a threshold value without a response to the user being output. If a response to the user is output before the required time has reached the threshold value, the time determination unit 110 determines that it does not take time to acquire a model answer and a confidence factor. If it is determined that the required time has reached the threshold value without a response to the user being output, the time determination unit 110 determines that it will take time to acquire a model answer and a confidence factor.

[0118] The determination method of the time determination unit 110 is not limited to the above example. For example, the time determination unit 110 may determine whether the required time has reached a threshold or more without an output from the large-scale language model being acquired. If an output from the large-scale language model is acquired before the required time has reached the threshold or more, the time determination unit 110 determines that no time will be required to acquire the model answer and the confidence factor. If the time determination unit 110 determines that the required time has reached the threshold or more without an output from the large-scale language model being acquired, the time determination unit 110 determines that time will be required to acquire the model answer and the confidence factor.

[0119] Alternatively, the time determination unit 110 may use a large-scale language model to estimate the time required to output the model answer and the confidence level, rather than measuring the time. The time determination unit 110 determines whether the required time estimated by the large-scale language model is equal to or greater than a threshold. If the time determination unit 110 determines that the required time estimated by the large-scale language model is less than the threshold, it determines that no time will be required to obtain the model answer and the confidence level. If the time determination unit 110 determines that the required time estimated by the large-scale language model is equal to or greater than the threshold, it determines that time will be required to obtain the model answer and the confidence level.

[0120] When it is determined that it will take time to acquire the model answer and the confidence factor, the other answer output unit 111 outputs to the user another answer that is different from the model answer. In the seventh modification, a case where the other answer is a predetermined answer is exemplified, but the other answer may also be generated by a large-scale language model. When the other answer is a predetermined answer, other answer information indicating the other answer is assumed to be stored in the data storage unit 100. The other answer output unit 111 outputs the other answer to the user based on the other answer information. The output of the other answer to the user includes not only audio output but also data output (transmission of data). For example, the other answer output unit 111 may output the other answer to the user terminal 20 by audio or text.

[0121] The inquiry answering system 1 of Variation 7 determines whether it will take time to acquire the model answer and the confidence level. If it is determined that it will take time to acquire the model answer and the confidence level, the inquiry answering system 1 outputs an answer other than the model answer to the user. This prevents the user from being confused by not receiving any response, thereby improving user convenience.

[0122] [6-8. Variation 8] For example, when the amount of model answers generated by the large-scale language model is large, the amount of information output to the user is large. Therefore, when the amount of model answers is equal to or greater than a threshold, the output control unit 104 may cause the large-scale language model to generate a summary of the model answers and output the summary to the user. In the eighth modification, an example is given in which the model answer is an answer indicating an appropriate contact point as described in the embodiment, but the model answer may also be an answer in the FAQ as described in the fourth modification.

[0123] The quantity of the model answer is the length of the model answer. For example, if the model answer is expressed in text, the quantity of the model answer is the number of characters, the number of lines, the number of paragraphs, or the number of pages. If the model answer is expressed in audio, the quantity of the model answer is the length of the audio playback time. If the model answer is expressed in a format other than text or audio, the quantity of the model answer is the data volume of the model answer.

[0124] For example, the instruction sentence input to the large-scale language model in Variation 8 includes a sentence indicating that a summary of the model answer should be generated if the amount of the model answer generated by the large-scale language model is equal to or greater than a threshold. For example, the instruction sentence includes a sentence such as, "If the amount of the model answer you generated is equal to or greater than a threshold, please output a summary of the model answer." The instruction sentence may be any sentence that instructs the generation of a summary of the model answer, and is not limited to the above example. The instruction sentence may include wording that instructs the generation of a summary of the model answer. Note that the large-scale language model of Variation 8 may include, in a subsequent model answer, content of the model answer that is not included in the summary of the model answer, in response to a request from the user in response to "Are there any other questions?" attached to the summary of the model answer.

[0125] For example, when the quantity of the model answer is less than a threshold, the output control unit 104 controls the output of the model answer based on the confidence level. When the quantity of the model answer is less than the threshold and the confidence level is equal to or greater than a threshold, the output control unit 104 controls the output of the model answer to the user. When the quantity of the model answer is equal to or greater than a threshold and the confidence level is equal to or greater than a threshold, the output control unit 104 outputs a summary of the model answer to the user. When the confidence level is less than a threshold, the same processing as in the embodiment may be performed.

[0126] The inquiry answering system 1 of Variation 8 generates a summary of the model answer in the large-scale language model when the volume of the model answer is equal to or greater than a threshold, and outputs the summary to the user. This prevents redundant model answers from being output to the user, thereby improving user convenience. Instead of checking the volume of the model answer, the inquiry answering system 1 of Variation 8 may check whether the model answer deviates from the flow of interaction (context) with the user by separately providing an instruction statement to the large-scale language model. The inquiry answering system 1 of Variation 8 may also check whether hallucination has occurred by separately providing an instruction statement to the large-scale language model. The instruction statement is intended to check whether the output from the large-scale language model contains text or terms not included in the FAQ database. If the inquiry answering system 1 of Variation 8 confirms that the deviation or hallucination has occurred, it may generate a response using the large-scale language model again. At this time, the inquiry answering system 1 of Variation 8 may, for example, call an escalation to a human, or a large-scale language model for escalation and its instructions.

[0127] [6-9. Other variations] For example, the above modifications may be combined.

[0128] For example, the functions described as being realized by the server 10 may be realized by the user terminal 20 or another computer. The functions described as being realized by the server 10 may be shared among multiple computers.

[0129] [7. Notes] For example, the inquiry response system 1 can also be configured as follows.

[0130] (1) an inquiry information acquisition unit that acquires inquiry information regarding an inquiry from a user regarding a predetermined service; a classification information acquisition unit that acquires classification information relating to the inquiry, the classification being predetermined in the service; a model answer acquisition unit that inputs the query information and the classification information into a large-scale language model and acquires a model answer regarding the classification generated by the large-scale language model and a confidence level of the classification; an output control unit that controls output of the model answer based on the confidence level; A query response system including. (2) The output control unit If the confidence is equal to or greater than a threshold, output the model answer to the user; If the confidence level is less than the threshold, obtaining a follow-up question for the user generated by the large-scale language model, and outputting the follow-up question to the user; the inquiry answering system further includes a user answer information acquiring unit that acquires user answer information regarding a user answer from the user to the additional question; the model answer acquisition unit inputs the user answer information into the large-scale language model, and again acquires the model answer and the confidence factor; the output control unit controls output of the reacquired model answer based on the reacquired confidence factor. (1) An inquiry response system according to the present invention. (3) the output control unit acquires the follow-up question based on an instruction statement indicating that the follow-up question for increasing the confidence level is to be generated when the confidence level is less than the threshold value; (2) The inquiry response system described above. (4) the inquiry answering system repeats the steps of acquiring and outputting the follow-up question, acquiring the user answer information, and acquiring the model answer and the confidence level until the confidence level becomes equal to or greater than the threshold value; the output control unit outputs an answer prepared in advance in the service to the user if the confidence level does not reach or exceed the threshold level even after the acquisition and output of the additional question, the acquisition of the user answer information, and the acquisition of the model answer and the confidence level have been repeated a predetermined number of times. An inquiry response system according to (2) or (3). (5) the inquiry answering system further includes a number determination unit that determines the predetermined number of times based on the confidence level. (4) An inquiry response system as described above. (6) the model answer acquisition unit further inputs a history related to the inquiry information into the large-scale language model, and again acquires the model answer and the confidence factor. An inquiry response system according to any one of (2) to (5). (7) The classification is a classification regarding a contact point for responding to the inquiry, the model answer acquisition unit acquires, as the model answer, an answer regarding the counter according to the classification to which the inquiry belongs; An inquiry response system according to any one of (1) to (6). (8) The model response acquisition unit further acquires considerations that should be taken into account by the contact point and that are generated by the large-scale language model; The inquiry response system further includes a consideration matter output unit that outputs the consideration matters to the contact point. (7) The inquiry response system described in (7). (9) the classification is a classification of FAQs (Frequently Asked Questions) in the service, the model answer acquisition unit acquires, as the model answer, an answer in the FAQ corresponding to the category to which the inquiry belongs; An inquiry response system according to any one of (1) to (8). (10) the classification information acquisition unit acquires the classification information by searching an FAQ database in which FAQ information related to the FAQ is stored, based on the inquiry information. (9) An inquiry response system according to (9). (11) The inquiry response system comprises: an input determination unit that determines whether a new input from the user has been received while the model answer and the confidence factor are being acquired or while the model answer is being output; an output limiting unit that limits the output of the model answer when it is determined that the new input has been received during the acquisition or the output; The inquiry response system according to any one of (1) to (10), further comprising: (12) The inquiry response system comprises: a time determination unit that determines whether or not time is required to acquire the model answer and the confidence factor; an alternative answer output unit that outputs an alternative answer different from the model answer to the user when it is determined that it will take time to acquire the model answer and the certainty factor; The inquiry response system according to any one of (1) to (11), further comprising: (13) the output control unit, when the amount of the model answer is equal to or greater than a threshold, causes the large-scale language model to generate a summary of the model answer and outputs the summary to the user. An inquiry response system according to any one of (1) to (12). [Explanation of symbols]

[0131] 1 Inquiry response system, N network, 10 server, 11, 21 control unit, 12, 22 memory unit, 13, 23 communication unit, 20 user terminal, 24 operation unit, 25 display unit, 100 data storage unit, 101 information acquisition unit, 102 classification information acquisition unit, 103 model response acquisition unit, 104 output control unit, 105 user response information acquisition unit, 106 number of times determination unit, 107 consideration item output unit, 108 input judgment unit, 109 output restriction unit, 110 time judgment unit, 111 other response output unit.

Claims

1. an inquiry information acquisition unit that acquires inquiry information regarding an inquiry from a user regarding a predetermined service; a classification information acquisition unit that acquires classification information relating to the inquiry, the classification being predetermined in the service; a model answer acquisition unit that inputs the query information and the classification information into a large-scale language model and acquires a model answer regarding the classification generated by the large-scale language model and a confidence level of the classification; an output control unit that controls output of the model answer based on the confidence level; A query response system including.

2. The output control unit If the confidence is equal to or greater than a threshold, output the model answer to the user; If the confidence level is less than the threshold, obtaining a follow-up question for the user generated by the large-scale language model, and outputting the follow-up question to the user; the inquiry answering system further includes a user answer information acquiring unit that acquires user answer information regarding a user answer from the user to the additional question; the model answer acquisition unit inputs the user answer information into the large-scale language model, and again acquires the model answer and the confidence factor; the output control unit controls output of the reacquired model answer based on the reacquired confidence factor. The inquiry response system according to claim 1 .

3. the output control unit acquires the follow-up question based on an instruction statement indicating that the follow-up question for increasing the confidence level is to be generated when the confidence level is less than the threshold value; The inquiry response system according to claim 2.

4. the inquiry answering system repeats the steps of acquiring and outputting the follow-up question, acquiring the user answer information, and acquiring the model answer and the confidence level until the confidence level becomes equal to or greater than the threshold value; the output control unit outputs an answer prepared in advance in the service to the user if the confidence level does not reach or exceed the threshold level even after the acquisition and output of the additional question, the acquisition of the user answer information, and the acquisition of the model answer and the confidence level have been repeated a predetermined number of times.

4. The inquiry response system according to claim 2 or 3.

5. the inquiry answering system further includes a number determination unit that determines the predetermined number of times based on the confidence level.

5. The inquiry response system according to claim 4.

6. the model answer acquisition unit further inputs a history related to the inquiry information into the large-scale language model, and again acquires the model answer and the confidence factor.

4. The inquiry response system according to claim 2 or 3.

7. The classification is a classification regarding a contact point for responding to the inquiry, the model answer acquisition unit acquires, as the model answer, an answer regarding the counter according to the classification to which the inquiry belongs; 4. An inquiry response system according to claim 1.

8. The model response acquisition unit further acquires considerations that should be taken into account by the contact point and that are generated by the large-scale language model; The inquiry response system further includes a consideration matter output unit that outputs the consideration matters to the contact point. The inquiry response system according to claim 7.

9. the classification is a classification of FAQs (Frequently Asked Questions) in the service, the model answer acquisition unit acquires, as the model answer, an answer in the FAQ corresponding to the classification to which the inquiry belongs; 4. An inquiry response system according to claim 1.

10. The classification information acquisition unit acquires the classification information by searching an FAQ database in which FAQ information related to the FAQ is stored, based on the inquiry information. The inquiry response system according to claim 9.

11. The inquiry response system comprises: an input determination unit that determines whether a new input from the user has been received while the model answer and the confidence factor are being acquired or while the model answer is being output; an output limiting unit that limits the output of the model answer when it is determined that the new input has been received during the acquisition or the output; 4. The inquiry response system according to claim 1, further comprising:

12. The inquiry response system comprises: a time determination unit that determines whether or not time is required to acquire the model answer and the confidence factor; an alternative answer output unit that outputs an alternative answer different from the model answer to the user when it is determined that it will take time to acquire the model answer and the certainty factor; 4. The inquiry response system according to claim 1, further comprising:

13. the output control unit, when the amount of the model answer is equal to or greater than a threshold, causes the large-scale language model to generate a summary of the model answer and outputs the summary to the user.

4. An inquiry response system according to claim 1.

14. an inquiry information acquisition step of acquiring inquiry information relating to an inquiry from a user regarding a predetermined service; a classification information acquisition step of acquiring classification information relating to the inquiry, the classification being predetermined in the service; a model answer acquisition step of inputting the query information and the classification information into a large-scale language model and acquiring a model answer regarding the classification generated by the large-scale language model and a confidence level of the classification; an output control step of controlling an output of the model answer based on the confidence level; How to answer inquiries, including:

15. an inquiry information acquisition unit that acquires inquiry information regarding an inquiry from a user regarding a predetermined service; a classification information acquisition unit that acquires classification information regarding the classification of the inquiry, the classification being predetermined in the service; a model answer acquisition unit that inputs the query information and the classification information into a large-scale language model and acquires a model answer regarding the classification generated by the large-scale language model and a confidence level of the classification; an output control unit that controls output of the model answer based on the confidence level; A program that allows a computer to function as a

Images