Program, method, information processing apparatus, and system

Abstract

To provide a program, a method, an information processing apparatus, and a system for outputting an answer (response) appropriate for a question.SOLUTION: A program executes: a question input step of inputting a question text received from a user terminal to a question answering system capable of outputting an answer text in accordance with the question test; an answer acquisition step of acquiring one or more answer texts according to the question text and degrees of association associated with each of the one or more answer texts; an answer transmission step of transmitting, when the degrees of association associated with the one or more answer texts satisfy a predetermined condition, the one or more answer texts satisfying the predetermined condition to the user terminal; and a response transmission step of transmitting, when the degrees of association associated with the one or more answer texts do not satisfy the predetermined condition, a response text to the user terminal, the response text being output by inputting the question text received in a question receiving step, as input data, to a large language model.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present disclosure relates to a program, a method, an information processing device, and a system. [Background technology]

[0002] 2. Description of the Related Art Systems (chatbot systems) that respond to online inquiries such as questions from users and output answers corresponding to those questions are known. Patent Document 1 discloses a technology that enables a chatbot to automatically respond to inquiries from users and, when transferring over to a human responder, seamlessly hand over the response to the respondent. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-189237 A Summary of the Invention [Problem to be solved by the invention]

[0004] There is an issue with the system not being able to output appropriate answers to questions. Therefore, the present disclosure has been made to solve the above problem, and has an objective to provide a technique for outputting an appropriate answer (response) to a question. [Means for solving the problem]

[0005] A program to be executed by a computer having a processor and a memory unit, the program executing the following steps: a question receiving step in which the processor receives a question from a user terminal; a question input step in which the question received in the question receiving step is input to a question answering system capable of outputting an answer in response to the input question; an answer acquisition step in which one or more answer sentences corresponding to the question input in the question input step and a relevance degree associated with each of the one or more answer sentences; an answer sending step in which, if the relevance degree associated with the one or more answer sentences acquired in the answer acquisition step satisfies a predetermined condition, the one or more answer sentences that satisfy a predetermined condition are sent to the user terminal; and, if the relevance degree associated with the one or more answer sentences acquired in the answer acquisition step does not satisfy the predetermined condition, the program executing the program. Effect of the Invention

[0006] According to the present disclosure, an appropriate answer (response) to a question can be output. [Brief description of the drawings]

[0007] [Figure 1] FIG. 2 is a block diagram showing the functional configuration of the system 1. [Diagram 2] FIG. 2 is a block diagram showing the functional configuration of the server 10. [Diagram 3] 2 is a block diagram showing the functional configuration of a user terminal 20. FIG. [Figure 4] 2 is a block diagram showing a functional configuration of an administrator terminal 30. FIG. [Diagram 5] FIG. 13 is a diagram showing the data structure of a user table 1012. [Figure 6] 13 is a diagram showing the data structure of a question and answer table 1013. FIG. [Figure 7] 13 is a flowchart showing an operation of a question process. [Figure 8]13 is a flowchart showing the operation of a management process. [Figure 9] 11 is a screen example showing a question processing operation. [Figure 10] 11 is a screen example showing the operation of a management process. [Figure 11] FIG. 2 is a block diagram showing the basic hardware configuration of a computer 90. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. In all the drawings explaining the embodiment, the same reference numerals are given to common components, and repeated explanations are omitted. Note that the following embodiment does not unduly limit the contents of the present disclosure described in the claims. In addition, not all of the components shown in the embodiment are essential components of the present disclosure. In addition, each figure is a schematic diagram and is not necessarily illustrated strictly.

[0009] <System 1 Configuration> The system 1 in the present disclosure is an information processing system that provides an information processing service (FAQ service) that outputs suitable answers to questions input by a user. For example, in a bank or the like, the system provides an information processing service that receives questions from users regarding account opening, domestic and international remittances, loan applications, ATMs, how to use cash cards, how to use mobile banking, etc., and provides the users with answers to solve their doubts and problems related to the questions. The application of the information processing service according to the present disclosure is not limited to banks, but can be applied to any business entity, such as public transportation such as railways, airplanes, and taxis, accommodation facilities such as hotels and guesthouses, software services, hospitals, etc. The system 1 includes an information processing device of a server 10, a user terminal 20, an administrator terminal 30, a generating AI 50, and a question and answer system 60, which are connected via a network N. FIG. 1 is a block diagram showing the functional configuration of the system 1. FIG. 2 is a block diagram showing the functional configuration of the server 10. As shown in FIG. FIG. 3 is a block diagram showing the functional configuration of the user terminal 20. As shown in FIG. FIG. 4 is a block diagram showing the functional configuration of the administrator terminal 30. As shown in FIG.

[0010] Each information processing device is configured by a computer equipped with an arithmetic unit and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. For each of the server 10, the user terminal 20, the administrator terminal 30, the generation AI 50, and the question and answer system 60, descriptions that overlap with the basic hardware configuration and basic functional configuration of the computer described later will be omitted.

[0011] <Server 10 Configuration> The server 10 is an information processing device that provides an information processing service (FAQ service) that outputs suitable answers to questions input by users by communicating with a generation AI 50, a question and answer system 60, and the like, which will be described later. The server 10 includes a storage unit 101 and a control unit 104 .

[0012] <Configuration of the storage unit 101 of the server 10> The storage unit 101 of the server 10 includes an application program 1011 , a user table 1012 , and a question and answer table 1013 .

[0013] The application program 1011 is a program for causing the control unit 104 of the server 10 to function as each functional unit. Application programs 1011 include applications such as a web browser application.

[0014] User table 1012 is a table that stores and manages information about member users (hereinafter, users) who use the service. When a user registers to use the service, the user's information is stored in a new record in user table 1012. This allows the user to use the service according to the present disclosure. The user table 1012 is a table having columns of user ID and user name, with the user ID as the primary key. FIG. 5 is a diagram showing the data structure of the user table 1012. As shown in FIG.

[0015] The user ID is an item for storing user identification information for identifying a user. The user identification information is an item for which a unique value is set for each user. The user name is an item for storing the name of the user. The user name may be set to any character string such as a nickname instead of a name.

[0016] The question and answer table 1013 is a table for storing and managing information relating to questions (question information). The question and answer table 1013 is a table having a question ID as a primary key, and columns of a question ID, a parent question ID, a user ID, a question text, answer data, a reply text, and a date and time. FIG. 6 is a diagram showing the data structure of the question and answer table 1013. As shown in FIG.

[0017] The question ID is an item for storing question identification information for identifying a question. The question identification information is an item for which a unique value is set for each piece of question information. The parent question ID is an item for storing parent question identification information for identifying a parent question. The user ID is an item for storing user identification information for identifying a user. The question sentence is an item for storing character string information (text information) related to the question sentence. The answer data is an item that stores character string information (text information) related to an output sentence (answer sentence) output from the question answering system 60. The answer data can store one or more answer sentences in association with an index indicating the degree of association between the one or more answer sentences and the question sentence associated with each of the one or more answer sentences. The response sentence is an item that stores character string information (text information) related to the output sentence (response sentence) output from the generation AI 50. The date and time is an item for storing the date and time when a record is newly stored or updated in the question and answer table 1013.

[0018] <Configuration of the control unit 104 of the server 10> The control unit 104 of the server 10 includes a user registration control unit 1041. The control unit 104 executes an application program 1011 stored in the storage unit 101, thereby realizing each functional unit.

[0019] The user registration control unit 1041 performs processing to store, in the user table 1012, information on users who wish to use the service according to the present disclosure. The information stored in the user table 1012 is generated when a user opens a web page operated by a service provider from any information processing terminal, enters information into a specific input form, and transmits the information to the server 10. The user registration control unit 1041 stores the received information in a new record in the user table 1012, completing the user registration. This allows the user stored in the user table 1012 to use the service. Before the user registration control unit 1041 registers user information in the user table 1012, the service provider may carry out a predetermined examination to restrict whether or not the user is permitted to use the service. The user ID may be any character string or number that can identify the user, any character string or number desired by the user, or an arbitrary character string or number may be automatically set by the user registration control unit 1041.

[0020] <Configuration of User Terminal 20> The user terminal 20 is an information processing device operated by a user who uses a service. The user terminal 20 may be, for example, a mobile terminal such as a smartphone or a tablet, a stationary PC (Personal Computer), or a laptop PC. It may also be a wearable terminal such as an HMD (Head Mount Display) or a wristwatch-type terminal. The user terminal 20 includes a storage unit 201 , a control unit 204 , an input device 206 , and an output device 208 .

[0021] <Configuration of the storage unit 201 of the user terminal 20> The storage unit 201 of the user terminal 20 includes a user ID 2011 and an application program 2012 .

[0022] The user ID 2011 is an account ID of the user. The user transmits the user ID 2011 from the user terminal 20 to the server 10. The server 10 identifies the user based on the user ID 2011 and provides the user with the service according to the present disclosure. The user ID 2011 includes information such as a session ID temporarily assigned by the server 10 when identifying the user who is using the user terminal 20.

[0023] The application program 2012 may be stored in advance in the storage unit 201, or may be configured to be downloaded from a web server operated by a service provider via a communication IF. The application programs 2012 include applications such as a web browser application. The application program 2012 includes an interpreted programming language such as JavaScript™ that runs on a web browser application stored on the user terminal 20 .

[0024] <Configuration of the control unit 204 of the user terminal 20> The control unit 204 of the user terminal 20 includes an input control unit 2041 and an output control unit 2042. The control unit 204 executes an application program 2012 stored in the storage unit 201, thereby realizing each functional unit.

[0025] <Configuration of the input device 206 of the user terminal 20> The input device 206 of the user terminal 20 includes a camera 2061 , a microphone 2062 , a position information sensor 2063 , a motion sensor 2064 , and a touch device 2065 .

[0026] <Configuration of the output device 208 of the user terminal 20> The output device 208 of the user terminal 20 includes a display 2081 and a speaker 2082 .

[0027] <Configuration of administrator terminal 30> The administrator terminal 30 is an information processing device operated by a user who uses the service. The administrator is a user who has the authority to edit the question and answer information stored in the question and answer system 60. The administrator terminal 30 may be, for example, a mobile terminal such as a smartphone or a tablet, or may be a stationary PC (Personal Computer) or a laptop PC. It may also be a wearable terminal such as an HMD (Head Mount Display) or a wristwatch-type terminal. The administrator terminal 30 includes a storage unit 301 , a control unit 304 , an input device 306 , and an output device 308 .

[0028] <Configuration of the storage unit 301 of the administrator terminal 30> The storage unit 301 of the administrator terminal 30 includes an administrator ID 3011 and an application program 3012 .

[0029] The administrator ID 3011 is the account ID of the administrator. The administrator transmits the administrator ID 3011 from the administrator terminal 30 to the server 10. The server 10 identifies the administrator based on the administrator ID 3011 and provides the administrator with the services disclosed herein. The administrator ID 3011 includes information such as a session ID temporarily assigned by the server 10 to identify the administrator using the administrator terminal 30.

[0030] The application program 3012 may be stored in advance in the storage unit 201, or may be configured to be downloaded from a web server operated by a service provider via a communication IF. Application programs 3012 include applications such as a web browser application. The application program 3012 includes an interpreted programming language such as JavaScript (registered trademark) that runs on a web browser application stored on the user terminal 20 .

[0031] <Configuration of the control unit 304 of the administrator terminal 30> The control unit 304 of the administrator terminal 30 includes an input control unit 3041 and an output control unit 3042. The control unit 304 executes an application program 3012 stored in the storage unit 301, thereby realizing each functional unit.

[0032] <Configuration of the input device 306 of the administrator terminal 30> The input device 306 of the administrator terminal 30 includes a camera 3061 , a microphone 3062 , a position information sensor 3063 , a motion sensor 3064 , and a touch device 3065 .

[0033] <Configuration of the output device 308 of the administrator terminal 30> The output device 308 of the administrator terminal 30 includes a display 3081 and a speaker 3082 .

[0034] <Configuration of generated AI50> Generative AI50 refers to large-scale artificial intelligence models used in the field of natural language processing (NLP). These models learn from large amounts of text data (web pages, books, articles, etc.) to understand patterns in human language and perform natural language generation (NLG) tasks effectively. The generation AI 50 is used in many NLP tasks, such as generating responses to specific questions, automatically generating sentences, summarizing text, translation, and sentiment analysis. It can also be used for a variety of purposes, such as education, entertainment, customer service, and product development. The generation AI 50 includes the following types. In this disclosure, a large-scale language model that mainly outputs text information as output information will be described as a type of generation AI 50. OpenAI ChatGPT ·Google Bard Stable Diffusion Midjourney The generation AI 50 may be realized as a part of the functions of the server 10.

[0035] <Configuration of question answering system 60> The question and answering system 60 is an information processing system that outputs a character string related to an answer in response to an input of a character string related to a question. For example, the question and answering system 60 includes an FAQ system. Specifically, the question and answering system 60 is an information processing system that stores pairs of frequently asked questions from users and their answers (question and answer information). When the question and answering system 60 receives an input of an input character string, it searches for at least one of the questions or answers included in the question and answer information based on the input character string by full-text search or the like, and outputs the question and answer information related to the search result in association with the relevance (score) to the input character string. By inputting a character string such as a search keyword or a question sentence into question answering system 60, it is possible to obtain an output of question answering information related to the character string. There does not need to be just one question answering system 60, and for example, a configuration may be adopted in which a question answering system A storing question answering information for bank A, a plurality of question answering systems 60 are provided for each business operator providing a service related to questions, such as bank B, bank C, accommodation facility A, accommodation facility B, etc. The question and answer system 60 includes an information processing device such as a full-text search AI, a full-text search system, and an AI chatbot (classification model). The question and answer system 60 may be realized as a part of the functions of the server 10.

[0036] <System 1 Operation> Each process of the system 1 will be described below. FIG. 7 is a flowchart showing the operation of the question processing. FIG. 8 is a flowchart showing the operation of the management process. FIG. 9 is a screen example showing the operation of question processing. FIG. 10 is a screen example showing the operation of the management process.

[0037] <Question Processing> The question process is a process of presenting to the user an output sentence from the question and answering system 60 based on a search keyword corresponding to a question received from the user, or an output sentence from the generation AI 50.

[0038] <Question processing overview> Question processing is a series of processes that accepts input of a question from a user, inputs the input question into question answering system 60, and based on the relevance between the answer sentence output from question answering system 60 and the answer sentence, if the relevance satisfies a specified condition, presents the answer sentence to the user, and if the relevance does not satisfy the specified condition, inputs the question sentence and the answer sentence into generation AI as input data, and presents the reply sentence output from generation AI 50 to the user.

[0039] <Question processing details> The question processing will be described in detail below. The user operates the input device 206 of the user terminal 20 to execute a browser application or the like, and opens the question page D1 by inputting the URL of a web page (question page) for executing a question process. The control unit 204 of the user terminal 20 transmits a request including a user ID 2011 for opening the question page to the server 10.

[0040] When the server 10 receives the request, it generates a question page and transmits it to the user terminal 20. The control unit 204 of the user terminal 20 displays the question page on the display 2081 of the user terminal 20 for presentation. 9 shows an example of a question screen in question processing. A question page D1 is displayed on the display 2081 of the user terminal 20. The question page D1 includes an input field D11, an output sentence D12 (an answer sentence or a response sentence), a question sentence D13, and a send button D14.

[0041] In step S101, the control unit 104 of the server 10 executes a question receiving step of receiving a question from a user terminal. The user operates the input device 206 of the user terminal 20 to input a question into the input field D11. The control unit 204 of the user terminal 20 transmits the question input into the input field D11 to the server 10. For example, the user can operate the input device 206 of the user terminal 20 to press the send button D14 to transmit the question input into the input field D11 to the server 10. The request transmitted to the server 10 includes the user ID 2011. The control unit 104 of the server 10 receives and accepts the question and user ID 2011. The control unit 104 of the server 10 stores the accepted question and user ID 2011 in the question and user ID fields of a new record in the question and answer table 1013, respectively. The control unit 204 of the user terminal 20 displays the question entered in the entry field D11 in question D13 on the question page D1, and presents it to the user.

[0042] In step S102, the control unit 104 of the server 10 executes a question input step of inputting the question received in the question receiving step to a question and answer system capable of outputting an answer in response to the input of the question. Specifically, the control unit 104 of the server 10 transmits the question received in step S101 to the question and answering system 60. The question and answering system 60 receives the question and accepts input.

[0043] In step S103, the control unit 104 of the server 10 executes an answer acquisition step of acquiring one or more answer sentences corresponding to the question sentence input in the question input step, and a relevance degree associated with each of the one or more answer sentences. Specifically, the control unit 104 of the server 10 acquires one or more answer sentences output from the question and answer system 60 in response to the input of the question sentence in step S102, and the relevance degree associated with each of the one or more answer sentences. The relevance is a numerical index that indicates the relevance of an answer to an input question. The relevance can be calculated based on the question and answer, using TF-IDF (Term Frequency-Inverse Document Frequency) score, cosine similarity, Euclidean distance, Jaccard coefficient, or any other machine learning model, deep learning model, artificial intelligence model, etc. In the present disclosure, question answering system 60 is a system that can output, in response to an input of a question sentence, answer sentences related to the question sentence in a form associated with the relevance to the question sentence. In particular, question answering system 60 is a system that outputs one or more answer sentences in descending order of relevance and the relevance associated with each of the one or more answer sentences.

[0044] In step S103, the control unit 104 of the server 10 executes an answer storage step of storing one or more answer sentences acquired in the answer acquisition step in association with the question sentence. Specifically, the control unit 104 of the server 10 stores one or more answer sentences and the relevance levels associated with each of the one or more answer data items in the answer data item of the record created in step S101 of the question and answer table 1013.

[0045] The answer storage step executes a step of storing a predetermined number of answer sentences in ascending order of relevance from among the one or more answer sentences acquired in the answer acquisition step. Specifically, the control unit 104 of the server 10 does not need to store all of the answer sentences or answer sentences output from the question and answer system 60, and may store a predetermined number of answer sentences in order of low or high relevance and the relevance degrees associated with the answer sentences in the answer data item of the question and answer table 1013. Furthermore, the control unit 104 of the server 10 may store answer sentences whose relevance degrees are less than or greater than a predetermined value and the relevance degrees associated with the answer sentences in the answer data item of the question and answer table 1013.

[0046] The answer storing step executes a step of storing one or more answer sentences when the relevance degree associated with the one or more answer sentences acquired in the answer acquiring step does not satisfy a predetermined condition. Specifically, the control unit 104 of the server 10 may store one or more answer sentences and the relevance associated with each of the one or more answer sentences in the answer data item of the question and answer table 1013 when at least some or all of the answer sentences output from the question and answer system 60 satisfy a predetermined condition, or when they do not satisfy a predetermined condition. Similarly, the control unit 104 of the server 10 may be configured not to store one or more answer sentences and the relevance levels associated with each of the one or more answer sentences output from the question and answer system 60 in the answer data item of the question and answer table 1013 when at least some or all of the answer sentences do not satisfy a specified condition or when they do satisfy a specified condition.

[0047] For example, the control unit 104 of the server 10 may determine that a specified condition is met when a statistical value such as the maximum value, minimum value, or average (mean, median, mode) of one or more relevance degrees output from the question and answering system 60 is greater than or equal to a specified value or less than a specified value. The control unit 104 of the server 10 may determine that a specified condition is not satisfied if a statistical value such as the maximum value, minimum value, average (mean, median, mode) of one or more relevance degrees output from the question and answering system 60 is below a specified value or above a specified value.

[0048] In addition, the control unit 104 of the server 10 may be configured to store some or all of the answer sentences, among one or more answer sentences, whose relevance meets a predetermined condition, in the answer data item of the question and answer table 1013, and not store some or all of the answer sentences, whose relevance does not meet the predetermined condition, in the answer data item of the question and answer table 1013. The control unit 104 of the server 10 may be configured to store some or all of the answer sentences, among one or more answer sentences, whose relevance level is above or below a predetermined value in the answer data item of the question and answer table 1013, and not store some or all of the answer sentences, whose relevance level is above or below a predetermined value, in the answer data item of the question and answer table 1013. Similarly, the control unit 104 of the server 10 may be configured to store some or all of the answer sentences, among one or more answer sentences, whose relevance does not satisfy a specified condition, in the answer data item of the question and answer table 1013, and not store some or all of the answer sentences, whose relevance does satisfy a specified condition, in the answer data item of the question and answer table 1013. The control unit 104 of the server 10 may be configured to store, in the answer data item of the question and answer table 1013, some or all of the answer sentences whose relevance level is equal to or less than a predetermined value among one or more answer sentences, and not to store, in the answer data item of the question and answer table 1013, some or all of the answer sentences whose relevance level is equal to or less than a predetermined value.

[0049] If the relevance level associated with one or more answer sentences acquired in the answer acquiring step does not satisfy a predetermined condition, a question storing step is executed to store the question sentence accepted in the question accepting step. Specifically, the control unit 104 of the server 10 may store the question sentence received in step S101 in the question sentence item of the question and answer table 1013 if at least some or all of the answer sentences among one or more answer sentences output from the question and answer system 60 satisfy a predetermined condition, or if they do not satisfy a predetermined condition. Similarly, the control unit 104 of the server 10 may be configured not to store the question received in step S101 in the question item of the question and answer table 1013 if at least some or all of the answer sentences among one or more answer sentences output from the question and answer system 60 do not satisfy a predetermined condition or if they do satisfy a predetermined condition.

[0050] In step S103, the control unit 104 of the server 10 determines whether the output content from the question and answering system 60 satisfies a predetermined condition based on one or more answer sentences output from the question and answering system 60 and the relevance associated with the answer sentences. Specifically, the control unit 104 of the server 10 determines that a specified condition is met when a statistical value such as the maximum value, minimum value, average (mean, median, mode) of one or more relevance degrees output from the question and answering system 60 is below a specified value or above a specified value. For example, the control unit 104 of the server 10 determines that the predetermined condition is satisfied when the maximum value of one or more relevance degrees output from the question and answering system 60 is equal to or greater than a predetermined value. Also, the control unit 104 may determine that the predetermined condition is satisfied when the relevance degree of one answer sentence output from the question and answering system 60 is equal to or greater than a predetermined value. Similarly, the control unit 104 of the server 10 determines that the predetermined condition is not satisfied when the maximum value of one or more relevance degrees output from the question and answering system 60 is less than a predetermined value. Also, when the relevance degree of one answer sentence output from the question and answering system 60 is less than a predetermined value, it may be determined that the predetermined condition is not satisfied. If the control unit 104 of the server 10 determines that the predetermined condition is satisfied, the process proceeds to step S104. On the other hand, if the control unit 104 of the server 10 determines that the predetermined condition is not satisfied, the process proceeds to step S105.

[0051] In step S104, the control unit 104 of the server 10 executes an answer sending step of sending one or more answer sentences that satisfy a specified condition to the user terminal if the relevance associated with the one or more answer sentences acquired in the answer acquisition step satisfies a specified condition. Specifically, the control unit 104 of the server 10 transmits one or more answer sentences to the user terminal 20 when at least some or all of the answer sentences output from the question and answer system 60 satisfy a predetermined condition.

[0052] In addition, the control unit 104 of the server 10 may be configured to send to the user terminal 20 some or all of the answer sentences whose relevance meets a specified condition among one or more answer sentences, and not send to the user terminal 20 some or all of the answer sentences whose relevance does not meet the specified condition. The control unit 104 of the server 10 may be configured to send to the user terminal 20 some or all of the one or more answer sentences whose relevance level is above or below a predetermined value, and not to send to the user terminal 20 some or all of the answer sentences whose relevance level is above or below a predetermined value. The control unit 104 of the server 10 may be configured to send to the user terminal 20 one answer sentence that has the highest or lowest relevance among one or more answer sentences, and not send other answer sentences to the user terminal 20.

[0053] The control unit 204 of the user terminal 20 presents the received one or more answer sentences as output sentences D12 on the display 2081 of the user terminal 20. The control unit 204 of the user terminal 20 may present the multiple answer sentences in a list on the display 2081 of the user terminal 20. Also, the answer sentences may be presented in a selectable manner.

[0054] In step S105, if the relevance associated with one or more answer sentences acquired in the answer acquisition step does not satisfy a predetermined condition, the control unit 104 of the server 10 inputs the question sentence received in the question reception step as input data into the large-scale language model. Specifically, when at least a part or all of the answer sentences among one or more answer sentences output from the question and answer system 60 do not satisfy a predetermined condition, the control unit 104 of the server 10 transmits the question sentence accepted in step S101 to the generation AI 50. The generation AI 50 receives the question sentence and accepts input. The control unit 104 of the server 10 generates a prompt by inserting the question received in step S101 into the [Question] section of the following prompt template as an input sentence (prompt) for the generation AI 50, which is a large-scale language model, and sends the prompt to the generation AI 50. [Prompt template] You are a very kind and capable AI assistant. It resides on FAQ sites and answers users' FAQ searches. When answering, please refer to the following reference answers. #User Questions [Question] #Reference answer text [Answer 1] [Relevance 1] [Answer 2] [Relevance 2] [Answer 3] [Relevance 3]

[0055] The reply sending step includes a step of inputting at least a part of the one or more reply sentences acquired in the reply acquiring step into a large-scale language model as input data. Specifically, the control unit 104 of the server 10 may include the answer sentence and the relevance level acquired from the question and answer system 60 in step S103 in the input data for the generation AI 50. For example, the control unit 104 of the server 10 generates a prompt by inserting one or more answer sentences and relevance into the prompt template's [answer sentence 1] [answer sentence 2] [answer sentence 3]..., [relevance 1] [relevance 2] [relevance 3]... as an input sentence (prompt) for the generation AI 50, which is a large-scale language model, and transmits the generated prompt to the generation AI 50. Note that the input data for the generation AI 50 does not need to include the relevance.

[0056] The reply sending step includes a step of inputting a predetermined number of reply sentences, from the one or more reply sentences acquired in the reply acquiring step, in order of decreasing relevance, as input data to the large-scale language model. Specifically, the control unit 104 of the server 10 does not need to include in the prompt all of one or more answer sentences output from the question and answer system 60, and may include in the prompt a predetermined number of answer sentences in order of low or high relevance and the relevance degrees associated with the answer sentences. Also, the control unit 104 of the server 10 may include in the prompt answer sentences whose relevance degrees are equal to or less than a predetermined value or equal to or more than a predetermined value and the relevance degrees associated with the answer sentences.

[0057] In step S106, if the relevance associated with one or more answer sentences acquired in the answer acquisition step does not satisfy a predetermined condition, the control unit 104 of the server 10 executes a reply sending step of sending to the user terminal a reply sentence that is output by inputting the question sentence received in the question receiving step as input data into a large-scale language model. Specifically, the control unit 104 of the server 10 acquires the response sentence output from the generation AI 50 in response to the input of the question sentence in step S105. The generation AI 50 may obtain one or more response sentences and the relevance level associated with each of the one or more response sentences. The control unit 104 of the server 10 transmits the acquired response message to the user terminal 20 .

[0058] The control unit 204 of the user terminal 20 presents the received reply message or messages on the display 2081 of the user terminal 20 as an output message D12.

[0059] By checking the display 2081 of the user terminal 20, the user can confirm the answer from the question and answer system 60 or the reply from the generation AI 50 to the question entered in step S101 as the output sentence D12.

[0060] In the present disclosure, a configuration is adopted in which one answer or response sentence is output for one input question sentence, but a configuration may also be adopted in which multiple answer or response sentences are output for one input question sentence.

[0061] The chatbot system may be configured to allow the user to input an additional question to the answer or response output from the information processing system. In this case, when the user inputs an additional question (additional question), the question input to the information processing system may be combined (joined) with at least one of the question, answer, and response already stored in the question and answer table 1013 to execute question processing. In this case, the user can obtain a more accurate answer or response by repeatedly checking the answer or response and asking an additional question. In this case, the question process may be terminated at the timing when the answer is presented in step S104. The question processing may be terminated when it is determined that an appropriate answer sentence with a high degree of relevance has been obtained from question and answering system 60.

[0062] <Management process> The management process is a process for improving the question and answer information (data) stored in the question and answering system 60 based on the question sentence received from the user in the question process and the answer sentence from the question and answering system 60.

[0063] <Overview of management process> The management process is a series of processes that presents a list of questions received from the user in the question processing and answers output from the question answering system 60, accepts the selection of a question from the user, accepts the selection of an answer to be associated with the selected question, and stores the question and answer in association with each other.

[0064] <Details of management process> The management process will be described in detail below.

[0065] In step S301, a user related to the administrator operates the input device 306 of the administrator terminal 30 to execute a browser application or the like, and opens the management page D3 by inputting the URL of a web page (management page) for executing management processing. The control unit 304 of the administrator terminal 30 transmits a request including the administrator ID 3011 for opening the management page to the server 10. The administrator is a user who has the authority to manage the question and answering system. Specifically, the administrator can execute processes such as editing and deleting the contents of the question and answering information stored in the question and answering system 60.

[0066] When the server 10 receives the request, it generates a management page and transmits it to the administrator terminal 30. The control unit 304 of the administrator terminal 30 displays the management page on the display 3081 of the administrator terminal 30 for presentation.

[0067] Specifically, the control unit 104 of the server 10 executes an answer presentation step of associating one or more answer data (including answer sentences) stored in the answer storage step with one or more question sentences, and presenting question answer information relating to pairs of questions and their answers stored in the question and answer system that can be viewed by the administrator on an editable management screen. Specifically, the control unit 104 of the server 10 refers to the question and answer table 1013 to obtain one or more question sentences and one or more answer data. The control unit 104 of the server 10 may be configured to refer to the relevance associated with one or more answer sentences included in the answer data, and acquire or present a predetermined number of answer sentences in order of decreasing relevance, or a predetermined number of answer sentences in order of increasing relevance, and not acquire or present any other answer sentences. The control unit 104 of the server 10 may be configured to refer to the relevance associated with one or more answer sentences included in the answer data, and acquire or present answer sentences whose relevance is below a predetermined value, or whose relevance is above a predetermined value, and not acquire or present other answer sentences. The control unit 104 of the server 10 may be configured to refer to the relevance associated with one or more answer sentences included in the answer data, and acquire or present question sentences corresponding to answer sentences whose relevance is less than a predetermined value, or questions corresponding to answer sentences whose relevance is greater than or equal to a predetermined value, and not acquire or present other question sentences. In the present disclosure, the management process is intended to improve the answer content of question and answer system 60, so it is preferable to configure it to obtain or present answer sentences whose relevance is below a predetermined value, and question sentences that correspond to answer sentences whose relevance is below a predetermined value. The control unit 104 of the server 10 transmits the question and answer obtained from the question and answer table 1013 to the administrator terminal 30 .

[0068] 10 is an example of a management screen in management processing. Management page D3 is displayed on display 3081 of administrator terminal 30. Management page D3 includes a list of questions D311, D312, D313..., answers D321, D322, D323..., and answers D331, D332, D333... stored in question and answer system 60. Based on the question and answer sentences obtained from question and answer table 1013, control unit 304 of administrator terminal 30 generates and draws question sentences D311, D311, D312..., answer sentences D321, D322, D323..., and check boxes D3311, D3321, D3331... indicating the relevance of each of the multiple answer sentences. This allows the administrator to check the answer sentence output from the question answering system together with the relevance when the question answering system cannot obtain an answer sentence with a sufficiently high relevance based on the question sentence received from the user.The administrator can check the question sentence when the question answering system cannot obtain an answer sentence with a sufficiently high relevance based on the question sentence received from the user.

[0069] In addition, in management page D3, questions D311, D312, D313,... and answer statements D321, D322, D323,... may each be presented in association with a specific question statement input by the user in the question processing and an answer statement output from question and answer system 60 in response to the question statement. Specifically, answer sentences output from question and answer system 60 to question sentences D311, D312, D313, . . . may be answer sentences D321, D322, D323, . . . .

[0070] In step S302, the control unit 304 of the manager terminal 30 executes a question selection step of accepting, from the manager, a selection of one or more questions presented in the question presentation step. Specifically, the administrator selects one or more predetermined questions D311, D312, D313 from among the questions D311, D312, D313, ... included in the management page D3 by operating the input device 306 of the administrator terminal 30. The control unit 304 of the administrator terminal 30 accepts the selection of the one or more selected questions. The control unit 304 of the administrator terminal 30 transmits the question ID of the selected question to the server 10.

[0071] In step S303, the control unit 104 of the server 10 acquires, from the question and answering system 60, question and answer information relating to the question sentence and the answer sentence that are stored in advance in the question and answering system 60. The control unit 104 of the server 10 may transmit to the administrator terminal 30 only the answer to question information related to the question selected in step S302, from among the answer to question information stored in the question and answering system 60. Specifically, the control unit 104 of the server 10 searches the question ID item in the question and answer table 1013 based on the question ID received in step S302, and acquires the question text. Based on the acquired question text, the control unit 104 of the server 10 transmits a request to the question and answer system 60 to acquire question and answer information related to the question text. The control unit 104 of the server 10 acquires the question and answer information related to the question text output from the question and answer system 60. The control unit 104 of the server 10 transmits the question and answer information acquired from the question and answer system 60 to the administrator terminal 30 .

[0072] In step S304, the control unit 104 of the server 10 executes an answer input step of accepting input of a predetermined answer sentence corresponding to the question sentence accepted in the question selection step. The control unit 104 of the server 10 executes an answer list step of presenting one or more answer sentences stored in the question and answer system to the administrator. The control unit 304 of the administrator terminal 30 generates answer sentences D331, D332, D333 . . . based on the question and answer information obtained from the question and answer system 60, and presents them in a list on the display 3081 of the administrator terminal 30.

[0073] The answer input step executes a step of accepting input of a predetermined answer sentence selected by the administrator from among one or more answer sentences presented in the answer listing step. Specifically, the administrator selects one or more predetermined answer sentences D331, D332, D333... from among the answer sentences D331, D332, D333... by operating the input device 306 of the administrator terminal 30. For example, the administrator selects check boxes D3311, D3321, D3331. The control unit 304 of the administrator terminal 30 accepts the selection of the one or more selected answer sentences. Note that the administrator may be configured to be able to select answer sentences D321, D322, D323... by operating the input device 306 of the administrator terminal 30. The control unit 304 of the administrator terminal 30 transmits to the server 10 answer-to-question identification information for identifying the answer-to-question information of the selected answer sentences D331, D332, D333 . . .

[0074] In step S305, the control unit 104 of the server 10 executes an association step in which, in the question and answer system, the answer sentence input in the answer input step is stored as an answer sentence to the question sentence selected in the question selection step. Specifically, the control unit 104 of the server 10 searches the question ID item of the question and answer table 1013 based on the question ID of the question selected in step S302, and acquires the question. The control unit 104 of the server 10 transmits a request to the question and answer system 60 to associate the acquired question with the question and answer identification information selected in step S304. The question and answering system 60 associates the question sentence selected in step S302 with the question and answer information selected in step S304 as question and answer information that is a new question and answer pair, and stores the associated information. In this way, when a question sentence with a sufficiently high relevance cannot be obtained from the question answering system, the question answering system can store the question sentence in association with the answer sentence stored in the question answering system. When a user newly asks a similar question, the question answering system can provide an appropriate answer.

[0075] Furthermore, the control unit 104 of the server 10 may associate the question sentences D311, D312, D313, ... with the answer sentences D321, D322, D323 in accordance with a selection by the administrator, and store them as question answering information in the question and answering system 60. In this way, when an answer sentence that has been determined to have a low degree of relevance in question processing is newly asked, it can be answered as an answer sentence with a high degree of relevance.

[0076] <Management process (variation example)> In the management process, an example has been disclosed in which a question and an answer are presented to an administrator, and based on the question and answer, additions, corrections, editing, etc. are performed on the question and answer information stored in the question and answer system 60; however, this is not limited to this. For example, a question and a response may be presented to the administrator, and the answer to question information stored in question and answering system 60 may be added to, modified, edited, etc. based on the question and response.

[0077] The control unit 104 of the server 10 executes a reply presentation step of presenting, on an editable management screen, a reply sentence output by the administrator through input to the large-scale language model, and question and answer information relating to pairs of questions and their answers stored in the question and answering system that can be viewed by the administrator. Specifically, in management page D3, instead of answer sentences D321, D322, D323, ..., one or more reply sentences stored in question and answer table 1013 may be presented to the administrator. In management page D3, question sentences D311, D312, D313, ... and one or more reply sentences may each be presented as a predetermined question sentence input by the user in the question process and one or more reply sentences output from question and answer system 60 in response to the question sentence, in association with each other.

[0078] The control unit 104 of the server 10 executes a reply selection step of accepting, from the administrator, a selection of a reply sentence to be output by inputting it into the large-scale language model. The control unit 104 of the server 10 executes a reply selection step of accepting, from the administrator, a selection of a reply sentence to be presented in the reply presentation step. Specifically, the administrator selects one or more response messages included in the management page D3 by operating the input device 306 of the administrator terminal 30. The control unit 304 of the administrator terminal 30 transmits the selected response messages to the server 10. The administrator may be configured to be able to amend or correct the response message displayed on the management page D3 by operating the input device 306 of the administrator terminal 30. This allows the administrator to amend the response message output by the generation AI 50 to an appropriate content even if the content contains an error or an insufficiency. In this case, the amended or corrected response message is also included in the response message selected by the administrator.

[0079] The control unit 104 of the server 10 executes an associating step of accepting an input of a second question sentence from the administrator, and associating the second question sentence with the reply sentence selected and accepted in the answer selection step, and storing the second question sentence in the question and answering system. The associating step is a step of associating the second question sentence selected in the question selection step with the reply sentence selected and accepted in the answer selection step, and storing the second question sentence in the question and answering system. Specifically, the administrator selects one or more predetermined questions D311, D312, D313 from among the questions D311, D312, D313, ... included in the management page D3 by operating the input device 306 of the administrator terminal 30. The control unit 304 of the administrator terminal 30 accepts the selection of the one or more selected questions. The control unit 304 of the administrator terminal 30 transmits the question ID of the selected question to the server 10. The administrator may input a new question by operating the input device 306 of the administrator terminal 30 to input a character string or the like in an input field or the like (not shown). In this case, the control unit 304 of the administrator terminal 30 transmits the input new question to the server 10. The control unit 104 of the server 10 searches the question ID item of the question and answer table 1013 based on the question ID of the received question text, and acquires the question text. Also, the control unit 104 acquires the received new question text from the administrator terminal 30.

[0080] The control unit 104 of the server 10 transmits to the question and answer system 60 a request for associating the acquired question with the reply selected in the reply selection step. The question answering system 60 associates the acquired question sentence with the reply sentence as question answering information, which is a new question and answer pair, and stores them. This allows the reply sentence output by the generation AI 50 to be stored as a new answer sentence in the question answering system. If a user asks a new similar question, the question answering system can provide an appropriate answer. This reduces the number of times requests are sent to the generation AI 50.

[0081] <Basic computer hardware configuration> 11 is a block diagram showing the basic hardware configuration of a computer 90. The computer 90 includes at least a processor 901, a main storage device 902, an auxiliary storage device 903, and a communication IF 991 (interface). These are electrically connected to each other by a communication bus 921.

[0082] The processor 901 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, a register, a peripheral circuit, and the like.

[0083] The main memory device 902 is for temporarily storing programs, data to be processed by the programs, etc. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0084] The auxiliary storage device 903 is a storage device for saving data and programs, such as a flash memory, a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory.

[0085] The communication IF 991 is an interface for inputting and outputting signals for communicating with other computers via a network using a wired or wireless communication standard. The network is composed of the Internet, a LAN, various mobile communication systems constructed by wireless base stations, etc. For example, the network includes 3G, 4G, 5G mobile communication systems, LTE (Long Term Evolution), wireless networks that can connect to the Internet via a specified access point (e.g., Wi-Fi (registered trademark)), etc. In the case of wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), etc. In the case of wired connection, the network also includes a network that is directly connected by a USB (Universal Serial Bus) cable or the like.

[0086] It should be noted that the computer 90 can be virtually realized by distributing all or part of each hardware configuration among multiple computers 90 and connecting them together via a network. In this way, the computer 90 is a concept that includes not only a computer 90 housed in a single housing or case, but also a virtualized computer system.

[0087] <Basic functional configuration of computer 90> A description will now be given of the functional configuration of a computer realized by the basic hardware configuration (FIG. 11) of a computer 90. The computer comprises at least the functional units of a control unit, a storage unit, and a communication unit.

[0088] The functional units of the computer 90 can also be realized by distributing all or part of the functional units among multiple computers 90 connected to each other via a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

[0089] The control unit is realized by the processor 901 reading out various programs stored in the auxiliary storage device 903, expanding the programs in the main storage device 902, and executing processes according to the programs. The control unit can realize functional units that perform various information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0090] The storage unit is realized by a main storage device 902 and an auxiliary storage device 903. The storage unit stores data, various programs, and various databases. Furthermore, the processor 901 can secure a storage area corresponding to the storage unit in the main storage device 902 or the auxiliary storage device 903 in accordance with a program. Furthermore, the control unit can cause the processor 901 to execute processes of adding, updating, and deleting data stored in the storage unit in accordance with the various programs.

[0091] The term database refers to a relational database, which is used to manage sets of data called masters and tables in a tabular format structurally defined by rows and columns, by associating them with each other. In a database, a table is called a table or master, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables and masters can be set and associated. Usually, a column that serves as a primary key for uniquely identifying a record is set in each table and each master, but setting a primary key in a column is not essential. The control unit can cause the processor 901 to add, delete, or update records in a specific table or master stored in the storage unit according to various programs. Furthermore, by storing data, various programs, and various databases in the storage unit, it can be considered that the information processing device and information processing system according to the present disclosure have been manufactured.

[0092] In addition, the database and master in this disclosure may include any data structure (such as a list, a dictionary, an associative array, or an object) in which information is structurally defined. The data structure also includes data that can be considered as a data structure by combining data with a function, class, method, or the like written in any programming language.

[0093] The communication unit is realized by the communication IF 991. The communication unit realizes a function of communicating with other computers 90 via a network. The communication unit can receive information transmitted from other computers 90 and input the information to the control unit. The control unit can cause the processor 901 to execute information processing on the received information in accordance with various programs. In addition, the communication unit can transmit information output from the control unit to other computers 90.

[0094] <Additional Notes> The matters described in the above embodiments will be supplemented below.

[0095] (Appendix 1) A program to be executed by a computer having a processor and a memory unit, the program comprising: a question receiving step (S101) of receiving a question from a user terminal, a question input step (S102) of inputting the question received in the question receiving step to a question answering system capable of outputting an answer in response to the input question sentence, an answer acquisition step (S103) of acquiring one or more answer sentences corresponding to the question sentence input in the question input step and a relevance degree associated with each of the one or more answer sentences, an answer sending step (S104) of sending one or more answer sentences that satisfy a predetermined condition to the user terminal if the relevance degree associated with the one or more answer sentences acquired in the answer acquisition step satisfies a predetermined condition, and a reply sending step (S106) of sending to the user terminal a reply sentence that is output by inputting the question received in the question receiving step into a large-scale language model as input data if the relevance degree associated with the one or more answer sentences acquired in the answer acquisition step does not satisfy the predetermined condition. In this way, if the question answering system obtains an answer sentence with a sufficiently high degree of relevance based on the question sentence received from the user, the answer sentence is output to the user, and if the question answering system does not obtain an answer sentence with a sufficiently high degree of relevance, a reply sentence from a large-scale language model can be output to the user. This makes it possible to output a more appropriate answer (response) to the question.

[0096] (Appendix 2) 2. The program of claim 1, wherein the question answering system is a system different from the large-scale language model. As a result, if the question answering system obtains an answer sentence with a sufficiently high degree of relevance based on the question received from the user, the answer sentence can be output to the user, and if the question answering system does not obtain an answer sentence with a sufficiently high degree of relevance, a reply sentence from a large-scale language model can be output to the user.

[0097] (Appendix 3) The program according to claim 1, wherein the reply sending step (S106) includes a step of inputting at least a part of the one or more reply sentences acquired in the reply acquisition step into a large-scale language model as input data. As a result, even if the question answering system is unable to obtain an answer sentence with a sufficiently high relevance based on the question received from the user, the obtained answer sentence can be included as reference information and posed to the large-scale language model, so that a more appropriate answer sentence obtained from the large-scale language model can be output to the user.

[0098] (Appendix 4) The program described in Appendix 1, wherein the reply sending step (S106) includes a step of inputting a predetermined number of answer sentences from the one or more answer sentences acquired in the answer acquisition step in order of relevance to a large-scale language model as input data. As a result, even if the question answering system is unable to obtain an answer sentence with a sufficiently high relevance based on the question received from the user, a more appropriate reply sentence obtained from the large-scale language model can be output to the user by asking the large-scale language model a question using the more relevant answer sentence from the obtained answer sentences as reference information.

[0099] (Appendix 5) A program described in any one of Appendices 1 to 4, wherein a processor executes an answer storage step (S103) of associating one or more answer sentences acquired in the answer acquisition step with a question sentence and storing them, and an answer presentation step (S301) of presenting the one or more answer sentences stored in the answer storage step on an editable management screen as question answer information relating to a pair of a question and its answer stored in the question answering system that can be viewed by an administrator, wherein the administrator is a user who has the authority to manage the question answering system. This allows the administrator to check the answers output from the question answering system in response to the questions received from users, and the administrator can use the contents of the answers as a reference to improve the question answering system.

[0100] (Appendix 6) The program according to claim 5, wherein the answer presenting step (S301) is a step of presenting a predetermined number of answer sentences in order of decreasing relevance from among the one or more answer sentences acquired in the answer acquiring step. This allows the administrator to check answers with low relevance that are output from the question and answer system in response to a question received from a user.

[0101] (Appendix 7) The program described in Appendix 5, wherein the answer presenting step (S301) is a step of presenting one or more answer sentences when the relevance associated with one or more answer sentences acquired in the answer acquisition step does not satisfy a predetermined condition. This allows the administrator to check the answer sentence output by the question answering system when the question answering system does not obtain an answer sentence with a sufficiently high degree of relevance based on a question received from a user.

[0102] (Appendix 8) A program described in any one of Appendices 1 to 4, in which a processor executes a question storage step (S103) of storing a question received in the question receiving step when the relevance associated with one or more answer sentences acquired in the answer acquisition step does not satisfy a predetermined condition, and a question presentation step (S301) of presenting to an administrator the one or more question sentences stored in the question storage step. This allows the administrator to check the question when the question and answering system does not provide an answer with a sufficiently high degree of relevance based on the question received from the user.

[0103] (Appendix 9) The program described in Appendix 8, wherein the processor executes a question selection step (S302) of accepting from an administrator a selection of one or more question sentences presented in the question presentation step, an answer input step (S304) of accepting input of a predetermined answer sentence corresponding to the question sentence accepted in the question selection step, and an association step (S305) of storing the answer sentence entered in the answer input step as an answer sentence to the question sentence selected in the question selection step in the question answering system. When the question answering system does not provide an answer with a sufficiently high relevance, the system can store an answer in association with the question. When a user asks a new similar question, the system can provide an appropriate answer.

[0104] (Appendix 10) The program described in Appendix 9, wherein the processor executes an answer list step (S301) of presenting one or more answer sentences stored in the question answering system to an administrator, and an answer input step (S304) of accepting input of a specific answer sentence selected by the administrator from among the one or more answer sentences presented in the answer list step. When a question sentence with a sufficiently high relevance cannot be obtained from the question answering system, the question answering system can store the question sentence in association with the answer sentence stored in the question answering system. When a user newly asks a similar question, the question answering system can provide an appropriate answer.

[0105] (Appendix 11) A program as described in any one of Appendices 1 to 4, wherein the processor executes a reply presentation step in which the processor presents, on an editable management screen, reply sentences output by an administrator through input to a large-scale language model, question and answer information relating to pairs of questions and their answers stored in the question and answering system that can be viewed by the administrator, wherein the administrator is a user who has the authority to manage the question and answering system. As a result, if the question answering system obtains an answer sentence with a sufficiently high degree of relevance based on the question received from the user, the answer sentence can be output to the user, and if the question answering system does not obtain an answer sentence with a sufficiently high degree of relevance, a reply sentence from a large-scale language model can be output to the user.

[0106] (Appendix 12) 12. The program according to claim 11, wherein the processor executes a reply selection step of accepting from an administrator a selection of a reply sentence presented in the reply presentation step, and an association step of accepting input of a second question sentence from the administrator, associating the second question sentence with the reply sentence selected in the reply selection step, and storing the second question sentence in the question answering system. As a result, if the question answering system obtains an answer sentence with a sufficiently high degree of relevance based on the question received from the user, the answer sentence can be output to the user, and if the question answering system does not obtain an answer sentence with a sufficiently high degree of relevance, a reply sentence from a large-scale language model can be output to the user.

[0107] (Appendix 13) A program as described in Appendix 12, wherein a processor executes a question presenting step (S301) of presenting to an administrator one or more questions received in the question receiving step, and a question selection step (S302) of accepting from the administrator a selection of one or more second questions presented in the question presenting step, and an associating step is a step of associating the second question selected in the question selection step with the answer selected in the answer selection step and storing the second question in the question answering system. As a result, if the question answering system obtains an answer sentence with a sufficiently high degree of relevance based on the question received from the user, the answer sentence can be output to the user, and if the question answering system does not obtain an answer sentence with a sufficiently high degree of relevance, a reply sentence from a large-scale language model can be output to the user. [Explanation of symbols]

[0108] 1 system, 10 server, 101 memory unit, 104 control unit, 106 input device, 108 output device, 20 user terminal, 201 memory unit, 204 control unit, 206 input device, 208 output device, 30 administrator terminal, 301 memory unit, 304 control unit, 306 input device, 308 output device, 50 generation AI, 501 memory unit, 504 control unit, 506 input device, 508 output device, 60 question answering system, 601 memory unit, 604 control unit, 606 input device, 608 output device

Claims

1. A program to be executed by a computer having a processor and a storage unit, The processor, a question receiving step of receiving a question from a user terminal; a question input step of inputting the question sentence received in the question receiving step to a question answering system capable of outputting an answer sentence in response to the input of the question sentence; an answer acquisition step of acquiring one or more answer sentences corresponding to the question sentence input in the question input step and a relevance degree associated with each of the one or more answer sentences; a response transmission step of transmitting, to the user terminal, the one or more response sentences that satisfy a predetermined condition when a relevance degree associated with the one or more response sentences acquired in the response acquisition step satisfies the predetermined condition; a response sending step of sending, to the user terminal, a response sentence that is output by inputting the question sentence accepted in the question accepting step as input data into a large-scale language model when a relevance degree associated with the one or more answer sentences acquired in the response acquiring step does not satisfy a predetermined condition; A program that executes.

2. The question answering system is a system different from the large-scale language model. The program according to claim 1.

3. the reply sending step includes a step of inputting at least a part of the one or more reply sentences acquired in the reply acquiring step into the large-scale language model as the input data; The program according to claim 1.

4. the reply sending step includes a step of inputting a predetermined number of the answer sentences, among the one or more answer sentences acquired in the answer acquiring step, in order of decreasing relevance to the large-scale language model as the input data; The program according to claim 1.

5. The processor, an answer storage step of storing the one or more answer sentences acquired in the answer acquisition step in association with the question sentence; an answer presenting step of presenting the one or more answer sentences stored in the answer storing step on a management screen that is viewable by an administrator and that is capable of editing question and answer information related to pairs of questions and their answers stored in the question and answering system; Run the administrator is a user who has authority to manage the question and answering system; The program according to claim 1.

6. the answer presentation step is a step of presenting a predetermined number of answer sentences from the one or more answer sentences acquired in the answer acquisition step in order of decreasing relevance; The program according to claim 5.

7. the answer presentation step is a step of presenting the one or more answer sentences when a relevance degree associated with the one or more answer sentences acquired in the answer acquisition step does not satisfy a predetermined condition; The program according to claim 5.

8. The processor, a question storage step of storing the question sentence received in the question receiving step when a relevance degree associated with the one or more answer sentences acquired in the answer acquiring step does not satisfy a predetermined condition; a question presenting step of presenting to an administrator one or more questions stored in the question storing step; Execute The program according to claim 1.

9. The processor, a question selection step of accepting, from the administrator, a selection of the one or more question sentences presented in the question presentation step; an answer input step of receiving an input of a predetermined answer sentence corresponding to the question sentence received in the question selection step; an associating step of storing the answer sentence inputted in the answer input step as an answer sentence to the question sentence selected in the question selection step in the question answering system; Execute The program according to claim 8.

10. The processor, an answer listing step of presenting one or more answers stored in the question and answering system to the administrator; Run the answer input step is a step of accepting an input of the predetermined answer sentence selected by the administrator from the one or more answer sentences presented in the answer listing step; The program according to claim 9.

11. The processor, a reply presentation step of presenting the reply sentence output by an administrator through input to the large-scale language model on a management screen that can be viewed by the administrator and that can edit question and answer information related to pairs of questions and their answers stored in the question and answering system; Run the administrator is a user who has authority to manage the question and answering system; The program according to claim 1.

12. The processor, a reply selection step of accepting, from the administrator, a selection of the reply sentences presented in the reply presentation step; an associating step of accepting an input of a second question from the administrator, and associating the second question with the answer selected in the answer selecting step and storing the second question in the question and answering system; Execute The program according to claim 11.

13. The processor, a question presenting step of presenting one or more questions received in the question receiving step to an administrator; a question selection step of accepting, from the administrator, a selection of the one or more second question sentences presented in the question presentation step; Run the associating step is a step of associating the second question selected in the question selecting step with the answer selected in the answer selecting step and storing the second question in the question and answering system. The program according to claim 12.

14. A method implemented on a computer having a processor and a memory, the processor performing all of the steps performed in any of the inventions according to claims 1 to 13.

15. 14. An information processing apparatus comprising a control unit and a storage unit, the control unit executing all steps executed in the invention according to any one of claims 1 to 13.

16. A system comprising means for performing all the steps performed in any one of claims 1 to 13.

Images