Information processing systems, information processing methods, and programs

Abstract

To improve the accuracy of responses based on the output of language models. [Solution] The information processing system obtains a question based on user input, retrieves answer basic information corresponding to the question from a database, requests a language model for determination to determine whether the question is related to the topic based on the topic-indicating information of the answer basic information corresponding to the question obtained from the database, obtains output from the language model for determination, and if the obtained output indicates that the question is related to the topic, requests a language model for answering to create an answer to the question based on the obtained answer basic information, obtains the answer from the language model, and sends information to the user that answers the question based on the obtained answer.

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background Art]

[0002] The performance of large language models (LLMs) has been greatly improved. Large language models can answer questions in natural text. Large language models are used in chatbots such as ChatGPT (registered trademark), for example.

[0003] Patent Document 1 discloses a technique for generating a summary sentence requested by a user using a large language model obtained by machine learning on a huge amount of unlabeled text. [Prior Art Documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Unexamined Patent Publication No. 2023-73095 [Summary of the Invention] [Problem to be Solved by the Invention]

[0005] Although language models such as large language models can output natural text in response to a question, inaccurate answers may be output depending on the technical field to which the question pertains. On the other hand, it is not easy to train a language model using knowledge in a specific field.

[0006] The present disclosure provides a technique for improving the accuracy of answers based on the output of a language model. [Means for Solving the Problem]

[0007] (1) An information processing system comprising: a question acquisition means for acquiring a question based on input from a user; a question determination means for acquiring a determination result of whether or not the question is subject to a predetermined field; an information acquisition means for acquiring basic answer information corresponding to the question from a database of the predetermined field; an answer acquisition means for requesting a language model for answers to create an answer to the question based on the acquired basic answer information when it is determined that the question is subject to the predetermined field, and acquiring an answer from the language model; and an answer means for sending information to the user to answer the question based on the acquired answer.

[0008] (2)(1) The information processing system wherein, when it is determined that the question does not cover the predetermined field, the answer acquisition means requests the answer language model to create an answer based on the question and obtains the answer from the language model.

[0009] (3)(1) The information processing system wherein the question determination means determines whether the question is subject to a predetermined field and whether the question is subject to rejection of answering, and the answer acquisition means, when it is determined that the question is not subject to a predetermined field and the question is not subject to rejection of answering, requests the answer language model to create an answer based on the question and acquires an answer from the language model.

[0010] (4)(3) If the response means determines that the question is subject to rejection of answering, it sends information to the user indicating that it cannot answer the question. Information processing system.

[0011] (5) An information processing system in which, in any of (1) to (4), the question determination means requests a language model for determination to determine whether the question is subject to the predetermined field, and determines whether the question is subject to the predetermined field based on the information output by the language model.

[0012] (6)(5) The question determination means requests the language model for determination to determine whether the question is subject to the predetermined field based on the predetermined field and information indicating the topic of the answer basic information corresponding to the question, obtained from the database, and determines whether the question is subject to the predetermined field based on the information output by the language model.

[0013] (7) An information processing system in which, in any of (1) to (6), the response acquisition means requests the response language model to create a response to the question based on the question and the acquired response basic information, and acquires the response from the language model.

[0014] (8) In any of (1) to (7), the database is an information processing system that generates feature vectors from the questions and obtains response basic information associated with feature vectors similar to the generated feature vectors.

[0015] (9) An information processing method comprising the steps of: obtaining a question based on input from a user; determining whether the question is related to a predetermined field; obtaining basic answer information corresponding to the question from a database of the predetermined field; if it is determined that the question is related to the predetermined field, requesting a language model for answers to create an answer to the question based on the obtained basic answer information and obtaining an answer from the language model; and sending information to answer the question to the user based on the obtained answer.

[0016] (10) A program for causing a computer to function as: question acquisition means for acquiring a question based on an input from a user; question determination means for determining whether or not said question targets a predetermined field; information acquisition means for acquiring answer basic information corresponding to said question from a database for said predetermined field; answer acquisition means for, when it is determined that said question targets said predetermined field, requesting an answer language model to generate an answer to said question based on said acquired answer basic information, and acquiring the answer from said language model; and answer means for transmitting information answering said question to said user based on said acquired answer. [Advantages of the Invention]

[0017] According to the present invention, the accuracy of answers based on the output of a language model can be improved. [Brief Description of the Drawings]

[0018] [Figure 1] It is a diagram showing an example of elements related to an inquiry management system according to an embodiment of the present invention. [Figure 2] It is a block diagram showing functions implemented by the inquiry management system. [Figure 3] It is a diagram showing an example of data stored in a feature database. [Figure 4] It is a diagram showing an example of document data. [Figure 5] It is a diagram showing an example of a screen displayed on a user terminal. [Figure 6] It is a diagram showing another example of a screen displayed on a user terminal. [Figure 7] It is a flowchart showing an example of processing of the inquiry management system. [Figure 8] It is a flowchart showing an example of processing of the inquiry management system. [Figure 9] It is a diagram showing an example of instruction text input to a large-scale language model by a question classification unit. [Figure 10]It is a diagram showing an example of instruction text input to a large language model by an answer acquisition unit. [Figure 11] It is a diagram showing an example of instruction text input to a large language model by an answer acquisition unit. MODE FOR CARRYING OUT THE INVENTION

[0019] Hereinafter, embodiments of the present invention will be described based on the drawings. Redundant descriptions of configurations denoted by the same reference numerals will be omitted.

[0020] FIG. 1 is a diagram showing an example of elements related to an inquiry management system 2 according to an embodiment of the present invention. The inquiry management system 2 receives question information from a user terminal 1 operated by a user, inputs a question sentence based on the information to a large language model service 3, and outputs an answer corresponding to the sentence output by the large language model service 3 to the user terminal 1.

[0021] The user terminal 1 is a computer having a user interface, such as a personal computer or a smartphone, for example.

[0022] The large language model service 3 includes a general-purpose large language model implemented by a computer. The large language model service 3 receives an instruction from the inquiry management system 2, and transfers an output obtained by inputting the instruction to the large language model to the inquiry management system 2. This instruction is in text format and is also called a prompt. Hereinafter, text-format instructions are also referred to as instruction text. This general-purpose large language model is trained with data from a wide range of fields. The large language model service 3 may be a service such as ChatGPT (registered trademark), for example.

[0023] In the following, when simply referred to as "large-scale language model," it refers to the large-scale language model included in the large-scale language model service 3. The inquiry management system 2 performs the process of inputting information into the large-scale language model and obtaining output from the large-scale language model by using the API provided by the large-scale language model service 3. The inquiry management system 2 is not required to input all information in a single API call; it may input information in parts through multiple API calls. The large-scale language model service 3 may also be provided within the inquiry management system 2. In this embodiment, the inquiry management system 2 inputs information requesting the creation of some kind of answer from the large-scale language model and obtains the output of the large-scale language model as that answer. In the following, inputting information requesting the creation of some kind of answer from the large-scale language model will also be referred to as requesting the large-scale language model to create an answer.

[0024] The inquiry management system 2 includes one or more computers (e.g., server computers). The inquiry management system 2 includes one or more processors 21, one or more storage 22, and one or more communication units 23. The inquiry management system 2 may include multiple computers, each containing one or more processors 21, storage 22, and communication units 23, or it may include one computer having one or more processors 21 and storage 22. The inquiry management system 2 may be implemented on one or more virtual servers or container infrastructure.

[0025] The processor 21 operates according to the program (also called instruction code) stored in the storage 22. The processor 21 also controls the communication unit 23. The processor 21 may include, for example, a CPU (Central Processing Unit), and may also include a GPU (Graphic Processing Unit) or an NPU (Neural Processing Unit). The program may be provided via the internet or the like, or it may be provided stored on a computer-readable storage medium such as flash memory or DVD-ROM.

[0026] The storage 22 consists of memory elements such as RAM and flash memory, and external storage devices such as hard disk drives (HDDs) and solid-state drives (SSDs). The storage 22 stores the above-mentioned program. The storage 22 also stores information and calculation results input from the processor 21 and the communication unit 23.

[0027] The communication unit 23 is a communication interface that communicates with other devices, such as a network interface card. The communication unit 23 is composed of integrated circuits, antennas, and communication terminals that implement wireless LAN or wired LAN, for example. Based on the control of the processor 21, the communication unit 23 inputs information received from other devices via the network to the processor 21 and storage 22, and transmits the information to the other devices.

[0028] Note that the hardware configuration of the inquiry management system 2 is not limited to the example above. For example, the inquiry management system 2 may include devices for reading computer-readable information storage media (e.g., optical disc drives or memory card slots) and devices for inputting and outputting data with external devices (e.g., USB ports). External devices may be input devices or output devices.

[0029] Next, the functions provided by the inquiry management system 2 will be described. Figure 2 is a block diagram showing the functions realized by the inquiry management system 2. Functionally, the inquiry management system 2 includes a management unit 50, a knowledge database 60, and an order management API 65. Functionally, the management unit 50 includes a question text acquisition unit 51, a question classification unit 52, a knowledge acquisition unit 53, an answer acquisition unit 54, an answer modification unit 55, a consistency determination unit 56, and an answer output unit 57. Functionally, the knowledge database 60 includes a search engine 61 and a knowledge document storage unit 62. The management unit 50, the knowledge database 60, and the order management API 65 are realized by the processor 21 executing programs corresponding to each function stored in the storage 22 and controlling the communication unit 23, etc.

[0030] The knowledge database 60 is a database that stores knowledge information for a predetermined knowledge field. The knowledge database 60 obtains information indicating a question from the user from the management unit 50, searches for knowledge information corresponding to that information, and passes the knowledge information found through the search back to the management unit 50. The knowledge database 60 has a different configuration from the large-scale language model of the large-scale language model service 3. Knowledge information may include knowledge documents and strings indicating the topic of those knowledge documents. The knowledge document storage unit 62 stores knowledge documents as knowledge information belonging to a certain knowledge field. Knowledge documents may be, for example, data of a document containing a set of questions and answers in an FAQ. Here, the questions in the FAQ may be treated as the topic of the knowledge document. Knowledge information may not be a document. Knowledge information including knowledge documents is used as the basis information (answer base information) when the large-scale language model creates an answer to a question.

[0031] The search engine 61 obtains information indicating a question from the user and searches for knowledge information corresponding to the search conditions. The information indicating the question may be the obtained question text. The search engine 61 includes a feature extraction model, which is a machine learning model, and a feature database that stores feature vectors extracted from each of the knowledge pieces. The search engine 61 inputs the information indicating the question into the feature extraction model and obtains the feature vector output from the feature extraction model as the query vector. The search engine 61 searches the feature database for feature vectors similar to the query vector. The search engine 61 outputs the similar feature vectors. The search engine 61 may calculate the cosine similarity between the query vector and the feature vectors in the feature database as a measure of similarity, select the feature vector with the largest similarity as the similar feature vector, and obtain the knowledge information corresponding to the selected feature vector. Here, a feature vector is extracted for each of the documents that constitute the knowledge information.

[0032] Figure 3 shows an example of data stored in the feature database. In the example in Figure 3, feature vectors are stored associated with the document ID of the knowledge document, the title of the knowledge document (corresponding to the FAQ question and topic), the API name, and the document usage. The meaning of the API name and document usage will be explained later.

[0033] Figure 4 shows an example of a knowledge document. In Figure 4, only the portion of the knowledge document corresponding to the FAQ answer is shown. This knowledge document is associated with the feature vector of the row titled "How to Cancel Your Order" in Figure 3. Also, some parts of the actual text are omitted in Figure 4. The number of characters in each knowledge document is greater than the number of characters that the inquiry management system 2 is allowed to output as an answer to a question.

[0034] Here, the feature extraction model is trained using training document data. During training, the feature extraction model extracts the feature vector of the query and the feature vector of the document data corresponding to that query. The feature extraction model is then trained so that the similarity between the feature vector of the query and the feature vector of the corresponding document data increases, and the similarity for uncorresponding document data decreases. The document data used for training may consist only of the parts of the knowledge document that correspond to the topic or question. This training may be carried out based on the methods shown in the following two papers.

[0035] Paper 1: Wataru Sakata, Tomohide Shibata, Ribeka Tanaka, and Sadao Kurohashi. 2019. FAQ Retrieval using Query-Question Similarity and BERT-Based Query-Answer Relevance. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR'19). Association for Computing Machinery, New York, NY, USA, 1113-1116.

[0036] Paper 2: Seo, J.; Lee, T.; Moon, H.; Park, C.; Eo, S.; Aiyanyo, ID; Park, K.; So, A.; Ahn, S.; Park, J. Dense-to-Question and Sparse-to-Answer: Hybrid Retriever System for Industrial Frequently Asked Questions. Mathematics 2022, 10, 1335.

[0037] The Order Management API 65 is an API for obtaining user-specific information. The Order Management API 65 may include, for example, an API for obtaining the current status (e.g., payment completed, shipped, delivered) of orders that a user has previously placed with the e-commerce system.

[0038] The management unit 50 acquires information about the question from the user terminal 1 and retrieves information related to that question from the knowledge database 60. The management unit 50 also generates an answer to the question using the large-scale language model service 3 and outputs the answer to the user terminal 1 via the network.

[0039] The question text acquisition unit 51 acquires a question text based on user input. Specifically, the user input is the question information entered by the user operating the user terminal 1 and received from the user terminal 1. The question information may be in sentence form or in the form of keywords. The question text acquisition unit 51 may acquire the question information entered in sentence form as is, or it may generate a question text by processing multiple keywords or sentences contained in the question information.

[0040] Figure 5 shows an example of a screen displayed on user terminal 1. Figure 5 shows a screen where a question is entered in a chat format. In the example in Figure 5, the user has entered the question itself.

[0041] The question text acquisition unit 51 may input a prompt to the large-scale language model to create a question text from multiple keywords or sentences acquired as question information, and acquire the output of the large-scale language model in response to that prompt as the question text. Here, it is desirable that the information input to the large-scale language model service 3 does not include personal information. Therefore, the question text acquisition unit 51 may input information from which personal information has been removed in advance into the large-scale language model.

[0042] The question classification unit 52 obtains a determination result regarding whether the acquired question can be answered by the large-scale language model alone, whether it belongs to a predetermined knowledge field, and whether it is subject to rejection of the answer. The predetermined knowledge field is a knowledge field to which the knowledge information stored in the knowledge database 60 belongs, and is a knowledge field to which the inquiry management system 2 can answer. Here, the question classification unit 52 requests the large-scale language model to determine whether the acquired question belongs to a knowledge field based on the acquired question and the knowledge field to which it can answer, and obtains a determination result from the large-scale language model. The question classification unit 52 may further request the large-scale language model to determine whether the question belongs to a knowledge field based on information indicating the topic of the answer basic information corresponding to the question, which is obtained from the database. The question classification unit 52 may simply request a determination as to whether the question belongs to a predetermined field, or simply request a determination as to whether it can be answered by the large-scale language model alone, or request a determination as to whether it belongs to any two of the above three categories.

[0043] In this embodiment, this determination is whether the question sentence falls into a specific category among several categories (types). The multiple categories include a category indicating that the question sentence belongs to a predetermined knowledge field (a knowledge field that can be answered) (corresponding to the specific category mentioned above), a category indicating that the answer will only be provided by a large-scale language model, and a category indicating that the question sentence is subject to rejection of an answer. Specifically, the information requested may be the text of the command being requested.

[0044] The knowledge acquisition unit 53 sends the question text as a query to the search engine 61 of the knowledge database 60, stores it in the knowledge document storage unit 62, and retrieves the knowledge information retrieved by the question text from the knowledge database 60.

[0045] The answer acquisition unit 54, when it determines that the question text targets a predetermined knowledge field, inputs information requesting the creation of an answer based on the question text and acquired knowledge information into a general-purpose large-scale language model, and retrieves the text output from the large-scale language model. Hereinafter, the text obtained by the answer acquisition unit 54 from the large-scale language model will be referred to as the model answer text. The number of characters in the model answer text is smaller than the number of characters in the knowledge information, and the request for answer creation is a kind of request for summarization. The information requesting answer creation does not have to include the question text, but including the question text increases the probability of obtaining a more appropriate model answer text.

[0046] Here, if the question does not target a predetermined knowledge field and is determined not to be subject to rejection of answers, the answer acquisition unit 54 requests a general-purpose large-scale language model to create an answer based on the question and obtains a model answer from the output of that large-scale language model. The answer acquisition unit 54 does not necessarily need to use whether or not the question is subject to rejection of answers. For example, if the multiple categories that the question classification unit 52 classifies do not include any category that indicates the question is subject to rejection of answers, the answer acquisition unit 54 may request a general-purpose large-scale language model to create an answer based on the question and obtain a model answer from the output of that large-scale language model when the question does not target a predetermined knowledge field. Also, if the question is determined to be subject to rejection of answers, the answer acquisition unit 54 requests a general-purpose large-scale language model to create an answer indicating that the question cannot be answered and obtains a model answer indicating that the question cannot be answered.

[0047] The response modification unit 55 modifies specific strings contained in the model response text. For example, the response modification unit 55 may filter out hyperlink strings contained in the model response text and decode specific strings contained in the model response text. When decoding specific strings, if the model response text contains a string that encodes personal information, the response modification unit 55 converts that string back to the original personal information.

[0048] The consistency determination unit 56 determines whether the acquired or modified model response sentence is consistent with the acquired knowledge information. Here, the consistency determination unit 56 determines consistency based on whether each of the multiple words contained in the model response sentence is also contained in the words included in the knowledge information. The method for determining consistency will be described later.

[0049] The response output unit 57 generates a response to be output to the user based on the result of the consistency determination and the model response sentence. If the response output unit 57 determines that the model response sentence is consistent with the knowledge information, it generates a response that includes the model response sentence and information that identifies the knowledge information (e.g., a hyperlink). If the response output unit 57 determines that the model response sentence is not consistent with the knowledge information, it generates a response that includes the model response sentence and information indicating the low reliability of the response. The response output unit 57 also outputs the generated response to the user's user terminal 1.

[0050] Figure 6 shows another example of the screen displayed on user terminal 1. Figure 6 shows an example of a response generated when the knowledge document shown in Figure 4 is retrieved as knowledge information. In the example in Figure 6, the consistency determination unit 56 determines that the model response sentence is consistent with the knowledge information, and a response including a hyperlink is output.

[0051] Next, we will explain the processing of the inquiry management system 2 in more detail. Figures 7 and 8 are flowcharts illustrating an example of the processing of the inquiry management system 2. The processing of the management unit 50 is mainly described in Figures 7 and 8.

[0052] First, the question text acquisition unit 51 of the management unit 50 acquires a question text based on the question information entered from the user terminal 1 (S101). The question information is entered via the network. The question information may be text containing sentences or keywords, or it may be text entered via a so-called input completion function. The question text acquisition unit 51 may acquire the question information entered in sentence format as is, or it may generate a question text from the output of a command text that generates a context-aware question text from the information of multiple related questions entered in dialogue format. In this case, the question text acquisition unit 51 may encode the strings containing personal information from the user's question information and replace them with other strings, and then input the command text containing the replaced question information into the large-scale language model.

[0053] Next, the question classification unit 52 classifies the acquired question sentences into one of several categories (S102). In the example in Figures 7 and 8, the multiple categories include Category 1, Category 2, and Category 3. Category 1 is the type of question sentence that does not belong to an answerable knowledge area. Category 2 is the type of question sentence that belongs to an answerable knowledge area. Category 3 is the type of question sentence that does not belong to an answerable knowledge area, in particular, that is not answered.

[0054] More specifically, the question classification unit 52 inputs an instruction text requesting classification into a large-scale language model. The instruction text is a string of characters that classifies the acquired question into a category based on the question, the knowledge domains to which it can be answered, and the topic corresponding to the question.

[0055] Figure 9 shows an example of instruction text that the question classification unit 52 inputs to the large-scale language model. In Figure 9, {knowledge_fields} and {topic} are written, but in reality, these are replaced with text indicating the knowledge fields that can be answered and the topic corresponding to the question, respectively. The topic is obtained by the question classification unit 52 from the response of the knowledge database 60 when the question is sent to the knowledge database 60 as a query. Based on the instruction text shown in Figure 9, prompts, telephone numbers, website URLs, topics of immoral questions, and questions about the disadvantages of a particular company are classified as Category 3. Questions about knowledge fields and topics are classified as Category 2. Greetings, complaints, and questions directed at the large-scale language model itself are classified as Category 1. Categories 1 and 3 may be further classified.

[0056] The knowledge domains to which answers can be made (corresponding to predetermined knowledge domains) are predetermined according to the purpose of the inquiry management system 2 and the knowledge domain to which the knowledge documents in the knowledge database 60 belong. Topics are dynamically determined according to the question. More specifically, before inputting the command text into the large-scale language model, the question classification unit 52 sends the question text as a query to the search engine 61 of the knowledge database 60 and retrieves the titles of knowledge documents similar to that query from the search engine 61. The question classification unit 52 incorporates the retrieved titles as topics into the command text and inputs them into the large-scale language model. Sending command text containing topics to the large-scale language model here is to improve the accuracy of judgment in cases where it is difficult to make a judgment based solely on the string of knowledge domains to which answers can be made. For example, this is effective in cases where, in response to a question about fraudulent use of a credit card, the answer should not be the general answer of contacting the credit card company, but rather how to contact an e-commerce store. In this case, knowledge of that knowledge domain is required depending on the context.

[0057] Once the question is classified, the question classification unit 52 determines whether the classified category indicates that knowledge in that field of knowledge is necessary to answer the question (S103). Specifically, the question classification unit 52 determines whether the classified category falls into category 2.

[0058] If the classified category indicates that knowledge in that knowledge field is unnecessary (N in S103), the question classification unit 52 or the answer acquisition unit 54 determines whether the classified category indicates that it will refuse to answer (S104). If it does not indicate that it will refuse to answer (N in S104), the answer acquisition unit 54 causes the large-scale language model to generate an answer sentence based on the question (S105). Specifically, the answer acquisition unit 54 inputs an instruction text to the large-scale language model to generate an answer sentence based on the question, and acquires the answer sentence output from the large-scale language model. This instruction text does not include knowledge information (especially knowledge documents) and may be category-specific.

[0059] Figure 10 shows an example of the command text that the response acquisition unit 54 inputs to the large-scale language model. The command text shown in Figure 10 contains the notations {name} and {assistant}, but in reality, these are replaced with the name of the chatbot and the name of the industry handled by this inquiry management system 2, respectively. The command text shown in Figure 10 does not include information from the knowledge database 60 and requests the large-scale language model to generate a response without using that information. The response is generated from general-purpose knowledge included in the large-scale language model.

[0060] The response output unit 57 then outputs the generated response text to the user terminal 1 (S106). The process shown in Figures 7 and 8 then ends.

[0061] On the other hand, if the classified category indicates that the answer is rejected (Y in S104), the answer acquisition unit 54 generates an answer statement indicating that it cannot answer the question (S107). In S106, the answer output unit 57 outputs the generated answer statement to the user terminal 1. The generated answer statement may be a predefined text stored in storage 22 in advance, or it may be a text obtained by the answer acquisition unit 54 requesting the large-scale language model to generate an answer. Alternatively, the answer acquisition unit 54 may request the large-scale language model to generate an answer statement using an instruction text that includes a request statement equivalent to S104, and obtain the answer statement from the large-scale language model.

[0062] On the other hand, if the classified category indicates that knowledge in that knowledge field is required (Y in S103), the process in S108 is executed. In S108, the knowledge acquisition unit 53 acquires dynamic information about the user via the order management API 65 if dynamic information about the user is required. Whether or not dynamic information about the user is required is determined by whether or not an API name exists, which is obtained as a response when the knowledge acquisition unit 53 sends a question to the search engine 61 of the knowledge database 60 as a query. If an API name exists, the knowledge acquisition unit 53 calls the API included in the order management API 65 that corresponds to the API name, along with parameters containing information that identifies the user. The API name may be the one obtained when acquiring a topic from the knowledge database 60 in S102. For example, according to the example in Figure 3, if the question is similar to the title of the order status that the user wants to know, the order status API is called.

[0063] Following S108, the process in S109 is executed. In S109, the knowledge acquisition unit 53 acquires a knowledge document corresponding to the question statement, which is output from the knowledge database 60, if a knowledge document is required as knowledge information. This knowledge document is acquired as a response to the search engine 61 of the knowledge database 60, which receives the question statement as a query. In the example in Figure 3, if the question statement is an inquiry about how to cancel an order, the knowledge document with document ID 000009739 shown in Figure 4 is acquired. Here, the knowledge acquisition unit 53 does not acquire a knowledge document if document use is not true. The knowledge acquisition unit 53 may decide whether to acquire a knowledge document simply based on whether or not the document ID exists, instead of document use. Cases in which a knowledge document is not acquired include, for example, cases where an answer can be generated from the information acquired from the order management API 65 and the question statement, such as simply checking the status of an order.

[0064] When processing in S108 and S109 is completed, the answer acquisition unit 54 inputs an instruction text to the large-scale language model to generate an answer sentence, and acquires its output as the model answer sentence (S110). The instruction text includes an instruction to generate an answer sentence from a question and dynamic information or knowledge document.

[0065] Figure 11 shows an example of the command text that the response acquisition unit 54 inputs to the large-scale language model. In the example in Figure 11, as in the example in Figure 9, the entries {knowledge_fields} and {topic} are actually replaced with text indicating the knowledge fields that can be answered and the topic corresponding to the question, respectively. Also, the entries {name} and {assistant} in Figure 10 are actually replaced with the name of the chatbot and the name of the industry that this inquiry management system 2 is responsible for, respectively. {observation_data} is replaced with the text of a knowledge document or dynamic information. The example in Figure 11 does not include a question, but it is included in the command text that is entered in advance and is not shown.

[0066] As shown in the example in Figure 11, the command text includes a command to generate a model response without using the information held by the large-scale language model if sufficient knowledge information such as knowledge documents is available, and a command to generate a model response of 400 characters or less. Therefore, the large-scale language model generates and outputs a model response consisting of a string of characters less than or equal to the specified number of characters.

[0067] When a model response is obtained, the response modification unit 55 modifies specific strings contained in the model response (S111). Specifically, if the model response contains a hyperlink string, the response modification unit 55 deletes that string. Also, if the model response contains a string that encodes personal information, the response modification unit 55 restores that string to its original personal information.

[0068] The consistency determination unit 56 then determines whether the model response sentence is consistent with the knowledge document (S112). Here, the consistency determination unit 56 extracts multiple words (for example, verbs and nouns) that satisfy certain conditions from the model response sentence by morphological analysis, and determines whether there is consistency based on whether each of those words is included in the knowledge document. More specifically, the consistency determination unit 56 may determine that there is no consistency if the proportion of those words that are not included in the knowledge document exceeds a threshold, and determine that there is consistency if it is below the threshold.

[0069] If consistency is determined (Y in S112), the response output unit 57 generates an output statement with information identifying the knowledge document added to the model response statement, and outputs the output statement to the user terminal 1 (S113). The information identifying the knowledge document may be, for example, a hyperlink to the knowledge document, or a string indicating how to refer to the knowledge document. On the other hand, if consistency is determined (N in S112), the response output unit 57 generates an output statement with a warning statement added to the model response statement, and outputs the output statement to the user terminal 1 (S114). This warning statement indicates the low reliability of the response statement, and more specifically, it may be a statement that the response statement is not necessarily correct and requires verification. The user terminal 1 receives the output statement output in S113 or S114 and presents it to the user.

[0070] If the instruction text in S110 does not contain a knowledge document, the process in S112 will not be executed, and the answer output unit 57 may output the model answer statement as is.

[0071] In this embodiment, the inquiry management system 2 retrieves knowledge documents from a knowledge database 60, which stores knowledge information about a specific knowledge domain, before having the large-scale language model generate an answer to a question. Then, it uses the question and its knowledge documents to have the large-scale language model generate a draft answer (i.e., a model answer). This improves the accuracy of the answer output from the large-scale language model, even if the large-scale language model's ability to answer questions in a given domain is insufficient.

[0072] The inquiry management system 2 verifies the consistency between its model response and the original knowledge information. This consistency verification makes it possible to address cases where the model response provides an answer unrelated to the question, thereby preventing users from misinterpreting the facts.

[0073] In this embodiment, the large-scale language model used by the question acquisition unit 51 to generate question sentences, the large-scale language model used by the question classification unit 52 to determine the acquired question sentences, and the large-scale language model used to generate model answer sentences are the same; however, different large-scale language models may be used for each other. For example, the large-scale language model used to determine the question sentences and the large-scale language model used to generate model answer sentences may be different.

[0074] In this embodiment, only a warning message was added when inconsistency was detected, but the answer message may be generated by other methods. For example, instead of a model answer message and a warning message, the answer output unit 57 may output links to multiple similar knowledge information items output by the knowledge database 60 in response to the question message and a standard phrase as the answer message.

[0075] Although a large-scale language model is used in this embodiment, there are no particular limitations on its implementation or the number of parameters. The present invention can be applied to machine learning models (language models) that handle natural language. [Explanation of Symbols]

[0076] 1. User terminal, 2. Inquiry management system, 3. Large-scale language model service, 21 Processor, 22 Storage, 23 Communication Unit, 50 Management Unit, 51 Question Text Acquisition Unit, 52 Question Classification Unit, 53 Knowledge Acquisition Unit, 54 Answer Acquisition Unit, 55 Answer Correction Unit, 56 Consistency Determination Unit, 57 Answer Output Unit, 60 Knowledge Database, 61 Search Engine, 62 Knowledge Document Storage Unit, 65 Order Management API.

Claims

1. A question acquisition means that acquires questions based on user input, A question determination means inputs a command text, which includes text indicating a predetermined field and the question, and causes the question to determine whether or not it pertains to the predetermined field, into a language model for determination, and obtains output from the language model for determination. When the acquired output indicates that the question pertains to a predetermined field, the answer acquisition means requests a language model for answers to create an answer to the question based on answer base information corresponding to the question, acquired from a database for the predetermined field, and acquires the answer from the language model. A response means that sends information to the user to answer the question based on the response obtained, An information processing system that includes this.

2. In the information processing system described in claim 1, The response acquisition means, when the acquired output indicates that it does not cover the predetermined field, requests the response language model to create a response based on the question and acquires the response from the language model. Information processing system.

3. In the information processing system described in claim 1, The question determination means requests a language model for determination to provide an instruction text that determines whether the question is subject to the predetermined field and whether the question is subject to rejection of answering. The response acquisition means, when the acquired output indicates that the question does not concern the predetermined field and that the question is not subject to rejection of an answer, requests the response language model to create an answer based on the question and acquires the answer from the language model. Information processing system.

4. In the information processing system described in claim 3, The response means, when the acquired output indicates that the question is subject to rejection of answering, sends information to the user indicating that it cannot answer the question. Information processing system.

5. In the information processing system described in claim 1, The response acquisition means requests the response language model to create an answer to the question based on the question and the acquired response basic information, and acquires the answer from the language model. Information processing system.

6. In the information processing system described in claim 1, The database generates feature vectors from the questions and retrieves response background information associated with feature vectors similar to the generated feature vectors. Information processing system.

7. In the information processing system described in claim 1, The language model for determination and the language model for response are the same language model. Information processing system.

8. An information processing system including one or more processors, Steps include obtaining questions based on user input, The steps include inputting a command text, which includes text indicating a predetermined field and the question, and which causes the question to determine whether or not it pertains to the predetermined field, into a language model for determination, and obtaining output from the language model for determination, If the obtained output indicates that the question pertains to the predetermined field, the step of requesting a language model for answers to create an answer to the question based on answer base information corresponding to the question obtained from a database for the predetermined field, and obtaining the answer from the language model, Based on the answers obtained, the step of sending information to the user to answer the question, Information processing methods including

9. Question acquisition means that acquires questions based on user input. Question determination means inputs a command text, which includes text indicating a predetermined field and the question, and causes the question to determine whether or not it pertains to the predetermined field, into a language model for determination, and obtains output from the language model for determination. When the acquired output indicates that the question pertains to a predetermined field, the answer acquisition means requests a language model for answers to create an answer to the question based on answer base information corresponding to the question, acquired from a database for the predetermined field, and acquires the answer from the language model; A response means that sends information to the user to answer the question based on the response obtained above. A program that makes a computer function.

Images