Information processing system, information processing method and program

Abstract

To improve the accuracy of an answer based on an output by a language model.SOLUTION: An information processing system: obtains a question based on an input from a user; obtains, from a database, answer basic information corresponding to the question; requests a determination language model to determine whether the question is relevant to a topic or not based on information representing the topic of the answer basic information which is obtained from the database, and which corresponds to the question; obtains an output from the determination language model; requests an answer language model to create an answer for the question based on the obtained answer basic information when the obtained output represents such that the question is relevant to the topic; obtains the answer from the language model; and transmits, to the user, information that is an answer to the question based on the obtained answer.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] The performance of large language models (LLMs) has improved significantly. LLMs can respond to questions in natural-sounding sentences. LLMs are used in chatbots such as ChatGPT (registered trademark).

[0003] Patent Document 1 discloses a technology for generating a summary sentence desired by a user using a large-scale language model obtained by machine learning a huge amount of unlabeled text. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-73095 Summary of the Invention [Problem to be solved by the invention]

[0005] While language models such as large-scale language models can output natural-sounding sentences in response to questions, they can sometimes output inaccurate answers depending on the domain the question is about. On the other hand, it is not easy to train a language model using knowledge from that domain.

[0006] The present disclosure provides techniques for improving the accuracy of answers based on the output of a language model. [Means for solving the problem]

[0007] (1) An information processing system including: a question acquisition means for acquiring a question based on an input from a user; a question determination means for acquiring a determination result as to whether the question is related to a predetermined field; an information acquisition means for acquiring answer basic information corresponding to the question from a database for the predetermined field; an answer acquisition means for, when it is determined that the question is related to the predetermined field, requesting an answer language model to create an answer to the question based on the acquired answer basic information and acquiring the answer from the language model; and an answering means for sending information answering the question to the user based on the acquired answer.

[0008] (2) In (1), the information processing system, wherein the answer acquisition means, when it is determined that the question does not target the specified field, requests the answer language model to create an answer based on the question and acquires the answer from the language model.

[0009] (3) In (1), the question determination means determines whether the question is related to a predetermined field and whether the question is a subject for which an answer is to be refused, and the answer acquisition means, if it is determined that the question is not related to a predetermined field and is not a subject for which an answer is to be refused, requests the answer language model to create an answer based on the question and acquires the answer from the language model.

[0010] (4) In (3), when the answering means determines that the question is one to which an answer cannot be refused, the answering means sends information indicating that the question cannot be answered to the user. Information processing system.

[0011] (5) In any of (1) to (4), the question determination means requests a language model for determination to determine whether the question is related to the specified field, and determines whether the question is related to the specified field based on information output by the language model.

[0012] (6) In (5), the question determination means requests the determination language model to determine whether the question targets the specified field based on the specified field and information indicating the topic of the answer basic information corresponding to the question, obtained from the database, and determines whether the question targets the specified field based on the information output by the language model.

[0013] (7) In any one of (1) to (6), the answer acquisition means requests the answer language model to create an answer to the question based on the question and the acquired answer basic information, and acquires the answer from the language model.

[0014] (8) In any of (1) to (7), the database generates a feature vector from the question and obtains answer basic information associated with a feature vector similar to the generated feature vector.

[0015] (9) An information processing method including the steps of: acquiring a question based on an input from a user; determining whether the question is related to a predetermined field; acquiring answer basic information corresponding to the question from a database for the predetermined field; if it is determined that the question is related to the predetermined field, requesting an answer language model to create an answer to the question based on the acquired answer basic information and acquiring the answer from the language model; and sending information answering the question to the user based on the acquired answer.

[0016] (10) A program for causing a computer to function as a question acquisition means for acquiring a question based on input from a user, a question determination means for determining whether the question is related to a predetermined field, an information acquisition means for acquiring basic answer information corresponding to the question from a database for the predetermined field, an answer acquisition means for requesting an answer language model to create an answer to the question based on the acquired basic answer information when it is determined that the question is related to the predetermined field, and acquiring the answer from the language model, and an answering means for sending information answering the question to the user based on the acquired answer. [Effects of the Invention]

[0017] The present invention can improve the accuracy of answers based on the output of a language model. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 2 is a diagram illustrating an example of elements related to an inquiry management system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing functions realized by the inquiry management system. [Figure 3] FIG. 10 is a diagram illustrating an example of data stored in a feature database. [Figure 4] FIG. 2 is a diagram illustrating an example of document data. [Figure 5] FIG. 10 is a diagram illustrating an example of a screen displayed on a user terminal. [Figure 6] FIG. 10 is a diagram illustrating another example of a screen displayed on the user terminal. [Figure 7] 10 is a flowchart illustrating an example of processing by an inquiry management system. [Figure 8] 10 is a flowchart illustrating an example of processing by an inquiry management system. [Figure 9] FIG. 10 is a diagram showing an example of instruction text that the question classification unit inputs to the large-scale language model. [Figure 10]FIG. 10 is a diagram showing an example of command text that the answer acquisition unit inputs to the large-scale language model. [Figure 11] FIG. 10 is a diagram showing an example of command text that the answer acquisition unit inputs to the large-scale language model. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Duplicate descriptions of components with the same reference numerals will be omitted.

[0020] 1 is a diagram showing an example of elements related to a query management system 2 according to an embodiment of the present invention. The query management system 2 receives question information from a user terminal 1 operated by a user, inputs a question sentence based on that information to a large-scale language model service 3, and outputs an answer corresponding to the sentence output by the large-scale language model service 3 to the user terminal 1.

[0021] The user terminal 1 is a computer having a user interface, such as a personal computer or a smartphone.

[0022] The large-scale language model service 3 includes a general-purpose large-scale language model implemented by a computer. The large-scale language model service 3 receives instructions from the query management system 2, inputs the instructions into the large-scale language model, and passes the output obtained by inputting the instructions to the query management system 2. This instruction is in text format and is also called a prompt. Hereinafter, instructions in text format will also be referred to as instruction text. This general-purpose large-scale language model is trained using data from a wide range of fields. The large-scale language model service 3 may be, for example, a service such as ChatGPT (registered trademark).

[0023] Hereinafter, when simply referred to as a "large-scale language model," this refers to a large-scale language model included in the large-scale language model service 3, and the query management system 2 executes the process of inputting information to the large-scale language model and obtaining output from the large-scale language model by using an API provided by the large-scale language model service 3. The query management system 2 does not necessarily input all information with a single API call, but may input information in parts using multiple API calls. The large-scale language model service 3 may be provided within the query management system 2. In this embodiment, the query management system 2 inputs information requesting the creation of an answer to the large-scale language model and obtains the output of the large-scale language model as that answer. Hereinafter, inputting information requesting the creation of an answer to the large-scale language model will also be referred to as requesting the large-scale language model to create an answer.

[0024] The inquiry management system 2 includes one or more computers (e.g., server computers). The inquiry management system 2 includes one or more processors 21, one or more storages 22, and one or more communication units 23. The inquiry management system 2 may include multiple computers each including one or more processors 21, storages 22, and communication units 23, or may include a single computer having one or more processors 21 and storages 22. The inquiry management system 2 may be implemented on one or more virtual server or container platforms.

[0025] The processor 21 operates in accordance with a program (also referred to as an instruction code) stored in the storage 22. The processor 21 also controls the communication unit 23. The processor 21 includes, for example, a CPU (Central Processing Unit), and may further include a GPU (Graphic Processing Unit) and an NPU (Neural Processing Unit). Note that the program may be provided via the Internet or the like, or may be provided by being stored in a computer-readable storage medium such as a flash memory or a DVD-ROM.

[0026] The storage 22 is composed of memory elements such as RAM and flash memory, and external storage devices such as a hard disk drive (HDD) and a solid state drive (SSD). The storage 22 stores the above programs. The storage 22 also stores information input from the processor 21 and the communication unit 23 and calculation results.

[0027] The communication unit 23 is a communication interface, such as a network interface card, that communicates with other devices. The communication unit 23 is configured with an integrated circuit, an antenna, a communication terminal, etc. that realize a wireless LAN or a wired LAN. Based on the control of the processor 21, the communication unit 23 inputs information received from other devices via a network to the processor 21 or the storage 22, and transmits the information to other devices.

[0028] The hardware configuration of the inquiry management system 2 is not limited to the above example. For example, the inquiry management system 2 may include a device for reading a computer-readable information storage medium (e.g., an optical disk drive or a memory card slot) or a device for inputting and outputting data to and from an external device (e.g., a USB port). The external device may be an input device or an output device.

[0029] Next, the functions provided by the inquiry management system 2 will be described. FIG. 2 is a block diagram showing the functions realized by the inquiry management system 2. The inquiry management system 2 functionally includes a management unit 50, a knowledge database 60, and an order management API 65. The management unit 50 functionally includes a question acquisition unit 51, a question classification unit 52, a knowledge acquisition unit 53, an answer acquisition unit 54, an answer correction unit 55, a consistency determination unit 56, and an answer output unit 57. The knowledge database 60 functionally includes a search engine 61 and a knowledge document storage unit 62. The management unit 50, the knowledge database 60, and the order management API 65 are realized by the processor 21 executing programs corresponding to each function stored in the storage 22 and controlling the communication unit 23, etc.

[0030] The knowledge database 60 is a database in which knowledge information in a predetermined knowledge field is stored. The knowledge database 60 acquires information indicating a user's question from the management unit 50, searches for knowledge information corresponding to the information, and passes the knowledge information found by the search to the management unit 50. The knowledge database 60 has a different configuration from the large-scale language model of the large-scale language model service 3. The knowledge information may include knowledge documents and character strings indicating the topics of the knowledge documents. The knowledge document storage unit 62 stores knowledge documents as knowledge information belonging to a certain knowledge field. The knowledge documents may be document data including sets of questions and answers in an FAQ, for example. Here, questions in the FAQ may be treated as topics of the knowledge documents. The knowledge information may be something other than documents. The knowledge information including knowledge documents is used as basic information (basic answer information) when the large-scale language model creates answers to questions.

[0031] The search engine 61 acquires information indicating a question from a user and searches for knowledge information corresponding to the search criteria. The information indicating the question may be the acquired question sentence. The search engine 61 includes a feature extraction model, which is a machine learning model, and a feature database that stores feature vectors extracted from each piece of knowledge information. The search engine 61 inputs the information indicating the question into the feature extraction model and acquires the feature vector output from the feature extraction model as a query vector. The search engine 61 searches the feature database for feature vectors similar to the query vector. The search engine 61 outputs the similar feature vector. The search engine 61 may calculate, for example, the cosine similarity between the query vector and the feature vectors in the feature database as the similarity, select the feature vector with the largest similarity as the similar feature vector, and acquire knowledge information corresponding to the selected feature vector. Here, a feature vector is extracted for each document that is knowledge information.

[0032] Fig. 3 is a diagram showing an example of data stored in the feature database. In the example of Fig. 3, the feature vector is stored in association with the document ID of the knowledge document, the title of the knowledge document (corresponding to the question and topic of the FAQ), the API name, and the document usage. The meaning of the API name and the document usage will be described later.

[0033] FIG. 4 is a diagram showing an example of a knowledge document. In FIG. 4, only the part of the knowledge document that corresponds to the answer to the FAQ is shown. This knowledge document is associated with the feature vector of the line with the title "How to cancel an order" in FIG. 3. Also, in FIG. 4, some of the actual content is omitted. The number of characters in each knowledge document is greater than the number of characters that the inquiry management system 2 is allowed to output as an answer to a question.

[0034] Here, the feature extraction model is trained using document data for training. During training, the feature extraction model extracts the feature vector of a query and the feature vector of document data corresponding to the query. The feature extraction model is then trained so that the similarity between the feature vector of the query and the feature vector of the corresponding document data increases and the similarity between the feature vector of the query and the feature vector of the corresponding document data decreases. The document data used in training may be only the part of the knowledge document that corresponds to the topic or question. This training may be performed based on the methods described in the following two papers:

[0035] Paper 1: Wataru Sakata, Tomohide Shibata, Ribeka Tanaka, and Sadao Kurohashi. 2019. FAQ Retrieval using Query-Question Similarity and BERT-Based Query-Answer Relevance. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR'19). Association for Computing Machinery, New York, NY, USA, 1113-1116.

[0036] Paper 2: Seo, J.; Lee, T.; Moon, H.; Park, C.; Eo, S.; Aiyanyo, ID; Park, K.; So, A.; Ahn, S.; Park, J. Dense-to-Question and Sparse-to-Answer: Hybrid Retriever System for Industrial Frequently Asked Questions. Mathematics 2022, 10, 1335.

[0037] The order management API 65 is an API for obtaining information specific to a user. The order management API 65 may include, for example, an API for obtaining the current status (e.g., payment completed, shipped, delivery completed) of an order that a user previously placed with the e-commerce system.

[0038] The management unit 50 acquires information about a question from the user terminal 1, and acquires information related to the question from the knowledge database 60. The management unit 50 also generates an answer to the question using the large-scale language model service 3, and outputs the answer to the user terminal 1 via the network.

[0039] The question sentence acquisition unit 51 acquires a question sentence based on an input from a user. Specifically, the input from the user is question information that is input by the user operating the user terminal 1 and received from the user terminal 1. The question information may be in the form of a sentence or keywords. The question sentence acquisition unit 51 may acquire the question information input in the form of a sentence as is as a question sentence, or may generate a question sentence by processing a plurality of keywords or sentences included in the question information.

[0040] Fig. 5 is a diagram showing an example of a screen displayed on the user terminal 1. Fig. 5 shows a screen for inputting a question in a chat format. In the example of Fig. 5, the question itself is input by the user.

[0041] The question acquisition unit 51 may input a prompt to the large-scale language model to create a question from multiple keywords or sentences acquired as question information, and acquire the output of the large-scale language model in response to the prompt as the question. Here, it is preferable that the information input to the large-scale language model service 3 does not include personal information. Therefore, the question acquisition unit 51 may input information from which personal information has been removed in advance to the large-scale language model.

[0042] The question classification unit 52 acquires a determination result as to whether the acquired question sentence can be answered using only the large-scale language model, whether it is targeted at a predetermined knowledge field, and whether it is a subject for which an answer is to be rejected. The predetermined knowledge field is a knowledge field to which the knowledge information stored in the knowledge database 60 belongs, and is a knowledge field to which the inquiry management system 2 can answer. Here, the question classification unit 52 requests the large-scale language model to determine whether the acquired question sentence belongs to the knowledge field based on the acquired question sentence and the knowledge fields to which the question sentence can be answered, and acquires a determination result from the large-scale language model. The question classification unit 52 may also request the large-scale language model to determine whether the question sentence belongs to the knowledge field based on information indicating the topic of the answer basic information corresponding to the question acquired from the database. Note that the question classification unit 52 may simply request a determination as to whether the question is targeted at a predetermined field, or may simply request a determination as to whether the question can be answered using only the large-scale language model, or may request a determination as to whether the question belongs to any two of the three categories.

[0043] In this embodiment, this determination is a determination as to whether the question sentence falls into a specific category among a plurality of categories (types). The plurality of categories include a category (corresponding to the specific category described above) indicating that the question sentence belongs to a predetermined knowledge field (a knowledge field that can be answered), a category indicating that an answer can be provided using only a large-scale language model, and a category indicating that the question sentence is a target for which an answer is to be refused. Specifically, the requested information may be the text of a command to be requested.

[0044] The knowledge acquisition unit 53 transmits the question as a query to the search engine 61 of the knowledge database 60, and acquires from the knowledge database 60 the knowledge information that is stored in the knowledge document storage unit 62 and searched for by the question.

[0045] When the answer acquisition unit 54 determines that the question sentence is targeted at a predetermined knowledge field, it inputs information requesting the creation of an answer based on the question sentence and the acquired knowledge information into a general-purpose large-scale language model and acquires a sentence output from the large-scale language model. Hereinafter, the sentence acquired by the answer acquisition unit 54 from the large-scale language model will be referred to as a model answer sentence. Note that the number of characters in the model answer sentence is smaller than the number of characters in the knowledge information, and the request for answer creation is a kind of request for a summary. The information requesting the creation of an answer does not have to include a question sentence, but including a question sentence increases the probability of obtaining a more appropriate model answer sentence.

[0046] Here, if it is determined that the question does not target a predetermined knowledge domain and is not a target for which an answer is to be rejected, the answer acquisition unit 54 requests a general-purpose large-scale language model to create an answer based on the question and acquires a model answer sentence from the output of the large-scale language model. The answer acquisition unit 54 does not need to use whether the question is a target for which an answer is to be rejected. For example, if the multiple categories to be classified by the question classification unit 52 do not include a category indicating a target for which an answer is to be rejected, the answer acquisition unit 54 may request a general-purpose large-scale language model to create an answer based on the question if the question does not target a predetermined knowledge domain and acquire a model answer sentence from the large-scale language model. On the other hand, if it is determined that the question is a target for which an answer is to be rejected, the answer acquisition unit 54 requests the general-purpose large-scale language model to create an answer indicating that the question cannot be answered and acquires a model answer sentence indicating that the question cannot be answered.

[0047] The answer correction unit 55 corrects a specific character string included in the model answer sentence. For example, the answer correction unit 55 may filter the character string of a hyperlink included in the model answer sentence and decode the specific character string included in the model answer sentence. In decoding the specific character string, if the model answer sentence includes a character string in which personal information is encoded, the answer correction unit 55 restores the character string to the original personal information.

[0048] The consistency determination unit 56 determines whether the acquired or corrected model answer sentence is consistent with the acquired knowledge information. Here, the consistency determination unit 56 determines consistency based on whether each of the multiple words included in the model answer sentence is included in the words included in the knowledge information. The method of determining consistency will be described later.

[0049] The answer output unit 57 generates an answer to be output to the user based on the result of the consistency determination and the model answer sentence. If the answer output unit 57 determines that the model answer sentence is consistent with the knowledge information, it generates an answer including the model answer sentence and information (e.g., a hyperlink) that identifies the knowledge information. If the answer output unit 57 determines that the model answer sentence is not consistent with the knowledge information, it generates an answer including the model answer sentence and information that indicates the low reliability of the answer. Furthermore, the answer output unit 57 outputs the generated answer to the user terminal 1 of the user.

[0050] Fig. 6 is a diagram showing another example of a screen displayed on the user terminal 1. Fig. 6 shows an example of an answer created when the knowledge document shown in Fig. 4 is searched for as knowledge information. In the example of Fig. 6, the consistency determination unit 56 determines that the model answer sentence is consistent with the knowledge information, and an answer including a hyperlink is output.

[0051] Next, a more detailed description will be given of the processing of the inquiry management system 2. Figures 7 and 8 are flowcharts showing an example of the processing of the inquiry management system 2. The processing of the management unit 50 is mainly described in Figures 7 and 8.

[0052] First, the question acquisition unit 51 of the management unit 50 acquires a question based on question information input from the user terminal 1 (S101). The question information is input via a network. The question information may be text including sentences or keywords, or may be text input via a so-called input completion function. The question acquisition unit 51 may acquire question information input in a sentence format as a question as is, or may generate a question from the output of a command text input into a large-scale language model that generates a context-aware question from information about multiple related questions input in a dialogue format. In this case, the question acquisition unit 51 may encode a character string containing personal information from the information about the user's question and replace it with another character string, and input the command text including the replaced question information into the large-scale language model.

[0053] Next, the question classification unit 52 classifies the acquired question sentence into one of a plurality of categories (S102). In the example of FIGS. 7 and 8, the plurality of categories include Category 1, Category 2, and Category 3. Category 1 is a type of question sentence that does not belong to an answerable knowledge field. Category 2 is a type of question sentence that belongs to an answerable knowledge field. Category 3 is a type of question sentence that does not belong to an answerable knowledge field, particularly a type of question sentence that does not accept an answer.

[0054] More specifically, the question categorization unit 52 inputs a command text for requesting classification into the large-scale language model. The command text is a character string that categorizes the acquired question into a category based on the acquired question, the knowledge field that can be answered, and the topic corresponding to the question.

[0055] FIG. 9 is a diagram showing an example of instruction text that the question classification unit 52 inputs to the large-scale language model. In FIG. 9, the words {knowledge_fields} and {topic} are actually replaced with text indicating the knowledge fields to which an answer can be given and the topic corresponding to the question, respectively. The topic is acquired by the question classification unit 52 from the response of the knowledge database 60 when the question is sent as a query to the knowledge database 60. According to the instruction text shown in FIG. 9, prompts, telephone numbers, website URLs, immoral question topics, and questions about the disadvantages of a particular company are classified into category 3. Questions related to knowledge fields and topics are classified into category 2. Greetings, complaints, and questions to the large-scale language model itself are classified into category 1. Categories 1 and 3 may be classified differently.

[0056] The answerable knowledge fields (corresponding to predetermined knowledge fields) are predetermined based on the purpose of the inquiry management system 2 and the knowledge fields to which the knowledge documents in the knowledge database 60 belong. The topics are dynamically determined based on the question text. More specifically, before inputting the command text into the large-scale language model, the question classification unit 52 sends the question text as a query to the search engine 61 of the knowledge database 60 and obtains from the search engine 61 the titles of knowledge documents similar to the query. The question classification unit 52 incorporates the obtained titles as topics into the command text and inputs it into the large-scale language model. Here, sending the command text including topics to the large-scale language model improves the accuracy of judgment in cases where judgment is difficult based solely on the character strings in the answerable knowledge fields. For example, in response to a question about fraudulent credit card use, this is effective in providing an answer on how to contact an e-commerce store rather than the general answer of contacting the credit card company. In this case, knowledge of the knowledge field is required depending on the context.

[0057] When the question is classified, the question classification unit 52 determines whether the classified category indicates that knowledge in that knowledge field is required to answer the question (S103). Specifically, the question classification unit 52 determines whether the classified category falls into category 2.

[0058] If the classified category indicates that knowledge in that knowledge field is unnecessary (N in S103), the question classification unit 52 or the answer acquisition unit 54 determines whether the classified category indicates that an answer is rejected (S104). If it does not indicate that an answer is rejected (N in S104), the answer acquisition unit 54 causes the large-scale language model to generate an answer sentence based on the question sentence (S105). Specifically, the answer acquisition unit 54 inputs command text for generating an answer sentence based on the question sentence into the large-scale language model, and acquires the answer sentence output from the large-scale language model. This command text may not include knowledge information (especially knowledge documents) and may be appropriate for the category.

[0059] Fig. 10 is a diagram showing an example of command text that the answer acquisition unit 54 inputs to the large-scale language model. The command text shown in Fig. 10 includes the words {name} and {assistant}, but in reality, these are replaced with the name of the chatbot and the name of the business type handled by this inquiry management system 2, respectively. The command text shown in Fig. 10 does not include information from the knowledge database 60, and requests the large-scale language model to generate a response sentence without using that information. The response sentence is generated from general-purpose knowledge contained in the large-scale language model.

[0060] Then, the answer output unit 57 outputs the generated answer sentence to the user terminal 1 (S106), and the process shown in FIGS.

[0061] On the other hand, if the classified category indicates that an answer is rejected (Y in S104), the answer acquisition unit 54 generates an answer sentence indicating that the question cannot be answered (S107). In S106, the answer output unit 57 outputs the generated answer sentence to the user terminal 1. The generated answer sentence may be a fixed phrase stored in advance in the storage 22, or may be a sentence acquired by the answer acquisition unit 54 requesting the large-scale language model to generate an answer. Alternatively, the answer acquisition unit 54 may request the large-scale language model to generate an answer sentence using command text including a request sentence equivalent to S104, and acquire the answer sentence from the large-scale language model.

[0062] On the other hand, if the classified category indicates that knowledge in that knowledge field is required (Y in S103), the process of S108 is executed. In S108, if dynamic information about the user is required, the knowledge acquisition unit 53 acquires the dynamic information about the user via the order management API 65. Whether or not dynamic information about the user is required is determined by whether or not an API name exists, which is acquired as a response to the question sent by the knowledge acquisition unit 53 to the search engine 61 of the knowledge database 60 as a query. If the API name exists, the knowledge acquisition unit 53 calls an API included in the order management API 65 and corresponding to the API name, along with a parameter including information identifying the user. The API name may be the one acquired when acquiring a topic from the knowledge database 60 in S102. For example, according to the example of FIG. 3, if the question is similar to a title that requests information about the status of an order, the order status API is called.

[0063] After S108, the process of S109 is executed. In S109, if a knowledge document is required as knowledge information, the knowledge acquisition unit 53 acquires a knowledge document corresponding to the question, output from the knowledge database 60. This knowledge document is acquired as a response to the question sent as a query to the search engine 61 of the knowledge database 60. According to the example of FIG. 3, if the question inquires about how to cancel an order, the knowledge acquisition unit 53 acquires the knowledge document with document ID 000009739 shown in FIG. 4. Here, if document use is not true, the knowledge acquisition unit 53 does not acquire the knowledge document. Instead of document use, the knowledge acquisition unit 53 may simply determine whether to acquire the knowledge document based on whether the document ID exists. A case in which a knowledge document is not acquired is, for example, a case in which an answer can be generated from the question and information acquired from the order management API 65, such as simply checking the status of an order.

[0064] After the processes of S108 and S109 are completed, the answer acquisition unit 54 inputs an instruction text for generating an answer sentence into the large-scale language model and acquires the output as a model answer sentence (S110). The instruction text includes an instruction for generating an answer sentence from the question sentence and the dynamic information or the knowledge document.

[0065] FIG. 11 is a diagram showing an example of command text that the answer acquisition unit 54 inputs to the large-scale language model. In the example of FIG. 11, similar to the example of FIG. 9, the descriptions {knowledge_fields} and {topic} are actually replaced with text indicating the knowledge field to which an answer can be given and a topic corresponding to the question, respectively. Furthermore, the descriptions {name} and {assistant} in FIG. 10 are actually replaced with the name of the chatbot and the name of the business type handled by this inquiry management system 2, respectively. {observation_data} is replaced with text from a knowledge document or dynamic information. Although the example of FIG. 11 does not include a question, it is included in command text (not shown) that is input in advance.

[0066] 11, the command text includes an instruction to generate a model answer sentence without using information held by the large-scale language model if there is sufficient knowledge information such as a knowledge document, and an instruction to generate a model answer sentence of 400 characters or less. Therefore, the large-scale language model generates and outputs a model answer sentence consisting of a character string of the specified number of characters or less.

[0067] When the model answer sentence is acquired, answer correction unit 55 corrects a specific character string included in the model answer sentence (S111). Specifically, if the model answer sentence contains a character string of a hyperlink, answer correction unit 55 deletes the character string. Furthermore, if the model answer sentence contains a character string in which personal information is encoded, answer correction unit 55 restores the character string to the original personal information.

[0068] Then, the consistency determination unit 56 determines whether or not there is consistency between the model answer sentence and the knowledge document (S112). Here, the consistency determination unit 56 extracts multiple words (e.g., verbs and nouns) that satisfy conditions from the model answer sentence by morphological analysis, and determines whether or not there is consistency based on whether or not each of these words is included in the knowledge document. More specifically, the consistency determination unit 56 may determine that there is no consistency if the proportion of words that are not included in the knowledge document among these words exceeds a threshold, and may determine that there is consistency if the proportion is equal to or less than the threshold.

[0069] If it is determined that there is consistency (Y in S112), the answer output unit 57 generates an output statement in which information identifying the knowledge document is added to the model answer statement, and outputs the output statement to the user terminal 1 (S113). The information identifying the knowledge document may be, for example, a hyperlink to the knowledge document or a character string indicating how to refer to the knowledge document. On the other hand, if it is determined that there is no consistency (N in S112), the answer output unit 57 generates an output statement in which a warning statement is added to the model answer statement, and outputs the output statement to the user terminal 1 (S114). This warning statement is information indicating the low reliability of the answer statement, and more specifically, may be a statement that the answer statement is not necessarily correct and requires confirmation. The user terminal 1 receives the output statement output in S113 or S114 and presents it to the user.

[0070] If the command text does not contain a knowledge document in S110, the process of S112 is not executed, and the answer output unit 57 may output the model answer sentence as is.

[0071] In this embodiment, before causing the large-scale language model to generate an answer to a question, the query management system 2 acquires knowledge documents that will be the basis for the answer from a knowledge database 60 that stores knowledge information about a specific knowledge field.The query and the knowledge documents are then used to cause the large-scale language model to generate a draft answer (i.e., a model answer).This makes it possible to improve the accuracy of the answer output from the large-scale language model even if the answering ability of the large-scale language model in the field to be answered is insufficient.

[0072] The inquiry management system 2 checks the consistency between the model answer sentence and the original knowledge information. This consistency check makes it possible to deal with cases where the model answer sentence is an answer unrelated to the question, and prevents factual errors by the user.

[0073] In this embodiment, the large-scale language model used by the question acquisition unit 51 to generate a question sentence, the large-scale language model used by the question classification unit 52 to determine the acquired question sentence, and the large-scale language model used to generate a model answer sentence are the same, but different large-scale language models may also be used. For example, the large-scale language model used to determine the question sentence and the large-scale language model used to generate a model answer sentence may be different.

[0074] In this embodiment, even if a mismatch is detected, only a warning message is added, but an answer may be generated by other methods. For example, instead of a model answer and a warning message, the answer output unit 57 may output, as an answer, a link to a plurality of similar knowledge information items output by the knowledge database 60 in response to the question message and a template message.

[0075] Although a large-scale language model is used in this embodiment, there are no particular limitations on the scale of its implementation and the number of parameters. The present invention can be applied to machine learning models (language models) that handle natural language. [Explanation of symbols]

[0076] 1 User terminal, 2 Inquiry management system, 3 Large-scale language model service, 21 Processor, 22 storage, 23 communication unit, 50 management unit, 51 question acquisition unit, 52 question classification unit, 53 knowledge acquisition unit, 54 answer acquisition unit, 55 answer correction unit, 56 consistency determination unit, 57 answer output unit, 60 knowledge database, 61 search engine, 62 knowledge document storage unit, 65 order management API.

Claims

1. a question acquisition means for acquiring a question based on an input from a user; an information acquisition means for acquiring answer basic information corresponding to the question from a database; a question determination means for requesting a determination language model to determine whether the question is related to the topic based on information indicating the topic of the answer basic information corresponding to the question, the information being obtained from the database, and obtaining an output from the determination language model; an answer acquisition means for requesting an answer language model to create an answer to the question based on the acquired answer basic information when the acquired output indicates that the question is related to the topic, and acquiring the answer from the language model; a replying means for sending information that answers the question to the user based on the acquired answer; An information processing system including:

2. 2. The information processing system according to claim 1, When the acquired output indicates that the question is not related to the topic, the answer acquisition means requests the language model for answers to create an answer based on the question, and acquires the answer from the language model. Information processing system.

3. 2. The information processing system according to claim 1, the question determination means requests a determination language model to determine whether the question is related to the topic and whether the question is a target for which an answer is to be rejected; When the acquired output indicates that the question is not related to the topic and the question is not a target for which an answer is to be rejected, the answer acquisition means requests the language model for answers to create an answer based on the question, and acquires the answer from the language model. Information processing system.

4. 4. The information processing system according to claim 3, the answering means, when the acquired output indicates that the question is a target for which an answer is refused, sends information to the user indicating that the question cannot be answered; Information processing system.

5. 2. The information processing system according to claim 1, the answer acquisition means requests the answer language model to create an answer to the question based on the question and the acquired answer basic information, and acquires the answer from the language model. Information processing system.

6. 2. The information processing system according to claim 1, The database generates a feature vector from the question, and acquires answer basis information associated with a feature vector similar to the generated feature vector. Information processing system.

7. 2. The information processing system according to claim 1, the database is a database for a predetermined field, the question determination means requests a determination language model to determine whether the question is related to the predetermined field and the topic, and obtains an output from the determination language model; When the acquired output indicates that the question is related to the predetermined field and the topic, the answer acquisition means requests an answer language model to create an answer to the question based on the acquired answer basic information, and acquires the answer from the language model. Information processing system.

8. an information processing system including one or more processors, obtaining a question based on input from a user; obtaining answer basic information corresponding to the question from a database; requesting a language model for determination to determine whether the question is related to the topic based on information indicating the topic of the answer basic information corresponding to the question, which is obtained from the database, and obtaining an output from the language model for determination; If the acquired output indicates that the question is related to the topic, requesting an answer language model to create an answer to the question based on the acquired answer basic information, and acquiring the answer from the language model; sending information to the user that answers the question based on the obtained answer; An information processing method including:

9. a question acquisition means for acquiring a question based on an input from a user; an information acquisition means for acquiring answer basic information corresponding to the question from a database; a question determination means for requesting a determination language model to determine whether the question is related to the topic based on information indicating the topic of the answer basic information corresponding to the question, the information being obtained from the database, and obtaining an output from the determination language model; an answer acquisition means for requesting an answer language model to create an answer to the question based on the acquired answer basic information when the acquired output indicates that the question is related to the topic, and acquiring the answer from the language model; and a reply means for sending information that answers the question to the user based on the acquired answer; A program that allows a computer to function as a

Images