Information processing system, information processing method, and program

Abstract

To improve the accuracy of a response based on the output of a language model.SOLUTION: An information processing system is configured to: acquire a question on the basis of input from a user; determine whether the question targets a predetermined field; acquire answer basis information corresponding to the question from a database about the predetermined field; when the question is determined to target the predetermined field, request a language model for answering to generate a response to the question on the basis of the answer basis information on the basis of the acquired answer basis information and acquire the response from the language model; and send information for answering the question to the user on the basis of the acquired response.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] The performance of large language models (LLMs) has improved significantly. Large language models can answer questions in natural sentences. Large language models are used in chatbots such as ChatGPT (registered trademark).

[0003] Patent Document 1 discloses a technology for generating a summary sentence desired by a user using a large-scale language model obtained by machine learning of a huge amount of unlabeled text. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2023-73095 Summary of the Invention [Problem to be solved by the invention]

[0005] Although language models such as large-scale language models can output natural-sounding sentences in response to questions, they may output inaccurate answers depending on the field the question is about. On the other hand, it is not easy to train a language model using knowledge of the field.

[0006] The present disclosure provides techniques for improving the accuracy of answers based on the output of a language model. [Means for solving the problem]

[0007] (1) An information processing system including: a question acquisition means for acquiring a question based on an input from a user; a question determination means for acquiring a determination result as to whether the question is related to a predetermined field; an information acquisition means for acquiring answer basic information corresponding to the question from a database for the predetermined field; an answer acquisition means for, when it is determined that the question is related to the predetermined field, requesting an answer language model to create an answer to the question based on the acquired answer basic information and acquiring the answer from the language model; and an answering means for sending information answering the question to the user based on the acquired answer.

[0008] (2) In the information processing system according to (1), when it is determined that the question does not target the specified field, the answer acquisition means requests the answer language model to create an answer based on the question and acquires the answer from the language model.

[0009] (3) In the information processing system of (1), the question determination means determines whether the question is related to a specified field and whether the question is related to a subject for which an answer is rejected, and if it is determined that the question is not related to a specified field and is not related to a subject for which an answer is rejected, the answer acquisition means requests the answer language model to create an answer based on the question and acquires the answer from the language model.

[0010] (4) In the method according to (3), when it is determined that the question is one for which an answer cannot be refused, the answering means sends information to the user indicating that the question cannot be answered. Information processing system.

[0011] (5) In any one of (1) to (4), the question determination means requests a language model for determination to determine whether the question targets the specified field, and determines whether the question targets the specified field based on information output by the language model.

[0012] (6) In (5), the question determination means requests the determination language model to determine whether or not the question targets the specified field based on the specified field and information indicating a topic of the answer basic information corresponding to the question, obtained from the database, and determines whether or not the question targets the specified field based on information output by the language model.

[0013] (7) In any one of (1) to (6), the answer acquisition means requests the answer language model to create an answer to the question based on the question and the acquired answer basic information, and acquires the answer from the language model.

[0014] (8) In any one of (1) to (7), the database generates a feature vector from the question and obtains answer basic information associated with a feature vector similar to the generated feature vector.

[0015] (9) An information processing method including the steps of: acquiring a question based on an input from a user; determining whether the question is related to a predetermined field; acquiring answer basic information corresponding to the question from a database for the predetermined field; if it is determined that the question is related to the predetermined field, requesting an answer language model to create an answer to the question based on the acquired answer basic information and acquiring the answer from the language model; and sending information answering the question to the user based on the acquired answer.

[0016] (10) A program for causing a computer to function as a question acquisition means for acquiring a question based on input from a user, a question determination means for determining whether the question is related to a specified field, an information acquisition means for acquiring answer basic information corresponding to the question from a database for the specified field, an answer acquisition means for requesting an answer language model to create an answer to the question based on the acquired answer basic information when it is determined that the question is related to the specified field, and acquiring the answer from the language model, and an answering means for sending information answering the question to the user based on the acquired answer. Effect of the Invention

[0017] The present invention can improve the accuracy of answers based on the output of a language model. [Brief description of the drawings]

[0018] [Figure 1] FIG. 2 is a diagram illustrating an example of elements related to an inquiry management system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing functions realized by the inquiry management system. [Diagram 3] FIG. 11 is a diagram illustrating an example of data stored in a feature database. [Figure 4] FIG. 2 is a diagram illustrating an example of document data. [Diagram 5] FIG. 13 is a diagram illustrating an example of a screen displayed on a user terminal. [Figure 6] FIG. 11 is a diagram showing another example of a screen displayed on the user terminal. [Figure 7] 13 is a flowchart illustrating an example of a process of an inquiry management system. [Figure 8] 13 is a flowchart illustrating an example of a process of an inquiry management system. [Figure 9] FIG. 13 is a diagram showing an example of an instruction text that the question classification unit inputs to the large-scale language model. [Figure 10]13 is a diagram showing an example of a command text that an answer acquisition unit inputs to a large-scale language model. FIG. [Figure 11] 13 is a diagram showing an example of a command text that an answer acquisition unit inputs to a large-scale language model. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Configurations with the same reference numerals will not be described repeatedly.

[0020] 1 is a diagram showing an example of elements related to a query management system 2 according to an embodiment of the present invention. The query management system 2 receives question information from a user terminal 1 operated by a user, inputs a question sentence based on the information to a large-scale language model service 3, and outputs an answer corresponding to a sentence output by the large-scale language model service 3 to the user terminal 1.

[0021] The user terminal 1 is a computer having a user interface, such as a personal computer or a smartphone.

[0022] The large-scale language model service 3 includes a general-purpose large-scale language model realized by a computer. The large-scale language model service 3 receives an instruction from the query management system 2, and passes the output obtained by inputting the instruction into the large-scale language model to the query management system 2. This instruction is in text format and is also called a prompt. Hereinafter, an instruction in text format is also referred to as an instruction text. This general-purpose large-scale language model is trained with data from a wide range of fields. The large-scale language model service 3 may be, for example, a service such as ChatGPT (registered trademark).

[0023] In the following, when simply described as "large-scale language model", it refers to the large-scale language model included in the large-scale language model service 3, and the query management system 2 executes the process of inputting information to the large-scale language model and acquiring the output from the large-scale language model by using the API provided by the large-scale language model service 3. The query management system 2 does not necessarily input all information with a single API call, and may input information part by part through multiple API calls. The large-scale language model service 3 may be provided in the query management system 2. In this embodiment, the query management system 2 inputs information requesting the large-scale language model to create a response, and acquires the output of the large-scale language model as the response. In the following, inputting information requesting the creation of a response into the large-scale language model is also described as requesting the large-scale language model to create a response.

[0024] The inquiry management system 2 includes one or more computers (e.g., server computers). The inquiry management system 2 includes one or more processors 21, one or more storages 22, and one or more communication units 23. The inquiry management system 2 may include multiple computers each including one or more processors 21, storages 22, and communication units 23, or may include a single computer having one or more processors 21 and storages 22. The inquiry management system 2 may be implemented on one or more virtual server or container platforms.

[0025] The processor 21 operates according to a program (also called an instruction code) stored in the storage 22. The processor 21 also controls the communication unit 23. The processor 21 includes, for example, a CPU (Central Processing Unit), and may further include a GPU (Graphic Processing Unit) and an NPU (Neural Processing Unit). The program may be provided via the Internet or the like, or may be provided by being stored in a computer-readable storage medium such as a flash memory or a DVD-ROM.

[0026] The storage 22 is composed of memory elements such as RAM and flash memory, and external storage devices such as a hard disk drive (HDD) and a solid state drive (SSD). The storage 22 stores the above programs. The storage 22 also stores information input from the processor 21 and the communication unit 23 and calculation results.

[0027] The communication unit 23 is a communication interface, such as a network interface card, for communicating with other devices. The communication unit 23 is composed of an integrated circuit, an antenna, a communication terminal, etc., which realize a wireless LAN or a wired LAN. The communication unit 23 inputs information received from other devices to the processor 21 or the storage 22 via a network under the control of the processor 21, and transmits information to other devices.

[0028] The hardware configuration of the inquiry management system 2 is not limited to the above example. For example, the inquiry management system 2 may include a device for reading a computer-readable information storage medium (e.g., an optical disk drive or a memory card slot) and a device for inputting and outputting data to and from an external device (e.g., a USB port). The external device may be an input device or an output device.

[0029] Next, the functions provided by the inquiry management system 2 will be described. Fig. 2 is a block diagram showing the functions realized by the inquiry management system 2. The inquiry management system 2 functionally includes a management unit 50, a knowledge database 60, and an order management API 65. The management unit 50 functionally includes a question sentence acquisition unit 51, a question classification unit 52, a knowledge acquisition unit 53, an answer acquisition unit 54, an answer correction unit 55, a consistency determination unit 56, and an answer output unit 57. The knowledge database 60 functionally includes a search engine 61 and a knowledge document storage unit 62. The management unit 50, the knowledge database 60, and the order management API 65 are realized by the processor 21 executing programs corresponding to the respective functions stored in the storage 22 and controlling the communication unit 23 and the like.

[0030] The knowledge database 60 is a database in which knowledge information of a predetermined knowledge field is stored. The knowledge database 60 acquires information indicating a question from a user from the management unit 50, searches for knowledge information corresponding to the information, and transfers the knowledge information found by the search to the management unit 50. The knowledge database 60 has a configuration different from that of the large-scale language model of the large-scale language model service 3. The knowledge information may include a knowledge document and a character string indicating a topic of the knowledge document. The knowledge document storage unit 62 stores the knowledge document as knowledge information belonging to a certain knowledge field. The knowledge document may be, for example, document data including a set of a question and an answer in an FAQ. Here, a question in an FAQ may be treated as a topic of the knowledge document. The knowledge information may be something other than a document. The knowledge information including the knowledge document is used as basic information (basic answer information) when the large-scale language model creates an answer to a question.

[0031] The search engine 61 acquires information indicating a question from a user, and searches for knowledge information corresponding to the search criteria. The information indicating the question may be an acquired question sentence. The search engine 61 includes a feature extraction model, which is a machine learning model, and a feature database that stores feature vectors extracted from each piece of knowledge information. The search engine 61 inputs the information indicating the question into the feature extraction model, and acquires the feature vector output from the feature extraction model as a query vector. The search engine 61 searches the feature database for a feature vector similar to the query vector. The search engine 61 outputs the similar feature vector. The search engine 61 may calculate, for example, a cosine similarity between the query vector and the feature vector in the feature database as the similarity, select the feature vector with the largest similarity as a similar feature vector, and acquire knowledge information corresponding to the selected feature vector. Here, a feature vector is extracted for each document that is knowledge information.

[0032] Fig. 3 is a diagram showing an example of data stored in the feature database. In the example of Fig. 3, the feature vector is stored in association with the document ID of the knowledge document, the title of the knowledge document (corresponding to the question and topic of the FAQ), the API name, and the document use. The meanings of the API name and the document use will be described later.

[0033] FIG. 4 is a diagram showing an example of a knowledge document. In FIG. 4, only the part of the knowledge document that corresponds to the answer to the FAQ is shown. This knowledge document is associated with the feature vector of the line with the title "How to cancel an order" in FIG. 3. Also, in FIG. 4, some of the actual content has been omitted. The number of characters in each knowledge document is greater than the number of characters that the inquiry management system 2 is capable of outputting as an answer to a question.

[0034] Here, the feature extraction model is trained using document data for training. During training, the feature extraction model extracts a feature vector of a query and a feature vector of document data corresponding to the query. The feature extraction model is trained so that the similarity between the feature vector of the query and the feature vector of the corresponding document data increases and the similarity between the non-corresponding document data decreases. The document data in training may be only parts of the knowledge document that correspond to topics or questions. This training may be performed based on the methods shown in the following two papers:

[0035] Paper 1: Wataru Sakata, Tomohide Shibata, Ribeka Tanaka, and Sadao Kurohashi. 2019. FAQ Retrieval using Query-Question Similarity and BERT-Based Query-Answer Relevance. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR'19). Association for Computing Machinery, New York, NY, USA, 1113-1116.

[0036] Paper 2: Seo, J.; Lee, T.; Moon, H.; Park, C.; Eo, S.; Aiyanyo, ID; Park, K.; So, A.; Ahn, S.; Park, J. Dense-to-Question and Sparse-to-Answer: Hybrid Retriever System for Industrial Frequently Asked Questions. Mathematics 2022, 10, 1335.

[0037] The order management API 65 is an API for acquiring information specific to a user. The order management API 65 may include, for example, an API for acquiring the current status (e.g., payment completed, shipped, delivery completed) of an order that a user has previously placed with the electronic commerce system.

[0038] The management unit 50 acquires information about a question from the user terminal 1, and acquires information related to the question from the knowledge database 60. The management unit 50 also generates an answer to the question using the large-scale language model service 3, and outputs the answer to the user terminal 1 via the network.

[0039] The question sentence acquisition unit 51 acquires a question sentence based on an input from a user. The input from the user is specifically question information that is input by the user operating the user terminal 1 and received from the user terminal 1. The question information may be in the form of a sentence or a keyword. The question sentence acquisition unit 51 may acquire the question information input in the form of a sentence as it is as a question sentence, or may generate a question sentence by processing a plurality of keywords or sentences included in the question information.

[0040] Fig. 5 is a diagram showing an example of a screen displayed on the user terminal 1. Fig. 5 shows a screen for inputting a question in a chat format. In the example of Fig. 5, the question itself is input by the user.

[0041] The question acquisition unit 51 may input a prompt to the large-scale language model to create a question from multiple keywords or sentences acquired as question information, and acquire the output of the large-scale language model in response to the prompt as a question. Here, it is preferable that the information input to the large-scale language model service 3 does not include personal information. Therefore, the question acquisition unit 51 may input information from which personal information has been removed in advance to the large-scale language model.

[0042] The question classification unit 52 acquires a determination result as to whether the acquired question sentence can be answered only by the large-scale language model, whether the question sentence is targeted at a predetermined knowledge field, and whether the question sentence is a target for which an answer is rejected. The predetermined knowledge field is a knowledge field to which the knowledge information stored in the knowledge database 60 belongs, and is a knowledge field to which the inquiry management system 2 can answer. Here, the question classification unit 52 requests the large-scale language model to determine whether the acquired question sentence belongs to the knowledge field based on the acquired question sentence and the answerable knowledge field, and acquires a determination result from the large-scale language model. The question classification unit 52 may request the large-scale language model to determine whether the question sentence belongs to the knowledge field based on information indicating a topic of the answer basic information corresponding to the question acquired from the database. Note that the question classification unit 52 may simply request a determination whether the question is targeted at a predetermined field, may simply request a determination whether the question can be answered only by the large-scale language model, or may request a determination whether the question belongs to any two of the above three categories.

[0043] In this embodiment, this determination is a determination as to whether the question sentence is classified into a specific category among a plurality of categories (types). The plurality of categories include a category (corresponding to the specific category described above) indicating that the question sentence belongs to a predetermined knowledge field (a knowledge field that can be answered), a category indicating that an answer is to be provided only using a large-scale language model, and a category indicating that the question sentence is a target for which an answer is to be refused. Specifically, the requested information may be the text of a requested command.

[0044] The knowledge acquisition unit 53 transmits the question as a query to a search engine 61 of a knowledge database 60, and acquires from the knowledge database 60 knowledge information that is stored in a knowledge document storage unit 62 and searched for by the question.

[0045] When it is determined that the question text is targeted at a predetermined knowledge field, the answer acquisition unit 54 inputs information requesting the creation of an answer based on the question text and the acquired knowledge information into a general-purpose large-scale language model, and acquires a sentence output from the large-scale language model. Hereinafter, the sentence acquired by the answer acquisition unit 54 from the large-scale language model is referred to as a model answer text. Note that the number of characters in the model answer text is smaller than the number of characters in the knowledge information, and the request for answer creation is a kind of request for summary. The information requesting the creation of an answer does not have to include a question text, but there is a high probability that a more appropriate model answer text can be obtained if the information includes a question text.

[0046] Here, when it is determined that the question does not target a predetermined knowledge field and that the question is not a target for which an answer is rejected, the answer acquisition unit 54 requests the general-purpose large-scale language model to create an answer based on the question, and acquires a model answer sentence from the output of the large-scale language model. The answer acquisition unit 54 does not need to use whether or not the question is a target for which an answer is rejected. For example, when the multiple categories to be classified by the question classification unit 52 do not include a category indicating that the question is a target for which an answer is rejected, the answer acquisition unit 54 may request the general-purpose large-scale language model to create an answer based on the question when the question does not target a predetermined knowledge field, and acquire a model answer sentence from the large-scale language model. Also, when it is determined that the question is a target for which an answer is rejected, the answer acquisition unit 54 requests the general-purpose large-scale language model to create an answer indicating that the question cannot be answered, and acquires a model answer sentence indicating that the question cannot be answered.

[0047] The answer correction unit 55 corrects a specific character string included in the model answer sentence. For example, the answer correction unit 55 may filter a character string of a hyperlink included in the model answer sentence and decode a specific character string included in the model answer sentence. In decoding a specific character string, when the model answer sentence includes a character string in which personal information is encoded, the answer correction unit 55 returns the character string to the original personal information.

[0048] The consistency determination unit 56 determines whether the acquired or corrected model answer sentence is consistent with the acquired knowledge information. Here, the consistency determination unit 56 determines the consistency based on whether each of the multiple words included in the model answer sentence is included in the words included in the knowledge information. The method of judging the consistency will be described later.

[0049] The answer output unit 57 generates an answer to be output to the user based on the result of the consistency determination and the model answer sentence. If the answer output unit 57 determines that the model answer sentence is consistent with the knowledge information, it generates an answer including the model answer sentence and information (e.g., a hyperlink) that identifies the knowledge information. If the answer output unit 57 determines that the model answer sentence is not consistent with the knowledge information, it generates an answer including the model answer sentence and information indicating low reliability of the answer. Furthermore, the answer output unit 57 outputs the generated answer to the user terminal 1 of the user.

[0050] Fig. 6 is a diagram showing another example of a screen displayed on the user terminal 1. Fig. 6 shows an example of an answer created when the knowledge document shown in Fig. 4 is searched for as knowledge information. In the example of Fig. 6, the consistency determination unit 56 determines that the model answer sentence is consistent with the knowledge information, and an answer including a hyperlink is output.

[0051] Next, a more detailed description will be given of the processing of the inquiry management system 2. Figures 7 and 8 are flowcharts showing an example of the processing of the inquiry management system 2. The processing of the management unit 50 is mainly described in Figures 7 and 8.

[0052] First, the question acquisition unit 51 of the management unit 50 acquires a question based on question information input from the user terminal 1 (S101). The question information is input via a network. The question information may be text including a sentence or a keyword, or may be text input via a so-called input completion function. The question acquisition unit 51 may acquire question information input in a sentence format as it is as a question, or may generate a question from an output when a command text for generating a question that is aware of a context from information of a plurality of related questions input in a dialogue format is input to a large-scale language model. In this case, the question acquisition unit 51 may encode a character string of personal information among the information of the question from the user and replace it with another character string, and input a command text including the replaced question information to the large-scale language model.

[0053] Next, the question classification unit 52 classifies the acquired question sentence into one of a plurality of categories (S102). In the example of FIGS. 7 and 8, the plurality of categories include category 1, category 2, and category 3. Category 1 is a type of question sentence that does not belong to an answerable knowledge field. Category 2 is a type of question sentence that belongs to an answerable knowledge field. Category 3 is a type of question sentence that does not belong to an answerable knowledge field, particularly a type of question sentence that does not allow an answer.

[0054] More specifically, the question categorization unit 52 inputs an instruction text for which classification is requested to the large-scale language model. The instruction text is a character string that categorizes the acquired question sentence into a category based on the acquired question sentence, the knowledge field that can be answered, and the topic corresponding to the question sentence.

[0055] FIG. 9 is a diagram showing an example of an instruction text that the question classification unit 52 inputs to the large-scale language model. In FIG. 9, {knowledge_fields} and {topic} are written, but in reality, they are replaced with text indicating a knowledge field that can be answered and a topic corresponding to a question, respectively. The topic is obtained by the question classification unit 52 from a response of the knowledge database 60 when a question is sent as a query to the knowledge database 60. According to the instruction text shown in FIG. 9, prompts, telephone numbers, website URLs, topics of immoral questions, and questions about the disadvantages of a particular company are classified into category 3. Questions about knowledge fields and topics are classified into category 2. Greetings, complaints, and questions to the large-scale language model itself are classified into category 1. Categories 1 and 3 may be classified into other categories.

[0056] The answerable knowledge field (corresponding to a predetermined knowledge field) is predefined according to the purpose of the inquiry management system 2 and the knowledge field to which the knowledge document in the knowledge database 60 belongs. The topic is dynamically determined according to the question sentence. More specifically, before inputting the command text to the large-scale language model, the question classification unit 52 sends the question sentence as a query to the search engine 61 of the knowledge database 60, and acquires the title of a knowledge document similar to the query from the search engine 61. The question classification unit 52 incorporates the acquired title as a topic into the command text, and inputs it to the large-scale language model. The command text including the topic is sent to the large-scale language model here in order to improve the judgment accuracy in cases where it is difficult to judge only by the character string of the answerable knowledge field. For example, this is effective in cases where a question about a credit card being fraudulently used is answered by answering how to contact an e-commerce store, rather than the general answer of contacting the credit card company. In this case, knowledge of the knowledge field is required depending on the context.

[0057] When the question is classified, the question classification unit 52 determines whether the classified category indicates that knowledge in the knowledge field is necessary to answer the question (S103). Specifically, the question classification unit 52 determines whether the classified category is classified into category 2.

[0058] If the classified category indicates that knowledge in that knowledge field is unnecessary (N in S103), the question classification unit 52 or the answer acquisition unit 54 determines whether the classified category indicates that an answer is rejected (S104). If it does not indicate that an answer is rejected (N in S104), the answer acquisition unit 54 causes the large-scale language model to generate an answer sentence based on the question sentence (S105). Specifically, the answer acquisition unit 54 inputs command text for generating an answer sentence based on the question sentence to the large-scale language model, and acquires the answer sentence output from the large-scale language model. This command text does not include knowledge information (particularly knowledge documents), and may be according to the category.

[0059] Fig. 10 is a diagram showing an example of command text input by the answer acquisition unit 54 to the large-scale language model. The command text shown in Fig. 10 includes the descriptions {name} and {assistant}, but in reality, these are replaced with the name of the chatbot and the name of the business type handled by this inquiry management system 2, respectively. The command text shown in Fig. 10 does not include information from the knowledge database 60, and requests the large-scale language model to generate a response sentence without using that information. The response sentence is generated from generic knowledge included in the large-scale language model.

[0060] Then, the answer output unit 57 outputs the generated answer sentence to the user terminal 1 (S106), and the process shown in FIGS.

[0061] On the other hand, if the classified category indicates that an answer is rejected (Y in S104), the answer acquisition unit 54 generates an answer sentence indicating that the question cannot be answered (S107). In S106, the answer output unit 57 outputs the generated answer sentence to the user terminal 1. The generated answer sentence may be a fixed phrase stored in advance in the storage 22, or may be a sentence acquired by the answer acquisition unit 54 requesting the large-scale language model to generate an answer. The answer acquisition unit 54 may also request the large-scale language model to generate an answer sentence by using a command text including a request sentence equivalent to S104, and acquire the answer sentence from the large-scale language model.

[0062] On the other hand, if the classified category indicates that knowledge of the knowledge field is required (Y in S103), the process of S108 is executed. In S108, if dynamic information about the user is required, the knowledge acquisition unit 53 acquires the dynamic information about the user via the order management API 65. Whether or not dynamic information about the user is required is determined based on whether or not an API name exists, which is acquired as a response to the question sent by the knowledge acquisition unit 53 to the search engine 61 of the knowledge database 60 as a query. If the API name exists, the knowledge acquisition unit 53 calls an API included in the order management API 65 and corresponding to the API name, together with a parameter including information identifying the user. The API name may be the one acquired when acquiring a topic from the knowledge database 60 in S102. For example, according to the example of FIG. 3, if the question is similar to a title that requests information about the status of an order, the order status API is called.

[0063] After S108, the process of S109 is executed. In S109, when a knowledge document is required as knowledge information, the knowledge acquisition unit 53 acquires a knowledge document corresponding to the question, output from the knowledge database 60. This knowledge document is acquired as a response to the question sent as a query to the search engine 61 of the knowledge database 60. According to the example of FIG. 3, when the question inquires about a method of canceling an order, the knowledge document with the document ID of 000009739 shown in FIG. 4 is acquired. Here, the knowledge acquisition unit 53 does not acquire a knowledge document when the document use is not true. Instead of using the document, the knowledge acquisition unit 53 may simply determine whether or not to acquire a knowledge document based on whether or not a document ID exists. A case in which a knowledge document is not acquired is, for example, a case in which an answer can be generated from information acquired from the order management API 65 and the question, such as simply checking the status of an order.

[0064] After the processes of S108 and S109 are completed, the answer acquisition unit 54 inputs the command text for generating an answer sentence to the large-scale language model, and acquires the output as a model answer sentence (S110). The command text includes a command for generating an answer sentence from a question sentence and dynamic information or a knowledge document.

[0065] FIG. 11 is a diagram showing an example of an instruction text input by the answer acquisition unit 54 to the large-scale language model. In the example of FIG. 11, as in the example of FIG. 9, the descriptions {knowledge_fields} and {topic} are actually replaced with text indicating the knowledge field that can be answered, and a topic corresponding to the question sentence, respectively. Furthermore, the descriptions {name} and {assistant} in FIG. 10 are actually replaced with the name of the chatbot and the name of the business type that this inquiry management system 2 handles, respectively. {observation_data} is replaced with text of a knowledge document or dynamic information. Although the example of FIG. 11 does not include a question sentence, it is included in the instruction text (not shown) that is input in advance.

[0066] 11, the command text includes a command to generate a model answer sentence without using information held by the large-scale language model when knowledge information such as a knowledge document is sufficient, and a command to generate a model answer sentence of 400 characters or less. Therefore, the large-scale language model generates and outputs a model answer sentence consisting of a character string of the specified number of characters or less.

[0067] When the model answer sentence is acquired, the answer correction unit 55 corrects a specific character string included in the model answer sentence (S111). Specifically, if the model answer sentence contains a character string of a hyperlink, the answer correction unit 55 deletes the character string. In addition, if the model answer sentence contains a character string in which personal information is encoded, the answer correction unit 55 returns the character string to the original personal information.

[0068] Then, the consistency determination unit 56 determines whether or not there is consistency between the model answer sentence and the knowledge document (S112). Here, the consistency determination unit 56 extracts a plurality of words (e.g., verbs and nouns) that satisfy a condition from the model answer sentence by morphological analysis, and determines whether or not there is consistency based on whether or not each of these words is included in the knowledge document. More specifically, the consistency determination unit 56 may determine that there is no consistency when the ratio of words that are not included in the knowledge document among these words exceeds a threshold, and may determine that there is consistency when the ratio is equal to or less than the threshold.

[0069] Then, if it is determined that there is consistency (Y in S112), the answer output unit 57 generates an output statement in which information identifying the knowledge document is added to the model answer statement, and outputs the output statement to the user terminal 1 (S113). The information identifying the knowledge document may be, for example, a hyperlink to the knowledge document, or a character string indicating a method for referencing the knowledge document. On the other hand, if it is determined that there is no consistency (N in S112), the answer output unit 57 generates an output statement in which a warning statement is added to the model answer statement, and outputs the output statement to the user terminal 1 (S114). This warning statement is information indicating that the reliability of the answer statement is low, and more specifically, may be a statement indicating that the answer statement is not necessarily correct and that confirmation is required. The user terminal 1 receives the output statement output in S113 or S114, and presents it to the user.

[0070] If the command text does not include a knowledge document in S110, the process of S112 is not executed, and the answer output unit 57 may output the model answer sentence as is.

[0071] In this embodiment, before the large-scale language model generates an answer to a question, the inquiry management system 2 acquires a knowledge document that is the source of the answer from a knowledge database 60 that stores knowledge information on a specific knowledge field. Then, the large-scale language model generates a draft answer sentence (i.e., a model answer sentence) using the question sentence and the knowledge document. This makes it possible to improve the accuracy of the answer output from the large-scale language model even if the answering ability of the large-scale language model in the field to be answered is insufficient.

[0072] The inquiry management system 2 checks the consistency between the model answer sentence and the original knowledge information. This check of consistency makes it possible to deal with cases where the model answer sentence gives an answer that is unrelated to the question, and prevents factual errors by the user.

[0073] In this embodiment, the large-scale language model used by the question acquisition unit 51 to generate a question sentence, the large-scale language model for determining the acquired question sentence by the question classification unit 52, and the large-scale language model for generating a model answer sentence are the same, but different large-scale language models may be used. For example, the large-scale language model for determining the question sentence and the large-scale language model for generating a model answer sentence may be different.

[0074] In this embodiment, even if it is determined that there is no consistency, only a warning message is added, but an answer sentence may be generated by other methods. For example, instead of the model answer sentence and the warning message, the answer output unit 57 may output, as an answer sentence, a link to a plurality of similar knowledge information items output by the knowledge database 60 in response to the question sentence and a fixed phrase.

[0075] In this embodiment, a large-scale language model is used, but there is no particular limitation on the scale of its implementation and the number of parameters. The present invention can be applied to a machine learning model (language model) that handles natural language. [Explanation of symbols]

[0076] 1 User terminal, 2 Inquiry management system, 3 Large-scale language model service, 21 Processor, 22 Storage, 23 Communication unit, 50 Management unit, 51 Question sentence acquisition unit, 52 Question classification unit, 53 Knowledge acquisition unit, 54 Answer acquisition unit, 55 Answer correction unit, 56 Consistency determination unit, 57 Answer output unit, 60 Knowledge database, 61 Search engine, 62 Knowledge document storage unit, 65 Order management API.

Claims

1. A question acquisition means for acquiring a question based on an input from a user; a question determination means for obtaining a determination result as to whether the question is related to a predetermined field; an information acquisition means for acquiring answer basic information corresponding to the question from a database for the predetermined field; an answer acquisition means for, when it is determined that the question is targeted at the predetermined field, requesting an answer language model to create an answer to the question based on the acquired answer basic information, and acquiring the answer from the language model; a reply means for sending information answering the question to the user based on the acquired answer; An information processing system comprising:

2. 2. The information processing system according to claim 1, when it is determined that the question does not target the predetermined field, the answer acquisition means requests the language model for answers to create an answer based on the question, and acquires the answer from the language model. Information processing system.

3. 2. The information processing system according to claim 1, The question determination means determines whether the question is related to a predetermined field and whether the question is related to a field to which an answer is refused; when it is determined that the question does not target a predetermined field and that the question is not a subject for which an answer is to be rejected, the answer acquisition means requests the language model for answers to create an answer based on the question and acquires the answer from the language model. Information processing system.

4. 4. The information processing system according to claim 3, When it is determined that the question is one for which an answer is refused, the answering means sends information to the user indicating that the question cannot be answered. Information processing system.

5. 2. The information processing system according to claim 1, the question determination means requests a language model for determination as to whether the question is targeted at the predetermined field, and determines whether the question is targeted at the predetermined field based on information output by the language model; Information processing system.

6. 6. The information processing system according to claim 5, the question determination means requests the language model for determination to determine whether or not the question is targeted at the predetermined field, based on the predetermined field and information indicating a topic of the answer basic information corresponding to the question, which is acquired from the database, and determines whether or not the question is targeted at the predetermined field, based on information output by the language model; Information processing system.

7. 2. The information processing system according to claim 1, the answer acquisition means requests the answer language model to create an answer to the question based on the question and the acquired answer basic information, and acquires the answer from the language model. Information processing system.

8. 2. The information processing system according to claim 1, The database generates a feature vector from the question, and obtains answer basis information associated with a feature vector similar to the generated feature vector. Information processing system.

9. obtaining a question based on input from a user; determining whether the query is directed to a predetermined subject area; obtaining answer basic information corresponding to the question from a database for the predetermined field; when it is determined that the question is targeted at the predetermined field, requesting an answer language model to create an answer to the question based on the acquired answer basic information, and acquiring the answer from the language model; sending information to the user answering the question based on the obtained answer; An information processing method comprising:

10. A question acquisition means for acquiring a question based on an input from a user; a question determination means for determining whether the question is directed to a predetermined field; an information acquisition means for acquiring answer basic information corresponding to the question from a database for the predetermined field; an answer acquisition means for, when it is determined that the question is targeted at the predetermined field, requesting an answer language model to create an answer to the question based on the acquired answer basic information, and acquiring the answer from the language model; and a reply means for sending information answering the question to the user based on the acquired reply; A program that makes a computer function as a

Images