Question response apparatus, method, and program

Abstract

To provide a question response apparatus, method, and program with which the accuracy of answers is improved.SOLUTION: A question response apparatus of an embodiment comprises an acquisition unit, a query generation unit, a search unit, a consistency determination unit, an integration unit, and an answer generation unit. The acquisition unit acquires a question inputted from a user. The query generation unit generates a query on the basis of the question. The search unit retrieves search results related to the query from a plurality of information sources. The consistency determination unit determines consistency of the search results. The integration unit generates an integrating prompt in which a search result determined as having consistency and the question are integrated. The answer generation unit generates an answer to the question by inputting the integrating prompt to a large language model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to a question answering device, method, and program. [Background technology]

[0002] In recent years, it has become possible to train large language models (LLMs) using large amounts of text data, making it possible for these models to answer questions in natural language. Systems using these large language models use pre-built knowledge bases or databases to search for information related to the input question, and then input the question and the searched information into the large language model to generate an appropriate answer. Such systems infer what actions should be taken to answer the question, execute the appropriate actions to obtain information related to the question, and then input the question and the retrieved information into the large language model to generate an appropriate answer.

[0003] Large-scale language models can answer questions about knowledge they have learned, but cannot answer questions about knowledge they have not learned. This can lead to a phenomenon known as hallucination, where large-scale language models generate answers that are counter to the facts.

[0004] To accurately answer questions about unlearned knowledge, one technique is to combine search results from multiple sources. For example, suppose you want to benchmark your company's image anomaly detection technology against other companies' technologies. In this case, an internal information search can obtain detailed information about your company's image anomaly detection technology, but not about other companies' technologies. Furthermore, it is not appropriate to answer that there are no similar technologies from other companies based on internal information search results; you must also search information on the Internet to generate an answer. On the other hand, an Internet search cannot obtain unpublished information about your company's image anomaly detection technology, but it can obtain information about similar technologies from other companies. You must generate a benchmark answer by combining the search results for your company's image anomaly detection technology obtained from an internal information search with the search results for similar technologies from other companies obtained from an Internet search.

[0005] However, when querying a large-scale language model using search results from multiple sources, simply combining the search results from multiple sources can result in a poor answer quality. This is because an appropriate answer cannot be generated if the search results are inconsistent. For this reason, there is a need to improve the answer accuracy of large-scale language models by using information from multiple sources. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Lewis Patrick et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, 2020 [Non-patent document 2] Shunyu Yao et al., “SYNERGIZING REASONING AND ACTING IN LANGUAGE MODELS”, 2022 Summary of the Invention [Problem to be solved by the invention]

[0007] The problem to be solved by the present invention is to provide a question answering device, method, and program with improved answer accuracy. [Means for solving the problem]

[0008] In order to solve such problems, a question answering device of an embodiment includes an acquisition unit, a query generation unit, a search unit, a consistency determination unit, an integration unit, and an answer generation unit. The acquisition unit acquires a question input by a user. The query generation unit generates a query based on the question. The search unit searches for search results related to the query from multiple information sources. The consistency determination unit determines the consistency of the search results. The integration unit generates an integrated prompt by integrating search results determined to be consistent with the question. The answer generation unit generates an answer to the question by inputting the integrated prompt into a large-scale language model. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a question-answering system according to a first embodiment. [Figure 2] FIG. 1 is a diagram showing an example of the configuration of a question answering device according to a first embodiment; [Figure 3] 1 is a flowchart illustrating a processing procedure of a question answering process according to a first embodiment; [Figure 4] FIG. 1 is a diagram showing a data flow in a question-answering process according to the first embodiment; [Figure 5] 10 is a flowchart illustrating a processing procedure for consistency determination processing according to the first embodiment; [Figure 6] 10 is a flowchart illustrating a processing procedure for consistency determination processing according to a modified example. [Figure 7] FIG. 10 is a diagram for explaining a consistency determination method according to a modified example. [Figure 8] FIG. 10 is a diagram showing an example of the configuration of a question answering device according to a second embodiment. [Figure 9]10 is a flowchart illustrating a processing procedure of a question answering process according to the second embodiment; [Figure 10] FIG. 10 is a diagram showing a data flow in a question-answering process according to a second embodiment. [Figure 11] FIG. 11 is a diagram showing a data flow in a question-answering process according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of a question answering device, method, and program will be described in detail with reference to the drawings. In the following description, components having substantially the same functions and configurations are designated by the same reference numerals, and redundant description will be given only when necessary.

[0011] (First embodiment) Fig. 1 is a diagram showing the configuration of a question answering system 1 including a question answering device 100 according to a first embodiment. The question answering system 1 is an interactive computer network system that uses a large-scale language model and searches information sources for information that matches a user's question and presents the search results to the user. As shown in Fig. 1, the question answering system 1 includes the question answering device 100, a client terminal 200, a first information source 300, and a second information source 400.

[0012] The question answering device 100 is connected to a client terminal 200, a first information source 300, and a second information source 400 via a network or the like. The network is, for example, a LAN (Local Area Network). The connection to the network may be a wired connection or a wireless connection. The network is not limited to a LAN, and may be the Internet, a public communication line, or the like.

[0013] The client terminal 200 is a computer used by a user of the question answering system 1. The client terminal 200 has, as hardware, a processor, an input device, a display device, and a communication device, and functions as a user interface for the question answering system 1. For example, the client terminal 200 accepts a question input by a user via the input device. The question is, for example, text data, and can also be called an inquiry from the user. The question may be input as natural language such as a question sentence, or as a single word such as a search word. The client terminal 200 transmits the question input by the user to the question answering device 100.

[0014] Furthermore, the client terminal 200 receives an answer to the question from the question answering device 100 and displays the received answer on a display device. The answer is text data, and can also be called a response to the inquiry from the user.

[0015] The first information source 300 and the second information source 400 are information sources capable of searching for the target data. The first information source 300 and the second information source 400 may be different types of information sources containing different information. The first information source 300 and the second information source 400 may be, for example, an in-house database, an academic society website, a research paper database, the Internet, or a social networking service (SNS). Simply combining the search results of the first information source 300 and the second information source 400 may result in hallucination.

[0016] Furthermore, a highly reliable information source may be used as the first information source 300, and a highly searchable information source may be used as the second information source 400. Examples of highly reliable information sources include an in-house database, an academic society's website, and a paper database. Information on the Internet may also be used as a highly reliable information source. An information source with high searchability is, for example, an information source with a large amount of information. Examples of information sources with high searchability are the Internet, SNS (Social Networking Service), etc. Information sources with high searchability may have low reliability.

[0017] The question answering device 100 functions as a server of the question answering system 1. The question answering device 100 combines information acquired from the first information source 300 and the second information source 400 to answer a question from a user. Specifically, the question answering device 100 receives a question entered by a user from the client terminal 200, searches for information related to the question from the first information source 300 and the second information source 400 based on the received question, generates an answer to the question using the search results, and transmits the generated answer to the client terminal 200. The question answering device 100 may also be called a question answering processing device. Furthermore, the question answering system 1 including the question answering device 100 may also be called a question answering processing system.

[0018] 2 is a diagram showing the configuration of the question answering device 100. The question answering device 100 is a computer having a processing circuit 11, a storage device 12, an input device 13, a communication device 14, and a display device 15. Data communication between the processing circuit 11, the storage device 12, the input device 13, the communication device 14, and the display device 15 is performed via a bus. The input device 13 and the display device 15 do not necessarily have to be provided.

[0019] The processing circuit 11 has a processor such as a CPU (Central Processing Unit) and a memory such as a RAM (Random Access Memory). The processing circuit 11 has an acquisition unit 111, a query generation unit 112, a search unit 113, a consistency determination unit 114, an integration unit 115, and an answer generation unit 116. By executing a program, the processing circuit 11 realizes the acquisition function, query generation function, search function, consistency determination function, integration function, and answer generation function of each of the above units.

[0020] The storage device 12 is configured by a ROM (Read Only Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), an integrated circuit storage device, etc. The storage device 12 stores programs and the like.

[0021] The program is stored in a non-transitory computer-readable recording medium such as the storage device 12. The program may be implemented as a single program that describes all the functions of the above-mentioned units, or as multiple modules divided into several functional units. The above-mentioned units may also be implemented by an integrated circuit such as an Application Specific Integrated Circuit (ASIC). In this case, they may be implemented in a single integrated circuit, or may be implemented individually in multiple integrated circuits. The above-mentioned units may also be implemented in a single device, or may be distributed across multiple devices. For example, some or all of the above-mentioned units may be implemented in the client terminal 200.

[0022] Various commands are input from an operator via the input device 13. A keyboard, a mouse, various switches, a touch pad, a touch panel display, etc. can be used as the input device 13. An output signal from the input device 13 is supplied to the processing circuit 11.

[0023] The communication device 14 is an interface for performing data communication with an external device connected to the question answering device 100 via a network. For example, the communication device 14 performs data communication with the first information source 300 and the second information source 400.

[0024] The display device 15 displays various information. As the display device 15, a CRT (Cathode-Ray Tube) display, a liquid crystal display, an organic EL (Electro Luminescence) display, an LED (Light-Emitting Diode) display, a plasma display, or any other display known in the art can be appropriately used. The display device 15 may also be a projector.

[0025] Next, the functions executed by each unit of the processing circuit 11 will be described in detail.

[0026] The acquisition unit 111 acquires a question input by a user from the client terminal 200.

[0027] The query generation unit 112 generates a query for searching for information related to the acquired question based on the acquired question. The query is used to search the first information source 300 and the second information source 400. The query is a keyword extracted from the question or an embedded expression of the question. The query generation unit 112 generates the query by, for example, querying a large-scale language model for keywords of the question. For example, the query generation unit 112 may generate an embedded expression as a query to be used for searching the first information source 300, and generate a keyword as a query to be used for searching the second information source 400.

[0028] In this way, query generation unit 112 may generate a query common to multiple information sources, or may generate a different query for each information source. Query generation unit 112 of the present embodiment is an example of a first query generation unit and a second query generation unit.

[0029] The search unit 113 executes a search using a query on multiple information sources, and searches each of the information sources for information related to the query. In this case, the search unit 113 searches each of the first information source 300 and the second information source 400 for information related to the query. For example, if the query is a keyword, the search unit 113 searches the information sources for text data or document data containing the keyword. Furthermore, if the query is an embedded expression, the search unit 113 generates an embedded expression for each piece of information included in the information source, and searches for information whose embedded expression is similar to the query. Hereinafter, the search results from the first information source 300 will be referred to as the first search result, and the search results from the second information source 400 will be referred to as the second search result.

[0030] The consistency determination unit 114 checks the consistency of multiple search results retrieved from different information sources. Specifically, the consistency determination unit 114 checks the consistency of a first search result retrieved from the first information source 300 and a second search result retrieved from the second information source 400. The consistency determination unit 114 may also be called a consistency checking unit.

[0031] For example, the consistency determination unit 114 determines the consistency of the search results using a consistency score between the search results. The consistency score is an index indicating the degree of consistency between the first search result and the second search result. The consistency score may also be called the consistency degree. For example, the cosine similarity between the first search result and the second search result may be used as the consistency score. In this case, the consistency determination unit 114 determines that the search results are consistent if the cosine similarity is equal to or greater than a threshold, and determines that the search results are inconsistent if the cosine similarity is less than the threshold.

[0032] The consistency score may also be calculated using a large-scale language model or a machine learning model. In this case, for example, the first search result, the second search result, and the sentence "Calculate the consistency score that indicates the consistency between these two pieces of information. The consistency score is a value between 0.0 and 1.0, and the closer to 1.0, the higher the consistency" are input to the large-scale language model, and the value output from the large-scale language model is used as the consistency score.

[0033] Alternatively, the consistency determination unit 114 may determine consistency using a large-scale language model or a machine learning model. In this case, the consistency determination unit 114 inputs a question inquiring whether the search results are consistent, together with the first search result and the second search result, to the large-scale language model, and determines consistency based on the output of the large-scale language model.

[0034] The integrating unit 115 generates an integrated prompt by integrating the question with the search results whose consistency has been confirmed. At this time, the integrating unit 115 generates an integrated prompt by integrating the question with a combination of the first search results whose consistency has been determined and the second search results whose consistency has been determined. The integrated prompt is used as a question sentence that the answer generating unit 116 inputs to the large-scale language model.

[0035] The answer generation unit 116 generates an answer to the user's question by inputting the integrated prompt into the large-scale language model. For example, the answer generation unit 116 queries the large-scale language model for the integrated prompt to generate an answer. The generated answer is sent to the client terminal 200 and displayed on a display device of the client terminal 200.

[0036] (Question-answering processing) Next, the operation of the question answering process executed by the question answering device 100 will be described. FIG. 3 is a flowchart showing an example of the procedure of the question answering process. FIG. 4 is a diagram showing an example of the data flow in the question answering process. The question answering process is a process of generating an answer to a question entered by a user at the client terminal 200. The processing circuitry 11 starts the question answering process based on receiving a question entered by the user from the client terminal 200.

[0037] The processing procedures for each process described below are merely examples, and each process can be modified as appropriate as possible. Furthermore, steps can be omitted, replaced, or added as appropriate for the processing procedures described below depending on the embodiment.

[0038] (Step S101) In the question answering process, first, the acquisition unit 111 acquires a question received from the client terminal 200 as a user's question.

[0039] (Step S102) Next, the query generation unit 112 generates a query using the question acquired in step S101. At this time, the query generation unit 112 generates a first query to be used for searching the first information source and a second query to be used for searching the second information source. The first query and the second query may be the same or different.

[0040] (Step S103) Next, the search unit 113 searches for information related to the question from multiple information sources using the query generated in step S102. In this case, the search unit 113 searches the first information source 300 for information related to the first query and acquires the searched information as a first search result. The search unit 113 also searches the second information source 400 for information related to the second query and acquires the searched information as a second search result. Here, it is assumed that one piece of information is searched from each of the first information source 300 and the second information source 400. The first search result includes one piece of information related to the first query from the first information source 300, and the second search result includes one piece of information related to the second query from the second information source 400.

[0041] (Step S104) Next, the consistency determination unit 114 executes a consistency determination process. The consistency determination process is a process for determining the consistency between the first search result and the second search result. Fig. 5 is a flowchart showing an example of the procedure of the consistency determination process of this embodiment.

[0042] (Step S111) In the consistency determination process, first, the consistency determination unit 114 calculates a consistency score between the first search result and the second search result. For example, the consistency determination unit 114 obtains the embedded representations of the first search result and the second search result using sBERT, and calculates the cosine similarity between the embedded representations of the first search result and the second search result as the consistency score.

[0043] (Step S112) Next, the consistency determination unit 114 determines whether the cosine similarity calculated as the consistency score is equal to or greater than a threshold value TH. The threshold value TH is preset and stored in the storage device 12. If the cosine similarity is smaller than the threshold value TH (step S112-No), the process returns to step S102, and steps S102, S103, and S111 are executed again. At this time, the query generation method in step S102 and the search conditions in step S103 are changed. For example, if the information most relevant to the second query among the second information source 400 was used as the second search result, the second search result is updated to the information second most relevant to the second query among the second information source 400. Then, in step S111, the cosine similarity between the first search result and the updated second search result is recalculated, and compared with the threshold value TH in step S112. The processing circuit 11 repeatedly executes searches with changed search conditions and consistency determinations until the cosine similarity becomes equal to or greater than the threshold value TH. Then, if the cosine similarity is greater than or equal to the threshold value TH (step S112-Yes), the consistency determination unit 114 determines that the combination of the first search result and the second search result, whose cosine similarity is greater than or equal to the threshold value TH, is a consistent search result.

[0044] (Step S105) The integrating unit 115 integrates the search results whose consistency has been confirmed with the question to generate an integrated prompt. At this time, the first search results determined to be consistent are used as first additional information, and the second search results determined to be consistent are used as second additional information. The integrating unit 115 inputs the question entered by the user and a combination of the first search results and second search results determined to be consistent into the large-scale language model, causing the large-scale language model to generate an integrated prompt.

[0045] As an example, assume that first information source 300 is Company A's internal database, second information source 400 is the Internet, and a query such as "Please benchmark the image anomaly detection technology developed by our company, Company A, against technologies from other companies." is input. A first search result is obtained from the internal database: "Company A's image anomaly detection technology uses generative adversarial network technology to automatically generate training data using AI, thereby achieving highly accurate inference even with a small amount of training data." A second search result is obtained from the Internet: "Company B's image anomaly detection technology detects anomalies by using an autoencoder to calculate reconstruction error." In this case, integration unit 115 integrates the query and the search results, and generates a query based on the information in the first and second search results below. The query is "Please benchmark the image anomaly detection technology developed by our company, Company A, against technologies from other companies." The first search result is "Company A's image anomaly detection technology uses generative adversarial network technology to automatically generate training data using AI, thereby achieving highly accurate inference even with a small amount of training data." The second search result is "Company B's image anomaly detection technology detects anomalies by using an autoencoder to calculate the reconstruction error.", generating the synthesized sentence.

[0046] (Step S106) The answer generation unit 116 generates an answer using the generated integrated prompt and the large-scale language model. At this time, the answer generation unit 116 queries the large-scale language model for the integrated prompt generated in step S105, causing the large-scale language model to output an answer to the integrated prompt.

[0047] The processing circuitry 11 then outputs the answer obtained from the large-scale language model to the client terminal 200 as an answer to the question entered by the user.

[0048] The effects of the question answering device 100 according to this embodiment will be described below.

[0049] The question answering device 100 according to this embodiment includes an acquisition unit 111, a query generation unit 112, a search unit 113, a consistency determination unit 114, an integration unit 115, and an answer generation unit 116. The acquisition unit 111 acquires a question input by a user. The query generation unit 112 generates a query based on the question. The search unit 113 searches multiple information sources for search results related to the query.

[0050] The consistency determination unit 114 determines the consistency of the search results. The integration unit 115 generates an integrated prompt by integrating the search results determined to be consistent with the question. The answer generation unit 116 generates an answer to the question by inputting the integrated prompt into a large-scale language model.

[0051] Conventionally, when a question is queried to a large-scale language model using search results from multiple information sources, simply combining the search results from the multiple information sources can result in a decrease in answer quality. This is because an appropriate answer cannot be generated if the search results are inconsistent. To address this issue, the question answering device 100 according to the present embodiment checks the consistency of the search results from the multiple information sources and integrates only the consistent search results into a prompt, thereby improving the answer quality of the large-scale language model.

[0052] The query is, for example, a keyword extracted from a question. In this case, the search unit 113 searches for information containing the keyword from the information sources. For example, the search unit 113 extracts the keyword by querying a large-scale language model for the keyword in the question. The query may also be an embedded expression of the question. In this case, the search unit 113 converts the information from each information source into an embedded expression and searches the information sources for information similar to the embedded expression of the question.

[0053] For example, the consistency determination unit 114 calculates a consistency score between the search results and determines whether the calculated consistency score is equal to or greater than a threshold, thereby determining the consistency of the search results. The consistency score is, for example, a cosine similarity between the embedded expressions.

[0054] For example, if it is determined that the search results are inconsistent, the query generation unit 112 generates a query and the search unit 113 searches again. At this time, the query generation conditions and search conditions are changed, and the search results are updated. Then, the query generation and search are repeatedly executed until it is determined that the search results are consistent. Note that the consistency of the search results may be determined by querying a large-scale language model as to whether the search results are consistent.

[0055] As an application example of this embodiment, a case will be described in which an answer is generated by combining in-house information and information from the Internet in response to a question inquiring about information related to the Export Trade Control Order. In this example, in-house information and the Internet are used as multiple information sources. If the Export Trade Control Order has been revised but in-house rules for export trade control have not been revised, in-house rules based on the Export Trade Control Order before the revision will be obtained as a search result for in-house information, and the revised Export Trade Control Order will be obtained as a search result for the Internet. If an answer is generated by simply combining the two pieces of information using a large-scale language model, the pre-revision information and the revised information will be combined, which may result in an answer containing hallucinations.

[0056] In contrast to the above application example, the question answering device 100 according to this embodiment checks the consistency between the acquired search results, does not use inconsistent information, and generates an answer using only consistent information, thereby improving the quality of the answer.

[0057] As another application example, a case will be described in which an answer is generated by combining in-house information and information from the Internet in response to a question inquiring about the career history of employee A of a company. In this example, for example, in-house information corresponds to first information source 300, and the Internet corresponds to second information source 400. In-house information is an information source that is more reliable than the Internet. The Internet is an information source that is less reliable than in-house information, but contains more information than in-house information.

[0058] An internal information search can retrieve technical documents reported by Employee A in their department within the company, but it cannot retrieve papers published by Employee A during their university days. For this reason, it is not appropriate to determine, based solely on the search results for internal information, that technical documents reported by Employee A are available but papers published by Employee A during their university days are not. For this reason, it is necessary to also search information on the Internet to generate an answer. On the other hand, an Internet search cannot retrieve technical documents reported by Employee A within the company, but it can retrieve papers published by Employee A during their university days. However, if two people with the same name as Employee A are searched and the information of a person other than Employee A is used in the answer, this can cause a decrease in the quality of the answer.

[0059] In the above application example, the question answering device 100 according to the present embodiment can check the consistency between the age of a person searched from the Internet and the age of employee A searched from highly reliable internal company information, and determine the information of a person whose age matches as a search result that is consistent with the internal company information, and determine the information of a person whose age does not match as a search result that is inconsistent with the internal company information. Then, by using information such as papers published by people of the same age as employee A and information such as technical documents reported by employee A internally as consistent search results, it is possible to provide answers with high answer quality.

[0060] If no consistent search results are found, the integrated prompt may be generated using only search results from highly reliable sources among the search results searched from multiple information sources. For example, if no combination of search results with a high similarity score is found, the integrated prompt may be generated using only search results and questions from highly reliable sources, without using search results from less reliable sources.

[0061] As an application example of this case, we will explain the case where a paper published by employee A is searched from the academic society's website (hereinafter referred to as the academic society's website) and an SNS. In this example, the academic society's website corresponds to the first information source 300, and the SNS corresponds to the second information source 400. The academic society's website is an information source that is more reliable than the SNS. The SNS is an information source that is less reliable than the academic society's website, but contains more information than the academic society's website.

[0062] For example, suppose only Paper 1, "Anomaly detection using random forests," is searched from an academic society's website, and only Paper 2, "Anomaly detection using machine learning," is searched from an SNS, and it is determined that there is no consistency between the first and second search results. In this case, because the academic society's website is more reliable than the SNS, the quality of the answer can be improved by generating an integrated prompt using only Paper 1 and the question searched from the academic society's website, without using Paper 2 searched from the SNS.

[0063] As another application example, let us consider a case where the best literature on a certain disease is searched for in PubMed (registered trademark) and the Internet. PubMed is a medical paper database available on the Internet. In this example, PubMed corresponds to the first information source 300, and the Internet corresponds to the second information source 400. If paper A is found in PubMed and an explanatory document for paper A is found on the Internet, the two search results are consistent. Therefore, an integrated prompt is generated using paper A and its explanatory document as the consistent search results, and an answer is generated. On the other hand, if paper A is found as a PubMed search result and internet article B is found as an internet search result, and the two search results are determined to be inconsistent, the quality of the answer can be improved by generating an integrated prompt using paper A and the question retrieved from the highly reliable PubMed without using internet article B.

[0064] The consistency of the search results may also be determined by taking into account the complementarity of the search results. For example, even if the consistency scores of the first and second search results are low, if both have high similarity to the content of the question, the first and second search results may be determined to be highly complementar and consistent.

[0065] For example, an application example will be described in which employee A's educational background, career history, and hobbies are searched from an in-house database and the Internet. For example, the in-house database corresponds to the first information source 300, and the Internet corresponds to the second information source 400. When employee A's educational background and career history are searched from the in-house database and employee A's hobbies are searched from the Internet, the degree of similarity between the educational background and career history searched from the in-house database and the hobbies searched from the Internet will be low. However, since each search result contains content that matches the question, the similarity of the search results to the question will be high. Therefore, it is advisable to determine that the search results are highly complementary and that the search results are consistent. In this case, by complementing the information in the in-house database with the information from the Internet, information about employee A's educational background, career history, and hobbies can be obtained.

[0066] Although the present embodiment describes a case where search results from two information sources are combined to generate an integrated prompt, search results from three or more information sources may be combined. For example, in addition to a first information source 300 such as an in-house database and a second information source 400 such as the Internet, a large-scale language model may be used as a third information source. In this case, answer quality can be further improved by generating an answer using only information that is consistent across all three information sources.

[0067] (First Modification) In the above embodiment, query generation and search are repeatedly performed until the search results are determined to be consistent. In this modification, the consistency determination unit 114 extracts the combination of search results that has the highest consistency score, and determines that the combination of search results is consistent.

[0068] (Consistency determination process) Fig. 6 is a flowchart showing an example of the procedure for consistency determination processing according to this modification. Fig. 7 is a diagram showing an example of first search results and second search results according to this modification. Here, in the processing of step S103, two pieces of information, search result 1-1 and search result 1-2, are searched for as the first search results, and three pieces of information, search result 2-1, search result 2-2, and search result 2-3, are searched for as the second search results.

[0069] (Step S121) In the consistency determination process, the consistency determination unit 114 first calculates a consistency score between the first search result and the second search result. The consistency determination unit 114 uses sBERT to obtain the embedded representation of the first search result and the embedded representation of the second search result, and calculates the cosine similarity between the embedded representation of the first search result and the embedded representation of the second search result. At this time, the consistency determination unit 114 calculates the cosine similarity for all combinations of the search results included in the first search result and the search results included in the second search result. In the example of FIG. 7, six cosine similarities are calculated as the consistency score: the cosine similarity between search result 1-1 and search result 2-1, the cosine similarity between search result 1-1 and search result 2-2, the cosine similarity between search result 1-1 and search result 2-3, the cosine similarity between search result 1-2 and search result 2-1, the cosine similarity between search result 1-2 and search result 2-2, and the cosine similarity between search result 1-2 and search result 2-3.

[0070] (Step S122) Next, the consistency determination unit 114 determines that the combination with the largest consistency score is a search result with consistency. Here, the combination with the largest cosine similarity among the six combinations is selected as the search result with consistency.

[0071] According to this variant, even if multiple pieces of information are searched from multiple information sources, an integrated prompt is generated using only consistent search results, thereby improving the quality of answers from the large-scale language model.

[0072] (Second embodiment) A second embodiment will be described. This embodiment is a modification of the configuration of the first embodiment as follows. Descriptions of the configuration, operation, and effects similar to those of the first embodiment will be omitted.

[0073] Fig. 8 is a diagram illustrating an example of the configuration of the question answering device 100 according to this embodiment. As illustrated in Fig. 8, the query generation unit 112 includes a first query generation unit 112A that generates a first query based on a question, and a second query generation unit 112B that generates a second query based on the question. In this embodiment, the second query generation unit 112B generates the second query using the first search result. For example, the second query may be an embedded expression of the first search result or a keyword extracted from the first search result.

[0074] The search unit 113 also includes a first search unit 113A that searches the first information source 300 for first search results related to the first query, and a second search unit 113B that searches the second information source 400 for second search results related to the second query.

[0075] The consistency determination unit 114 determines the consistency between the first search result and the second search result. The integration unit 115 generates an integrated prompt by integrating the question, the first search result determined to be consistent, and the second search result determined to be consistent. The answer generation unit 116 generates an answer to the question using the integrated prompt and the large-scale language model. The consistency determination method, the integrated prompt generation method, and the answer generation method using the integrated prompt are the same as those in the first embodiment, and therefore will not be described again.

[0076] Next, a description will be given of the operation of the question answering process executed by the question answering device 100 according to this embodiment. Fig. 9 is a flowchart showing an example of the procedure of the question answering process. Fig. 10 is a diagram showing an example of the flow of data in the question answering process.

[0077] In the question answering process of this embodiment, the first query generation unit 112A uses the question acquired in the process of step S201 to generate a first query to be used for searching the first information source 300 (step S202). For example, the first query generation unit 112A queries a large-scale language model for keywords of the question and acquires the extracted keywords as the first query. Thereafter, the first search unit 113A searches the first information source 300 using the first query and acquires first search results related to the question (step S203). For example, the first search unit 113A searches the first information source 300 for information including the first query, thereby searching the first information source 300 for information related to the question.

[0078] Next, second query generation unit 112B generates a second query to be used for searching second information source 400 using the first search result generated in the process of step S203 (step S204). For example, second query generation unit 112B obtains, as the second query, keywords extracted by querying a large-scale language model for keywords related to the information obtained as the first search result. Thereafter, second search unit 113B searches second information source 400 using the second query and obtains second search results related to the first search results (step S205). Because the first search results are information related to a question searched using a question, the second search results related to the first search results are also information related to the question.

[0079] Next, the consistency determination unit 114 determines the consistency between the first search result obtained in the process of step S203 and the second search result obtained in the process of step S205 (step S206). The integrating unit 115 integrates the question with the combination of the first search result and the second search result determined to be consistent, to generate an integrated prompt (step S207). Then, the answer generation unit 116 generates an answer to the question by querying the large-scale language model using the generated integrated prompt (step S208).

[0080] The question answering device 100 of this embodiment can generate a highly reliable second query by using the search results of the highly reliable first information source 300. Then, by searching the second information source 400 using the second query, highly reliable information can be searched for from the second information source 400. Furthermore, by generating an answer to a question using an integrated prompt generated using the highly reliable search results, the quality of the answer can be further improved.

[0081] In this embodiment, as in the first embodiment and its variants, the combination of search results with the highest consistency score may be determined to be consistent, or if no combination of search results with a high similarity score is found, only search results from highly reliable information sources may be used, or the consistency of search results may be determined taking into account the complementarity of the search results, or search results obtained from three or more information sources may be combined.

[0082] (Third embodiment) A third embodiment will be described. This embodiment is a modification of the configuration of the second embodiment as follows. Descriptions of the configuration, operation, and effects similar to those of the second embodiment will be omitted.

[0083] In this embodiment, second query generation unit 112B generates a second query using both the question and the first search results retrieved from first information source 300 using the question. As the second query, for example, an embedded expression generated using the question and the first search results, or a keyword extracted from the question and the first search results can be used.

[0084] 11 is a diagram illustrating an example of a data flow in the question answering process according to this embodiment. In the question answering process according to this embodiment, in the process of step S204, the second query generation unit 112B uses both the question acquired in the process of step S201 and the first search results acquired in the process of step S203 to generate a second query to be used in searching the second information source 400. For example, the second query generation unit 112B queries a large-scale language model for keywords related to both the question and the first search results, and acquires the keywords output by the large-scale language model as the second query. Thereafter, the second search unit 113B searches the second information source 400 using the second query, thereby acquiring information related to both the question and the first search results as the second search results (step S205).

[0085] According to the question answering device 100 of this embodiment, by generating a second query using a question in addition to search results from the highly reliable first information source 300, it is possible to generate a second query that is highly reliable and highly relevant to the question. Then, by searching the second information source 400 using the second query, it is possible to search the second information source 400 for information that is highly reliable and highly relevant to the question. Furthermore, by generating an answer to a question using an integrated prompt generated using search results that are highly relevant to the question and highly reliable, it is possible to further improve the quality of the answer.

[0086] In this embodiment, as in the first embodiment and its variants, the combination of search results with the highest consistency score may be determined to be consistent, or if no combination of search results with a high similarity score is found, only search results from highly reliable information sources may be used, or the consistency of search results may be determined taking into account the complementarity of the search results, or search results obtained from three or more information sources may be combined.

[0087] Thus, according to any of the above-described embodiments, it is possible to provide a question answering device, method, and program with improved answer accuracy.

[0088] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0089] 1...Question answering system, 100...Question answering device, 200...Client terminal, 300, 400...Information source, 11...Processing circuit, 12...Memory device, 13...Input device, 14...Communication device, 15...Display device, 111...Acquisition unit, 112, 112A, 112B...Query generation unit, 113, 113A, 113B...Search unit, 114...Consistency determination unit, 115...Integration unit, 116...Answer generation unit.

Claims

1. an acquisition unit that acquires a question input by a user; a query generation unit that generates a query based on the question; a search component that searches multiple sources for search results related to the query; a consistency determination unit that determines the consistency of the search results; an integration unit that generates an integrated prompt by integrating the search results determined to be consistent with the question; an answer generator that generates an answer to the question by inputting the integrated prompt into a large-scale language model; A question answering device comprising:

2. the plurality of information sources includes a first information source and a second information source; the query generation unit includes a first query generation unit that generates a first query based on the question, and a second query generation unit that generates a second query based on the question; the search unit includes a first search unit that searches the first information source for first search results related to the first query, and a second search unit that searches the second information source for second search results related to the second query, the consistency determination unit determines consistency between the first search result and the second search result; the integrating unit generates an integrated prompt by integrating the first search result and the second search result determined to be consistent with the question. The question answering device according to claim 1 .

3. the first information source is more reliable than the second information source; The second information source is more searchable than the first information source. The question answering device according to claim 2 .

4. the second query generation unit generates the second query based on the first search result. The question answering device according to claim 2 or 3.

5. the second query generation unit generates the second query based on the question and the first search result. The question answering device according to claim 2 or 3.

6. the query is a keyword extracted from the question; the search unit searches the information source for information including the keyword; The question answering device according to claim 1 .

7. the query generation unit extracts the keywords in the question by querying a large-scale language model; The question answering device according to claim 6.

8. the query is an embedded representation of the question; the search unit searches the information source for information similar to the embedded expression of the question; The question answering device according to claim 1 .

9. the consistency determination unit determines the consistency of the search results by querying a large-scale language model as to whether the search results have consistency. The question answering device according to claim 1 .

10. the consistency determination unit determines consistency of the search results based on a similarity between the search results. The question answering device according to claim 1 .

11. the question answering device repeatedly executes the generation of the query by the query generation unit and the search by the search unit until it is determined that the search result has consistency. The question answering device according to claim 1 .

12. the consistency determination unit determines that a combination of search results having a maximum consistency score is consistent; The question answering device according to claim 1 .

13. the integrating unit generates an integrated prompt by integrating the question and the first search result when it is determined that the first search result and the second search result are not consistent. The question answering device according to claim 3 .

14. The consistency determination unit determines that a combination of search results having a high consistency score or a combination of search results having a high degree of complementarity is consistent. The question answering device according to claim 1 .

15. an acquisition unit acquiring a question input by a user; a query generation unit generating a query based on the question; a search unit searching a plurality of information sources for search results related to the query; a consistency determination unit determining consistency of the search results; an integration unit generating an integrated prompt by integrating the search results determined to be consistent with the question; an answer generation unit generating an answer to the question by inputting the integrated prompt into a large-scale language model; A method for providing the above.

16. On the computer, A function to obtain questions entered by users, generating a query based on the question; a function of searching search results related to said query from multiple sources; a function for determining the consistency of the search results; a function of generating an integrated prompt that integrates the search results determined to be consistent with the question; generating an answer to the question by inputting the integrated prompt into a large-scale language model; A program to achieve this.

Images