Information processing device, information processing method, and program

Abstract

The objective is to provide an information processing device, an information processing method, and a program that can improve the accuracy of answers to questions. [Solution] The information processing device according to the embodiment comprises a first generation means, a search means, a determination means, a second generation means, and a third generation means. The first generation means generates a query for searching for data related to a question specified by the user. The search means obtains a first search result by performing a search on a pre-prepared first database using the query. The determination means determines whether there is consistency between the question and the first search result. The second generation means generates a prompt that integrates the question and the first search result which has been determined to be consistent with the question. The third generation means generates an answer to the question by inputting the prompt into a pre-prepared first language model.

Description

Technical Field

[0001] Embodiments of the present invention relate to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In recent years, it has become possible to train large language models (LLMs) using a large amount of text data, and it has been realized to generate answers to questions in natural language using such LLMs.

[0003] The above-mentioned LLM can generate an appropriate answer to a question by inputting a question regarding pre-learned knowledge (text data). On the other hand, when a question regarding unlearned knowledge is input, the LLM may not be able to generate an appropriate answer to the question, and a phenomenon called hallucination may occur, in which an answer contrary to the facts is generated.

[0004] [[ID=2*]]Here, in order to generate an answer to a question regarding knowledge that the LLM has not learned, it is conceivable to use a pre-constructed database (a database in which the knowledge is accumulated). Specifically, the LLM can generate an appropriate answer even for matters that have not been learned by inputting a question regarding unlearned knowledge and data related to the question retrieved from the database.

[0005] However, when the LLM generates an answer to a question using the data retrieved from the above-mentioned database, the accuracy (quality) of the answer to the question may decrease depending on the retrieved data.

Prior Art Documents

Non-Patent Documents

[0006]

Non-Patent Document 1

[0007] Therefore, the problem that the present invention aims to solve is to provide an information processing device, an information processing method, and a program that can improve the accuracy of answers to questions. [Means for solving the problem]

[0008] The information processing device according to the embodiment comprises a first generation means, a search means, a determination means, a second generation means, and a third generation means. The first generation means generates a query for searching for data related to a question specified by a user. The search means obtains a first search result by performing a search on a pre-prepared first database using the generated query. The determination means determines whether there is consistency between the question and the obtained first search result. The second generation means generates a prompt that integrates the question and the first search result which has been determined to be consistent with the question. The third generation means generates an answer to the question by inputting the generated prompt into a pre-prepared first language model. [Brief explanation of the drawing]

[0009] [Figure 1] A diagram showing an example of the configuration of the information processing system in the first embodiment. [Figure 2] A diagram showing an example of the configuration of an information processing device. [Figure 3] A diagram illustrating the general operation of an information processing system. [Figure 4] A flowchart illustrating an example of the processing procedure for an information processing device. [Figure 5]A diagram showing an example of the configuration of the information processing system in the second embodiment. [Figure 6] A diagram showing an example of the configuration of an information processing device. [Figure 7] A diagram illustrating the general operation of an information processing system. [Figure 8] A flowchart illustrating an example of the processing procedure for an information processing device. [Figure 9] A diagram showing an example of the configuration of an information processing device according to the third embodiment. [Figure 10] A diagram illustrating the general operation of an information processing system. [Figure 11] A flowchart illustrating an example of the processing procedure for an information processing device. [Figure 12] A diagram illustrating the general operation of the information processing system in the fourth embodiment. [Figure 13] A flowchart illustrating an example of the processing procedure for an information processing device. [Modes for carrying out the invention]

[0010] The embodiments will be described below with reference to the drawings. (First Embodiment) First, the first embodiment will be described. The information processing device according to this embodiment corresponds to, for example, a question answering device or question answering processing device that has the function of responding to a question specified by a user.

[0011] Figure 1 shows an example of the configuration of the information processing system in this embodiment. As shown in Figure 1, the information processing system 1 comprises a client terminal 10, a database 20, and an information processing device 30.

[0012] The client terminal 10 is a terminal device (computer) used by the user who specifies (creates) the above-mentioned question, and is connected to the information processing device 30 via a network. The client terminal 10 transmits the question specified by the user to the information processing device 30.

[0013] The database 20 corresponds to an information source in which various data are accumulated and is constructed to be accessible from an information processing apparatus 30 connected via a network. Although only one database 20 is shown in FIG. 1, the information processing system 1 may include a plurality of databases 20.

[0014] The information processing apparatus 30 receives a question transmitted from the client terminal 10 and generates an answer to the question. When generating an answer to the above-described question, the information processing apparatus 30 uses data retrieved from the LLM (Large Language Model) and the database 20 that matches the question. The answer generated in the information processing apparatus 30 is transmitted to the client terminal 10.

[0015] That is, it can be said that the information processing system 1 in the present embodiment is an interactive computer network system using an LLM.

[0016] The information processing apparatus 30 is connected to the client terminal 10 and the database 20 via a network. The network in this case is, for example, a LAN (Local Area Network), but may also be the Internet or a public communication line, etc. Further, the connection to the network may be wired or wireless.

[0017] FIG. 2 shows an example of the configuration of the information processing apparatus 30 shown in FIG. 1. As shown in FIG. 2, the information processing apparatus 30 includes a processing circuit 31, a storage device 32, and a communication device 33.

[0018] The processing circuit 31 has a processor such as a CPU (Central Processing Unit) and a memory such as a RAM (Random Access Memory). Note that the number of processors included in the processing circuit 31 may be one or plural.

[0019] The storage device 32 corresponds to a non-temporary computer-readable recording medium and includes, for example, ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), or integrated circuit storage device. The storage device 32 stores various programs, etc. The programs stored in the storage device 32 may be implemented as a single program, or as multiple modules divided into predetermined functional units.

[0020] The communication device 33 corresponds to an interface for communicating with external devices connected to the information processing device 30 via a network.

[0021] In the example shown in Figure 2, the information processing device 30 is described as comprising a processing circuit 31, a storage device 32, and a communication device 33. However, the information processing device 30 may further include input devices such as a keyboard, mouse, various switches, a touchpad, or a touch panel display, and display devices such as a CRT (Cathode-Ray Tube) display, a liquid crystal display, an organic EL (Electro-Luminescence) display, an LED (Light-Emitting Diode) display, a plasma display, or any other display. The display device may also be, for example, a projector.

[0022] As shown in Figure 2, the processing circuit 31 includes a query generation unit 311, a search unit 312, a consistency determination unit 313, a prompt generation unit 314, and an answer generation unit 315.

[0023] In this embodiment, each of the parts 311 to 315 is realized by the execution of a program stored in the above-mentioned storage device 32 by the processing circuit 31 (processor) (i.e., software).

[0024] Each of the parts 311 to 315 may be implemented, for example, by an Application Specific Integrated Circuit (ASIC). In this case, each of the parts 311 to 315 may be implemented on a single integrated circuit, or they may be implemented individually on multiple integrated circuits.

[0025] Here, the client terminal 10 described above has a processor, memory, storage device, communication device, input device, and display device as hardware, and functions as a user interface in the information processing system 1. Specifically, the client terminal 10 receives input of a question specified by the user via the input device. The question is text data corresponding to an inquiry from the user. The question may be a natural sentence such as a question statement, or it may be a word such as a search keyword. When the client terminal 10 receives input of a question specified by the user as described above, it transmits the question to the information processing device 30.

[0026] The information processing device 30 receives a question sent from the client terminal 10 via the communication device 33. The query generation unit 311 generates a query to search for data related to the received question.

[0027] The search unit 312 performs a search on the database 20 using the query generated by the query generation unit 311. As a result, the search unit 312 obtains data (information related to the query) that matches the query generated by the query generation unit 311 as search results. The database 20 only needs to be structured in a way that allows searching of data related to the above-mentioned question; for example, an internal company database or a research paper database can be used as the database 20.

[0028] The consistency determination unit 313 determines whether there is consistency between the question specified by the user and the search results obtained by the search unit 312 (data acquired by the search unit 312 as search results).

[0029] The prompt generation unit 314 generates a prompt that integrates the question and the search results determined to be consistent with the question by the consistency determination unit 313.

[0030] The answer generation unit 315 generates an answer to the above-mentioned question based on the prompt generated by the prompt generation unit 314.

[0031] In this embodiment, the answer generation unit 315 includes the LLM described above. The LLM is a language model generated by learning a large amount of text data (questions and answers), and is constructed to generate (output) an answer to a question when a question is input. In other words, in this embodiment, the answer generation unit 315 can generate an answer to a question by inputting a prompt generated by the prompt generation unit 314 (i.e., a query generated from the question and search results) into the LLM.

[0032] As described above, the response generated by the response generation unit 315 (LLM) is transmitted to the client terminal 10 via the communication device 33. In this case, the client terminal 10 receives the response transmitted from the information processing device 30 and displays the received response on the display device of the client terminal 10. The response is text data corresponding to a response to an inquiry from the user.

[0033] In Figure 2, all of the parts 311 to 315 are described as being included in a single device (i.e., the information processing device 30), but these parts 311 to 315 may be distributed and implemented in multiple devices, or some of these parts 311 to 315 may be implemented in the client terminal 10.

[0034] Now, with reference to Figure 3, an overview of the operation of the information processing system 1 in this embodiment will be described.

[0035] In the information processing system 1 of this embodiment, a search is performed on the database 20 using a query generated based on a question specified by the user using the client terminal 10, and the search results are obtained.

[0036] As mentioned above, if the question is text data, the query is, for example, a keyword extracted from the question or an embedded expression of the question (a vector representing the question). If the query is a keyword, text data (document data) containing that keyword is obtained as the search result. If the query is an embedded expression, the database 20 stores text data and the embedded expression of that text data in association, and text data stored in association with an embedded expression similar to the query is obtained as the search result. Note that the search process in this embodiment may be performed on multiple databases 20, or it may be performed using multiple queries (both keywords and embedded expressions).

[0037] In this embodiment, by inputting a prompt (hereinafter referred to as the integrated prompt) that integrates the question and the search results (data retrieved from the database 20) consistent with the question into the LLM, the answer to the question output by the LLM is sent to the client terminal 10 as a response.

[0038] The following describes an example of the processing procedure of the information processing device 30 (processing circuit 31) according to this embodiment, with reference to the flowchart in Figure 4. The processing shown in Figure 4 (question answering processing) is executed when a question sent from the client terminal 10 is received by the information processing device 30.

[0039] First, the query generation unit 311 obtains the question received by the information processing device 30 (step S1).

[0040] Next, the query generation unit 311 generates a query based on the question obtained in step S1 (step S2). In step S2, a query is generated in a format that allows data related to the question to be retrieved from the database 20.

[0041] The search unit 312 performs a search on the database 20 using the query generated in step S2 (step S3).

[0042] When the process in step S3 is executed, the consistency determination unit 313 determines whether there is consistency between the question obtained in step S1 and the search results obtained in step S3 (i.e., whether the content is consistent or not) (step S4).

[0043] The following describes the process of step S4 (the process of determining whether or not consistency exists). In this embodiment, the process of step S4 is performed using, for example, an LLM. Specifically, whether or not consistency exists is determined by inputting, for example, a question, search results, and a prompt to inquire about the presence or absence of consistency into the LLM. The LLM used for determining whether or not consistency exists may be the LLM included in the answer generation unit 315 described above (hereinafter referred to as the answer generation LLM), or it may be a different LLM.

[0044] Here, the presence or absence of the aforementioned consistency shall be determined, for example, based on predetermined evaluation criteria. Possible predetermined evaluation criteria include similarity, presence or absence of contradictions, or the feasibility of answering the question (i.e., whether or not the question can be answered using the search results).

[0045] If the evaluation criterion is similarity, for example, a prompt such as "Determine if the content of the question and search results are similar" is input to the LLM, and the consistency between the question and search results is determined based on the similarity of the question and search results output by the LLM. According to this, for example, if the similarity between the question and search results is high (above a threshold), it can be determined that the question and search results are consistent.

[0046] If the evaluation criterion is the presence or absence of contradictions, then, for example, a prompt such as "Check for contradictions between the question and search results" is entered into the LLM, and the consistency between the question and search results is determined based on the presence or absence of contradictions between the question and search results output by the LLM. According to this, for example, if the content of the question and search results are not contradictory, then the question and search results are determined to be consistent.

[0047] If the evaluation criterion is the possibility of answering a question, then, for example, a prompt such as "Determine whether it is possible to answer the question using the search results" is input to the LLM, and the consistency between the question and the search results is determined based on the possibility of answering the question output by the LLM. According to this, for example, if the numerical value representing the possibility of answering the question is high (above the threshold), it can be determined that the question and the search results are consistent.

[0048] Furthermore, the evaluation criteria used to determine consistency may be one of the following: similarity, presence or absence of contradictions, and the ability to answer the question; however, consistency may be determined based on two or more evaluation criteria.

[0049] Furthermore, although the process in step S4 has been described here as being performed using LLM, the process in step S4 may be any process that determines whether or not the search results are consistent with the question, and may be performed, for example, based on the results of morphological analysis performed on the question and search results. Specifically, for example, consistency may be determined using the similarity of keywords extracted from the question and search results through morphological analysis as an evaluation criterion. Alternatively, consistency may be determined using the presence or absence of contradictions between keywords extracted from the question and search results through morphological analysis as an evaluation criterion. Furthermore, consistency may be determined using the likelihood of answering the question based on the frequency of occurrence of keywords extracted from the question in the search results through morphological analysis as an evaluation criterion. Note that the process in step S4 described above may also be performed using, for example, embedded expressions generated from the question and search results.

[0050] If it is determined in step S4 above that the question and search results are consistent, the prompt generation unit 314 generates an integrated prompt using the question and search results (step S5). The integrated prompt is, for example, a prompt that states that the question should be answered using the search results, but any prompt that reflects the question and search results is acceptable.

[0051] In step S5, for example, an integrated prompt may be generated by inputting the question and search results into the LLM. The LLM used to generate such an integrated prompt may be the answer generation LLM described above, or it may be a different LLM.

[0052] When the process in step S5 is executed, the answer generation unit 315 generates an answer to the question using the integrated prompt and the answer generation LLM generated in step S5 (step S6). In this case, the answer generation unit 315 inputs the integrated prompt to the answer generation LLM and obtains the answer output from the answer generation LLM.

[0053] Although not shown in Figure 4, the answer generated in step S6 (i.e., the answer to the question specified by the user) is sent from the information processing device 30 (processing circuit 31) to the client terminal 10.

[0054] Note that while this explanation describes the case where the question and search results are consistent, if the question and search results are inconsistent, steps S5 and S6 will not be executed, and the process shown in Figure 4 will terminate.

[0055] Furthermore, the results of the consistency determination described above may be sent to the client terminal 10 and notified to the user. In addition, the reasons for the consistency determination may be notified to the user along with the consistency determination results. The reasons for the consistency determination may include, for example, the similarity of the keywords described above, keywords that serve as the basis for determining whether there is a contradiction, or numerical values ​​that represent the likelihood of answering the question.

[0056] By the way, in step S3 described above, multiple data may be retrieved. When multiple data are retrieved in this way (i.e., when the search results include multiple data), in step S4, the consistency between the question and each of the multiple data will be determined. According to this, in step S5, for example, an integrated prompt can be generated using only the data that has been determined to be consistent with the question from among the multiple data included in the search results. In other words, the process in step 4 can be said to be the process of selecting data that matches the evaluation criteria (i.e., data appropriate for generating an answer) from the search results (multiple data). In addition, the results of the determination of consistency between multiple data included in the search results may be taken into consideration when selecting such data. Specifically, for example, if there is no consistency between multiple data, an integrated prompt will not be generated using the multiple data that do not have consistency (i.e., only the multiple data that do have consistency will be used).

[0057] The process shown in Figure 4 above (question answering process) is just one example, and at least a part of it may be modified. Specifically, at least a part of the processes in steps S1 to S6 shown in Figure 4 may be omitted or replaced with other processes. In addition, processes other than steps S1 to S6 shown in Figure 4 may be added.

[0058] As described above, the information processing device 30 according to this embodiment generates a query to search for data related to a question specified by the user, and obtains search results (data retrieved from database 20) by performing a search on a pre-prepared database 20 (first database) using the generated query. Furthermore, the information processing device 30 according to this embodiment determines whether there is consistency between the question and the search results, generates an integrated prompt (a prompt that integrates the question and the search results) using the question and the search results that are determined to be consistent with the question, and generates an answer to the question by inputting the generated integrated prompt into a pre-prepared answer generation LLM (first language model).

[0059] In this embodiment, the above configuration makes it possible to improve the accuracy of the answers to questions specified by the user.

[0060] Specifically, for example, if search results that are inconsistent with the question are obtained, and the answer is then queried to the answer generation LLM using such search results, the accuracy (quality) of the answer to the question may decrease due to the influence of the inconsistent search results. In contrast, in this embodiment, the answer to the question is generated using search results that are consistent with the question, thus suppressing a decrease in the accuracy of the answer (i.e., improving the accuracy of the answer).

[0061] In this embodiment, consistency is determined based on evaluation criteria such as the similarity between the question and the search results, the presence or absence of contradictions between the question and the search results, or whether the question can be answered using the search results (i.e., the possibility of answering the question). However, multiple evaluation criteria may be combined for determination, or other evaluation criteria may be used for determination.

[0062] Furthermore, the presence or absence of the aforementioned consistency is determined, for example, by inputting a question, search results, and a prompt to inquire about consistency into the LLM (a pre-prepared second language model). However, it may also be determined based on the results of morphological analysis of the question and search results (keywords extracted from text data), etc.

[0063] Furthermore, in this embodiment, the user may be notified of the result of the determination of consistency. This would allow the user to easily understand, for example, the search results used to generate an answer to a question (i.e., data that was determined to be consistent with the question).

[0064] Furthermore, although this embodiment has described the questions and search results as primarily being text data, the questions and search results may be in a format other than text data, for example, if it is possible to generate answers to the questions using LLM.

[0065] (Second Embodiment) Next, a second embodiment will be described. In this embodiment, the description of parts that are the same as those of the first embodiment described above will be omitted, and the parts that differ from the first embodiment will be described mainly.

[0066] Figure 5 shows an example of the configuration of the information processing system in this embodiment. In Figure 5, the same reference numerals are used for the same parts as in Figure 1 described above, and their detailed explanations are omitted.

[0067] As shown in Figure 5, the information processing system 1 includes a first database 20a and a second database 20b instead of the database 20 shown in Figure 1.

[0068] The first database 20a and the second database 20b each correspond to information sources where various types of data are stored, and are configured to be accessible from the information processing device 30 connected via a network.

[0069] Furthermore, the first database 20a and the second database 20b only need to be databases in which at least a portion of the stored data is different.

[0070] Figure 6 shows an example of the configuration of the information processing device 30 shown in Figure 5. In Figure 6, the same reference numerals are used for the same parts as in Figure 2, and their detailed explanations are omitted.

[0071] As shown in Figure 6, in this embodiment, the processing circuit 31 includes a keyword extraction unit 316 in addition to the parts 311 to 315 shown in Figure 2.

[0072] In this embodiment, the keyword extraction unit 316 is implemented by the execution of a program stored in the storage device 32 by the processing circuit 31 (processor) (i.e., software). The keyword extraction unit 316 may also be implemented by, for example, an application-specific integrated circuit.

[0073] In this embodiment, the search unit 312 obtains search results (hereinafter referred to as "first search results") by performing a search on, for example, the first database 20a using the query generated by the query generation unit 311.

[0074] The keyword extraction unit 316 extracts keywords from the question specified by the user and the first search result.

[0075] In this case, the search unit 312 obtains search results (hereinafter referred to as "second search results") by performing a search on the second database 20b using the keywords extracted by the keyword extraction unit 316.

[0076] In this embodiment, the consistency determination unit 313 determines whether or not there is consistency between the question, the first and second search results.

[0077] Now, with reference to Figure 7, an overview of the operation of the information processing system 1 in this embodiment will be described. Note that this section will primarily focus on the differences from Figure 3 described above.

[0078] The information processing system 1 in this embodiment differs from the first embodiment described above in that it obtains a first search result from a first database 20a using a query generated based on a question, and obtains a second search result from a second database 20b using the question and keywords extracted from the first search result.

[0079] In this embodiment, the information processing system 1 generates an answer to a question by inputting a prompt (integrated prompt) into the LLM that combines the question with the first and second search results that are consistent with the question.

[0080] Hereinafter, an example of the processing procedure of the information processing device 30 (processing circuit 31) according to this embodiment will be described with reference to the flowchart in Figure 8.

[0081] First, steps S11 and S12 are executed, which are similar to the processes in steps S1 and S2 shown in Figure 4 above.

[0082] Next, the search unit 312 performs a search on the first database 20a using the query generated in step S12 (step S13). The search unit 312 obtains the first search result by executing the process in step S13.

[0083] When the process in step S13 is executed, the keyword extraction unit 316 extracts keywords from the question and the first search result (step S14).

[0084] Keywords may be extracted, for example, by performing morphological analysis on the question and the first search result, or by inputting the question, the first search result, and a prompt such as "Extract keywords important to answer the user's question from the question and the first search result" into the LLM (i.e., querying the LLM for keywords using the prompt). The LLM used for such keyword extraction may be the answer generation LLM or a different LLM.

[0085] In step S14, it is sufficient to extract keywords that are highly relevant to the user's question, and these keywords may be extracted by other methods.

[0086] Next, the search unit 312 performs a search on the second database 20b using the keywords extracted in step S14 (step S15). The search unit 312 obtains the second search result by executing the process in step S15.

[0087] When the process in step S15 is executed, the consistency determination unit 313 determines whether the question, the first and second search results are consistent (step S16).

[0088] In step S16, for example, consistency between the question and the first search result is determined, as is consistency between the question and the second search result. However, the process for determining consistency is as described in the first embodiment above, so a detailed explanation is omitted here.

[0089] When the process in step S16 is executed, the processes in steps S17 and S18, which correspond to the processes in steps S5 and S6 shown in Figure 4 above, are executed.

[0090] In step S17, for example, search results determined to be consistent with the question are selected from the first and second search results, and an integrated prompt is generated using these selected search results. When selecting such search results, the results of the consistency determination in the first and second search results may be considered.

[0091] As described above, the information processing device 30 according to this embodiment extracts keywords from the first search results obtained by performing a search on the question and the first database 20a, and performs a search on the second database 20b using the extracted keywords. The information processing device 30 according to this embodiment determines whether there is consistency between the question, the first search results, and the second search results obtained by performing a search on the second database 20b, and generates an integrated prompt using the question and the search results determined to be consistent with the question.

[0092] In this embodiment, as described above, keywords highly relevant to the question extracted from the first search result are used for searching the second database 20b. Therefore, it is possible to generate answers to the question using more consistent data (i.e., improve the accuracy of the answers).

[0093] In this embodiment, keywords can be extracted, for example, by inputting a question, a first search result, and a prompt to query for keywords into an LLM (a pre-prepared third language model). However, they may also be extracted by other methods that do not utilize the LLM (for example, morphological analysis).

[0094] (Third embodiment) Next, a third embodiment will be described. In this embodiment, the description of parts that are the same as those of the second embodiment described above will be omitted, and the parts that differ from the second embodiment will be described mainly.

[0095] The configuration of the information processing system in this embodiment is the same as that of the second embodiment described above, and will be explained using Figure 5 as appropriate.

[0096] Figure 9 shows an example of the configuration of the information processing device 30 according to this embodiment. In Figure 9, the same reference numerals are used for the same parts as in Figure 6 described above, and their detailed explanations are omitted.

[0097] As shown in Figure 9, in this embodiment, the processing circuit 31 includes a keyword selection unit 317 in addition to the parts 311 to 316 shown in Figure 6.

[0098] In this embodiment, the keyword selection unit 317 is implemented by the execution of a program stored in the memory device 32 by the processing circuit 31 (processor) (i.e., software). The keyword selection unit 317 may also be implemented by, for example, an application-specific integrated circuit.

[0099] In this embodiment, assuming that the keyword extraction unit 316 has extracted multiple keywords, the keyword selection unit 317 selects from among these multiple keywords the keyword that is most appropriate for generating an answer.

[0100] In this case, the search unit 312 performs a search on the second database 20b using the keywords selected by the keyword selection unit 317.

[0101] Now, with reference to Figure 10, an overview of the operation of the information processing system 1 in this embodiment will be described. Note that this explanation will primarily focus on the differences from Figure 7 described above.

[0102] In the information processing system 1 of this embodiment, keywords are extracted from the question and the first search result, similar to the second embodiment described above. However, here we assume that multiple keywords (hereinafter referred to as the first keyword) are extracted.

[0103] The information processing system 1 in this embodiment differs from the second embodiment described above in that it classifies the above-mentioned multiple first keywords into multiple classification classes and performs a search on the second database 20b using the second keywords classified into a classification class selected from among the multiple classification classes. A classification class corresponds to, for example, a category into which a predetermined keyword (similar keywords, etc.) is classified.

[0104] Hereinafter, an example of the processing procedure of the information processing device 30 (processing circuit 31) according to this embodiment will be described with reference to the flowchart in Figure 11.

[0105] First, steps S21 to S24, which correspond to steps S11 to S14 shown in Figure 8 above, are executed. In step S24, it is assumed that multiple first keywords have been extracted from the question and the first search result.

[0106] When the process in step S24 is executed, the keyword selection unit 317 generates multiple classification classes for classifying the multiple first keywords extracted in step S24 (step S25).

[0107] In step S5, the classification class may be generated manually, for example, based on instructions from a user who specified a question, or it may be generated by inputting a question and a prompt such as "Generate a class for classifying keywords as a preprocessing step to search for data to answer the user's question" into the LLM (i.e., querying the LLM for the classification class using the prompt). The LLM used to generate such a classification class may be the answer generation LLM or a different LLM.

[0108] In step S25, other methods may be employed, such as automatically generating a classification class to which a keyword belongs using the results of morphological analysis of the question (i.e., keywords extracted from the question). Also, although it has been explained here that a classification class is generated, this classification class may be prepared in advance.

[0109] Next, the keyword selection unit 317 classifies the multiple first keywords extracted in step S24 into multiple classification classes generated in step S25 (step S26).

[0110] In step S26, each of the multiple first keywords may be classified by inputting, for example, the first keyword, multiple classification classes, and a prompt such as "select the classification class to which the first keyword belongs from among the multiple classification classes" into the LLM (i.e., querying the LLM for the classification class to which the first keyword belongs using the prompt). The LLM used for such classification of first keywords may be the answer generation LLM or a different LLM.

[0111] When the process in step S26 is executed, the keyword selection unit 317 selects a second keyword from among the multiple first keywords based on the classification results of the multiple first keywords in step S26 (step S27).

[0112] In step S27, for example, a question, multiple classification classes, and a prompt such as "Select the classification class necessary to answer the user's question from among the multiple classification classes" are input to the LLM (i.e., the LLM is queried using the prompt to determine the classification class necessary to answer the question), and a second keyword classified into the selected classification class is selected. The LLM used for selecting such a second keyword may be the answer generation LLM or a different LLM. The selection of the classification class for selecting the second keyword may be based on user instructions, for example.

[0113] Next, the search unit 312 performs a search on the second database 20b using the second keyword selected in step S27 (step S28).

[0114] When the process in step S28 is executed, the processes in steps S29 to S31, which correspond to the processes in steps S16 to S18 shown in Figure 8 above, are executed.

[0115] As described above, the information processing device 30 according to this embodiment extracts a plurality of first keywords from the question and the first search result, classifies the extracted plurality of first keywords into a plurality of classification classes, selects a second keyword that is classified into a selected classification class from among the plurality of classification classes, and performs a search on the second database 20b using the selected second keyword.

[0116] As described in the second embodiment above, when keywords are extracted from a question and a first search result, some of these keywords may not be suitable for searching for data to generate an answer to the question. In contrast, in this embodiment, the first keywords extracted from the question and the first search result are first classified into classification classes, and the search is narrowed down to second keywords classified into the classification class that is presumed to be necessary for the search (i.e., keywords belonging to that classification class). This makes it possible to generate a more accurate answer using more consistent data than in the second embodiment above.

[0117] In this embodiment, the selection of a classification class is described as being achieved by inputting a question, multiple classification classes, and a prompt to query the classification class necessary to generate an answer to the question into the LLM (a pre-prepared fourth language model). However, the selection of the classification class may be achieved by other methods that do not utilize the LLM. Similarly, the generation of the classification class and the classification of the first keyword may be performed using the LLM or by other methods.

[0118] (Fourth Embodiment) Next, a fourth embodiment will be described. In this embodiment, the description of parts that are the same as those in the second embodiment described above will be omitted, and the parts that differ from the second embodiment will be described mainly.

[0119] The configuration of the information processing system and information processing device in this embodiment is the same as that of the second embodiment described above, and will be explained using Figures 5 and 6 as appropriate.

[0120] Now, with reference to Figure 12, an overview of the operation of the information processing system 1 in this embodiment will be described. Note that this section will primarily focus on the differences from Figure 7 described above.

[0121] The information processing system 1 in this embodiment differs from the second embodiment described above in that, for example, it generates a search class corresponding to keywords for searching for data necessary to generate an answer to a question based on the question, and extracts keywords corresponding to the search class from the question and the first search result.

[0122] Hereinafter, an example of the processing procedure of the information processing device 30 (processing circuit 31) according to this embodiment will be described with reference to the flowchart in Figure 13.

[0123] First, the processes S41 to S43, which correspond to the processes S11 to S13 shown in Figure 8 above, are executed.

[0124] When the process in step S43 is executed, the keyword extraction unit 316 generates a search class (step S44). The search class may be generated based on instructions from, for example, a user who has specified a question (i.e., manually), or it may be generated by inputting a question and a prompt such as "Generate the search class necessary to answer the user's question" into the LLM (i.e., querying the LLM for a search class using the prompt). The LLM used to generate such a search class may be an answer generation LLM or a different LLM.

[0125] In step S44, it is sufficient for a search class to be generated based on a question, for example, and this search class may be generated using other methods.

[0126] Next, the keyword extraction unit 316 extracts keywords corresponding to the search class generated in step S44 from the question obtained in step S41 and the first search result obtained by executing the process in step S43 (step S45). The keywords may also be extracted by inputting a prompt such as "extract keywords corresponding to the search class from the question and the first search result" into the LLM (i.e., querying the LLM for keywords using the prompt).

[0127] When the process in step S45 is executed, the processes in steps S46 to S49, which correspond to the processes in steps S15 to S18 shown in Figure 8 above, are executed.

[0128] As described above, the information processing device 30 according to this embodiment extracts keywords corresponding to the search class generated based on the question from the question and the first search result, and performs a search on the second database 20b using the extracted keywords.

[0129] As mentioned above, in the second embodiment, the keywords extracted from the question and the first search result may include keywords that are not suitable for searching for data to generate an answer to the question. However, in this embodiment, as described above, by extracting keywords corresponding to the search class from the question and the first search result and using them for the search, it is possible to generate a more accurate answer using more consistent data than in the second embodiment described above.

[0130] In this embodiment, for example, the search class is generated by inputting prompts to query the question and search class into the LLM (a pre-prepared fifth language model). However, the generation of the search class may be achieved by other methods that do not utilize the LLM. Similarly, the extraction of keywords corresponding to the search class may be performed using the LLM or by other methods.

[0131] According to at least one embodiment described above, it is possible to provide an information processing device, an information processing method, and a program that can improve the accuracy of answers to questions.

[0132] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0133] 1... Information processing system, 10... Client terminal, 20... Database, 20a... First database, 20b... Second database, 30... Information processing device, 31... Processing circuit, 32... Storage device, 33... Communication device, 311... Query generation unit, 312... Search unit, 313... Consistency determination unit, 314... Prompt generation unit, 315... Answer generation unit, 316... Keyword extraction unit, 317... Keyword selection unit.

Claims

1. A first generation means that generates a query for searching data related to a question specified by the user, A search means that obtains a first search result by performing a search on a pre-prepared first database using the generated query, A determination means for determining whether the aforementioned question and the obtained first search result are consistent, A second generation means for generating a prompt that integrates the aforementioned question and the first search result which is determined to be consistent with the question, A third generation means generates an answer to the question by inputting the generated prompt into a pre-prepared first language model. An information processing device equipped with the following.

2. The information processing apparatus according to claim 1, wherein the determination means determines whether or not the consistency exists based on the similarity between the question and the first search result, whether or not there is a contradiction between the question and the first search result, or whether or not it is possible to answer the question using the first search result.

3. The information processing apparatus according to claim 1 or 2, wherein the determination means determines whether or not the consistency exists by inputting the question, the first search result, and a prompt to inquire about the existence or absence of consistency into a pre-prepared second language model.

4. The aforementioned question and the first search result include text data, The determination means determines whether or not the consistency exists based on the keywords extracted from the text data. The information processing apparatus according to claim 1 or 2.

5. Further equipped with extraction means, The aforementioned question and the first search result include text data, The extraction means extracts keywords from the text data, The search means obtains a second search result by performing a search on a second database different from the first database using the extracted keywords. The determination means determines whether there is consistency between the question and the obtained first and second search results, The second generation means generates a prompt that integrates the question and the search results determined to be consistent with the question. The information processing apparatus according to claim 1.

6. The information processing apparatus according to claim 5, wherein the extraction means extracts the keyword by inputting the question, the first search result, and a prompt to inquire about the keyword into a pre-prepared third language model.

7. Further equipped with sorting means, The extraction means extracts a plurality of first keywords from the text data, The selection means classifies the extracted first keywords into a plurality of classification classes, and selects the second keyword that is classified into the classification class selected from among the plurality of classification classes. The search means performs a search on the second database using the selected second keyword. The information processing apparatus according to claim 5.

8. The information processing apparatus according to claim 7, wherein the selection means selects the classification class by inputting the question, the plurality of classification classes, and a prompt to query the classification class necessary to generate an answer to the question into a pre-prepared fourth language model.

9. The information processing apparatus according to claim 5, wherein the extraction means extracts keywords corresponding to the search class based on the question from the text data.

10. The information processing apparatus according to claim 9, wherein the extraction means generates the search class by inputting the question and the prompt for querying the search class into a pre-prepared fifth language model.

11. The information processing apparatus according to claim 1, wherein the result of the consistency determination is notified to the user.

12. An information processing method performed by an information processing device, Based on a question specified by the user, generate a query to search for data related to that question, The first search result is obtained by performing a search on a pre-prepared first database using the generated query, To determine whether the aforementioned questions and the obtained first search results are consistent, To generate a prompt that integrates the aforementioned question and the first search result which is determined to be consistent with the question, The generated prompt is input to a pre-prepared first language model to generate an answer to the question. An information processing method comprising the following.

13. A program executed by a computer of an information processing device, To the aforementioned computer, Based on a question specified by the user, generate a query to search for data related to that question, The first search result is obtained by performing a search on a pre-prepared first database using the generated query, To determine whether the above question and the first search result obtained are consistent, To generate a prompt that integrates the aforementioned question and the first search result which is determined to be consistent with the question, The generated prompt is input to a pre-prepared first language model to generate an answer to the question. A program to execute.

Images