Search program, search method, and search device

Abstract

A search device (100) divides a set of structured documents related to external knowledge into a plurality of partial data items for each prescribed unit. The search device (100) generates a prompt including the plurality of partial data items, designated input data, and an instruction to select a partial data item related to the input data. The search device (100) acquires a selection result of a large language model by inputting the prompt into the large language model. The search device (100) searches, on the basis of the selection result, for one partial data item from the plurality of partial data items, and outputs a search result.

Description

Search program, search method, and search device

[0001] The present invention relates to a search program and the like.

[0002] Retrieval-Augmented Generation (RAG) is used as a method to improve the quality of the generated results of Large Language Models (LLMs). RAG extracts external knowledge using a retriever and explicitly adds the extracted external knowledge to the prompt, performing in-context learning for LLMs. RAG has been reported to improve the performance of LLMs and reduce illusions.

[0003] In RAG, external knowledge is stored as a set of characters that can be input into LLM. Retriever is a system that searches for external knowledge that is closest to a question input by the system or a user, based on the surface information of the sentence and the sentence embedding.

[0004] Fig. 6 is a diagram illustrating a conventional Retriever. As shown in Fig. 6, an external knowledge document 10 is input to an external knowledge divider 30 and divided into sentence units or sentence units of an appropriate length. In the following explanation, for the sake of convenience, it is assumed that the external knowledge document 10 is divided into sentence units, and that each divided sentence is assigned an ID.

[0005] The external knowledge document 10 divided into sentence units is input from the external knowledge divider 30 to the retriever 50 as an external knowledge sentence set 11. The retriever 50 also receives as input a question sentence 12 from the system or a user.

[0006] The sentence embedding model 51 of the retriever 50 acquires the external knowledge sentence set 11 and converts each sentence included in the external knowledge sentence set 11 into a vector to generate an external knowledge vector set 13. Each sentence vector is associated with the ID of the sentence that is the source of the conversion. The retriever 50 also converts the question sentence 12 into a vector to generate a question sentence vector 14. The sentence embedding model 51 inputs the external knowledge vector set 13 and the question sentence vector 14 to an approximate nearest neighbor search unit 52.

[0007] The approximate nearest neighbor search unit 52 performs an approximate nearest neighbor search based on the external knowledge vector set 13 and the question statement vector 14. For example, the approximate nearest neighbor search unit 52 outputs the ID of the sentence vector that is closest to the question statement vector 14 among the sentence vectors included in the external knowledge vector set 13 as the search result 15. The retriever 50 retrieves the external knowledge sentence 16 corresponding to the sentence ID included in the search result from the external knowledge sentence set 11, and outputs the retrieved external knowledge sentence 16.

[0008] As explained in FIG. 6, the Retriever 50 executes the above process, thereby making it possible to extract external knowledge that is similar to the question 12 from the external knowledge document 10.

[0009] Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, Tat-Seng Chua “Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering”, arXiv:2101.00774v3 [cs.AI] 8 May 2021 “QA using a Retriever”, [online], searched October 11, 2020, Internet, <URL: https: / / python.langchain.com / docs / use_cases / question_answering / how_to / vector_db_qa>

[0010] However, the above-mentioned conventional techniques have a problem in that they are unable to search for information suitable for a question with high accuracy from a collection of structured documents that serve as external knowledge.

[0011] When using a "set of structured documents that fit into a prompt (number of characters)" as external knowledge, the following problems arise depending on the granularity when dividing the external knowledge into sets of sentences to be searched. As an example, we will explain the set of structured documents related to external knowledge as an Excel file (Excel document). Excel files are structured in a complex manner for each sheet.

[0012] For example, if external knowledge is divided into sentence units or equivalently smaller units such as cells or lines, the divided external knowledge will lack the contextual information and interrelationships between sentences that were contained in the structured document before division. As a result, the external knowledge searched by the Retriever will lack the information necessary for LLM answers.

[0013] On the other hand, when dividing external knowledge into coarse units such as tables or sheets, it is difficult to set the correct vector for the information that has been converted into a string of characters for the entire table or sheet, which reduces the search accuracy of Retriever using sentence embedding.

[0014] That is, with the conventional technology, it is not possible to search with high accuracy for information suitable for input data such as a question from a collection of structured documents that serve as external knowledge.

[0015] In one aspect, the present invention aims to provide a search program, a search method, and a search device that can search for information suitable for input data with high accuracy from a set of structured documents that serve as external knowledge.

[0016] In the first proposal, a computer executes the following process. The computer divides a set of structured documents related to external knowledge into a plurality of partial data sets each of which is a predetermined unit. The computer generates a prompt including the plurality of partial data sets, specified input data, and an instruction to select partial data sets related to the input data. The computer inputs the prompt into a large-scale language model to obtain a selection result of the large-scale language model. Based on the selection result, the computer searches for one partial data set from the plurality of partial data sets and outputs the search result.

[0017] It is possible to search for information suitable for input data with high accuracy from a collection of structured documents that serve as external knowledge.

[0018] FIG. 1 is a diagram for explaining the processing of a search device according to this embodiment. FIG. 2 is a diagram showing an example of a search prompt template. FIG. 3 is a functional block diagram showing the configuration of a search device according to this embodiment. FIG. 4 is a flowchart showing the processing procedure of the search device according to this embodiment. FIG. 5 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the search device of the embodiment. FIG. 6 is a diagram for explaining a conventional Retriever.

[0019] Hereinafter, embodiments of the search program, search method, and search device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments.

[0020] An example of processing by the search device 100 according to this embodiment will be described. FIG. 1 is a diagram illustrating processing by the search device according to this embodiment. The external knowledge document 60 in FIG. 1 is a collection of structured documents such as Excel files, CSV files, and Markdown files. In this embodiment, the external knowledge document 60 will be described as an Excel file having multiple sheets as an example. The size (number of characters) of the external knowledge document 60 is adjusted to be less than the size that fits within a preset prompt.

[0021] When the sheet dividing unit 151 receives the input of the external knowledge document 60, it divides the external knowledge document 60 into sheets. In the following description, the data of the external knowledge document 60 divided into sheets will be referred to as "sheet data." Sheet data is an example of partial data.

[0022] The sheet dividing unit 151 generates an external knowledge sheet set 61 including a plurality of sheet data. The sheet dividing unit 151 assigns IDs to the plurality of sheet data included in the external knowledge sheet set 61. The sheet dividing unit 151 inputs the external knowledge sheet set 61 to the search device 100.

[0023] The search device 100 includes a prompt builder 152, an LLM 153, an answer ID extraction unit 154, and an answer ID aggregation unit 155. The search device 100 may also include the sheet division unit 151. The search device 100 is a Retriever or the like.

[0024] When the prompt builder 152 receives the external knowledge sheet set 61 and the question 62, it generates a search prompt 64 based on a search prompt template 63. For example, the question 62 is a query specified by the system or a user, such as "Please tell me the apple production volume in Aomori Prefecture."

[0025] 2 is a diagram showing an example of a search prompt template. As shown in FIG. 2, the search prompt template includes areas 70, 71, 72, 73, 74, and 75.

[0026] Area 70 is an area where a role is presented to LLM 153. For example, a character string such as "(system) You are a retriever to be used in retrieval augmented generation. We give you the sentences the user has input into the language model and external knowledge documents. Your task is to select one of the external knowledge documents needed so that the language model can generate factual, high-performance sentences in response to user input" is preset in area 70. For example, area 70 contains character strings instructing LLM 153 to be given the role of Retriever, to provide sentences specified by the user, external knowledge, etc.

[0027] Area 71 is an area where preliminary knowledge of the external knowledge document 60 is presented to the LLM 153. For example, area 71 is preset with a string such as "(system)(Step 1) First, I will give you some documents. If it was an Excel file, it will be ill-formatted and converted to CSV, so some values ​​are missing or duplicated. They are enclosed with triple-double quotation. Each document will be written in the following format. No.(x):"""(document_name) / (sheet_name):\\n(content)"""". For example, area 71 instructs the LLM 153 to provide multiple external knowledge documents (multiple sheet data), that each sheet data is enclosed in """", and that each sheet data is assigned an ID "No.(x)".

[0028] Area 72 is an area where each piece of sheet data is set. In the example shown in FIG. 2, areas 72a, 72b, and 72c are included as areas where each piece of sheet data is set. For example, the prompt builder 152 sets the sheet data included in the external knowledge sheet collection 61 and assigned the ID "0" in area 72a. The prompt builder 152 sets the sheet data included in the external knowledge sheet collection 61 and assigned the ID "1" in area 72b. The prompt builder 152 sets the sheet data included in the external knowledge sheet collection 61 and assigned the ID "2" in area 72c.

[0029] 2 shows only areas 72a to 72c, but other areas for setting sheet data may be included. The prompt builder 152 performs similar processing for other sheet data included in the external knowledge sheet set 61. When setting each piece of sheet data in an area, the prompt builder 152 follows the format described in area 71.

[0030] Area 73 is an area where prior knowledge for a query is presented to the LLM 153. For example, a character string such as "(system) (Step 2) The user's input into the language model is below. The language model will generate an answer. But, for it to generate a high-performance and correct answer, he needs one external knowledge document that you will search for." is set in advance in area 73. For example, area 73 instructs the LLM 153 that a sentence (question) specified by the system or the user is set in area 74.

[0031] Area 74 is an area for presenting a query to LLM 153. For example, if question 62 is "Please tell me the apple production volume in Aomori Prefecture," prompt builder 152 sets area 74 to "Please tell me the apple production volume in Aomori Prefecture."

[0032] Area 75 is an area for issuing instructions to the LLM 153. For example, a character string such as "(system) (Step 3) Please select one document that is most relevant to the user's input. The answer should contain a number of documents as 'No. (x)'." is preset in area 75. Area 75 instructs the LLM 153 to output the ID of the sheet data that is most similar to the query set in area 74, from among the multiple sheet data set in area 72.

[0033] As described above, when the prompt builder 152 receives the external knowledge sheet set 61 and the question 62, it generates a search prompt 64 by setting each piece of information in the search prompt template 63. For example, the prompt builder 152 sets each piece of sheet data included in the external knowledge sheet set 61 in area 72 (areas 72a to 72c) of the search prompt template 63 using a predetermined format. The prompt builder 152 also sets the character string of the question 62 in area 74. The prompt builder 152 inputs the generated search prompt 64 into the LLM 153 multiple times.

[0034] Note that the search prompt template 63 described in Figure 2 is a template created on the assumption that ChatGPT (Generative Pretrained Transformer) is used as the LLM. Even if an LLM other than ChatGPT is used as the LLM, it is sufficient to set information equivalent to areas 70 to 76 described in Figure 2 according to the corresponding LLM.

[0035] LLM153 is a natural language processing model trained using large amounts of text data and is suitable for various natural language processing (NLP) tasks. Examples of LLM153 include BERT and ChatGPT. The temperature parameter of LLM153 can take a value between 0 and 1, but in this embodiment, the temperature parameter of LLM153 is set to "0.5 or more (preferably, 0.7 or more)."

[0036] For example, when a search prompt 64 is input, the LLM 153 outputs an LLM generation result. The LLM generation result contains the ID of the sheet data most similar to the query set in area 74, among the multiple sheet data set in area 72. When the search prompt 64 is input N times, the LLM 153 outputs N LLM generation results. In the example shown in FIG. 1 , the LLM 153 outputs LLM generation results 65a, 65b, and 65c to the answer ID extraction unit 154. The LLM generation result is an example of a selection result.

[0037] The answer ID extraction unit 154 extracts the ID of the sheet data included in the LLM generation result as the "answer ID." The answer ID extraction unit 154 compares the LLM generation result with a predetermined regular expression, "No.\((1-9)(0-9)*\)," and sets the value of the first match as the answer ID. For example, if the LLM generation result is "No. (2)," the answer ID will be "2." If the LLM generation result is "No. (3) and No. (5)," the answer ID will be "3."

[0038] 1, the answer ID extraction unit 154 generates an answer ID 66a from the LLM generation result 65a. The answer ID extraction unit 154 generates an answer ID 66b from the LLM generation result 65b. The answer ID extraction unit 154 generates an answer ID 66c from the LLM generation result 65c. The answer ID extraction unit 154 outputs the answer IDs 66a to 66c to the answer ID aggregation unit 155.

[0039] As explained in FIG. 2, the search prompt template 63 is defined in advance so as to output the answer ID in a format that allows the answer ID to be extracted using the regular expression.

[0040] The answer ID aggregating unit 155 aggregates multiple answer IDs into one answer ID 67. For example, the answer ID aggregating unit 155 generates a final answer ID by taking a majority vote on the multiple answer IDs. For example, the answer ID aggregating unit 155 acquires answer IDs 66a to 66c, and if their values ​​are "2, 6, 6," the answer ID aggregating unit 155 sets the value of the final answer ID 67 to "6." If there are two or more most frequent values, the answer ID aggregating unit 155 randomly selects one value from the multiple most frequent values ​​and sets it as the final answer ID 67.

[0041] The answer ID aggregating unit 155 outputs the aggregated answer ID 67. The search device 100 searches the external knowledge sheet collection 61 for sheet data corresponding to the answer ID 67. The search device 100 outputs the searched external knowledge sheet 68. Note that the answer ID aggregating unit 155 in this embodiment aggregates up to five answer IDs into one answer ID.

[0042] The search device 100 searches the external knowledge sheet set 61 for sheet data corresponding to the answer ID 67 , and outputs the sheet data resulting from the search as an external knowledge sheet 68 .

[0043] As described above, the search device 100 generates an external knowledge sheet set 61 by dividing the external knowledge document 60 into sheets, and generates a search prompt 64 based on the external knowledge sheet set 61, the question 62, and the search prompt template 63. The search prompt 64 is set to instruct the LLM 153 to select sheet data similar to the question from multiple sheet data included in the external knowledge sheet set 61. The search device 100 inputs the search prompt 64 to the LLM 153, searches for sheet data from the external knowledge document 60 based on the LLM generation result output from the LLM 153, and outputs the search results. This makes it possible to highly accurately search for information suitable for input data such as a question from a set of structured documents that constitute external knowledge.

[0044] Next, an example of the configuration of the search device 100 that executes the above-mentioned processing will be described. Fig. 3 is a functional block diagram showing the configuration of the search device according to this embodiment. As shown in Fig. 3, the search device 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0045] The communication unit 110 executes data communication with an external device or the like via a network. The communication unit 110 is a network interface card (NIC) or the like. For example, the search device 100 may acquire an external knowledge document 60 or the like from the external device.

[0046] The input unit 120 is an input device that inputs various pieces of information to the control unit 160 of the search device 100. For example, the input unit 120 corresponds to a keyboard, a mouse, a touch panel, etc. The user may operate the input unit 120 to input the question sentence 62.

[0047] The display unit 130 is a display device that displays information output from the control unit 150 .

[0048] The storage unit 140 includes an external knowledge document 60, a set of external knowledge sheets 61, a question sentence 62, a search prompt template 63, a search prompt 64, a set of LLM-generated results 65, and a set of answer IDs 66. The storage unit 140 is a memory or the like.

[0049] The external knowledge document 60 is a collection of structured documents such as excel files, csv files, markdown files, etc. Other explanations regarding the external knowledge document 60 are the same as those regarding the external knowledge document 60 described in FIG.

[0050] 1, the external knowledge sheet set 61 is generated by the sheet dividing unit 151. The external knowledge sheet set 61 includes a plurality of divided sheet data and an ID for identifying each sheet data.

[0051] For Excel files, each sheet converted to CSV format is considered to be one unit of external knowledge (one sheet of data). When converting to CSV, line breaks within cells are converted to the "\n" string. When converting to CSV, spaces are entered after the CSV commas. If the external knowledge document 60 is a Markdown file, one file is considered to be one unit of external knowledge (equivalent to one sheet of data).

[0052] The question 62 is a question (character string) designated by the user operating the system or the input unit 120 .

[0053] The search prompt template 63 is a template used when generating the search prompt 64. Other explanations regarding the search prompt template 63 are the same as those given in FIG.

[0054] Search prompt 64 is a prompt generated by prompt builder 152. The rest of the description of search prompt 64 is the same as that given above.

[0055] The LLM generation result set 65 includes a plurality of LLM generation results output from the LLM 153. The plurality of LLM generation results output from the LLM 153 are the LLM generation results 65a, 65b, 65c, etc. shown in FIG.

[0056] The answer ID set 66 has a plurality of answer IDs output from the answer ID extraction unit 154. The plurality of answer IDs output from the answer ID extraction unit 154 are the answer IDs 66a, 66b, and 66c shown in FIG.

[0057] We now move on to explaining the control unit 150. The control unit 150 includes a sheet dividing unit 151, a prompt builder 152, an LLM 153, an answer ID extraction unit 154, an answer ID aggregation unit 155, and a search result output unit 156. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0058] The sheet dividing unit 151 acquires the external knowledge document 60 and divides the external knowledge document 60 into sheets, thereby generating an external knowledge sheet set 61. The sheet dividing unit 151 assigns an ID to each piece of divided sheet data. Other explanations regarding the sheet dividing unit 151 are the same as those for the sheet dividing unit 151 described in FIG. 1. Note that, in the example of FIG. 3, a case will be described in which the sheet dividing unit 151 is included in the control unit 150, but the function of the sheet dividing unit 151 may be provided in another external device. In this case, the search device 100 acquires the external knowledge sheet set 61 from the external device.

[0059] The prompt builder 152 acquires the external knowledge sheet collection 61 and the question 62, and generates a search prompt 64 based on the search prompt template 63. For example, the prompt builder 152 sets each sheet data included in the external knowledge sheet collection 61 in a predetermined format in an area 72 (areas 72a to 72c) of the search prompt template 63. The prompt builder 152 also sets the character string of the question 62 in an area 74.

[0060] The prompt builder 152 inputs the generated search prompt 64 into the LLM 153 multiple times.

[0061] The LLM 153 is a natural language processing model trained using large amounts of text data and is suitable for various natural language processing tasks. Examples of the LLM 153 include BERT and ChatGPT. For example, the LLM 153 is assigned the role of Retriever by the search prompt 64, and outputs the ID of the sheet data included in the search prompt 64 that is most similar to the question as the LLM generation result. When the search prompt 64 is input N times, the LLM 153 outputs N LLM generation results. Other details regarding the LLM 153 are the same as those described in FIG. 1. The N LLM generation results are registered in the LLM generation result set 65.

[0062] The answer ID extraction unit 154 extracts, as "answer IDs," the IDs of the sheet data included in the multiple LLM generation results included in the LLM generation result set 65. The multiple extracted answer IDs are registered in the answer ID set 66. Other explanations regarding the answer ID extraction unit 154 are the same as those regarding the answer ID extraction unit 154 described in FIG. 1.

[0063] The answer ID aggregating unit 155 aggregates multiple answer IDs included in the answer ID set 66 into one answer ID. The answer ID aggregating unit 155 outputs the aggregated answer ID to the search result output unit 156. Other explanations regarding the answer ID aggregating unit 155 are the same as those regarding the answer ID aggregating unit 155 described in FIG. 1 .

[0064] The search result output unit 156 compares the answer ID acquired from the answer ID aggregation unit 155 with the ID of each sheet data included in the external knowledge sheet set 61, and searches for sheet data having the same ID as the answer ID. The search result output unit 156 outputs the sheet data as the search result to an external device that executes the functions of the RAG.

[0065] Next, an example of the processing procedure of the search device 100 according to this embodiment will be described. Fig. 4 is a flowchart showing the processing procedure of the search device according to this embodiment. As shown in Fig. 4, the sheet dividing unit 151 of the search device 100 divides the external knowledge document 60 into sheets to generate an external knowledge sheet set 61 (step S101).

[0066] The prompt builder 152 of the search device 100 acquires the question 62 from the input unit 120 or the memory unit 140 (step S102). The prompt builder 152 generates a search prompt 64 based on the external knowledge sheet set 61, the question 62, and the search prompt template 63 (step S103).

[0067] The prompt builder 152 inputs the search prompt 64 to the LLM 153 multiple times and obtains multiple LLM generation results from the LLM 153 (step S104). The answer ID extraction unit 154 of the search device 100 extracts multiple answer IDs from the multiple LLM generation results (step S105).

[0068] The answer ID aggregation unit 155 of the search device 100 aggregates multiple answer IDs into one answer ID (step S106). The search result output unit 156 of the search device 100 acquires sheet data corresponding to the aggregated answer IDs from the external knowledge sheet collection 61 (step S107). The search result output unit 156 outputs the acquired sheet data as the search result (step S108).

[0069] Next, the effects of the search device 100 according to this embodiment will be described. The search device 100 generates an external knowledge sheet set 61 by dividing an external knowledge document 60 into sheets, and generates a search prompt 64 based on the external knowledge sheet set 61, a question 62, and a search prompt template 63. The search prompt 64 is set to instruct the LLM 153 to select sheet data similar to the question from multiple sheet data included in the external knowledge sheet set 61. The search device 100 inputs the search prompt 64 to the LLM 153, searches for sheet data from the external knowledge document 60 based on the LLM generation result output from the LLM 153, and outputs the search results. This enables information suitable for input data such as a question to be searched for with high accuracy from a set of structured documents that constitute external knowledge.

[0070] In addition, since it is possible to search with high accuracy for information suitable for input data such as questions from a collection of structured documents that serve as external knowledge, it is expected that the answering performance of RAG systems that utilize Retriever will be improved and confusion will be reduced.

[0071] The search device 100 divides the external knowledge document 60 into a plurality of sheet data and assigns IDs to the divided sheet data to generate an external knowledge sheet set 61. The search device 100 sets an instruction in the search prompt 64 to output the ID of the sheet data that is most relevant to the question 62 among the plurality of sheet data. This makes it possible to identify the ID of the sheet data that is most relevant to the question 62 using the LLM 153.

[0072] The search device 100 obtains multiple LLM generation results by inputting the search prompt 64 multiple times to the LLM 153. The search device 100 also identifies a final answer ID by majority vote of the answer IDs corresponding to the multiple LLM generation results, and outputs the sheet data corresponding to the identified answer ID as the search result. This allows for a more accurate search for sheet data related to the question.

[0073] Next, a description will be given of an example of the hardware configuration of a computer that realizes the same functions as the above-described search device 100. Fig. 5 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the search device of the embodiment.

[0074] 5, computer 200 has a CPU 201 that executes various types of arithmetic processing, an input device 202 that accepts data input from a user, and a display 203. Computer 200 also has a communication device 204 that exchanges data with external devices via a wired or wireless network, and an interface device 205. Computer 200 also has a RAM 206 that temporarily stores various types of information, and a hard disk drive 207. Each of devices 201 to 207 is connected to a bus 208.

[0075] The hard disk drive 207 stores a sheet division program 207a, a prompt builder program 207b, an LLM program 207c, an answer ID extraction program 207d, an answer ID aggregation program 207e, and a search result output program 207f. The CPU 201 reads out each of the programs 207a to 207f and loads them into the RAM 206.

[0076] The sheet division program 207a functions as the sheet division process 206a. The prompt builder program 207b functions as the prompt builder process 206b. The LLM program 207c functions as the LLM process 206c. The answer ID extraction program 207d functions as the answer ID extraction process 206d. The answer ID aggregation program 207e functions as the answer ID aggregation process 206e. The search result output program 207f functions as the search result output process 206f.

[0077] The processing of the sheet division process 206a corresponds to the processing of the sheet division unit 151. The processing of the prompt builder process 206b corresponds to the processing of the prompt builder 152. The processing of the LLM process 206b corresponds to the processing of the LLM 153. The processing of the answer ID extraction process 206d corresponds to the processing of the answer ID extraction unit 154. The processing of the answer ID aggregation process 206e corresponds to the processing of the answer ID aggregation unit 155. The processing of the search result output process 206f corresponds to the processing of the search result output unit 156.

[0078] It should be noted that each of the programs 207a to 207f does not necessarily have to be stored in the hard disk drive 207 from the beginning. For example, each program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207f.

[0079] 60 External knowledge document 61 External knowledge sheet set 62 Question sentence 63 Search prompt template 64 Search prompt 65 LLM generation result set 66 Answer ID set 100 Search device 110 Communication unit 120 Input unit 130 Display unit 140 Memory unit 150 Control unit 151 Sheet division unit 152 Prompt builder 153 LLM 154 Answer ID extraction unit 155 Answer ID aggregation unit 156 Search result output unit

Claims

1. A collection of structured documents relating to external knowledge is divided into multiple subdata units, each of which is a predetermined unit. A prompt is generated that includes the aforementioned plurality of partial data, specified input data, and an instruction to select partial data related to the input data. By inputting the aforementioned prompt into the large-scale language model, the selection result of the large-scale language model is obtained. Based on the selection results, a single partial data point is searched from the multiple partial data points, and the search result is output. A search program characterized by having a computer perform the processing.

2. The search program according to claim 1, wherein the process of assigning identification information to each of the plurality of partial data is further performed by the computer, and the process of generating the prompt is characterized in that the prompt is set to an instruction to select the identification information of the partial data that is most related to the input data from the plurality of partial data.

3. The search program according to claim 2, characterized in that the process of obtaining the selection results is performed by executing the process of inputting the prompt to the large-scale language model multiple times, thereby obtaining multiple selection results from the large-scale language model.

4. The search process is characterized by aggregating the identification information included in the plurality of selection results, searching for one partial data corresponding to the largest number of identification pieces from the plurality of partial data, and outputting the searched one partial data, as described in claim 3.

5. A collection of structured documents relating to external knowledge is divided into multiple subdata units, each of which is a predetermined unit. A prompt is generated that includes the aforementioned plurality of partial data, specified input data, and an instruction to select partial data related to the input data. By inputting the aforementioned prompt into the large-scale language model, the selection result of the large-scale language model is obtained. Based on the selection results, a single partial data point is searched from the multiple partial data points, and the search result is output. A search method characterized by the processing being performed by a computer.

6. A collection of structured documents relating to external knowledge is divided into multiple subdata units, each of which is a predetermined unit. A prompt is generated that includes the aforementioned plurality of partial data, specified input data, and an instruction to select partial data related to the input data. By inputting the aforementioned prompt into the large-scale language model, the selection result of the large-scale language model is obtained. Based on the selection results, a single partial data point is searched from the multiple partial data points, and the search result is output. A search device having a control unit that performs processing.