Document processing program, document processing device, and document processing method

Abstract

To provide a document processing program which generates a retrieval result for obtaining an accurate answer regarding a retrieval result used for prompts of a large-scale language model.SOLUTION: A document processing program of an embodiment causes a computer to achieve a document chunk division function, a document block division function, a document block extraction function, and a document block rearrangement function. The document chunk division function generates a document chunk by dividing a retrieval object document. The document block division function divides the document chunk into document blocks each having a number of characters being a prescribed number of characters or less. The document block extraction function generates a retrieval result in which document blocks having high similarity with a question document are arranged in an order of higher similarity. The document block rearrangement function rearranges the order of the document blocks so that the order of document blocks belonging to a same document chunk becomes sequential, and further orders of document blocks belonging to a same document chunk become the same orders in the document chunk.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a document processing program, a document processing device, and a document processing method. [Background technology]

[0002] In recent years, it has become possible to train large language models (LLMs) using large amounts of text data, and these models are now able to answer questions entered in natural language. Systems using such large language models search for information related to the question from a pre-built document database, and then input the question and search results into the large language model to generate an appropriate answer.

[0003] Generally, there is a limit to the number of characters in a prompt that can be sent to a large-scale language model. Therefore, in order to achieve high-performance search responses with a small number of characters, the document to be searched is divided into small text fragments (document blocks), and prompts are generated using search results that extract parts similar to the question (query) from those fragments. However, if the text fragment included in the prompt only contains part of the document content, an incomplete answer to the question may be generated. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] "An overview of the RAG framework LlamaIndex", [Online], February 27, 2024, [Retrieved May 14, 2024], Internet<URL: https: / / zenn.dev / nomhiro / articles / llama-index-abstract> Summary of the Invention [Problem to be solved by the invention]

[0005] The problem to be solved by the present invention is to provide a document processing program, a document processing device, and a document processing method that generate search results for obtaining highly accurate answers regarding search results used for prompting large-scale language models. [Means for solving the problem]

[0006] To solve these problems, a document processing program according to an embodiment causes a computer to implement a document chunk division function, a document block division function, a document block extraction function, and a document block sorting function. The document chunk division function divides a document to be searched into content units to generate document chunks. The document block division function divides the document chunks into document blocks, each having a predetermined number of characters or less, and assigns an order within the document chunks belonging to the document blocks. The document block extraction function calculates the similarity between each document block and the question sentence, extracts document blocks with high similarity from the document blocks, and generates search results in which the extracted document blocks are sorted in order of highest similarity. The document block sorting function sorts the order of document blocks included in the search results so that the order of document blocks belonging to the same document chunk is consecutive and so that the order of document blocks belonging to the same document chunk is the same as the order within the document chunk. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a question answering system according to a first embodiment. [Figure 2] FIG. 1 is a diagram showing an example of the configuration of a document processing apparatus according to a first embodiment. [Figure 3] FIG. 2 is a view showing an example of a document division method according to the first embodiment. [Figure 4] 10 is a flowchart illustrating a processing procedure of a prompt generation process by the document processing apparatus according to the first embodiment. [Figure 5] FIG. 3 is a diagram showing an example of a data flow in a prompt generation process by the document processing apparatus according to the first embodiment. [Figure 6]FIG. 4 is a diagram showing an example of a search result according to the first embodiment. [Figure 7] FIG. 4 is a diagram showing an example of sorting of search results according to the first embodiment. [Figure 8] FIG. 10 is a diagram showing an example of an experiment result on answer accuracy according to the first embodiment. [Figure 9] FIG. 10 is a histogram of the experimental results of FIG. 9. [Figure 10] FIG. 10 is a diagram showing an example of the configuration of a document processing apparatus according to a second embodiment. [Figure 11] 10 is a flowchart illustrating a processing procedure of a prompt generation process by a document processing apparatus according to a second embodiment. [Figure 12] FIG. 10 is a diagram showing an example of a data flow in a prompt generation process by a document processing apparatus according to a second embodiment. [Figure 13] FIG. 10 is a diagram showing an example of complementing missing document blocks according to the second embodiment. [Figure 14] FIG. 10 is a diagram showing an example of an experiment result on answer accuracy according to the second embodiment. [Figure 15] FIG. 15 is a histogram of the experimental results of FIG. 14. [Figure 16] FIG. 10 is a diagram showing an example of completing a document block according to a modified example of the second embodiment. [Figure 17] FIG. 10 is a diagram showing an example of the configuration of a document processing apparatus according to a third embodiment. [Figure 18] 10 is a flowchart illustrating a processing procedure of a prompt generation process by a document processing apparatus according to a third embodiment. [Figure 19] FIG. 11 is a diagram showing an example of a data flow in a prompt generation process by a document processing apparatus according to a third embodiment. [Figure 20] FIG. 13 is a diagram showing an example of inserting a referenced document block according to the third embodiment. [Figure 21] FIG. 13 is a diagram showing an example of inserting a referenced document block according to a first modified example of the third embodiment. [Figure 22] FIG. 13 is a diagram showing an example of inserting a referenced document block according to a second modified example of the third embodiment. [Figure 23]FIG. 13 is a diagram showing an example of inserting a referenced document block according to a third modified example of the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of a document processing program, a document processing device, and a document processing method will be described in detail with reference to the drawings. In the following description, components having substantially the same functions and configurations are designated by the same reference numerals, and redundant description will be given only when necessary.

[0009] (First embodiment) 1 is a diagram showing the configuration of a question answering system 1 including a document processing device 100 according to a first embodiment. The question answering system 1 is an interactive computer network system that, based on a question input by a user, searches for sentences related to the question from documents to be searched, and generates an answer to the question based on the search results. As shown in FIG. 1, the question answering system 1 includes the document processing device 100, a document database 200, a client terminal 300, and an answer generation device 400.

[0010] The document processing device 100 is connected to a document database 200, a client terminal 300, and a response generating device 400 via a network or the like. The client terminal 300 and the response generating device 400 are also connected via a network or the like. The network is, for example, a LAN (Local Area Network). The connection to the network may be either a wired connection or a wireless connection. The network is not limited to a LAN, and may be the Internet, a public communication line, or the like.

[0011] The document database 200 is a computer that maintains a database that stores multiple document data to be searched. The document data may be, for example, document data in HTML format or PDF format, but may also be data in any other format. In the following description, "document data" will also be simply referred to as "document." Each document is composed of multiple sentences, words, symbols, etc. The types of documents to be searched may be QA (Question and Answer) documents, FAQ (Frequently Asked Question) documents, reports, manuals, or any other type of document.

[0012] The client terminal 300 is a computer used by a user of the question answering system 1. The client terminal 300 has, as hardware, a processor, an input device, a display device, and a communication device, and functions as a user interface for the question answering system 1. For example, the client terminal 300 accepts a question input by a user via the input device. The question is text data related to the content to be searched. The question can also be called a search query from the user. The question may be in natural language or may consist of only words. The client terminal 300 transmits the question input by the user to the document processing device 100 via a network.

[0013] The response generation device 400 is a computer that stores a large-scale language model. The response generation device 400 has a processor and a communication device as hardware. For example, ChatGPT can be used as the large-scale language model. The response generation device 400 receives a prompt sent from the document processing device 100 and inputs the received prompt into the large-scale language model. The response generation device 400 acquires an answer sentence output from the large-scale language model and transmits the acquired answer sentence to the client terminal 300. The answer sentence sent from the response generation device 400 is displayed on a display device of the client terminal 300.

[0014] The document processing device 100 functions as a server device of the question answering system 1. Specifically, the document processing device 100 receives a question from the client terminal 300, searches for sentences related to the question from documents stored in the document database 200 based on the received question, generates a prompt including the question and the search results, and transmits the generated prompt to the answer generation device 400. Since there is a limit to the number of characters that can be input to a prompt in a large-scale language model, the document processing device 100 generates a prompt of a size according to the specifications of the large-scale language model to be used.

[0015] Fig. 2 is a diagram showing an example of the configuration of the document processing device 100. As shown in Fig. 2, the document processing device 100 is a computer having a processing circuit 11, a storage device 12, an input device 13, a communication device 14, and a display device 15. Data communication between the processing circuit 11, the storage device 12, the input device 13, the communication device 14, and the display device 15 is performed via a bus. The input device 13 and the display device 15 do not necessarily have to be provided.

[0016] The processing circuit 11 includes a processor such as a CPU (Central Processing Unit) and a memory such as RAM (Random Access Memory). The processing circuit 11 includes a document chunk division unit 111, a document block division unit 112, a document block extraction unit 113, a document block rearrangement unit 114, and a document block transmission unit 115. By executing a document processing program, the processing circuit 11 realizes the document chunk division function, document block division function, document block extraction function, document block rearrangement function, and document block transmission function of the above-mentioned units. The document processing program is stored in a non-transitory computer-readable recording medium such as the storage device 12. The document processing program may be implemented as a single program that describes all the functions of the above-mentioned units, or as multiple modules divided into several functional units. Furthermore, the above-mentioned units may be implemented by an integrated circuit such as an application-specific integrated circuit (ASIC). In this case, the units may be implemented on a single integrated circuit or individually on multiple integrated circuits.

[0017] The storage device 12 is configured by a ROM (Read Only Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), an integrated circuit storage device, etc. The storage device 12 stores a document processing program and the like.

[0018] Various commands are input from an operator via the input device 13. A keyboard, a mouse, various switches, a touchpad, a touch panel display, etc. can be used as the input device 13. An output signal from the input device 13 is supplied to the processing circuit 11.

[0019] The communication device 14 is an interface for performing data communication with external devices connected to the document processing device 100 via a network. As an example, the communication device 14 performs data communication with the document database 200, the client terminal 300, and the response generation device 400.

[0020] The display device 15 displays various information. As the display device 15, a CRT (Cathode-Ray Tube) display, a liquid crystal display, an organic EL (Electro Luminescence) display, an LED (Light-Emitting Diode) display, a plasma display, or any other display known in the art can be appropriately used. The display device 15 may also be a projector.

[0021] Next, the functions performed by each unit of the processing circuit 11 will be described in detail. The document chunk division unit 111 divides each document to be searched, acquired from the document database 200, into multiple document chunks. A document chunk is a unit obtained by dividing a document into groups of content. FIG. 3 is a diagram showing an example of a method for dividing document chunks. A document chunk is made up of, for example, one chapter or one section within a document. Note that a document chunk may be made up of multiple chapters or sections, or may be made up of a combination of chapters and sections.

[0022] The document block division unit 112 divides each document chunk into multiple document blocks. A document block is a unit obtained by dividing a document chunk so that the number of characters is equal to or less than a predetermined number, and is composed of, for example, sentences. FIG. 3 illustrates an example of a method for dividing document blocks. The number of characters included in a document block is set according to the specifications of the large-scale language model used in the response generation device 400. For example, the maximum size of a prompt that can be input to ChatGPT is several thousand to several tens of thousands of characters. If a prompt exceeding this number of characters is input to ChatGPT, proper processing will not be performed, and the accuracy of ChatGPT's answers will decrease. For this reason, the document block division unit 112 generates document blocks so that a prompt composed of text with a number of characters equal to or less than the number of characters that can be processed by the large-scale language model used in the response generation device 400 is generated. For example, the size of the document block is set according to the size of the prompt that can be used in the large-scale language model and the number of document blocks to be included in the prompt search results.

[0023] Furthermore, the document block division unit 112 assigns to each generated document block information indicating the original document chunk and information indicating the order of the document block within the original document chunk. At this time, the document block division unit 112 generates, for each document block, a set of three pieces of information: the generated document block, information identifying the document chunk to which the document block belongs, and the order of the document block within the document chunk to which it belongs. For example, in the example of FIG. 3, for the second document block of the "Chapter 3" document chunk, a set of the text data included in the document block, the information "Chapter 3," and the information "Second" is generated. As a result, each document block is assigned information indicating where in the original document chunk it was located.

[0024] The document block extraction unit 113 searches for document blocks related to the question entered by the user. To do so, the document block extraction unit 113 first calculates a similarity score between each document block and the question. The similarity score is an index indicating the similarity between multiple texts. Examples of similarity scores that can be used include well-known scores such as TF-IDF between words and cosine similarity between feature vectors. Hereinafter, the similarity score will be simply referred to as "similarity." Next, the document block extraction unit 113 extracts document blocks with high similarity scores as document blocks related to the question. The document block extraction unit 113 then extracts a predetermined number of document blocks in descending order of similarity score and generates search results in which the extracted document blocks are sorted in descending order of similarity score. The search results include text data in which the text of the document blocks is sorted in descending order of similarity score and information including the similarity ranking of each document block used. In other words, the search results include text in which sentences are arranged in descending order of relevance to the question. The search results may also be referred to as extraction results. By using text data in which the text of document blocks is arranged in descending order of similarity as a prompt for a large-scale language model, it is possible to cause the large-scale language model to preferentially use sentences related to the question.

[0025] The document block sorting unit 114 sorts the order of document blocks included in the search results so that document blocks belonging to the same document chunk are arranged together and so that the document blocks belonging to the same document chunk are in the same order as in the original document chunk. At this time, the document block sorting unit 114 changes the order of the document blocks in the search results so that document blocks belonging to the same document chunk are adjacent to each other among the document blocks included in the search results. Then, the document block sorting unit 114 changes the order of the document blocks in the search results so that the order of the document blocks belonging to the same document chunk is the same as the order of the document blocks in the original document chunk. At this time, the document block sorting unit 114 changes the order of document blocks with low similarity among the document blocks belonging to the same document chunk to the same order as document blocks with high similarity among the document blocks belonging to the same document chunk, thereby moving the positions. The sorted search results are stored in the storage device 12. The storage device 12 that stores the sorted search results is an example of a storage unit and may be called a document chunk storage unit.

[0026] The document block sending unit 115 generates a prompt to be input to the large-scale language model by combining the rearranged search results with the question sentence, and sends the generated prompt to the answer generation device 400. The prompt is input to the answer generation device 400 as an instruction to the large-scale language model.

[0027] (Prompt generation process) Next, the operation of the prompt generation process executed by the document processing device 100 will be described. The document processing device 100 starts the prompt generation process based on input of a question sentence at the client terminal 300. FIG. 3 is a flowchart showing an example of the procedure of the prompt generation process. FIG. 4 is a diagram showing an example of the data flow in the prompt generation process. Here, as an example, a case will be described in which sentences related to a question sentence are extracted from a single document consisting of "Chapter A," "Chapter B," "Chapter C," "Chapter D," and "Chapter E," and a prompt is generated using the extracted sentences. Note that the processing procedures in each process described below are merely examples, and each process can be modified as appropriate as possible. Furthermore, steps in the processing procedures described below can be omitted, replaced, or added as appropriate depending on the embodiment.

[0028] (Step S101) In the prompt generation process, first, the document chunk division unit 111 acquires the document to be searched from the document database 200 and divides the acquired document into a plurality of document chunks. Here, document chunk A consisting of all the text contained in chapter A, document chunk B consisting of all the text contained in chapter B, document chunk C consisting of all the text contained in chapter C, document chunk D consisting of all the text contained in chapter D, and document chunk E consisting of all the text contained in chapter E are generated.

[0029] (Step S102) Next, the document block division unit 112 divides each document chunk into multiple document blocks. Note that if the total number of characters included in a document chunk is equal to or less than the number of characters that can be input to the large-scale language model, only one document block containing all the text included in that document chunk is generated.

[0030] (Step S103) Next, the document block division unit 112 generates, for each document block, a set consisting of the document block, the document chunk to which the document block belonged, and the order of the document block within the document chunk to which the document block belonged.

[0031] Here, five document blocks (document block A-1 to document block A-5) are generated from document chunk A, two document blocks (document block B-1 to document block B-2) are generated from document chunk B, one document block (document block C-1) is generated from document chunk C, four document blocks (document block D-1 to document block D-4) are generated from document chunk D, and three document blocks (document block E-1 to document block E-3) are generated from document chunk E. For example, for document block A-1, a set is generated consisting of the text contained in document block A-1, "A" indicating the document chunk to which document block A-1 belongs, and "1" indicating the order of document block A-1 among document blocks A1 to A5 included in document chunk A. Similarly, for all document blocks, a combination of the document chunk to which it belongs and the order in that document chunk is generated.

[0032] (Step S104) Next, the document block extraction unit 113 acquires the question sentence input by the user in the answer generation device 400 from the answer generation device 400 .

[0033] (Step S105) Next, the document block extraction unit 113 calculates the similarity between each document block and the question sentence.

[0034] (Step S106) Next, the document block extraction unit 113 extracts document blocks with high similarity and generates a search result in which the extracted document blocks are sorted in descending order of similarity. Here, an example will be described in which five document blocks with high similarity are extracted and sorted in descending order of similarity. Search result 1 in FIG. 6 shows an example of the search result. In FIG. 6, it can be seen that, of all the document blocks, document block A-1 has the highest similarity, document block E-1 has the second highest similarity, document block A-4 has the third highest similarity, document block C-1 has the fourth highest similarity, and document block A-2 has the fifth highest similarity.

[0035] (Step S107) Next, the document block sorting unit 114 sorts the order of the document blocks included in the search result so that document blocks belonging to the same document chunk are arranged together. At this time, the document block sorting unit 114 changes the order of the document blocks included in the search result so that the document blocks belonging to the same document chunk are arranged in consecutive positions and in the same order as in the document chunk to which they belong. When sorting, the sorting is determined in descending order of similarity among the document blocks included in the search result 1.

[0036] Here, an example will be described in which search result 1 is rearranged. FIG. 7 is a diagram showing search result 2 obtained by rearranging search result 1. When rearranging, first, for document block A-1, which has the highest similarity among the document blocks included in search result 1, document blocks A-4 and A-2 belonging to the same document chunk are extracted. Then, the extracted document blocks A-4 and A-2 are moved to positions contiguous to document block A-1. At this time, document blocks A-1, A-4, and A-2 belonging to the same document chunk A are rearranged so that they are in the same order as they were in document chunk A. As a result, document blocks A-1, A-4, and A-2 are rearranged so that they are contiguous in the order of document block A-1, document block A-2, and document block A-4. Also, in search result 2, among the document blocks A-1, A-4, and A-2 that belong to the same document chunk A, the rankings of the document blocks A-4 and A-2, which have low similarity, are changed to the same ranking (1st place) as the document block A-1, which has high similarity, and are moved to the position of document block A-1.

[0037] Next, for document block E-1, which has the second highest similarity, document blocks belonging to the same document chunk are extracted. Here, since there are no other document blocks belonging to document chunk E in search result 1, document blocks belonging to the same document chunk are not extracted and no reordering is performed. For document block A-4, which has the third highest similarity, reordering is omitted because it has already been reordered. For document block C-1, which has the fourth highest similarity, there are no other document blocks belonging to document chunk C in search result 1, document blocks belonging to the same document chunk are not extracted and no reordering is performed. For document block A-4, which has the fifth highest similarity, reordering is omitted because it has already been reordered.

[0038] By rearranging search result 1 through the above process, search result 2 is generated, in which the document blocks are arranged in the following order: A-1, A-2, A-4, E-1, C-1. As a result, in search result 2, the document blocks are arranged together according to the document chunk to which they belong. Also, in search result 2, document blocks A-1, A-2, and A-4, which belong to the same document chunk A, are arranged in the same order as in document chunk A.

[0039] (Step S108) Next, the document block sending unit 115 generates a prompt by combining the sorted search results with the question sentence, and sends the generated prompt to the response generation device 400.

[0040] The response generation device 400 inputs the prompt received from the document processing device 100 into a large-scale language model to output an answer sentence, and transmits the acquired answer sentence to the client terminal 300. The client terminal 300 displays the answer sentence received from the response generation device 400 on a display device.

[0041] The effects of the document processing device 100 that executes the document processing program and document processing method according to this embodiment will be described below.

[0042] The document processing device 100 according to this embodiment includes a document chunk division unit 111, a document block division unit 112, a document block extraction unit 113, a document block sorting unit 114, and a document block transmission unit 115. The document chunk division unit 111 divides a document to be searched into content units to generate document chunks. For example, a document chunk may be composed of a chapter or a section of a document. The document block division unit 112 divides the document chunk into document blocks each having a predetermined number of characters or less and assigns an order within the document chunks belonging to the document block. The document block extraction unit 113 calculates the similarity between each document block and the question sentence, extracts document blocks with high similarity from among the document blocks, and generates search results in which the extracted document blocks are sorted in descending order of similarity. The number of characters in a document block and the number of document blocks to be extracted are set to satisfy the size of the prompt that can be input to the large-scale language model used. The document block sorting unit 114 sorts the order of the document blocks included in the search results so that the order of the document blocks belonging to the same document chunk is continuous and the order of the document blocks belonging to the same document chunk is the same as the order within the document chunk.

[0043] The document processing device 100 further includes a document block sending unit 115. The document block sending unit 115 generates a prompt using the rearranged search results and the question sentence, and sends the generated prompt to the answer generation device 400. The prompt is input to a large-scale language model in the answer generation device 400. The large-scale language model searches the text of the search results for a description related to the question sentence included in the prompt, and generates an answer sentence including an answer to the question sentence. The answer sentence is sent from the answer generation device 400 to the client terminal 300 and presented to the user.

[0044] Large-scale language models typically analyze the content of search results by checking the text of search results included in prompts from top to bottom. Therefore, by using a prompt that sorts the document blocks of the search results in order of similarity, the large-scale language model can prioritize sentences with high similarity to the question. However, since the text of the search results is simply a concatenation of the text of multiple document blocks, when sentence blocks belonging to different document chunks are concatenated, the content changes at the concatenated portions of the document blocks, making it difficult for the large-scale language model to properly analyze the content, and reducing the accuracy of the large-scale language model. Therefore, if the search results contain many concatenated document blocks belonging to different document chunks, prompts with many content changes are generated, reducing the accuracy of the answers provided by the large-scale language model. By arranging document blocks belonging to the same document chunk consecutively, the document processing device 100 of this embodiment can generate search results in which sentences describing the same content are arranged together.

[0045] For example, in search result 1 shown in Figures 6 and 7, the document blocks are arranged in descending order of similarity to the question, so document blocks A-1, A-2, and A-4 belonging to the same document chunk A are arranged in the first, third, and fifth positions in the search results, and are not arranged in consecutive positions. Furthermore, there are four places where document blocks from different original document chunks are connected, such as the first document block A-1 and the second document block E-1, resulting in a sentence containing four difficult-to-understand passages. On the other hand, in search result 2 shown in Figure 7, document blocks A-1, A-2, and A-4 belonging to the same document chunk A are grouped together, reducing the number of places where document blocks from different original document chunks are connected to two. Therefore, the prompt using search result 2 provides a prompt with higher answer accuracy from a large-scale language model than the prompt using search result 1.

[0046] Thus, in search result 1, document blocks E-1 and C-1, which belong to different document chunks, are placed between document blocks A-1 and A-4 and A-2, which belong to the same document chunk A. This increases the likelihood that document block A-4 will not be treated as having the same content as document block A-1. Furthermore, because the content changes when moving from document block A-1 to document block E-1, the large-scale language model may not use the text from document block E-1 onward to generate an answer. In this case, even though document blocks A-4 and A-2 belong to document chunk A with the same content as document block A-1, which has the highest similarity to the question, they are not used to generate an answer, resulting in reduced answer accuracy. Meanwhile, in search result 2, document blocks A-1, A-2, and A-4, which belong to the same document chunk A, are consecutive. Therefore, document blocks A-1, A-2, and A-4 are appropriately treated as text with the same content, and the text in document blocks A-4 and A-2, which belong to document chunk A and have the same content as document block A-1, which has the highest similarity to the question, are appropriately used to generate the answer.

[0047] In search result 2, document blocks A-1, A-2, and A-4 are arranged in the same order as in the original document chunk A. This results in higher response accuracy when applied to a large-scale language model compared to when document blocks belonging to the same document chunk are arranged in order of similarity.

[0048] Figure 8 shows an example of the change in accuracy of a large-scale language model when search results 1 and 2 are used as prompts for the large-scale language model. The vertical axis in Figure 8 represents the average subjective evaluation score. The average subjective evaluation score is the average of the subjective evaluation scores. The subjective evaluation score is the result of an experiment verifying the accuracy of the answer sentences output by the large-scale language model. In this experiment, evaluators subjectively evaluated the answers to the questions on a five-point scale. The evaluators compared the answers from the large-scale language model with the correct answers prepared in advance and determined the scores for the answers using the following scoring criteria table. In this experiment, the scores were set so that the higher the accuracy of the answer sentence, the higher the score. The average subjective evaluation score was then calculated as the average of the scores from multiple evaluators. Note that if the answer did not contain the correct answer and only contained incorrect information, a negative score was assigned. In addition, OpenAI's "ChatGPT-3.5-Turbo" was used as the large-scale language model in this experiment.

[0049] [Table 1]

[0050] As shown in Figure 8, when search result 1 was used as the prompt, the average subjective evaluation score was 3.7. On the other hand, when search result 2 was used as the prompt, the average subjective evaluation score was 3.83, which was higher than the score for search result 1. Figure 9 shows a histogram of the scores for the experimental results shown in Figure 8. The vertical axis in Figure 9 represents the number of samples. Figure 9 shows that when search result 2 was used as the prompt, the number of scores given a rating of "5" increased and the number of scores given a rating of "3" decreased compared to when search result 1 was used as the prompt. Figures 8 and 9 confirm that using search result 2, in which document blocks were sorted in consideration of the document chunks to which they belong, as the prompt can obtain more accurate answers from a large-scale language model than when search result 1, in which document blocks were sorted in order of similarity, was used as the prompt.

[0051] As described above, the document processing device 100 of this embodiment can generate search results that are easy to read and that can be properly interpreted by a large-scale language model by rearranging the document blocks included in the search results so that document blocks belonging to the same document chunk are arranged together and in the same order as in the original document chunk, even when the search results contain scattered document blocks. Using prompts that utilize the search results generated in this manner can improve the response accuracy of the large-scale language model. In other words, because a prompt is generated that consolidates information from the same document chunk in one place, a prompt can be generated in which text strings are arranged in an easy-to-understand order, allowing the large-scale language model to provide an appropriate response with fewer gaps, thereby improving the response performance of the large-scale language model. In other words, by restoring the text fragments (document blocks) included in the search results to the order in the original manual, the large-scale language model can provide a complete response to the query (question sentence).

[0052] (Second embodiment) A second embodiment will be described. This embodiment is a modification of the configuration of the first embodiment as follows. Descriptions of the same configuration, operation, and effects as those of the first embodiment will be omitted. In this embodiment, missing document blocks in search results are supplemented.

[0053] Fig. 10 is a diagram showing an example of the configuration of a document processing device 100 according to this embodiment. As shown in Fig. 10, in this embodiment, the processing circuitry 11 further includes a document block completion unit 116. The processing circuitry 11 realizes the document block completion function by executing a document processing program.

[0054] If the search results show that multiple document blocks belonging to the same document chunk are not consecutive within the document chunk, the document block completion unit 116 extracts document blocks between these document blocks and inserts the extracted document blocks between the document blocks belonging to the same document chunk.

[0055] (Prompt generation process) Next, the operation of the prompt generation process executed by the document processing device 100 according to this embodiment will be described. Fig. 11 is a flowchart showing an example of the procedure of the prompt generation process. Fig. 12 is a diagram showing an example of the data flow in the prompt generation process. The processes of steps S201 to S207 in Fig. 11 are the same as the processes of steps S101 to S107 in Fig. 4, and therefore will not be described again.

[0056] (Step S208) In the processing of step S207, when the rearrangement of the document blocks included in the search results is completed, if the document blocks belonging to the same document chunk are not in consecutive order within the document chunk, the document block completion unit 116 completes the missing document blocks by inserting document blocks of the discontinuous parts.

[0057] Here, an example will be described in which the above processing is performed on search result 2, which is obtained by rearranging search result 1. FIG. 13 is a diagram showing search result 3, in which document blocks are supplemented for search result 2. First, the document block supplementing unit 116 determines that there is a gap between document block A-2 and document block A-4 because the document blocks A-2 and A-4, which belong to the same document chunk A, are not consecutive in order, and therefore document block A-3 is missing. Then, the document block supplementing unit 116 reads the missing document block A-3 from the storage device 12 and inserts it between document block A-2 and document block A-4 in the search results.

[0058] (Step S208) Next, the text block sending unit 115 generates a prompt by combining the search result in which the missing text blocks have been completed with the question text, and sends the generated prompt to the response generation device 400.

[0059] Note that the search results after the missing document blocks have been supplemented have a larger number of characters than the initial search results. To prevent the total number of characters in the search results from exceeding a specified number, document blocks with low similarity may be excluded from the search results after the document blocks have been supplemented. For example, it is recommended to exclude document block C-1, which has the lowest similarity among search results 3 in Figure 13, and keep the number of document blocks included in the search results below the initial number (five or less).

[0060] The effects of the document processing device 100 that executes the document processing program and document processing method according to this embodiment will be described below.

[0061] Generally, search results generated by extracting document blocks in descending order of similarity may lack elements between the document blocks. For example, as shown in search result 2 in Figure 13, a search result may be generated that lacks the element (document block A-3) between document block A-2 and document block A-4. If a prompt generated using such search results is input to a large-scale language model, an answer sentence will be generated that does not include the content of the missing element.

[0062] On the other hand, the document processing device 100 of this embodiment further includes a document block completion unit 116. When document blocks included in the search results that belong to the same document chunk are not consecutive in order in the document chunk, the document block completion unit 116 extracts document blocks between these document blocks and inserts the extracted document blocks between the document blocks. With the above configuration, the document processing device 100 of this embodiment can generate search results in which missing document blocks are completed, thereby enabling the large-scale language model to output appropriate answer sentences with few deficiencies, thereby improving the answering performance of the large-scale language model.

[0063] FIG. 14 is a diagram showing an example of changes in the accuracy of a large-scale language model when search results 1, 2, and 3 are used as prompts for the large-scale language model. The vertical axis of FIG. 14 shows the average subjective evaluation score. FIG. 15 is a diagram showing a histogram by rating in the experimental results shown in FIG. 14. The vertical axis of FIG. 15 shows the number of samples. FIGS. 14 and 15 show experimental results when an experiment was conducted under the same experimental conditions as those in FIGS. 8 and 9.

[0064] As shown in Figure 14, when search result 3 was used as the prompt, the average subjective evaluation score was 4.48, which was higher than the scores for search result 1 and search result 2. This confirmed that using search result 3, which complements the elements between discontinuous document blocks, as the prompt can obtain more accurate answers from the large-scale language model than using search result 2, which rearranges the document blocks taking document chunks into account, as the prompt.

[0065] Additionally, Figure 15 shows that when search result 3 was used as the prompt, the number of responses rated a "5" increased and the number of responses rated a "4" or lower decreased compared to when search result 2 was used as the prompt. This shows that by compensating for missing document blocks in the search results, the large-scale language model is able to process the search results accurately, as the number of responses rated a "5" or lower, indicating a complete answer, decreased, while the number of responses rated a "4" or lower, indicating a complete answer, increased.

[0066] From the above, according to the document processing device 100 of this embodiment, by rearranging the document blocks included in the search results in the same order as in the document chunks to which they belong, and further by supplementing the intervening elements when the element numbers in the document chunks are discontinuous, the large-scale language model can more easily process prompts and provide an accurate answer. In other words, when the numbers of the document blocks included in the search results are discontinuous, by supplementing the intervening text, the search results become complete manual text, and the large-scale language model can provide a complete answer.

[0067] (Modification of the second embodiment) In the above embodiment, when the element numbers of the document blocks included in the search results are not consecutive, the document blocks between them are complemented. In this modified example, all document blocks that belong to the same document chunk as the document block included in the search results are complemented.

[0068] The document block supplementing unit 116 of this modified example acquires document blocks belonging to the same document chunk for each document block included in the search result, and inserts the acquired document blocks into the search result.

[0069] For example, the above processing will be described below for search result 2. FIG. 16 is a diagram showing search result 4 in which document blocks are supplemented for search result 2. The document block supplementing unit 116 reads document blocks A-3 and A-5, which are included in the same document chunk A as document blocks A-1, A-2, and A-4, from the storage device 12, and inserts them into the search result. In this case, document block A-3 is inserted between document blocks A-2 and A-4, and document block A-5 is inserted after document block A-4, so that the order is the same as in the original document chunk A. Similarly, the document block supplementing unit 116 reads document blocks E-2 and E-3, which are included in the same document chunk E as document block E-1, from the storage device 12, and inserts the read document blocks E-2 and E-3 after document block E-1 in the search result. Note that in the example of FIG. 16, document chunk C is composed only of document block C-1, and therefore document blocks belonging to the same document chunk C as document block C-1 are not inserted into the search result.

[0070] With the above configuration, the document processing device 100 of this modified example complements all information contained in the same document chunk as the highly similar document block, making it easier for the large-scale language model to process prompts and enabling the large-scale language model to provide more accurate answers, thereby further improving the answering performance of the large-scale language model.

[0071] (Third embodiment) A third embodiment will be described. This embodiment is a modification of the configuration of the second embodiment as follows. Descriptions of the same configuration, operation, and effects as those of the second embodiment will be omitted. In this embodiment, if the search results contain information indicating a reference destination, the document block of the reference destination is inserted into the search results.

[0072] Fig. 17 is a diagram showing an example of the configuration of a document processing device 100 according to this embodiment. As shown in Fig. 17, in this embodiment, the processing circuitry 11 further includes a reference block inserting unit 117. The processing circuitry 11 realizes the reference block inserting function by executing a document processing program.

[0073] If the search results contain reference information indicating the reference destination, the reference block insertion unit 117 inserts the reference destination document block into the search results. The reference information is, for example, text indicating the reference destination. The text indicating the reference destination is, for example, text such as "See Chapter X" or text such as "... is discussed in Chapter X." The reference block insertion unit 117 detects the reference information indicating the reference destination from the text contained in the search results, and inserts the reference destination document block after the document block containing the reference information in the search results.

[0074] (Prompt generation process) Next, the operation of the prompt generation process executed by the document processing device 100 according to this embodiment will be described. Fig. 18 is a flowchart showing an example of the procedure for the prompt generation process. Fig. 19 is a diagram showing an example of the data flow in the prompt generation process. The processes of steps S301 to S308 in Fig. 18 are the same as the processes of steps S201 to S208 in Fig. 12, and therefore will not be described here.

[0075] (Step S309) In step S308, when the missing document blocks have been completed, the reference block inserting unit 117 detects reference information indicating the reference destination from the text data included in the search results. The reference block inserting unit 117 reads, from the storage device 12, the document block corresponding to the reference destination indicated in the detected reference information, and inserts the read document block into the search results.

[0076] Here, an example will be described in which the above processing is performed on search result 3 shown in FIG. 13. FIG. 20 is a diagram showing an example of search result 5 generated by inserting a referenced document block into search result 3. In FIG. 20, it is assumed that document block A-1 contains text indicating the reference, "See Chapter B for details." The reference block insertion unit 117 detects the text "See Chapter B for details" as reference information indicating the reference, and detects "Chapter B" as the reference. The reference block insertion unit 117 reads all document blocks B-1 and B-2 included in document chunk B from the storage device 12 as document blocks corresponding to "Chapter B," and inserts them into the search result. In this case, the reference block insertion unit 117 concatenates document blocks B-1 and B-2 after document blocks A-1 to A-4 of document chunk A to which document block A-1, for which reference information was detected, belongs.

[0077] (Step S310) Next, the document block sending unit 115 generates a prompt by combining the search result into which the referenced document block has been inserted with the question text, and sends the generated prompt to the response generation device 400.

[0078] Note that the search results after inserting the referenced document block will have more characters than the original search results. To prevent the total number of characters in the search results from exceeding the specified number of characters, document blocks with low similarity may be excluded from the search results.

[0079] The effects of the document processing device 100 that executes the document processing program and document processing method according to this embodiment will be described below.

[0080] Even if a document block is highly relevant to a question sentence, the explanation may not be accurately understood without referring to descriptions or diagrams in other chapters. The document processing device 100 of this embodiment further includes a reference block insertion unit 117. When the search results contain reference information indicating the reference destination, the reference block insertion unit 117 inserts the referenced document block into the search results. With the above configuration, the document processing device 100 of this embodiment can generate search results that complement information from other document chunks related to the content included in the search results, thereby allowing the large-scale language model to output appropriate answer sentences with few deficiencies, thereby improving the answering performance of the large-scale language model.

[0081] (First modified example of the third embodiment) Instead of text indicating the referenced document chunk, a hyperlink to display the referenced location may be used as reference information. In HTML and PDF documents, hyperlinks that display pre-defined related sections when selected may be embedded within the document. The content of the reference embedded in the hyperlink is often related to the content of the section where the hyperlink is written. Therefore, by inserting into the search results a document block corresponding to the reference of the hyperlink detected from the document block of the search results, the content related to the search results can be supplemented.

[0082] Here, an example will be described in which the above processing is performed on search result 3 shown in FIG. 13. FIG. 21 is a diagram showing an example of search result 6 generated by inserting a referenced document block into search result 3. In FIG. 21, a hyperlink that moves to chapter B when selected is embedded in document block A-3. The reference block insertion unit 117 detects the hyperlink as reference information indicating the reference destination, and detects "chapter B" as the reference destination. The reference block insertion unit 117 reads all document blocks B-1 and B-2 included in document chunk B from the storage device 12 as document blocks corresponding to "chapter B," and inserts them into the search result. In this case, the reference block insertion unit 117 inserts document blocks B-1 and B-2 after document blocks A-1 to A-4 of document chunk A to which document block A-3, whose reference information was detected, belongs.

[0083] (Second modified example of the third embodiment) Furthermore, instead of text indicating the referenced document chunk, text indicating the referenced figure or table may be used as reference information. Examples of text indicating a figure or table include the figure number and title, or the table number and title. In general documents, descriptions indicating the referenced figure or table, such as "As in Table X..." or "The result was... (Figure X)," are sometimes used. The chapters containing these figures or tables are often related to the content of the sections containing the text indicating the referenced figure or table. Therefore, by inserting the document block containing the detected figure or table into the search results, the relevant content of the search results can be supplemented.

[0084] Here, the above processing will be described using an example of search result 3 shown in FIG. 13. FIG. 22 is a diagram showing an example of search result 7 generated by inserting a referenced document block into search result 3. In FIG. 22, document block A-4 includes text indicating the referenced diagram, such as "As in FIG. 6," and document block B-2 displays "FIG. 6." The reference block insertion unit 117 detects the text "As in FIG. 6" as reference information indicating the reference, and detects document block B-2 in which "FIG. 6" is written as the reference. The reference block insertion unit 117 reads all document blocks B-1 and B-2 included in document chunk B to which document block B-2 belongs from the storage device 12 as document blocks corresponding to document block B-2, and inserts them into the search result. In this case, the reference block insertion unit 117 inserts document blocks B-1 and B-2 after document blocks A-1 to A-4 of document chunk A to which document block A-4, whose reference information was detected, belongs.

[0085] (Third Modification of the Third Embodiment) Note that if a document block can be identified as the reference, only the identified document block may be inserted. FIG. 23 is a diagram showing an example of search result 8 generated by inserting the referenced document block into search result 3 shown in FIG. 13. For example, if document block B-2 containing "FIG. 6" is detected as the reference, reference block inserter 117 reads only document block B-2 from storage device 12 as the document block corresponding to document block B-2 and inserts it into the search result. In this case, reference block inserter 117 inserts document block B-2 after document blocks A-1 to A-4 of document chunk A to which document block A-4, whose reference information was detected, belongs. In this case, compared to inserting all document blocks B-1 and B-2 included in document chunk B to which document block B-2 belongs as document blocks corresponding to document block B-2, as shown in FIG. 22, search results can be generated that complement information from other document chunks related to the content included in the search result, and the total amount of text included in the search result can be reduced.

[0086] Thus, according to any of the above-described embodiments, it is possible to provide a document processing program, a document processing device, and a document processing method that generate search results for obtaining highly accurate answers regarding search results used for prompting a large-scale language model.

[0087] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0088] 1...question answering system, 100...document processing device, 11...processing circuit, 12...memory device, 13...input device, 14...communication device, 15...display device, 111...document chunk division unit, 112...document block division unit, 113...document block extraction unit, 114...replacement unit, 115...document block transmission unit, 116...document block completion unit, 117...reference block insertion unit, 200...document database, 300...client terminal, 400...response generation device.

Claims

1. On the computer, A document chunk division function that divides documents to be searched into content blocks and generates document chunks; a document block division function that divides the document chunk into document blocks each having a predetermined number of characters or less and assigns an order within the document chunks that belong to the document block; a document block extraction function that calculates the similarity between each document block and the question sentence, extracts document blocks with high similarity from among the document blocks, and generates search results in which the extracted document blocks are sorted in descending order of similarity; and a document block sorting function that sorts the order of the document blocks included in the search results so that the order of the document blocks belonging to the same document chunk is continuous and the order of the document blocks belonging to the same document chunk is the same as the order within the document chunk; A document processing program to achieve this.

2. a sending function for generating a prompt using the rearranged search results and the question sentence, and sending the generated prompt to a large-scale language model; 2. The document processing program according to claim 1.

3. To further realize a document block completion function that completes document blocks that are missing in the search results, 2. The document processing program according to claim 1.

4. the document block completion function, when document blocks belonging to the same document chunk are not consecutive in order in the document chunk, extracts document blocks between the document blocks belonging to the same document chunk in the document chunk, and inserts the extracted document blocks between the document blocks belonging to the same document chunk in the search results; 4. The document processing program according to claim 3.

5. The document block completion function acquires a document block belonging to the same document chunk for each document block included in the search results, and inserts the acquired document block into the search results.

4. The document processing program according to claim 3.

6. and further realizing a reference block insertion function that, when the search result includes reference information indicating a reference destination, inserts a document block corresponding to the reference destination after the document block including the reference information.

2. The document processing program according to claim 1.

7. the reference information is text indicating the referenced document chunk, the reference block insertion function inserts a document block included in a document chunk corresponding to the text into the search results; 7. The document processing program according to claim 6.

8. the reference information is a hyperlink for displaying the reference destination, the reference block insertion function inserts a document block included in the document chunk corresponding to the hyperlink into the search results; 7. The document processing program according to claim 6.

9. the reference information is text indicating the referenced figure or table, the reference block insertion function inserts a document block included in a document chunk including the diagram into the search results; 7. The document processing program according to claim 6.

10. the document block sorting function changes the ranking of a document block having a low degree of similarity among the document blocks belonging to the same document chunk to the same ranking as a document block having a high degree of similarity among the document blocks belonging to the same document chunk; 2. The document processing program according to claim 1.

11. a document chunk division unit that divides a document to be searched into content units and generates document chunks; a document block division unit that divides the document chunks into document blocks each having a predetermined number of characters or less and assigns an order within the document chunks that belong to the document blocks; a document block extraction unit that calculates a similarity between each of the document blocks and the question sentence, extracts document blocks with high similarity from among the document blocks, and generates search results in which the extracted document blocks are sorted in descending order of similarity; a document block sorting unit that sorts the order of the document blocks included in the search results so that the order of the document blocks belonging to the same document chunk is continuous and the order of the document blocks belonging to the same document chunk is the same as the order within the document chunk; a storage unit that stores the sorted search results; A document processing device comprising:

12. Dividing a document to be searched into chunks of content to generate document chunks; Dividing the document chunks into document blocks each having a predetermined number of characters or less, and assigning an order within the document chunks belonging to the document blocks; calculating a similarity between each of the document blocks and the question sentence, extracting document blocks with high similarity from among the document blocks, and generating search results in which the extracted document blocks are sorted in descending order of similarity; rearranging the order of the document blocks included in the search results so that the order of the document blocks belonging to the same document chunk is continuous and the order of the document blocks belonging to the same document chunk is the same as the order within the document chunk; A document processing method comprising:

Images