Information processing apparatus, information processing method, and program

Abstract

To provide an information processor capable of accurately extracting a character string corresponding to a desired item from image data of a document.SOLUTION: The information processing apparatus includes acquisition means for acquiring document information (character string and position information) included in image data of a document, conversion means for converting the acquired document information into a distributed representation, extraction means for extracting a character string corresponding to an item instructed by the acquired document information by inputting a prompt including the acquired document information to a large-scale language model and performing inference by the large-scale language model, storage means for storing the document information related to the document and the extraction result by the information extraction means in a storage unit in association with each other, and selection means for selecting a reference document from previously processed documents based on the distributed representation related to the document to be processed and the distributed representation related to the previously processed document stored in the storage unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] There is known a technique for extracting handwritten or printed character strings from image data such as documents using optical character recognition (OCR). There is also a technique for extracting character strings corresponding to specified items by performing optical character recognition on image data such as documents. Methods for extracting character strings corresponding to specified items include, for example, a method for extracting character strings located in a predetermined direction relative to the item name in the document, and a method for extracting character strings that satisfy conditions (constraints) contained in the image data as candidate character strings, and determining and extracting one character string from the candidate character strings based on the number of times the character string appears in the image data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-46684 Summary of the Invention [Problem to be solved by the invention]

[0004] However, if the item name of the item from which a character string is to be extracted is not included in the image data, it is not always possible to accurately extract the character string corresponding to the item. Furthermore, if the item name of the item from which a character string is to be extracted differs from the item name included in the image data, such as when the item from which a character string is to be extracted is "delivery location" and the item name in the image data indicates the delivery location but is "destination," it is not always possible to accurately extract the character string corresponding to the item. The present invention aims to provide an information processing device, an information processing method, and a program capable of accurately extracting character strings corresponding to desired items from image data of a document. [Means for solving the problem]

[0005] The information processing device of the present invention comprises an acquisition means for acquiring a character string contained in image data of a document and location information indicating the location of the character string as document information; a conversion means for converting the acquired document information into a distributed representation; an information extraction means for inputting a prompt including the acquired document information into a large-scale language model to perform inference using the large-scale language model and extracting a character string corresponding to an item indicated by the prompt; a storage means for associating the document information related to the document, the distributed representation, and the extraction results by the information extraction means and storing them in a storage unit; and a selection means for selecting a reference document from previously processed documents based on the distributed representation related to the document to be processed and the distributed representation related to previously processed documents stored in the storage unit, wherein the prompt input for the document to be processed includes information of the selected reference document. [Effects of the Invention]

[0006] According to the present invention, it is possible to provide an information processing device, an information processing method, and a program that are capable of extracting character strings corresponding to desired items from image data of a document with high accuracy. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an example of a hardware configuration of an information processing device. [Figure 2] FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing device according to the first embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of processing performed by the information processing device according to the first embodiment. [Figure 4] 4 is a flowchart showing an example of processing by the information processing device according to the first embodiment. [Figure 5] FIG. 1 is a diagram illustrating an example of a document, document information, and a distributed representation. [Figure 6] FIG. 10 is a diagram illustrating correction of extraction results. [Figure 7] FIG. 10 illustrates an example of a prompt for extracting information. [Figure 8A] FIG. 10 illustrates an example of a prompt for extracting information. [Figure 8B] FIG. 10 illustrates an example of a prompt for extracting information. [Figure 8C] FIG. 10 illustrates an example of a prompt for extracting information. [Figure 9] FIG. 10 is a diagram illustrating an example of an output of a processing result. [Figure 10] FIG. 10 is a diagram illustrating an example of a functional configuration of an information processing device according to a second embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of processing performed by an information processing device according to a second embodiment. [Figure 12] 10 is a flowchart illustrating an example of processing by an information processing device according to the second embodiment. [Figure 13] 10 is a flowchart illustrating an example of a process for obtaining IoU for a character region. [Figure 14] FIG. 10 is a diagram illustrating IoU for a character region. [Figure 15] FIG. 10 is a diagram illustrating information about documents that have been processed in the past. [Figure 16] FIG. 10 is a diagram illustrating an example of a functional configuration of an information processing device according to a third embodiment. [Figure 17] 10 is a flowchart illustrating an example of processing by an information processing device according to the third embodiment. [Figure 18] FIG. 10 is a diagram illustrating information about documents that have been processed in the past. [Figure 19] FIG. 10 is a diagram illustrating an example of information relating to documents processed in the past. [Figure 20] FIG. 10 illustrates an example of a prompt for extracting information. DETAILED DESCRIPTION OF THE INVENTION

[0008] An embodiment of the present invention will be described with reference to the drawings.

[0009] First Embodiment An information processing device in a first embodiment described below uses optical character recognition (OCR) processing and a large-scale language model (LLM) to extract information (character strings) corresponding to specified items from image data of a document. The information processing device in this embodiment inputs information about the item (item name and item description) from which information (character strings) is to be extracted, a character string contained in the image data of the document, and a prompt including location information indicating the location of the character string within the document into the large-scale language model, thereby performing inference using the large-scale language model to extract information (character strings) corresponding to the desired item. Inputting the prompt including the character string and location information contained in the image data of the document into the large-scale language model allows the large-scale language model to grasp the structure of the document, making it possible to extract information (character strings) taking into account the layout, etc., of the document.

[0010] However, when so-called tacit knowledge or business knowledge is required to extract information (character strings) from a document, it may be difficult to properly extract information (character strings) corresponding to desired items from the document image data. Therefore, in this embodiment, documents similar in format to the document to be processed (hereinafter also referred to as "target document") are selected from previously processed documents and used as reference documents (hereinafter also referred to as "reference documents") for information extraction processing of the target document. Information from the reference documents is then included in prompts as examples for few-shot learning and input into a large-scale language model. This allows knowledge equivalent to tacit knowledge or business knowledge to be easily and efficiently learned, allowing information (character strings) corresponding to desired items to be extracted, thereby improving the accuracy of information extraction.

[0011] 1 is a diagram showing an example of the hardware configuration of an information processing device 100 according to this embodiment. The information processing device 100 includes a CPU 101, a ROM 102, a RAM 103, an auxiliary storage device 104, an output device 105, an input device 106, and a network I / F 107. The CPU 101, the ROM 102, the RAM 103, the auxiliary storage device 104, the output device 105, the input device 106, and the network I / F 107 are communicatively connected via a system bus 108.

[0012] The CPU (Central Processing Unit) 101 is a central processing unit that controls various operations of the information processing device 100. For example, the CPU 101 may control the operation of the entire information processing device 100. The ROM (Read Only Memory) 102 stores control programs, boot programs, etc. that can be executed by the CPU 101. The RAM (Random Access Memory) 103 is the main storage memory of the CPU 101 and is used as a work area or a temporary storage area for expanding various programs.

[0013] The auxiliary storage device 104 stores various data, various programs, etc. The auxiliary storage device 104 is realized by a storage device capable of temporarily or permanently storing various data, such as a non-volatile memory represented by an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0014] The output device 105 is a device that outputs various types of information and is used to present various types of information to a user. For example, the output device 105 is realized by a display device such as a display. The output device 105 may present information to a user by displaying various types of display information. As another example, the output device 105 may be realized by an audio output device that outputs sounds such as voice and electronic sounds. In this case, the output device 105 may present information to a user by outputting sounds such as voice and electronic sounds. Furthermore, the device used as the output device 105 may be changed as appropriate depending on the medium used to present information to a user.

[0015] The input device 106 is used to receive various instructions from a user. For example, the input device 106 may include an input device such as a mouse, a keyboard, or a touch panel. As another example, the input device 106 may include a sound collection device such as a microphone to collect voices uttered by the user. In this case, various analytical processes such as acoustic analysis and natural language processing may be performed on the collected voice, and the content of the voice may be recognized as an instruction from the user. Furthermore, the device used as the input device 106 may be changed as appropriate depending on the method for recognizing instructions from the user. Furthermore, multiple types of devices may be used as the input device 106.

[0016] The network I / F 107 is used for communication with external devices etc. via a network. The device used as the network I / F 107 may be changed as appropriate depending on the type of communication path and the communication method used.

[0017] The CPU 101 loads a program stored in the ROM 102 or the auxiliary storage device 104 into the RAM 103 and executes the program, thereby realizing the functions and processes of the information processing device described below. The program of the information processing device 100 may be provided to the information processing device 100 by a recording medium such as a CD-ROM, or may be downloaded via a network or the like. When the program of the information processing device 100 is provided by a recording medium, the program recorded on the recording medium is installed in the auxiliary storage device 104 by inserting the recording medium into a predetermined drive device.

[0018] 1 is merely an example and does not necessarily limit the hardware configuration of the information processing device 100 in this embodiment. As an example, some components such as the output device 105 and the input device 106 may not be included. As another example, components according to the functions realized by the information processing device 100 may be added as appropriate.

[0019] 2 is a diagram showing an example of the functional configuration of the information processing device 100 according to the first embodiment. The information processing device 100 according to the first embodiment includes a control unit 201, an input / output control unit 202, a storage unit 203, an acquisition unit 204, a conversion unit 205, a selection unit 206, an instruction generation unit 207, an information extraction unit 208, and a correction unit 209.

[0020] The control unit 201 controls each component of the information processing device 100. The input / output control unit 202 executes various processes related to presenting various information to the user and accepting information input (e.g., instructions, etc.) from the user. For example, the input / output control unit 202 may execute processes related to presenting a UI (User Interface) and accepting input via the UI. This enables the information processing device 100 to recognize instructions from the user and present the results of processing according to those instructions to the user.

[0021] The storage unit 203 schematically illustrates a storage area for storing various data, various programs, and the like. For example, the storage unit 203 may store data and programs for each component of the information processing device 100 to execute processing. The storage unit 203 may also store a trained model that has undergone machine learning (deep learning) and that is used for inference regarding information extraction performed in the information processing device 100. The storage unit 203 may also store document information regarding documents, distributed representations obtained by converting the document information, and extraction results of document information.

[0022] The acquisition unit 204 optically scans or photographs a document to acquire image data of the document. The acquisition unit 204 also performs optical character recognition (OCR) processing on the acquired document image data to extract and acquire document information related to the document. The document information includes character strings included in the document image data and location information indicating the location of the character strings within the document. The acquisition unit 204 may also acquire document information by receiving as input image data of the document obtained by externally scanning or photographing the document in advance, without scanning or photographing the document. The acquisition unit 204 may also acquire document information by receiving as input character strings included in the document and their location information obtained as a result of external optical character recognition processing.

[0023] The conversion unit 205 converts the document information about the document acquired by the acquisition unit 204 into a distributed representation (embedded representation). The distributed representation of the document information represents the document information as a multidimensional real vector, and vectors indicated by the distributed representation of document information about similar documents are close (have a small distance). The conversion unit 205 converts the document information into a distributed representation, for example, using a trained embedding model that converts natural language into a numerical vector. The trained embedding model may be a trained embedding model such as Sentence-BERT or OpenAI's text embedding models. In addition to these trained embedding models, a trained embedding model may be generated by performing machine learning to convert document information into a distributed representation (embedded representation), and the generated trained embedding model may be applied. For example, a trained embedding model may be generated by preparing a large number of pairs of a certain sentence and similar sentences as training data, and training the system so that multidimensional vectors generated from similar sentences are similar vectors.

[0024] As an example of this embodiment, the conversion unit 205 converts character strings and location information (combinations of character strings and location information) in document information related to a document into a distributed representation for each page of the document (on a page-by-page basis). However, this is not limiting, and the conversion unit 205 may convert the entire document (on a document-by-document basis) into a distributed representation. Furthermore, the conversion unit 205 may convert only character strings in document information related to a document into a distributed representation, or may convert only location information of character strings in document information related to a document into a distributed representation. Furthermore, although a distributed representation is used in this embodiment, a method other than a distributed representation may be used as long as it is possible to evaluate similarity.

[0025] The selection unit 206 selects documents (reference documents) to be used as references in information extraction processing for a document to be processed (target document) from among previously processed documents whose document information, distributed representations, and information extraction results are stored in the storage unit 203. The selection unit 206 selects reference documents from among documents previously processed by machine learning (e.g., K-Nearest Neighbor: KNN) based on the distributed representations related to the document. Specifically, the selection unit 206 compares the distributed representations related to the target document obtained by conversion in the conversion unit 205 with the distributed representations related to previously processed documents stored in the storage unit 203, and selects a predetermined number of documents whose format is similar to that of the target document (documents whose distributed representations are close to that of the target document) from among previously processed documents, and sets them as reference documents.

[0026] Here, documents with similar formats have similar distributed representations due to similarities in the types of character strings and the arrangement of positional information in the documents, and the selection unit 206 can identify documents with similar formats using machine learning such as k-nearest neighbor. Furthermore, by evaluating similarity using the distributed representations of documents, the selection unit 206 can acquire highly similar documents even if the formats are slightly different or the contents differ. Note that the method for comparing the distributed representation of a target document with the distributed representation of previously processed documents is not limited to k-nearest neighbor, and any other method that compares distributed representations (vectors) can be applied. For example, the comparison of distributed representations may be performed using a support vector machine (SVM), an artificial neural network (ANN), cosine similarity, or the like.

[0027] In this embodiment, the instruction generation unit 207 generates instructions for extracting information (character strings) from a target document and inputs them to the information extraction unit 208, which then uses a large-scale language model (LLM) to extract information (character strings) according to the input instructions. The large-scale language model (LLM) is a language model constructed using a large amount of text data (such as a large-scale corpus) and deep learning technology, and when text data called a prompt indicating an instruction or the like is input, it performs inference based on the prompt and generates and outputs text data according to the input prompt.

[0028] The instruction generation unit 207 generates a prompt including instructions for extracting information (character strings) corresponding to a desired item from the target document and document information of the target document, and inputs the generated prompt to the information extraction unit 208. Inference using a large-scale language model (LLM) involves a method called few-shot learning, which utilizes the characteristics of large-scale language models, namely, including examples (samples) in the prompt to highly accurately return an output adapted to the presented examples (samples). Therefore, in this embodiment, if the reference document selected by the selection unit 206 is highly similar to the target document (e.g., a document whose distributed representation distance from the target document is equal to or less than a predetermined threshold), the instruction generation unit 207 generates a prompt using document information and extraction results related to the selected reference document as examples for few-shot learning, and inputs the generated prompt to the information extraction unit 208. In this way, by providing information (document information and extraction results) of reference documents highly similar to the target document to the large-scale language model (LLM) as examples for few-shot learning, knowledge equivalent to tacit knowledge or business knowledge can be easily and efficiently learned, enabling accurate extraction of information (character strings) corresponding to a desired item.

[0029] The information extraction unit 208 performs inference using a large-scale language model (LLM) based on the prompt generated and input by the instruction generation unit 207, and extracts information (character strings) corresponding to the items instructed to be extracted by the prompt from the image data of the target document. The correction unit 209 accepts a correction request from the user regarding the extraction result of information about the target document by the information extraction unit 208, and corrects the extraction result in accordance with the correction request. The extraction result of information about the target document extracted by the information extraction unit 208 (or the corrected extraction result if correction has been made by the correction unit 209) is associated with the document information acquired by the acquisition unit 204 and the shared information acquired by conversion by the conversion unit 205, and stored in the storage unit 203.

[0030] The processing in the information processing device 100 in the first embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram illustrating an example of processing in the information processing device 100 in the first embodiment.

[0031] The information processing device 100 performs document information acquisition processing 302 to acquire document information (character strings included in image data and their position information) from image data of a document to be processed (target document) 301. The information processing device 100 performs document information conversion processing 303 on the acquired document information related to the target document, and converts the document information (combination of character strings and position information) into a distributed representation using a trained embedding model or the like.

[0032] Next, the information processing device 100 performs a reference document acquisition process 304 to select documents (reference documents) to be used as references in the information extraction process from the target document from among previously processed documents stored in a database (DB) 312. The information processing device 100 compares the distributed representations of the target document obtained in the conversion process 303 with the distributed representations of previously processed documents stored in the database 312, and acquires a predetermined number of documents from the previously processed documents in descending order of similarity (closest distance) between the distributed representations and the target document as reference documents. Note that in this embodiment, the information processing device 100 selects reference documents from documents stored in the database 312 by a storage process 307 (described later) among previously processed documents stored in the database 312. This is because the extraction results stored by the storage process 307 (described later) are extraction results that have been modified as necessary in accordance with a user instruction, and are generally considered to be of higher quality than the extraction results stored by the storage process 311.

[0033] After acquiring the reference documents in the acquisition process 304, the information processing device 100 determines whether the acquired reference documents include a document with a similar format to the target document. For example, if there is a reference document whose distance of the distributed representation from the target document is equal to or less than a predetermined threshold, the information processing device 100 determines that there is a document with a similar format to the target document; otherwise, it determines that there is no document with a similar format to the target document.

[0034] If it is determined that the acquired reference documents do not include documents with a similar format to the target document, the information processing device 100 performs an information extraction process 305 from the target document. In this extraction process 305, the information processing device 100 performs inference regarding information extraction by inputting a prompt including instructions for extracting information (character strings) corresponding to specified items and document information of the target document into a large-scale language model (LLM), and extracts information (character strings) corresponding to the specified items in the target document. In this case, since previously processed documents do not include documents with a similar format to the target document, the information processing device 100, for example, generates a prompt using examples of documents showing how to extract information from documents as examples for few-shot learning without using the information of the reference documents acquired in the acquisition process 304, and inputs the prompt into the large-scale language model (LLM). Note that although the similarity to the target document is low, the similarity to the target document is high compared to other documents processed in the past. Therefore, a prompt using the information of the reference document acquired in the acquisition process 304 as examples for few-shot learning may be generated and input into the large-scale language model (LLM).

[0035] After performing the extraction process 305, the information processing device 100 performs an extraction result correction process 306, correcting the extraction results of the extraction process 305 as necessary based on a correction request from the user, etc. The information processing device 100 presents the extraction results of the extraction process 305 to the user and accepts a correction request from the user for the presented extraction results. For example, suppose that the extraction result of the extraction process 305 shows "no information" because the item name is different from the item instructed to be extracted in the prompt, but information (character string) corresponding to that item is included in the image data of the target document (document information). In this case, if the user, after checking the presented extraction results, requests a correction to correct the extraction result of "no information" to information included in the image data of the target document (document information), the extraction results are corrected to the user-specified information in accordance with the correction request from the user.

[0036] After performing the correction process 306, the information processing device 100 performs a storage process 307 of document information, distributed representations, and extraction results related to the target document, and stores the document information, distributed representations, and extraction results related to the target document in association with each other in a database 312. For example, the information processing device 100 assigns an identifier (ID) that can uniquely identify the target document, and associates the identifier (ID) with the document information acquired in the acquisition process 302, the distributed representations obtained in the conversion process 303, and the corrected extraction results after the correction process 306, and stores them in the database 312.

[0037] If it is determined that the acquired reference documents include a document with a similar format to the target document, the information processing device 100 performs a prompt generation process 308 including information about the reference document, and generates a prompt to be provided to a large-scale language model (LLM) in an extraction process 310 (described later). The information processing device 100 references an item name and an item description 309 to generate a prompt including instructions for extracting information (character strings) corresponding to the specified item from the target document and document information about the target document. The information processing device 100 also generates a prompt that includes information about the reference document (document information and extraction results) acquired in the acquisition process 304 as examples for few-shot learning. For example, the information processing device 100 generates a prompt by incorporating the document information and extraction results of the reference document stored in the database 312 as examples for few-shot learning.

[0038] After performing the generation process 308, the information processing device 100 performs an information extraction process 310 from the target document. In this extraction process 310, the information processing device 100 performs inference regarding information extraction by inputting the prompt generated in the generation process 308 into a large-scale language model (LLM), and extracts information (character strings) corresponding to the specified items.

[0039] After performing extraction process 310, information processing device 100 performs storage process 311 of document information, distributed representations, and extraction results related to the target document, and associates and stores the document information, distributed representations, and extraction results related to the target document in database 312. For example, information processing device 100 assigns an identifier (ID) that can uniquely identify the target document, and associates the identifier (ID) with the document information acquired in acquisition process 302, the distributed representations obtained in conversion process 303, and the extraction results in extraction process 310, and stores them in database 312. Here, because reference documents are acquired in acquisition process 304 from documents stored in storage process 307, it is preferable that the document information, distributed representations, and extraction results related to the target document stored in storage process 307 and the document information, distributed representations, and extraction results related to the target document stored in storage process 311 be stored separately in respective databases (or memory areas).

[0040] FIG. 4 is a flowchart showing an example of processing by the information processing device 100 according to the first embodiment. In step S401, the acquisition unit 204 performs optical character recognition on the image data of the document to be processed (target document) 301, thereby acquiring document information of the target document (character strings included in the image data and their position information). For example, the acquisition unit 204 scans or photographs a target document such as that shown in FIG. 5A to acquire image data, performs optical character recognition processing on the acquired image data, and acquires document information of the target document (character strings and their position information) as shown in FIG. 5B. In FIG. 5B, as an example, the document information of the target document is shown in the format of "character string:position information." The position information of the character string indicates four values: the minimum x-coordinate, the minimum y-coordinate, the maximum x-coordinate, and the maximum y-coordinate of the character string in an XY coordinate system with one end point of the document (e.g., the upper left point or the lower left point) as the origin. In the case of a document including a table such as that shown in FIG. 5A, the acquisition unit 204 recognizes and excludes the borders of the table in the document image data to acquire the character string. Alternatively, the character string may be acquired by dividing an area based on the recognized borders of the table.

[0041] In step S402, the conversion unit 205 converts the document information of the target document acquired in step S401 (a combination of character strings and position information) into a distributed representation using a trained embedding model, etc. For example, the conversion unit 205 inputs the document information of the target document as shown in Fig. 5(B) into the trained embedding model, thereby converting it into a distributed representation (multi-dimensional real vector) as shown in Fig. 5(C).

[0042] In step S403, the selection unit 206 compares the distributed representation of the target document acquired in step S402 with the distributed representation of previously processed documents using machine learning (k-nearest neighbor method), and selects, from the previously processed documents, documents to be used as references in the information extraction process from the target document (reference documents). For example, the selection unit 206 compares the distributed representation of the target document with the distributed representation of previously processed documents that have been subjected to correction processing on the extraction results and stored in the storage unit 203, and acquires, from the previously processed documents, a predetermined number of documents as reference documents in descending order of similarity (nearest distance) between the distributed representation of the target document and the document.

[0043] In step S404, the instruction generation unit 207 determines whether or not the reference documents selected in step S403 include a document with a high format similarity to the target document. The instruction generation unit 207 determines whether or not there is a reference document with a high format similarity based on the distance between the distributed representation of the target document and the distributed representation of the reference document. For example, if the distance between the distributed representation of the target document and the distributed representation of the reference document is equal to or less than a predetermined threshold, the instruction generation unit 207 determines that the format similarity is high; otherwise, it determines that the format similarity is not high. If the instruction generation unit 207 determines that the selected reference documents include a document with a high format similarity to the target document (YES), the process of step S409 is executed. On the other hand, if the instruction generation unit 207 determines that the selected reference documents do not include a document with a high format similarity to the target document (NO), the process of step S405 is executed.

[0044] In step S405, which is executed when the instruction generation unit 207 determines that there is no document with a format highly similar to that of the target document among the selected reference documents, the information extraction unit 208 extracts information from the target document using a large-scale language model (LLM). Specifically, the information extraction unit 208 performs inference regarding information extraction by inputting a prompt including instructions for extracting information (character strings) corresponding to specified items generated by the instruction generation unit 207 and document information of the target document into the large-scale language model (LLM), and extracts information (character strings) corresponding to the specified items from the target document. In this case, for example, the instruction generation unit 207 generates a prompt using example documents showing how to extract information from documents as examples for few-shot learning, and inputs the prompt to the large-scale language model (LLM).

[0045] In step S406, the control unit 201 outputs the extraction results of the information about the target document obtained in the processing of step S405. Note that the control unit 201 may output the document information of the target document acquired in step S401 in addition to the information extraction results.

[0046] In step S407, the correction unit 209 corrects the extraction results in response to a correction request from the user, etc. For example, if a user who has confirmed the extraction results of the information related to the target document output in step S406 requests that the extraction results be corrected, the correction unit 209 accepts the correction request and corrects the extraction results in accordance with the correction request. As an example, assume that the extraction results shown in FIG. 6(A) are obtained in step S406 for the target document shown in FIG. 5(A) based on the document information of the target document shown in FIG. 5(B). In the example shown in FIG. 6(A), the presence or absence of information for the item "Case Name" is "None" and the content of the information is "Unknown." However, assume that the information described in the item "Business Content" in the target document shown in FIG. 5(A) corresponds to the item "Case Name." In this case, if the user requests a correction to be made to the content of the "Job Description" item in the target document (the complete "Basic Plan for Renovating a Factory Production System") as information corresponding to the "Project Name" item, the correction unit 209 corrects the extraction results by correcting the presence or absence of information for the "Project Name" item in the extraction results for the target document, and the content of the information, as shown in FIG. 6(B). By providing these corrected extraction results as examples for few-shot learning, it is possible to train the large-scale language model to recognize that the "Job Description" item corresponds to the "Project Name" item in a document such as that shown in FIG. 5(A). In this way, labeling of the extraction results can be completed by correcting only the item information portion, which allows labeling to be performed with minimal effort and is easier than conventional labeling. Furthermore, by providing these corrected extraction results to the large-scale language model as examples for few-shot learning via prompts, it is possible to simply incorporate the corrected extraction results without having to input them as sentences into the prompts each time processing is performed, thereby enabling information equivalent to tacit knowledge, business knowledge, and the like to be easily and efficiently learned.

[0047] In step S408, the control unit 201 associates the document information, the distributed representations, and the extraction results related to the target document and stores them in the database (storage unit 203). For example, the control unit 201 assigns an identifier (ID) that can uniquely identify the target document, and associates the identifier (ID) with the document information acquired in step S401, the distributed representations obtained in step S402, and the corrected extraction results corrected in step S407, and stores them in the database. After executing the process of step S408, the information processing device 100 ends the process shown in FIG. 4.

[0048] In step S409, which is executed when the instruction generation unit 207 determines that the selected reference documents include a document whose format is highly similar to that of the target document, the instruction generation unit 207 generates a prompt to be given to the large-scale language model (LLM). The instruction generation unit 207 generates a prompt that includes information about the reference document selected in step S403 (document information and extraction results) along with instructions for extracting information (character strings) corresponding to specified items from the target document and the document information about the target document.

[0049] An example of the prompt generated in step S409 is shown in FIG. 7. As shown in FIG. 7, prompt 700 includes an information (character string) extraction method description 710, item names and item descriptions to be extracted 720, example document information and answers 730, example output formats 740, and document information of the target document 750. Extraction method description 710 is an explanation of how to extract information from the document information of the target document, and item names and item descriptions to be extracted 720 are item names and descriptions of one or more items to be extracted. Here, the item description describes, for example, general knowledge (explanation) that can be commonly used in documents to be processed. Including the item description in the prompt makes it possible to update the knowledge of items held by the large-scale language model (LLM) to common knowledge and to learn knowledge of items not held by the large-scale language model (LLM). Updating to common knowledge and learning common knowledge makes it possible to accurately extract character strings corresponding to specified items. Document information and answer example 730 is an example (sample) for few-shot learning, and includes document information of the reference document selected in step S403 and extraction results 731, 732. Output format example 740 is an example of the output format when information (character strings) corresponding to the item to be extracted is output as the extraction result. Document information 750 of the target document is the document information of the target document acquired in step S401, and includes document information 751 for each page of the target document.

[0050] Specific examples of the prompt shown in Figure 7 are shown in Figures 8A to 8C. Combining the prompts shown in Figures 8A to 8C corresponds to one prompt. In Figures 8A to 8C, element 810 corresponds to the explanation of the extraction method, and element 820 corresponds to the item names to be extracted and the item explanation group 720. Furthermore, element 830 corresponds to document information and answer example 730, which includes information on three reference documents 831, 832, and 833. Furthermore, element 840 corresponds to the output format example 740, and element 850 corresponds to document information 750 of the target document.

[0051] 4, in step S410, the information extraction unit 208 extracts information from the target document using a large-scale language model (LLM). Specifically, the information extraction unit 208 performs inference regarding information extraction by inputting a prompt including an instruction for extracting information (character strings) corresponding to the desired item generated in step S409, the document information of the target document, the document information of the reference document, and the extraction result into the large-scale language model (LLM), and extracts information (character strings) corresponding to the instructed item from the target document.

[0052] In step S411, the control unit 201 outputs the extraction result of information about the target document obtained in the process of step S410. Note that the control unit 201 may output, in addition to the extraction result of information, the document information of the target document obtained in step S401, the similarity (distance) between the distributed representations of the reference document selected in step S403 and the target document, and the like.

[0053] In step S412, the control unit 201 associates the document information, the distributed representations, and the extraction results related to the target document and stores them in the database (storage unit 203). For example, the control unit 201 assigns an identifier (ID) that can uniquely identify the target document, and associates the identifier (ID) with the document information acquired in step S401, the distributed representations obtained in step S402, and the extraction results obtained in step S410, and stores them in the database. After executing the process of step S412, the information processing device 100 ends the process shown in FIG. 4.

[0054] FIG. 9 is a diagram illustrating an example of an output of a processing result related to information extraction from a target document by the information processing device 100 according to the first embodiment. The processing result shown in FIG. 9 is displayed by the output device 105 via the input / output control unit 202 and presented to the user, for example. In the processing result 900 shown in FIG. 9, the few-shot distance 910 indicates the distance between the embedded representation of a reference document selected from previously processed documents and the embedded representation of the target document. In this example, the distances between the embedded representation of the target document and three reference documents are shown. The extraction result 920 indicates the extraction result of information related to the target document. The extraction result may include, for example, the name of the item to be extracted, the presence or absence and content of information corresponding to the item, etc. The processed document information 930 indicates document information (character strings and position information) of the target document acquired based on the image data of the target document. Note that the output example of the processing result shown in FIG. 9 is merely an example and is not limited thereto; other information related to the information extraction process may also be displayed.

[0055] According to the first embodiment, the information processing device 100 can extract information (character strings) corresponding to desired items from image data of a target document by inputting a prompt containing instructions for extracting information (character strings) corresponding to specified items and document information of the target document into a large-scale language model. Furthermore, by acquiring previously processed documents with a format similar to that of the target document as reference documents and incorporating the previously obtained information of the reference documents (document information and extraction results) into the prompt as examples for few-shot learning, the information equivalent to tacit knowledge, business knowledge, etc. can be easily and efficiently learned without having to transcribe everything as text, and information (character strings) corresponding to desired items can be accurately extracted from image data of the target document.

[0056] Note that, as the position information of the character string, the coordinate values ​​of the character string in an XY coordinate system with one end point of the document as the origin (minimum and maximum x-coordinate values, minimum and maximum y-coordinate values) are used, but the coordinate values ​​may be normalized using the width and height of the document so that the values ​​fall within the range of 0 to 1 and used as the position information of the character string. By using normalized position information, even if the documents are not the same size, documents with similar layouts when viewed as a whole page are more likely to be selected as reference documents, thereby improving the accuracy of selecting reference documents.

[0057] <Second embodiment> A second embodiment will be described. In the first embodiment, reference documents are obtained from previously processed documents based on the distributed representations of the documents, but in the second embodiment described below, reference documents are obtained from previously processed documents based on the intersection over union (IoU) of character string areas between documents instead of using distributed representations. This is because documents with the same layout will generally have similar character string positions within the documents, and a high IoU is considered to indicate documents with similar formats.

[0058] The hardware configuration of the information processing device 100 in the second embodiment is similar to the hardware configuration of the information processing device in the first embodiment shown in FIG. 1, and therefore a description thereof will be omitted.

[0059] 10 is a diagram showing an example of the functional configuration of an information processing device 100 according to the second embodiment. The information processing device 100 according to the second embodiment includes a control unit 1001, an input / output control unit 1002, a storage unit 1003, an acquisition unit 1004, a calculation unit 1005, a selection unit 1006, an instruction generation unit 1007, an information extraction unit 1008, and a correction unit 1009.

[0060] The control unit 1001 controls the various components of the information processing device 100. The input / output control unit 1002 executes various processes related to presenting various types of information to the user and accepting information input (e.g., instructions, etc.) from the user. For example, the input / output control unit 1002 may execute processes related to presenting a UI and accepting input via the UI. This enables the information processing device 100 to recognize instructions from the user and present the results of processing in accordance with those instructions to the user.

[0061] The storage unit 1003 schematically illustrates a storage area for storing various data, various programs, etc. For example, the storage unit 1003 may store data and programs for each component of the information processing device 100 to execute processing. The storage unit 1003 may also store a trained model that has undergone machine learning (deep learning) used for inference regarding information extraction, or may store document information regarding documents and extraction results of document information.

[0062] The acquisition unit 1004 optically scans or photographs a document to acquire image data of the document. The acquisition unit 1004 also performs optical character recognition (OCR) processing on the acquired document image data to extract and acquire document information related to the document. The document information includes character strings included in the document image data and location information indicating the location of the character strings within the document. The acquisition unit 1004 may also acquire document information by receiving as input image data of the document obtained by externally scanning or photographing the document in advance, without scanning or photographing the document. The acquisition unit 1004 may also acquire document information by receiving as input character strings included in the document and their location information obtained as a result of external optical character recognition processing.

[0063] The calculation unit 1005 calculates the overlap ratio (IoU) of character string areas (character areas) between a document to be processed (target document) and a previously processed document. The overlap ratio (IoU) of character areas is calculated as {(area of ​​the overlapping area between character area A and character area B) / (total area of ​​the two areas, character area A and character area B)}, where the character area to be calculated in the target document is character area A and the character area to be calculated in the previously processed document is character area B, and the value is in the range of 0 to 1. In this embodiment, the calculation unit 1005 determines the overlap ratio (IoU) of a bounding box (rectangle) surrounding a character string, which is specified by character string position information in the document information, as the overlap ratio (IoU) of the character string area (character area). The calculation unit 1005 also calculates an average overlap ratio (average IoU) for each page of the document. Alternatively, the calculation unit 1005 may calculate an average overlap ratio (average IoU) for each document. The average overlap ratio (average IoU) will be described in detail later.

[0064] Based on the overlap ratio (IoU) of the character string area (character area), the selection unit 1006 selects documents (reference documents) to be used as references in information extraction processing of the document to be processed (target document) from among previously processed documents whose document information, etc. is stored in the storage unit 1003. Specifically, based on the average overlap ratio (average IoU) acquired by calculation in the calculation unit 1005, the selection unit 1006 selects a predetermined number of documents in descending order of average overlap ratio (average IoU) from among previously processed documents whose average overlap ratio (average IoU) is equal to or greater than a predetermined threshold, and sets these documents as reference documents.

[0065] In this embodiment, too, the instruction generation unit 1007 generates instructions for extracting information (character strings) from the target document and inputs them to the information extraction unit 1008, which then uses a large-scale language model (LLM) to extract the information (character strings) according to the input instructions.

[0066] The instruction generation unit 1007 generates a prompt including instructions for extracting information (character strings) corresponding to desired items from the target document and document information about the target document, and inputs the generated prompt to the information extraction unit 1008. Furthermore, if there is a reference document selected by the selection unit 1006 as a document highly similar to the target document, the instruction generation unit 1007 generates a prompt in which the document information about the reference document and the extraction results are used as examples for few-shot learning, and inputs the generated prompt to the information extraction unit 1008. In this way, by providing the information (document information and extraction results) of the reference document highly similar to the target document to the large-scale language model (LLM) as examples for few-shot learning, knowledge equivalent to tacit knowledge, business knowledge, etc. can be easily and efficiently learned, and information (character strings) corresponding to desired items can be accurately extracted.

[0067] The information extraction unit 1008 performs inference using a large-scale language model (LLM) based on the prompt generated and input by the instruction generation unit 1007, and extracts information (character strings) corresponding to the items instructed to be extracted by the prompt from the image data of the target document. The correction unit 1009 accepts a correction request from the user regarding the information extraction result of the information about the target document by the information extraction unit 1008, and corrects the extraction result in accordance with the correction request. The extraction result of the information about the target document extracted by the information extraction unit 1008 (or the corrected extraction result if correction has been made by the correction unit 1009) is stored in the memory unit 1003 in association with the document information acquired by the acquisition unit 1004.

[0068] The processing in the information processing device 100 in the second embodiment will be described with reference to Fig. 11. Fig. 11 is a diagram illustrating an example of processing in the information processing device 100 in the second embodiment.

[0069] The information processing device 100 performs document information acquisition processing 1102 to acquire document information (character strings included in image data and their position information) from image data of a document (target document) 1101 to be processed.

[0070] Next, the information processing device 100 performs IoU acquisition processing 1103 for the character area, and calculates and acquires the overlap ratio (IoU) of the character string area (character area) between the document to be processed (target document) and previously processed documents stored in the database (DB) 1111. Furthermore, the information processing device 100 calculates and acquires the average overlap ratio (average IoU) based on the calculated overlap ratio (IoU) of the character string area (character area).

[0071] After acquiring the average overlap ratio (average IoU) in the acquisition process 1103, the information processing device 100 determines whether or not any document with the acquired average overlap ratio (average IoU) equal to or greater than a threshold is included in the documents processed in the past.

[0072] If it is determined that there is no document with an average overlap ratio (average IoU) equal to or greater than the threshold, the information processing device 100 performs information extraction processing 1104 from the target document. In this extraction processing 1104, the information processing device 100 performs inference regarding information extraction by inputting a prompt including instructions for extracting information (character strings) corresponding to specified items and document information of the target document into a large-scale language model (LLM), and extracts information (character strings) corresponding to the specified items from the target document. In this case, since there are no documents processed in the past that have a layout similar to the target document such that the average overlap ratio (average IoU) is equal to or greater than the threshold, the information processing device 100 generates a prompt using example documents showing how to extract information from documents as examples for few-shot learning and inputs the prompt to the large-scale language model (LLM).

[0073] After performing the extraction process 1104, the information processing device 100 performs a correction process 1105 of the extraction result, and corrects the extraction result obtained in the extraction process 1104 as necessary based on a correction request from the user, etc. As with the correction process 306 in the first embodiment shown in Fig. 3, the information processing device 100 presents the extraction result obtained in the extraction process 1104 to the user, accepts a correction request from the user regarding the presented extraction result, etc., and corrects the extraction result to information specified by the user in accordance with the correction request from the user.

[0074] After performing the correction process 1105, the information processing device 100 performs a process 1106 of storing document information and extraction results related to the target document, and stores the document information related to the target document and the extraction results in association with each other in a database 1111. For example, the information processing device 100 assigns an identifier (ID) that can uniquely identify the target document, and stores the identifier (ID) in association with the document information acquired in the acquisition process 1102 and the extraction results after the correction process 1105 in the database 1111.

[0075] If it is determined that there is a document with an average overlap ratio (average IoU) equal to or greater than a threshold, the information processing device 100 acquires a predetermined number of documents from previously processed documents in descending order of average overlap ratio (average IoU) as reference documents, performs prompt generation process 1107 including information about the reference documents, and generates a prompt to be provided to a large-scale language model (LLM) in extraction process 1109, which will be described later. The information processing device 100 references item names and item descriptions 1108 to generate a prompt including instructions for extracting information (character strings) corresponding to the specified items from the target document and document information about the target document. The information processing device 100 also generates a prompt that includes information about the acquired reference documents (document information and extraction results) as examples for few-shot learning. For example, the information processing device 100 generates a prompt by incorporating the document information and extraction results of the reference documents stored in database 1111 as examples for few-shot learning. Note that the process of acquiring reference documents from previously processed documents may be for one document or multiple documents.

[0076] After performing the generation process 1107, the information processing device 100 performs an information extraction process 1109 from the target document. In this extraction process 1109, the information processing device 100 performs inference regarding information extraction by inputting the prompt generated in the generation process 1107 into a large-scale language model (LLM), and extracts information (character strings) corresponding to the specified items.

[0077] After performing extraction process 1109, information processing device 100 performs storage process 1110 of document information and extraction results related to the target document, and stores the document information related to the target document and the extraction results in association with each other in database 1111. For example, information processing device 100 assigns an identifier (ID) that can uniquely identify the target document, and stores the identifier (ID) in database 1111 in association with the document information acquired in acquisition process 1102 and the extraction results in extraction process 1109.

[0078] FIG. 12 is a flowchart showing an example of processing by the information processing device 100 according to the second embodiment. In step S1201, the acquisition unit 1004 scans or photographs a document to be processed (target document) to acquire image data, performs optical character recognition processing on the acquired image data, and acquires document information of the target document (character strings included in the image data and their position information). Here, the character string position information is acquired, for example, by acquiring four values: the minimum x-coordinate, the minimum y-coordinate, the maximum x-coordinate, and the maximum y-coordinate of the character string in an XY coordinate system with one end point (e.g., the upper left or lower left point) of the document as the origin. The rectangle specified by these four values ​​becomes a bounding box indicating the area of ​​the character string (character area). Furthermore, in the optical character recognition processing, information indicating the width (horizontal length) and height (vertical length), which are the size of the document (corresponding to the maximum x-coordinate and the maximum y-coordinate that can be taken on the document), is acquired. The character string position information is normalized using the acquired width and height of the document so that the value falls within the range of 0 to 1. By using normalized position information, even if the documents are not the same size, a high average IoU value can be obtained for documents that have similar layouts when viewed on a page-by-page basis, thereby improving the accuracy of selecting documents to use as reference documents.

[0079] In step S1202, the calculation unit 1005 calculates and obtains the overlap ratio (IoU) for the character string area (character area) between the target document obtained in step S1201 and previously processed documents. Furthermore, the calculation unit 1005 calculates and obtains the average overlap ratio (average IoU) based on the calculated overlap ratio (IoU) of the character string area (character area).

[0080] Here, the process of acquiring the overlap ratio (IoU) of a character string area (character area) will be described with reference to Fig. 13. Fig. 13 is a flowchart illustrating an example of the process of acquiring the overlap ratio (IoU) of a character string area (character area). Fig. 13 shows an example of acquiring the overlap ratio (IoU) of a character string area (character area) on corresponding pages between a target document and a previously processed document. In the following description of Fig. 13, "target document" refers to a page to be processed in the target document, and "previous document" refers to a page in a previously processed document that corresponds to the page to be processed in the target document.

[0081] In step S1301, the calculation unit 1005 acquires a character string included in the target document and position information of a bounding box surrounding the character string, based on the document information of the target document acquired in step S1201 of Fig. 12. Here, as described above, the bounding box indicating the character area is a rectangle specified by the value of the character string position information, and therefore the position information of the bounding box and the character string position information are the same information (the same applies hereinafter). That is, in the processing of step S1301, the calculation unit 1005 acquires document information about the page to be processed (a character string and its position information) from the document information of the target document (a character string and its position information) acquired in step S1201 of Fig. 12.

[0082] In step S1302, the calculation unit 1005 acquires character strings included in each past document and position information of the bounding box surrounding the character string, based on information about previously processed documents stored in the storage unit 1003. FIG. 15 is a diagram illustrating information about previously processed documents (past documents) stored in the storage unit 1003. Information about previously processed documents is saved, for example, in a json format as shown in FIG. 15. For each character string included on a page, the character string and position information of the bounding box surrounding the character string (character string position information) are saved for each page. The bounding box position information is normalized so that the coordinate value is a value between 0 and 1 based on the height and width of the document (the vertical and horizontal sizes of the image data). In addition, as the extraction results, item names and the contents of one or more items corresponding to the item names are saved.

[0083] The processing of steps S1301 and S1302 may be performed in any order, and the processing of step S1301 may be performed after the processing of step S1302.

[0084] In step S1303, the calculation unit 1005 selects one unprocessed bounding box from among the bounding boxes in the target document.

[0085] Next, in step S1304, the calculation unit 1005 calculates and obtains the overlap ratio (IoU) between the bounding box selected in step S1303 and each bounding box in the previous document. For example, assume that page X1400 shown in FIG. 14 is the page to be processed in the target document, page A1410 is the page corresponding to the page to be processed in the target document in previously processed document A, and page B1420 is the page corresponding to the page to be processed in the target document in previously processed document B. Also assume that bounding box 1401 in page X1400 was selected in step S1303 described above. In this case, in step S1304, the calculation unit 1005 obtains the overlap ratio (IoU) between bounding box 1401 and bounding box 1411 based on the position information of bounding box 1401 and the position information of bounding box 1411 in page A1410. Similarly, the calculation unit 1005 obtains the overlap ratios (IoU) between the bounding box 1401 and other bounding boxes on page A1410, that is, the overlap ratios (IoU) between the bounding box 1401 and the bounding box 1412, and the overlap ratios (IoU) between the bounding box 1401 and the bounding box 1413. Furthermore, the calculation unit 1005 obtains the overlap ratios (IoU) between the bounding box 1401 and the bounding box 1421 based on the position information of the bounding box 1401 and the position information of the bounding box 1421 on page B1420. Similarly, the calculation unit 1005 obtains the overlap ratios (IoU) between the bounding box 1401 and other bounding boxes on page B 1420, i.e., the overlap ratio (IoU) between the bounding box 1401 and the bounding box 1422, and the overlap ratio (IoU) between the bounding box 1401 and the bounding box 1423. Similar processing is performed for other past documents. That is, the processing of step S1304 described above is performed for each of the past documents.

[0086] In step S1305, the calculation unit 1005 selects, for each past document, the largest overlap ratio (IoU) as the overlap ratio (IoU) for the bounding box selected in step S1303. For example, in the example shown in Fig. 14, the largest overlap ratio (IoU) among the overlap ratios (IoU) between the bounding box 1401 on page X 1400 and the bounding boxes 1411, 1412, and 1413 on page A 1410 is selected as the overlap ratio (IoU) for the bounding box 1401 in past document A. Furthermore, the largest overlap ratio (IoU) among the overlap ratios (IoU) between the bounding box 1401 on page X 1400 and the bounding boxes 1421, 1422, and 1423 on page B 1420 is selected as the overlap ratio (IoU) for the bounding box 1401 in past document B. Similar processing is performed for other past documents. That is, the processing of step S1305 described above is performed for each past document. Note that if the largest inter-unit-of-union (IoU) is 0, that is, if there is no bounding box in the past document page that overlaps with the bounding box selected in step S1303, the inter-unit-of-union (IoU) for the selected bounding box is set to 0.

[0087] Next, in step S1306, the calculation unit 1005 determines whether or not there are any unprocessed bounding boxes in the target document. If the calculation unit 1005 determines that there are any unprocessed bounding boxes in the target document (YES), the process returns to step S1303, and the processing from step S1303 onwards is executed again. If the calculation unit 1005 determines that there are no unprocessed bounding boxes in the target document, that is, that the processing from step S1303 onwards has been executed for all bounding boxes (NO), the process proceeds to step S1307.

[0088] In step S1307, the calculation unit 1005 calculates and acquires an average overlap ratio (average IoU), which is the average value of the selected overlap ratios (IoU), for each past document, based on the overlap ratios (IoU) selected for each bounding box in the target document in step S1305. For example, in the example shown in FIG. 14, if the overlap ratios (IoU) for bounding boxes 1401, 1402, and 1403 in past document A are a, b, and c, respectively, then (a+b+c) / 3 is set as the average overlap ratio (average IoU) for past document A (more specifically, the page in past document A that corresponds to the page to be processed in the target document). Similar processing is performed for other past documents. That is, the processing of step S1307 described above is performed for each past document.

[0089] As described above, the calculation unit 1005 obtains the average overlap ratio (average IoU) of pages in past documents that correspond to the page to be processed in the target document, calculates the average value of the average overlap ratios (average IoU) of each page in the past documents, and obtains the average overlap ratio (average IoU) for all past documents. Note that when obtaining the average overlap ratio (average IoU) for all past documents, if there is no page in the past documents that corresponds to the page to be processed in the target document, that page is excluded from the calculation of the average. This process is performed for each document processed in the past. In this way, the calculation unit 1005 obtains the average overlap ratio (average IoU) for each document processed in the past.

[0090] In the above description, the average overlap ratio (average IoU) for the entire document is obtained for each previously processed document. However, if the processing from step S1203 onward in the flowchart shown in FIG. 12 is performed on a page-by-page basis rather than on a document-by-document basis, the average overlap ratio (average IoU) for each page may be obtained for each previously processed document. Furthermore, the average overlap ratio (average IoU) is obtained by comparing the page to be processed in the target document with a page in a past document corresponding to the page to be processed in the target document. However, the average overlap ratio (average IoU) may also be obtained by comparing the page to be processed with each page in the past document. Comparing a past document with a page corresponding to the page to be processed in the target document can reduce calculation time and improve accuracy compared to comparing with each page in the past document. Therefore, whether to compare a past document with a page in the past document corresponding to the page to be processed in the target document or with each page in the past document may be selected appropriately depending on, for example, the time required for processing, the load, accuracy, etc.

[0091] 12, in step S1203, the instruction generation unit 1007 determines whether or not any of the previously processed documents has an average overlap ratio (average IoU) equal to or greater than a threshold, based on the average overlap ratio (average IoU) for each previously processed document acquired in step S1202. If the instruction generation unit 1007 determines that any of the previously processed documents has an average overlap ratio (average IoU) equal to or greater than the threshold (YES), the process proceeds to step S1208. On the other hand, if the instruction generation unit 1007 determines that any of the previously processed documents does not have an average overlap ratio (average IoU) equal to or greater than the threshold (NO), the process proceeds to step S1204.

[0092] In step S1204, which is executed when the instruction generation unit 1007 determines that there is no document with an average overlap ratio (average IoU) equal to or greater than a threshold among previously processed documents, the information extraction unit 1008 extracts information from the target document using a large-scale language model (LLM). Specifically, the information extraction unit 1008 performs inference regarding information extraction by inputting a prompt, generated by the instruction generation unit 1007, for extracting information (character strings) corresponding to specified items and including document information of the target document into the large-scale language model (LLM), and extracts information (character strings) corresponding to the specified items from the target document. In this case, for example, the instruction generation unit 1007 generates a prompt using example documents showing how to extract information from documents as examples for few-shot learning, and inputs the prompt to the large-scale language model (LLM).

[0093] In step S1205, the control unit 1001 outputs the extraction results of the information about the target document obtained in the processing of step S1204. Note that the control unit 1001 may output the document information about the target document acquired in step S1201 in addition to the information extraction results.

[0094] In step S1206, the correction unit 1009 corrects the extraction results in response to a correction request from the user, in the same manner as in step S407 in the first embodiment shown in Fig. 4. For example, if a user who has confirmed the extraction results of the information related to the target document output in step S1205 requests correction of the extraction results, the correction unit 1009 accepts the correction request and corrects the extraction results in accordance with the correction request.

[0095] In step S1207, the control unit 1001 associates the document information about the target document with the extraction results and stores them in the database (storage unit 1003). For example, the control unit 1001 assigns an identifier (ID) that can uniquely identify the target document, and associates the identifier (ID) with the document information acquired in step S1201 and the corrected extraction results corrected in step S1206 and stores them in the database. After executing the process of step S1207, the information processing device 100 ends the process shown in FIG. 12.

[0096] In step S1208, which is executed when the instruction generating unit 1007 determines that there is a document with an average overlap ratio (average IoU) equal to or greater than a threshold among the documents processed in the past, the selecting unit 1006 selects the document with the highest average overlap ratio (average IoU) from among the documents processed in the past as the reference document. Note that in this example, the document with the highest average overlap ratio (average IoU) is selected as the reference document, but multiple documents may also be selected in descending order of average overlap ratio (average IoU).

[0097] In step S1209, the instruction generation unit 1007 generates a prompt to be given to the large-scale language model (LLM). The instruction generation unit 1007 generates a prompt including instructions for extracting information (character strings) corresponding to specified items from the target document, document information of the target document, and information on the reference document selected in step S1208 (document information and extraction results). The prompt generated in step S1209 is similar to the prompt generated in step S409 in the first embodiment shown in FIG. 4, and therefore a description thereof will be omitted.

[0098] In step S1210, the information extraction unit 1008 extracts information from the target document using a large-scale language model (LLM). Specifically, the information extraction unit 1008 performs inference regarding information extraction by inputting a prompt including an instruction for extracting information (character strings) corresponding to the desired item generated in step S1209, document information of the target document, document information of the reference document, and the extraction result into the large-scale language model (LLM), and extracts information (character strings) corresponding to the instructed item from the target document.

[0099] In step S1211, the control unit 1001 outputs the extraction results of information about the target document obtained in the processing of step S1210. Note that the control unit 1001 may output, in addition to the extraction results of information, document information about the target document and the average IoU of documents that have been processed in the past and selected as reference documents.

[0100] In step S1212, control unit 1001 associates the document information about the target document with the extraction results and stores them in the database (storage unit 1003). For example, control unit 1001 assigns an identifier (ID) that can uniquely identify the target document, and associates the identifier (ID) with the document information acquired in step S1201 and the extraction results obtained in step S1210 and stores them in the database. After executing the process of step S1212, information processing device 100 ends the process shown in FIG. 12.

[0101] According to the second embodiment, the information processing device 100 can extract information (character strings) corresponding to desired items from image data of a target document by inputting a prompt containing instructions for extracting information (character strings) corresponding to specified items and document information of the target document into a large-scale language model. Furthermore, documents with similar layouts based on the average overlap ratio (average IoU) of the character string areas between the target document and the previously processed documents are acquired as reference documents from previously processed documents, and information on the previously acquired reference documents (document information and extraction results) is included (incorporated) in the prompt as an example for few-shot learning and input to the large-scale language model. This allows information equivalent to tacit knowledge, business knowledge, etc. to be easily and efficiently learned without having to transcribe everything as text, thereby enabling information (character strings) corresponding to desired items to be accurately extracted from image data of the target document.

[0102] <Third embodiment> A third embodiment will be described. The hardware configuration of the information processing device 100 in the third embodiment is similar to the hardware configuration of the information processing device in the first embodiment shown in FIG. 1, and therefore a description thereof will be omitted.

[0103] 16 is a diagram showing an example of the functional configuration of an information processing device 100 according to the third embodiment. The information processing device 100 includes a control unit 1601, an input / output control unit 1602, a storage unit 1603, an acquisition unit 1604, a calculation unit 1605, a selection unit 1606, an instruction generation unit 1607, a first information extraction unit 1608, a modification unit 1609, and a second information extraction unit 1610.

[0104] The control unit 1601 controls the various components of the information processing device 100. The input / output control unit 1602 executes various processes related to presenting various types of information to the user and accepting input of information (e.g., instructions, etc.) from the user. For example, the input / output control unit 1602 may execute processes related to presenting a UI and accepting input via the UI. This enables the information processing device 100 to recognize instructions from the user and present the results of processing in accordance with those instructions to the user.

[0105] The storage unit 1603 schematically illustrates a storage area for storing various data, various programs, etc. For example, the storage unit 1603 may store data and programs for each component of the information processing device 100 to execute processing. The storage unit 1603 may also store a trained model that has undergone machine learning (deep learning) used for inference regarding information extraction, or may store various information related to documents, such as document information and information extraction results.

[0106] In the third embodiment, document information refers to information extracted from a document based on the items to be extracted (items to be extracted), in which the item names corresponding to the items to be extracted, the contents of the items corresponding to the item names (item contents), and location information relating to each of the item names and item contents are associated with the items to be extracted. The location information is information indicating the location in the document of the character strings indicating the item names and item contents.

[0107] The acquisition unit 1604 optically scans or photographs a document to acquire image data of the document. The acquisition unit 1604 also performs optical character recognition (OCR) processing on the acquired image data of the document to acquire character strings included in the image data of the document and position information related to the character strings. The acquisition unit 1604 may also be configured to accept and acquire, as input, image data of the document obtained by scanning or photographing the document in advance externally, without scanning or photographing the document itself. The acquisition unit 1604 may also be configured to accept and acquire, as input, character strings included in the document and their position information obtained as a result of external optical character recognition processing.

[0108] The calculation unit 1605 calculates the overlap ratio (IoU) of the character string area (character area) between the document to be processed (target document) and previously processed documents. In this embodiment, the calculation unit 1605 sets the overlap ratio (IoU) of the bounding box (rectangle) surrounding the character string, which is identified by the position information of the character string in the document information, as the overlap ratio (IoU) of the character string area (character area). Furthermore, the calculation unit 1605 calculates the average overlap ratio (average IoU) for each page of the document. Furthermore, the calculation unit 1605 may be configured to calculate the average overlap ratio (average IoU) for each document.

[0109] Based on the overlap ratio (IoU) of the character string area (character area), the selection unit 1606 selects a document (reference document) to be used as a reference in information extraction processing of the document to be processed (target document) from among previously processed documents whose document information and the like is stored in the storage unit 1603. Specifically, based on the average overlap ratio (average IoU) acquired by calculation in the calculation unit 1605, the selection unit 1606 selects, as the reference document, a document whose average overlap ratio (average IoU) is equal to or greater than a predetermined threshold and has the highest average overlap ratio (average IoU).

[0110] In the information processing device 100 of this embodiment, it is possible to extract information relating to the specified extraction target item from the image data of the document using a large-scale language model (LLM) by the instruction generation unit 1607 and the first information extraction unit 1608. For example, the instruction generation unit 1607 generates instructions for extracting various information relating to the extraction target item from the target document and inputs the instructions to the first information extraction unit 1608, and the first information extraction unit 1608 extracts information according to the input instructions using the large-scale language model (LLM).

[0111] The instruction generation unit 1607 generates a prompt including instructions for extracting various information related to the items to be extracted from the target document, and character strings included in the image data of the target document and their location information, and inputs the generated prompt to the first information extraction unit 1608. In this embodiment, the various information related to the items to be extracted from the target document includes the item names (character strings) corresponding to the items to be extracted, the item contents (character strings) corresponding to the item names, and location information (coordinate information) of the item names and the item contents. The instruction generation unit 1607 may also generate a prompt using document information of documents highly similar to the target document (e.g., documents with an average IoU equal to or greater than a threshold) as examples for few-shot learning, and input the generated prompt to the first information extraction unit 1608. In this way, by providing information about documents highly similar to the target document to a large-scale language model (LLM) as examples for few-shot learning, the accuracy of extracting item names and item contents corresponding to the items to be extracted from the target document can be improved.

[0112] The first information extraction unit 1608 performs inference using a large-scale language model (LLM) based on the prompt generated and input by the instruction generation unit 1607, and extracts various pieces of information related to the items to be extracted as instructed by the prompt from the image data of the target document. The correction unit 1609 accepts a correction request from the user regarding the results of extraction of information related to the target document by the first information extraction unit 1608, and corrects the extraction results in accordance with the correction request.

[0113] Furthermore, in the information processing device 100 of this embodiment, the second information extraction unit 1610 can use document information of previously processed documents to extract information about specified items to be extracted from the image data of the documents based on the item names and location information of the item contents. The second information extraction unit 1610 extracts various pieces of information about the specified items to be extracted from the image data of the target document based on the item names and location information of the item contents in the document information of the previously processed documents. Specifically, the second information extraction unit 1610 refers to a reference document selected from previously processed documents by the selection unit 1606, and extracts various pieces of information about the items to be extracted from the target document based on the item names and location information of the item contents in the reference document.

[0114] FIG. 17 is a flowchart showing an example of processing by the information processing device 100 according to the third embodiment. In step S1701, the acquisition unit 1604 scans or photographs a document to be processed (target document) to acquire image data, performs optical character recognition (OCR) processing on the acquired image data, and acquires character strings and their position information included in the image data of the target document. Here, the character string position information is acquired, for example, as four values: the minimum x-coordinate, the minimum y-coordinate, the maximum x-coordinate, and the maximum y-coordinate of the character string in an XY coordinate system with one end point (e.g., the upper left or lower left) of the document as the origin. The rectangle specified by these four values ​​becomes a bounding box indicating the area of ​​the character string (character area). Furthermore, in the OCR processing, information indicating the width and height of the document is acquired. The character string position information is normalized using the acquired width and height of the document so that the value falls within the range of 0 to 1. By using the normalized position information, a high average IoU value can be obtained for documents with similar layouts when viewed page by page, even if the documents are not the same size, thereby improving the accuracy of selecting documents to be used as reference documents.

[0115] In step S1702, the calculation unit 1605 calculates and obtains the overlap ratio (IoU) for the character string area (character area) between the target document acquired in step S1701 and previously processed documents. Furthermore, the calculation unit 1605 calculates and obtains the average overlap ratio (average IoU) based on the calculated overlap ratio (IoU) for the character string area (character area). The calculation unit 1605 obtains the overlap ratio (IoU) and average overlap ratio (average IoU) for the character string area (character area) based on the document information of the target document acquired in step S1701 and the information of previously processed documents stored in the storage unit 1603, in the same manner as in the second embodiment described above.

[0116] Here, in this embodiment, information about previously processed documents stored in the storage unit 1603 will be described. FIG. 18 is a diagram illustrating information about previously processed documents stored in the storage unit 1603. Information about previously processed documents is saved, for example, in a json format as shown in FIG. 18. For each character string included in a page, the character string and position information of the bounding box surrounding the character string (character string position information) are saved for each page. The bounding box position information is normalized so that coordinate values ​​range from 0 to 1 based on the height and width of the document (the vertical and horizontal dimensions of the image data). In addition, as an extraction result, item names and information about one or more items corresponding to the item names are saved. The information about the items includes the content of the item, the character string in which the content of the item existed (a pair of the character string and its position information), and the position information of the item. The item position information is normalized so that coordinate values ​​range from 0 to 1 based on the height and width of the document (the vertical and horizontal dimensions of the image data).

[0117] As an example, FIG. 19(B) shows an example of information on a previously processed document that was extracted from the document shown in FIG. 19(A) and stored in storage unit 1603. FIG. 19(B) shows information extracted and stored from the document information shown in FIG. 19(A) in which deliverables were selected as the extraction target item. The information listed in column 1901 for "Delivery" in the document shown in FIG. 19(A) is extracted and stored as information corresponding to the extraction target item "Delivery." As shown in FIG. 19(B), information corresponding to the extraction target item "Delivery" exists in the document, so the presence or absence of information is stored as "present." Furthermore, "Delivery" is stored as the content of the item name corresponding to the extraction target item "Delivery," and the character string "Delivery" and its location information, "Delivery: A, A, A, A," are stored as the character string that existed in the item name. "A', A', A', A'" is stored as the location information for the item name. Here, "A',A',A',A'" is the value of "A,A,A,A" normalized using the width and height of the document, i.e., "A',A',A',A'" = "A / (width),A / (height),A / (width),A / (height)". The contents of this item are stored as "DB design document", "Similar function examination result report", "Business function comparison table between old and new", and "Major table relationship diagram". The character strings that existed in the item contents are stored as "DB design document: B,B,B,B", "Similar function examination result report: C,C,C,C", "Business function comparison table between old and new: D,D,D,D", and "Major table relationship diagram: E,E,E,E", which indicate the character strings of each item and their location information. The location information of each item name is stored as the location information normalized based on the size of the document.

[0118] 17, in step S1703, the instruction generating unit 1607 determines whether or not any of the previously processed documents has an average overlap ratio (average IoU) equal to or greater than a threshold, based on the average overlap ratio (average IoU) for each previously processed document acquired in step S1702. If the instruction generating unit 1607 determines that any of the previously processed documents has an average overlap ratio (average IoU) equal to or greater than the threshold (YES), the process proceeds to step S1704. On the other hand, if the instruction generating unit 1607 determines that any of the previously processed documents does not have an average overlap ratio (average IoU) equal to or greater than the threshold (NO), the process proceeds to step S1708.

[0119] In step S1704, which is executed when the instruction generation unit 1607 determines that there is a document among the documents processed in the past whose average overlap ratio (average IoU) is equal to or greater than a threshold value, the selection unit 1606 selects the document with the highest average overlap ratio (average IoU) from among the documents processed in the past as the reference document.

[0120] In step S1705, the control unit 1601 determines, based on the reference document selected in step S1704, whether or not there is an item among the items to be extracted from the target document that may contain multiple pieces of information as item content. Whether or not an item may contain multiple pieces of information can be determined, for example, by setting in advance for each item whether or not there is a possibility that a single item may contain multiple pieces of information as item content, and determining whether or not there is an item that may contain multiple pieces of information based on that information. If the control unit 1601 determines that there is an item that may contain multiple pieces of information (YES), the process of step S1706 is executed. On the other hand, if the control unit 1601 determines that there is no item that may contain multiple pieces of information (NO), the process of step S1706 is not executed and the process of step S1707 is executed.

[0121] In step S1706, the instruction generation unit 1607 and the first information extraction unit 1608 use a large-scale language model (LLM) to extract from the target document item names and item contents corresponding to items to be extracted, which may include multiple pieces of information. Specifically, the instruction generation unit 1607 generates a prompt to be provided to the large-scale language model (LLM) for extracting information for items that may include multiple pieces of information from the target document. The instruction generation unit 1607 generates a prompt including instructions for extracting information corresponding to items to be extracted, which may include multiple pieces of information, from the target document, and character strings in the target document and their location information. The instruction generation unit 1607 generates a prompt that further includes information on the reference document selected in step S1704 (character strings in the document and their location information, and the extraction result) as an example for few-shot learning.

[0122] An example of a prompt generated by the instruction generation unit 1607 is shown in Fig. 20. As shown in Fig. 20, the prompt 2000 includes an explanation 2010 of a method for extracting information from a document, a group of item names and item descriptions to be extracted 2020, an example of character strings and their location information and answers from previously processed documents 2030, an example of an output format 2040, and a group of character strings and their location information from a target document. The explanation 2010 of the extraction method is an explanation of a method for extracting information on items to be extracted from a target document, and the group of item names and item descriptions to be extracted 2020 is the name of one or more items to be extracted and a description of each item. By including the item description in the prompt, it is possible to update the knowledge of items held by the large-scale language model (LLM) and to have the large-scale language model (LLM) learn knowledge of items not held by the LLM, thereby enabling accurate extraction of information corresponding to specified items. An example of a character string and its position information of a previously processed document and an answer 2030 is an example (sample) for few-shot learning, and includes information on a reference document selected from documents previously processed (document character string and its position information, and extraction result) 2031. An example of an output format 2040 is an example of an output format when information corresponding to an item to be extracted is output as an extraction result. A character string of a target document and its position information 2050 is a character string and its position information included in the target document acquired in step S1701, and includes a character string of a page of the target document and its position information 2051.

[0123] After the instruction generation unit 1607 generates a prompt to be given to the large-scale language model (LLM), the first information extraction unit 1608 performs inference regarding information extraction by inputting the generated prompt into the large-scale language model (LLM), and extracts information corresponding to the item to be extracted, which may include multiple pieces of information, from the target document.

[0124] In step S1707, second information extraction unit 1610 extracts information corresponding to the remaining items (items that may not include multiple pieces of information as item content) of the items to be extracted from the target document based on the location information in the reference document selected in step S1704 and the location information in the target document. For example, second information extraction unit 1610 extracts, as information (item content) corresponding to the item to be extracted, a character string whose center point of a rectangle (bounding box) calculated based on the location information acquired in step S1701 exists within a rectangle (bounding box) based on the location information of the item in the reference document. Here, if the location information of an item includes multiple pieces of information, information can be extracted using a rectangle that includes all of the character strings of the corresponding item content within it, i.e., a rectangle defined by the minimum x-coordinate, minimum y-coordinate, maximum x-coordinate, and maximum y-coordinate within the location information.

[0125] In step S1708, the control unit 1601 outputs the extraction results of the target document corresponding to the items to be extracted obtained in the processes of steps S1706 and S1707. Note that the control unit 1601 may also output the character strings and their position information in the target document obtained in step S1701. After executing the process of step S1708, the information processing device 100 ends the process shown in FIG.

[0126] In the above description, a large-scale language model (LLM) is used to extract item names and item contents corresponding to items to be extracted, which may contain multiple pieces of information. However, the large-scale language model (LLM) may be used to extract item names and item contents corresponding to all items to be extracted, not just items that may contain multiple pieces of information. Furthermore, regardless of whether there are items that may contain multiple pieces of information, a prompt containing information about the reference document selected in step S1704 as an example for few-shot learning may be provided to the large-scale language model (LLM), and the large-scale language model (LLM) may be used to extract item names and item contents corresponding to the items to be extracted.

[0127] In step S1709, which is executed when the instruction generation unit 1607 determines that there is no document with an average overlap ratio (average IoU) equal to or greater than a threshold among previously processed documents, the instruction generation unit 1607 and the first information extraction unit 1608 extract information corresponding to each item to be extracted from the target document using a large-scale language model (LLM). Specifically, the instruction generation unit 1607 generates a prompt to be provided to the large-scale language model (LLM) for extracting information corresponding to each item to be extracted from the target document, as shown in FIG. 20 . At this time, the instruction generation unit 1607 may generate the prompt by including, for example, an example document showing how to extract information from a document or an example of a previously processed document similar in format to the target document as an example for few-shot learning. The first information extraction unit 1608 performs inference regarding information extraction by inputting the prompt generated by the instruction generation unit 1607 into the large-scale language model (LLM) and extracts information corresponding to the item to be extracted from the target document. In this embodiment, the information corresponding to the items to be extracted is extracted as the item name and item content strings and their location information. That is, the content of the item name corresponding to the item to be extracted, the string where the item name existed (and its location information), the location information of the item name, the item content, the string where the item existed (and its location information), and the location information of the item are extracted from the target document as the information corresponding to the items to be extracted. In addition to this information, other information may also be extracted from the target document.

[0128] In step S1710, the control unit 1601 outputs the extraction results of the target document corresponding to the items to be extracted obtained in the process of step S1709. Note that the control unit 1601 may also output the character strings and their position information in the target document obtained in step S1701.

[0129] In step S1711, the control unit 1601 determines whether or not there has been any correction input to the extraction results of the target document output in step S1710. The control unit 1601 determines that there has been any correction input when, for example, a correction request is confirmed from a user who has confirmed the extraction results of the output target document. If the control unit 1601 determines that there has been any correction input to the extraction results (YES), the process of step S1712 is executed. On the other hand, if the control unit 1601 determines that there has been no correction input to the extraction results (NO), the process of step S1712 is not executed and the process of step S1713 is executed.

[0130] In step S1712, the correction unit 1609 corrects the extraction results of the target document in accordance with the input for correcting the extraction results. In step S1713, control unit 1601 associates the character string and its position information related to the target document with the extraction result and stores them in a database (storage unit 1603). For example, control unit 1601 assigns an identifier (ID) that can uniquely identify the target document, and associates the identifier (ID), the character string and its position information acquired in step S1701, and the extraction result obtained in step S1708 (or the corrected extraction result corrected in step S1712) with each other and stores them in the database. After executing the process of step S1713, information processing device 100 ends the process shown in FIG. 17.

[0131] According to the third embodiment, the information processing device 100 acquires, as a reference document, a document with a similar layout based on the average overlap ratio (average IoU) of the character string areas between the target document and the previously processed document from among the document information of previously processed documents. Then, using the information of the reference document, it extracts the item names and item contents corresponding to the items to be extracted. This enables accurate extraction of the item names and item contents corresponding to the items to be extracted from the target document. Furthermore, by appropriately inputting a prompt containing instructions for extracting information (character strings) corresponding to specified items and document information of the target document into a large-scale language model, it is possible to extract information (character strings) corresponding to desired items from the image data of the target document. Furthermore, by including (incorporating) information of reference documents acquired from previously processed documents into the prompt as examples for few-shot learning and inputting it into the large-scale language model, it is possible to easily and efficiently learn information equivalent to tacit knowledge, business knowledge, etc., without having to transcribe all of the information into sentences. This enables accurate extraction of information (character strings) corresponding to desired items from the image data of the target document.

[0132] In the above description of the third embodiment, the determination of whether a character string exists at the same position in a target document and a previously processed document is based on whether the center point of a rectangle calculated based on the position information of the character string included in the target document is located within a rectangle based on the position information of the previously processed document. However, this is not limited to this, and other methods may be used to determine whether a character string exists at the same position in a target document and a previously processed document. For example, if the distance between the center point of a rectangle calculated based on the position information of the previously processed document and the center point of a rectangle calculated based on the position information of the character string included in the target document is less than a predetermined threshold, it may be determined that a character string exists at the same position. Alternatively, the character string with the shortest distance between the center point of a rectangle calculated based on the position information of the previously processed document and the center point of a rectangle calculated based on the position information of the character string included in the target document may be determined to exist at the same position. Alternatively, the character string with the closest cosine similarity among a predetermined number of character strings from the shortest distance may be determined to exist at the same position. Alternatively, an image of a character string near the character string may be input into a large-scale language model to determine whether a character string exists at the same position.

[0133] In the above description of the third embodiment, various information related to the items to be extracted is extracted by inputting the character string extracted from the image and the information on the document processed in the past. However, this is not limiting, and information related to the items to be extracted may be extracted by inputting the acquired image (hereinafter referred to as the "current image") and the image of the document processed in the past (hereinafter referred to as the "past image") together with the character string extracted from the image and the information on the document processed in the past. Furthermore, in order to distinguish between the current image and the past image, each image may be input with a character string indicating the respective image. This makes it possible to accurately extract various information related to the items to be extracted using a large-scale language model (LLM).

[0134] In the above description of the third embodiment, various information related to the items to be extracted is extracted using a large-scale language model (LLM). This makes it possible to accurately extract the item names and item contents even when the target document contains multiple character strings corresponding to the item names and the item contents. Furthermore, in the above description, the prompts input to the large-scale language model (LLM) include information including the item descriptions and extraction results of previously processed documents. This makes it possible to learn based on information from previously processed documents and accurately extract the items of the target document as the items to be extracted, even when the item names in the target document differ from those in the target document.

[0135] In the above description of the third embodiment, when a character string for an item to be extracted cannot contain multiple pieces of information as its item content (contains only a single piece of information as its item content), the extraction target is extracted using location information in the target document and location information in previously processed documents, without using a large-scale language model (LLM). This improves processing speed and reduces the amount of processing (calculation amount) compared to when a large-scale language model (LLM) is used.

[0136] In the second and third embodiments described above, the calculation of the overlap ratio (IoU) with the bounding box in the target document is performed for all bounding boxes in previously processed documents, but it may also be performed only for those bounding boxes in previously processed documents that have overlapping portions with the bounding box in the target document. (Condition 1) The minimum x-coordinate of the bounding box in previously processed documents is less than or equal to the maximum x-coordinate of the bounding box in the target document. (Condition 2) The maximum x-coordinate of the bounding box in previously processed documents is greater than or equal to the minimum x-coordinate of the bounding box in the target document. (Condition 3) The minimum y-coordinate of the bounding box in previously processed documents is less than or equal to the maximum y-coordinate of the bounding box in the target document. (Condition 2) The maximum y-coordinate of the bounding box in previously processed documents is greater than or equal to the minimum y-coordinate of the bounding box in the target document. Of all the bounding boxes in previously processed documents, those that satisfy all of these conditions (Condition 1) to (Condition 4) overlap with the bounding box in the target document. This can be achieved simply by comparing the coordinate values ​​of the position information in the target document and previously processed documents, so by narrowing down the bounding boxes in previously processed documents for which the overlap ratio (IoU) is to be calculated, the calculation time can be reduced without reducing accuracy.

[0137] Furthermore, since documents with similar layouts have roughly the same number of pages, the documents to be subjected to the calculation may be narrowed down based on the difference in page count between the target document and previously processed documents. For example, previously processed documents whose page count difference with respect to the target document is less than a predetermined threshold may be subjected to the calculation. Furthermore, meta-information may be assigned to each document, and the documents to be subjected to the calculation may be narrowed down based on that meta-information. For example, for company documents, the company name or the like may be assigned to the document as meta-information, and previously processed documents that have the same company name assigned as meta-information may be subjected to the calculation. Narrowing down the documents to be subjected to the calculation in this manner reduces processing time.

[0138] <Other embodiments> The present invention can also be realized by executing the following process. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various recording media. The computer (or CPU, MPU, etc.) of the system or device then reads and executes the program. Furthermore, a computer-readable recording medium on which the program is recorded and a computer program product such as the program can also be applied as an embodiment of the present invention. Examples of recording media that can be used include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, magnetic tapes, non-volatile memory cards, and ROMs.

[0139] It should be noted that the above-described embodiments are merely examples of specific embodiments of the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be embodied in various forms without departing from its technical concept or main features.

[0140] The following devices, methods, etc. are also included in the disclosure of this embodiment. (1) an acquisition means for acquiring, as document information, a character string included in image data of a document and position information indicating the position of the character string; a conversion means for converting the acquired document information into a distributed representation; an information extraction means for inputting a prompt including the acquired document information into a large-scale language model, performing inference using the large-scale language model, and extracting a character string corresponding to an item indicated by the prompt; a storage unit that stores the document information, the distributed representations, and the extraction results of the information extraction unit in association with each other in a storage unit; a selection unit that selects a reference document from the previously processed documents based on a distributed representation of the document to be processed and a distributed representation of the previously processed documents stored in the storage unit; 10. An information processing apparatus according to claim 9, wherein the prompt input for the document to be processed includes information about the selected reference document. (2) The information processing device according to (1), characterized in that the prompt input for the document to be processed includes the document information and the extraction result relating to the selected reference document. (3) The information processing device described in (1) or (2) is characterized in that the selection means selects a predetermined number of documents from the previously processed documents in order of the degree of similarity between the distributed representation of the document to be processed and the distributed representation of the previously processed documents, and sets them as the reference documents. (4) The information processing device described in (1) or (2) is characterized in that the selection means selects a predetermined number of documents from the previously processed documents in order of the closest distance between the distributed representation of the document to be processed and the distributed representation of the previously processed documents, and sets them as the reference documents. (5) a correction means for receiving a correction request for the extraction result by the information extraction means and correcting a character string corresponding to the item in the extraction result in response to the correction request; The information processing device described in any one of (1) to (4), characterized in that when correction is made by the correction means, the document information regarding the document, the distributed representation, and the corrected extraction result are stored in association with each other in the memory unit. (6) The information processing device described in (5) is characterized in that the selection means selects the reference document from previously processed documents in which the document information, the distributed representation, and the modified extraction result related to the document are stored in the memory unit. (7) The information processing device described in any one of (1) to (6), characterized in that the selection means selects the reference document by comparing the distributed representation of the document to be processed with the distributed representation of the document processed in the past using a k-nearest neighbor method. (8) An information processing device described in any one of (1) to (7), characterized in that the prompt input for the document to be processed is generated by incorporating the document information stored in the memory unit regarding the reference document and the extraction result. (9) The information processing device according to any one of (1) to (8), wherein the prompt includes an explanation of the item. (10) The storage means stores the document information about the document and the extraction result by the information extraction means, including the position information about the extracted character string, in association with each other in the storage unit; The information processing device described in (1) or (2) is characterized in that the selection means selects the reference document from the previously processed documents based on, instead of the distributed representation, the area of ​​the character string in the document to be processed indicated by the location information regarding the document to be processed, and the area of ​​the character string in the previously processed document indicated by the location information regarding the previously processed document stored in the memory unit. (11) A calculation means for calculating an overlap ratio between the character string area in the document to be processed and the character string area in the previously processed document is provided, The information processing device according to (10), wherein the selection means selects the reference document from the previously processed documents based on the overlap ratio calculated by the calculation means. (12) The calculation means calculates an overlap ratio between each of the character string areas in the document to be processed and the character string areas in the previously processed documents, and calculates an average overlap ratio, which is an average value of the maximum overlap ratios for each of the calculated character string areas, for each of the previously processed documents; The information processing device described in (11) is characterized in that the selection means selects the reference document from the previously processed documents based on the average overlap ratio calculated by the calculation means for each previously processed document. (13) The information processing device according to (12), wherein the selection means selects, as the reference document, the document with the highest calculated average overlap ratio from among the previously processed documents. (14) The information processing device described in (12), characterized in that the selection means selects a predetermined number of documents as the reference documents from the documents processed in the past in order of the highest calculated average overlap ratio. (15) The information processing device according to any one of (11) to (14), characterized in that the calculation means calculates the overlap ratio of the character string area only for corresponding pages in the document to be processed and the document processed in the past. (16) The information extraction means If the item to be extracted is an item that may contain multiple character strings, extracting character strings corresponding to the item to be extracted from the document to be processed using the large-scale language model; The information processing device according to any one of (10) to (15), characterized in that if the item to be extracted is an item containing only one string, a string corresponding to the item to be extracted is extracted from the document to be processed based on position information of the string in the selected reference document and position information of the string in the document to be processed. (17) An information processing method executed by an information processing device, an acquisition step of acquiring, as document information, a character string included in image data of the document and position information indicating a position of the character string; a conversion step of converting the acquired document information into a distributed representation; an information extraction step of inputting the obtained prompt including the document information into a large-scale language model, performing inference using the large-scale language model, and extracting character strings corresponding to items indicated by the prompt; a storage step of storing the document information related to the document, the distributed representation, and the extraction result in the information extraction step in a storage unit in association with each other; a selection step of selecting a reference document from the previously processed documents based on a distributed representation of the document to be processed and a distributed representation of the previously processed documents stored in the storage unit, 10. An information processing method, wherein the prompt input for the document to be processed includes information about the selected reference document. (18) In the storing step, the document information about the document and the extraction result in the information extracting step, including position information about the extracted character string, are stored in the storing unit in association with each other; The information processing method described in (17), characterized in that in the selection process, instead of the distributed representation, the reference document is selected from the previously processed documents based on the area of ​​the character string in the document to be processed indicated by the location information regarding the document to be processed and the area of ​​the character string in the previously processed document indicated by the location information regarding the previously processed document stored in the memory unit. (19) In the computer of the information processing device, an acquiring step of acquiring, as document information, a character string included in image data of the document and position information indicating a position of the character string; a conversion step of converting the acquired document information into a distributed representation; an information extraction step of inputting the obtained prompt including the document information into a large-scale language model, performing inference using the large-scale language model, and extracting character strings corresponding to items indicated by the prompt; a storing step of storing the document information related to the document, the distributed representation, and the extraction result in the information extracting step in a storage unit in association with each other; a selection step of selecting a reference document from the previously processed documents based on a distributed representation of the document to be processed and a distributed representation of the previously processed documents stored in the storage unit; The prompts entered for the document to be processed are A program (computer program product) characterized by containing information. (20) In the storing step, the document information about the document and the extraction result in the information extracting step, including position information about the extracted character string, are stored in the storage unit in association with each other; The program (computer program product) described in (19) is characterized in that in the selection step, instead of the distributed representation, the reference document is selected from the previously processed documents based on the area of ​​the character string in the document to be processed indicated by the location information regarding the document to be processed and the area of ​​the character string in the previously processed document indicated by the location information regarding the previously processed document stored in the memory unit. (21) In the computer of the information processing device, an acquiring step of acquiring, as document information, a character string included in image data of the document and position information indicating a position of the character string; a conversion step of converting the acquired document information into a distributed representation; an information extraction step of inputting the obtained prompt including the document information into a large-scale language model, performing inference using the large-scale language model, and extracting character strings corresponding to items indicated by the prompt; a storing step of storing the document information related to the document, the distributed representation, and the extraction result in the information extracting step in a storage unit in association with each other; a selection step of selecting a reference document from the previously processed documents based on a distributed representation of the document to be processed and a distributed representation of the previously processed documents stored in the storage unit; 10. A computer-readable recording medium having a program recorded thereon, wherein the prompt input for the document to be processed includes information about the selected reference document. (22) In the storing step, the document information about the document and the extraction result in the information extracting step, including position information about the extracted character string, are stored in the storage unit in association with each other; The computer-readable recording medium described in (21) is characterized in that in the selection step, instead of the distributed representation, the reference document is selected from the previously processed documents based on the area of ​​the character string in the document to be processed indicated by the location information regarding the document to be processed and the area of ​​the character string in the previously processed document indicated by the location information regarding the previously processed document stored in the memory unit. [Explanation of symbols]

[0141] 100 Information processing device 101 CPU 102 ROM 103 RAM 104 Auxiliary storage 105 Output Device 106 Input Device 107 Network I / F 201, 1001, 1601 Control unit 202, 1002, 1602 Input / output control unit 203, 1003, 1603 Storage section 204, 1004, 1604 Acquisition Department 205 Conversion Unit 206, 1006 Selection section 207, 1007, 1607 Instruction generation section 208, 1008 Information extraction section 209, 1009, 1609 correction section 1005, 1605 Arithmetic unit 1608 First Information Extraction Unit 1610 Second Information Extraction Unit

Claims

1. an acquisition means for acquiring, as document information, a character string included in image data of the document and position information indicating the position of the character string; a conversion means for converting the acquired document information into a distributed representation; an information extraction means for inputting a prompt including the acquired document information into a large-scale language model, performing inference using the large-scale language model, and extracting a character string corresponding to an item indicated by the prompt; a storage unit that stores the document information, the distributed representations, and the extraction results of the information extraction unit in association with each other in a storage unit; a selection unit that selects a reference document from the previously processed documents based on a distributed representation of the document to be processed and a distributed representation of the previously processed documents stored in the storage unit; 10. An information processing apparatus according to claim 9, wherein the prompt input for the document to be processed includes information about the selected reference document.

2. 2. The information processing apparatus according to claim 1, wherein the prompt input for the document to be processed includes the document information and the extraction result relating to the selected reference document.

3. The information processing device according to claim 1, characterized in that the selection means selects a predetermined number of documents from the previously processed documents in order of the degree of similarity between the distributed representation of the document to be processed and the distributed representation of the previously processed documents, and sets them as the reference documents.

4. The information processing device according to claim 1, characterized in that the selection means selects a predetermined number of documents from the previously processed documents in order of proximity between the distributed representation of the document to be processed and the distributed representation of the previously processed documents, and sets them as the reference documents.

5. a correction means for receiving a correction request for the extraction result by the information extraction means and correcting a character string corresponding to the item in the extraction result in response to the correction request; The information processing device according to claim 1, characterized in that, when correction is made by the correction means, the document information regarding the document, the distributed representation, and the corrected extraction result are stored in the storage unit in association with each other.

6. The information processing device described in claim 5, characterized in that the selection means selects the reference document from previously processed documents in which the document information, the distributed representation, and the modified extraction result related to the document are stored in the memory unit.

7. 2. The information processing apparatus according to claim 1, wherein the selection means selects the reference document by comparing a distributed representation of the document to be processed with a distributed representation of the previously processed document using a k-nearest neighbor method.

8. 2. The information processing apparatus according to claim 1, wherein the prompt input for the document to be processed is generated by incorporating the document information stored in the storage unit relating to the reference document and the extraction result.

9. The information processing apparatus of claim 1 , wherein the prompt includes a description of the item.

10. the storage means stores the document information relating to the document and the extraction result by the information extraction means, including position information relating to the extracted character string, in the storage unit in association with each other; The information processing device described in claim 1, characterized in that the selection means selects the reference document from the previously processed documents based on, instead of the distributed representation, the area of ​​the character string in the document to be processed indicated by the location information regarding the document to be processed, and the area of ​​the character string in the previously processed document indicated by the location information regarding the previously processed document stored in the memory unit.

11. a calculation means for calculating an overlap ratio between the character string area in the document to be processed and the character string area in the previously processed document, 11. The information processing apparatus according to claim 10, wherein the selection means selects the reference document from the previously processed documents based on the overlap ratio calculated by the calculation means.

12. the calculation means calculates an overlap ratio between each of the character string areas in the document to be processed and the character string areas in the previously processed documents, and calculates an average overlap ratio for each of the previously processed documents, which is an average value of the maximum overlap ratios for each of the calculated character string areas; 12. The information processing apparatus according to claim 11, wherein the selection means selects the reference document from the previously processed documents based on the average overlap ratio calculated for each previously processed document by the calculation means.

13. 13. The information processing apparatus according to claim 12, wherein the selection means selects, as the reference document, a document having the highest calculated average overlap ratio from among the previously processed documents.

14. 13. The information processing apparatus according to claim 12, wherein the selection means selects a predetermined number of documents as the reference documents from the previously processed documents in descending order of the calculated average overlap ratio.

15. 12. The information processing apparatus according to claim 11, wherein the calculation means calculates the overlapping ratio of the character string area only on corresponding pages of the document to be processed and the previously processed document.

16. The information extraction means If the item to be extracted is an item that may contain multiple character strings, extracting character strings corresponding to the item to be extracted from the document to be processed using the large-scale language model; The information processing device described in claim 10, characterized in that if the item to be extracted is an item that contains only one string, a string corresponding to the item to be extracted is extracted from the document to be processed based on the position information of the string in the selected reference document and the position information of the string in the document to be processed.

17. An information processing method executed by an information processing device, an acquisition step of acquiring, as document information, a character string included in image data of the document and position information indicating a position of the character string; a conversion step of converting the acquired document information into a distributed representation; an information extraction step of inputting the obtained prompt including the document information into a large-scale language model, performing inference using the large-scale language model, and extracting character strings corresponding to items indicated by the prompt; a storage step of storing the document information related to the document, the distributed representation, and the extraction result in the information extraction step in a storage unit in association with each other; a selection step of selecting a reference document from the previously processed documents based on a distributed representation of the document to be processed and a distributed representation of the previously processed documents stored in the storage unit, 10. An information processing method, wherein the prompt input for the document to be processed includes information about the selected reference document.

18. In the storing step, the document information relating to the document and the extraction result in the information extracting step, including position information relating to the extracted character string, are stored in the storage unit in association with each other; The information processing method described in claim 17, characterized in that in the selection process, instead of the distributed representation, the reference document is selected from the previously processed documents based on the area of ​​the character string in the document to be processed indicated by the location information regarding the document to be processed and the area of ​​the character string in the previously processed document indicated by the location information regarding the previously processed document stored in the memory unit.

19. The computer of the information processing device an acquiring step of acquiring, as document information, a character string included in image data of the document and position information indicating a position of the character string; a conversion step of converting the acquired document information into a distributed representation; an information extraction step of inputting the obtained prompt including the document information into a large-scale language model, performing inference using the large-scale language model, and extracting character strings corresponding to items indicated by the prompt; a storing step of storing the document information related to the document, the distributed representation, and the extraction result in the information extracting step in a storage unit in association with each other; a selection step of selecting a reference document from the previously processed documents based on a distributed representation of the document to be processed and a distributed representation of the previously processed documents stored in the storage unit; The program, wherein the prompt input for the document to be processed includes information about the selected reference document.

20. In the storing step, the document information relating to the document and the extraction result in the information extracting step, including position information relating to the extracted character string, are stored in the storage unit in association with each other; The program described in claim 19, characterized in that in the selection step, instead of the distributed representation, the reference document is selected from the previously processed documents based on the area of ​​the character string in the document to be processed indicated by the location information regarding the document to be processed and the area of ​​the character string in the previously processed document indicated by the location information regarding the previously processed document stored in the memory unit.

Images