Information processing device, information processing method, computer program product, and recording medium

Abstract

The invention provides an information processing apparatus, an information processing method, a computer program product, and a recording medium. The information processing device includes: an acquisition unit that acquires document information included in image data of a document; a conversion means for converting the acquired document information into a dispersed representation; an information extraction unit that inputs a cue including the acquired document information into the large-scale language model, performs reasoning based on the large-scale language model, and extracts a character string corresponding to an item indicated by the cue; a storage unit that associates and stores in a storage unit document information relating to the document, the dispersion performance, and the extraction result by the information extraction unit; and a selection means for selecting a reference document from the documents that have been processed in the past on the basis of the distributed representation relating to the document that has been processed in the past and the distributed representation relating to the documents that have been processed in the past and stored in the storage unit, the presentation input to the document that has been processed including information relating to the selected reference document.

Description

Technical Field

[0001] This invention relates to information processing apparatus, information processing method, computer program product, and recording medium. Background Technology

[0002] Techniques are known for using Optical Character Recognition (OCR) processing to extract handwritten or printed strings from image data such as documents. Furthermore, there are techniques for performing OCR processing on image data such as documents to extract strings corresponding to specified items. Examples of methods for extracting strings corresponding to specified items include: extracting strings located in a predetermined direction relative to the item name in the document; or extracting strings that meet certain conditions (restrictions) contained in the image data as candidate strings, and selecting and extracting a string from the candidate strings based on the frequency of occurrence in the image data.

[0003] However, if the image data does not contain the item name related to the item from which the string is to be extracted, it may not be possible to extract the string corresponding to the item with high accuracy. Additionally, for example, if the item from which the string is to be extracted is "delivery location," but the item name in the image data is "destination," indicating a delivery location, the string corresponding to the item may not be extracted with high accuracy.

[0004] Patent Document 1: Japanese Patent Application Publication No. 2023-46684 Summary of the Invention

[0005] The purpose of this invention is to provide an information processing device, information processing method, computer program product, and recording medium capable of extracting strings corresponding to desired items from image data of a document with high precision.

[0006] The information processing apparatus of the present invention comprises: an acquisition unit that acquires a string contained in image data of a document and position information indicating the position of the string as document information; a conversion unit that converts the acquired document information into a discrete representation; an information extraction unit that inputs a prompt containing the acquired document information into a large-scale language model, performs inference using the large-scale language model, and extracts a string corresponding to the item indicated by the prompt; a storage unit that stores the document information related to the document, the discrete representation, and the extraction result of the information extraction unit in a storage unit; and a selection unit that selects a reference document from the previously processed documents based on the discrete representation related to the document to be processed and the discrete representation related to previously processed documents stored in the storage unit, wherein the prompt input to the document to be processed includes information of the selected reference document.

[0007] According to the present invention, an information processing apparatus, information processing method, computer program product, and recording medium capable of extracting strings corresponding to desired items from image data of a document with high precision can be provided. Attached Figure Description

[0008] Figure 1 This is a diagram illustrating an example of the hardware structure of an information processing device.

[0009] Figure 2 This is a diagram illustrating an example of the functional structure of the information processing device in the first embodiment.

[0010] Figure 3 This is a diagram illustrating a processing example of the information processing apparatus in the first embodiment.

[0011] Figure 4 This is a flowchart illustrating a processing example of the information processing apparatus in the first embodiment.

[0012] Figure 5 It is a diagram illustrating the documentation, document information, and examples of distributed presentation.

[0013] Figure 6 This is a diagram illustrating the corrections made to the extraction results.

[0014] Figure 7 This is a diagram illustrating an example of information extraction.

[0015] Figure 8A This is a diagram illustrating an example of information extraction.

[0016] Figure 8B This is a diagram illustrating an example of information extraction.

[0017] Figure 8CThis is a diagram illustrating an example of information extraction.

[0018] Figure 9 This is a diagram illustrating an example of the output of the processing results.

[0019] Figure 10 This is a diagram illustrating an example of the functional structure of the information processing device in the second embodiment.

[0020] Figure 11 This is a diagram illustrating a processing example of the information processing apparatus in the second embodiment.

[0021] Figure 12 This is a flowchart illustrating a processing example of the information processing apparatus in the second embodiment.

[0022] Figure 13 This is a flowchart illustrating an example of the process of obtaining the IoU associated with a character region.

[0023] Figure 14 This is a diagram illustrating the IoUs related to the character region.

[0024] Figure 15 It is a diagram that illustrates information related to documents that have been processed in the past.

[0025] Figure 16 This is a diagram illustrating an example of the functional structure of the information processing device in the third embodiment.

[0026] Figure 17 This is a flowchart illustrating a processing example of the information processing apparatus in the third embodiment.

[0027] Figure 18 It is a diagram that illustrates information related to documents that have been processed in the past.

[0028] Figure 19 This is a diagram illustrating examples of information related to documents that have been processed in the past.

[0029] Figure 20 This is a diagram illustrating examples related to information extraction. Detailed Implementation

[0030] The embodiments of the present invention will be described with reference to the accompanying drawings.

[0031] <First Implementation Method>

[0032] The information processing apparatus in the first embodiment described below uses Optical Character Recognition (OCR) processing and a Large Language Model (LLM) to extract information (strings) corresponding to the indicated item from the document's image data. In this embodiment, the information processing apparatus inputs information (item name or description) of the item from which the desired information (string) is extracted, along with a prompt containing the string in the document's image data and location information indicating the string's position within the document, into the Large Language Model. This prompt then performs reasoning using the Large Language Model to extract the information (string) corresponding to the desired item. By inputting the string contained in the document's image data and the prompt containing location information into the Large Language Model, the Large Language Model can grasp the document's structure and extract information (strings) that takes into account the document's layout, etc.

[0033] However, when extracting information (strings) from a document requires so-called tacit knowledge or business knowledge, it is considered impossible to properly extract information (strings) corresponding to the desired item from the document's image data. Therefore, in this embodiment, a document with a format similar to the document being processed (hereinafter also referred to as the "object document") is obtained from previously processed documents and used as a reference document (hereinafter also referred to as the "reference document") in the information extraction processing of the object document. Furthermore, by including the information from the reference document as an example for few-shot learning in the prompt and inputting it into a large-scale language model, it can easily and efficiently learn knowledge equivalent to tacit knowledge or business knowledge, and extract information (strings) corresponding to the desired item, thereby improving the accuracy of information extraction.

[0034] Figure 1 This diagram illustrates an example of the hardware structure of the information processing apparatus 100 in this embodiment. The information processing apparatus 100 includes a CPU 101, a ROM 102, a RAM 103, an auxiliary storage device 104, an output device 105, an input device 106, and a network I / F 107. The CPU 101, ROM 102, RAM 103, auxiliary storage device 104, output device 105, input device 106, and network I / F 107 are communicatively connected via a system bus 108.

[0035] The CPU (Central Processing Unit) 101 is the central arithmetic unit that controls various operations of the information processing device 100. For example, the CPU 101 can control the operation of the entire information processing device 100. The ROM (Read-Only Memory) 102 stores control programs, boot programs, etc., that can be executed by the CPU 101. The RAM (Random Access Memory) 103 is the main memory of the CPU 101 and is used as a working area or a temporary storage area for expanding various programs.

[0036] The auxiliary storage device 104 stores various data, programs, etc. The auxiliary storage device 104 is implemented by a storage device that can temporarily or continuously store various data, such as a non-volatile memory represented by HDD (Hard Disk Drive) or SSD (Solid State Drive).

[0037] Output device 105 is a device that outputs various information, used to provide various information prompts to the user. For example, output device 105 is implemented by a display device such as a monitor. Output device 105 can provide information to the user by displaying various display information. As another example, output device 105 can also be implemented by an audio output device that outputs voice, electronic sounds, etc. In this case, output device 105 can also provide information to the user by outputting voice, electronic sounds, etc. The device used as output device 105 can be appropriately changed depending on the medium used to provide information to the user.

[0038] Input device 106 is used to receive various instructions from the user. For example, input device 106 may also include input devices such as a mouse, keyboard, and touch panel. As another example, input device 106 may include a microphone sound collection device, which can collect the voice emitted by the user. In this case, the content represented by the voice can be identified as an instruction from the user by performing various analysis processes such as audio analysis and natural language processing on the collected voice. The device used as input device 106 can be appropriately changed according to the method of recognizing the instructions from the user. Various devices can also be used as input device 106.

[0039] The Network I / F107 is used for communication with external devices via a network. The device used as the Network I / F107 can also be appropriately changed depending on the type of communication path or the communication method used.

[0040] The CPU 101 expands the program stored in the ROM 102 or auxiliary storage device 104 into the RAM 103 and executes the program, thereby realizing the various functions and processes of the information processing device described later. The program of the information processing device 100 can be provided to the information processing device 100 via a recording medium such as a CD-ROM, or downloaded via a network. When the program of the information processing device 100 is provided via a recording medium, the program recorded on the recording medium is installed in the auxiliary storage device 104 by placing the recording medium on a predetermined drive device.

[0041] Figure 1 The structure shown is merely an example and does not necessarily limit the hardware structure of the information processing device 100 in this embodiment. As an example, it may be possible to omit certain structures such as the output device 105 and the input device 106. As another example, structures corresponding to the functions implemented by the information processing device 100 may be appropriately added.

[0042] Figure 2 This is a diagram illustrating an example of the functional structure of the information processing apparatus 100 in the first embodiment. The information processing apparatus 100 in the first embodiment includes a control unit 201, an input / output control unit 202, a storage unit 203, an acquisition unit 204, a conversion unit 205, a selection unit 206, an instruction generation unit 207, an information extraction unit 208, and a correction unit 209.

[0043] The control unit 201 is responsible for controlling each component of the information processing device 100. The input / output control unit 202 performs various processes related to prompting the user with various information and accepting input from the user (e.g., instructions). For example, the input / output control unit 202 may also perform processes related to prompting the user (UI) and accepting input via the UI. As a result, the information processing device 100 can recognize instructions from the user and display the result of the processing corresponding to the instruction to the user.

[0044] Storage unit 203 schematically illustrates a storage area for storing various data, programs, etc. For example, storage unit 203 may also store data and programs used for processing by the various components of the information processing apparatus 100. Storage unit 203 may store learned models that have undergone machine learning (deep learning), which are used for inference regarding information extraction performed in the information processing apparatus 100. Storage unit 203 may store document information related to documents, distributed representations obtained by transforming document information, and the results of document information extraction.

[0045] The acquisition unit 204 acquires image data of a document by optically scanning or photographing it. Furthermore, the acquisition unit 204 performs optical character recognition (OCR) processing on the acquired document image data and extracts and acquires document information. The document information includes strings contained in the document's image data and positional information indicating the location of those strings within the document. Alternatively, the acquisition unit 204 may acquire document image data obtained through pre-scanning or photographing externally, without scanning or photographing the document itself. The acquisition unit 204 may also acquire document information by accepting strings contained in the document and their positional information obtained as a result of external optical character recognition processing.

[0046] The conversion unit 205 converts the document-related information acquired by the acquisition unit 204 into a distributed representation (embedded representation). The distributed representation of document information is expressed as a multi-dimensional real-valued vector, with vectors representing the distributed representation of document information related to similar documents being the closest (smallest distance) vectors. The conversion unit 205 uses, for example, a learned embedding model that numerically vectorizes natural language to convert the document information into a distributed representation. The learned embedding model can be, for example, Sentence-BERT, OpenAI's text embedding models, etc. Not limited to these learned embedding models, a learned embedding model can be generated by performing machine learning to convert document information into a distributed representation (embedded representation), and then applied. For example, multiple groups of a certain article and similar articles can be prepared as learning data to learn how to make multi-dimensional vectors generated from similar articles similar vectors, thereby generating a learned embedding model.

[0047] In this embodiment, as an example, the conversion unit 205 converts the strings and position information (combinations of strings and position information) in the document-related information into a discrete representation for each page of the document (on a page-by-page basis). However, the conversion unit 205 is not limited to this; it can convert the entire document (on a document-by-document basis) into a discrete representation. The conversion unit 205 may convert only the strings in the document-related information into a discrete representation, or it may convert only the position information of the strings in the document-related information into a discrete representation. While a discrete representation is used in this embodiment, other methods different from discrete representation may be used as long as similarity can be evaluated.

[0048] The selection unit 206 selects reference documents (reference documents) from the previously processed documents stored in the storage unit 203, based on document-related information, dispersion representation, and information extraction results. The selection unit 206 uses machine learning (e.g., k-nearest neighbor, KNN) to select reference documents from the previously processed documents, based on the document-related dispersion representation. Specifically, the selection unit 206 compares the dispersion representation related to the object document obtained through the transformation in the transformation unit 205 with the dispersion representation related to the previously processed documents stored in the storage unit 203, and selects a predetermined number of documents with formats similar to the object document (documents whose dispersion representation is close to that of the object document) from the previously processed documents as reference documents.

[0049] Here, based on the similarity of the types and positional information of strings in the documents, documents with similar formats have similar dispersion patterns. The selection unit 206 can identify documents with similar formats using machine learning methods such as k-nearest neighbors. Furthermore, the selection unit 206 uses the dispersion patterns of documents to evaluate similarity, thus obtaining documents with high similarity even if the format deviates slightly or the content differs. The method for comparing the dispersion patterns related to the target document with those related to previously processed documents is not limited to k-nearest neighbors; any method that compares dispersion patterns (vectors) can be used. For example, Support Vector Machines (SVMs), Artificial Neural Networks (ANNs), cosine similarity, etc., can also be used for dispersion pattern comparison.

[0050] In this embodiment, the instruction generation unit 207 generates an instruction for extracting information (strings) from the object document and inputs it into the information extraction unit 208. The information extraction unit 208 uses a large-scale language model (LLM) to extract the information (strings) corresponding to the input instruction. A large-scale language model (LLM) is a language model constructed using a large amount of text data (large-scale corpora, etc.) and deep learning techniques. When text data called prompts representing instructions, etc., is input, it performs reasoning based on the prompts, generates and outputs text data corresponding to the input prompts.

[0051] The instruction generation unit 207 generates a prompt containing instructions (strings) for extracting information (strings) corresponding to the desired item from the object document and document information of the object document, and inputs it to the information extraction unit 208. In reasoning using a large-scale language model (LLM), there is a method called few-shot learning: the accuracy of the answer is improved by utilizing the characteristic of a high-precision large-scale language model that returns an output adapted to the prompted example (sample) by including the example (sample) in the prompt. Therefore, in this embodiment, if there is a document in the reference document selected by the selection unit 206 that is highly similar to the object document (e.g., a document whose distance from the dispersion representation of the object document is below a predetermined threshold), the instruction generation unit 207 generates a prompt that uses the document information related to the selected reference document and the extraction result as an example for few-shot learning, and inputs it to the information extraction unit 208. In this way, by providing information (document information and extraction results) from reference documents that are highly similar to the object document as examples for few-shot learning to a large-scale language model (LLM), it is possible to learn knowledge equivalent to tacit knowledge, business knowledge, etc. in a simple and efficient manner, and to extract information (strings) corresponding to the desired item with high accuracy.

[0052] Information extraction unit 208 performs large-scale language model (LLM)-based reasoning based on prompts generated and input by instruction generation unit 207, and extracts information (strings) corresponding to the items indicated by the prompts from the image data of the object document. Correction unit 209 receives a correction request from the user regarding the results of information extraction by information extraction unit 208 on the object document, and corrects the extraction results according to the correction request. The extraction results of information about the object document extracted by information extraction unit 208 (the corrected extraction results in the case of correction by correction unit 209) are stored in storage unit 203 in association with the document information obtained by acquisition unit 204 and the distributed representation obtained through conversion in conversion unit 205.

[0053] Reference Figure 3 The processing in the information processing apparatus 100 of the first embodiment will be described. Figure 3 This is a diagram illustrating a processing example of the information processing apparatus 100 in the first embodiment.

[0054] The information processing device 100 performs document information acquisition processing 302, acquiring document information (strings and their position information contained in the image data) from the image data of the document (object document) 301, which is the object of processing. The information processing device 100 performs document information conversion processing 303 on the document information related to the acquired object document, and uses a learned embedding model or the like to convert the document information (the combination of strings and position information) into a discrete representation.

[0055] Next, the information processing device 100 performs a reference document acquisition process 304, selecting documents (reference documents) that were previously processed and used as references in the information extraction process from the object document from the database (DB) 312. The information processing device 100 compares the dispersion representation related to the object document obtained in the conversion process 303 with the dispersion representation related to previously processed documents stored in the database 312, and acquires a predetermined number of documents from the previously processed documents in descending order of similarity to the object document's dispersion representation (from closest to furthest distance) as reference documents. Furthermore, in this embodiment, the information processing device 100 selects reference documents from the documents stored in the database 312 from the previously processed documents stored in the database 312 through the storage process 307 described later. This is because the extraction result stored through the storage process 307 described later is an extraction result that has undergone corrections based on user instructions as needed, and is generally considered to be of higher quality than the extraction result stored through the storage process 311.

[0056] After obtaining the reference document through acquisition process 304, the information processing device 100 determines whether there is a document in the obtained reference document that has a similar format to the object document. For example, the information processing device 100 determines that there is a document with a similar format to the object document if there is a reference document whose distance from the dispersion representation of the object document is less than a predetermined threshold; otherwise, it determines that there is no document with a similar format to the object document.

[0057] When it is determined that no document with a similar format to the object document exists in the obtained reference documents, the information processing device 100 performs an extraction process 305 to extract information from the object document. In this extraction process 305, the information processing device 100 inputs a prompt containing an instruction (string) for extracting information corresponding to the specified item and document information of the object document into a large-scale language model (LLM), performs reasoning related to information extraction, and extracts the information (string) corresponding to the indicated item from the object document. In this case, since no document with a similar format to the object document exists in the documents processed in the past, the information processing device 100, for example, without using the information from the reference document obtained in the acquisition process 304, generates a prompt that represents a document example of a method for extracting information from a document as an example for few-shot learning, and inputs it into the large-scale language model (LLM). Additionally, although the similarity to the object document is low, the similarity to the object document is high when compared with other documents processed in the past. Therefore, information from the reference document obtained in processing 304 can be generated as a hint for few-shot learning and input into the large-scale language model (LLM).

[0058] After extraction processing 305, the information processing device 100 performs extraction result correction processing 306, correcting the extraction results in extraction processing 305 as needed based on correction requests from users, etc. The information processing device 100 displays the extraction results from extraction processing 305 to the user and accepts correction requests from the user regarding the displayed extraction results. For example, because the extracted item name is different from the item name indicated in the prompt, the extraction result in extraction processing 305 shows "no information," but the information (string) corresponding to that item is contained in the image data (document information) of the object document. In this case, if the user who confirmed the displayed extraction result requests correction to change the "no information" extraction result to the information contained in the image data (document information) of the object document, the extraction result is corrected to the information specified by the user according to the correction request.

[0059] After the correction process 306, the information processing apparatus 100 performs a storage process 307 for document information / distributed representation / extraction results related to the object document, and stores the document information, distributed representation, and extraction results related to the object document in the database 312. For example, the information processing apparatus 100 assigns a unique identifier (ID) to the object document, and associates the identifier (ID) with the document information obtained in the acquisition process 302, the distributed representation obtained in the conversion process 303, and the corrected extraction results after the correction process 306, and stores them in the database 312.

[0060] If it is determined that a document with a similar format to the object document exists in the obtained reference document, the information processing device 100 performs a prompt generation process 308 that includes information from the reference document, and generates a prompt for the Large Scale Language Model (LLM) in the extraction process 310 described later. Referring to the item name and item description 309, the information processing device 100 generates a prompt containing instructions for extracting information (strings) corresponding to the specified item from the object document, as well as document information of the object document. Furthermore, the information processing device 100 generates a prompt that includes information (document information and extraction results) from the reference document obtained in the acquisition process 304 as an example for few-shot learning. For example, the information processing device 100 directly incorporates the document information and extraction results of the reference document stored in the database 312 as an example for few-shot learning to generate a prompt.

[0061] After generation process 308, information processing device 100 performs extraction process 310 to extract information from the object document. In this extraction process 310, information processing device 100 performs reasoning related to information extraction by inputting the prompts generated in generation process 308 into a large-scale language model (LLM) to extract information (strings) corresponding to the indicated items.

[0062] After extraction processing 310, the information processing apparatus 100 performs storage processing 311 for document information / distributed representation / extraction results related to the object document, storing the document information, distributed representation, and extraction results related to the object document in a database 312. For example, the information processing apparatus 100 assigns a unique identifier (ID) to the object document and associates this identifier (ID) with the document information obtained in acquisition processing 302, the distributed representation obtained in conversion processing 303, and the extraction results in extraction processing 310, storing them in the database 312. Here, since a reference document is obtained from the document stored in storage processing 307 in acquisition processing 304, it is preferable to set up separate databases (or storage areas) to separately store the document information / distributed representation / extraction results related to the object document stored in storage processing 307 and the document information / distributed representation / extraction results related to the object document stored in storage processing 311.

[0063] Figure 4 This is a flowchart illustrating a processing example of the information processing apparatus 100 in the first embodiment.

[0064] In step S401, the acquisition unit 204 performs optical character recognition on the image data of the document (object document) 301, which is the object of processing, to acquire document information of the object document (strings and their position information included in the image data). For example, the acquisition unit 204 scans or captures images such as... Figure 5 Image data is obtained from the object document exemplified by (A), and optical character recognition processing is performed on the obtained image data, such as... Figure 5 As shown in (B), the document information (string and its position information) of the object document is obtained. Figure 5 In (B), as an example, the document information of the object document is represented in the format "string: position information". The position information of the string represents four values ​​in the XY coordinate system with an endpoint (e.g., top left or bottom left) as the origin: the minimum x-coordinate, the minimum y-coordinate, the maximum x-coordinate, and the maximum y-coordinate. (Including...) Figure 5 In the case of a document with the table shown in (A), the acquisition unit 204 identifies and excludes the table borders in the document's image data and acquires the string. Alternatively, the string can be acquired by dividing the region based on the identified table borders.

[0065] In step S402, the conversion unit 205 uses a learned embedding model or the like to convert the document information (a combination of string and location information) of the object document obtained in step S401 into a distributed representation. For example, the conversion unit 205 converts the document information (a combination of string and location information) of the object document obtained in step S401 into a distributed representation. Figure 5 The document information of the object document shown in (B) is input into the learned embedding model and converted into... Figure 5 The dispersed representation shown in (C) is a multidimensional real vector.

[0066] In step S403, the selection unit 206 compares the dispersion performance of the object document obtained in step S402 with the dispersion performance of previously processed documents using machine learning (e.g., k-nearest neighbor method), and selects documents (reference documents) from the previously processed documents that will be used as references in the information extraction process from the object document. For example, the selection unit 206 compares the dispersion performance of the object document with the dispersion performance of previously processed documents stored in the storage unit 203 after performing correction processing on the extraction results, and selects a predetermined number of documents from the previously processed documents as reference documents in descending order of similarity to the dispersion performance of the object document (from closest to furthest distance).

[0067] In step S404, the instruction generation unit 207 determines whether there is a document among the reference documents selected in step S403 that has a high similarity in format to the object document. The instruction generation unit 207 determines whether there is a reference document with high format similarity based on the distance between the dispersion representation of the object document and the dispersion representation of the reference document. For example, if the distance between the dispersion representation of the object document and the dispersion representation of the reference document is below a predetermined threshold, the instruction generation unit 207 determines that the format similarity is high; otherwise, it determines that the format similarity is not high. If the instruction generation unit 207 determines that there is a document among the selected reference documents that has a high similarity in format to the object document (yes), then the processing of step S409 is performed. If the instruction generation unit 207 determines that there is no document among the selected reference documents that has a high similarity in format to the object document (no), then the processing of step S405 is performed.

[0068] In step S405, when the instruction generation unit 207 determines that there is no document in the selected reference document that is highly similar in format to the object document, the information extraction unit 208 extracts information from the object document using a large-scale language model (LLM). Specifically, the information extraction unit 208 inputs a prompt containing an instruction (string) generated by the instruction generation unit 207 for extracting information corresponding to the specified item and document information of the object document into the large-scale language model (LLM), performs reasoning related to information extraction, and extracts the information (string) corresponding to the indicated item from the object document. In this case, for example, the instruction generation unit 207 generates a prompt and inputs the prompt into the large-scale language model (LLM), which uses an example of a document representing a method for extracting information from a document as an example for few-shot learning.

[0069] In step S406, the control unit 201 outputs the extraction results of information related to the object document obtained in step S405. Alternatively, the control unit 201 may also output, in addition to the information extraction results, document information of the object document obtained in step S401.

[0070] In step S407, the correction unit 209 corrects the extraction results based on correction requests from users. For example, when a user who has confirmed the extraction results related to the object document output in step S406 submits a correction request for the extraction results, the correction unit 209 accepts the correction request and corrects the extraction results accordingly. As an example, regarding... Figure 5 The object document shown in (A) is based on Figure 5 The document information of the object document shown in (B) is obtained in step S406 as follows. Figure 6 The extraction results are shown in (A). Figure 6 In the example shown in (A), regarding the project "Case Name", the presence or absence of information is "None", and the content of the information is "Unknown". Figure 5 In the object document shown in (A), the information recorded in the "Business Content" section is assumed to correspond to the "Case Name". In this case, if a user requests a correction to change the information corresponding to the "Case Name" in the "Business Content" section of the object document (e.g., "A set of basic plans for factory production system refresh"), the correction unit 209 will, based on the information extracted from the object document, retrieve the "Case Name" from the "Business Content" section. Figure 6 The extraction results are corrected by adjusting the presence or absence of information and the content of the information, as shown in (B). By presenting this corrected extraction result as an example for few-shot learning, large-scale language model learning can be improved. Figure 5In the document shown in (A), the item "Business Content" is equivalent to the item "Case Name". By modifying only the item information in this way, tagging related to the extraction results can be completed with minimal effort, making it easier than usual tagging. Furthermore, by providing the modified extraction results as examples for few-shot learning to a large-scale language model, even if they are not input as text into the prompts every time processing is performed, as long as the modified extraction results are included, information such as tacit knowledge and business knowledge can be learned simply and efficiently.

[0071] In step S408, the control unit 201 stores document information, dispersion representation, and extraction results related to the object document in a database (storage unit 203) in a mutually associated manner. For example, the control unit 201 assigns a unique identifier (ID) to the object document and stores this identifier (ID) in the database in association with the document information obtained in step S401, the dispersion representation obtained in step S402, and the corrected extraction results obtained in step S407. After executing step S408, the information processing device 100 terminates. Figure 4 The processing is shown.

[0072] In step S409, when the instruction generation unit 207 determines that the selected reference document has a format highly similar to the object document, the instruction generation unit 207 generates a prompt for the Large Scale Language Model (LLM). The instruction generation unit 207 generates a prompt that includes instructions for extracting information (strings) corresponding to the specified item from the object document, document information of the object document, and information (document information and extraction results) of the reference document selected in step S403.

[0073] Figure 7 An example of the prompt generated in step S409 is shown. Figure 7As shown, prompt 700 includes a description 710 of the information (string) extraction method, a group of descriptions of the items to be extracted 720, an example of document information and response 730, an example of output format 740, and document information 750 of the object document. The description 710 describes a method for extracting information from the document information of the object document. The group of descriptions of the items to be extracted 720 describes the item names and descriptions of one or more items to be extracted. Here, the item description is, for example, general knowledge (description) that can be commonly used in the documents to be processed. By including item descriptions in the prompt, the item-specific knowledge possessed by the Large Scale Language Model (LLM) can be updated to common knowledge, and item-specific knowledge not possessed by the Large Scale Language Model (LLM) can be learned. By updating to common knowledge and learning common knowledge, strings corresponding to the specified items can be extracted with high precision. The example of document information and response 730 is an example (sample) for few-shot learning, containing document information of the reference document selected in step S403 and extraction results 731, 732. Example 740 of the output format is an example of the output format when the information (string) corresponding to the item to be extracted is output as the extraction result. The document information 750 of the object document is the document information of the object document obtained in step S401, which includes the document information 751 of each page of the object document.

[0074] Figure 7 The specific examples shown are as follows: Figures 8A to 8C As shown. Figures 8A to 8C The parts shown, when combined, constitute a prompt. Figures 8A to 8C In this context, element 810 corresponds to the description of the extraction method 710, and element 820 corresponds to the item name and description group of the item to be extracted 720. Element 830 corresponds to document information and an example of the answer 730, containing information from three reference documents 831, 832, and 833. Element 840 corresponds to an example of the output format 740, and element 850 corresponds to the document information 750 of the object document.

[0075] return Figure 4 In step S410, the information extraction unit 208 extracts information from the object document using a large-scale language model (LLM). Specifically, the information extraction unit 208 inputs the instruction (string) generated in step S409, which contains information (string) corresponding to the desired item, document information of the object document, document information of the reference document, and a prompt of the extraction result, into the large-scale language model (LLM), performs reasoning related to information extraction, and extracts the information (string) corresponding to the indicated item from the object document.

[0076] In step S411, the control unit 201 outputs the extraction results of information related to the object document obtained in the processing of step S410. In addition to the information extraction results, the control unit 201 may also output the document information of the object document obtained in step S401, the similarity (distance) of the dispersion representation between the reference document selected in step S403 and the object document, etc.

[0077] In step S412, the control unit 201 stores document information, dispersion representation, and extraction results related to the object document in a database (storage unit 203) in a mutually associated manner. For example, the control unit 201 assigns a unique identifier (ID) to the object document and stores this identifier (ID) in the database in association with the document information obtained in step S401, the dispersion representation obtained in step S402, and the extraction results obtained in step S410, respectively. After performing the processing in step S412, the information processing apparatus 100 terminates. Figure 4 The processing is shown.

[0078] Figure 9 This is a diagram illustrating an example of the output of the processing result related to the information processing apparatus 100 extracting information from an object document in the first embodiment. Figure 9 The processing results are displayed, for example, via the input / output control unit 202 and output device 105, and prompts are given to the user. Figure 9 In the processing result 900 shown, the distance between the dispersion representation of the reference documents selected from previously processed documents and the dispersion representation of the object document is output as at least sample distances 910. In this example, the distances between the dispersion representations of the reference documents and the object document are represented for three reference documents. In the extraction result 920, the extraction results related to the object document are output. As extraction results, for example, the project name of the item to be extracted, the presence or absence of information corresponding to the item, and the content are output. In the processed document information 930, the document information (string and location information) of the object document obtained based on the image data of the object document is output. Figure 9 The output example shown is just one example and is not limited to this; other information related to information extraction and processing can also be displayed.

[0079] According to the first embodiment, the information processing apparatus 100 inputs a prompt containing instructions for extracting information (strings) corresponding to a specified item and document information of the object document into a large-scale language model, thereby enabling the extraction of information (strings) corresponding to the desired item from the image data of the object document. Furthermore, by obtaining a document with a similar format to the object document from previously processed documents as a reference document, and incorporating information (document information and extraction results) from the previously obtained reference document as an example for few-shot learning into the prompt, this information is input into the large-scale language model. This allows for the simple and efficient learning of information equivalent to tacit knowledge or business knowledge, even without writing everything as an article, and enables the high-precision extraction of information (strings) corresponding to the desired item from the image data of the object document.

[0080] Additionally, as positional information for strings, the coordinate values ​​of the string in an XY coordinate system with one endpoint of the document as the origin (minimum and maximum x-coordinates, minimum and maximum y-coordinates) are used. However, to ensure the values ​​are within the range of 0 to 1, the coordinate values ​​are standardized using the document's width and height, and can then be used as the string's positional information. By using standardized positional information, even if documents are of different sizes, documents with similar layouts when viewed as a whole on the page are more likely to be selected as reference documents, thus improving the accuracy of reference document selection.

[0081] <Second Implementation Method>

[0082] The second embodiment will be described.

[0083] In the first embodiment, a reference document is obtained from previously processed documents based on the document-related dispersion representation. However, in the second embodiment described below, instead of dispersion representation, a reference document is obtained from previously processed documents based on the Intersection over Union (IoU) of string regions between documents. If the documents have the same layout, the strings are positioned substantially the same within the documents. If the IoU of string regions is high, the documents can be considered to have similar formats.

[0084] The hardware structure of the information processing device 100 in the second embodiment is similar to... Figure 1 The hardware structure of the information processing device in the first embodiment shown is the same, so the description is omitted.

[0085] Figure 10This is a diagram illustrating an example of the functional structure of the information processing apparatus 100 in the second embodiment. The information processing apparatus 100 in the second embodiment includes a control unit 1001, an input / output control unit 1002, a storage unit 1003, an acquisition unit 1004, an arithmetic unit 1005, a selection unit 1006, an instruction generation unit 1007, an information extraction unit 1008, and a correction unit 1009.

[0086] The control unit 1001 is responsible for controlling each component of the information processing device 100. The input / output control unit 1002 performs various processes related to prompting the user with various information and receiving information input (e.g., instructions) from the user. For example, the input / output control unit 1002 can perform processes related to prompts from the UI and processes related to receiving input via the UI. As a result, the information processing device 100 can recognize instructions from the user and can prompt the user with the result of the processing corresponding to the instruction.

[0087] Storage unit 1003 schematically represents a storage area used to store various data, programs, etc. For example, storage unit 1003 may also store data and programs used for processing by the various components of information processing device 100. Storage unit 1003 may store learned models that have undergone machine learning (deep learning) for reasoning related to information extraction, and may also store document information related to documents and the extraction results of document information.

[0088] The acquisition unit 1004 acquires image data of a document by optically scanning or photographing it. Furthermore, the acquisition unit 1004 performs optical character recognition (OCR) processing on the acquired document image data and extracts and acquires document information related to the document. The document information includes strings contained in the document's image data and positional information indicating the position of those strings within the document. Alternatively, the acquisition unit 1004 may acquire document information by accepting and processing image data of a document pre-scanned or photographed externally, without scanning or photographing the document itself. The acquisition unit 1004 may also acquire document information by accepting strings contained in the document and their positional information obtained as a result of external optical character recognition processing.

[0089] The calculation unit 1005 calculates the overlap ratio (IoU) of string regions (character regions) between the document being processed (object document) and previously processed documents. When character regions in the object document that are the objects of calculation are designated as character region A, and character regions in previously processed documents that are the objects of calculation are designated as character region B, the overlap ratio (IoU) of the character regions is calculated by {(area of ​​overlapping regions in character region A and character region B) / (area of ​​the two regions A and B combined)}, with a value ranging from 0 to 1. In this embodiment, the calculation unit 1005 sets the overlap ratio (IoU) of the bounding box (rectangle) surrounding the string, determined by the string position information in the document information, as the overlap ratio (IoU) of the string region (character region). Furthermore, the calculation unit 1005 calculates the average overlap ratio (average IoU) on a page-by-page basis. Alternatively, the calculation unit 1005 can also calculate the average overlap ratio (average IoU) per document unit. A detailed explanation of the average overlap ratio (average IoU) will be provided later.

[0090] The selection unit 1006 selects reference documents (reference documents) from previously processed documents stored in the storage unit 1003, based on the overlap ratio (IoU) of string regions (character regions). Specifically, the selection unit 1006 selects a predetermined number of reference documents from previously processed documents whose average overlap ratio (average IoU) is above a predetermined threshold, based on the average overlap ratio (average IoU) obtained by the calculation unit 1005.

[0091] In this embodiment, the instruction generation unit 1007 generates an instruction for extracting information (string) from the object document and inputs it into the information extraction unit 1008. The information extraction unit 1008 uses a large-scale language model (LLM) to extract the information (string) corresponding to the input instruction.

[0092] The instruction generation unit 1007 generates a prompt containing instructions for extracting information (strings) corresponding to the desired item from the object document and document information of the object document, and inputs it to the information extraction unit 1008. Additionally, if a reference document exists that is selected by the selection unit 1006 as having high similarity to the object document, the instruction generation unit 1007 generates a prompt that uses document information related to the reference document and the extraction results as examples for few-shot learning, and inputs it to the information extraction unit 1008. In this way, by providing information (document information and extraction results) from a reference document that is highly similar to the object document as examples for few-shot learning to a large-scale language model (LLM), knowledge such as tacit knowledge and business knowledge can be learned simply and efficiently, and information (strings) corresponding to the desired item can be extracted with high accuracy.

[0093] The information extraction unit 1008, based on the prompts generated and input by the instruction generation unit 1007, performs reasoning using a large-scale language model (LLM) and extracts information (strings) corresponding to the items extracted via the prompts from the image data of the object document. The correction unit 1009 receives correction requests from users regarding the extraction results of the information related to the object document by the information extraction unit 1008 and corrects the extraction results according to the correction requests. The extraction results of the information related to the object document extracted by the information extraction unit 1008 (the corrected extraction results when the correction unit 1009 performs corrections) are stored in the storage unit 1003 in association with the document information obtained by the acquisition unit 1004.

[0094] Reference Figure 11 The processing in the information processing apparatus 100 of the second embodiment will be described. Figure 11 This is a diagram illustrating a processing example of the information processing apparatus 100 in the second embodiment.

[0095] The information processing device 100 performs document information acquisition processing 1102, and obtains document information (strings and their position information contained in the image data) from the image data of the processing object document (object document) 1101.

[0096] Next, the information processing device 100 performs an IoU acquisition process 1103 related to the character region, calculating and obtaining the overlap ratio (IoU) of the string regions (character regions) between the processing target document (object document) and previously processed documents stored in the database (DB) 1111. Furthermore, based on the calculated overlap ratio (IoU) of the string regions (character regions), the information processing device 100 calculates and obtains the average overlap ratio (average IoU).

[0097] After obtaining the average overlap ratio (average IoU) in the processing 1103, the information processing device 100 determines whether a document with an average overlap ratio (average IoU) of 1 or higher than a threshold exists in a previously processed document.

[0098] When it is determined that no document has an average overlap ratio (average IoU) greater than or equal to a threshold, the information processing device 100 performs an extraction process 1104 to extract information from the object document. In this extraction process 1104, the information processing device 100 inputs a prompt containing an instruction (string) for extracting information corresponding to a specified item and document information of the object document into a large-scale language model (LLM), performs inference related to information extraction, and extracts the information (string) corresponding to the indicated item from the object document. In this case, since there are no documents with a layout similar to the object document with an average overlap ratio (average IoU) greater than or equal to a threshold among the documents processed in the past, the information processing device 100, for example, generates a prompt that represents a document example of how information is extracted from the document as an example for few-shot learning and inputs it into the large-scale language model (LLM).

[0099] After extraction processing 1104, the information processing device 100 performs extraction result correction processing 1105, correcting the extraction results in extraction processing 1104 as needed, based on correction requests from users, etc. Figure 3 Similarly, in the correction process 306 of the first embodiment shown, the information processing device 100 prompts the user with the extraction results in the extraction process 1104, accepts correction requests from the user regarding the prompted extraction results, and corrects the extraction results to the information specified by the user according to the correction requests from the user.

[0100] After the correction process 1105, the information processing device 100 performs a storage process 1106 for document information / extraction results related to the object document, storing the document information related to the object document and the extraction results in the database 1111. For example, the information processing device 100 assigns a unique identifier (ID) to the object document, and associates the identifier (ID) with the document information obtained in the acquisition process 1102 and the extraction results after the correction process 1105, storing it in the database 1111.

[0101] When a document is determined to have an average overlap ratio (average IoU) exceeding a threshold, the information processing device 100 retrieves a predetermined number of documents from previously processed documents in descending order of average overlap ratio (average IoU) as reference documents. It then performs a prompt generation process 1107, which generates a prompt containing information from the reference documents, and generates a prompt that will be provided to the large-scale language model (LLM) in the extraction process 1109 described later. The information processing device 100 generates a prompt containing instructions for extracting information (strings) corresponding to the specified item from the object document, along with document information of the object document, based on the item name and item description 1108. Furthermore, the information processing device 100 generates the prompt by including information from the retrieved reference documents (document information and extraction results) as examples for few-shot learning. For example, the information processing device 100 generates the prompt by directly incorporating the document information and extraction results of the reference documents stored in the database 1111 as examples for few-shot learning. Additionally, the document retrieved as a reference document from previously processed documents can be a single document or multiple documents.

[0102] After the generation process 1107, the information processing device 100 performs an extraction process 1109 to extract information from the object document. In this extraction process 1109, the information processing device 100 performs information extraction-related reasoning by inputting the prompts generated by the generation process 1107 into a large-scale language model (LLM) to extract information (strings) corresponding to the indicated items.

[0103] After the extraction process 1109, the information processing device 100 performs a storage process 1110 for document information / extraction results related to the object document, storing the document information related to the object document and the extraction results in the database 1111. For example, the information processing device 100 assigns a unique identifier (ID) to the object document and stores the identifier (ID) in the database 1111 in association with the document information obtained in the acquisition process 1102 and the extraction results in the extraction process 1109.

[0104] Figure 12 This is a flowchart illustrating a processing example of the information processing apparatus 100 in the second embodiment.

[0105] In step S1201, the acquisition unit 1004 scans or photographs the target document (object document) to acquire image data, performs optical character recognition (OCR) processing on the acquired image data, and acquires document information of the target document (strings and their position information contained in the image data). The string position information is obtained by acquiring, for example, four values ​​in an XY coordinate system with an endpoint (e.g., the top left or bottom left point) as the origin: the minimum x-coordinate, the minimum y-coordinate, the maximum x-coordinate, and the maximum y-coordinate. The rectangle defined by these four values ​​becomes the bounding box representing the region (character region) of the string. Additionally, in the OCR processing, information representing the size of the document, i.e., its width (horizontal length) and height (vertical length), is acquired (equivalent to the maximum x-coordinate and maximum y-coordinate that can be taken in the document). The string position information is standardized using the acquired document width and height, with values ​​ranging from 0 to 1. By using standardized location information, even if the documents are of different sizes, a high average IoU can be obtained among documents with similar layouts when viewed on a page-by-page basis, thereby improving the accuracy of selecting documents as reference documents.

[0106] In step S1202, the calculation unit 1005 calculates and obtains the overlap ratio (IoU) of the string regions (character regions) between the object document obtained in step S1201 and the previously processed documents. Furthermore, based on the calculated overlap ratio (IoU) of the string regions (character regions), the calculation unit 1005 calculates and obtains the average overlap ratio (average IoU).

[0107] Reference Figure 13 This section explains how to obtain the overlap ratio (IoU) of a string region (character region). Figure 13 This is a flowchart illustrating an example of how the overlap ratio (IoU) of a string region (character region) is obtained. Figure 13 This illustrates an example of the overlap ratio (IoU) of the regions (character regions) where strings are retrieved in the corresponding pages between the object document and previously processed documents. (Regarding...) Figure 13 In the following description, "object document" refers to the processing object page in the object document, and "past document" refers to the page in a previously processed document that corresponds to the processing object page in the object document.

[0108] In step S1301, the arithmetic unit 1005 is based on... Figure 12In step S1201, the document information of the object document is obtained, including the string contained in the object document and the position information of the bounding box surrounding the string. As described above, since the bounding box representing the character region is a rectangle determined by the value of the string's position information, the position information of the bounding box and the position information of the string are the same information (hereinafter, the same applies). That is, in the processing of step S1301, the arithmetic unit 1005 obtains the document information of the object document from the object document. Figure 12 In step S1201, the document information (string and position information of the string) of the object document obtained is used to obtain the document information (string and position information of the string) of the object page to be processed.

[0109] In step S1302, the arithmetic unit 1005 obtains the string contained in each past document and the position information of the bounding box surrounding the string based on the information of the previously processed documents stored in the storage unit 1003. Figure 15 This diagram illustrates the information stored in storage unit 1003 regarding previously processed documents (past documents). The information about previously processed documents includes, for example,... Figure 15 The data is saved in the JSON format shown. For each string contained in a page, the position information of the string and the bounding box surrounding it is saved for each page. The bounding box position information is normalized to coordinate values ​​of 0 to 1 based on the document's height and width (the dimensions of the image data). Additionally, as part of the extraction results, the project name and the content of one or more projects corresponding to that project name are also saved.

[0110] The execution order of steps S1301 and S1302 is different. Alternatively, the process shown in step S1301 can be performed after the process shown in step S1302.

[0111] In step S1303, the arithmetic unit 1005 selects an unprocessed bounding box from the bounding boxes in the object document.

[0112] Next, in step S1304, the calculation unit 1005 calculates and obtains the overlap ratio (IoU) between the bounding box selected in step S1303 and each bounding box in the previous document. For example, assuming... Figure 14Page X1400 is the processing object page in the object document, page A1410 is the page in document A that was previously processed that corresponds to the processing object page in the object document, and page B1420 is the page in document B that was previously processed that corresponds to the processing object page in the object document. Assume that bounding box 1401 in page X1400 was selected in step S1303 above. In this case, in step S1304, the calculation unit 1005 obtains the overlap ratio (IoU) between bounding boxes 1401 and 1411 based on the position information of bounding box 1401 and the position information of bounding box 1411 in page A1410. Similarly, the calculation unit 1005 obtains the overlap ratio (IoU) between bounding box 1401 and other bounding boxes in page A1410, namely the overlap ratio (IoU) between bounding box 1401 and bounding box 1412, and the overlap ratio (IoU) between bounding box 1401 and bounding box 1413. Furthermore, the calculation unit 1005 obtains the overlap ratio (IoU) between bounding box 1401 and bounding box 1421 based on the position information of bounding box 1401 and the position information of bounding box 1421 in page B1420. Similarly, the calculation unit 1005 obtains the overlap ratio (IoU) between bounding box 1401 and other bounding boxes in page B1420, namely the overlap ratio (IoU) between bounding box 1401 and bounding box 1422, and the overlap ratio (IoU) between bounding box 1401 and bounding box 1423. The same processing is performed on other past documents. That is, the processing of step S1304 described above is performed on each past document.

[0113] In step S1305, the calculation unit 1005 selects the largest overlap ratio (IoU) for each past document as the overlap ratio (IoU) associated with the bounding box selected in step S1303. For example, in Figure 14 In the example shown, the largest overlap ratio (IoU) among the bounding boxes 1401 in page X1400 and 1411, 1412, and 1413 in page A1410 is selected as the overlap ratio (IoU) related to the bounding box 1401 in the past document A. Similarly, the largest overlap ratio (IoU) among the bounding boxes 1401 in page X1400 and 1421, 1422, and 1423 in page B1420 is selected as the overlap ratio (IoU) related to the bounding box 1401 in the past document B. The same process is performed on other past documents. That is, the process of step S1305 described above is performed on each past document. Furthermore, if the largest overlap ratio (IoU) is 0, meaning there is no bounding box overlapping the bounding box selected in step S1303 within the page in a past document, the overlap ratio (IoU) related to that selected bounding box is set to 0.

[0114] Next, in step S1306, the calculation unit 1005 determines whether there are any unprocessed bounding boxes in the object document. If the calculation unit 1005 determines that there are unprocessed bounding boxes in the object document (yes), it returns to step S1303 and executes the processing after step S1303 again. If the calculation unit 1005 determines that there are no unprocessed bounding boxes in the object document, that is, the processing after step S1303 has been performed on all bounding boxes (no), then the processing in step S1307 is executed.

[0115] In step S1307, the calculation unit 1005 calculates the average overlap ratio (IoU) for each past document based on the overlap ratio (IoU) selected for the bounding boxes in the object document in step S1305, and obtains the average overlap ratio (average IoU). For example, in Figure 14 In the example shown, if the overlap ratios (IoU) related to the bounding boxes 1401, 1402, and 1403 in the past document A are a, b, and c, respectively, then (a+b+c) / 3 is set as the average overlap ratio (average IoU) of the past document A (specifically, the page in the past document A corresponding to the processing object page in the object document). The same processing is performed on other past documents. That is, the processing described in step S1307 is performed on each of the past documents.

[0116] As described above, the calculation unit 1005 obtains the average overlap ratio (average IoU) of pages corresponding to processing target pages in the target document in past documents, calculates the average of the average overlap ratios (average IoU) of each page in past documents, and obtains the average overlap ratio (average IoU) of the entire past documents. Furthermore, when obtaining the average overlap ratio (average IoU) of the entire past documents, if there is no page in the past documents corresponding to a processing target page in the target document, that page is excluded from the average calculation. This processing is performed on each document that has been processed in the past. Thus, the calculation unit 1005 obtains the average overlap ratio (average IoU) of each document that has been processed in the past.

[0117] In the above explanation, the average overlap ratio (average IoU) of the entire document is obtained for each document that has been processed in the past, but the calculation is performed on a page-by-page basis instead of on a document-by-document basis. Figure 12In the process following step S1203 in the flowchart shown, the average overlap ratio (average IoU) can also be obtained for each previously processed document, on a page-by-page basis. The average overlap ratio (average IoU) can be obtained by comparing the processed object page in the object document with the corresponding pages in past documents, or by comparing the processed object page with each page in past documents. When comparing with the corresponding pages in past documents, compared to comparing with each page in past documents, a reduction in computation time and an improvement in accuracy are expected. Therefore, whether to compare with the corresponding pages in past documents or with each page in past documents can be appropriately selected based on factors such as processing time, workload, and accuracy requirements.

[0118] Return to Figure 12 In step S1203, the generation unit 1007 is instructed to determine, based on the average overlap ratio (average IoU) of each previously processed document obtained in step S1202, whether a document with an average overlap ratio (average IoU) of 1007 or higher exists among the previously processed documents. If the generation unit 1007 determines that a document with an average overlap ratio (average IoU) of 1007 or higher exists among the previously processed documents (yes), the processing in step S1208 is executed. If the generation unit 1007 determines that no document with an average overlap ratio (average IoU) of 1007 or higher exists among the previously processed documents (no), the processing in step S1204 is executed.

[0119] In step S1204, which is executed when the instruction generation unit 1007 determines that there are no documents in the previously processed documents with an average overlap ratio (average IoU) of a threshold or higher, the information extraction unit 1008 extracts information from the object document using a large-scale language model (LLM). Specifically, the information extraction unit 1008 inputs a prompt containing instructions generated by the instruction generation unit 1007 for extracting information (strings) corresponding to a specified item and document information of the object document into the large-scale language model (LLM), thereby performing reasoning related to information extraction and extracting information (strings) corresponding to the indicated item from the object document. In this case, for example, the instruction generation unit 1007 generates a prompt that uses an example of a document representing a method for extracting information from a document as an example for few-shot learning and inputs it into the large-scale language model (LLM).

[0120] In step S1205, the control unit 1001 outputs the extraction results of information related to the object document obtained in step S1204. In addition to the information extraction results, the control unit 1001 may also output document information of the object document obtained in step S1201.

[0121] In step S1206, with Figure 4 Similarly, in step S407 of the first embodiment shown, the correction unit 1009 corrects the extraction results based on correction requests from users. For example, when a user who has confirmed the extraction results related to the object document output in step S1205 makes a correction request for the extraction results, the correction unit 1009 accepts the correction request and corrects the extraction results according to the correction request.

[0122] In step S1207, the control unit 1001 stores the document information related to the object document and the extraction results in a database (storage unit 1003). For example, the control unit 1001 assigns a unique identifier (ID) to the object document and stores this identifier (ID) in the database in association with the document information obtained in step S1201 and the corrected extraction results modified in step S1206. After executing step S1207, the information processing device 100 terminates the process. Figure 12 The processing is shown.

[0123] In step S1208, which is executed when the instruction generation unit 1007 determines that there are documents among the previously processed documents with an average overlap ratio (average IoU) of a threshold or higher, the selection unit 1006 selects the document with the highest average overlap ratio (average IoU) from the previously processed documents as a reference document. In this example, the document with the highest average overlap ratio (average IoU) is selected as the reference document, but multiple documents may also be selected in descending order of average overlap ratio (average IoU).

[0124] In step S1209, the instruction generation unit 1007 generates a prompt for the Large Scale Language Model (LLM). The instruction generation unit 1007 generates a prompt containing instructions for extracting information (strings) corresponding to specified items from the object document, document information of the object document, and information (document information and extraction results) of the reference document selected in step S1208. The prompt generated in step S1209 is consistent with... Figure 4 The prompt generated in step S409 of the first embodiment shown is the same, so the description is omitted.

[0125] In step S1210, the information extraction unit 1008 extracts information from the object document using a large-scale language model (LLM). Specifically, the information extraction unit 1008 inputs the instruction (string) generated in step S1209, which contains information corresponding to the desired item, the document information of the object document, the document information of the reference document, and a prompt of the extraction result, into the large-scale language model (LLM), performs reasoning related to information extraction, and extracts the information (string) corresponding to the indicated item from the object document.

[0126] In step S1211, the control unit 1001 outputs the extraction results of information related to the object document obtained in the processing of step S1210. In addition to outputting the information extraction results, the control unit 1001 may also output document information of the object document, the average IoU of previously processed documents selected as reference documents, etc.

[0127] In step S1212, the control unit 1001 stores the document information related to the object document and the extraction results in a database (storage unit 1003). For example, the control unit 1001 assigns a unique identifier (ID) to the object document and stores this identifier (ID) in the database in association with the document information obtained in step S1201 and the extraction results obtained in step S1210. After executing step S1212, the information processing device 100 terminates. Figure 12 The processing is shown.

[0128] According to the second embodiment, the information processing apparatus 100 can extract information (strings) corresponding to a desired item from the image data of the object document by inputting a prompt including an instruction for extracting information (strings) corresponding to a specified item and document information of the object document into a large-scale language model. Furthermore, based on the average overlap ratio (average IoU) related to the regions of strings in the object document and the previously processed documents, a document with a similar layout is obtained as a reference document. Information from the previously obtained reference document (document information and extraction results) is included (incorporated) in the prompt as an example for few-shot learning and input into the large-scale language model. Thus, even if not all of it is written as an article, information equivalent to tacit knowledge, business knowledge, etc., can be learned simply and efficiently, and information (strings) corresponding to the desired item can be extracted from the image data of the object document with high accuracy.

[0129] <Third Implementation Method>

[0130] The third embodiment will be described.

[0131] The hardware structure of the information processing device 100 in the third embodiment and Figure 1 The hardware structure of the information processing device in the first embodiment shown is the same, so the description is omitted.

[0132] Figure 16 This is a diagram illustrating an example of the functional structure of the information processing apparatus 100 in the third embodiment. The information processing apparatus 100 includes a control unit 1601, an input / output control unit 1602, a storage unit 1603, an acquisition unit 1604, an arithmetic unit 1605, a selection unit 1606, an instruction generation unit 1607, a first information extraction unit 1608, a correction unit 1609, and a second information extraction unit 1610.

[0133] The control unit 1601 is responsible for controlling each component of the information processing device 100. The input / output control unit 1602 performs various processes related to prompting the user with various information and accepting input from the user (e.g., instructions). For example, the input / output control unit 1602 may also perform processes related to prompting the user via the UI or processes related to accepting input via the UI. Thus, the information processing device 100 can recognize instructions from the user and display the results of the processing corresponding to those instructions to the user.

[0134] Storage unit 1603 schematically represents a storage area used to store various data, programs, etc. For example, storage unit 1603 may also store data and programs used for processing by the various components of information processing device 100. Storage unit 1603 may store learned models that have undergone machine learning (deep learning) for reasoning related to information extraction, and may also store various document-related information such as document information and information extraction results.

[0135] In the third embodiment, document information refers to the information that is extracted from the document based on the item to be extracted (extraction object item), and associates the item name corresponding to the extraction object item, the content of the item corresponding to the item name (item content), and the location information related to the item name and item content with the extraction object item, respectively. The location information is information indicating the position of the string showing the item name or item content in the document.

[0136] The acquisition unit 1604 acquires image data of a document by optically scanning or photographing it. Furthermore, the acquisition unit 1604 performs optical character recognition (OCR) processing on the acquired document image data to obtain the string contained in the document image data and the positional information associated with that string. Alternatively, the acquisition unit 1604 may accept and acquire image data of a document pre-scanned or photographed externally as input, without scanning or photographing the document itself. The acquisition unit 1604 may also accept and acquire the string contained in the document and its positional information obtained as a result of external optical character recognition processing.

[0137] The calculation unit 1605 calculates the overlap ratio (IoU) of the string regions (character regions) between the processing target document (object document) and previously processed documents. In this embodiment, the calculation unit 1605 sets the overlap ratio (IoU) of the bounding box (rectangle) surrounding the string, determined by the string position information in the document information, as the overlap ratio (IoU) of the string region (character region). Furthermore, the calculation unit 1605 calculates the average overlap ratio (average IoU) on a page-by-page basis. Alternatively, the calculation unit 1605 can also calculate the average overlap ratio (average IoU) per document unit.

[0138] The selection unit 1606 selects a reference document (reference document) from previously processed documents stored in the storage unit 1603, based on the overlap ratio (IoU) of regions (character regions) of the string. Specifically, the selection unit 1606 selects the document with the highest average overlap ratio (average IoU) that is above a predetermined threshold, based on the average overlap ratio (average IoU) obtained through calculations in the calculation unit 1605.

[0139] In the information processing apparatus 100 of this embodiment, the instruction generation unit 1607 and the first information extraction unit 1608 can use a large-scale language model (LLM) to extract information related to the indicated extraction target item from the image data of the document. For example, the instruction generation unit 1607 generates instructions for extracting various information related to the extraction target item from the object document and inputs them into the first information extraction unit 1608, which uses the large-scale language model (LLM) to extract information corresponding to the input instructions.

[0140] The instruction generation unit 1607 generates instructions containing various information related to the extracted object items from the object document, as well as prompts containing strings and their location information contained in the image data of the object document, and inputs them to the first information extraction unit 1608. Here, in this embodiment, the various information related to the extracted object items extracted from the object document includes the item name (string) corresponding to the extracted object item, the item content (string) corresponding to the item name, and the location information (coordinate information) of the item name and item content. Alternatively, the instruction generation unit 1607 may also generate prompts that use document information of documents highly similar to the object document (e.g., documents with an average IoU of more than a threshold) as examples for few-shot learning, and input them to the first information extraction unit 1608. In this way, by providing information of documents highly similar to the object document as examples for few-shot learning to the large-scale language model (LLM), the extraction accuracy of the item name and item content corresponding to the extracted object items from the object document can be improved.

[0141] The first information extraction unit 1608 performs reasoning using a large-scale language model (LLM) based on the prompt generated and input by the instruction generation unit 1607, and extracts various information about the extracted object items indicated by the prompt from the image data of the object document. The correction unit 1609 receives a correction request from the user regarding the extraction results of the information about the object document extracted by the first information extraction unit 1608, and corrects the extraction results according to the correction request.

[0142] In the information processing apparatus 100 of this embodiment, the second information extraction unit 1610 can use document information from previously processed documents to extract information related to the indicated extraction target item from the document's image data based on the item name and item content location information. The second information extraction unit 1610 extracts various information related to the indicated extraction target item from the image data of the object document based on the item name or item content location information in the document information of previously processed documents. Specifically, the second information extraction unit 1610 refers to a reference document selected by the selection unit 1606 from previously processed documents and extracts various information related to the extraction target item from the object document based on the item name or item content location information in the reference document.

[0143] Figure 17 This is a flowchart illustrating a processing example of the information processing apparatus 100 in the third embodiment.

[0144] In step S1701, the acquisition unit 1604 scans or photographs the target document (object document) to acquire image data, performs optical character recognition (OCR) processing on the acquired image data, and acquires the string and its position information contained in the image data of the target document. The string position information includes, for example, four values: the minimum x-coordinate, the minimum y-coordinate, the maximum x-coordinate, and the maximum y-coordinate of the string in an XY coordinate system with one endpoint (e.g., the top-left or bottom-left point) as the origin. The rectangle defined by these four values ​​becomes the bounding box representing the region (character region) of the string. Furthermore, in the OCR processing, information representing the width and height of the document is acquired. The string position information is standardized using the acquired document width and height, with values ​​ranging from 0 to 1. By using the standardized position information, even if the document sizes are different, a high average IoU can be obtained among documents with similar layouts when viewed on a page-by-page basis, thus improving the accuracy of selecting a document as a reference document.

[0145] In step S1702, the calculation unit 1605 calculates and obtains the overlap ratio (IoU) related to the string regions (character regions) between the object document obtained in step S1701 and previously processed documents. Furthermore, based on the calculated overlap ratio (IoU) of the string regions (character regions), the calculation unit 1605 calculates and obtains the average overlap ratio (average IoU). The calculation unit 1605, based on the document information of the object document obtained in step S1701 and the information of previously processed documents stored in the storage unit 1603, similarly obtains the overlap ratio (IoU) and average overlap ratio (average IoU) related to the string regions (character regions) in the second embodiment described above.

[0146] In this embodiment, information about previously processed documents stored in the storage unit 1603 will be described. Figure 18 This diagram illustrates the information of previously processed documents stored in storage unit 1603. The information of previously processed documents includes, for example,... Figure 18The data is saved in JSON format as shown. For each string contained in a page, the position information of the string and the bounding box surrounding it is saved for each page. The bounding box position information is normalized to coordinate values ​​of 0 to 1 based on the document's height and width (the dimensions of the image data). Additionally, as an extraction result, information related to the project name and one or more projects corresponding to that project name is saved. The project-related information includes the project content, the string containing the project content (a group of strings and their position information), and the project's position information. The project's position information is normalized to coordinate values ​​of 0 to 1 based on the document's height and width (the dimensions of the image data).

[0147] As an example, in Figure 19 (B) shows from Figure 19 The example shown in (A) illustrates the extraction and storage of previously processed document information in storage unit 1603. Figure 19 (B) shows Figure 19 The information in document (A) shows the information about extracting and storing deliverables as extraction object items. Figure 19 In the document shown in (A), the information recorded in column 1901, "Delivery of Goods," is extracted and stored as the information corresponding to the "Delivery of Goods" item. For example... Figure 19 As shown in (B), the information corresponding to the "deliverables" of the extraction object exists in the document, so the presence or absence of the information is stored as "present". Additionally, as the content of the item name corresponding to the extraction object, "delivery", "delivery" is stored. As the string representing the item name, "delivery: A, A, A, A" and its position information are stored. "A', A', A', A'" is stored as the position information of the item name. Here, "A', A', A', A'" is the value obtained by normalizing "A, A, A, A" using the document's width and height, i.e., "A', A', A', A'" = "A / (width), A / (height), A / (width), A / (height)". Furthermore, as the content of this item, "DB Design Document", "Similar Function Research Results Report", "Business Function New and Old Comparison Table", and "Main Table Relationship Diagram" are stored. The strings representing the content of the project are "DB Design Book: B, B, B, B", "Similar Function Research Results Report: C, C, C, C", "Business Function New and Old Comparison Table: D, D, D, D", and "Main Table Relationship Diagram: E, E, E, E". The position information of the project name is stored separately and is standardized based on the document size.

[0148] Return to Figure 17 In step S1703, the generation unit 1607 determines, based on the average overlap ratio (average IoU) of each previously processed document obtained in step S1702, whether a document with an average overlap ratio (average IoU) of 1 or higher exists among the previously processed documents. If the generation unit 1607 determines that a document with an average overlap ratio (average IoU) of 1 or higher exists among the previously processed documents (yes), the processing in step S1704 is executed. If the generation unit 1607 determines that no document with an average overlap ratio (average IoU) of 1 or higher exists among the previously processed documents (no), the processing in step S1709 is executed.

[0149] In step S1704, which is executed when the instruction generation unit 1607 determines that there is a document in the previously processed documents with an average overlap ratio (average IoU) of a threshold or higher, the selection unit 1606 selects the document with the highest average overlap ratio (average IoU) from the previously processed documents as a reference document.

[0150] In step S1705, the control unit 1601, based on the reference document selected in step S1704, determines whether, within the extraction object items for extracting information from the object document, there exists an item that can contain multiple pieces of information as item content. Regarding whether an item can contain multiple pieces of information, for example, it can be pre-defined whether each item might contain multiple pieces of information as item content, and the existence of such an item can be determined based on this information. If the control unit 1601 determines that an item that can contain multiple pieces of information exists (Yes), the processing in step S1706 is executed. If the control unit 1601 determines that no item can contain multiple pieces of information exists (No), the processing in step S1706 is not executed, and the processing in step S1707 is executed instead.

[0151] In step S1706, the instruction generation unit 1607 and the first information extraction unit 1608 use a large-scale language model (LLM) to extract from the object document the item name and item content corresponding to the extracted object item, which can contain multiple pieces of information. Specifically, the instruction generation unit 1607 generates a prompt for the large-scale language model (LLM) to extract information about items that can contain multiple pieces of information from the object document. The instruction generation unit 1607 generates a prompt containing an instruction for extracting information corresponding to the extracted object item that can contain multiple pieces of information from the object document, as well as the string and position information of the object document. The instruction generation unit 1607 also includes information from the reference document selected in step S1704 (the string and position information of the document, and the extraction result) as an example for few-shot learning to generate the prompt.

[0152] Figure 20 An example of a prompt generated by the instruction generation unit 1607 is shown. For example... Figure 20 As shown, prompt 2000 includes a description 2010 of the method for extracting information from a document, a group of descriptions of the item to be extracted 2020, strings of previously processed documents and their positional information and an example of the response 2030, an example of the output format 2040, and strings of the object document and their positional information 2050. The description 2010 of the extraction method describes the method for extracting information about the desired item from the object document, while the group of descriptions of the desired item 2020 describes the item name and its description for one or more items to be extracted. By including the item's description in the prompt, the knowledge of the item held by the Large Scale Language Model (LLM) can be updated, and knowledge of the item not held by the LLM can be learned, enabling high-precision extraction of information corresponding to the specified item. The example 2030 of strings of previously processed documents and their positional information and the response is an example (sample) for few-shot learning, containing information about a reference document selected from previously processed documents (the document's string, its positional information, and the extraction result) 2031. Example 2040 of the output format is an example of the output format when the information corresponding to the item to be extracted is output as the extraction result. The string and position information of the object document 2050 are the strings and position information contained in the object document obtained in step S1701, including the string and position information of the page of the object document 2051.

[0153] After the instruction generation unit 1607 generates a prompt that is provided to the large-scale language model (LLM), the first information extraction unit 1608 performs reasoning about information extraction by inputting the generated prompt into the large-scale language model (LLM), and extracts information from the object document that corresponds to the extraction object item, which may include multiple pieces of information.

[0154] In step S1707, the second information extraction unit 1610 extracts information from the object document corresponding to the remaining items (items whose content does not contain multiple pieces of information) of the extracted object item, based on the location information in the reference document selected in step S1704 and the location information in the object document. For example, the second information extraction unit 1610 extracts the string whose center point of the rectangle (boundary box) calculated based on the location information obtained in step S1701 exists in the rectangle (boundary box) based on the location information of the item in the reference document, as the information (item content) corresponding to the extracted object item. Here, when multiple pieces of information are contained as the location information of the item, it is sufficient to extract the information by using a rectangle that contains all the strings of the corresponding item content, that is, a rectangle defined by the minimum value of the x-coordinate, the minimum value of the y-coordinate, the maximum value of the x-coordinate, and the maximum value of the y-coordinate of these location information.

[0155] In step S1708, the control unit 1601 outputs the extraction results of the object document corresponding to the extraction object item obtained in steps S1706 and S1707. Additionally, the control unit 1601 may also output strings and their position information from the object document obtained in step S1701. After executing step S1708, the information processing device 100 terminates. Figure 17 The processing is shown.

[0156] Furthermore, in the above description, a large-scale language model (LLM) is used to extract the item name and item content corresponding to the extraction target item that can contain multiple pieces of information. However, it is not limited to items that can contain multiple pieces of information; a large-scale language model (LLM) can also be used to extract the item name and item content corresponding to all items that are extraction targets. Alternatively, regardless of whether there are items that can contain multiple pieces of information, the large-scale language model (LLM) can be provided with information from the reference document selected in step S1704 as a prompt for few-shot learning, and the large-scale language model (LLM) can be used to extract the item name and item content corresponding to the extraction target item.

[0157] In step S1709, which is executed when the instruction generation unit 1607 determines that there are no documents in the previously processed documents with an average overlap ratio (average IoU) of a threshold or higher, the instruction generation unit 1607 and the first information extraction unit 1608 use a large-scale language model (LLM) to extract information corresponding to each item as the extraction target from the object document. Specifically, the instruction generation unit 1607 generates information such as... Figure 20The prompt provided to a large-scale language model (LLM) for extracting information corresponding to each item of the extracted object from the object document is an example of a prompt. In this case, the instruction generation unit 1607 may generate the prompt by including, for example, an example of a document representing a method for extracting information from a document, or an example of a previously processed document with a format similar to the object document, as an example for few-shot learning. The first information extraction unit 1608 performs reasoning about information extraction by inputting the prompt generated by the instruction generation unit 1607 into the large-scale language model (LLM), and extracts information corresponding to the extracted object items from the object document. Here, in this embodiment, as information corresponding to the extracted object items, strings related to the item name and item content, as well as their position information, are extracted. That is, the content of the item name, the string containing the item name (and its position information), and the position information of the item name, the item content, the string containing the item (and its position information), and the position information of the item are extracted from the object document as information corresponding to the extracted object items. In addition to this information, other information may also be extracted from the object document.

[0158] In step S1710, the control unit 1601 outputs the extraction results of the object document corresponding to the extraction object item obtained in step S1709. Additionally, the control unit 1601 may also output the strings and their position information from the object document obtained in step S1701.

[0159] In step S1711, the control unit 1601 determines whether there is any correction input for the extraction result of the object document output in step S1710. For example, if the control unit 1601 confirms a correction request from a user who has confirmed the extraction result of the output object document, it determines that there is correction input. If the control unit 1601 determines that there is correction input for the extraction result (yes), the processing in step S1712 is executed. On the other hand, if the control unit 1601 determines that there is no correction input for the extraction result (no), the processing in step S1712 is not executed, and the processing in step S1713 is executed instead.

[0160] In step S1712, the correction unit 1609 corrects the extraction result of the object document based on the correction input of the extraction result.

[0161] In step S1713, the control unit 1601 stores the string and its location information related to the object document in the database (storage unit 1603) in association with the extraction result. For example, the control unit 1601 assigns a unique identifier (ID) to the object document and stores this identifier (ID), the string and its location information obtained in step S1701, and the extraction result obtained in step S1710 (or the corrected extraction result modified in step S1712) in association with the database. After executing step S1713, the information processing device 100 terminates. Figure 17 The processing is shown.

[0162] According to the third embodiment, the information processing apparatus 100 obtains a document with a similar layout as a reference document from the document information of previously processed documents, based on the average overlap ratio (average IoU) related to the regions of strings in the object document and the previously processed documents. Using the information in the reference document, it extracts the item name and item content corresponding to the extracted item, thereby enabling high-precision extraction of the item name and item content corresponding to the extracted item from the object document. Furthermore, by appropriately inputting a prompt containing an instruction for extracting information (strings) corresponding to the specified item and the document information of the object document into a large-scale language model, it is possible to extract information (strings) corresponding to the desired item from the image data of the object document. Moreover, by including (incorporating) the information (document information and extraction results) from the reference document obtained from previously processed documents as an example for few-shot learning into the prompt and inputting it into the large-scale language model, even if not all of it is written as an article, it is possible to easily and efficiently learn information equivalent to tacit knowledge or business knowledge, enabling high-precision extraction of information (strings) corresponding to the desired item from the image data of the object document.

[0163] Furthermore, in the description of the third embodiment, the determination of whether a string exists at the same position in the object document and previously processed documents is based on whether the center point of a rectangle calculated from the position information of the string contained in the object document exists within a rectangle based on the position information in the previously processed documents. However, this is not a limitation; other methods can also be used to determine whether a string exists at the same position in the object document and previously processed documents. For example, when the distance between the center point of the rectangle calculated based on the position information in the previously processed documents and the center point of the rectangle calculated based on the position information of the string included in the object document is below a predetermined threshold, it can be determined that the string exists at the same position. Additionally, the string with the shortest distance between the center point of the rectangle calculated based on the position information in the previously processed documents and the center point of the rectangle calculated based on the position information of the string included in the object document can be considered a string existing at the same position. Alternatively, the string with the highest cosine similarity among a predetermined number of strings from the side with the shortest distance can be considered a string existing at the same position. Furthermore, images near the string can be input into a large-scale language model to determine whether a string exists at the same position.

[0164] Furthermore, in the above description of the third embodiment, various information related to the extraction target item is extracted by inputting a string extracted from the image and information from previously processed documents. However, it is not limited to this; other methods can also be used to extract information related to the extraction target item by inputting the obtained image (hereinafter referred to as the "current image") and images of previously processed documents (hereinafter referred to as "past images") together with the string extracted from the image and information from previously processed documents. In addition, to distinguish between the current image and past images, each image can be assigned a string representing each image as input. Thus, it is possible to extract various information related to the extraction target item with high accuracy using a large-scale language model (LLM).

[0165] Furthermore, in the description of the third embodiment above, a large-scale language model (LLM) is used to extract various information related to the extraction target item. Therefore, even when the object document contains multiple strings corresponding to item names and multiple strings corresponding to item content, the item names and item content can be extracted with high accuracy. Additionally, in the description above, the prompts input to the large-scale language model (LLM) include information including item descriptions and extraction results from previously processed documents. Therefore, even when the extraction target item differs from the item name in the object document, learning can be performed based on information from previously processed documents, allowing for high-precision extraction of the object document's items as the extraction target item.

[0166] Furthermore, in the above description of the third embodiment, when the string of the extracted object item cannot contain multiple pieces of information as item content (only a single piece of information as item content), a large-scale language model (LLM) is not used. Instead, positional information in the object document and positional information in previously processed documents are used to extract the extracted object. As a result, compared with the case of using a large-scale language model (LLM), the processing speed is improved and the processing amount (computational cost) is reduced.

[0167] In the second and third embodiments described above, the overlap ratio (IoU) between the bounding boxes in the previously processed documents and the bounding boxes in the object document is calculated. However, the calculation can be performed only on the bounding boxes in the previously processed documents that have a portion that overlaps with the bounding boxes in the object document.

[0168] (Condition 1) The minimum x-coordinate of the bounding box in the previously processed document is less than the maximum x-coordinate of the bounding box in the object document.

[0169] (Condition 2) The maximum x-coordinate of the bounding box in the previously processed document is greater than the minimum x-coordinate of the bounding box in the object document.

[0170] (Condition 3) The minimum y-coordinate of the bounding box in the previously processed document is less than the maximum y-coordinate of the bounding box in the object document.

[0171] (Condition 4) The maximum y-coordinate of the bounding box in the previously processed document is greater than the minimum y-coordinate of the bounding box in the object document.

[0172] In all bounding boxes of previously processed documents, those that satisfy all conditions 1 through 4 overlap with the bounding boxes in the object document. This can be achieved simply by comparing the coordinate values ​​of the positional information in the object document and the previously processed documents. Therefore, by performing the overlap ratio (IoU) calculation on the bounding boxes in the previously processed documents, the computation time can be reduced without sacrificing accuracy.

[0173] Furthermore, documents with similar layouts generally have roughly the same number of pages. Therefore, the number of documents selected for calculation can be reduced based on the difference in page count between the target document and previously processed documents. For example, previously processed documents whose page count difference relative to the target document is below a predetermined threshold can be selected as calculation objects. Alternatively, metadata can be assigned to each document, and the number of documents selected as calculation objects can be reduced based on this metadata. For example, if it is a company document, the company name and other information can be assigned to the document as metadata, and previously processed documents with the same company name as their metadata can be selected as calculation objects. By reducing the number of documents selected as calculation objects in this way, processing time can be reduced.

[0174] <Other Implementation Methods>

[0175] Furthermore, the present invention is also achieved by performing the following process: Software (programs) that implement the functions of the above-described embodiments are provided to the system or device via a network or various recording media. The computer (or CPU, MPU, etc.) of the system or device then reads and executes the program. Additionally, computer-readable recording media containing the program and computer program products such as the program can also be used as embodiments of the present invention. Examples of recording media include floppy disks, hard disks, optical disks, optical discs, CD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0176] Furthermore, the above embodiments are merely examples illustrating specific implementations of the present invention, and the scope of the invention should not be limited by them. That is, the present invention can be implemented in various ways without departing from its technical concept or its main features.

[0177] The following apparatus, methods, etc. are also included in the disclosure of this embodiment.

[0178] (1) An information processing device, comprising:

[0179] The acquisition unit acquires the string contained in the image data of the document and the position information indicating the position of the string as document information;

[0180] A conversion unit that converts the acquired document information into a distributed representation;

[0181] An information extraction unit inputs a prompt containing the obtained document information into a large-scale language model, uses the large-scale language model to perform reasoning, and extracts a string corresponding to the item indicated by the prompt.

[0182] A storage unit that associates and stores the document information related to the document, the distributed representation, and the extraction results of the information extraction unit in a storage unit; and

[0183] The selection unit selects a reference document from the previously processed documents based on the dispersion representation related to the document being processed and the dispersion representation related to previously processed documents stored in the storage unit.

[0184] The prompt for inputting the document being processed includes information about the selected reference document.

[0185] (2) The information processing apparatus according to (1), wherein the prompt input to the processing object document includes the document information related to the selected reference document and the extraction result.

[0186] (3) The information processing apparatus according to (1) or (2), wherein the selection unit selects a predetermined number of documents as reference documents from the previously processed documents in descending order of similarity between the dispersion representation related to the processing object document and the dispersion representation related to the previously processed documents.

[0187] (4) The information processing apparatus according to (1) or (2), wherein the selection unit selects a predetermined number of documents as reference documents from the previously processed documents in order of distance from the dispersion representation related to the processing object document and the dispersion representation related to the previously processed documents.

[0188] (5) The information processing apparatus according to any one of (1) to (4), wherein,

[0189] The information processing device further includes: a correction unit, which accepts a correction request for the extraction result of the information extraction unit, and corrects the string corresponding to the item in the extraction result according to the correction request.

[0190] When the correction unit makes a correction, the document information related to the document, the dispersion representation, and the corrected extraction result are associated and stored in the storage unit.

[0191] (6) The information processing apparatus according to (5), wherein the selection unit selects the reference document from previously processed documents in the storage unit that store the document information related to the document, the dispersion representation and the corrected extraction result.

[0192] (7) The information processing apparatus according to any one of (1) to (6), wherein the selection unit selects the reference document by comparing the dispersion performance related to the processing object document and the dispersion performance related to previously processed documents using the k-nearest neighbor method.

[0193] (8) The information processing apparatus according to any one of (1) to (7), wherein the prompt for inputting the document to be processed is generated by incorporating the document information stored in the storage unit related to the reference document and the extraction result.

[0194] (9) The information processing apparatus according to any one of (1) to (8), wherein the prompt includes a description of the item.

[0195] (10) The information processing apparatus according to (1) or (2), wherein,

[0196] The storage unit stores the document information related to the document and the extraction results of the information extraction unit, which include location information related to the extracted strings, in the storage unit in association.

[0197] The selection unit, instead of the distributed representation, selects the reference document from the previously processed documents based on the region of strings in the processing object document represented by the location information related to the processing object document, and the region of strings in the previously processed documents represented by the location information related to the previously processed documents stored in the storage unit.

[0198] (11) The information processing apparatus according to (10), wherein,

[0199] The information processing device further includes: a calculation unit, which calculates the overlap ratio between the region of the string in the processing object document and the region of the string in the previously processed document.

[0200] The selection unit selects the reference document from the previously processed documents based on the overlap ratio calculated by the calculation unit.

[0201] (12) The information processing apparatus according to (11), wherein,

[0202] The processing unit calculates the overlap ratio between each region of the string in the processing object document and the regions of the string in previously processed documents, and for each of the previously processed documents, calculates the average of the maximum overlap ratios of each region of the string, i.e., the average overlap ratio.

[0203] The selection unit selects the reference document from the previously processed documents based on the average overlap ratio calculated by the calculation unit for each of the previously processed documents.

[0204] (13) The information processing apparatus according to (12), wherein,

[0205] The selection unit selects the document with the highest calculated average overlap ratio from the previously processed documents as the reference document.

[0206] (14) The information processing apparatus according to (12), wherein,

[0207] The selection unit selects a predetermined number of documents as reference documents from the previously processed documents in descending order of the calculated average overlap ratio.

[0208] (15) The information processing apparatus according to any one of (11) to (14), wherein,

[0209] The calculation unit calculates the overlap ratio of the string regions only on the corresponding pages in the processing object document and the previously processed documents.

[0210] (16) The information processing apparatus according to any one of (10) to (15), wherein,

[0211] If the extracted object item is one that can contain multiple strings, then the information extraction unit uses the large-scale language model to extract the string corresponding to the extracted object item from the processed object document.

[0212] If the extracted object item is an item containing only one string, the information extraction unit extracts the string corresponding to the extracted object item from the processing object document based on the position information of the string in the selected reference document and the position information of the string in the processing object document.

[0213] (17) An information processing method executed by an information processing device, the information processing method comprising the following steps:

[0214] The acquisition process involves obtaining the string contained in the image data of the document and the location information indicating the position of that string as document information;

[0215] The conversion process transforms the obtained document information into a distributed representation;

[0216] The information extraction process involves inputting a prompt containing the obtained document information into a large-scale language model, using the large-scale language model for reasoning, and extracting the string corresponding to the item indicated by the prompt.

[0217] The storage process associates and stores the document information related to the document, the distributed representation, and the extraction results from the information extraction process in the storage unit; and

[0218] The selection process involves choosing a reference document from the previously processed documents based on the dispersion representation related to the document being processed and the dispersion representation related to previously processed documents stored in the storage unit.

[0219] The prompt for inputting the document being processed includes information about the selected reference document.

[0220] (18) According to the information processing method described in (17), wherein,

[0221] In the storage process, the document information related to the document and the extraction results from the information extraction process, which include location information related to the extracted string, are stored together in the storage unit.

[0222] In the selection process, instead of the dispersed representation, the reference document is selected from the previously processed documents based on the region of strings in the processing object document represented by the location information related to the processing object document, and the region of strings in previously processed documents represented by the location information related to previously processed documents stored in the storage unit.

[0223] (19) A program (computer program product) that causes a computer of an information processing device to perform the following steps:

[0224] The acquisition step involves obtaining the string contained in the image data of the document and the location information representing the position of that string as document information;

[0225] The conversion step transforms the obtained document information into a distributed representation;

[0226] The information extraction step involves inputting a prompt containing the obtained document information into a large-scale language model, using the large-scale language model for reasoning, and extracting the string corresponding to the item indicated by the prompt.

[0227] The storage step associates the document information related to the document, the distributed representation, and the extraction results from the information extraction step and stores them in the storage unit; and

[0228] The selection step involves choosing a reference document from the previously processed documents based on the dispersion representation related to the document being processed and the dispersion representation related to previously processed documents stored in the storage unit.

[0229] The prompt for inputting the document being processed includes information about the selected reference document.

[0230] (20) According to the program (computer program product) described in (19), wherein,

[0231] In the storage step, the document information related to the document and the extraction results from the information extraction step, which include location information related to the extracted strings, are stored together in the storage unit.

[0232] In the selection step, instead of the dispersed representation, the reference document is selected from the previously processed documents based on the region of strings in the document of the processing object represented by the location information related to the document of the processing object, and the region of strings in the previously processed documents represented by the location information related to the previously processed documents stored in the storage unit.

[0233] (21) A computer-readable recording medium having a program that causes a computer of an information processing device to perform the following steps:

[0234] The acquisition step involves obtaining the string contained in the image data of the document and the location information representing the position of that string as document information;

[0235] The conversion step transforms the obtained document information into a distributed representation;

[0236] The information extraction step involves inputting a prompt containing the obtained document information into a large-scale language model, using the large-scale language model for reasoning, and extracting the string corresponding to the item indicated by the prompt.

[0237] The storage step associates the document information related to the document, the distributed representation, and the extraction results from the information extraction step and stores them in the storage unit; and

[0238] The selection step involves choosing a reference document from the previously processed documents based on the dispersion representation related to the document being processed and the dispersion representation related to previously processed documents stored in the storage unit.

[0239] The prompt for inputting the document being processed includes information about the selected reference document.

[0240] (22) The computer-readable recording medium according to (21), wherein,

[0241] In the storage step, the document information related to the document and the extraction results from the information extraction step, which include location information related to the extracted strings, are stored together in the storage unit.

[0242] In the selection step, instead of the dispersed representation, the reference document is selected from the previously processed documents based on the region of strings in the processing object document represented by the location information related to the processing object document, and the region of strings in the previously processed documents represented by the location information related to the previously processed documents stored in the storage unit.

Claims

1. An information processing device, characterized in that, have: The acquisition unit acquires the string contained in the image data of the document and the position information indicating the position of the string as document information; A conversion unit that converts the acquired document information into a distributed representation; An information extraction unit inputs a prompt containing the obtained document information into a large-scale language model, uses the large-scale language model to perform reasoning, and extracts a string corresponding to the item indicated by the prompt. A storage unit that associates the document information related to the document, the distributed representation, and the extraction results of the information extraction unit in the storage unit; as well as The selection unit selects a reference document from the previously processed documents based on the dispersion representation related to the document being processed and the dispersion representation related to previously processed documents stored in the storage unit. The prompt for inputting the document being processed includes information about the selected reference document.

2. The information processing device according to claim 1, characterized in that, The prompts for inputting the document to be processed include document information related to the selected reference document and the extraction results.

3. The information processing apparatus according to claim 1 or 2, characterized in that, The selection unit selects a predetermined number of documents as reference documents from the previously processed documents, in descending order of similarity between the dispersion representation related to the document being processed and the dispersion representation related to the previously processed documents.

4. The information processing apparatus according to claim 1 or 2, characterized in that, The selection unit selects a predetermined number of documents as reference documents from the previously processed documents in order of distance from the dispersion representation related to the processing target document and the dispersion representation related to the previously processed documents, from near to far.

5. The information processing apparatus according to any one of claims 1 to 4, characterized in that, The information processing device further includes: a correction unit, which accepts a correction request for the extraction result of the information extraction unit, and corrects the string corresponding to the item in the extraction result according to the correction request. When the correction unit makes a correction, the document information related to the document, the dispersion representation, and the corrected extraction result are associated and stored in the storage unit.

6. The information processing apparatus according to claim 5, characterized in that, The selection unit selects the reference document from previously processed documents stored in the storage unit, which contain the document information related to the document, the dispersion representation, and the corrected extraction results.

7. The information processing apparatus according to any one of claims 1 to 6, characterized in that, The selection unit uses the k-nearest neighbor method to compare the dispersion performance related to the document being processed with the dispersion performance related to previously processed documents to select the reference document.

8. The information processing apparatus according to any one of claims 1 to 7, characterized in that, The prompt for inputting the document to be processed is generated by incorporating the document information stored in the storage unit related to the reference document and the extraction results.

9. The information processing apparatus according to any one of claims 1 to 8, characterized in that, The prompt contains a description of the project.

10. The information processing apparatus according to claim 1 or 2, characterized in that, The storage unit stores the document information related to the document and the extraction results of the information extraction unit, which include location information related to the extracted strings, in the storage unit in association. The selection unit, instead of the distributed representation, selects the reference document from the previously processed documents based on the region of strings in the processing object document represented by the location information related to the processing object document, and the region of strings in the previously processed documents represented by the location information related to the previously processed documents stored in the storage unit.

11. The information processing apparatus according to claim 10, characterized in that, The information processing device further includes: a calculation unit, which calculates the overlap ratio between the region of the string in the processing object document and the region of the string in the previously processed document. The selection unit selects the reference document from the previously processed documents based on the overlap ratio calculated by the calculation unit.

12. The information processing apparatus according to claim 11, characterized in that, The processing unit calculates the overlap ratio between each region of the string in the processing object document and the regions of the string in previously processed documents, and for each of the previously processed documents, calculates the average of the maximum overlap ratios of each region of the string, i.e., the average overlap ratio. The selection unit selects the reference document from the previously processed documents based on the average overlap ratio calculated by the calculation unit for each of the previously processed documents.

13. The information processing apparatus according to claim 12, characterized in that, The selection unit selects the document with the highest calculated average overlap ratio from the previously processed documents as the reference document.

14. The information processing apparatus according to claim 12, characterized in that, The selection unit selects a predetermined number of documents as reference documents from the previously processed documents in descending order of the calculated average overlap ratio.

15. The information processing apparatus according to any one of claims 11 to 14, characterized in that, The calculation unit calculates the overlap ratio of the string regions only on the corresponding pages in the processing object document and the previously processed documents.

16. The information processing apparatus according to any one of claims 10 to 15, characterized in that, If the extracted object item is one that can contain multiple strings, then the information extraction unit uses the large-scale language model to extract the string corresponding to the extracted object item from the processed object document. If the extracted object item is an item containing only one string, the information extraction unit extracts the string corresponding to the extracted object item from the processing object document based on the position information of the string in the selected reference document and the position information of the string in the processing object document.

17. An information processing method executed by an information processing device, characterized in that, The information processing method includes the following steps: The acquisition process involves obtaining the string contained in the image data of the document and the location information indicating the position of that string as document information; The conversion process transforms the obtained document information into a distributed representation; The information extraction process involves inputting a prompt containing the obtained document information into a large-scale language model, using the large-scale language model for reasoning, and extracting the string corresponding to the item indicated by the prompt. The storage process associates the document information related to the document, the distributed representation, and the extraction results from the information extraction process and stores them in the storage unit; as well as The selection process involves choosing a reference document from the previously processed documents based on the dispersion representation related to the document being processed and the dispersion representation related to previously processed documents stored in the storage unit. The prompt for inputting the document being processed includes information about the selected reference document.

18. The information processing method according to claim 17, characterized in that, In the storage process, the document information related to the document and the extraction results from the information extraction process, which include location information related to the extracted string, are stored together in the storage unit. In the selection process, instead of the dispersed representation, the reference document is selected from the previously processed documents based on the region of strings in the processing object document represented by the location information related to the processing object document, and the region of strings in previously processed documents represented by the location information related to previously processed documents stored in the storage unit.

19. A computer program product, characterized in that, The computer of the information processing device performs the following steps: The acquisition step involves obtaining the string contained in the image data of the document and the location information representing the position of that string as document information; The conversion step transforms the obtained document information into a distributed representation; The information extraction step involves inputting a prompt containing the obtained document information into a large-scale language model, using the large-scale language model for reasoning, and extracting the string corresponding to the item indicated by the prompt. The storage step involves associating the document information related to the document, the distributed representation, and the extraction results from the information extraction step and storing them in the storage unit. as well as The selection step involves choosing a reference document from the previously processed documents based on the dispersion representation related to the document being processed and the dispersion representation related to previously processed documents stored in the storage unit. The prompt for inputting the document being processed includes information about the selected reference document.

20. The computer program product according to claim 19, characterized in that, In the storage step, the document information related to the document and the extraction results from the information extraction step, which include location information related to the extracted strings, are stored together in the storage unit. In the selection step, instead of the dispersed representation, the reference document is selected from the previously processed documents based on the region of strings in the document of the processing object represented by the location information related to the document of the processing object, and the region of strings in the previously processed documents represented by the location information related to the previously processed documents stored in the storage unit.

21. A computer-readable recording medium, characterized in that, The program contains a record of a procedure that causes a computer, an information processing device, to perform the following steps: The acquisition step involves obtaining the string contained in the image data of the document and the location information representing the position of that string as document information; The conversion step transforms the obtained document information into a distributed representation; The information extraction step involves inputting a prompt containing the obtained document information into a large-scale language model, using the large-scale language model for reasoning, and extracting the string corresponding to the item indicated by the prompt. The storage step involves associating the document information related to the document, the distributed representation, and the extraction results from the information extraction step and storing them in the storage unit. as well as The selection step involves choosing a reference document from the previously processed documents based on the dispersion representation related to the document being processed and the dispersion representation related to previously processed documents stored in the storage unit. The prompt for inputting the document being processed includes information about the selected reference document.

22. The computer-readable recording medium according to claim 21, characterized in that, In the storage step, the document information related to the document and the extraction results from the information extraction step, which include location information related to the extracted strings, are stored together in the storage unit. In the selection step, instead of the dispersed representation, the reference document is selected from the previously processed documents based on the region of strings in the processing object document represented by the location information related to the processing object document, and the region of strings in the previously processed documents represented by the location information related to the previously processed documents stored in the storage unit.

Images