Information processing device, information processing method, and program

Abstract

The present invention provides an information processing device capable of accurately extracting item names and item contents corresponding to target items from image data of documents. [Solution] The information processing device includes storage means for storing in a storage unit as document information, associated with the item to be extracted, the item name and item content corresponding to the item to be extracted, and position information indicating the respective locations of the item name and item content, which are extracted from documents processed in the past based on the item to be extracted; search means for searching for the document information of documents processed in the past that correspond to the document to be processed, from among the document information stored in the storage unit, based on the string contained in the document to be processed and the position information indicating the location of the string; and information extraction means for inputting a prompt containing the document information obtained by the search into a large-scale language model, performing inference by the large-scale language model, and extracting the item name and item content corresponding to the item to be extracted from the document to be processed.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Techniques for extracting handwritten or printed character strings from image data such as documents using optical character recognition (OCR) processing are known. There is also a technique for performing optical character recognition processing on image data such as documents and extracting character strings corresponding to specified items. As a method for extracting a character string corresponding to a specified item, for example, there is a method of extracting a character string located in a predetermined direction with respect to an item name in a document, or a method of extracting a character string in the vicinity of an item name included in image data as a candidate character string and determining and extracting one character string from the candidate character strings based on the number of occurrences in the image data.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, if the item name for the item to be extracted is not included in the document, or if the item name specified as the item to be extracted differs from the item name included in the document, the content of the item corresponding to the item (hereinafter also referred to as "item content") cannot always be extracted with high accuracy. Furthermore, if the item name or item content corresponding to the item to be extracted is written across multiple lines in the document, or if the item content is written in a divided manner, the item name or item content cannot always be extracted with high accuracy. The present invention aims to provide an information processing device, an information processing method, and a program that can accurately extract the item name and item content corresponding to the target item from image data of a document. [Means for solving the problem]

[0005] The information processing device according to the present invention is characterized by comprising: a storage means for storing in a storage unit as document information, associated with the item to be extracted, the item name and item content corresponding to the item to be extracted, and position information indicating the respective locations of the item name and item content, which are extracted from documents processed in the past based on the item to be extracted; a search means for searching for the document information of a previously processed document corresponding to the document to be processed from among the document information stored in the storage unit, based on a string contained in the document to be processed and position information indicating the location of the string; and an information extraction means for inputting a prompt containing the document information obtained by the search into a large-scale language model, performing inference by the large-scale language model, and extracting the item name and item content corresponding to the item to be extracted from the document to be processed. [Effects of the Invention]

[0006] According to the present invention, it is possible to provide an information processing device, an information processing method, and a program that can accurately extract item names and item contents corresponding to target items from image data of a document. [Brief explanation of the drawing]

[0007] [Figure 1]This figure shows an example of the hardware configuration of an information processing device. [Figure 2] This figure shows an example of the functional configuration of an information processing device. [Figure 3] This is a flowchart showing an example of processing performed by an information processing device. [Figure 4] This diagram illustrates the information stored in the document information database. [Figure 5] This diagram illustrates an example of information stored in the document information database. [Figure 6] This is a diagram illustrating the extraction of information from documents. [Figure 7] This diagram illustrates an example of a prompt related to information extraction. [Figure 8] This diagram illustrates the correction of the extraction results. [Figure 9] This diagram illustrates the correction of the extraction results. [Figure 10A] This diagram illustrates an example of a prompt related to information extraction. [Figure 10B] This diagram illustrates an example of a prompt related to information extraction. [Figure 10C] This diagram illustrates an example of a prompt related to information extraction. [Figure 11] This diagram illustrates an example of information extraction results. [Modes for carrying out the invention]

[0008] Embodiments of the present invention will be described based on the drawings.

[0009] Figure 1 shows an example of the hardware configuration of an information processing device 100 in one embodiment of the present invention. The information processing device 100 includes a CPU 101, ROM 102, RAM 103, auxiliary storage device 104, output device 105, input device 106, and network I / F 107. The CPU 101, ROM 102, RAM 103, auxiliary storage device 104, output device 105, input device 106, and network I / F 107 are communicated together via a system bus 108.

[0010] The CPU (Central Processing Unit) 101 is a central processing unit that controls various operations of the information processing device 100. For example, the CPU 101 may control the operation of the entire information processing device 100. The ROM (Read Only Memory) 102 stores control programs, boot programs, etc., that can be executed by the CPU 101. The RAM (Random Access Memory) 103 is the main memory of the CPU 101 and is used as a work area or a temporary storage area for deploying various programs.

[0011] The auxiliary storage device 104 stores various data and programs. For example, the auxiliary storage device 104 is implemented by a storage device capable of temporarily or permanently storing various data, such as an HDD (Hard Disk Drive) or non-volatile memory such as an SSD (Solid State Drive).

[0012] The output device 105 is a device that outputs various types of information and is used to present various types of information to the user. For example, the output device 105 may be implemented by a display device such as a display. The output device 105 may present information to the user by displaying various types of display information. As another example, the output device 105 may be implemented by an acoustic output device that outputs sounds such as voice or electronic sounds. In this case, the output device 105 may present information to the user by outputting sounds such as voice or electronic sounds. Furthermore, the device to which the output device 105 is applied may be appropriately changed depending on the medium used to present information to the user.

[0013] The input device 106 is used to receive various instructions from the user. For example, the input device 106 may include input devices such as a mouse, a keyboard, and a touch panel. As another example, the input device 106 may include a sound collection device such as a microphone, and may collect the voice spoken by the user. In this case, various analysis processes such as acoustic analysis and natural language processing may be performed on the collected voice, and the content indicated by this voice may be recognized as an instruction from the user. Further, the device applied as the input device 106 may be appropriately changed according to the method of recognizing an instruction from the user. Also, a plurality of types of devices may be applied as the input device 106.

[0014] The network I / F 107 is used for communication with an external device or the like via a network. Note that the device applied as the network I / F 107 may be appropriately changed according to the type of communication path and the applied communication method.

[0015] The CPU 101 expands the program stored in the ROM 102 or the auxiliary storage device 104 into the RAM 103 and executes the program, thereby realizing each function and each process of the information processing apparatus described later. The program of the information processing apparatus 100 may be provided to the information processing apparatus 100 by a recording medium such as a CD-ROM, or may be downloaded via a network or the like. When the program of the information processing apparatus 100 is provided by a recording medium, the recording medium is set in a predetermined drive device, and the program recorded in the recording medium is installed in the auxiliary storage device 104.

[0016] The configuration shown in FIG. 1 is merely an example, and does not necessarily limit the hardware configuration of the information processing apparatus 100 in the present embodiment. As an example, some configurations such as the output device 105 and the input device 106 may not be included. Also, as another example, a configuration corresponding to the function realized by the information processing apparatus 100 may be appropriately added.

[0017] Figure 2 shows an example of the functional configuration of the information processing device 100 in this embodiment. The information processing device 100 includes a control unit 201, an input / output control unit 202, a storage unit 203, an acquisition unit 204, a search unit 205, an instruction generation unit 206, a first information extraction unit 207, a modification unit 208, and a second information extraction unit 209.

[0018] The control unit 201 is responsible for controlling each component of the information processing device 100. The input / output control unit 202 performs various processes related to presenting various information to the user and receiving information input from the user (e.g., instructions). For example, the input / output control unit 202 may perform processes related to presenting a UI (User Interface) and processes related to receiving input via the UI. This enables the information processing device 100 to recognize instructions from the user and present the user with the results of processing corresponding to those instructions.

[0019] The memory unit 203 schematically represents a memory area for storing various data and programs. For example, the memory unit 203 may store data and programs for each component of the information processing device 100 to execute processing. The memory unit 203 may also store a trained model that has undergone machine learning (deep learning) and is used for inference related to information extraction performed in the information processing device 100. Furthermore, the memory unit 203 may store various information related to documents, such as document information and information extraction results.

[0020] In this embodiment, document information refers to information that associates the item name, the content of the item corresponding to the item name (item content), and the positional information for the item name and item content, respectively, with the item to be extracted, based on the item to be extracted (item to be extracted). The positional information refers to information that indicates the position of the string representing the item name and item content within the document.

[0021] The acquisition unit 204 acquires image data of a document by optically scanning or photographing it. The acquisition unit 204 also performs optical character recognition (OCR) processing on the acquired image data of the document to acquire the strings contained in the image data of the document and the positional information related to those strings. The acquisition unit 204 may also accept and acquire image data of a document that has been scanned or photographed externally in advance as input, without scanning or photographing the document itself. Alternatively, the acquisition unit 204 may accept and acquire the strings contained in the document and their positional information obtained as a result of external optical character recognition processing as input.

[0022] The search unit 205 searches the document information database (document information DB), which stores and stores document information about documents processed in the past, based on the information obtained from the document to be processed by the acquisition unit 204 (hereinafter also referred to as the "target document"), and searches for documents that are similar in format to the target document from among the documents that have been processed in the past. Specifically, the search unit 205 searches the document information DB based on information about item names and searches for documents that have a high degree of matching with the target document in terms of the strings contained in the document and their positional information, as documents that correspond to the target document that are similar in format to the target document. The previously processed documents that are similar in format to the target document obtained as a search result are referenced in the information extraction process for the target document.

[0023] In the information processing device 100 of this embodiment, the instruction generation unit 206 and the first information extraction unit 207 can use a Large Language Model (LLM) to extract information about the specified items to be extracted from the image data of the document. For example, the instruction generation unit 206 generates instructions to extract various information about the items to be extracted from the target document and inputs them to the first information extraction unit 207, and the first information extraction unit 207 uses the Large Language Model (LLM) to extract information according to the input instructions. The Large Language Model (LLM) is a language model constructed using a large amount of text data (such as a large corpus) and deep learning technology. When text data called prompts indicating instructions are input, the LLM performs inference based on the prompts and generates and outputs text data corresponding to the input prompts.

[0024] The instruction generation unit 206 generates a prompt that includes instructions for extracting various information about the items to be extracted from the target document, a string of characters and their positional information contained in the image data of the target document, and inputs it to the first information extraction unit 207. In this embodiment, the various information about the items to be extracted from the target document includes the item name (string of characters) corresponding to the item to be extracted, the item content (string of characters) corresponding to the item name, and the positional information (coordinate information) of the item name and item content.

[0025] The first information extraction unit 207 performs inference using a large-scale language model (LLM) based on prompts generated and input by the instruction generation unit 206, and extracts various information related to the items to be extracted as instructed by the prompts from the image data of the target document. The modification unit 208 receives a modification request from the user regarding the extraction results of information about the target document by the first information extraction unit 207, and modifies the extraction results according to the modification request.

[0026] Furthermore, in the information processing device 100 of this embodiment, the second information extraction unit 209 can use document information from documents processed in the past to extract information about the specified target items from the image data of the documents, based on the location information of the item names and item contents. The second information extraction unit 209 extracts various information about the specified target items from the image data of the target document based on the location information of the item names and item contents in the document information of documents processed in the past. Specifically, the second information extraction unit 209 searches for documents processed in the past and refers to documents with a similar format to the target document, and extracts various information about the target items from the target document based on the location information of the item names and item contents in those documents.

[0027] Furthermore, in inference using a Large-Scale Language Model (LLM), there is a method called few-shot learning that improves the accuracy of responses by taking advantage of the characteristic of the LLM that it has a high accuracy in returning output adapted to the presented example (sample) by including an example (sample) in the prompt. Therefore, the instruction generation unit 206 may generate a prompt using document information of documents highly similar to the target document (for example, documents obtained by searching in the search unit 205, or documents whose similarity to the target document in terms of document format is below a threshold) as an example for few-shot learning, and input it to the first information extraction unit 207. In this way, by providing the LLM with information of documents highly similar to the target document as an example for few-shot learning, it is possible to improve the accuracy of extracting item names and item contents corresponding to the items to be extracted from the target document.

[0028] Referring to Figure 3, the processing in the information processing device 100 in this embodiment will be described. Figure 3 is a flowchart showing an example of processing in the information processing device 100 in this embodiment.

[0029] In step S301, the acquisition unit 204 scans or photographs the document to be processed (target document) to acquire image data, performs optical character recognition processing on the acquired image data, and obtains the string and its position information contained in the image data of the target document. Here, the position information of the string is obtained as four values: the minimum x-coordinate, minimum y-coordinate, maximum x-coordinate, and maximum y-coordinate of the string in an XY coordinate system with one endpoint of the document (e.g., the top-left or bottom-left point) as the origin. In addition, during the optical character recognition processing, information indicating the width (horizontal length) and height (vertical length), which are the dimensions of the document (corresponding to the maximum x-coordinate and maximum y-coordinate values ​​that the document can take), is also obtained.

[0030] In step S302, the search unit 205 searches the document information database to obtain documents from previously processed documents that are similar in format to the target document, based on information about the item names in the document information. The document information database is located, for example, in the storage unit 203 and stores document information of documents that have been processed in the past.

[0031] Figure 4 illustrates the document information of previously processed documents stored in the document information database. Document information for previously processed documents is stored in the document information database, for example, for each page of the document. The document information may include the document type, a uniquely assigned identifier (ID), document information, extraction results, and information indicating the document's width and height. The extraction results may include, for example, the item name indicating the item to be extracted, whether or not information about that item exists, the content of the item name extracted from the document, the string where the item name existed in the document (a pair of the string and its position information), the position information of the item name, and information about the item. The string where the item name existed, the position information of the item name, and the information about the item are in a list structure, allowing for the storage of multiple pieces of information. The information about the item includes the content of the item extracted from the document, the string where the item content existed in the document (a pair of the string and its position information), and the position information of the item. The string where the item content existed and the position information of the item are also in a list structure, allowing for the storage of multiple pieces of information. For example, if an item name or information corresponding to a single item in a document is written across multiple lines or separated by lines, multiple pieces of information are stored as strings where the item name exists and strings where the item content exists. These multiple pieces of information are then combined to form a string which is stored as the item name content and item content. Furthermore, the position information of the item name and the position information of the item are stored using the width and height of the document so that the values ​​are within the range of 0 to 1. By storing this normalized position information, it is possible to restore the position information of strings that are appropriate for the size of the document, even for documents of different sizes. In this embodiment, the item name corresponding to the item to be extracted, the content of the item corresponding to the item name (item content), and the position information for the item name and item content, each of which is extracted from the document based on the item to be extracted, are stored as document information associated with the item to be extracted. Note that in the example shown in Figure 4, there is one item to be extracted (one item name indicating the item to be extracted), but this is just an example, and the number of items to be extracted can be arbitrary.

[0032] As an example, Figure 5(B) shows an example of document information extracted from the document shown in Figure 5(A) and stored in the document information database. Figure 5(B) shows the document information of the document shown in Figure 5(A) that has been extracted and stored with "Deliverables" as the extraction target item. In the document shown in Figure 5(A), the information described in column 501 of "Deliverables" is extracted as the information corresponding to the extraction target item "Deliverables" and stored in the document information database. As shown in Figure 5(B), since the information corresponding to the extraction target item "Deliverables" exists in the document, the presence or absence of the information is stored as "Yes". In addition, "Deliverables" is stored as the content of the item name corresponding to the extraction target item "Deliverables", and the string "Deliverables" and its position information "Deliverables:A,A,A,A" are stored as the string in which the item name existed. "A',A',A',A'" is stored as the position information of the item name. Here, “A',A',A',A'” is a value normalized using the width and height of the document, i.e., “A',A',A',A'” = “A / (width),A / (height),A / (width),A / (height)”. The contents of this item are stored as “DB design document”, “Similar function study results report”, “New and old business function comparison table”, and “Main table relationship diagram”. The strings indicating the existence of the item's contents are stored as the strings of each item and their positional information, such as “DB design document:B,B,B,B”, “Similar function study results report:C,C,C,C”, “New and old business function comparison table:D,D,D,D”, and “Main table relationship diagram:E,E,E,E”. The positional information of the item names is normalized based on the size of the document.

[0033] In step S302, when searching the document information database, the search unit 205 compares the strings and their positional information of each item name in previously processed documents with the strings and their positional information of the target document obtained in step S301. This allows the search unit to find documents with a similar format to the target document from among previously processed documents. Since item names are expected to be in the same position in documents of the same format, comparing the item names makes it possible to search for documents with a similar format to the target document from among previously processed documents. For example, the search unit 205 determines whether the strings exist in the same position in the target document for all item names included in the document information of a given previously processed document. For example, the search unit 205 determines that the strings exist in the same position if the center point of a rectangle calculated based on the positional information of the strings included in the target document exists within a rectangle based on the positional information of the item names in previously processed documents (more specifically, a rectangle based on the positional information after conversion to match the size of the target document). In this way, if the center point of a rectangle calculated based on the position information of the strings contained in the target document is within a rectangle based on the position information of the item name, it is determined that the string exists at the same position. This makes it possible to treat even if the position of the string is slightly shifted as the same position. If it is determined that the string exists at the same position in the target document for all item names, the search unit 205 determines whether those strings match the item name strings for each position. As a result, if all of the strings match the item name strings, that is, if all the strings related to the item names and the positions in which they exist match, it is determined that the previously processed document is a document with the same format as the target document. This search process is performed on all previously processed documents stored in the document information DB. Alternatively, previously processed documents stored in the document information DB may be classified according to attributes such as the type of document, and the above search process may be performed only on documents classified according to the attributes corresponding to the target document. Alternatively, the process in step S302 may be terminated when a document with the same format as the target document is found among the previously processed documents, and the process in step S303 may be executed.

[0034] In step S303, the control unit 201 determines, based on the results of searching the document information DB in step S302, whether there are any documents with a similar format to the target document among the documents processed in the past. If the control unit 201 determines that there are documents with a similar format to the target document (YES), the process in step S304 is executed. On the other hand, if the control unit 201 determines that there are no documents with a similar format to the target document (NO), the process in step S308 is executed.

[0035] In step S304, the control unit 201 determines whether there are any items among the items to be extracted from the target document that may contain multiple pieces of information as item content, based on previously processed documents that are similar in format to the target document. To determine whether an item may contain multiple pieces of information, for example, one can pre-set whether an item may potentially contain multiple pieces of information as item content for each item, and then determine whether there are any items that may contain multiple pieces of information based on that information. If the control unit 201 determines that there are items that may contain multiple pieces of information (YES), the process in step S305 is executed. On the other hand, if the control unit 201 determines that there are no items that may contain multiple pieces of information (NO), the process in step S305 is not executed, and the process in step S306 is executed.

[0036] The reason for determining whether or not an item among the items to be extracted may contain multiple pieces of information as its content will be explained with reference to Figures 6(A) and 6(B). Let's assume that Figure 6(A) is a previously processed document that was determined to be similar in format to the target document, and Figure 6(B) is the target document. For example, when extracting information by matching location information as will be described later, let's assume that information is extracted if the center point of the bounding rectangle for the string calculated based on the location information of the string in the target document is included in the bounding rectangle of the string in the previously processed document. In this case, the center point 615 of the bounding rectangle 611 for the string "DB design document" and the center point 620 of the bounding rectangle 619 for the string "hand-delivered" in the target document shown in Figure 6(B) are included in the bounding rectangles 601 and 602 for the string in the previously processed document shown in Figure 6(A), so information regarding the strings "DB design document" and "hand-delivered" can be extracted based on location information. On the other hand, the center point 616 of the bounding rectangle 612 for the string "Report on Similar Functions," the center point 617 of the bounding rectangle 613 for the string "Comparison Table of Old and New Business Functions," and the center point 618 of the bounding rectangle 614 for the string "Main Table Relationship Diagram" in the target document shown in Figure 6(B) are not included in any of the bounding rectangles for strings in the previously processed document shown in Figure 6(A). Therefore, this information is not extracted based on location information. In order to prevent information omissions and acquisition errors in this way, it is determined whether or not an item can contain multiple pieces of information as item content, and whether or not to extract the information based on location information.

[0037] Returning to Figure 3, in step S305, the instruction generation unit 206 and the first information extraction unit 207 use a large-scale language model (LLM) to extract item names and item contents corresponding to items to be extracted, which may contain multiple pieces of information, from the target document. Specifically, the instruction generation unit 206 generates a prompt to be given to the large-scale language model (LLM) for extracting information of items that may contain multiple pieces of information from the target document. The instruction generation unit 206 generates a prompt that includes instructions for extracting information corresponding to items to be extracted, which may contain multiple pieces of information, from the target document, as well as the string of the target document and its position information. As an example for few-shot learning, the instruction generation unit 206 further generates a prompt that includes information (the string of the document and its position information, and the extraction results) of previously processed documents that have been determined to be similar in format to the target document.

[0038] An example of a prompt generated by the instruction generation unit 206 is shown in Figure 7. As shown in Figure 7, the prompt 700 includes a description of how to extract information from the document 710, a group of item names and descriptions to be extracted 720, a string of previously processed documents and their location information and an example of a response 730, an example of an output format 740, and a string of the target document and its location information 750. The description of the extraction method 710 is a description of how to extract information on the items to be extracted from the target document, and the group of item names and descriptions to be extracted 720 is the item name and description for one or more items to be extracted. By including this item description in the prompt, it is possible to update the knowledge of items held by the Large-Scale Language Model (LLM) and to train the Large-Scale Language Model (LLM) to acquire knowledge of items it does not have, enabling accurate extraction of information corresponding to the specified items. The strings of previously processed documents, their location information, and example responses 730 are examples (samples) for few-shot learning, and include, for example, information (strings of documents and their location information, and extraction results) 731 of previously processed documents that were determined to be similar in format to the target document. The example output format 740 is an example of the output format when outputting information corresponding to the items to be extracted as extraction results. The strings of the target document and their location information 750 are the strings and their location information contained in the target document obtained in step S301, and include the strings and their location information of the pages of the target document 751.

[0039] After the instruction generation unit 206 generates a prompt to be given to the large-scale language model (LLM), the first information extraction unit 207 inputs the generated prompt to the large-scale language model (LLM) to perform inference regarding information extraction and extracts information from the target document that corresponds to the item to be extracted, which may contain multiple pieces of information.

[0040] In step S306, the second information extraction unit 209 extracts information from the target document that corresponds to the remaining items of the item to be extracted (items that cannot contain multiple pieces of information as item content) based on the location information of previously processed documents that have been determined to be similar in format to the target document, and the location information of the target document. For example, the second information extraction unit 209 extracts a string of characters in which the center point of a rectangle calculated based on the location information obtained in step S301 lies within a rectangle based on the location information of an item in a previously processed document, as the information (item content) corresponding to the item to be extracted. If the location information of an item contains multiple pieces of information, the information can be extracted using a rectangle that contains all of the corresponding item content strings, that is, a rectangle defined by the minimum x-coordinate, minimum y-coordinate, maximum x-coordinate, and maximum y-coordinate within that location information.

[0041] In step S307, the control unit 201 outputs the extraction results of target documents corresponding to the items to be extracted obtained in steps S305 and S306. The control unit 201 may also output the strings and their position information in the target documents acquired in step S301. After executing the process in step S307, the information processing device 100 terminates the process shown in Figure 3.

[0042] In the explanation above, a Large-Scale Language Model (LLM) is used to extract item names and content corresponding to items to be extracted that may contain multiple pieces of information. However, it is also possible to use the LLM to extract item names and content corresponding to all items to be extracted, not just items that may contain multiple pieces of information. Furthermore, regardless of whether or not there are items that may contain multiple pieces of information, as an example for few-shot learning, the LLM may be given a prompt containing information from previously processed documents that have been determined to be similar in format to the target document, and the LLM may be used to extract item names and content corresponding to the items to be extracted.

[0043] In step S308, which is executed when the control unit 201 determines in step S303 that there are no documents with the same format as the target document among the documents processed in the past, the instruction generation unit 206 and the first information extraction unit 207 extract information corresponding to each item to be extracted from the target document using a large-scale language model (LLM). Specifically, the instruction generation unit 206 generates prompts to be given to the large-scale language model (LLM) for extracting information corresponding to each item to be extracted from the target document, as an example shown in Figure 7. At this time, the instruction generation unit 206 may generate prompts that include, for example, an example of a document showing how to extract information from a document, or an example of a previously processed document with a similar format to the target document, as examples for few-shot learning. The first information extraction unit 207 performs inference regarding information extraction by inputting the prompts generated by the instruction generation unit 206 to the large-scale language model (LLM) and extracts information corresponding to the items to be extracted from the target document. In this embodiment, the information corresponding to the items to be extracted includes the item name, a string relating to the item content, and its position information. In other words, the content of the item name corresponding to the item to be extracted, the string in which the item name existed (and its location information), and the location information of the item name, along with the content of the item, the string in which the item existed (and its location information), and the location information of the item, are extracted from the target document as information corresponding to the item to be extracted. In addition to this information, other information may also be extracted from the target document.

[0044] In step S309, the control unit 201 outputs the extraction results of target documents corresponding to the items to be extracted obtained in the processing of step S308. The control unit 201 may also output the strings and their position information in the target documents acquired in step S301.

[0045] In step S310, the control unit 201 determines whether or not there is any modification input for the extraction results of the target document output in step S309. The control unit 201 determines that there is modification input if, for example, a modification request is confirmed from a user who has reviewed the extraction results of the output target document. If the control unit 201 determines that there is modification input for the extraction results (YES), the process in step S311 is executed. On the other hand, if the control unit 201 determines that there is no modification input for the extraction results (NO), the process in step S311 is not executed, and the process in step S312 is executed.

[0046] In step S311, the correction unit 208 corrects the extraction results of the target documents in accordance with the correction input for the extraction results. In step S312, the control unit 201 associates the string and its location information related to the target document with the extraction result and stores it in the document information DB. For example, the control unit 201 assigns a unique identifier (ID) to the target document, and stores the identifier (ID), the string and its location information obtained in step S301, and the extraction result obtained in step S308 (or the corrected extraction result modified in step S311) in the document information DB. After executing the process in step S312, the information processing device 100 terminates the process shown in Figure 3.

[0047] Referring to Figures 8 and 9, we will explain the correction process applied to the extracted results of the target documents obtained by extracting information using a Large-Scale Language Model (LLM). Figure 8(A) shows an example of UI800 displaying the extraction results before correction. In the example shown in Figure 8(A), "DB design document" is extracted as the item content corresponding to the item "deliverables" and "store the electronic file on our designated server" is extracted as the item content corresponding to the item "delivery method". In the image of the target document, the extracted item content is displayed identifiable using borders 801 and 811. In addition, the "deliverables" button 820 and the "delivery method" button 830 corresponding to the items are displayed. Figure 9(A) shows the extraction results before correction corresponding to this. In this case, only "DB design document" is extracted as the item content corresponding to the item "deliverables" and "similar function study result report", "business function old and new comparison table", and "main table relationship diagram" are not extracted as item content.

[0048] Users can modify the extraction results by performing input operations related to the modifications on the UI displaying the extraction results. For example, they can modify the extraction results by adding or moving borders on the UI displaying the extraction results. As an example, to add "Similar Function Review Results Report," "Business Function Old vs. New Comparison Table," and "Main Table Relationship Diagram" as item contents corresponding to the extraction target item "Deliverables," as shown in Figure 8(B), on the UI 850 displaying the extraction results, borders 851, 852, and 853 are added to the image of the target document so that the text "Similar Function Review Results Report," "Business Function Old vs. New Comparison Table," and "Main Table Relationship Diagram" are included, respectively, and then the "Deliverables" button 820 is pressed. As a result, the modification unit 208 modifies the item contents corresponding to the extraction target item "Deliverables," for example, as shown in Figure 9(B), so that "Similar Function Review Results Report," "Business Function Old vs. New Comparison Table," and "Main Table Relationship Diagram" are included in addition to "DB Design Document." The above example explained the modification of item contents, but item names may also be modified in the same way. In this way, users can modify the item names, item content strings, and positional information corresponding to the items to be extracted in the extraction results simply by performing operations on the UI that displays the extraction results. In the UI examples shown in Figures 8(A) and 8(B), the information included in the extraction results is displayed in a way that makes it identifiable by a border, but this is not the only way. For example, highlighting or selection display may be used, and display methods commonly used in UIs can be applied.

[0049] In this embodiment, specific examples of prompts generated by the instruction generation unit 206 are shown in Figures 10A to 10C. The elements shown in Figures 10A to 10C are combined to form a single prompt. In Figures 10A to 10C, element 1010 corresponds to the extraction method description 710, and element 1020 corresponds to the item name to be extracted and the item description group 720. Element 1030 corresponds to the document string, location information, and example answer 730. Element 1040 corresponds to the output format example 740, and element 1050 corresponds to the target document string and location information 750. By providing the prompts shown in Figures 10A to 10C to the Large-Scale Language Model (LLM), inference regarding information extraction is performed, and the extraction results shown in Figure 11 are obtained.

[0050] According to this embodiment, the information processing device 100 searches for document information of previously processed documents that corresponds to the target document, based on the string of characters and their positional information contained in the target document. Then, using the document information of the documents corresponding to the target document obtained through the search, it extracts the item name and item content corresponding to the item to be extracted, thereby enabling accurate extraction of the item name and item content corresponding to the item to be extracted from the target document.

[0051] As explained above, the determination of whether a string exists at the same location in the target document and previously processed documents is made by checking whether the center point of a rectangle calculated based on the location information of the string contained in the target document exists within a rectangle based on the location information of the previously processed document. However, this is not the only way to determine whether a string exists at the same location in the target document and previously processed documents. For example, if the distance between the center point of a rectangle calculated based on the location information of a previously processed document and the center point of a rectangle calculated based on the location information of the string contained in the target document is less than or equal to a predetermined threshold, it may be determined that the strings exist at the same location. Alternatively, for example, if the IoU (Intersection over Union) between a rectangle based on the location information of a previously processed document and a rectangle based on the location information of the string contained in the target document is greater than or equal to a predetermined threshold, it may be determined that the strings exist at the same location. Furthermore, the string whose distance between the center point of a rectangle calculated based on the location information of a previously processed document and the center point of a rectangle calculated based on the location information of the string contained in the target document is shortest may be considered as the string that exists at the same location. Alternatively, from a predetermined number of strings, starting from the shortest distance, the string with the closest cosine similarity may be considered as existing at the same location. Alternatively, images of the vicinity of a string may be input into a large-scale language model to determine whether or not a string exists at the same location.

[0052] Furthermore, the explanation above describes how to extract various information about the target item by inputting strings extracted from images and information from previously processed documents. However, the method is not limited to this; information about the target item may also be extracted by inputting the acquired image (hereinafter referred to as the "current image") and images of previously processed documents (hereinafter referred to as "past images") along with strings extracted from images and information from previously processed documents using other methods. In addition, to distinguish between the current image and past images, each image may be assigned a string to indicate its respective image before input. This makes it possible to accurately extract various information about the target item using a Large-Scale Language Model (LLM).

[0053] Furthermore, as described above, a Large-Scale Language Model (LLM) is used to extract various information about the items to be extracted. This makes it possible to accurately extract item names and item contents even if the target document contains multiple strings corresponding to item names and item contents. In addition, as described above, the prompts input to the Large-Scale Language Model (LLM) include information including item descriptions and extraction results from previously processed documents. This makes it possible to accurately extract items from the target document as items to be extracted, even if the item names in the target document differ from those in the target document, by learning from information in previously processed documents.

[0054] Furthermore, as explained above, if the string for the item to be extracted cannot contain multiple pieces of information as item content (i.e., it contains only a single piece of information as item content), the item to be extracted is not used with a Large-Scale Language Model (LLM), but rather with location information in the target document and location information in previously processed documents. This improves processing speed and reduces processing load (computational load) compared to using a Large-Scale Language Model (LLM).

[0055] It should be noted that the embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0056] The following devices and methods are also included in the disclosure of this embodiment. (1) A storage means that stores in a storage unit as document information, associated with the item to be extracted, the item name and item content corresponding to the item to be extracted, and position information indicating the respective locations of the item name and item content, extracted from documents processed in the past based on the item to be extracted. A search means that searches for document information of previously processed documents corresponding to the document to be processed from among the document information stored in the storage unit, based on a string of characters contained in the document to be processed and location information indicating the location of said string, An information processing apparatus characterized by having an information extraction means that inputs a prompt containing the document information obtained by search into a large-scale language model, performs inference by the large-scale language model, and extracts the item name and item content corresponding to the item to be extracted from the document to be processed. (2) The information processing apparatus according to (1), wherein the search means compares the item name and location information of the item name contained in the document information stored in the storage unit with the string and location information of the string contained in the document to be processed, and searches for the document information of the document corresponding to the document to be processed. (3) The information extraction means is If the item to be extracted is an item that may contain multiple item contents, the large-scale language model is used to extract the item name and item contents corresponding to the item to be extracted from the document to be processed. If the item to be extracted is an item that contains only one item content, the information processing device according to (1) or (2) is characterized in that it extracts the item name and item content corresponding to the item to be extracted from the document to be processed based on the location information contained in the document information obtained by the search and the location information in the document to be processed. (4) The information processing apparatus according to (3), wherein if the item to be extracted is an item that contains only one item content, the information extraction means performs a comparison using the location information contained in the document information obtained by the search and the location information in the document to be processed, and extracts a string of characters located at the position corresponding to the location information contained in the document information as the item name and item content corresponding to the item to be extracted from the document to be processed. (5) The information processing device according to any one of (1) to (4), characterized in that it has a correction means for receiving correction input for the extracted document information and correcting the document information in accordance with the correction input. (6) The information processing device according to any one of (1) to (5), characterized in that the document information is stored in a list structure in which the item contents corresponding to the item to be extracted can be associated with a single item. (7) The information processing apparatus according to (4), wherein the information extraction means is characterized in that the center point of a rectangle calculated based on the positional information of a string contained in the document to be processed is located within the rectangle based on the positional information of the document information obtained by the search, and the strings located at the corresponding positions are extracted as strings located at the corresponding positions. (8) The information processing apparatus according to (4), wherein the information extraction means extracts strings as strings present at the corresponding positions, wherein the distance between the center point of a rectangle calculated based on the positional information of strings contained in the document to be processed and the center point of a rectangle calculated based on the positional information of the document information obtained by searching is less than or equal to a predetermined threshold. (9) The information processing apparatus according to (4), wherein the information extraction means extracts strings that exist at the corresponding positions, wherein the degree of overlap between a rectangle based on the positional information of a string contained in the document to be processed and a rectangle based on the positional information of the document information obtained by searching is greater than or equal to a predetermined threshold. (10) An information processing method performed by an information processing device, A storage step involves storing in a storage unit as document information, associated with the item to be extracted, the item name and item content corresponding to the item to be extracted, and position information indicating the respective locations of the item name and item content, which are extracted from documents processed in the past based on the item to be extracted. A search step of searching for document information of previously processed documents corresponding to the document to be processed from among the document information stored in the storage unit, based on a string of characters contained in the document to be processed and location information indicating the location of said string, An information processing method characterized by comprising: an information extraction step of inputting a prompt containing the document information obtained by search into a large-scale language model, performing inference by the large-scale language model, and extracting the item name and item content corresponding to the item to be extracted from the document to be processed. (11) The computer of the information processing device, A storage step involves storing in a storage unit as document information, associated with the item to be extracted, the item name and item content corresponding to the item to be extracted, and position information indicating the respective locations of the item name and item content, which are extracted from documents processed in the past based on the item to be extracted. A search step of searching for document information of previously processed documents corresponding to the document to be processed from among the document information stored in the storage unit, based on a string of characters contained in the document to be processed and location information indicating the location of said string, A program for inputting a prompt containing the document information obtained through a search into a large-scale language model, performing inference using the large-scale language model, and executing an information extraction step that extracts the item name and item content corresponding to the item to be extracted from the document to be processed. (12) The computer of the information processing device, A storage step involves storing in a storage unit as document information, associated with the item to be extracted, the item name and item content corresponding to the item to be extracted, and position information indicating the respective locations of the item name and item content, which are extracted from documents processed in the past based on the item to be extracted. A search step of searching for document information of previously processed documents corresponding to the document to be processed from among the document information stored in the storage unit, based on a string of characters contained in the document to be processed and location information indicating the location of said string, A computer-readable recording medium characterized by recording a program that causes a program to perform an information extraction step in which a prompt containing the document information obtained by a search is input to a large-scale language model, inference is performed by the large-scale language model, and the item name and item content corresponding to the item to be extracted are extracted from the document to be processed. [Explanation of Symbols]

[0057] 100 Information Processing Devices 101 CPU 102 ROM 103 RAM 104 Auxiliary storage 105 Output device 106 Input device 107 Network Interface 201 Control Unit 202 Input / Output Control Unit 203 Storage section 204 Acquisition Department 205 Search Section 206 Instruction generation section 207 First Information Extraction Unit 208 Correction section 209 Second Information Extraction Unit

Claims

1. A storage means that stores in a storage unit as document information, associated with the item to be extracted, the item name and item content corresponding to the item to be extracted, and position information indicating the respective locations of the item name and item content, extracted from documents processed in the past based on the item to be extracted. A search means that searches for document information of previously processed documents corresponding to the document to be processed from among the document information stored in the storage unit, based on a string of characters contained in the document to be processed and location information indicating the location of said string, An information processing apparatus characterized by having an information extraction means that inputs a prompt containing the document information obtained by search into a large-scale language model, performs inference by the large-scale language model, and extracts the item name and item content corresponding to the item to be extracted from the document to be processed.

2. The information processing apparatus according to claim 1, wherein the search means compares the item name and location information of the item name contained in the document information stored in the storage unit with the string and location information of the string contained in the document to be processed, and searches for the document information of the document corresponding to the document to be processed.

3. The aforementioned information extraction means is If the item to be extracted is an item that may contain multiple item contents, the large-scale language model is used to extract the item name and item contents corresponding to the item to be extracted from the document to be processed. If the item to be extracted is an item that contains only one item content, the information processing device according to claim 1 is characterized in that it extracts the item name and item content corresponding to the item to be extracted from the document to be processed based on the location information contained in the document information obtained by the search and the location information in the document to be processed.

4. The information extraction means, if the item to be extracted is an item that contains only one item content, performs a comparison using the location information contained in the document information obtained by the search and the location information in the document to be processed, and extracts the string located at the position corresponding to the location information contained in the document information from the document to be processed as the item name and item content corresponding to the item to be extracted, as described in claim 3.

5. The information processing apparatus according to claim 1, characterized in that it has a correction means for receiving correction input for the extracted document information and correcting the document information in accordance with the correction input.

6. The information processing device according to claim 1, characterized in that the document information is stored in a list structure in which the content of items corresponding to the items to be extracted can be associated with a single item.

7. The information processing apparatus according to claim 4, characterized in that the information extraction means extracts strings that exist within a rectangle based on the position information contained in the document information obtained by searching, where the center point of the rectangle calculated based on the position information of the strings contained in the document to be processed is the corresponding string.

8. The information processing apparatus according to claim 4, wherein the information extraction means extracts strings as strings present at the corresponding positions, wherein the distance between the center point of a rectangle calculated based on the positional information of strings contained in the document to be processed and the center point of a rectangle calculated based on the positional information contained in the document information obtained by searching is less than or equal to a predetermined threshold.

9. The information processing apparatus according to claim 4, wherein the information extraction means extracts strings as strings present at the corresponding positions, wherein the degree of overlap between a rectangle based on the positional information of a string contained in the document to be processed and a rectangle based on the positional information of the document information obtained by searching is greater than or equal to a predetermined threshold.

10. An information processing method performed by an information processing device, A storage step involves storing in a storage unit as document information, associated with the item to be extracted, the item name and item content corresponding to the item to be extracted, and position information indicating the respective locations of the item name and item content, which are extracted from documents processed in the past based on the item to be extracted. A search step of searching for document information of previously processed documents corresponding to the document to be processed from among the document information stored in the storage unit, based on a string of characters contained in the document to be processed and location information indicating the location of said string, An information processing method characterized by comprising: an information extraction step of inputting a prompt containing the document information obtained by search into a large-scale language model, performing inference by the large-scale language model, and extracting the item name and item content corresponding to the item to be extracted from the document to be processed.

11. In the computer of the information processing device, A storage step involves storing in a storage unit as document information, associated with the item to be extracted, the item name and item content corresponding to the item to be extracted, and position information indicating the respective locations of the item name and item content, which are extracted from documents processed in the past based on the item to be extracted. A search step of searching for document information of previously processed documents corresponding to the document to be processed from among the document information stored in the storage unit, based on a string of characters contained in the document to be processed and location information indicating the location of said string, A program for inputting a prompt containing the document information obtained through a search into a large-scale language model, performing inference using the large-scale language model, and executing an information extraction step that extracts the item name and item content corresponding to the item to be extracted from the document to be processed.

Images