Information processing device, information processing method, and program

Abstract

The ability to automatically fill in missing information when extracting text information from document images. [Solution] The information processing device comprises: a character information recognition unit that recognizes first character information contained in a document image; a determination unit that determines whether or not there is a gap in the first character information contained in the document image based on the recognition result recognized by the character information recognition unit; and a completion unit that, if the determination unit determines that there is a gap in the first character information, completes the second character information contained in the recognition result.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Generally, when obtaining information from a document image using an information processing apparatus, it is not sufficient to simply extract all the described characters by optical character recognition (OCR). It is necessary to organize and output the information required by the user in a desirable format for the user. In recent years, with the emergence of large language models (LLMs), the accuracy of appropriately summarizing long texts has been significantly improved even without human intervention.

[0003] As an example of a technique for obtaining information from a document image, when character recognition of the name for identifying the financial institution of the payee entered in a transfer slip by OCR, an apparatus that detects abbreviations such as those of financial institutions, refers to a target table of official names and abbreviations to complement the omitted parts, and corrects them to the official names is known (Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When using an information processing device to extract text information from a document image, extracting information faithfully to the written text may result in the loss of information on items the user wants. For example, a menu image is a document image that generally contains a lot of text, including the menu name, the category to which the menu belongs, and a description of the menu. In menu images, as shown in Figure 2, prices and parts of the dish names are often omitted to avoid redundant expressions, making information loss likely. Next to "potato salad," the price is not written, but from the context, it can be inferred that this is a 600 yen menu item. If text information is extracted faithfully to the written text from the menu image shown in Figure 3, the output data will be missing the price of "potato salad." In this case, the user needs to look at the menu image to confirm the information loss and then compare it with the output from the information processing device to fill in the 600 yen price. When extracting text information from document images, there is a need to automatically fill in any missing information.

[0006] The present invention has been made in view of the above points, and provides an information processing device, an information processing method, and a program that can automatically compensate for missing information when acquiring character information from a document image. [Means for solving the problem]

[0007] (1) One aspect of the present invention is an information processing device comprising: a character information recognition unit that recognizes first character information contained in a document image; a determination unit that determines whether or not there is a void in the first character information contained in the document image based on the recognition result recognized by the character information recognition unit; and a completion unit that completes second character information contained in the recognition result if the determination unit determines that there is a void in the first character information. (2) One aspect of the present invention is an information processing apparatus as described in (1) above, further comprising a formatting unit that formats the second character information recognized by the character information recognition unit into an arbitrary format, wherein the determination unit determines whether or not there is an information deficiency in the first character information, which is a deficiency of necessary character information in the format, when the formatting unit formats the second character information, and the completion unit, if the determination unit determines that there is an information deficiency in the first character information, completes the second character information based on the position of the information deficiency in the format and the relationship between the second character information arranged around the information deficiency in the format and the information deficiency. (3) In one aspect of the present invention, in the information processing device described in (1) above, the completion unit completes the second character information using machine learning.

[0008] (4) One aspect of the present invention is an information processing method comprising: a character information recognition step of recognizing first character information contained in a document image; a determination step of determining whether or not there is a void in the first character information contained in the document image based on the recognition result recognized by the character information recognition step; and a completion step of supplementing second character information contained in the recognition result if the determination step determines that there is a void in the first character information.

[0009] (5) One aspect of the present invention is a program that causes a computer to perform a character information recognition step of recognizing first character information contained in a document image; a determination step of determining whether or not there is a void in the first character information contained in the document image based on the recognition result recognized by the character information recognition step; and a completion step of completing second character information contained in the recognition result if the determination step determines that there is a void in the first character information. [Effects of the Invention]

[0010] According to the present invention, when acquiring text information from a document image, missing information can be automatically compensated for. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an example of an overview of information processing by the information processing device 1 according to the first embodiment of the present invention. [Figure 2] This figure shows an example of document image A1 according to the first embodiment of the present invention. [Figure 3] This figure shows an example of OCR result B1 according to the first embodiment of the present invention. [Figure 4] This figure shows an example of the LLM reading result C1 according to the first embodiment of the present invention. [Figure 5] This figure shows an example of the functional configuration of the information processing device 1 according to the first embodiment of the present invention. [Figure 6] This figure shows an example of the flow of the missing data interpolation process according to the first embodiment of the present invention. [Figure 7] This figure shows an example of an output screen D1 according to the first embodiment of the present invention. [Figure 8] This figure shows an example of an overview of information processing by the information processing device 1a according to the second embodiment of the present invention. [Figure 9] This figure shows an example of the output screen D1a according to a second embodiment of the present invention. [Figure 10] This figure shows an example of the functional configuration of the information processing device 1a according to a second embodiment of the present invention. [Figure 11] This figure shows an example of the flow of the highlighting process according to the second embodiment of the present invention. [Figure 12] This figure shows an example of an overview of information processing by the information processing apparatus 1b according to the third embodiment of the present invention. [Figure 13] This figure shows an example of a trimmed image AT1 according to the third embodiment of the present invention. [Figure 14] This figure shows an example of the LLM partial reading result CT1 according to the third embodiment of the present invention. [Figure 15] This figure shows another example of the overview of information processing by the information processing apparatus 1b according to the third embodiment of the present invention. [Figure 16]It is a diagram showing an example of the functional configuration of an information processing apparatus 1b according to a third embodiment of the present invention. [Figure 17] It is a diagram showing an example of the flow of information processing when preprocessing according to a third embodiment of the present invention is performed. [Figure 18] It is a diagram showing an example of the flow of information processing when postprocessing according to a third embodiment of the present invention is performed. [Figure 19] It is a diagram showing an example of the outline of information processing by an information processing apparatus 1c according to a fourth embodiment of the present invention. [Figure 20] It is a diagram showing an example of text format output according to a fourth embodiment of the present invention. [Figure 21] It is a diagram showing an example of image format output according to a fourth embodiment of the present invention. [Figure 22] It is a diagram showing an example of the functional configuration of an information processing apparatus 1c according to a fourth embodiment of the present invention. [Figure 23] It is a diagram showing an example of the flow of region extraction processing according to a fourth embodiment of the present invention. [Figure 24] It is a diagram showing an example of the flow of information processing when region extraction according to a fourth embodiment of the present invention is applied. [Figure 25] It is a diagram showing an example of the outline of information processing by an information processing apparatus 1d according to a fifth embodiment of the present invention. [Figure 26] It is a diagram showing an example of a tool screen P1 according to a fifth embodiment of the present invention. [Figure 27] It is a diagram showing an example of an OCR result B1 according to a fifth embodiment of the present invention. [Figure 28] It is a diagram showing an example of a matching result according to a fifth embodiment of the present invention. [Figure 29] It is a diagram showing an example of a menu registration UI52 when a click is performed according to a fifth embodiment of the present invention. [Figure 30] It is a diagram showing an example of a menu registration UI52 when a range is selected by drag-and-drop according to a fifth embodiment of the present invention. [Figure 31]This figure shows an example of range selection by drag and drop according to the fifth embodiment of the present invention. [Figure 32] This figure shows an example of a tool screen P2 including a menu registration UI according to a modified example of the fifth embodiment of the present invention. [Figure 33] This figure shows an example of the functional configuration of the information processing device 1d according to the fifth embodiment of the present invention. [Figure 34] This figure shows an example of the flow of the reading result display process according to the fifth embodiment of the present invention. [Modes for carrying out the invention]

[0012] (First embodiment) Embodiments of the present invention will be described in detail below with reference to the drawings. Figure 1 is a diagram showing an example of an overview of information processing by the information processing device 1 according to this embodiment. The information processing device 1 acquires character information contained in document image A1 in a format desired by the user. The information processing device 1 has a character recognition function based on optical character recognition (OCR) and a document generation function based on a large language model (LLM).

[0013] Document image A1 is a menu image of a menu served at a restaurant, as shown in Figure 2. While this embodiment uses a restaurant menu for explanation, document image A1 is not limited to restaurant menus; it may be any type of document image, such as a treatment menu at a beauty salon or chiropractic clinic. Document image A1 includes textual information such as the menu name, price, category, and description. Document image A1 is an image captured by user terminal 2. While this embodiment uses an example of capture by user terminal 2, the terminal used for capture may be a scanner or similar device. Document image A1 is also referred to as a scanned image. Document image A1 is uploaded from user terminal 2 to information processing device 1.

[0014] The information processing device 1 obtains character information from document image A1 by applying LLM to the OCR result B1 for document image A1. Alternatively, LLM may be applied directly to document image A1. Here, the OCR result B1 is data consisting of an OCR string B11 and coordinates B12, as shown in Figure 3. The OCR string B11 is a string extracted from document image A1 using OCR. Coordinates B12 are the coordinates of the OCR string B11 in document image A1. Coordinates B12 are represented by the row and column numbers of the string in each grid when document image A1 is divided into a grid. Alternatively, the coordinates of the bounding box of the string may be used as coordinates B12.

[0015] The LLM is prompted with a command to read the text information from the OCR result B1 in the format desired by the user, and then performs inference. An example of such a prompt is, "Based on the OCR result, read the category name, menu name, price excluding tax, price including tax, and description for each menu item." The LLM outputs the LLM reading result C1 as the result of the inference. The LLM reading result C1 contains the read string C11. The read string C11 is a string generated by the LLM from the OCR result B1. In the reading result C1, the read string C11 is structured item by item based on the format desired by the user. An example of this format is the CSV format shown in Figure 4.

[0016] Document image A1 may have the following missing information, as shown in Figure 2. In document image A1, the price is not listed to the right of "potato salad." Among the "menu names," "fried chicken" is not listed to the left of "large." Also, the description "Freshly fried and crispy!" is a description for both the large and small "fried chicken," but only one is listed. These missing pieces of information are the result of intentional omissions to avoid redundant expressions.

[0017] Information processing device 1 automatically completes any missing information originally contained in document image A1 using LLM. As shown in Figure 4, in the LLM reading result C1, the string C11 "600" is completed in the "Tax-inclusive price" field for "Potato salad". The dish "Karaage" has different prices depending on its size (small or large). Therefore, in LLM inference, "Karaage Small" and "Karaage Large" are defined and distinguished as dish names. Accordingly, the string C11 "Karaage" is completed in the "Menu Name" field, such as "Karaage Large". In addition, the string C11 "Freshly fried and crispy!" is completed in the "Description" field, and a description is added to both "Karaage Small" and "Karaage Large", not just one of them.

[0018] Generally, two typical methods for improving the performance of an LLM are prompt engineering, which involves optimizing the prompts, and fine-tuning the model itself. In document images, there are many different ways to omit parts of a string, and it is not easy to describe the method of completing the omissions within the character limit of the prompt. Therefore, the information processing device 1 employs the fine-tuning method.

[0019] To achieve this, training data is created by pairing LLM reading results with data loss and LLM reading results with data completion performed by the user. The information processing device 1 then fine-tunes the LLM using the created training data. In other words, the LLM acquires a generalized completion method through learning. The training data database 3 shown in Figure 1 is the database of this training data.

[0020] Information processing device 1 is, for example, a cloud server. Information processing device 1 may be implemented as a single computer or as a virtual server.

[0021] User terminal 2 is a terminal device used by the user. User terminal 2 is a computer device such as a personal computer, tablet, or smartphone. User terminal 2 comprises a display unit 21, an operation unit 22, and an imaging unit 23. The display unit 21, for example, is equipped with a liquid crystal display and displays various screens. In the following description, the act of the display unit 21 displaying a screen is also referred to as presenting a screen to the user. The control unit 22, for example, is equipped with a touch panel and detects user operations. The imaging unit 23, for example, is equipped with a camera and performs imaging.

[0022] User terminal 2 comprises an arithmetic unit 200 and a storage unit 250. The arithmetic unit 200 includes, for example, a central processing unit (CPU), and operates based on programs and data stored in the storage unit 250, providing various functions. The storage unit 250 is composed of, for example, one or both of a hard disk drive and semiconductor memory (flash memory), RAM (Random Access Memory), ROM (Read Only Memory), etc., and stores various types of information, such as programs and data read by the arithmetic unit 200. The storage unit 250 may also be implemented by a virtual storage device such as a cloud server located outside the user terminal 2.

[0023] The information processing device 1 and the user terminal 2 can communicate with each other via a network. This network may include the internet, a WAN (Wide Area Network), a LAN (Local Area Network), a public network, provider equipment, a dedicated line, a wireless base station, and the like.

[0024] [Functional configuration of the information processing device 1] Figure 5 shows an example of the functional configuration of the information processing device 1 according to this embodiment. The information processing device 1 comprises a control unit 10 and a storage unit 11.

[0025] The control unit 10 performs various processes. The control unit 10 includes, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an FPGA (Field-programmable Gate Array).

[0026] The control unit 10 comprises a document image acquisition unit 100, a character information recognition unit 101, and a generation unit 102. Each of these functional units is implemented, for example, by the CPU reading a program from ROM (Read Only Memory), loading it into RAM (Random Access Memory), and executing processing according to that program. The ROM and RAM are included in the storage unit 11.

[0027] The document image acquisition unit 100 acquires document image A1.

[0028] The character information recognition unit 101 recognizes the string A11 contained in the document image A1. The string A11 is, for example, a string displayed in a predetermined font. The string A11 may also consist of handwritten characters. In this embodiment, the character information recognition unit 101 recognizes the string A11 based on OCR. The OCR result B1 is the recognition result by the character information recognition unit 101. The OCR result B1 consists of the OCR string B11 and the coordinates B12.

[0029] The generation unit 102, as an example, generates an LLM reading result C1 from the OCR result B1 based on the LLM. The generation unit 102 includes a formatting unit 103, a determination unit 104, a completion unit 105, and an output unit 106. The formatting unit 103, determination unit 104, completion unit 105, and output unit 106 are realized when predetermined instructions are input to the LLM and each process is executed using the LLM.

[0030] The formatting unit 103 formats the OCR string B11 recognized by the character information recognition unit 101 into an arbitrary format.

[0031] The determination unit 104 determines whether or not there is missing information in the string A11 contained in the document image A1, based on the OCR result B1 recognized by the character information recognition unit 101. The determination unit 104 determines at least the missing information that originally existed in the string A11 contained in the document image A1.

[0032] If the determination unit 104 determines that there is a missing value in the document image A1, the completion unit 105 completes the OCR string B11 contained in the OCR result B1. The completion unit 105 completes the OCR string B11 contained in the OCR result B1 by completing the read string C11 in the LLM read result C1.

[0033] The output unit 106 outputs the OCR string B11, which has been interpolated by the interpolation unit 105, as the LLM reading result C1.

[0034] The memory unit 11 stores various types of information. The memory unit 11 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. For example, the memory unit 11 stores a trained model L1. The trained model L1 is trained to output output data corresponding to input data. The trained model L1 is an LLM (Low-Level Model). The memory unit 11 may also store instruction information L2. The instruction information L2 is information indicating a predetermined instruction to be input to the LLM.

[0035] Furthermore, each functional unit of the control unit 10 may be distributed across multiple servers. For example, the character information recognition unit 101, the formatting unit 103, the determination unit 104, the completion unit 105, and the output unit 106 may each be located on a different server.

[0036] [Missing data interpolation] Next, we will describe the missing character interpolation process, which is the process by which the information processing device 1 interpolates the missing characters in the string A11. Figure 6 is a diagram showing an example of the flow of the missing character interpolation process according to this embodiment.

[0037] Step S10: The document image acquisition unit 100 acquires document image A1. The user terminal 2 transmits the document image A1 captured by the imaging unit 23 to the information processing device 1. Note that document image A1 may be an image acquired from a web page by the user terminal 2. The document image acquisition unit 100 supplies the acquired document image A1 to the character information recognition unit 101.

[0038] Step S20: The character information recognition unit 101 recognizes the string A11 contained in the document image A1 based on OCR. The character information recognition unit 101 generates the OCR string B11 as the recognition result of string A11. The character information recognition unit 101 also recognizes the positional relationship between strings A11 in the document image A1, along with the strings A11 contained in the document image A1, based on OCR. The character information recognition unit 101 indicates the positional relationship between strings A11 using the coordinates B12 of strings A11 in the document image A1. Note that the representation of the positional relationship between strings A11 is not limited to coordinates B12, as long as it indicates the positional relationship between strings A11. Information indicating the relative positional relationship between strings A11 may be used instead of coordinates B12. The character information recognition unit 101 generates OCR result B1 as a result of the recognition. The character information recognition unit 101 supplies the generated OCR result B1 to the formatting unit 103.

[0039] Step S30: The control unit 10 reads the OCR result B1 based on the LLM. The control unit 10 inputs the OCR result B1 and the instruction information L2 into the trained model L1, which is the LLM.

[0040] The LLM formats the OCR string B11 contained in the OCR result B1 into an arbitrary format based on the instruction information L2 and outputs it. If the LLM determines that there is a missing value in the string A11 contained in the document image A1, it supplements the OCR string B11 contained in the OCR result B1 and formats it into that format.

[0041] The formatting unit 103 formats the OCR string B11 contained in the OCR result B1 into an arbitrary format specified by the instruction information L2. The determination unit 104 determines whether there is any information loss in the string A11 contained in the document image A1, which is the loss of necessary character information in the formatting unit 103 when formatting the OCR string B11. Here, the determination unit 104 determines whether there is any loss in the string A11 contained in the document image A1 based on the content that the string A11 represents in the document image A1 and the positional relationship between the strings A11. The determination unit 104 obtains the positional relationship between the strings A11 from coordinates B12.

[0042] If the determination unit 104 determines that there is missing information in document image A1, the completion unit 105 completes the OCR string B11 based on the position of the missing information in the format and the relationship between the missing information and the OCR string B11 that is positioned around the missing information in the format. The completion unit 105 obtains the position of the missing information in the format and the relationship between the missing information and the OCR string B11 that is positioned around the missing information in the format from the OCR string B11 and coordinates B12 included in the OCR result B1.

[0043] An arbitrary format specified by instruction information L2 is, as an example, the CSV format shown in Figure 4. This format has columns for the following items: <Category name to which the menu belongs>, <Menu name>, <Price excluding tax>, <Price including tax>, and <Description>. This format is a two-dimensional tabular form consisting of rows and columns in which <Menu name to which the menu belongs>, <Price excluding tax>, <Price including tax>, and <Description> are stored.

[0044] An example of a prompt indicated by instruction information L2 for outputting the above format to LLM is, as mentioned above, "Read the category name, menu name, price excluding tax, price including tax, and description for each menu based on the OCR results." Note that this prompt is not limited to the text described above, as long as the user can specify their desired format. Instruction information L2 specifies the user's desired format based on the user's instructions from the user terminal 2. Note that instruction information L2 may be pre-stored in the storage unit 11 as a predetermined instruction. Instruction information L2 may also be changed based on the user's instructions from the user terminal 2 if it is a pre-stored instruction.

[0045] The completion unit 105 completes the OCR string B11 contained in the OCR result B1 by completing the read string C11 in the LLM read result C1.

[0046] Step S40: The output unit 106 outputs the LLM reading result C1. The LLM reading result C1 contains the read string C11. The read string C11 contains the string obtained by interpolating the OCR string B11 by the interpolation unit 105.

[0047] In this LLM, the trained model L1 acquires a generalized interpolation method through fine-tuning using training data. Therefore, when the trained model L1 receives the OCR result B1 as input, it outputs the LLM reading result C1 in the format indicated by the instruction information L2, with missing information interpolated based on the instruction information L2. The aforementioned training data database 3 is used as the training data. The control unit 10 performs processing based on the trained model L1 to realize the functions of the formatting unit 103, the determination unit 104, the interpolation unit 105, and the output unit 106.

[0048] In this embodiment, the original missing information in the string A11 contained in document image A1 is compensated for by LLM, specifically the missing information resulting from the omission of common information within the same item. Missing information resulting from omission between items occurs when multiple first items are associated with multiple other second items that indicate similar content, and some of these other second items are omitted.

[0049] If some of the second items are omitted, the remaining second items are associated with multiple first items in common. Even if some of the second items are omitted, the multiple first items and the common second items are arranged in document image A1 in such a way that humans can understand, based on the meaning the items represent and the positional relationships between them, that the multiple first items and the common second items that correspond to those multiple first items are indeed related.

[0050] For example, in document image A1 shown in Figure 2, the price "600 yen each" corresponds to multiple menu items "potato salad" and "chicken cartilage" because the price includes the string "each" (e.g., "600 yen each"), and no price is listed next to "potato salad." Similarly, the menu item "fried chicken" corresponds to multiple prices "600 yen" and "1200 yen" because the price "600 yen each" is listed next to the menu item "small," while no price is listed next to the menu item "large."

[0051] Menu images are an example of data that does not have a fixed structure for its layout, where the text of each item is not necessarily arranged according to a predetermined format (referred to as unstructured data).

[0052] Menus can be presented in both vertical and horizontal formats. In some cases, a single menu may contain a mix of vertical and horizontal text. Furthermore, menus contain a lot of text information because they cover all the dishes, services, or products offered at the restaurant. For example, an invoice only lists text information such as product names and prices. In addition, menus may include category names and descriptions in addition to dish names.

[0053] Menus exhibit diverse layouts and lack a consistent structure. For example, in many menus, the price or description of a dish is not listed in the same row or column as the dish name. Menus often omit a lot of textual information, requiring users to fill in the blanks when reading them. For example, when prices are listed together, such as "2100 yen each," product sizes and other details may be omitted.

[0054] The menus often use unusual fonts, such as handwritten or brushstroke styles, and also contain many difficult characters, including kanji for Chinese and traditional Japanese restaurant dishes, as well as foreign language menus. Therefore, errors are likely to occur when OCR is used to recognize the text information. The menu has a complex hierarchical structure of textual information to be extracted, such as multiple levels of categories. A description where "Course Meals" contains "Course XX," and "Course YY" further contains "Course YY," is one example of this hierarchical structure. Additionally, in menu images where the menu itself has been photographed, objects other than the menu (such as lights, shadows, fingers, or parts of other pages) may be captured during the photography process.

[0055] The information processing device 1 is preferably used when reading strings from unstructured data as described above and arranging the read strings in an arbitrary format.

[0056] In addition, any missing information inherent in document image A1 resulting from the omission of information items may be corrected, as may any omissions or errors in text string A11. For example, if "menu name" is omitted, it may be corrected to its original name. If "menu name" contains an error, it may be corrected by LLM.

[0057] Furthermore, if an error or omission occurs in the OCR string B11 during OCR reading, such error or omission may be corrected by LLM. Omissions that occur during OCR reading are also described as information omissions.

[0058] The output unit 106 outputs the LLM reading result C1 to, for example, the user terminal 2. The display unit 21 of the user terminal 2 displays the LLM reading result C1. Figure 7 shows an example of the output screen D1 displayed on the display unit 21. In the output screen D1, as an example, the LLM reading result C1 and the original document image A1 are displayed side by side. Comparing the LLM reading result C1 and the document image A1, the reading string C11 included in the LLM reading result C1 has been completed. Since the LLM reading result C1 includes categories, the output screen D1 aggregates the "menu names" for each category and displays each category in the LLM reading result C1.

[0059] The output unit 106 may store the LLM reading result C1 in the storage unit 11. Alternatively, the output unit 106 may output the LLM reading result C1 to an external server. In that case, the user terminal 2 may retrieve the LLM reading result C1 from the external server and display it on the display unit 21. With this, the information processing device 1 terminates the missing data interpolation process.

[0060] In this embodiment, an example has been described in which the character information recognition unit 101 recognizes the string A11 contained in the document image A1 based on OCR, but it is not limited to this. The string A11 may also be recognized directly by LLM without going through OCR. In that case, the function of the character information recognition unit 101 is realized by performing character recognition using LLM.

[0061] As described above, the generation unit 102 generates the LLM reading result C1 from the OCR result B1 based on machine learning. The formatting unit 103 performs formatting using machine learning. The judgment unit 104 performs the judgment process using machine learning. The completion unit 105 completes the OCR string B11 using machine learning.

[0062] As described above, the information processing device 1 according to this embodiment comprises a character information recognition unit 101, a determination unit 104, and a completion unit 105. The character information recognition unit 101 recognizes the first character information (in this embodiment, the string A11) contained in the document image A1. The determination unit 104 determines whether or not there is a missing character in the first character information (string A11 in this embodiment) contained in the document image A1, based on the recognition result (OCR result B1 in this embodiment) recognized by the character information recognition unit 101. If the determination unit 104 determines that there is a gap in the first character information (in this embodiment, string A11), the completion unit 105 completes the second character information (in this embodiment, OCR string B11) included in the recognition result (in this embodiment, OCR result B1).

[0063] With this configuration, if the information processing device 1 according to this embodiment determines that there is a gap in the first character information contained in the document image A1, it can supplement the second character information (OCR string B11 in this embodiment) contained in the recognition result (OCR result B1 in this embodiment), thus enabling automatic supplementation of missing information when acquiring character information from a document image.

[0064] (Second embodiment) A second embodiment of the present invention will be described in detail below with reference to the drawings. As explained in the output screen D1 (Figure 7) of the first embodiment described above, LLM can complete the string based on the OCR result B1. On the other hand, hallucination may occur when LLM completes the string, potentially generating incorrect information.

[0065] The following are specific examples of generating incorrect information: (i) Reading text characters that are not present in document image A1; (b) The string A11 contained in document image A1 cannot be read; (h) In the LLM reading result C1, the order of the read string C11 differs from the order of the string A11 in the document image A1; 2) The linking of read strings C11 in the LLM reading result C1 (for example, the linking of "category name" and "menu name") differs from the linking of strings A11 in document image A1.

[0066] In this case, the user needs to check whether the LLM output is correct and correct any errors. Menu images, in particular, are document images that generally contain a lot of text, such as the category name of the dish, the menu name, the price, and the description. Also, as shown in Figure 2, to avoid redundant expressions, the price or part of the dish name is often omitted, making menu images prone to information loss. Therefore, if a user corrects the output from the LLM, the correction work may be time-consuming.

[0067] Therefore, it is desirable to be able to quickly recognize the completed portion of the string read by LLM. For this reason, this embodiment describes a case in which the information processing device identifies the completed portion of the string included in the LLM reading result and highlights the corresponding portion.

[0068] The information processing device according to this embodiment will be referred to as information processing device 1a. Note that components identical to those in the first embodiment described above are denoted by the same reference numerals, and descriptions of identical components and operations may be omitted.

[0069] Figure 8 shows an example of an overview of information processing by the information processing device 1a according to this embodiment. The information processing device 1a identifies the portion of the OCR string B11 contained in the OCR result B1 that has been completed by LLM through a matching process. In the matching process, the completed portion is identified by associating the OCR string B11 contained in the OCR result B1 with the read string C11 contained in the LLM read result C1 and taking the difference.

[0070] In the matching process, for a given read string C11, the OCR string B11 that is most similar to C11 is determined. To do this, the string similarity between C11 and all OCR strings B11 included in the OCR result B11 is calculated, and the OCR string B11 with the highest string similarity is determined. If there are multiple strings with the highest string similarity, any of them may be selected, or a selection may be made according to a predetermined rule. One example of a predetermined rule is to select the one with the smallest coordinates corresponding to the OCR string. The string similarity is, for example, based on the Jarot-Winkler distance. Other similarity measures may also be used. Before calculating string similarity, one or more normalization processes can be performed on a given read string C11 and a given OCR string B11. Examples of normalization processes include the removal of whitespace and unnecessary symbols, the standardization of hiragana or katakana, the standardization of full-width or half-width characters, or Unicode normalization.

[0071] The OCR string B11 that is determined to be the most similar to a given read string C11 is recorded as a correspondence between the two. Alternatively, the range in document image A1 in which the string A11 corresponding to the determined OCR string B11 is written may be recorded as a correspondence with a given read string C11.

[0072] The above process is repeated for all read strings C11 contained in the LLM read result C1. However, during the repetition of this process, the correspondence is not assigned to OCR strings B11 for which a correspondence has already been recorded. In other words, OCR strings B11 for which a correspondence has already been recorded are excluded from the candidates for the OCR string B11 that is most similar to read string C11.

[0073] As a result of repeatedly performing the above process for all read strings C11, the parts of read string C11 included in the LLM read result C1 that do not have a correspondence assigned to them with OCR string B11 are identified as strings that have been completed by LLM.

[0074] On the other hand, any OCR string B11 that does not have a correspondence assigned to it with the read string C11 may be identified by the LLM as a string that failed to be read.

[0075] In addition, during the matching process, if the relationship between the read string C11 and the OCR string B11 is reversed in the above matching process and repeated for all OCR strings B11, the portion of the OCR string B11 included in the OCR result B1 for which no correspondence has been assigned to the read string C11 may be identified by the LLM as a string that failed to be read. In that case, the portion of the read string C11 for which no correspondence has been assigned to the OCR string B11 may be identified as a string that has been completed by the LLM.

[0076] Furthermore, the method for achieving matching is not limited to matching processing; it may also be achieved by outputting the line number of the OCR result B1 corresponding to the LLM. Specifically, the format specified by instruction information L2 can be specified as <category name to which the menu belongs>, <menu name>, <price excluding tax>, <price including tax>, <description>, and <list of line numbers of the corresponding OCR string>. However, in this case, hallucination may occur when the LLM assigns line numbers, resulting in incorrectly assigned or missing line numbers in the OCR result B1. In other words, if hallucination occurs when the LLM completes the information and generates incorrect information, it may be impossible to identify the location of the error. Therefore, matching may be achieved by combining outputting the number of the OCR result B1 corresponding to the LLM with matching processing.

[0077] In the example output screen D1a shown in Figure 9, if a correspondence is recorded for the menu name "Fried Chicken Small", the correspondences for "Fried Chicken" and "Small" in OCR string B11 will not be assigned redundantly in subsequent processing. Therefore, in the process of determining the OCR string B11 that is most similar to the menu name "Fried Chicken Large", "Fried Chicken" in OCR string B11 will be excluded from the candidates for the OCR string B11 that is most similar to the read string C11 in subsequent processing.

[0078] In the example shown in Figure 9, as a result of the above processing, no correspondence was assigned to "French fries" from the menu name, "fried chicken" from "large fried chicken," and "freshly fried and crispy!" from the description. In other words, "French fries," "fried chicken" from "large fried chicken," and "freshly fried and crispy!" are the read string C11 that was completed by LLM. Note that "French fries" is a string that is not included in document image A1, and therefore read string C11 was incorrectly completed due to hallucination by LLM. The incorrectly completed read string C11 needs to be corrected by the user.

[0079] Alternatively, LLM can be used to output the completion portion simultaneously with the read string C11. In this case, an instruction to output the completion portion will be added to the prompt. As a result, the output from LLM will be in a format such as "Fried Chicken Small:Small:600 Yen:Freshly Fried and Crispy!". By checking the output from LLM itself, the user can identify the completion portion.

[0080] [Functional configuration of information processing device 1a] Figure 10 shows an example of the functional configuration of the information processing device 1a according to this embodiment. The information processing device 1a comprises a control unit 10a and a storage unit 11.

[0081] The control unit 10a comprises a document image acquisition unit 100, a character information recognition unit 101, a generation unit 102, an emphasis unit 207a, and an identification unit 208a. Comparing the control unit 10a according to this embodiment (Figure 10) with the control unit 10 according to the first embodiment (Figure 5), the emphasis unit 207a and the identification unit 208a are different. Here, the functions of the other components (character information recognition unit 101 and generation unit 102) are the same as in the first embodiment.

[0082] If the generation unit 102 determines that the OCR result B1 generated by the character information recognition unit 101 contains one or more errors or omissions, it corrects those omissions and generates the LLM reading result C1 from the OCR result B1. In this case, the character information recognition unit 101 is an example of an acquisition unit that acquires first document information, the OCR result B1 is an example of first document information, and the LLM reading result C1 is an example of second document information. Therefore, if the generation unit 102 determines that the first document information acquired by the acquisition unit contains one or more errors or omissions, it corrects those omissions and generates the second document information from the first document information.

[0083] The emphasis unit 207a adds highlighting to the corrected sections that the generation unit 102 has determined to contain one or more errors or omissions.

[0084] The identification unit 208a identifies the modified portion included in the LLM reading result C1 based on whether or not the OCR string B11 included in the OCR result B1 corresponds to the reading string C11 included in the LLM reading result C1.

[0085] [Highlighting process] Next, we will describe the highlighting process, which is a process in which the information processing device 1a adds highlighting to the modified parts. Figure 11 is a diagram showing an example of the flow of the highlighting process according to this embodiment. Note that the processes in steps S110, S120, and S130 are the same as the processes in steps S10, S20, and S30 in Figure 6, so their explanations are omitted.

[0086] Step S140: The identification unit 208a identifies the modified parts included in the LLM reading result C1. The identification unit 208a identifies the modified parts based on the matching process described above. Therefore, the identification unit 208a identifies the modified parts included in the LLM reading result C1 based on whether or not the OCR string B11 included in the OCR result B1 corresponds to the reading string C11 included in the LLM reading result C1.

[0087] Step S150: The emphasis unit 207a adds highlighting to the corrected areas where the generation unit 102 has determined to contain one or more errors and / or omissions. The emphasis unit 207a adds highlighting to the LLM reading result C1 based on the identification result by the identification unit 208a. The emphasis unit 207a adds highlighting to the corrected areas by making one or more of the font (bold or italic), text color, background color, and the addition of an underline or border different from the areas in the LLM reading result C1 other than the corrected areas.

[0088] Step S160: The output unit 106 outputs the LLM reading result C1. In the output LLM reading result C1, the modified areas are highlighted by making one or more of the following highlights different from the areas that have been modified: font (bold or italic), text color, background color, and the addition of an underline or border. An example of the LLM reading result C1 with added highlighting is shown on the right side of the output screen D1a in Figure 9 above. With this, the information processing device 1a terminates the highlighting process.

[0089] As described above, the corrected areas highlighted by the information processing device 1a are areas where one or more errors and omissions have been corrected by the information processing device 1a. One or more of these errors and omissions may be present in the original document image A1, occur when OCR results B1 are generated from document image A1 by OCR, or occur when LLM reading results C1 are generated from OCR results B1 by LLM.

[0090] Furthermore, if the generation unit 102 determines that the document image A1 acquired by the document image acquisition unit 100 contains one or more errors or omissions, it may correct those locations and generate the LLM reading result C1 from the document image A1. In this case, the document image acquisition unit 100 is an example of an acquisition unit that acquires first document information, the document image A1 is an example of first document information, and the LLM reading result C1 is an example of second document information.

[0091] In this embodiment, an example has been described in which the first document information is document image A1, but the embodiment is not limited to this. The first document information may also be data consisting of a string, such as the OCR result B1.

[0092] As described above, the information processing device 1a according to this embodiment comprises an acquisition unit (in this embodiment, a document image acquisition unit 100), a generation unit 102, and an enhancement unit 207a. The acquisition unit (in this embodiment, the document image acquisition unit 100) acquires the first document information (in this embodiment, the document image A1). If the generation unit 102 determines that the first document information (document image A1 in this embodiment) acquired by the acquisition unit (document image acquisition unit 100 in this embodiment) contains one or more errors and / or omissions, it corrects those locations and generates second document information (LLM reading result C1 in this embodiment) from the first document information (document image A1 in this embodiment). The emphasis unit 207a adds emphasis to the modified portion that the generation unit 102 has modified.

[0093] With this configuration, the information processing device 1a according to this embodiment can acquire necessary information even from document information that does not have a defined format. Furthermore, if the acquired information contains missing information or if it is determined that the acquired information is incorrect considering the context, it can automatically complete or correct the missing or incorrect information, and it can also make it easier for the user to notice the completed or corrected parts. In other words, because the corrected parts can be highlighted, when acquiring information from document information and correcting one or more errors and missing parts to output the information, the corrected parts can be easily noticed by the user.

[0094] (Third embodiment) A third embodiment of the present invention will be described in detail below with reference to the drawings. This embodiment describes a case where one or more of the following are performed: pre-processing of the input document image or post-processing of the reading result. In pre-processing, the document image is cropped on the user interface before it is input to the LLM. In post-processing, the read string included in the LLM reading result generated from the document image is cropped on the user interface. The following explanation will first describe the pre-processing steps, followed by the post-processing steps.

[0095] The information processing device according to this embodiment will be referred to as information processing device 1b. Note that components identical to those in the first or second embodiment described above will be denoted by the same reference numerals, and descriptions of identical components and operations may be omitted.

[0096] Figure 12 is a diagram showing an example of the information processing overview by the information processing device 1b according to this embodiment. This information processing includes preprocessing. The information processing device 1b performs trimming as preprocessing on the document image A1 uploaded from the user terminal 2. The trimmed image AT1 is the image obtained by trimming the document image A1. In the illustrated example, as a result of trimming the document image A1, four trimmed images AT1 are generated: trimmed image AT1-1, trimmed image AT1-2, trimmed image AT1-3, and trimmed image AT1-4.

[0097] The subsequent processes of reading the string A11 using OCR and generating the LLM reading result C1 from the OCR result B1 using LLM are performed for each trimmed image AT1. Here, we will explain using trimmed image AT1-1 as an example. From the trimmed image AT1-1 shown in Figure 13, the LLM reading result C1-1 (also referred to as LLM partial reading result CT1) shown in Figure 14 is generated by LLM via the OCR result. The LLM reading result C1-1 contains the read string C11 that corresponds to the string A11 contained in trimmed image AT1-1 from the entire string A11 contained in document image A1.

[0098] In this embodiment, the trimming process is performed by selecting a rectangular area in the document image A1 displayed on the display unit 21. The user selects the rectangular area from the operation unit 22 of the user terminal 2. For example, the operation to select the rectangular area is performed by drawing a diagonal line by dragging and dropping on the display unit 21 where the document image A1 is displayed. As a result, the color of the rectangular area changes, indicating that the area has been selected as the target for trimming. The pixel values ​​included in this rectangular area are copied from the original document image A1 to generate the trimmed image AT1.

[0099] Here, we will explain the effects of preprocessing. Without preprocessing, the user would need to directly check and correct the read string C11 contained in the LLM read result C1. In this case, if LLM reading is performed on all image areas of document image A1 without cropping associated with the selection of a rectangular area, the user cannot directly determine which part of document image A1 corresponds to the read string C11 (such as "category name" and "menu name") contained in the LLM read result C1. In other words, the user cannot directly grasp the correspondence between the read string C11 contained in the LLM read result C1 and the areas of document image A1.

[0100] Here, we will explain using the LLM reading result C1 shown in Figure 9 above as an example. In the LLM reading result C1, the "Standard" category includes three "menu names" ("Potato Salad," "French Fries," and "Chicken Cartilage"). The LLM reading result C1 itself does not include which areas in document image A1 these "menu names" correspond to.

[0101] The user must verify whether the menu name "French fries" in the read string C11 contained in the LLM read result C1 is a read string C11 that LLM mistakenly generated for hallucination, or whether string A11, which was in a category other than "Standard," was mistakenly included in the "Standard" category. To do this, it is necessary to examine the entire document image A1, not just a specific area of ​​it. In other words, in order to determine which parts of the LLM read result C1 are correct and which parts are incorrect, the user must understand all the read strings C11 contained in the LLM read result C11 and compare them with all the strings A11 contained in document image A1. In other words, in order to determine whether correction or addition is needed to the LLM read result C1, it is necessary to compare all the read strings C11 contained in the LLM read result C11 with all the strings A11 contained in document image A1.

[0102] In this matching process, the user's workload and working time increase as the number of strings A11 in document image A1 increases. For example, suppose a document image A1 is a menu image containing 100 menu names, and after reading the strings, the LLM reading result C1 contains 95 menu name strings C11. In this case, to check whether a particular menu name string A11 in document image A1 has been read, it is necessary to sequentially compare all 95 menu name strings C11 in LLM reading result C1. Since this matching process must be repeated for all name strings A11 in document image A1, the user's workload and working time increase.

[0103] On the other hand, as in this embodiment, if trimming is performed as a preprocessing step and the reading is limited to the area of ​​document image A1 to the "category name" and the "menu name" included in the "category name", the correspondence between the LLM reading result C1 and document image A1 is limited to that range, making the user's matching work easier and shortening the working time.

[0104] For example, consider a menu image in Figure 9 above, document image A1, which contains 100 menu names. Suppose this menu image contains 10 category names, and each category contains 10 menu names as string A11. In this case, first, a rectangular area corresponding to one category name can be selected and cropped, and the reading can be limited to the string A11 contained in that area. Suppose the LLM reading result C1 contains 9 menu names as read string C11. In this case, the matching process only requires searching among the 9 menu names to see if the read string C11 corresponding to a particular menu name in the menu image is included in the LLM reading result C11. In other words, the number of read strings C11 that the user needs to match in the matching process for each menu name in document image A1 has been reduced to approximately one-tenth.

[0105] Furthermore, trimming as a preprocessing step has the following effect, separate from the effects mentioned above: In LLM reading, by trimming the document image A1 in units of selected areas before reading, and then having the LLM read the trimmed image AT1, the amount of information (such as the number of menu names) contained in a single image read by the LLM can be reduced. As a result, the number of strings A11 to which price and category are associated with menu names when the LLM reads is reduced, making it easier for the LLM to analyze. Accordingly, the accuracy of the LLM reading improves. Improved reading accuracy reduces the amount of correction required for the LLM reading result C1, and as a result, the user's creation time is shortened.

[0106] Figure 15 shows another example of the overview of information processing by the information processing device 1b according to this embodiment. This information processing includes post-processing. The information processing shown in Figure 15 is the same as the information processing shown in Figure 8 up to the point where OCR result B1 is generated from document image A1 by OCR, the generated OCR result B1 is input to LLM to generate LLM reading result C1, and highlighting is added to LLM reading result C1.

[0107] In the information processing device 1b, after the LLM reading result C1 is generated, a rectangular area is selected in the document image A1. This allows the device to select the reading string C11 associated with the string A11 contained within the selected rectangular area from the reading string C11 included in the LLM reading result C1. In other words, the LLM reading result C1 is limited to the range corresponding to the selected area in the document image A1.

[0108] Specifically, the OCR result B1 is linked to the LLM reading result C1. This linking allows the coordinates B12 in document image A1 of the string A11 corresponding to the read string C11 to be determined. When a rectangular area is selected in document image A1, the string A11 contained within the selected area is identified using coordinates B12. This limits the LLM reading result C1 to the read string C11 corresponding to the identified string A11. The method for displaying the limited LLM reading result C1 will be described later in the fifth embodiment.

[0109] [Functional configuration of information processing device 1b] Figure 16 shows an example of the functional configuration of the information processing device 1b according to this embodiment. The information processing device 1b comprises a control unit 10b and a storage unit 11.

[0110] The control unit 10b comprises a document image acquisition unit 100, a character information recognition unit 101, a generation unit 102, an emphasis unit 207a, an identification unit 208a, a linking unit 300b, a region designation unit 301b, an operation reception unit 302b, and a reflection unit 303b. Comparing the control unit 10b according to this embodiment (Figure 16) with the control unit 10a according to the second embodiment (Figure 10), the linking unit 300b, region designation unit 301b, operation reception unit 302b, and reflection unit 303b are different. Here, the functions of the other components (document image acquisition unit 100, character information recognition unit 101, generation unit 102, emphasis unit 207a, and identification unit 208a) are the same as in the second embodiment.

[0111] The generation unit 102 analyzes document image A1 containing the string A11 using LLM and generates an LLM reading result C1 in which the string A11 is formatted to an arbitrary format. In one example of this embodiment, similar to the first and second embodiments described above, the generation unit 102 analyzes the OCR result B1 obtained by performing OCR processing on document image A1 and generates an LLM reading result C1.

[0112] The linking unit 300b links the string A11 contained in the document image A1 with the read string C11 contained in the LLM read result C1 generated by the generation unit 102.

[0113] The area designation unit 301b designates a portion of the document image A1. This area is the rectangular area described above. In other words, the area designation unit 301b performs a trimming process on the document image A1.

[0114] The operation reception unit 302b extracts the string A11 contained within the area specified by the area designation unit 301b and the read string C11 contained in the LLM read result C1 that has been linked to the string A11 by the linking unit 300b, and accepts an operation on at least one of the extracted string A11 and read string C11.

[0115] The reflection unit 303b reflects the operations received by the operation reception unit 302b to the read string C11 extracted by the operation reception unit 302b all at once.

[0116] [Pre-processing] Next, we will describe the information processing performed by the information processing device 1b when preprocessing is performed. Figure 17 is a diagram showing an example of the flow of information processing when preprocessing is performed according to this embodiment. Note that the process in step S210 is the same as the process in step S110 in Figure 11, so its explanation will be omitted.

[0117] Step S220: The area selection unit 301b performs a trimming process on the document image A1. The area selection unit 301b generates a trimmed image AT1 as a result of the trimming process. At this point, the user performs an operation to select a rectangular area from the operation unit 22 of the user terminal 2. Based on the operation received by the operation unit 22, the area selection unit 301b specifies a rectangular area of ​​a part of the document image A1. The area selection unit 301b performs trimming on the specified rectangular area.

[0118] Step S230: The character information recognition unit 101 and the generation unit 102 start the process of generating the LLM partial reading result CT1 from each trimmed image AT1. The processes from Step S240 to Step S280 are the same as the processes from Step S110 to Step S160 in Figure 11, except that the target of the processing is the trimmed image AT1 and the LLM partial reading result CT1 is output, so the explanation is omitted.

[0119] Step S290: The generation unit 102 completes the process of generating the LLM portion reading result CT1 from the trimmed image AT1 for each trimmed image AT1. With this, the information processing device 1b terminates the information processing when preprocessing is performed.

[0120] [Post-processing] Next, we will describe the information processing performed by the information processing device 1b when pre-processing is performed. Figure 18 is a diagram showing an example of the information processing flow when post-processing is performed according to this embodiment. Note that the processes from step S310 to step S350 are the same as the processes from step S110 to step S150 in Figure 11, so their explanation will be omitted.

[0121] Step S360: The linking unit 300b links the string A11 contained in the document image A1 with the read string C11 contained in the LLM reading result C1 generated by the generation unit 102. In this embodiment, the linking unit 300b links the read string C11 to the string A11 based on the coordinates B12 contained in the OCR result B1.

[0122] Step S370: The region selection unit 301b selects a portion of the document image A1. The process in step S370 is the same as the process in step S220 in Figure 17, so a detailed explanation is omitted.

[0123] Step S380: The operation reception unit 302b extracts the string A11 contained within the area specified by the area designation unit 301b and the read string C11 contained in the LLM read result C1 that has been linked to the string A11 by the linking unit 300b, and accepts an operation for at least one of the extracted string A11 and read string C11.

[0124] For example, LLM may fail to read string A11. In other words, string A11, which is contained within a specified area of ​​document image A1, may not be included in the read string C11 contained in the LLM read result C1. In that case, the user performs an operation from the operation unit 22 to add the string A11 that failed to be read to the read string C11. The operation reception unit 302b accepts this operation.

[0125] In another example, LLM hallucination may incorrectly generate a read string C11 that is not included in document image A1. In other words, a read string C11 that is not included in the string A11 contained within a specified area of ​​document image A1 may be included in the LLM read result C1. In that case, the operation unit 22 performs an operation to delete the incorrectly generated read string C11 from the LLM read result C1. The operation reception unit 302b receives this operation.

[0126] Step S390: The reflection unit 303b reflects the operations received by the operation reception unit 302b to the read string C11 extracted by the operation reception unit 302b all at once.

[0127] Step S3100: The output unit 106 outputs the LLM partial reading result CT1. The LLM partial reading result CT1 includes the reading string C11 included in the LLM reading result C11 that is associated with the string A11 included in the region of document image A1 specified by the region specification unit 301b. With this, the information processing device 1b terminates the information processing if post-processing is to be performed.

[0128] Furthermore, steps S260 and S270 in the information processing when pre-processing is performed (Figure 17), or steps S340 and S350 in the information processing when post-processing is performed (Figure 18), may be omitted. In other words, the process of identifying the modified parts by LLM and adding highlighting to the identified modified parts may be omitted. In that case, the highlighting unit 207a and the identification unit 208a may be omitted from the control unit 10b.

[0129] Furthermore, the operation of selecting a rectangular area in the document image A1 displayed on the display unit 21 may be performed using an operation other than dragging and dropping to draw a diagonal line. The rectangular area may be selected by tracing the perimeter of the rectangular area with the user's finger on the document image A1 displayed on the display unit 21. The rectangular area may also be selected by touching the four corners of the rectangular area with the user's finger or clicking them with a mouse on the document image A1 displayed on the display unit 21.

[0130] In this embodiment and its modifications, an example has been described in which pre-processing is performed on the input document image A1 and post-processing is performed on the reading result C1, but it is not limited to this. Both pre-processing and post-processing may be performed. That is, after the LLM partial reading result CT1 is generated from the trimmed image AT1 generated by pre-processing, a part of the trimmed image AT1 may be specified by post-processing, and a reading string C11 associated with the string A11 contained in the area of ​​the trimmed image AT1 specified by the area specification unit 301b may be extracted from the LLM partial reading result CT1.

[0131] As described above, the information processing device 1b according to this embodiment comprises a generation unit 102, a linking unit 300b, a region designation unit 301b, an operation reception unit 302b, and a reflection unit 303b. The generation unit 102 analyzes a document image (document image A1 in this embodiment) containing first character information (string A11 in this embodiment) and generates output information (LLM reading result C1 in this embodiment) by formatting the first character information (string A11 in this embodiment) into an arbitrary format. The linking unit 300b links the first character information (in this embodiment, string A11) contained in the document image (in this embodiment, document image A1) with the second character information (in this embodiment, read string C11) contained in the output information (in this embodiment, LLM read result C1) generated by the generation unit 102. The region designation unit 301b designates a portion of the document image (in this embodiment, document image A1). The operation reception unit 302b extracts the first character information (string A11 in this embodiment) contained within the area specified by the area designation unit 301b, and the second character information (read string C11 in this embodiment) contained in the output information (LLM reading result C1 in this embodiment) that is linked to the first character information (string A11 in this embodiment) by the linking unit 300b, and accepts an operation on at least one of the extracted first character information (string A11 in this embodiment) and second character information (read string C11 in this embodiment). The reflection unit 303b reflects the operations received by the operation reception unit 302b to the extracted second character information (in this embodiment, the read string C11) all at once.

[0132] With this configuration, the information processing device 1b according to this embodiment can apply operations to at least one of the first character information contained in a region of a document image and the second character information contained in the output information to the second character information all at once. This reduces the burden on the user in the process of comparing and correcting the output information generated from the document image with the document image itself.

[0133] Here, the document information and the reading results are compared to check for errors or omissions that may occur when extracting text information from a document image. When comparing with the reading results, it is easier to compare by dividing the document image into smaller areas rather than comparing the entire document image. Also, in the process of extracting text information from a document image, dividing the document image into smaller areas tends to improve the accuracy of the extraction compared to comparing the entire document image.

[0134] (Fourth embodiment) A fourth embodiment of the present invention will be described in detail below with reference to the drawings. Document images generally have a layout, and the area is often divided for each paragraph. For example, in document image A1, which is a menu image shown in Figure 2, the menu names are grouped by category, such as "Just in case" and "Standard." In the third embodiment described above, the case in which a region in a document image is selected by the user was explained. In this embodiment, as an example of the application of paragraph-by-paragraph region extraction in a document image, the case in which a region in a document image is automatically extracted based on a multimodal model, assuming the technology described in the third embodiment, will be explained.

[0135] The information processing device according to this embodiment will be referred to as information processing device 1c. Note that components identical to those in the first, second, or third embodiments described above will be denoted by the same reference numerals, and descriptions of identical components and operations may be omitted.

[0136] Figure 19 shows an example of the information processing overview by the information processing device 1c according to this embodiment. The information processing device 1c takes document image A1 as input and extracts regions for each paragraph using a multimodal model. The multimodal model is, as an example, the multimodal model LLM. A prompt for the multimodal model LLM to extract regions is, for example, "Please extract regions for each paragraph based on the image."

[0137] The output from the multimodal model can be in text format, image format, or both text and image format.

[0138] Figure 20 shows an example of text-format output. Text-format output T1 is an example of text-format output. Text-format output T1 consists of reading results for each region. Text-format output T1-1 and text-format output T1-2 are examples of reading results for each region. Note that in the example shown, only reading results for two regions are shown, but reading results for each region are output according to the number of regions.

[0139] The text format output T1 has columns for the following items: <paragraph name>, <menu name>, <price excluding tax>, <price including tax>, and <description>. The text format output T1 is a two-dimensional tabular format consisting of rows and columns, where the <paragraph name> column stores a string summarizing the text information within the area, and each <menu name> also stores <price excluding tax>, <price including tax>, and <description>. Note that the <paragraph name> column may be omitted.

[0140] In the text format output T1-1, regions containing the strings "for now," "edamame," "400 yen," "ume suisho," and "600 yen" are extracted from the document image A1 shown in Figure 2. The strings contained in these regions are classified into a single group. Among the strings contained in these regions, "for now" is determined to be the string corresponding to the group name. The string "for now," which has been determined to be the group name, is stored in the "paragraph name" column.

[0141] These reading results are read, for example, using an LLM or multimodal model provided in the information processing device according to each embodiment described above. When reading using a multimodal model, the image can be directly input, eliminating the need for OCR processing. In other words, document image A1 is directly input into the multimodal model and reading is performed.

[0142] Alternatively, instead of reading the string, the coordinates of the bounding boxes for each region may be output. These bounding box coordinates are obtained by inputting document image A1 into a multimodal model and allowing it to infer the bounding box coordinates.

[0143] Alternatively, the coordinates of the bounding box for a string may be obtained using OCR. The OCR result B1 includes the reading result of the string's coordinates, as shown in coordinate B12. The coordinates of the bounding box for each region may be obtained using the reading result of the string's coordinates.

[0144] Figure 21 shows an example of image format output. Image format output M1 is an image as an example of image format output. In image format output M1, boxes corresponding to each region are overwritten on the original document image A1. In other words, each region is color-coded. Instead of boxes, borders corresponding to each region may be overwritten and displayed. A multimodal model capable of generating images is used for image format output.

[0145] As mentioned above, the output from the multimodal model may be in both text and image formats. In that case, for example, the text format output T1 shown in Figure 20 and the image format output M1 shown in Figure 21 will be output simultaneously.

[0146] To display the extracted regions, the following processing is performed. If the output is in image format, the image is displayed as is. If the output is in text format, for example, an image is displayed in which the regions are drawn on the original document image A1 using the coordinates of the bounding boxes that enclose each region. This image is the same as the image format output M1 shown in Figure 21.

[0147] By extracting regions from document image A1, it becomes easier for users to verify the content indicated by the text contained in document image A1. For example, as shown in Figure 21, color-coding each region makes it easy to see which region in the image contains which category of menu name.

[0148] Furthermore, extracting a region from document image A1 can be applied to information processing by the information processing device 1b according to the third embodiment described above. As described above, by trimming document image A1, the user's burden in the process of comparing the output information generated from the document image with the document image can be reduced. However, in the third embodiment, the region for trimming had to be selected by the user. By providing the user with the region extraction result by the information processing device 1c according to this embodiment, the user's comparison work can be made more efficient. [Functional configuration of information processing device 1c] Figure 22 shows an example of the functional configuration of the information processing device 1c according to this embodiment. The information processing device 1c comprises a control unit 10c and a storage unit 11.

[0149] The control unit 10c comprises a document image acquisition unit 100, a character information recognition unit 101, a generation unit 102c, an emphasis unit 207a, an identification unit 208a, a linking unit 300b, a region designation unit 301b, an operation reception unit 302b, and a reflection unit 303b. Comparing the control unit 10c according to this embodiment (Figure 22) with the control unit 10b according to the third embodiment (Figure 16), the differences are that the generation unit 102c is included and the region designation unit 301b is not included. Here, the functions of the other components (document image acquisition unit 100, character information recognition unit 101, emphasis unit 207a, identification unit 208a, linking unit 300b, operation reception unit 302b, and reflection unit 303b) are the same as in the third embodiment.

[0150] The generation unit 102c comprises a formatting unit 103, a determination unit 104, a completion unit 105, an output unit 106, a region division unit 107c, a grouping unit 108c, and a structured data generation unit 109c.

[0151] The region division unit 107c divides the document image A1 into multiple regions based on at least one of the content, size, and layout of the string A11 contained in the document image A1, or at least one of the content, size, and layout of the image contained in the document image A1.

[0152] The grouping unit 108c extracts the string A11 contained within each region divided by the region division unit 107c and classifies it into the group corresponding to that region.

[0153] The structured data generation unit 109c generates structured data. The structured data is data in which the string A11 included in the group classified by the grouping unit 108c is structured.

[0154] The memory unit 11 stores the trained model L1c. The trained model L1c is a multimodal model (for example, a multimodal LLM).

[0155] [Region extraction process] Next, we will describe the region extraction process, which is the process by which the information processing device 1c extracts a region from the document image A1. Figure 23 is a diagram showing an example of the flow of the region extraction process according to this embodiment. Note that the process in step S410 is the same as the process in step S10 in Figure 6, so the explanation is omitted.

[0156] Step S420: The region division unit 107c divides the document image A1 into multiple regions. Here, the region division unit 107c divides the document image A1 into multiple regions based on at least one of the content, size, and layout of the string A11 contained in the document image A1, or at least one of the content, size, and layout of the image contained in the document image A1. In one example of this embodiment, the region division unit 107c divides the document image A1 into multiple regions based on a multimodal model (e.g., multimodal LLM).

[0157] Step S430: The grouping unit 108c extracts the string A11 contained within each region divided by the region division unit 107c and classifies it into the group corresponding to that region.

[0158] Step S440: The structured data generation unit 109c generates structured data. The structured data generation unit 109c generates structured data by structuring the string A11 included in the group classified by the grouping unit 108c.

[0159] The structured data generation unit 109c may also determine which string A11 corresponds to the group name from among the strings A11 classified into groups by the grouping unit 108c. In that case, the structured data generation unit 109c may generate structured data by making the strings A11 other than the string A11 corresponding to the group name into element data for that group.

[0160] As an example, the structured data generation unit 109c determines the string A11, which corresponds to the group name, when a prompt instructing the LLM to determine the string A11, which corresponds to the group name, is input.

[0161] Furthermore, if the structured data generation unit 109c determines that there is no string A11 corresponding to the group name among the strings A11 classified into groups by the grouping unit 108c, it may infer and complete the group name from the content of the element data included in that group.

[0162] As an example, the structured data generation unit 109c determines the string A11 corresponding to the group name by inputting a prompt to the LLM that instructs it to determine the string A11 corresponding to the group name, and if it determines that the string A11 corresponding to the group name does not exist, instructs it to infer and complete the group name from the content of the element data included in the group.

[0163] Step S450: The output unit 106 outputs the structured data generated by the structured data generation unit 109c. Here, the output unit 106 structures the structured data generated by the structured data generation unit 109c based on the structure of the structured data and outputs it. The output unit 106 outputs the structured data to the user terminal 2. The display unit 21 of the user terminal 2 displays the structured data. Note that the text format output T1 and the image format output M1 described above are examples of structured data output based on the structure of the structured data. With this, the information processing device 1c terminates the region extraction process.

[0164] [Application to pretreatment] Next, we will describe the information processing when region extraction is applied in the information processing by the information processing device 1c. Figure 24 is a diagram showing an example of the information processing flow when region extraction is applied according to this embodiment. Note that steps S510 and steps S530 through S590 are the same as steps S210 and steps S230 through S290 in Figure 17, so their explanation will be omitted.

[0165] Step S520: The region division unit 107c divides the document image A1 into multiple regions. Each of the following processes from step S540 to step S580 is performed for each of the multiple regions into which the document image A1 has been divided by the region division unit 107c. With this, the information processing device 1c terminates the information processing when region extraction is applied.

[0166] Alternatively, instead of preprocessing, automatic region extraction in document image A1 may be applied in postprocessing. In that case, for example, in the information processing when postprocessing is performed (Figure 18), in step S370, the region division unit 107c divides document image A1 into multiple regions.

[0167] As described above, the information processing device 1c according to this embodiment includes a region division unit 107c. The region division unit 107c divides the document image A1 into a plurality of regions based on at least one of the content, size, and layout of the character information (in this embodiment, the string A11) contained in the document image A1, or at least one of the content, size, and layout of the image contained in the document image A1.

[0168] With this configuration, the information processing device 1c according to this embodiment can divide the document image A1 into multiple regions, making it easier to verify the content indicated by the character information contained in the document image A1.

[0169] (Fifth embodiment) A fifth embodiment of the present invention will be described in detail below with reference to the drawings. As mentioned above, the LLM reading result C1 generated from document image A1 using the information processing device may contain errors. Therefore, the user is required to verify whether or not the LLM reading result C1 contains errors.

[0170] In this embodiment, a user interface (UI) is described to support a series of operations, including a matching operation, performed by the user when reading a string from document image A1 using LLM and formatting it into an arbitrary format.

[0171] The information processing device according to this embodiment will be referred to as information processing device 1d. Note that components identical to those in the first, second, third, or fourth embodiments described above will be denoted by the same reference numerals, and descriptions of identical components and operations may be omitted.

[0172] Figure 25 shows an example of an overview of information processing by the information processing device 1d according to this embodiment. Figure 25 describes an example in which the LLM reading result C1 shown in Figure 4 is generated from the document image A1, which is a menu image shown in Figure 2.

[0173] The display unit 21 of the user terminal 2 displays a reading screen (not shown) provided by the information processing device 1d. When the read execution button is pressed on this reading screen, the information processing device 1d starts reading the document image A1. Once the reading is complete, the information processing device 1d outputs the tool screen P1 to the user terminal 2. The tool screen P1 is displayed on the display unit 21 of the user terminal 2. The user operates from the UI displayed on the tool screen P1.

[0174] Figure 26 shows an example of the tool screen P1. The tool screen P1 includes a reading result display UI 51 and a menu registration UI 52. In the tool screen P1, as an example, the reading result display UI 51 and the menu registration UI 52 are displayed side by side. The reading result display UI 51 is displayed on the left side of the tool screen P1. The menu registration UI 52 is displayed on the right side of the tool screen P1. Depending on the shape and size of the display unit 21 screen, the reading result display UI 51 and the menu registration UI 52 may be displayed side by side vertically.

[0175] The reading result display UI 51 is a screen that displays the LLM reading result C1. The reading result display UI 51 displays the LLM reading result C1. In this embodiment, as an example, in the reading result display UI 51, the LLM reading result C1 is displayed by superimposing a predetermined display of the LLM reading result C1 onto the document image A1. In the reading result display UI 51, the string A11 contained in the document image A11 is highlighted in different ways for each category classified based on the content that the string A11 indicates. As an example, this highlighting is done by surrounding the string A11 with a frame of a predetermined color.

[0176] The string A11 classified as a "category name" by LLM is enclosed in a blue rectangular frame. The string A11 classified as a "menu name" by LLM is enclosed in an orange rectangular frame. Note that in Figure 26, the blue and orange frames are indicated by dashed and chained lines, respectively.

[0177] Note that the frame color is just an example, and the frame color may be other than blue or orange. Also, the highlighting may be in a form other than surrounding the string A11 with a frame of a predetermined color. As another example, as highlighting, one or more of the following may be changed: the font (bold or italic), text color, background color, and the addition of an underline or border to the string A11. As highlighting, a predetermined mark may be displayed near the string A11.

[0178] Furthermore, a predetermined display showing the LLM reading result C1 can be hidden. The reading result display UI 51 includes a button to toggle the display of this display on and off. Temporarily hiding the predetermined display showing the LLM reading result C1 makes it easier to compare the menu image (document image A1) with the information registered in the menu registration UI.

[0179] The menu registration UI52 displays the read strings C11 that have been read by LLM and categorized as "Category Name".

[0180] In the reading result display UI51, a frame is displayed around only strings A11 that are associated with the reading string C11 classified as "category name" or "menu name," and where the reading string C11 included in the LLM reading result C1 matches the OCR string B11 included in the OCR result B1. The match between the reading string C11 and the OCR string B11 is determined based on the following matching process. This matching process is the same as the matching process algorithm described in the second embodiment.

[0181] In the matching process, for all the "category names" and "menu names" read by the LLM, it is determined whether or not the list of OCR strings B11 included in the OCR results performed on the menu image contains an OCR string B11 with a similarity of above a threshold.

[0182] The similarity is, for example, based on the Jarot-Winkler distance. A Jarot-Winkler distance similarity score of 0 indicates that the strings are not a perfect match, while a value of 1 indicates that the strings are a perfect match. In this example embodiment, the determination is based on whether the similarity is above a threshold; therefore, even if the strings are not a perfect match, they are considered a match if they are similar. The similarity threshold can be set as appropriate. Furthermore, other similarity measures may be used.

[0183] For example, suppose the OCR result B1 shown in Figure 27 is obtained as the OCR result B1. Suppose the LLM reading result C1 shown in Figure 4 is obtained as the read string C11 classified as "category name" and "menu name" from the LLM reading result C1. Note that in the OCR result B1 shown in Figure 27, coordinate B12 is omitted.

[0184] Figure 28 shows an example of the matching results obtained through the matching process. In this matching result, for each of the read strings C11 of the "category name" and "menu name" contained in the LLM read result C1, the OCR string B11 that was determined to have the highest similarity based on the Jaro-Winkler distance is shown along with the similarity value. In this matching result, the similarity threshold based on the Jaro-Winkler distance is set to 0.9.

[0185] Looking at the similarity values ​​based on the Jaro-Winkler distance, all of the read string C11 except "French fries" were successfully matched. In other words, the read string C11 other than "French fries" was determined to match the OCR string B11. "French fries" is a string that is not included in the original menu image, while the other strings are included in the original menu image. Therefore, it can be seen that the matching process of this embodiment is effective.

[0186] As described above, the information processing device 1d according to this embodiment performs matching between the read string C11 included in the LLM read result C1 and the OCR string B11 included in the OCR result B1 based on their similarity. As a result of the matching, only the read string C11 that is determined to match the OCR string B11 is displayed in the read result display UI 51. This ensures that even if a read string C11 that was incorrectly read by the LLM is included in the LLM read result C1, that read string C11 is not displayed. Therefore, it does not confuse the user. This reduces the workload on the user in verifying whether or not the LLM read result C1 contains errors.

[0187] Next, the menu registration UI 52 will be explained. In the menu registration UI 52, the user selects the "category name" to which the "menu name" will be registered. Then, from the "menu names" displayed in the reading result display UI 51, the user selects the "menu name" to be registered in the "category name". The user selects the "menu name" by performing a predetermined operation on the area enclosed by the orange frame surrounding the "menu name". A predetermined operation is, for example, clicking or selecting a range by drag and drop.

[0188] Figure 29 shows the menu registration UI 52 when a click is made in the area enclosed by the "menu name". In the example shown in Figure 29, "edamame" is selected as the "menu name". The selected "menu name" is added below the previously selected "category name" in the menu registration UI 52. In the example shown in Figure 29, "just for now" is selected as the "category name", and the selected "edamame" is added below "just for now".

[0189] Figure 30 shows the menu registration UI52 when a range is selected by drag and drop in the area surrounding the "menu name". In the example shown in Figure 29, "Edamame" and "Ume Suisho" are selected as the "menu names". In the example shown in Figure 29, "Just for now" is selected as the "category name", and the selected "Edamame" and "Ume Suisho" are added below "Just for now".

[0190] Here, when dragging and dropping a menu name, it is necessary to move the pointing device, such as a mouse, along the direction of the text (the direction of arrow X1), as shown in Figure 31. This movement is used to determine whether the text in the document image A1 being read is written vertically or horizontally.

[0191] When reading restaurant menus using OCR, the "reading order" of each item in the menu is diverse. "Reading order" refers to the intended order or direction in which each item, such as a menu item, is read. For example, in horizontally written menus where menu items are grouped in the same column, there is a tendency for them to be designed to be read from top to bottom. Similarly, in vertically written menus where menu items are grouped in the same line, there is a tendency for them to be designed to be read from right to left. If menu items are listed across multiple lines, the "reading order" includes not only the order within a region but also the order between regions. For example, one might read the menu items in the same column in the above order before moving on to the menu items in the column to the right. Furthermore, there are exceptions, such as when vertical and horizontal writing are mixed, resulting in more complex orders. Thus, when reading menus, both the "direction of the text" and the "reading order" are important, and determining these is considered difficult.

[0192] If either the "text orientation" or the "reading order" is incorrectly determined during the reading process, the menu may not be read correctly. To resolve this issue, either a complex program is required, or the user must specify the document orientation before starting the OCR reading process. Similar issues can arise when using LLM (Language-Likely Memory) for reading.

[0193] Due to the above challenges, when extracting strings from a menu, it is necessary to arrange the strings in the reading result according to the correct "reading order." However, the accuracy of the string arrangement in the reading result can be improved by having the user specify in advance the area where the strings are grouped and information about the "reading order," or by inferring this information from mouse operations.

[0194] Therefore, the information processing device 1d may determine the direction indicated by the user's operation of dragging and dropping the string A11 displayed in the reading result display UI 51 as the "string orientation" and "reading order" of the string A11 in the document image A1, and automatically correct the order of the read strings C11 in the LLM reading result C1 based on the determined "string orientation" and "reading order". By tracing the mouse movement during the drag-and-drop operation, the information processing device 1d can grasp the string orientation in a natural flow of operation without requiring the user to specify the string orientation.

[0195] Similarly, it is known that errors are more likely to occur when reading document images with complex layouts, such as menus with many images, using OCR. This is thought to be due to the difficulty in determining "where to start and end to determine if they belong to the same category." Therefore, using logic similar to that used to determine the orientation of text strings, as described above, the "orientation of text strings" and the "reading order" of text string A11 in document image A1 can be determined by tracking the mouse movement when performing a "range selection" operation using drag and drop. Alternatively, this tracking can be used to determine the area in document image A1 where text string A11 is grouped together.

[0196] Specifically, Figure 31 shows the result of selecting area X2 by dragging and dropping the mouse from the upper left of the area containing the text "edamame" to the lower right of the area containing the text "ume suisho". In the example shown, the order in which the selected area X2 covers the rectangle is "edamame" followed by "ume suisho".

[0197] Therefore, the information processing device 1d may determine the order in which the rectangle surrounding the string A11 displayed in the reading result display UI 51 was covered during range selection as the "string orientation" and "reading order" of the string A11 in document image A1, and register the read string C11 in the determined "string orientation" and "reading order" as the LLM reading result C1. Thus, as long as the user understands the relationship between the order in which the rectangle surrounding the string was covered during range selection and the order in which the string is registered in the menu registration UI 52, the menu can be registered in the menu registration UI 52 in the order intended by the user during range selection, while reducing reading errors.

[0198] As described above, the information processing device 1d can determine the order of the text strings A11 in document image A1 (for example, whether it is vertically or horizontally written) by tracking mouse operations (direction of mouse movement or the order in which text strings are selected by the mouse), thereby improving the reading accuracy that OCR and LLM struggle with. In this tracking, instead of tracking the movement of a pointing device such as a mouse, the movement of the user's finger on a touch panel provided on the operation unit 22 of the user terminal 2 may be tracked.

[0199] The orange frame contains information about the "menu name" it encloses (price excluding tax, price including tax, and description). This information is added to the menu registration UI52 along with the "menu name" when the "menu name" is selected. In other words, in the LLM reading result C1, the read strings C11 are linked to each other based on their content, and when a read string C11 is selected, that read string C11 and the read strings C11 linked to it are registered together in the menu registration UI52.

[0200] The display of the frame surrounding the selected "menu name" may be changed after the "menu name" has been selected. In Figure 29 or Figure 30, as an example, the intensity of the frame's color is increased compared to before the selection. A mark or other indicator showing that the "menu name" has been selected may also be displayed near the selected "menu name".

[0201] The reading result display UI 51 and menu registration UI 52 provided by the information processing device 1d do not display all LLM reading results C1 after LLM reading is complete, and the user does not search for and delete or correct any reading errors. The reading result display UI 51 and menu registration UI 52 do not display the reading string C11 itself that is included in the LLM reading result C11. In the reading result display UI 51 and menu registration UI 52, the reading string C11 to be registered is added to the menu registration UI 52 by being selected using the reading result display UI 51.

[0202] Since LLM accuracy is not perfect, errors such as typos or omissions, as well as errors in the order of menu items, may occur. If there are menu names that are displayed incorrectly in the LLM reading result C1, the user will need to check each time whether that menu name exists in the original menu image, or where in the original menu image that menu name is written, and correct it. However, especially when there are many menu items, manually correcting these name or order errors is extremely difficult and significantly reduces work efficiency.

[0203] In the information processing device 1d according to this embodiment, instead of displaying the LLM reading result C1 as is, the user can select and add the target reading string C11 to be registered in the menu registration UI 52 from the LLM reading result C1, thereby enabling more efficient work compared to the automatic method.

[0204] Furthermore, because there is a limit to the amount of information a person can process in their brain at once, displaying all of the LLM reading results C1 at once would exceed the user's processing capacity and make the work time-consuming. Therefore, limiting the amount of information the user can process to what they can handle leads to increased work efficiency.

[0205] As described above, the information processing device 1d according to this embodiment makes it easier for the user to organize information in their mind, improving work efficiency. Furthermore, adding a "menu name" to the menu registration UI52 can be done with simple operations such as clicking or dragging and dropping to select a range in the reading result display UI51. Also, since the verification of the LLM reading result C1 is a necessary step from the start, adding a "menu name" to the menu registration UI52 also serves as this verification step. For these reasons, adding a "menu name" to the menu registration UI52 requires almost no effort from the user compared to performing the LLM reading result C1 verification step that was originally required.

[0206] Furthermore, the reading result display UI 51 and the menu registration UI 52 allow for the effective incorporation of human knowledge into the LLM reading result C1 without requiring extra effort. Specifically, in these UIs, the user directly specifies the "category name" to which the "menu name" is added, as well as the order in which the "menu names" are added. Therefore, even if an error occurs in the LLM reading result C1 regarding the linking of "category names" and "menu names," or the order in which the "menu names" are arranged, the user can correct it and register the "menu names" with almost no effort. In addition, as mentioned above, by relating the order in which the rectangle surrounding the string is covered during range selection to the order in which the string is registered in the menu registration UI 52, menus can be registered in the menu registration UI 52 in the order intended by the user during range selection, while reducing reading errors.

[0207] Figure 31 shows an example of tool screen P2, including a modified version of the menu registration UI. Tool screen P2 includes a reading result display UI 51 and a menu registration UI 53. The menu registration UI 53 is a table-style UI. In the menu registration UI 53, the menu name is added to the menu registration UI 53 when the area surrounding the menu name is selected in the reading result display UI 51. This selection is performed by clicking or by dragging and dropping the range. Therefore, unlike the menu registration UI 52, the menu registration UI 53 does not require the user to select a category name to which the menu name will be added.

[0208] Furthermore, the "category name," "price," and "description" associated with the "menu name" are also added to the menu registration UI53 and displayed together when the "menu name" is selected. The menu registration UI53 also has editing functions similar to those of spreadsheet software.

[0209] [Functional configuration of information processing device 1d] Figure 32 shows an example of the functional configuration of the information processing device 1d according to this embodiment. The information processing device 1d comprises a control unit 10d and a storage unit 11.

[0210] The control unit 10d comprises a document image acquisition unit 100, a character information recognition unit 101, a generation unit 102d, an emphasis unit 207a, an identification unit 208a, a linking unit 300b, an operation reception unit 302b, and a reflection unit 303b. Comparing the control unit 10d according to this embodiment (Figure 32) with the control unit 10b according to the third embodiment (Figure 16), the generation unit 102d is different. Here, the functions of the other components (document image acquisition unit 100, character information recognition unit 101, emphasis unit 207a, identification unit 208a, linking unit 300b, operation reception unit 302b, and reflection unit 303b) are the same as in the third embodiment.

[0211] The generation unit 102d includes a formatting unit 103, a determination unit 104, a completion unit 105, an output unit 106, a classification unit 400d, and a highlighting unit 401d.

[0212] The classification unit 400d classifies the string A11 recognized by the character information recognition unit 101 into categories based on the content that the string A11 represents.

[0213] The highlighting unit 401d highlights the string A11 in a different manner for each category classified by the classification unit 400d.

[0214] The operation reception unit 302d receives operations on at least one of the string A11 contained in the document image A1 and the read string C11 contained in the LLM reading result C1 that has been linked to the string A11 by the linking unit 300b.

[0215] [Processing to display reading results] Next, we will describe the reading result display process, which is the process by which the information processing device 1d displays the LLM reading result C1. Figure 33 is a diagram showing an example of the flow of the reading result display process according to this embodiment. Note that the processes from step S610 to step S650 are the same as the processes from step S310 to step S340 and the process in step S360 in Figure 18, so their explanation will be omitted.

[0216] Step S660: The classification unit 400d classifies the string A11 recognized by the character information recognition unit 101 into categories based on the content represented by string A11. Here, in step S650, the linking unit 300b has linked string A11 to the read string C11. The classification unit 400d classifies the read string C11 included in the LLM reading result C1 into categories based on the content represented by said read string C11. In this way, the classification unit 400d classifies the string A11 linked to read string C11 into categories based on the content represented by the read string C11 linked to string A11.

[0217] Furthermore, the classification unit 400d classifies string A11 into category names and element data. Here, the classification unit 400d classifies string A11 into category names and element data by classifying the read string C11 associated with string A11 into category names and element data. Element data is string A11 other than the "category name". Element data includes "menu name".

[0218] Step S670: The highlighting unit 401d highlights the string A11 in a different manner for each category classified by the classification unit 400d. Since the string A11 and the read string C11 are linked, highlighting the string A11 is equivalent to highlighting the read string C11.

[0219] Here, the highlighting unit 401d highlights the string A11 by surrounding the string A11 classified as a "category name" with a frame of a predetermined color (for example, blue), and the string A11 classified as a "menu name" with a frame of a different color (for example, orange). The blue and orange frames shown in the reading result display UI 51 (Figure 26) described above are examples of highlighting by the highlighting unit 401d.

[0220] Therefore, the highlighting unit 401d highlights the string A11 in the document image A1 by applying a first highlight to the string A11 classified as a "category name" and a second highlight, different from the first highlight, to the string A11 classified as element data. This configuration makes it easy for the user to distinguish between "category names" and element data.

[0221] Furthermore, the highlighting unit 401d may highlight string A11 that is not classified as either a "category name" or element data by applying a third highlight that is different from both the first and second highlights. For example, the highlighting unit 401d may surround at least one of the strings A11 that is either "price" or "description" and is not classified as either a "category name" or "menu name" with a green frame. This configuration makes it easier for users to distinguish between strings A11 that are not classified as either a "category name" or "menu name".

[0222] Step S680: The operation reception unit 302b accepts an operation to select element data. As an example of such an operation, the operation reception unit 302b accepts a predetermined operation (click or drag-and-drop) on the area enclosed by the orange frame surrounding the "menu name". This operation is performed by the user on the reading result display UI 51 displayed on the display unit 21 of the user terminal 2. Furthermore, this operation may also be performed on the string A11, which indicates the "menu name" contained in the document image A1 displayed in the reading result display UI51.

[0223] The operation reception unit 302b stores the element data selected based on the received operation in the output information. The element data added to the output information is displayed, for example, in the menu registration UI 52. The output information also stores the read string C11 of the "category name". In the output information, the read strings C11 of both the "category name" and the "menu name" are formatted in any format specified by the user.

[0224] Step S690: The output unit 106 outputs output information. For example, the output unit 106 outputs output information to the user terminal 2. The display unit 21 of the user terminal 2 displays the output information in the menu registration UI 52. The output unit 106 may also store the output information in the storage unit 11. Alternatively, the output unit 106 may output the output information to an external server. In that case, the user terminal 2 may obtain the output information from the external server and display it on the display unit 21. With this, the information processing device 1d terminates the reading result display process.

[0225] Furthermore, the process of extracting regions of document image A1 using a multimodal model, as described in the fourth embodiment, may be combined with the reading result display process in this embodiment. In that case, the generation unit 102d of the information processing device 1d may include a region division unit 107c, a grouping unit 108c, and a structured data generation unit 109c.

[0226] As described above, the information processing device 1d according to this embodiment comprises a character information recognition unit 101, a classification unit 400d, and a highlighting unit 401d. The character information recognition unit 101 recognizes the character information (in this embodiment, the character string A11) contained in the document image A1. The classification unit 400d classifies the character information (string A11 in this embodiment) recognized by the character information recognition unit 101 into categories ("category name" in this embodiment) based on the content indicated by the character information (string A11 in this embodiment). The highlighting unit 401d highlights the text information (in this embodiment, the string A11) in a different manner (in this embodiment, by enclosing it in a frame of a predetermined color) for each category classified by the classification unit 400d.

[0227] With this configuration, the information processing device 1d according to this embodiment highlights character information (string A11 in this embodiment) in different ways for each category, thereby assisting the user in understanding the content of the character information when displaying the character information contained in the document image A1.

[0228] In each embodiment, the case where document image A1 is a menu image for a restaurant was described as an example, but the invention is not limited to this. As mentioned above, document image A1 may be a menu image provided in a business other than the food and beverage industry. Due to the nature of price lists, menu images from other businesses may also have similar information omissions as menu images for restaurants. In other words, by training a completion method with document images provided in a specific business, the accuracy of information acquisition for document images provided in other businesses can be improved.

[0229] In each embodiment, an example of the use of LLM as the machine learning model has been described, but it is not limited to this. Other machine learning models besides LLM may be used. With machine learning models other than LLM, training is performed according to the type of document image A1 to create a trained model. Various tables may be used instead of machine learning. Various tables may be used in combination with machine learning. As various tables, tables corresponding to the type of document image A1 are prepared. In order to ensure generality regardless of the type of document image A1, it is preferable to use LLM.

[0230] Although embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments, and design modifications and the like are also included within the scope of the gist of the present invention. For example, a computer program to implement the functions of each of the above-mentioned devices may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed. The term "computer system" here may include hardware such as an operating system and peripheral devices.

[0231] Furthermore, "computer-readable recording media" refers to writable non-volatile memory such as flexible disks, magneto-optical disks, ROMs, and flash memory, portable media such as DVDs (Digital Versatile Discs), and storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording media" also includes volatile memory (such as DRAM (Dynamic Random Access Memory)) within computer systems that act as servers or clients when programs are transmitted via networks such as the Internet or communication lines such as telephone lines, which retain programs for a certain period of time.

[0232] Furthermore, the above program may be transmitted from a computer system that stores the program in a memory device or the like to another computer system via a transmission medium or by transmission waves within the transmission medium. Here, the "transmission medium" used to transmit the program refers to a medium that has the function of transmitting information, such as a network (communication network) like the Internet or a communication line (communication line) like a telephone line. Furthermore, the above program may be intended to implement some of the functions described above. It may also be a so-called differential file (differential program) that can implement the aforementioned functions in combination with programs already recorded in the computer system. [Explanation of symbols]

[0233] 1...Information processing device, 101...Character information recognition unit, 104...Determination unit, 105...Completion unit, A1...Document image, A11...String, B1...OCR result, B11...OCR string

Claims

1. A character information recognition unit that recognizes the first character information contained in the document image, A determination unit determines whether or not there is a defect in the first character information contained in the document image, based on the recognition result recognized by the character information recognition unit, If the determination unit determines that the first character information has a deficiency, the completion unit completes the second character information included in the recognition result, An information processing device equipped with the following features.

2. The system further comprises a formatting unit that formats the second character information recognized by the character information recognition unit into an arbitrary format, The determination unit determines whether or not there is an information deficiency in the first character information, which is a deficiency of character information necessary in the format, when the formatting unit formats the second character information. If the determination unit determines that the missing information is in the first character information, the completion unit completes the second character information based on the position of the missing information in the format and the relationship between the second character information arranged around the missing information in the format and the missing information. The information processing apparatus according to claim 1.

3. The completion unit completes the second character information using machine learning. The information processing apparatus according to claim 1 or claim 2.

4. A character information recognition step that recognizes the first character information contained in the document image, A determination step, based on the recognition result recognized by the character information recognition step, determines whether or not there is a missing character in the first character information contained in the document image. If the determination step determines that the first character information has the missing information, the completion step is to complete the second character information included in the recognition result, An information processing method having

5. On the computer, A character information recognition step that recognizes the first character information contained in the document image, A determination step, based on the recognition result recognized by the character information recognition step, determines whether or not there is a missing character in the first character information contained in the document image. If the determination step determines that the first character information has the missing information, the completion step is to complete the second character information included in the recognition result, A program to execute.

Images