Information processing device and program

Abstract

To provide an information processing device and a program configured to associate attribute values for multiple items with each other even if a table of a form image has an irregular layout.SOLUTION: In a terminal device 10 serving as an information processing device, a KV extraction unit 33 performs a character recognition process on a form image containing data in a table format, on the basis of a result of an analysis of a table by a table analysis unit 32, thereby executing KV extraction to acquire character strings of item names representing items of the data in a table format and character strings of attribute values for the items. An extraction result changing unit 34 changes, in a case where a correspondence relationship between the attribute values for multiple items is undetermined in the KV extraction by a table KV extraction unit 33, the correspondence relationship between the attribute values for the multiple items by using a height in the form image of the character string of the attribute value for which the correspondence relationship with the attribute value of another item is undetermined.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an information processing device and a program. [Background technology]

[0002] Patent Document 1 discloses a form recognition device that detects an area containing a character string from a form image, detects the character string within the detected area, recognizes the characters of the detected character string using a recognition dictionary, compares the character recognition results of the character string with item name words in an item name word registration dictionary, determines a character string that is successfully matched as an item name character string, and determines a character string that is not successfully matched as a data character string, and determines the attributes of the data character string from the positional relationship between the item name character string and the data character string. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4996940 Summary of the Invention [Problem to be solved by the invention]

[0004] When performing character recognition processing on a form image containing tabular data to obtain the character strings of the item names representing the items in the tabular data and the character strings of the attribute values ​​for those items, the table may have an irregular layout, such as no ruled lines or including explanatory text within the table. In such cases, the correspondence between the acquired attribute value items may not be one-to-one, and the correspondence may not be able to be determined.

[0005] The object of the present invention is to provide an information processing device and program that can perform character recognition processing on a form image containing tabular data, thereby obtaining strings of item names representing items of tabular data and strings of attribute values ​​for those items, and can associate attribute values ​​between multiple items even when the table layout is irregular. [Means for solving the problem]

[0006] An information processing device according to a first aspect of the present invention includes a processor, and the processor performs character recognition processing on a form image including tabular data to obtain character strings of item names representing items of the tabular data and character strings of attribute values ​​for the items; When the correspondence between attribute values ​​in multiple items cannot be determined, the correspondence between attribute values ​​in multiple items is changed using the height in the form image of the character string of the attribute value whose correspondence with the attribute value in other items cannot be determined.

[0007] A second aspect of the information processing device of the present invention is the information processing device of the first aspect, wherein the processor, when the height of the character string of an attribute value of an item is an integer multiple of at least twice the height of the attribute value of another item, changes the correspondence between attribute values ​​among multiple items by dividing the character string of the attribute value of an item in the height direction.

[0008] A third aspect of the information processing device of the present invention is the information processing device of the second aspect, wherein the processor divides the string of an attribute value of a certain item in the vertical direction so that the height of the string of the attribute value of the certain item after division is approximately the same as the height of the string of the attribute value of another item.

[0009] A fourth aspect of the information processing device of the present invention is the information processing device of the second aspect, wherein when the processor divides the character string of an attribute value of a certain item based on the height of the character string of the attribute value of another item, and the character string cannot be matched one-to-one with the character string of the attribute value of another item, the processor divides the character string of the attribute value of a certain item in the height direction to match the vertical position of the top end of each character string of the attribute value of another item.

[0010] An information processing device of a fifth aspect of the present invention is an information processing device of any one of the first to fourth aspects, wherein the processor changes the correspondence between attribute values ​​between multiple items by adding a string indicating that no corresponding attribute value exists for an item that does not have a corresponding attribute value for that item.

[0011] An information processing device of a sixth aspect of the present invention is an information processing device of any one of the first to fourth aspects, wherein the processor changes the correspondence between attribute values ​​among multiple items by deleting an attribute value in an item when an attribute value corresponding to an attribute value in the item does not exist in any of the other items.

[0012] An information processing device according to a seventh aspect of the present invention is the information processing device according to the sixth aspect, wherein the processor notifies that the attribute value has been deleted.

[0013] A program according to an eighth aspect of the present invention includes a step of performing character recognition processing on a form image including tabular data to obtain character strings of item names representing items of the tabular data and character strings of attribute values ​​of the items; If the correspondence between attribute values ​​in multiple items cannot be determined, the computer executes a step of changing the correspondence between attribute values ​​in multiple items using the height in the form image of a character string of an attribute value whose correspondence with attribute values ​​in other items cannot be determined. [Effects of the Invention]

[0014] According to the information processing device of the first aspect of the present invention, by performing character recognition processing on a form image containing tabular data, when obtaining strings of item names representing items of tabular data and strings of attribute values ​​for those items, it is possible to associate attribute values ​​between multiple items even when the table layout is irregular.

[0015] According to the information processing device of the second aspect of the present invention, by performing character recognition processing on a form image containing tabular data, when obtaining a string of an item name representing an item of tabular data and a string of attribute values ​​for that item, even if multiple attribute value strings for a certain item are combined and obtained as a single string, it is possible to associate attribute values ​​between multiple items.

[0016] According to the information processing device of the third aspect of the present invention, by performing character recognition processing on a form image containing tabular data, when obtaining a string of an item name representing an item of tabular data and a string of attribute values ​​for that item, even if multiple attribute value strings for a certain item are combined and obtained as a single string, it is possible to associate attribute values ​​between multiple items.

[0017] According to the information processing device of the fourth aspect of the present invention, by performing character recognition processing on a form image containing tabular data, when obtaining a string of an item name representing an item of tabular data and a string of attribute values ​​for that item, even if multiple attribute value strings for a certain item are combined and obtained as a single string, it is possible to associate attribute values ​​between multiple items.

[0018] According to the information processing device of the fifth aspect of the present invention, by performing character recognition processing on a form image containing tabular data, when obtaining the item name string representing the item of the tabular data and the attribute value string for that item, it is possible to associate attribute values ​​between multiple items even if there is an item for which there is no attribute value string corresponding to the attribute value string of another item.

[0019] According to the information processing device of the sixth aspect of the present invention, by performing character recognition processing on a form image containing tabular data, when obtaining the item name string representing the item of the tabular data and the attribute value string for that item, it is possible to associate attribute values ​​between multiple items even if there is an item for which there is no attribute value string corresponding to the attribute value string of another item.

[0020] According to the information processing device of the seventh aspect of the present invention, it is possible to prevent occurrence of a situation in which a character string is deleted without the user's knowledge.

[0021] According to the program of the eighth aspect of the present invention, by performing character recognition processing on a form image containing tabular data, when obtaining the string of item names representing the items of tabular data and the string of attribute values ​​for those items, it is possible to associate attribute values ​​between multiple items even if the table layout is irregular. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a diagram illustrating a system configuration of an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 10 is a diagram illustrating an example of a document image including data in a table format. [Figure 3] 3 is a diagram showing an example of the result of KV extraction performed on the form image of the purchase order shown in FIG. 2. FIG. [Figure 4] FIG. 10 is a diagram showing an example of a form image in which it is difficult to associate attribute values ​​with each other by simply performing KV extraction. [Figure 5] FIG. 5 is a diagram showing a state in which normal KV extraction is performed on a table included in the form image shown in FIG. [Figure 6] FIG. 10 is a diagram illustrating a case where three attribute values ​​are combined and extracted as a character string of one attribute value. [Figure 7]1 is a diagram illustrating a hardware configuration of a terminal device 10 in an information processing system according to an embodiment of the present invention. [Figure 8] 1 is a block diagram showing a functional configuration of a terminal device 10 in an information processing system according to an embodiment of the present invention. [Figure 9] 10 is a flowchart showing the overall operation of the terminal device 10 when performing KV extraction. [Figure 10] 10 is a flowchart for explaining details of the correction process of the KV extraction result in step S104 shown in the flowchart of FIG. 9. [Figure 11] FIG. 10 is a diagram showing an example of adding a null character string as described in step S207. [Figure 12] FIG. 10 is a diagram showing an example of a KV extraction result when a character string for which no corresponding character string exists is deleted. [Figure 13] FIG. 10 is a diagram showing an example of dividing a character string of an attribute value described in step S206. [Figure 14] 10A and 10B are diagrams showing examples of the positions and sizes of character strings of item names and attribute values ​​extracted by the in-table KV extraction unit 33. FIG. [Figure 15] FIG. 10 is a diagram for explaining how the height of the character string in the item "product name" is compared with the height of the character string in the items "quantity" and "unit price." [Figure 16] This is a diagram for explaining how the character string in the item "product name" is divided into three parts in the vertical direction to form three character strings each 20 pixels high. [Figure 17] FIG. 10 is a diagram illustrating how a character string in the item "product name" is divided into two character strings at the top two-thirds of the way. [Figure 18] FIG. 10 is a diagram illustrating how a character string in the item "product name" is divided into two character strings at the top third of the string. DETAILED DESCRIPTION OF THE INVENTION

[0023] Next, an embodiment of the present invention will be described in detail with reference to the drawings.

[0024] FIG. 1 is a diagram showing the system configuration of an information processing system according to an embodiment of the present invention.

[0025] 1, an information processing system according to one embodiment of the present invention is configured with an image forming device 20 and a terminal device 10, which are interconnected via a network 30. The image forming device 20 is a so-called multifunction device that has multiple functions such as printing, scanning, copying, and facsimile functions.

[0026] The terminal device 10 is an information processing device having a function of accepting image data scanned by the image forming device 20, and performing OCR (Optical Character Recognition) processing on the accepted image data to convert it into character data.

[0027] However, the image data to be subjected to OCR processing may include not only document image data consisting of characters but also tabular data consisting of multiple rows and multiple columns.

[0028] For example, an example of a document image containing such tabular data is shown in FIG.

[0029] The example document image shown in Figure 2 is an example of image data obtained by scanning a purchase order form, and it can be seen that this document image contains tabular data consisting of the product name, quantity, unit price, and amount of the product to be ordered.

[0030] In the case of a form such as this purchase order, the top row of the table generally contains a string of item names (keys) that indicate the attributes of the attribute values ​​(values) contained in the rows below. Then, multiple attribute values ​​are listed below this item name. Generally, there are multiple attribute values ​​for one item name.

[0031] For example, in the table in the center of the purchase order image shown in Figure 2, the four item names "Item Name," "Quantity," "Unit Price," and "Amount" are written in the top row, and below each item name, a string of characters is written as an attribute value.

[0032] For example, below the item name "Product Name," four character strings, "LCD TV," "MP3 player," "monitor," and "installation cost," are listed as attribute values.

[0033] For example, below the item name "quantity," four character strings, "1," "2," "3," and "1 set," are written as attribute values.

[0034] Similarly, under the item name "Unit Price" four character strings "200,000", "10,000", "5,000", and "1,000" are written as attribute values, and under the item name "Amount" four character strings "200,000", "20,000", "15,000", and "1,000" are written as attribute values.

[0035] The process of performing character recognition on a form image containing such tabular data to obtain the string of the item name (key) representing the item of the tabular data and the string of the attribute value (value) for that item is called KV (Key Value) extraction.

[0036] Figure 3 shows an example of the results of KV extraction performed on the image of the purchase order form shown in Figure 2. Referring to Figure 3, we can see that four character strings have been acquired as attribute values ​​for each of the item name character strings: "No.", "Item name", "Quantity", "Unit price", and "Amount".

[0037] By extracting KVs in this way, character strings of item names and attribute values ​​are converted into character data by character recognition processing, and can then be used as data.

[0038] Therefore, in the data obtained as a result of KV extraction, the character strings of each attribute value on the same line must correspond. For example, in the KV extraction results shown in Figure 3, the attribute values ​​of each item name, "Monitor," "3 units," "5,000," and "15,000," must correspond as the third attribute value, respectively.

[0039] However, depending on the layout of the table in the form image, the number of character strings of the attribute values ​​obtained for each item name may differ, making it impossible to determine which attribute values ​​should be associated with each other as being in the same row.

[0040] For example, Figure 4 shows an example of a form image where it is difficult to associate attribute values ​​with each other simply by extracting KVs.

[0041] In the table included in the form image shown in Figure 4, the first line of the "Item Name" column contains the explanatory string "XX Business Support Costs (July to September 2020)," which causes the number of attribute values ​​for "Item Name" to differ from those for "Quantity" and "Unit Price." Note that this type of supplementary explanatory information is often written in columns with item names such as "Item Name."

[0042] Figure 5 shows how normal KV extraction works when using a table in this format.

[0043] In the table included in the report image shown in Figure 4, as shown in Figure 5, the string corresponding to the attribute value string "XX business support costs (July to September 2020)" written in the first row of the "Product name" column does not exist as an attribute value for "Quantity" or "Unit price."

[0044] As a result, the number of attribute values ​​for each item name will differ, making it impossible to determine which attribute values ​​should be associated with each other and extracted as the same row, and there is a possibility that attribute values ​​that should not be associated with each other will be extracted as associated with each other.

[0045] In addition, in the example table shown in Figure 4, there are no horizontal lines and the spacing between characters is narrow, so the "product name" character string may be combined and acquired as a single attribute value.

[0046] For example, as shown in Figure 6, three attribute values ​​for the item name "Product Name"—"July 2020 (July 1st to end of July)," "August 2020 (August 1st to end of August)," and "September 2020 (September 1st to end of September)"—may be combined and extracted as a single attribute value string. When multiple strings are combined and extracted as a single attribute value string in this way, it becomes impossible to determine their correspondence with attribute values ​​in other item names. In the case shown in Figure 6, it is impossible to determine how the single attribute value string resulting from the combination of three attribute values ​​should be associated with the three attribute values ​​"Set" and "100,500" for the item names "Quantity" and "Unit Price."

[0047] In addition, compared to the attribute values ​​of the items "quantity" and "unit price," the attribute values ​​of "item name" tend to be written freely, so there is a high possibility that multiple character strings will be combined and extracted as a single character string.

[0048] Therefore, the terminal device 10 of this embodiment executes KV extraction as described below, so that even if the table layout is irregular, attribute values ​​among a plurality of items can be associated with each other.

[0049] Next, the hardware configuration of the terminal device 10 in the information processing system of this embodiment is shown in FIG.

[0050] 7, the terminal device 10 has a CPU 11, a memory 12, a storage device 13 such as a hard disk drive, a communication interface (abbreviated as IF) 14 for transmitting and receiving data to and from external devices via a network 30, and a user interface (abbreviated as UI) device 15 including a liquid crystal display and a keyboard. These components are connected to one another via a control bus 16.

[0051] The CPU 11 is a processor that controls the operation of the image forming apparatus 10 by executing predetermined processes based on a control program stored in the memory 12 or the storage device 13. While the present embodiment has been described as the CPU 11 reading and executing the control program stored in the memory 12 or the storage device 13, this is not limiting. The control program may be provided in a form recorded on a computer-readable recording medium. For example, the program may be provided in a form recorded on an optical disc such as a CD (Compact Disc)-ROM or a DVD (Digital Versatile Disc)-ROM, or in a semiconductor memory such as a USB (Universal Serial Bus) memory or a memory card. The control program may also be obtained from an external device via a communication line connected to the communication interface 14.

[0052] FIG. 8 is a block diagram showing the functional configuration of the terminal device 10 realized by executing the above control program.

[0053] As shown in FIG. 8, the terminal device 10 of this embodiment includes an image data receiving unit 31, a table analyzing unit 32, an in-table KV extracting unit 33, an extraction result changing unit 34, and a KV data storing unit 35.

[0054] The image data receiving unit 31 receives image data such as a form image from an external device such as the image forming device 20 via the network 30 .

[0055] The table analysis unit 32 analyzes the table contained in the image data received by the image data reception unit 31, and acquires information such as the structure of the table, such as the number of rows and columns, and its position in the image.

[0056] The table KV extraction unit 33 performs character recognition processing on the form image containing tabular data based on the table analysis results by the table analysis unit 32, thereby extracting KVs to obtain strings of item names representing items of tabular data and strings of attribute values ​​for those items.

[0057] When the correspondence between attribute values ​​in multiple items cannot be determined in the KV extraction by the in-table KV extraction unit 33, the extraction result change unit 34 changes the correspondence between attribute values ​​in multiple items using the height in the form image of the character string of the attribute value whose correspondence with the attribute value in other items cannot be determined. Here, the height in the form image is expressed as the length from the bottom to the top of the character when viewed in the direction in which the character is upright. Specifically, when the character string is upright, it is expressed as the length of the character string in the axial direction from the bottom to the top of the rectangular form image. When the form image is not upright but is horizontal, the height in the form image is expressed as the length of the character string in the axial direction from the right side to the left side or from the left side to the right side of the rectangular form image.

[0058] The extraction result change unit 34 acquires information on the coordinates, height, and width in the image of the character string of each attribute value extracted by the in-table KV extraction unit 33, and changes the correspondence between attribute values ​​between multiple items by comparing the coordinates, height, etc. of attribute values ​​between adjacent items. Here, the coordinates in the form image are expressed as positions in the X-axis and Y-axis directions relative to an origin set inside or outside the form image arranged on the XY plane, for example.

[0059] For example, if the height of the character string of an attribute value of a certain item is an integer multiple of twice or more the height of the attribute value of another item, the extraction result change unit 34 changes the correspondence between attribute values ​​among a plurality of items by dividing the character string of the attribute value of the certain item in the height direction. Here, the integer multiple does not necessarily have to be an exact integer multiple, and an error within a range of 90 to 110% from the integer multiple value is acceptable.

[0060] Specifically, the extraction result change unit 34 divides the character string of an attribute value of a certain item in the height direction so that the height of the divided character string of the attribute value of a certain item approximately matches the height of the character string of the attribute value of another item. Here, "approximately match" does not only mean a perfect match, but also includes, for example, a case where the height of one character string is within a range of 90 to 110% of the height of the other character string.

[0061] In addition, if the extraction result change unit 34 divides the character string of an attribute value of a certain item based on the height of the character string of the attribute value of another item, but cannot establish a one-to-one correspondence with the character string of the attribute value of another item, it divides the character string of the attribute value of a certain item in the height direction to match the height position of the top end of each character string of the attribute value of another item.

[0062] This is because when the number of attribute values ​​in a plurality of items is not one-to-one, the character strings of the attribute values ​​are often aligned to the top in the table.

[0063] In addition, for an item for which no attribute value exists that corresponds to an attribute value in a certain item, the extraction result change unit 34 changes the correspondence between attribute values ​​among multiple items by adding a character string indicating that no corresponding attribute value exists, for example, an empty character string.

[0064] Furthermore, if an attribute value in an item does not have a corresponding attribute value in any other item, the extraction result change unit 34 may change the correspondence between attribute values ​​among multiple items by deleting the attribute value in the item.

[0065] When an attribute value in a certain item is deleted, the extraction result change unit 34 may notify the user that the attribute value has been deleted.

[0066] A specific example of how the extraction result change unit 34 changes the correspondence between attribute values ​​among a plurality of items will be described later.

[0067] The KV data storage unit 35 stores the KV extraction results after the changes have been made by the extraction result change unit 34 as KV data.

[0068] Next, the operation of the terminal device 10 of this embodiment when extracting KVs from input image data will be described in detail with reference to the drawings.

[0069] First, the overall operation of the terminal device 10 when extracting KVs is shown in the flowchart of FIG.

[0070] First, when image data including data in a table format is transmitted from an external device, the image data receiving unit 31 receives the transmitted image data in step S101.

[0071] Then, in step S102, the table analysis unit 32 analyzes the table in the received image data. Next, in step S103, the table KV extraction unit 33 performs KV extraction to extract character strings of item names and character strings of attribute values ​​of each item from the table in the image data.

[0072] Finally, in step S104, the extraction result change unit 34 changes the correspondence between attributes in the KV extraction results performed by the in-table KV extraction unit 33, correcting the correspondence between attribute values ​​among multiple items so that they are one-to-one.

[0073] Next, the details of the correction process of the KV extraction result in step S104 shown in the flowchart of Fig. 9 will be described with reference to the flowchart of Fig. 10. Note that the flowchart of Fig. 10 specifically describes the case of processing for the document image shown in Fig. 4.

[0074] First, in step S201, the extraction result change unit 34 determines whether evaluation of all the character strings of the attribute values ​​in the "product name" item has been completed for the KV extraction result for the table format in the text image shown in Fig. 4. If evaluation of all the character strings of the attribute values ​​in the "product name" item has been completed, the extraction result change unit 34 ends the process.

[0075] If evaluation of all the character strings of the attribute values ​​in the "product name" item has not been completed, the extraction result change unit 34 selects one character string of the "product name" attribute value that has not been evaluated in step S202.

[0076] That is, the extraction result change unit 34 evaluates the character strings of the attribute values ​​in the "product name" item one by one in order, and when evaluation has been performed on the character strings of all the attribute values, ends the correction process of the KV extraction result.

[0077] Next, in step S203, the extraction result change unit 34 acquires information about the height of the character string of the selected attribute value.

[0078] Then, in step S204, the extraction result change unit 34 determines whether the height of the character string of the attribute value in the "product name" item matches the character string of the attribute values ​​in the "quantity" and "unit price" items.

[0079] In step S204, if the height of the attribute value character string in the "item name" item matches the attribute value character string in the "quantity" and "unit price" items, the extraction result change unit 34 determines that there is no problem with the correspondence between the attribute value character string in the "item name" item and the attribute value character string in the "quantity" and "unit price" items, and returns to the processing of step S201.

[0080] If, in step S204, the height of the character string of the attribute value in the "product name" item does not match the character string of the attribute value in the "quantity" and "unit price" items, the extraction result change unit 34 determines in step S205 whether the height of the character string of the attribute value in the selected "product name" item is a multiple of the height of the character string of the attribute value in the "quantity" and "unit price" items. In other words, the extraction result change unit 34 determines whether the height of the character string of the attribute value in the selected "product name" item is an integer multiple that is two or more times the height of the character string of the attribute value in the "quantity" and "unit price" items. Here, an integer multiple that is two or more times is considered to be a value that allows for an error of 90% to 110% from the exact integer multiple, as described above.

[0081] If it is determined in step S205 that the height of the character string of the attribute value in the selected "product name" item is multiple times the height of the character string of the attribute value in the "quantity" and "unit price" items, the extraction result change unit 34 divides the character string of the attribute value in the selected "product name" item in step S206 so that the height matches the height of the character string of the attribute value in the "quantity" and "unit price" items.

[0082] Furthermore, if it is determined in step S205 that the height of the character string of the attribute value in the selected "product name" item is not a multiple of the height of the character string of the attribute value in the "quantity" and "unit price" items, the extraction result change unit 34 adds or inserts an empty string, indicating that no corresponding string exists, as the attribute value in the "quantity" and "unit price" items so that the height of the character string of the attribute value in the selected "product name" item matches.

[0083] An example of adding such an empty string is shown in FIG.

[0084] Referring to Figure 11, we can see that the string corresponding to the string "XX business support costs (July to September 2020)", which is the attribute value of "Product name" in the tabular data that is the target of KV extraction, does not exist as an attribute value for the "Quantity" or "Unit price" items.

[0085] Therefore, the extraction result change unit 34 adds the character string "" "" as an empty character string indicating that there is no corresponding character string in the attribute values ​​of the "quantity" and "unit price" items, in correspondence with the character string "XXX business support fee (July to September 2020)" in the "product name". Note that the empty character string is not limited to the character string "" "", and various character strings may be used as the empty character string.

[0086] Figure 11 shows the KV extraction results after adding the empty string in this way, and it can be seen that there is a one-to-one correspondence between all the strings of the attribute values ​​in the three items, "product name," "quantity," and "unit price."

[0087] In addition, in step S207, if there is no string in the "quantity" or "unit price" items that corresponds to the string "XX business support costs (July to September 2020)", which is the attribute value in the "item name" item, the extraction result change unit 34 may change the correspondence between attribute values ​​between multiple items by deleting the string "XX business support costs (July to September 2020)" in the "item name" item.

[0088] An example of the KV extraction results when strings with no corresponding strings are deleted in this way is shown in Figure 12. In Figure 12, the string "XX business support costs (July to September 2020)" has been deleted, for which no corresponding strings exist in the "quantity" and "unit price" fields. Therefore, in the KV extraction results, it can be seen that there is a one-to-one correspondence between the strings of all attribute values ​​in the three fields "item name," "quantity," and "unit price."

[0089] In this way, when an attribute value in a certain item is deleted, the extraction result change unit 34 may notify the user that the attribute value has been deleted. For example, the extraction result change unit 34 may include a comment indicating that the attribute value has been deleted in the KV extraction result stored in the KV data storage unit 35. Alternatively, the extraction result change unit 34 may display a comment indicating that the attribute value has been deleted on a display device when KV extraction is performed. For example, the extraction result change unit 34 may display a comment such as "The character string corresponding to the character string 'XX business support fee (July to September 2020)' in 'Product Name' does not exist in 'Quantity' and 'Unit Price', so it has been deleted" on a display device or include the comment in the KV extraction result.

[0090] Next, an example of dividing the character string of the attribute value, which has been described in step S206 of the flowchart in FIG. 10, is shown in FIG.

[0091] Referring to Figure 13, when the three attribute values ​​for the item name "Product Name", "July 2020 (July 1st to end of July)", "August 2020 (August 1st to end of August)", and "September 2020 (September 1st to end of September)", are combined and extracted as a single attribute value string, this string can be divided vertically into three parts, resulting in three strings: "July 2020 (July 1st to end of July)", "August 2020 (August 1st to end of August)", and "September 2020 (September 1st to end of September)".

[0092] In this way, when multiple character strings are combined vertically to form a single character string, dividing this character string vertically makes it possible to improve the correspondence between the character strings of all attribute values ​​in multiple items.

[0093] Next, with reference to FIGS. 14 to 16, a specific process for dividing a character string in which multiple character strings are joined together will be described.

[0094] The positions and sizes of the character strings of the item names and the character strings of the attribute values ​​extracted by the in-table KV extraction unit 33 are identified, for example, by a method such as that shown in Fig. 14. Specifically, the position and size of each character string in the image can be determined from the X and Y coordinates of the reference point 90 of the character string in the image and the width and height of the character string.

[0095] Therefore, the extraction result change unit 34 can compare, for example, the height of the character string in the item "product name" with the height of the character string in the items "quantity" and "unit price" based on the height information of each character string. Specifically, as shown in Fig. 15, if the height of the character string in the item "product name" is 60 pixels and the height of the character string in the items "quantity" and "unit price" is 20 pixels, the extraction result change unit 34 can determine that the height of the character string in the item "product name" is three times the height of the character string in the items "quantity" and "unit price".

[0096] Therefore, the extraction result change unit 34 divides the character string in the item "product name" into three parts in the vertical direction, resulting in three character strings with a height of 20 pixels, as shown in Fig. 16. As a result, the character strings of all attribute values ​​in the three items "product name," "quantity," and "unit price" have a one-to-one correspondence.

[0097] Note that the correspondence between the strings of attribute values ​​of multiple items is irregular, so there are cases where a one-to-one correspondence cannot be achieved even if the height of the string of one item is divided by the height of the string of another item.

[0098] For example, as shown in Figure 17, this is the case when the height of the string in the "product name" item is three times the height of the string in the "quantity" and "unit price" items, but there are no two strings in the attribute values ​​of the "quantity" and "unit price" items. In this case, even if the strings in the "quantity" and "unit price" items are divided into three in the height direction, there is no one-to-one correspondence with the strings in the "quantity" and "unit price" items.

[0099] In such a case, as shown in FIG. 17, the extraction result change unit 34 determines that the character strings of the attribute values ​​in the items "quantity" and "unit price" are written aligned at the top, divides the character string in the item "product name" at two-thirds of the way from the top into two character strings, and associates the two divided character strings with the character strings in the items "quantity" and "unit price", respectively.

[0100] 18 also corresponds to a case where the height of the character string in the "item name" field is three times the height of the character strings in the "quantity" and "unit price" fields, but two character strings do not exist in the attribute values ​​of the "quantity" and "unit price" fields. In the case shown in Fig. 18, the extraction result change unit 34 determines that the character strings in the attribute values ​​of the "quantity" and "unit price" fields are written aligned to the top, divides the character string in the "item name" field at one-third of the way down from the top, creates two character strings, and associates the two divided character strings with the character strings in the "quantity" and "unit price" fields, respectively.

[0101] In each of the above embodiments, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0102] Furthermore, the operations of the processor in each of the above embodiments may be performed not only by a single processor but also by multiple processors located at physically separate locations working together. Furthermore, the order of the operations of the processor is not limited to the order described in each of the above embodiments and may be changed as appropriate.

[0103] [Variations] In the above embodiment, we have described a case where KV extraction is performed in the terminal device 10 on image data scanned in the image forming device 20, but the present invention is not limited to this, and can be similarly applied to cases where KV extraction is performed on a form image in various information processing devices. [Explanation of symbols]

[0104] 10 Terminal Equipment 11 CPU 12 Memory 13 Storage device 14 Communication Interface 15 User Interface Device 16 Control Bus 20 Image forming device 30 Network 31 Image data reception section 32 Table analysis section 33 KV extraction part in the table 34 Extraction result change section 35 KV data storage unit 90 reference points

Claims

1. a processor; The processor: By performing character recognition processing on a form image including tabular data, character strings of item names representing items of the tabular data and character strings of attribute values ​​of the items are obtained; When the correspondence between attribute values ​​in a plurality of items is not one-to-one and the correspondence cannot be determined, the height of the character string of the first attribute value in the form image, whose correspondence with the attribute values ​​in other items cannot be determined, is obtained, and the height of the first attribute value is changed so that the correspondence between the attribute values ​​in the plurality of items becomes one-to-one. Information processing device.

2. 2. The information processing device according to claim 1, wherein the processor changes the correspondence between attribute values ​​among a plurality of items by dividing the character string of the first attribute value in the height direction when the height of the character string of the first attribute value is an integer multiple of at least twice the height of the attribute value in another item.

3. 3. The information processing device according to claim 2, wherein the processor divides the character string of the first attribute value in the height direction so that the height of the character string of the first attribute value after division is approximately the same as the height of the character string of the attribute value in another item.

4. 3. The information processing device according to claim 2, wherein when the character string of the first attribute value cannot be matched one-to-one with the character string of the attribute value in another item even if the character string of the first attribute value is divided based on the height of the character string of the attribute value in another item, the processor divides the character string of the first attribute value in the height direction to match the height position of the top end of each character string of the attribute value in the other item.

5. 5. An information processing device according to claim 1, wherein the processor changes the correspondence between attribute values ​​among a plurality of items by adding a character string indicating that no corresponding attribute value exists for an item for which no attribute value corresponding to the first attribute value exists.

6. 5. An information processing device according to claim 1, wherein the processor changes the correspondence between attribute values ​​among a plurality of items by deleting the first attribute value if no attribute value corresponding to the first attribute value exists in any other item.

7. The information processing apparatus according to claim 6 , wherein the processor notifies that the attribute value has been deleted.

8. a step of performing character recognition processing on a form image including tabular data to obtain character strings of item names representing items of the tabular data and character strings of attribute values ​​of the items; when the correspondence between attribute values ​​in the plurality of items is not one-to-one and the correspondence cannot be determined, acquiring a height in the form image of a character string of a first attribute value whose correspondence with attribute values ​​in other items cannot be determined, and changing the height of the first attribute value so that the correspondence between attribute values ​​in the plurality of items becomes one-to-one; A program that causes a computer to execute the following.

Images