Image processing system and image processing program

Abstract

To provide an image processing system and an image processing program capable of reducing the burden for specifying a range of an unruled table included in an image.SOLUTION: An image processing system is configured to: specify (S102) a position of a title row in a target image including a table by template matching that uses a template image including a title row portion of a table; determine an alignment direction of a plurality of character strings, as an alignment direction for alignment with a specific portion, when lateral coordinates of the specific portion of a left end portion, a central portion and a right end portion in lateral direction of each of bounding boxes of multiple character strings below the title row in the target image exist within a specific distance from each other; and specify (S105) character strings constituting the table based on the alignment direction and specify a lower end position of the table on the basis of the arrangement of the character strings constituting the table.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing system and an image processing program for identifying an area of ​​a table without rules included in an image. [Background technology]

[0002] Conventionally, as an image processing system for identifying the range of an unlined table contained in an image, a system is known that converts an unlined table into a ruled table by using a machine learning device that has been trained to convert unlined tables into ruled tables (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-82814 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional image processing systems, a large amount of training data must be prepared for learning by the machine learning device, which imposes a large burden on the machine learning device to learn how to convert tables without lines into tables with lines.

[0005] SUMMARY OF THE PRESENT EMBODIMENTS In view of the above, an object of the present invention is to provide an image processing system and an image processing program that can reduce the burden of identifying the range of a table without rules contained in an image. [Means for solving the problem]

[0006] The image processing system of the present invention identifies the position of the title line in an image including a table by template matching using a template image that includes a portion of the title line of the table, and if the horizontal coordinates of specific portions among the left end portions, the middle portion in the horizontal direction, and the right end portions of the bounding boxes of multiple character strings below the title line in the image are within a specific distance from each other, determines the alignment direction of the multiple character strings as the alignment direction for aligning the specific portions, identifies the character strings that make up the table based on the alignment direction, and identifies the position of the bottom end of the table based on the arrangement of the character strings that make up the table.

[0007] With this configuration, the image processing system of the present invention locates the position of the title row in the image by template matching using a template image that includes part of the title row of the table, determines the alignment direction of multiple character strings below the title row in the image, identifies the character strings that make up the table based on the alignment direction, and locates the position of the bottom edge of the table based on the arrangement of the character strings that make up the table, so there is no need to prepare large amounts of training data in order to identify the area of ​​a table without lines, and as a result, the burden associated with identifying the area of ​​a table without lines can be reduced.

[0008] The image processing system of the present invention may determine the shift direction by using a plurality of tables of the same type.

[0009] With this configuration, the image processing system of the present invention determines the alignment direction of character strings using a plurality of tables of the same type, thereby improving the accuracy of determining the alignment direction of character strings.

[0010] When the image processing system of the present invention identifies the position of the title line in the image by template matching using the template image, it is not necessary to identify the position of the title line again within the scope of the template image when placing the template image relative to the image so that the title line in the template image is located at the identified position.

[0011] With this configuration, when the image processing system of the present invention identifies the position of the title line in an image by template matching using a template image, it does not re-identify the position of the title line within the template image when placing the template image relative to the image so that the title line in the template image is located at the identified position, thereby reducing the possibility of erroneously identifying the position of the title line in an image and, as a result, improving the accuracy of identifying the area of ​​a table without lines.

[0012] The image processing program of the present invention is characterized in that it causes a computer to perform the following operations: identify the position of a title line in an image including a table by template matching using a template image including a portion of the title line of the table; determine the justification direction of a plurality of character strings below the title line in the image as the justification direction for aligning the plurality of character strings at the specific portions when the coordinates in the left-right direction of a specific portion among the left end portion, the central portion in the left-right direction, and the right end portion of each bounding box of the plurality of character strings below the title line in the image are within a specific distance from each other; identify the character strings that make up the table based on the justification direction; and identify the position of the bottom end of the table based on the arrangement of the character strings that make up the table.

[0013] With this configuration, a computer executing the image processing program of the present invention identifies the position of the title row in the image by template matching using a template image that includes part of the title row of the table, determines the alignment direction of multiple character strings below the title row in the image, identifies the character strings that make up the table based on the alignment direction, and identifies the position of the bottom edge of the table based on the arrangement of the character strings that make up the table. Therefore, there is no need to prepare large amounts of training data to identify the area of ​​a table without lines, and as a result, the burden of identifying the area of ​​a table without lines can be reduced. Effect of the Invention

[0014] The image processing system and image processing program of the present invention can reduce the burden of identifying the range of a table without rules contained in an image. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram of an image processing system according to an embodiment of the present invention when configured with one computer. [Diagram 2] 2 is a diagram showing an example of a target image handled by the image processing system shown in FIG. 1; [Diagram 3] 2 is a diagram showing an example of a template image handled by the image processing system shown in FIG. 1; [Figure 4] 4 is a flowchart of the operation of the image processing system shown in FIG. 1 when identifying the range of a table included in an image. [Diagram 5] 5 is a flowchart of a title line position specifying process shown in FIG. 4. [Figure 6] FIG. 2 is a diagram showing an example of a correlation image generated by the image processing system shown in FIG. [Figure 7] 7 is an example of the correlation image shown in FIG. 6 in a state where a mask with the size of the template image is applied to the position of the title line. [Figure 8] 3 is a diagram showing an example of the target image shown in FIG. 2 in a state in which the range of the title line has been specified. FIG. [Figure 9] 5 is a flowchart of the character string specification process shown in FIG. 4. [Figure 10] 3 is a diagram showing an example of the target image shown in FIG. 2 in a state in which each character is boxed. FIG. [Figure 11] 3 is a diagram showing an example of the target image shown in FIG. 2 in a state in which each character string is boxed. FIG. [Figure 12] 5 is a flowchart of a table-constituting character string specification process shown in FIG. 4. [Figure 13] 2 is a diagram showing an example of an upper left point X coordinate list generated by the image processing system shown in FIG. 1. [Figure 14]13 is a flowchart of the character string grouping process shown in FIG. 12. [Figure 15] FIG. 14 is a diagram showing an example of the upper left point X coordinate list shown in FIG. 13 in a state in which the coordinates of a plurality of points that exist within a specific distance from each other have been replaced. [Figure 16] FIG. 16 is a diagram showing an example of the upper left point X coordinate list shown in FIG. 15 in a state in which character strings having the same coordinates are grouped together. [Figure 17] 3 is a diagram showing an example of the target image shown in FIG. 2 in a state in which the position of the bottom edge of the table has been specified. FIG. [Figure 18] 3 is a diagram showing an example of the target image shown in FIG. 2 in a state in which the range of the table has been specified. FIG. [Figure 19] 7 is a diagram showing an example of a correlation image generated by the image processing system, which is different from the example shown in FIG. 6. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0017] First, the configuration of the image processing system according to the present embodiment will be described.

[0018] FIG. 1 is a block diagram of an image processing system 10 according to the present embodiment, which is configured using one computer.

[0019] As shown in Fig. 1, the image processing system 10 includes an operation unit 11, which is an operation device such as a keyboard or a mouse through which various operations are input, a display unit 12, which is a display device such as an LCD (Liquid Crystal Display) that displays various information, a communication unit 13, which is a communication device that communicates with an external device via a network such as a LAN (Local Area Network) or the Internet, or directly by wire or wirelessly without using a network, a storage unit 14, which is a non-volatile storage device such as a semiconductor memory or an HDD (Hard Disk Drive) that stores various information, and a control unit 15 that controls the entire image processing system 10. The image processing system 10 may be configured by, for example, a PC (Personal Computer). The image processing system 10 may be arranged on a cloud.

[0020] The storage unit 14 can store an image processing program 14a for executing a specific process on an input image and outputting the execution result. The image processing program 14a may be installed in the image processing system 10 during the manufacturing stage of the image processing system 10, or may be additionally installed in the image processing system 10 from an external storage medium such as a USB (Universal Serial Bus) memory, or may be additionally installed in the image processing system 10 from a network.

[0021] The control unit 15 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory) that stores programs and various data, and a RAM (Random Access Memory) that serves as a memory used as a working area for the CPU of the control unit 15. The CPU of the control unit 15 executes programs stored in the storage unit 14 or the ROM of the control unit 15.

[0022] The control unit 15 executes the image processing program 14a to realize an image processing unit 15a that performs a specific process on an input image and outputs the result of the process.

[0023] Methods of inputting an image into the image processing system 10 include, for example, a method in which image data is input into a specific folder monitored by the image processing system 10, and the image processing system 10 acquires the image data from this folder; a method in which image data read from a document by an image reading device such as an MFP (Multifunction Peripheral) or a dedicated scanner is transmitted from the image reading device to the image processing system 10; a method in which image data is acquired from an email containing image data when the email is received to an email address monitored by the image processing system 10; a method in which image data is transmitted to the image processing system 10 from a web browser on a computer such as a PC to a website provided by the image processing system 10; and a method in which image data is transmitted to the image processing system 10 from a dedicated application installed on a mobile device such as a smartphone.

[0024] The specific processing of an image by the image processing system 10 includes, for example, processing for converting the contents of a table included in an image into data in CSV (Comma-Separated Values) format. The execution result of the processing for converting the contents of a table included in an image into data in CSV format is, for example, data in CSV format into which the contents of the table included in the image have been converted.

[0025] Methods for transmitting execution results from the image processing system 10 include, for example, inputting the execution results into a specific cloud service, inputting the execution results into a specific folder, and sending an email including the execution results to a specific email address.

[0026] Next, the operation of the image processing system 10 when identifying the range of a table included in an image will be described.

[0027] An image (hereinafter referred to as a "target image") for which the image processing system 10 is to identify the range of a table is input as the target image to the image processing system 10 by, for example, any of the input methods described above. In addition, a template image (hereinafter simply referred to as a "template image") used by the image processing system 10 to identify the range of a table included in an image is also input as a template image to the image processing system 10 in the same manner as the target image.

[0028] In the following, a target image 20 (see FIG. 2) and a template image 30 (see FIG. 3) will be used as examples of a target image and a template image, respectively.

[0029] Fig. 2 is a diagram showing an example of a target image handled by the image processing system 10. Fig. 3 is a diagram showing an example of a template image handled by the image processing system 10.

[0030] Target image 20 shown in Fig. 2 is an image read from a document by an image reading device. Target image 20 includes table 21 and table 22. Table 21 has title row 21a at the top and has no lines separating rows or columns. Table 22 has title row 22a at the top and has no lines separating rows or columns. Title row 21a and title row 22a have the same appearance.

[0031] The template image 30 shown in FIG. 3 is the same as the image of the title row 21a of the table 21 and the title row 22a of the table 22.

[0032] FIG. 4 is a flowchart showing the operation of the image processing system 10 when identifying the range of a table.

[0033] As shown in Fig. 4, the image processing unit 15a executes pre-processing on the target image (S101). The pre-processing in S101 includes, for example, tilt correction for reducing the tilt of the target image that occurs when the image reading device reads the document at an angle, and size correction for adjusting the size of the target image to the size of the document when the size of the target image changes from the size of the document when the document is read by the image reading device. After the process of S101 is completed, the target image that has been pre-processed in S101 is used.

[0034] When the process of S101 ends, the image processing unit 15a executes a title line position specifying process for specifying the position of the title line in the target image by template matching using a template image including a portion of the title line of the table (S102).

[0035] FIG. 5 is a flowchart of the title line position specifying process shown in FIG.

[0036] As shown in FIG. 5, the image processing unit 15a generates an image (hereinafter referred to as a “correlation image”) showing the correlation between each position in the target image and the template image by calculating the correlation between each position in the target image and the template image, i.e., the cumulative error, by convolution (S131).

[0037] FIG. 6 is a diagram showing an example of a correlation image generated by the image processing system 10. As shown in FIG.

[0038] Correlation image 40 shown in FIG. 6 is an image in which the correlation with template image 30 (see FIG. 3) at each position in target image 20 (see FIG. 2), i.e., the accumulated error, is expressed as brightness. Each position in correlation image 40 corresponds to each position in target image 20. In correlation image 40, the higher the brightness of a pixel position, the lower the accumulated error, i.e., the stronger the correlation. Correlation image 40 shown in FIG. 6 includes regions 41, 42 formed by adjacent pixels in which the accumulated error calculated in S131 is equal to or less than a specific threshold value (hereinafter referred to as "strong correlation regions")

[0039] As shown in Fig. 5, when the process of S131 is completed, the image processing unit 15a extracts a strongly correlated region in the correlation image (S132). That is, the image processing unit 15a extracts a region formed by adjacent pixels whose brightness is equal to or higher than a specific threshold as a strongly correlated region. In the example of the correlation image 40 shown in Fig. 6, the image processing unit 15a extracts a strongly correlated region 41 and a strongly correlated region 42 in the process of S132.

[0040] When the process of S132 is completed, the image processing unit 15a determines whether or not the number of strong correlation regions extracted in the immediately preceding S132 is one or more (S133).

[0041] When the image processing unit 15a determines in S133 that the number of strongly correlated regions extracted in the immediately preceding S132 is one or more, it determines whether or not the number of strongly correlated regions extracted in the immediately preceding S132 is one (S134).

[0042] When the image processing unit 15a determines in S134 that the number of strongly correlated regions extracted in the immediately preceding S132 is one, it specifies the position of the title line based on the strongly correlated region extracted in the immediately preceding S132 (S135). For example, the image processing unit 15a may specify the position in the strongly correlated region of interest where the accumulated error calculated in S131 is the smallest as the position of the title line. The image processing unit 15a may also specify the center of gravity of the strongly correlated region of interest, i.e., the simple average value of the coordinates of all pixels included in the strongly correlated region of interest, as the coordinates of the position of the title line. The image processing unit 15a may also specify the average value of the coordinates of all pixels included in the strongly correlated region of interest, weighted by the accumulated error calculated in S131 for each pixel, as the coordinates of the position of the title line.

[0043] When the image processing unit 15a determines in S134 that the number of strongly correlated regions extracted in the previous S132 is not one, it identifies the position containing the smallest cumulative error calculated in S131, i.e., the position with the highest brightness, among all the positions contained in all the strongly correlated regions extracted in the previous S132 (S136).

[0044] When the process of S136 is completed, the image processing unit 15a specifies the position of the title line based on the strong correlation region including the position specified in S136 among all the strong correlation regions extracted in the immediately preceding S132 (S137). The method of specifying the position of the title line in S137 is the same as the method of specifying the position of the title line in S135.

[0045] When the process of S137 is completed, the image processor 15a applies a mask to the correlation image with the size of the template image at the position of the title line specified in the previous S137 (S138). That is, the image processor 15a applies a mask to the correlation image so that the center of gravity of the mask with the size of the template image is located at the position of the title line specified in the previous S137. After the process of S138 is completed, the correlation image after the masking in S138 is used.

[0046] FIG. 7 is an example of a correlation image 40 in which a mask is applied to the title line position with the size of the template image.

[0047] The correlation image 40 shown in FIG. 7 is obtained by applying a mask 43 to the correlation image 40 shown in FIG. 6. When the image processing unit 15a specifies the position of the title line based on the strongly correlated region 41 shown in FIG. 6 in S137, the image processing unit 15a applies a mask 43 with the size of the template image to the position of the title line specified in the immediately preceding S137 as shown in FIG. 7. It is assumed that the position of the title line specified in the immediately preceding S137 was inside the strongly correlated region 41 shown in FIG. 6. Here, the mask 43 is an image in which the brightness of all pixels is lower than the brightness corresponding to the accumulated error corresponding to the threshold value in S132. For example, the mask 43 may be an image in which the brightness of all pixels is 0. Applying the mask 43 to the position of the title line means that the mask 43 is synthesized with the correlation image 40 so that the center of gravity of the mask 43, that is, the simple average value of the coordinates of all pixels included in the mask 43, coincides with the position of the title line. In the example shown in FIG. 7, the strongly correlated region 41 shown in FIG. 6 cannot be seen due to the mask 43.

[0048] As shown in FIG. 5, upon completion of the process of S138, the image processing unit 15a executes the process of S132.

[0049] When the image processing unit 15a determines in S133 that the number of strong correlation regions extracted in the immediately preceding S132 is not one or more, or when the process of S135 ends, the title line position specifying process shown in FIG. 5 ends.

[0050] FIG. 8 is a diagram showing an example of the target image 20 in a state where the range of the title line has been specified.

[0051] By identifying the position of the title line by the title line position identifying process shown in Fig. 5, range 21b of title line 21a of table 21 and range 22b of title line 22a of table 22 are identified as shown in Fig. 8. It can be seen from target image 20 shown in Fig. 8 that target image 20 contains two tables. However, at the time when the title line position identifying process shown in Fig. 5 is completed, the position of the bottom end of each table included in target image 20 is unknown.

[0052] As shown in FIG. 4, when the title line position specifying process of S102 ends, the image processing unit 15a executes a character string specifying process for specifying a character string included in the target image (S103).

[0053] FIG. 9 is a flowchart of the character string specification process shown in FIG.

[0054] 9, the image processing unit 15a converts each character included in the target image into a box (S161). That is, the image processing unit 15a surrounds each character included in the target image with a bounding box.

[0055] FIG. 10 is a diagram showing an example of the target image 20 in which each character is boxed.

[0056] By the process of S161, each character in the target image 20 shown in FIG. 2 is boxed as shown in FIG.

[0057] As shown in FIG. 9, when the process of S161 ends, the image processing unit 15a measures the distance between the bounding boxes of adjacent characters, among the characters included in the target image, in both the horizontal and vertical directions (S162).

[0058] When the process of S162 is completed, the image processing unit 15a derives the direction of the character string in the target image and the typical distance between adjacent characters in the direction of the character string based on the statistical results of the measurement results in S162 (S163).

[0059] When the process of S163 is completed, the image processing unit 15a identifies the character string by connecting adjacent bounding boxes in the direction of the character string derived in S163 within a distance obtained by multiplying the typical distance derived in S163 by a specific coefficient (S164). The coefficient in S164 is a value slightly larger than 1.

[0060] When the process of S164 ends, the image processing unit 15a ends the character string identification process shown in FIG.

[0061] FIG. 11 is a diagram showing an example of the target image 20 in which each character string is boxed.

[0062] By the character string identification process shown in FIG. 9, each character string in the target image 20 shown in FIG. 2 is boxed as shown in FIG.

[0063] As shown in Fig. 4, when the character string identification process of S103 ends, the image processing unit 15a executes a table-composing character string identification process (S104) to identify character strings that are likely to constitute a table from character strings included in a range below the title line range that can become part of a table (hereinafter referred to as the "table candidate range"). If there is another title line range below the title line range, the bottom end of the table candidate range is the top end of the uppermost title line range among the other title line ranges present below the title line range. If there is no other title line range below the title line range, the bottom end of the table candidate range is the bottom end of the target image.

[0064] FIG. 12 is a flowchart of the table-constituting character string specification process shown in FIG.

[0065] As shown in FIG. 12, for each character string included in the table candidate range, the image processing unit 15a generates a list of numerical sequences obtained by extracting only the X coordinate of the upper left point of the bounding box (hereinafter referred to as the "upper left point X coordinate list"), a list of numerical sequences obtained by extracting only the Y coordinate of the upper left point of the bounding box (hereinafter referred to as the "upper left point Y coordinate list"), a list of numerical sequences obtained by extracting only the X coordinate of the center of gravity of the bounding box (hereinafter referred to as the "center of gravity point X coordinate list"), a list of numerical sequences obtained by extracting only the Y coordinate of the center of gravity of the bounding box (hereinafter referred to as the "center of gravity point Y coordinate list"), a list of numerical sequences obtained by extracting only the X coordinate of the lower right point of the bounding box (hereinafter referred to as the "lower right point X coordinate list"), and a list of numerical sequences obtained by extracting only the Y coordinate of the lower right point of the bounding box (hereinafter referred to as the "lower right point Y coordinate list") (S191). The coordinates of the center of gravity of a bounding box are the average of the coordinates of the upper left corner and the lower right corner of the bounding box. The X coordinate is the horizontal coordinate, i.e., the left-right coordinate. The Y coordinate is the vertical coordinate, i.e., the up-down coordinate.

[0066] FIG. 13 is a diagram showing an example of the upper left point X coordinate list generated by the image processing system 10. As shown in FIG.

[0067] 13, the top left point X coordinate list includes, for each string, a string ID as identification information for the string, the X coordinate of the top left point of the bounding box of the string, and group information indicating the group of the string. When the process of S191 is completed, nothing is indicated in the group information.

[0068] The above describes the configuration of the top left point X coordinate list. However, the configurations of the top left point Y coordinate list, the center of gravity point X coordinate list, the center of gravity point Y coordinate list, the bottom right point X coordinate list, and the bottom right point Y coordinate list are the same as the configuration of the top left point X coordinate list.

[0069] As shown in FIG. 12, when the process of S191 ends, the image processing unit 15a executes a string grouping process for grouping strings on the top left point X coordinate list generated in S191 (S192).

[0070] FIG. 14 is a flowchart of the character string grouping process shown in FIG.

[0071] As shown in FIG. 14, the image processing unit 15a determines whether or not there are multiple points in the target list whose coordinates are within a specific distance from each other (S221).

[0072] When the image processing unit 15a determines in S221 that there are multiple points whose coordinates are within a specific distance from each other, it calculates the average value of the coordinates of the multiple points whose coordinates are within a specific distance from each other (S222).

[0073] When the process of S222 is completed, the image processing unit 15a replaces the coordinates of each of a plurality of points that exist within a specific distance from each other with the average value calculated for these points in S222 (S223). That is, the image processing unit 15a eliminates slight coordinate errors that have occurred as numerical differences in the target list.

[0074] FIG. 15 is a diagram showing an example of the upper left point X coordinate list shown in FIG. 13 in a state in which the coordinates of a plurality of points that exist within a specific distance from each other have been replaced.

[0075] In the upper left point X coordinate list shown in Fig. 13, the X coordinates of the point with the character string ID "C101", the point with the character string ID "C102", and the point with the character string ID "C103" are within a specific distance from each other. Therefore, the X coordinates of these points are replaced by the average value of the X coordinates of these points by the process of S223, as shown in Fig. 15.

[0076] As shown in FIG. 14, when the process of S223 ends, the image processing unit 15a groups character strings having the same coordinates in the target list (S224).

[0077] FIG. 16 is a diagram showing an example of the upper left point X coordinate list shown in FIG. 15 in a state in which character strings having the same coordinates are grouped together.

[0078] In the upper left point X coordinate list shown in Fig. 15, character strings with the same X coordinate, i.e., a character string with a character string ID of "C101", a character string with a character string ID of "C102", and a character string with a character string ID of "C103", are recorded in the group information as the same group, as shown in Fig. 16. Here, the group information records group identification information.

[0079] As shown in FIG. 14, when the image processing unit 15a determines in S221 that there are no multiple points whose coordinates are within a specific distance from each other, or when the processing of S224 is completed, the image processing unit 15a ends the execution of the character string grouping processing shown in FIG. 14.

[0080] As shown in FIG. 12, when the process of S192 is completed, the image processing unit 15a executes a character string grouping process on each of the top left point Y coordinate list, the center of gravity point X coordinate list, the center of gravity point Y coordinate list, the bottom right point X coordinate list, and the bottom right point Y coordinate list generated in S191, in the same manner as for the top left point X coordinate list (S193 to S197).

[0081] When the process of S197 is completed, the image processing unit 15a determines the alignment direction of the character strings in each column of the table based on the group information in the top left X coordinate list, the center of gravity X coordinate list, and the bottom right X coordinate list (S198). That is, when the coordinates in the left and right directions of a specific part among the left end part, the center part in the left and right directions, and the right end part of each bounding box of a plurality of character strings below the title line in the target image are within a specific distance from each other, the image processing unit 15a determines the alignment direction of these character strings as the alignment direction for aligning the character strings at the above-mentioned specific parts. For example, when the X coordinates of the left ends of the bounding boxes of a plurality of character strings below the title line in the target image are within a specific distance from each other, the image processing unit 15a determines the alignment direction of these character strings as the alignment direction for aligning the character strings at the left ends, that is, left alignment. Furthermore, when the X coordinates of the centers in the left-right direction of the bounding boxes of a plurality of character strings below the title line in the target image are within a specific distance from each other, image processing unit 15a determines the alignment direction of these character strings to be an alignment direction that aligns the character strings at the center in the left-right direction, i.e., center alignment. Furthermore, when the X coordinates of the right ends of the bounding boxes of a plurality of character strings below the title line in the target image are within a specific distance from each other, image processing unit 15a determines the alignment direction of these character strings to be an alignment direction that aligns the character strings at the right ends, i.e., right alignment.

[0082] When the process of S198 is completed, the image processing unit 15a identifies character strings that are likely to form a table from among the character strings within the table candidate range, based on the alignment direction determined in S198 and the group information in the top left point X coordinate list, the center of gravity point X coordinate list, and the bottom right point X coordinate list (S199).

[0083] A character string constituting a table may have the same X coordinates of the left end, center in the horizontal direction, or right end as a character string belonging to another cell in the same column as the cell to which the character string belongs. For example, when the alignment direction of the character string in each cell in the column to which the cell to which the character string belongs is set to left alignment, the X coordinate of the left end of the character string is the same as that of the character string belonging to the other cells in the column. When the alignment direction of the character string in each cell in the column to which the cell to which the character string belongs is set to center alignment, the X coordinate of the center in the horizontal direction of the character string is the same as that of the character string belonging to the other cells in the column. When the alignment direction of the character string in each cell in the column to which the cell to which the character string belongs is set to right alignment, the X coordinate of the right end of the character string is the same as that of the character string belonging to the other cells in the column. Therefore, among the character strings in the table candidate range, multiple character strings having the same X coordinates of the left end, center in the horizontal direction, or right end are likely to form a table.

[0084] In addition, among the character strings in the table candidate range, even if the X coordinate of the left end, center in the horizontal direction, or right end of other character strings in the table candidate range is the same as that of other character strings in the table candidate range, if the justification direction of the character strings in the column to which these character strings are likely to belong is different, it is likely that the character string does not constitute a table. For example, the character string "above" located at the bottom of the target image 20 (see FIG. 11) has the same X coordinate of the right end as the X coordinate of the right end of the character string above, "12 x 900 ml plastic bottles of black tea", but since the justification direction of the character string in the column to which "12 x 900 ml plastic bottles of black tea" is likely to belong is left justification, it is likely that the character string does not constitute table 22.

[0085] When the processing of S199 is completed, the image processing unit 15a identifies additional character strings within the table candidate range that are likely to form a table, based on the character strings identified in S199 and the group information in the top left point Y coordinate list, the center of gravity point Y coordinate list, and the bottom right point Y coordinate list (S200).

[0086] Character strings that make up a table are likely to be on the same line as other character strings that make up a table. Therefore, among character strings within a table candidate range, a character string that has the same Y coordinate of its top edge, center in the vertical direction, or bottom edge as a character string that is likely to make up a table is also likely to make up a table.

[0087] When the process of S200 ends, the image processing unit 15a ends the table-constituting character string specification process shown in FIG.

[0088] 4, when the table-constituting string identification process of S104 ends, the image processing unit 15a identifies the position of the bottom end of each table included in the target image 20 based on the arrangement of the character strings identified in the table-constituting string identification process of S104 as character strings that are likely to constitute a table (S105). That is, the image processing unit 15a identifies the bottom end position of the bottommost character string among the character strings identified in the table-constituting string identification process of S104 as character strings that are likely to constitute a table, as the bottom end position of this table.

[0089] FIG. 17 is a diagram showing an example of the target image 20 in a state where the position of the bottom edge of the table has been specified.

[0090] By identifying the position of the bottom end of each table through the process of S105, a bottom end position 21c of table 21 and a bottom end position 22c of table 22 are identified as shown in FIG.

[0091] As shown in FIG. 4, when the process of S105 is completed, the image processing unit 15a identifies, based on the X coordinate of each character string, which of the character strings in the range below the range of the title row of the table and above the bottom end position identified in S105 of the table is included in which column corresponding to each character string in the title row (S106), and then terminates the operation shown in FIG. 4.

[0092] FIG. 18 is a diagram showing an example of the target image 20 in a state where the range of the table has been specified.

[0093] By the operation shown in FIG. 4, the table range of the target image 20 shown in FIG. 2 can be specified as shown in FIG.

[0094] As described above, image processing system 10 locates the position of the title line in the image by template matching using a template image including part of the title line of the table (S102), determines the alignment direction of multiple character strings below the title line in the image (S198), identifies the character strings that make up the table based on the alignment direction (S199), and locates the position of the bottom edge of the table based on the arrangement of the character strings that make up the table (S105). Therefore, there is no need to prepare a large amount of training data in order to identify the area of ​​a table without lines, and as a result, the burden of identifying the area of ​​a table without lines can be reduced.

[0095] A plurality of character strings each having the same horizontal length and the same X coordinate of the left end of the bounding box of each character string also have the same X coordinate of the center in the left-right direction of the bounding box of each character string, and also have the same X coordinate of the right end of the bounding box of each character string. Therefore, the image processing system 10 cannot determine the alignment direction of these character strings based only on a plurality of character strings each having the same horizontal length and the same X coordinate of the left end of the bounding box of each character string. However, the image processing system 10 determines the alignment direction of the character strings in a table using two tables of the same type, that is, Table 21 and Table 22, and therefore can improve the accuracy of determining the alignment direction of the character strings in a table compared to the case where only one table is used to determine the alignment direction of the character strings in a table. In the above, the image processing system 10 determines the alignment direction of the character strings in a table using two tables of the same type in the same target image. However, the image processing system 10 may determine the justification direction of character strings in a table by using multiple tables of the same type in multiple target images.

[0096] FIG. 19 is a diagram showing an example of a correlation image generated by the image processing system 10, which is different from the example shown in FIG.

[0097] The correlation image 50 shown in FIG. 19 is generated in S131, similarly to the correlation image 40 shown in FIG. 6. The correlation image 50 includes strong correlation regions 51, 52, 53, 54, 55, and 56. In a specific case, such as when the template image has a simple structure, the correlation image may include strong correlation regions at a plurality of positions where the Y coordinates are the same or almost the same, but the X coordinates are different. For example, the strong correlation regions 51 to 53 in the correlation image 50 are arranged at a plurality of positions where the Y coordinates are the same or almost the same, but the X coordinates are different. In addition, the strong correlation regions 54 to 56 in the correlation image 50 are also arranged at a plurality of positions where the Y coordinates are the same or almost the same, but the X coordinates are different. Among the pixels included in the strong correlation regions 51, 52, and 53, the pixel with the highest brightness is included in the strong correlation region 52. Among the pixels included in the strong correlation regions 54, 55, and 56, the pixel with the highest brightness is included in the strong correlation region 55.

[0098] The image processing system 10 specifies the position with the smallest cumulative error calculated in S131 among all positions included in all strongly correlated regions extracted in the previous S132 (S136), specifies the position of the title line based on the strongly correlated region including the position specified in S136 among all strongly correlated regions extracted in the previous S132 (S137), and applies a mask to the position of the title line specified in S137 in the correlation image with the size of the template image (S138). That is, when the image processing system 10 specifies the position of the title line in the image by template matching using the template image (S137), it does not specify the position of the title line again within the range of the template image when placing the template image on the image so that the title line in the template image exists at the specified position (S138). Therefore, the image processing system 10 can reduce the possibility of erroneously specifying the position of the title line based on the strongly correlated regions 51, 53, 54, and 56 in the correlation image 50 shown in FIG. 19, and as a result, can improve the accuracy of specifying the range of the table without ruled lines.

[0099] In this embodiment, image processing system 10 uses a template image that includes only the title line, so the accuracy of identifying the position of the title line is high. However, image processing system 10 can identify the position of the title line even if a template image that includes characters in the vicinity of the title line is used.

[0100] In this embodiment, the image processing system 10 uses the upper left point and the lower right point of the bounding box of the character string. However, the image processing system 10 may use the lower left point and the upper right point of the bounding box of the character string instead of the upper left point and the lower right point of the bounding box of the character string.

[0101] In this embodiment, the image processing system 10 is configured with one computer. However, the image processing system 10 may be configured with multiple computers. [Explanation of symbols]

[0102] 10 Image Processing System 14a Image processing program 20 Target image (image) 21 table 21a Title line 21c position (bottom edge of table) 22 Table 22a Title line 22c position (bottom edge of table) 30 Template Images

Claims

1. identifying a position of the title line in an image including a table by template matching using a template image including a portion of the title line of the table; when the coordinates of specific portions among the left end portion, the center portion in the left-right direction, and the right end portion of each of the bounding boxes of the plurality of character strings below the title line in the image are within a specific distance from each other, the alignment direction of the plurality of character strings is determined to be the alignment direction for aligning the plurality of character strings at the specific portions; identifying the character strings constituting the table based on the alignment direction; An image processing system comprising: an image processing unit that specifies a position of a bottom end of the table based on an arrangement of the character strings that constitute the table.

2. 2. The image processing system according to claim 1, wherein the shift direction is determined by using a plurality of the tables of the same type.

3. 3. The image processing system according to claim 1, wherein when the position of the title line in the image is identified by template matching using the template image, the position of the title line is not identified again within the range of the template image when the template image is placed relative to the image so that the title line in the template image is located at the identified position.

4. identifying a position of the title line in an image including a table by template matching using a template image including a portion of the title line of the table; an operation of determining, when coordinates of specific portions among a left end portion, a central portion in the left-right direction, and a right end portion of each of a plurality of character strings below the title line in the image are within a specific distance from one another, a direction in which the plurality of character strings are aligned is the direction in which the plurality of character strings are aligned at the specific portions; an operation of identifying the character strings constituting the table based on the justification direction; an operation of identifying a position of the bottom end of the table based on an arrangement of the character strings constituting the table; An image processing program characterized by causing a computer to realize the above.

Images