Methods of understanding

Abstract

To provide a table recognition method capable of recognizing a table with high accuracy.SOLUTION: In a table recognition method executed by a table recognition device as a computer such as a server, a plurality of character strings respectively arrayed in a first direction are extracted from document image data 130. A reference point P is set at each of the plurality of character strings, and the plurality of character strings are divided into a plurality of groups G#1 to G#4 on the basis of coordinates of the reference points in a second direction orthogonal to the first direction. Two or more groups in which the number of the character strings is the same and differences between the coordinates of the reference points in the first direction are a predetermined value or less among the plurality of groups are recognized as a table, so that table data 131 is generated.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a table recognition method. [Background technology]

[0002] For example, Patent Document 1 describes how a table can be recognized by registering groups of characters whose coordinates in the arrangement direction overlap as columns. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-331763 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the above method, even if a character group other than a table has a longer length in the arrangement direction than other character groups, the coordinates may overlap with the character group in the table, which may result in the table being mistakenly recognized.

[0005] SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned problems, and has as its object to provide a table recognition method that can recognize tables with high accuracy. [Means for solving the problem]

[0006] The table recognition method of the present invention includes extracting a plurality of character strings arranged in a first direction from an image, and From one end, the other end, and the center in the first direction Reference point Select The method is executed by a computer to divide the plurality of character strings into a plurality of groups based on the coordinates of the reference points in a second direction perpendicular to the first direction, and recognize as tables two or more groups among the plurality of groups in which the number of character strings is the same and the difference between the coordinates of the reference points in the first direction is less than a predetermined value.

[0008] In the above table recognition method, in the process of recognizing the two or more groups as a table, if there are no two or more groups among the multiple groups in which the number of character strings is the same and the coordinates of the reference point in the first direction are the same, the reference point may be switched to an unselected position between the end and the center of the character string in the first direction.

[0009] In the above table recognition method, in the process of extracting the multiple character strings, the multiple character strings may be extracted from the image using an extraction model that has been machine-learned through supervised learning, which takes the image as input and outputs characters in the image.

[0010] In the table recognition method, in the process of extracting the plurality of character strings, the plurality of character strings may be extracted based on distances between characters in the image along the first direction. [Effects of the Invention]

[0011] According to the present invention, tables can be recognized with high accuracy. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a configuration diagram showing an example of a table recognition device. [Figure 2] FIG. 2 is a diagram showing an example of setting the reference point. [Figure 3] FIG. 3 shows an example of successful table recognition. [Figure 4] FIG. 4 shows an example of a table recognition failure. [Figure 5] FIG. 5 shows an example of successful table recognition when the character strings in the table are centered. [Figure 6] FIG. 6 is a flowchart showing an example of the table recognition process. DETAILED DESCRIPTION OF THE INVENTION

[0013] (Configuration of table recognition device) Figure 1 is a configuration diagram showing an example of a table recognition device 1. The table recognition device 1 is a computer such as a server. The table recognition device 1 has a CPU (Central Processing Unit) 10, a ROM (Read Only Memory) 11, a RAM (Random Access Memory) 12, an HDD (Hard Disk Drive) 13, and a communication port 14. The CPU 10 is electrically connected to the ROM 11, the RAM 12, the HDD 13, and the communication port 14 via a bus 19 so that signals can be input and output between them.

[0014] The ROM 11 stores a program that drives the CPU 10. The RAM 12 functions as a working memory for the CPU 10. The communication port 14 is connected to a communication network NW such as the Internet or a LAN (Local Area Network). The communication port 14 is, for example, a wireless LAN (Local Area Network) card or a NIC (Network Interface Card), and performs communication processing between the CPU 10 and a terminal 9 such as a personal computer.

[0015] The HDD 13 also stores document image data 130 and table data 131. The document image data 130 is an example of an image. The document image data 130 is an image file converted from a document file such as a PDF (Portable Document Format). The table data 131 is an image file or document file of a table recognized from the document image data 130. The CPU 10 recognizes the table from the document image data 130 and generates the table data 131.

[0016] When the CPU 10 reads the program from the ROM 11, it forms, as software functions, an operation control unit 100, a character string extraction unit 101, a reference point setting unit 102, a group generation unit 103, and a table recognition unit 104. Note that the operation control unit 100, the character string extraction unit 101, the reference point setting unit 102, the group generation unit 103, and the table recognition unit 104 may be realized by hardware such as an LSI.

[0017] The operation control unit 100 controls the operation of the table recognition device 1. The operation control unit 100 receives a document file from the terminal 9 via, for example, the communication port 14, converts it into document image data 130, and stores it in the HDD 13. The operation control unit 100 instructs the character string extraction unit 101, the reference point setting unit 102, the group generation unit 103, and the table recognition unit 104 to operate in accordance with a predetermined sequence defined by a program.

[0018] The character string extraction unit 101 extracts a plurality of character strings arranged in a predetermined arrangement direction from the document image data 130. The character string extraction unit 101 uses, for example, optical character recognition (OCR) using AI (artificial intelligence). For example, the character string extraction unit 101 extracts a plurality of character strings from the document image data 130 using an extraction model that has been machine-learned through supervised learning and that takes the document image data 130 as input and outputs the characters in the image. This increases the accuracy of character string extraction compared to when AI is not used.

[0019] The extraction model used by the character string extraction unit 101 is generated by machine learning such as deep learning using training data. Specifically, the extraction model is a neural network that mathematically models human brain functions, and is constructed by determining the weight coefficients of the activation functions of parts corresponding to neurons based on the correlation between an image as input and characters as output. Note that the characters are not limited to Japanese characters, and may be characters of other languages ​​such as English.

[0020] Furthermore, the character string extraction unit 101 extracts multiple character strings based on the distance between characters, and therefore, the character string extraction unit 101 can extract character strings by recognizing the boundaries between character strings using the spaces between characters as starting points.

[0021] The reference point setting unit 102 sets a reference point for each extracted character string. The reference point is used to identify character strings included in a table. The reference point setting unit 102 selects a reference point from the right end, left end, or center of the character string arrangement direction. Because the reference point can be selected in this way, it is possible to recognize a table regardless of whether each character string in the table is left-justified, right-justified, or center-justified with respect to the table lines.

[0022] The reference point setting unit 102 may select a reference point by receiving a selection instruction from the terminal 9, for example, via the communication port 14, or may sequentially switch the selection of reference points when the table cannot be recognized, as described below.

[0023] The group generation unit 103 generates groups from a plurality of character strings based on the reference point. The group generation unit 103 divides the plurality of character strings into a plurality of groups based on the coordinates of the reference point in an orthogonal direction perpendicular to the arrangement direction. In this way, the group generation unit 103 selects character string candidates corresponding to rows of the table. Note that the arrangement direction is an example of a first direction, and the orthogonal direction is an example of a second direction.

[0024] The table recognition unit 104 recognizes, as a table, two or more groups among the multiple groups that have the same number of character strings and whose reference point coordinates in the array direction are equal to or less than a predetermined value. First, the table recognition unit 104 identifies two or more groups that contain the same number of character strings from the multiple groups, and excludes other groups with a different number of character strings from the candidates for table rows. Therefore, for example, if there is a group that contains non-table character strings that are longer and have more characters than the character strings in the table, that group can be excluded from the candidates for table rows.

[0025] Next, the table recognition unit 104 identifies the character strings that correspond to the columns of the table by checking the coordinates of the reference points in the arrangement direction for each group of character strings. In this way, the table recognition unit 104 determines the character strings that make up the rows and columns of the table, and can recognize the table with high accuracy.

[0026] (Example of setting the reference point) 2 is a diagram showing an example of setting the reference point P. In the following example, a table included in the document image data 130 is given, with the fields "Name," "Age," "Sex," "Occupation," and "Location," and containing data on two people, "Toyota Taro" and "Yamada Hanako." Each character string in the table is aligned left-justified with respect to the ruled line L that separates the columns. In this example, the left-right direction corresponds to the arrangement direction of each character string.

[0027] The document image data 130 also includes a character string other than the table that reads, "The following 30 applicants have been accepted. Please confirm." The left end (beginning) of this character string matches the left end of the leftmost column of character strings in the table.

[0028] Furthermore, the character string extraction unit 101 extracts character strings based on the spacing S between adjacent characters in the arrangement direction of the character string. For example, the character string extraction unit 101 separates and identifies character strings by detecting spacing S that is equal to or larger than the size of one character.

[0029] Symbol G1a indicates a case where the reference point setting unit 102 sets the left end of each character string as the reference point P. The reference point P is set at the left end of the arrangement direction of each character string and at the center in the height direction. In this example, each character string is arranged in a left-justified style, so the table recognition unit 104 can recognize a table using the reference point P at the left end, as will be described later.

[0030] Symbol G1b indicates a case where the reference point setting unit 102 sets the center of each character string as the reference point P. The reference point P is set at the center of the arrangement direction and the center of the height direction of each character string. In this example, each character string is arranged in a left-justified style, so the table recognition unit 104 cannot recognize a table using the reference point P in the center, as will be described later.

[0031] Symbol G1c indicates a case where the reference point setting unit 102 sets the right end of each character string as the reference point P. The reference point P is set at the right end of each character string in the arrangement direction and at the center in the height direction. In this example, each character string is arranged in a left-justified style, so the table recognition unit 104 cannot recognize a table using the reference point P at the right end, as will be described later. Note that the reference point P is not limited to the above three points, and may be set at the upper right corner or the upper left corner of each character string, for example.

[0032] (Successful example of table recognition) Figure 3 shows an example of successful table recognition. In this example, a procedure for recognizing a table from a reference point P on the left edge indicated by symbol G1a in Figure 2 will be described. In the following description, the arrangement direction will be referred to as the X direction, and the direction perpendicular to the arrangement direction will be referred to as the Y direction.

[0033] First, the group generation unit 103 detects the Y coordinate of the reference point P of each character string, as indicated by symbol G2a. The group generation unit 103 selects character string candidates corresponding to table rows by dividing the character strings into groups G#1 to G#4 for each Y coordinate. Specifically, the group generation unit 103 recognizes character strings whose difference in the Y coordinate of the reference point P is within a predetermined value as being in the same group G#1 to G#4.

[0034] For example, the character strings "Yamada Hanako," "45 years old," "female," "self-employed," and "Osaka department" have a Y coordinate of Y1 at the reference point P and belong to group G#1. The character strings "Toyota Taro," "50 years old," "male," "company employee," and "Tokyo" have a Y coordinate of Y2 at the reference point P and belong to group G#2. The character strings "Name," "Age," "Gender," "Occupation," and "Residence" have a Y coordinate of Y3 at the reference point P and belong to group G#4. Furthermore, the character string other than the table, "The following 30 people are applying. Please confirm," has a Y coordinate of Y4 at the reference point P and belongs to group G#4.

[0035] In this way, the group generating unit 103 divides the multiple character strings into multiple groups G#1 to G#4 based on the Y coordinate of the reference point P. This allows the selection of character string candidates corresponding to the rows of the table. Note that the Y coordinate is an example of the coordinate of the reference point P in the orthogonal direction.

[0036] Next, table recognition unit 104 counts the number of character strings in each of groups G#1 to G#4 (see "Number of Character Strings"), as indicated by symbol G3a. Since the number of character strings in groups G#1 to G#3 is 5 and the number of character strings in group G#4 is 1, table recognition unit 104 excludes the character strings in group G#4 from the candidates for table rows.

[0037] Because the length (number of characters) of the strings in group G#4 is longer than the lengths of the strings in the other groups G#1 to G#3, the strings in group G#4 overlap with the strings in the other groups G#1 to G#3 in the X direction. Therefore, if table recognition unit 104 were to recognize strings whose coordinates in the X direction overlap as columns of a table, as in Patent Document 1, it would erroneously recognize the strings in group G#4 as part of a table, even though it is not a table. However, table recognition unit 104 can accurately recognize a table by excluding group G#4, whose number of character strings does not match those of groups G#1 to G#3, from row candidates.

[0038] Next, the table recognition unit 104 identifies character strings that correspond to table columns by comparing the X coordinates of the reference points P of each character string in groups G#1 to G#3 that have the same number of character strings. Specifically, the table recognition unit 104 detects the X coordinates of the reference points P of each character string in groups G#1 to G#3, sequentially starting with the character string in the leftmost column, for example, and determines whether the difference between the X coordinates for each column is equal to or less than a predetermined value. In this example, the difference between the X coordinates for character strings in all columns is equal to or less than the predetermined value, so the table recognition unit 104 recognizes each character string in groups G#1 to G#3 as a table. On the other hand, if groups G#1 to G#3 include character strings whose difference between the X coordinates of the reference points P exceeds the predetermined value, the table recognition unit 104 cannot recognize a table.

[0039] In this way, the table recognition unit 104 determines that the difference between the X coordinates of the reference points P of the multiple character strings belonging to each of the groups G#1 to G#3 is equal to or less than a predetermined value. As a result, the table recognition unit 104 determines that the X coordinates of the reference points P of the multiple character strings belonging to each of the groups G#1 to G#3 are aligned, and recognizes the multiple character strings belonging to each of the groups G#1 to G#3 as table rows.

[0040] Specifically, the X coordinate of each reference point P of "Name," "Toyota Taro," and "Yamada Hanako" is X1, and the X coordinate of each reference point P of "Age," "50 years old," and "45 years old" is X2. The X coordinate of each reference point P of "Sex," "Male," and "Female" is X3, the X coordinate of each reference point P of "Occupation," "Company employee," and "Self-employed" is X4, and the X coordinate of each reference point P of "Residence," "Tokyo," and "Osaka Prefecture" is X5. Therefore, the table recognition unit 104 determines that the X coordinates of the reference points P of the character strings in each group G#1 to G#3 are aligned with each other at X1 to X5. Note that the X coordinates are an example of the coordinate of the reference point P in the arrangement direction.

[0041] Next, the table recognition unit 104 generates table data 131 from the character strings of each group G#1 to G#3, as indicated by the symbol G4a. Specifically, the table recognition unit 104 generates table data 131 of a 3×5 table by taking X1 to X5, which are the X coordinates of the reference point P, as column positions and Y1 to Y3, which are the Y coordinates of the reference point P, as column positions. Here, the table data 131 may be image data or text data. The table recognition unit 104 displays the table data 131 on a display (not shown) or the like, or transmits it to the terminal 9 via the communication network NW. This allows the user to check the table in the document image data 130.

[0042] In this way, table recognition unit 104 recognizes as a table two or more groups G#1 to G#3 that have the same number of character strings and the same X coordinate of reference point P. For this reason, table recognition unit 104 can recognize tables with high accuracy.

[0043] (Example of table recognition failure) 4 is a diagram showing an example of a table recognition failure. Symbol G2b shows a case where the reference point P is in the center of each character string. Since each character string in groups G#1 to G#3 is aligned left, if there is a character string with a different number of characters among groups G#1 to G#3, the X coordinate of the reference point P will also be different.

[0044] For example, "Name" has two characters, while "Toyota Taro" and "Yamada Hanako" have four characters. Therefore, X1, the X coordinate of the reference point P of "Name", is located to the left of X2, the X coordinate of the reference points P of "Toyota Taro" and "Yamada Hanako". Here, the difference between X1 and X2 is greater than the above-mentioned predetermined value. Therefore, even though "Name", "Toyota Taro", and "Yamada Hanako" are in the same column in the table, the table recognition unit 104 cannot recognize the table because the X coordinates of their reference points P are different.

[0045] Furthermore, the X coordinates X3, X6, and X8 of the reference points P for "age," "occupation," and "residence" are different from the X coordinates X4, X7, and X9 of the other character strings in the same column. Note that "gender," "male," and "female" each contain the same number of characters, two, so the X coordinate is also the same, X3. However, the table recognition unit 104 cannot recognize a table unless the X coordinates of the reference points P for all the character strings in groups G#1 to G#3 are substantially the same.

[0046] Moreover, symbol G2c indicates a case where the reference point P is at the right end of each character string. Since the character strings of groups G#1 to G#3 are arranged left-justified, in this case as well, the X coordinates of the reference points P of the character strings of groups G#1 to G#3 do not align, and table recognition unit 104 cannot recognize the table.

[0047] Therefore, if the table recognition unit 104 fails to recognize a table, the reference point setting unit 102 switches the reference point P to an unselected position among the right end, left end, and center in the arrangement direction of the character string. For example, if the reference point setting unit 102 initially sets the center of each character string as the reference point P, table recognition will fail, so the reference point setting unit 102 newly sets the right end of each character string as the reference point P. In this case, table recognition will also fail, so the reference point setting unit 102 further sets the left end of each character string as the reference point P. This will result in successful table recognition.

[0048] In this way, if there are no two or more groups with the same number of character strings and the same X coordinate of the reference point P, the reference point setting unit 102 switches the reference point P to an unselected position. This makes it possible to recognize the table regardless of the arrangement of the character strings in the table (left-justified, center-justified, right-justified).

[0049] 5 is a diagram showing an example of successful table recognition when character strings in a table are centered. In FIG. 5, a description of the configuration common to FIG. 3 will be omitted.

[0050] As indicated by symbol G3b, the character strings in the table are centered relative to the column ruled line L, so when the reference point setting unit 102 sets the center of each character string as the reference point P, the X coordinates of each character string in groups G#1 to G#3 are aligned to X1 to X5. Therefore, the table recognition unit 104 can recognize the table and generate table data 131, as indicated by symbol G4b.

[0051] (Table Recognition Processing) 6 is a flowchart showing an example of table recognition processing. This processing is an example of a table recognition method, and is executed when a document file to be subjected to table recognition processing is input to the CPU 10, for example.

[0052] First, the operation control unit 100 generates document image data 130 from a document file (step St1). The document image data 130 is stored in the HDD 13. Next, the character string extraction unit 101 extracts a plurality of character strings from the document image data 130 (step St2).

[0053] Next, the reference point setting unit 102 sets a reference point P for each character string (step St3). The reference point P is set at any one of the left end, the center, and the right end of the character string.

[0054] Next, the group generation unit 103 generates groups of character strings based on the Y coordinate of the reference point P (step St4). If the number of groups is one (No in step St5), no table exists in the document image data 130, and this process ends.

[0055] If there are multiple groups (Yes in step St5), the table recognition unit 104 performs table recognition based on the number of character strings in each group and the X coordinate of the reference point P of each character string (step St6). As described above, the table recognition unit 104 compares the number of character strings in each group, selects two or more groups with the same number of character strings as candidates for table rows, and compares the X coordinates of the reference points P of the character strings in each group corresponding to the columns.

[0056] If table recognition is successful (Yes in step St7), the table recognition unit 104 generates table data 131 (step St8). If table recognition is unsuccessful (No in step St7), the reference point setting unit 102 determines whether there are any unselected reference points P at the left end, center, or right end of the character string (step St9). If there are any unselected reference points P (Yes in step St9), the reference point setting unit 102 sets the unselected reference points P (step St10). Thereafter, the processes from step St4 onwards are executed again.

[0057] If there is no unselected reference point P (No in step St9), table recognition is not possible, and this process ends. In this way, the table recognition process is performed.

[0058] In this example, a list is used as a table, but the table recognition process is not limited to this and can also be applied to, for example, a specification table published in an automobile owner's manual. Also, in this example, a horizontally written table is used, but even in the case of a vertically written table, it is possible to recognize the table by switching the X and Y directions and executing the table recognition process.

[0059] The above-described embodiment is a preferred example of the present invention, but the present invention is not limited to this and can be modified in various ways without departing from the spirit of the present invention. [Explanation of symbols]

[0060] 1 table recognition device, 10 CPU, 101 character string extraction unit, 102 reference point setting unit, 103 group generation unit, 104 table recognition unit, 130 document image data

Claims

1. extracting a plurality of character strings arranged in a first direction from the image; selecting reference points from one end, the other end, and a center in the first direction of each of the plurality of character strings; Dividing the plurality of character strings into a plurality of groups based on coordinates of the reference points in a second direction perpendicular to the first direction; Two or more groups among the plurality of groups, in which the number of character strings is the same and the difference between the coordinates of the reference points in the first direction is equal to or less than a predetermined value, are recognized as a table. The processing is performed by a computer. Table recognition method.

2. in the process of recognizing the two or more groups as a table, if there are no two or more groups among the plurality of groups in which the number of character strings is the same and the coordinates of the reference points in the first direction are aligned, the reference points are switched to an unselected position between the end portion and the center portion of the character strings in the first direction. The table recognition method according to claim 1 .

3. In the process of extracting the plurality of character strings, the image is input, and characters in the image are output, and the plurality of character strings are extracted from the image using an extraction model that has been machine-learned through supervised learning.

3. The table recognition method according to claim 1 or 2.

4. In the process of extracting the plurality of character strings, the plurality of character strings are extracted based on distances between characters in the image along the first direction.

3. The table recognition method according to claim 1 or 2.

Images