Table recognition device

Abstract

To provide a table recognition device that recognizes a table with high precision.SOLUTION: In a table recognition device 1, a CPU 10 functions as: a frame identification unit which reads a program out of a ROM 11 and identifies a plurality of frames from an image; an endpoint detection unit which detects endpoints of respective frame lines of the plurality of identified frames; a group generation unit which compares coordinates of detected endpoints with each other between the plurality of identified frames to generate a group of frames having inter-endpoint distances equal to or less than a threshold; and a table recognition unit which recognizes tables by generated groups.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a table recognition device. [Background technology]

[0002] Regarding technology for recognizing tables from within an image, for example, Patent Document 1 describes a technique for setting rectangular subgroups in the row and column directions so as to surround multiple character strings arranged in the row and column directions of an image, and recognizing the intersection range as an entry frame for a form. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-179743 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned technology cannot recognize boxes that have a width corresponding to multiple columns or a height corresponding to multiple rows, and therefore cannot recognize tables with high accuracy.

[0005] SUMMARY OF THE PRESENT DISCLOSURE The present invention has been made in consideration of the above problems, and has an object to provide a table recognition device that can recognize tables with high accuracy. [Means for solving the problem]

[0006] The table recognition device of the present invention has an identification unit that identifies a plurality of frames from an image, a detection unit that detects the end points of each of the plurality of frames, a generation unit that generates groups of frames whose end points are spaced apart by a threshold value by comparing the coordinates of the end points between the plurality of frames, and a recognition unit that recognizes a table for each of the groups.

[0007] In the above table recognition device, the identification unit may identify the multiple frames defined by the ruled lines from the image using an output model trained on machine learning through supervised learning, which takes the image as input and outputs the ruled lines within the image.

[0008] In the above table recognition device, the identification unit may identify the multiple frames surrounding the character strings from the image using an output model that has been machine-learned through supervised learning, which takes the image as input and outputs character strings within the image.

[0009] In the above table recognition device, when the ruled lines are interrupted, the identification unit may identify the plurality of boxes by completing the interrupted portions of the ruled lines.

[0010] In the above table recognition device, the identification unit may identify, as the multiple frames, a multiple first frames surrounding characters in the image and a multiple second frames defined by lines in the image, the detection unit may detect the endpoints of each of the frame lines of the multiple first frames and the endpoints of each of the frame lines of the multiple second frames, and the generation unit may generate the group of the first frames whose endpoints are spaced apart by a distance less than or equal to the threshold value, and generate the group of the second frames whose endpoints are spaced apart by a distance less than or equal to the threshold value. Effect of the Invention

[0011] According to the present invention, tables can be recognized with high accuracy. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a configuration diagram showing an example of a table recognition device. [Diagram 2] FIG. 2 is a plan view showing an example of the table recognition process. [Diagram 3] FIG. 3 shows an example of a process for generating a plurality of groups. [Figure 4] FIG. 4 is a flowchart showing an example of the table recognition process. [Diagram 5]FIG. 5 is a flowchart showing an example of the group generation process. [Figure 6] FIG. 6 is a plan view of a table showing an example of identifying a frame from a character string. [Figure 7] FIG. 7 is a flowchart showing an example of a process for recognizing a table based on both ruled lines and character strings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] (Configuration of table recognition device) 1 is a configuration diagram showing an example of a table recognition device 1. The table recognition device 1 is a computer such as a server. The table recognition device 1 has a CPU (Central Processing Unit) 10, a ROM (Read Only Memory) 11, a RAM (Random Access Memory) 12, a HDD (Hard Disk Drive) 13, and a communication port 14. The CPU 10 is electrically connected to the ROM 11, the RAM 12, the HDD 13, and the communication port 14 via a bus 19 so as to be able to input and output signals to and from each of them.

[0014] The ROM 11 stores a program that drives the CPU 10. The RAM 12 functions as a working memory for the CPU 10. The communication port 14 is connected to a communication network NW such as the Internet or a LAN (Local Area Network). The communication port 14 is, for example, a wireless LAN (Local Area Network) card or a NIC (Network Interface Card), and performs communication processing between the CPU 10 and a terminal 9 such as a personal computer.

[0015] HDD 13 also stores image data 130 and table data 131. Image data 130 is an example of an image. Image data 130 is an image file converted from a document file such as PDF (Portable Document Format). Table data 131 is an image file or document file of a table recognized from image data 130. CPU 10 recognizes the table from image data 130 and generates table data 131.

[0016] When the CPU 10 reads a program from the ROM 11, it forms, as software functions, an operation control unit 100, an image acquisition unit 101, a frame identification unit 102, an end point detection unit 103, a group generation unit 104, and a table recognition unit 105. Note that the operation control unit 100, the image acquisition unit 101, the frame identification unit 102, the end point detection unit 103, the group generation unit 104, and the table recognition unit 105 may be realized by hardware such as an IC (Integrated Circuit).

[0017] The operation control unit 100 controls the operation of the table recognition device 1. The operation control unit 100 instructs the image acquisition unit 101, the box identification unit 102, the end point detection unit 103, the group generation unit 104, and the table recognition unit 105 to operate in accordance with a predetermined sequence defined by a program. When the operation control unit 100 receives a table recognition request from the terminal 9 via the communication network NW, for example, it starts instructing each operation.

[0018] The image acquisition unit 101 receives a document file from the terminal 9 via the communication port 14 , for example, converts it into image data 130 , and stores it in the HDD 13 .

[0019] The box identification unit 102 is an example of an identification unit. The box identification unit 102 identifies a plurality of boxes forming a table in the image from the image data 130. The box identification unit 102 extracts, for example, ruled lines of a table in the image, and identifies, for each area surrounded by a plurality of ruled lines, a part of each ruled line that defines the area as a box line. The box identification unit 102 also extracts character strings within the boxes.

[0020] The frame identification unit 102 extracts a plurality of character strings and a plurality of ruled lines arranged in the vertical or horizontal direction on the paper surface of the document from the image data 130. The frame identification unit 102 uses, for example, optical character recognition (OCR) using artificial intelligence (AI). For example, the frame identification unit 102 extracts a plurality of character strings and a plurality of ruled lines from the image data 130 using an output model that has been machine-learned through supervised learning, which takes the image data 130 as input and outputs the character strings and ruled lines within the image. Since the frame identification unit 102 identifies a plurality of frames defined by ruled lines, the accuracy of frame identification is higher than when AI is not used.

[0021] The output model used by the frame identification unit 102 is generated by machine learning such as deep learning using teacher data. Specifically, the output model is a neural network that mathematically models human brain functions, and is constructed by determining the weight coefficients of the activation function of the parts corresponding to neurons based on the correlation between an image as input and characters and ruled lines as output. Note that the characters are not limited to Japanese characters, and may be characters of other languages ​​such as English.

[0022] Furthermore, the box identification unit 102 extracts a plurality of character strings based on the distance between characters, and therefore the box identification unit 102 can extract character strings by recognizing boundaries between character strings using white spaces between characters as starting points.

[0023] The end point detection unit 103 is an example of a detection unit. The end point detection unit 103 detects the end points of the frame lines of each frame. For example, the end point detection unit 103 separates a frame into a plurality of frame lines and detects the coordinates in the image of each end point located at both ends of each frame line. For example, in the case of a rectangular frame, the end points are located at the four corners of the rectangle.

[0024] The group generating unit 104 is an example of a generating unit. The group generating unit 104 generates a group of frames whose end points are separated by a threshold or less by comparing the coordinates of the end points of the frames. For example, in the case of rectangular frames, rectangular frames whose corners are close to each other belong to a common group. In this way, multiple frames arranged vertically and horizontally in a table are treated as one group.

[0025] Table recognition unit 105 is an example of a recognition unit. Table recognition unit 105 recognizes a table for each group. Table recognition unit 105 stores the table as table data 131 in HDD 13 and outputs it to terminal 9 via communication network NW.

[0026] 2 is a plan view showing an example of table recognition processing. Symbol Ga indicates an example of image data 130, and symbol Gb indicates an example of a process for generating groups. The image data 130 includes an event participant list TD as an example of a table. In FIG. 2, the horizontal direction of the paper is the X direction, and the vertical direction is the Y direction.

[0027] The participant list TD lists participant information by event date and location. Participant information includes name, age, and gender arranged in the X direction. The event dates are "Saturday, August 12th" and "Sunday, August 13th", and the event locations are "Shiodome Venue" and "Makuhari Venue". Participant information is grouped by date in the X direction and by location in the Y direction.

[0028] As indicated by the symbol Ga, the frame identification unit 102 extracts ruled lines Lx, Ly and character strings from the participant list TD. The ruled lines Lx extend in the X direction, and the ruled lines Ly extend in the Y direction. The frame identification unit 102 identifies a plurality of frames f defined by the ruled lines Lx, Ly. The frame identification unit 102 identifies frames f by detecting a rectangular area surrounded by a pair of adjacent ruled lines Lx and a pair of adjacent ruled lines Ly. Therefore, the vertical and horizontal frame lines of each frame f correspond to a part of the ruled lines Lx, Ly. Note that, when a ruled line is interrupted, the frame identification unit 102 identifies the frame f by completing the interrupted portion of the ruled line. The frame identification unit 102 also extracts character strings within each frame f.

[0029] The end point detection unit 103 detects four end points, indicated by white circles, for each frame as indicated by the reference symbol Gb. The four end points correspond to the four corners of a rectangular region.

[0030] The group generating unit 104, for example, selects one frame f0 from all frames to generate a group. The group generating unit 104 compares the coordinates of the end points between the frame f0 that belongs to the group and the other frames that do not belong to the group, and adds the other frames f1 to f4 whose distances between the end points are equal to or less than a threshold to the group. Here, the distance is calculated from the coordinates of each frame as, for example, a straight-line distance between the end points. Also, the threshold may be, for example, 0.5 points when the size of characters in a table is expressed in points.

[0031] For example, the distance between the end points p1 and p3 of the frame f0 and the end points p11 and p13 of the frame f1 is less than or equal to the threshold, so the frame f1 is added to the group. Here, the end points p1 and p11, and the end points p3 and p13 correspond to each other in the participant list TD. However, depending on the width of the ruled lines Lx and Ly, the respective coordinates do not necessarily match, so the presence or absence of a correspondence relationship in the participant list TD is determined by comparing the distance between the end points p1 and p11, and the distance between the end points p3 and p13 with the threshold.

[0032] Furthermore, the distance between end point p2 of frame f0 and end point p32 of frame f2 is less than the threshold, so frame f2 is added to the group. Furthermore, the distance between end point p3 of frame f0 and end point p33 of frame f3 and end point p43 of frame f4 are less than the threshold, so frames f3 and f4 are added to the group. In this way, the group generation unit 104 generates a group to which frames f0 to f4 with mutually adjacent end points p1 to p3, p11, p13, p32, and p33 belong.

[0033] Furthermore, the group generating unit 104 adds to the group the frames f5 to f7 whose end points are less than or equal to the threshold distance from the frame f2 belonging to the group. In this way, every time the group generating unit 104 adds frames f1 to f7 to the group, it compares the coordinates of the end points of the frames f1 to f7 that belong to the group with the coordinates of the end points of other frames that do not belong to the group. By repeating this process, a group that matches the frame f in the participant list TD is generated.

[0034] (Multiple group generation process) Fig. 3 shows an example of a process for generating a plurality of groups G1 and G2. In Fig. 3, white circles indicate the end points of a frame f. The group generation unit 104 generates a total set S including all frames identified from the image data 130. The group generation unit 104 selects one frame f10 from the total set S and adds it to the group G1. The group generation unit 104 compares the coordinates of each end point of the frame f10 in the group G1 with the coordinates of each end point of the other frames f in the total set S.

[0035] The group generation unit 104 adds to group G1 all frames f in the entire set S whose end point distances are equal to or less than the threshold. The group generation unit 104 compares the coordinates of each end point of the frame f added to group G1 with the coordinates of each end point of the other frames f in the entire set S, and adds to group G1 all frames f in the entire set S whose end point distances are equal to or less than the threshold, in the same manner as described above. The group generation unit 104 repeats adding frames f to group G1 until there are no frames f in the entire set S whose end point distances are equal to or less than the threshold.

[0036] Next, the group generation unit 104 selects one frame f20 from the entire set S and adds it to the new group G2. As with group G1, the group generation unit 104 compares the coordinates of each end point of the frame f10 in group G2 with the coordinates of each end point of the other frames f in the entire set S, and repeats adding the frames f to group G2 until there are no frames f in the entire set S whose inter-end point distances are less than or equal to the threshold.

[0037] In this way, the group generating unit 104 generates groups G1 and G2 of frames f whose end point distances are equal to or less than a threshold value by comparing the coordinates of the end points between the frames f. The table recognizing unit 105 recognizes tables for each of the groups G1 and G2. Note that, although this example shows an example in which the image data 130 contains two tables, groups are generated and tables are recognized in the same manner as above even when three or more tables are present.

[0038] (Table Recognition Processing) 4 is a flowchart showing an example of the table recognition process. This process is executed when the operation control unit 100 receives a request for table recognition from the terminal 9 via the communication network NW, for example.

[0039] First, the image acquisition unit 101 acquires a document file from the terminal 9 via the communication network NW (step St1). The document file is stored in the HDD 13 as image data 130.

[0040] Next, the frame identification unit 102 extracts character strings and lines from the image of the image data 130 by OCR using AI (step St2). Next, the frame identification unit 102 determines whether or not there is a break r in the ruled line Lx, as indicated by the symbol Gh (step St3). For example, the frame identification unit 102 determines that there is a break r when the end point t of the ruled line Lx is located at a position other than the intersection of the vertical and horizontal ruled lines Lx and Ly, or when the ruled line Lx is faint. The break r in the vertical ruled line Ly is also determined in a similar manner.

[0041] If there is a break r in the ruled lines Lx, Ly (Yes in step St3), the box identification unit 102 complements the break r (step St4). This enables the box identification unit 102 to extract the ruled lines Lx, Ly with high accuracy and identify the box. If there is no break r (No in step St3), the processing in step St4 is not executed.

[0042] Next, the box identification unit 102 identifies the boxes from the ruled lines Lx and Ly (step St5). Next, the end point detection unit 103 detects the end points of each box (step St6). Next, the group generation unit 104 generates a group of boxes whose end point distances are equal to or less than a threshold (step St7). The group generation process will be described in detail later.

[0043] Next, the table recognition unit 105 recognizes a table for each group (step St8) and generates table data 131 (step St9). In this manner, the table recognition process is executed.

[0044] 5 is a flowchart showing an example of the group generation process, which is executed in step St7 described above.

[0045] First, the group generation unit 104 generates a total set S including all frames (step St10). Next, the group generation unit 104 selects one frame from the total set S (step St11) and adds it to a group Gi (i=1, 2, . . . ) (step St12).

[0046] Next, the group generation unit 104 selects another frame from the entire set S (step St13) and compares the coordinates of the end points with the frames in group Gi (step St14). Next, the group generation unit 104 determines whether or not there is a pair of end points whose distance is less than or equal to the threshold (step St15). If there is a pair of end points whose distance is less than or equal to the threshold (Yes in step St15), the group generation unit 104 adds the selected frame to group Gi (step St16). If there is no pair of end points whose distance is less than or equal to the threshold (No in step St15), the process of step St16 is not executed.

[0047] Next, the group generating unit 104 determines whether or not there are any frames in the entire set S whose coordinates between their end points have not been compared (hereinafter, referred to as uncompared frames) (step St17). If there are any uncompared frames (Yes in step St17), the processes in and after step St13 are executed again.

[0048] If there are no uncompared frames (No in step St17), the group generation unit 104 determines whether there are multiple frames in the entire set S (step St18). That is, the group generation unit 104 determines whether there are multiple frames remaining that have not been grouped.

[0049] If there are multiple frames for the entire set S (Yes in step St18), the processes from step St11 onwards are executed again. This causes the generation process of a new group Gi to be performed. At this time, although not shown in the figure, i is incremented by 1. If there are no frames for the entire set S (No in step St18), this process ends. In this manner, the group generation process is executed.

[0050] (Other Examples) The box identification unit 102 identifies multiple boxes from the ruled lines Lx and Ly, but is not limited to this. The box identification unit 102 may extract a character string from an image and identify multiple boxes surrounding the character string.

[0051] 6 is a plan view of a table showing an example of identifying a box from a character string. The box identification unit 102 identifies a plurality of boxes surrounding a character string in the participant list TD. At this time, as described above, the box identification unit 102 extracts the character string using an output model that has been machine-learned by supervised learning, which receives the image data 130 as input and outputs the character string in the image data 130. This makes it possible to extract the character string and identify the box with high accuracy.

[0052] When identifying a frame from a character string, the identification result may differ from when identifying a frame from ruled lines Lx, Ly. In this example, a frame fs surrounding the character string "Participant List" is identified, but when identifying a frame from ruled lines Lx, Ly, it is not identified. Also, in this example, since there is no character string in the blank space where the horizontal date column and the vertical location column intersect, the blank space is not identified as a frame, but when identifying a frame from ruled lines Lx, Ly, it is identified.

[0053] Therefore, the table recognition device 1 may recognize each table based on both the ruled lines Lx, Ly and the character strings, and if there is a difference between the tables, output any one of the tables.

[0054] Fig. 7 is a flow chart showing an example of a process for recognizing a table based on both the ruled lines Lx and Ly and the character strings. First, the table recognition device 1 recognizes a table based on the ruled lines Lx and Ly as shown in Fig. 2 (step St21). Next, the table recognition device 1 recognizes a table based on the character strings as shown in Fig. 6 (step St22).

[0055] In steps St21 and St22, the box identification unit 102 identifies a plurality of boxes (first boxes) surrounding characters in the image, and a plurality of boxes (second boxes) defined by ruled lines in the image. The end point detection unit 103 detects end points of the frame lines of each box based on the ruled lines Lx, Ly, and each box based on the character string. The group generation unit 104 generates groups of boxes whose end points are separated by a distance equal to or less than the threshold value for each box based on the ruled lines Lx, Ly, and each box based on the character string.

[0056] Next, the table recognition unit 105 compares the tables recognized in steps St21 and St22 (step St23). If all the tables match each other (Yes in step St24), the table recognition unit 105 transmits the table data 131 to the terminal 9 (step St25).

[0057] If there is a mismatched table (No in step St24), the table recognition unit 105 transmits the matching table to the terminal 9 (step St26). Next, the table recognition unit 105 transmits the mismatched table to the terminal 9 (step St27). This allows the user to select the correct table by checking the terminal 9.

[0058] The above-described embodiment is a preferred example of the present invention, but the present invention is not limited to this embodiment and can be modified in various ways without departing from the scope of the present invention. [Explanation of symbols]

[0059] 1 table recognition device, 10 CPU, 102 frame identification unit, 103 end point detection unit, 104 group generation unit, 105 table recognition unit, 130 image data

Claims

1. An identification unit that identifies a plurality of frames from an image; A detection unit that detects end points of each of the frame lines of the plurality of frames; a generating unit that generates a group of the frames whose distances between the endpoints are equal to or less than a threshold by comparing the coordinates of the endpoints between the frames; A recognition unit for recognizing a table for each of the groups. Table recognition device.

2. the identification unit receives the image as an input, outputs ruled lines in the image, and identifies the plurality of frames defined by the ruled lines from the image using an output model that has been machine-learned through supervised learning; 2. The table recognition apparatus according to claim 1.

3. the identification unit receives the image as an input, outputs character strings in the image, and identifies the plurality of frames surrounding the character strings from the image using an output model that has been machine-learned through supervised learning; 2. The table recognition apparatus according to claim 1.

4. the identification unit, when the ruled lines are interrupted, identifies the plurality of frames by completing the interrupted portions of the ruled lines.

4. The table recognition apparatus according to claim 3.

5. The identification unit identifies, as the plurality of frames, a plurality of first frames surrounding characters in the image and a plurality of second frames defined by ruled lines in the image; the detection unit detects the end points of a frame line of each of the plurality of first frames and the end points of a frame line of each of the plurality of second frames; the generation unit generates the group of the first frames whose mutual end points are spaced apart by a distance equal to or less than the threshold, and generates the group of the second frames whose mutual end points are spaced apart by a distance equal to or less than the threshold.

2. The table recognition apparatus according to claim 1.

Images