Information processing device, information processing method, and program

Abstract

To provide an information processing device, an information processing method and a program capable of coping with even an atypical business form in reading image data of a table having a nest structure to output information of an item of the nest structure.SOLUTION: An information processing device includes: an acquisition part for acquiring image data of a business form including a table composed of a plurality of cells; an extraction part for extracting a ruled line of the table on the image data; a recognition part for recognizing a plurality of cells of the table; a combination specification part for specifying one or more candidates for a combination of a parent cell and a child cell in a parent-child relation in the plurality of recognized cells; a selection part for selecting a combination to be adopted from among candidates on the basis of a positional relation between the parent cell and the child cell and an arrangement state of the ruled line between the parent cell and the child cell; and an output part for outputting information showing the parent cell and information showing the child cell on the basis of the selected combination.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Conventionally, there is a technology called OCR (Optical Character Recognition) that analyzes image data of a form, recognizes the table structure of the form and each item contained in the table, and outputs information about each item. Furthermore, many forms employ tables with a so-called nested structure, in which multiple areas (hereinafter, these areas are also referred to as "cells") that correspond to multiple item names and constitute the table have a parent-child relationship, and there is also technology that recognizes such nested structures (see, for example, Patent Document 1). With this technology, for example, a cell is extracted, and if the extracted area has a nested structure, the area is considered to be a table and a recursive process of cell extraction is performed to extract the areas that constitute the nested structure. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-317107 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned Patent Document 1, it is necessary to set in advance that the extracted region has a nested structure, which poses a problem that it cannot be used for, for example, non-standard forms.

[0005] Therefore, the present invention aims to provide an information processing device, information processing method, and program that can read image data of a table with a nested structure and output information on the items of this nested structure, even if the form is non-standard. [Means for solving the problem]

[0006] An information processing device according to one embodiment of the present invention includes an acquisition unit that acquires image data of a document including a table composed of multiple cells, an extraction unit that extracts the table's borders from the image data, a recognition unit that recognizes multiple cells in the table, a combination identification unit that identifies one or more candidate combinations of parent cells and child cells in a parent-child relationship from the recognized multiple cells, a selection unit that selects a combination to adopt from the candidates based on the positional relationship between the parent cell and the child cell and the arrangement of the borders between the parent cell and the child cell, and an output unit that outputs information indicating the parent cell and information indicating the child cell based on the selected combination.

[0007] A program according to one embodiment of the present invention provides a computer with the following functions: an acquisition function for acquiring image data of a document including a table made up of multiple cells; an extraction function for extracting the table's borders from the image data; a recognition function for recognizing multiple cells in the table; a combination identification function for identifying one or more candidate combinations of parent cells and child cells in a parent-child relationship from the recognized multiple cells; a selection function for selecting a combination to adopt from the candidates based on the positional relationship between the parent cell and the child cell and the arrangement of the borders between the parent cell and the child cell; and an output function for outputting information indicating the parent cell and information indicating the child cell based on the selected combination.

[0008] In one aspect of the information processing method of the present invention, a computer acquires image data of a document including a table composed of multiple cells, extracts the table lines from the image data, recognizes multiple cells in the table, identifies one or more candidate combinations of parent and child cells in a parent-child relationship among the recognized multiple cells, selects a combination to adopt from the candidates based on the positional relationship between the parent and child cells and the arrangement of the lines between the parent and child cells, and outputs information indicating the parent cell and information indicating the child cell based on the selected combination. [Effects of the Invention]

[0009] According to the present invention, an information processing device, information processing method, and program can be provided that can read image data of a table with a nested structure and output information on the items of this nested structure, even if the form is non-standard. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of the system configuration of a document reading system according to an embodiment of the present invention; [Figure 2] FIG. 2 is a diagram illustrating an example of a functional configuration of a server device according to the present embodiment. [Figure 3] 10A and 10B are diagrams for explaining a process for extracting pixels of candidate ruled lines and pixels of candidate intersections of a form in a server device according to the present embodiment. [Figure 4] 10A and 10B are diagrams for explaining a process for dividing a plurality of linked tables in the server device according to the embodiment. [Figure 5] 10A and 10B are diagrams for explaining a process for recognizing a nested structure in the server device according to the embodiment; [Figure 6] 10A and 10B are diagrams for explaining a process for recognizing a nested structure in the server device according to the embodiment; [Figure 7] 10A and 10B are diagrams for explaining a process for recognizing a nested structure in the server device according to the embodiment; [Figure 8] 10A and 10B are diagrams for explaining a process for recognizing a nested structure in the server device according to the embodiment; [Figure 9] FIG. 10 is a diagram illustrating an example of the operation of the server device according to the embodiment. [Figure 10] FIG. 2 is a diagram illustrating an example of a hardware configuration of a server device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A preferred embodiment of the present invention will be described with reference to the accompanying drawings. In the drawings, components with the same reference numerals have the same or similar configurations.

[0012] In this embodiment, an example will be described in which OCR (Optical Character Recognition) is used to analyze an image of a form read by a scanner or the like, and to recognize the characters on the form (character recognition) and the structure of a table included in the form.

[0013] In this embodiment, an example will be described in which an image of an unconventional paper form is read, and from this image, the item names and item value strings of each item in a table included in the form are identified and converted into electronic text. This "conversion to electronic text" refers to, for example, outputting information read from an image of the form as text data (hereinafter also referred to as "text data"). Note that the data output by the form reading system 1 according to this embodiment is not limited to this text data. Furthermore, "unconventional" here refers to a state in which the format of the target form is not fixed, and unconventional forms also include forms whose layout is not fixed because there are multiple issuing agencies and therefore multiple patterns.

[0014] In this embodiment, an example will be described in which the document reading system 1 evaluates (scores) the validity of at least one of the following (a) to (c) items, which are structures in which one item is represented by multiple item names in a parent-child relationship, i.e., nested structure items, and recognizes them based on the results of this evaluation. (A) The validity of the nesting structure itself (a) The appropriateness of the position within a nested table (c) The appropriateness of the positional relationship with keywords such as table titles and units

[0015] In this embodiment, an area constituting a table included in a form (hereinafter simply referred to as a "table") is also referred to as a "cell." A cell may correspond to, for example, each item included in a table. Cells include cells surrounded by visually recognizable lines and cells that are not surrounded by lines. Specifically, one cell may correspond to one item name or one item value. Each of multiple cells in a parent-child relationship, i.e., a nested structure according to this embodiment, typically corresponds to an item name, but is not limited to this. In the following description, each of multiple cells in a parent-child relationship may be read as an item name. Furthermore, a form may include, for example, a composite table consisting of multiple adjacent tables.

[0016] <1. System configuration> An example of the system configuration of a document reading system 1 according to this embodiment will be described with reference to FIG.

[0017] The form reading system 1 is a system for reading paper forms and converting them into electronic text. As shown in Fig. 1, the form reading system 1 includes a server device 100, a terminal 200 used by a user, and a scanner 300. The server device 100, the terminal 200, and the scanner 300 are connected to each other via a network N.

[0018] The network N is composed of a wireless network or a wired network, examples of which include a mobile phone network, a PHS (Personal Handy-phone System) network, a wireless LAN (Local Area Network), 3G (3rd Generation), LTE (Long Term Evolution), 4G (4th Generation), 5G (5th Generation), WiMax (registered trademark), infrared communication, Bluetooth (registered trademark), a wired LAN, a telephone line, a power line network, and a network conforming to IEEE1394 or the like.

[0019] The server device 100 is an information processing device capable of communicating with the terminal 200 and the scanner 300, and performing image analysis processing, etc. By executing a predetermined program (hereinafter also referred to as a "server program"), the server device 100 cooperates with the scanner 300 to realize functions such as analyzing image data (hereinafter also simply referred to as "image data") of a paper form read by the scanner 300, and identifying the layout of the form table, the item names of multiple items included in the table, and the item values ​​corresponding to these item names.

[0020] The terminal 200 is a smartphone, laptop, desktop, or the like, and is an information processing device that can communicate with the server device 100 and output the item name and item value of each item in a form to the user in association with each other. By executing a predetermined program, the terminal 200 realizes the function of transmitting and receiving information related to the form in cooperation with the server device 100 and outputting a screen related to the form.

[0021] Scanner 300 is an image input device that converts original documents such as text, photographs, and pictures into digital image data. Note that scanner 300 may be any device that can read paper forms and convert them into image data, and may be a terminal device such as a smartphone or laptop with a camera function. Scanner 300 may also be capable of character recognition in image data.

[0022] <2. Functional configuration> The functional configuration of the server device 100 according to this embodiment will be described with reference to FIG. 2. As shown in FIG. 2, the server device 100 includes a control unit 110, a storage unit 120, and a communication unit 130. The server device 100 recognizes each of the multiple cells in a table included in the acquired form from the image data of the form. Furthermore, if multiple cells included in the table have a parent-child relationship, i.e., a nested structure, the server device 100 identifies and associates the multiple cells in this nested structure as a parent cell and a child cell. The server device 100 outputs the results of this identification and association to the terminal 200.

[0023] [Control Unit] The control unit 110 includes an acquisition unit 111, an extraction unit 112, a recognition unit 113, a combination identification unit 114, a selection unit 115, and an output unit .

[0024] [Acquisition Department] The acquisition unit 111 acquires image data of a form. This form includes a table having a plurality of cells. The acquisition unit 111 may acquire, for example, image data obtained by scanning a paper form using the scanner 300 and converting the scanned data via the communication unit 130, or may acquire image data of the form that has been stored in advance in the storage unit 120. The acquisition unit 111 may also acquire image data of the form via the communication unit 130 from, for example, the terminal 200 or another external device (not shown).

[0025] [Extraction part] The extraction unit 112 extracts ruled lines and tables from the image data acquired by the acquisition unit 111 .

[0026] For example, as shown in FIG. 3, the extraction unit 112 may extract ruled lines from the image data based on ruled line candidates determined by a ruled line determination unit 112a (described later) and intersection positions extracted by an intersection extraction unit 112b (described later). Specifically, the extraction unit 112 first identifies a combination of two intersection positions from the extracted intersection positions. Next, the extraction unit 112 identifies a straight line connecting the two combined intersection positions. Next, the extraction unit 112 determines whether each pixel is on the identified straight line and is a ruled line candidate. The extraction unit 112 then extracts, as a ruled line, pixels that are on the identified straight line and are ruled line candidates. That is, the extraction unit 112 extracts, as a ruled line pixel, a pixel on the straight line connecting the two intersection positions that coincides with a pixel determined to be a ruled line candidate.

[0027] According to the above configuration, the extraction unit 112 can extract ruled lines from the image data of the form based on the pixels of the ruled line candidates and the intersection positions. Therefore, even if the form is an unstandardized form, the extraction unit 112 can extract ruled lines of a table included in the form and analyze the layout of the table.

[0028] The extraction unit 112 may include, for example, a ruled line determination unit 112a. The ruled line determination unit 112a determines for each pixel of the image data whether or not it is a ruled line candidate (hereinafter simply referred to as a "ruled line candidate") that forms a ruled line on the image data.

[0029] Ruled line determination unit 112a may use, for example, machine learning techniques to determine whether or not each pixel should be considered a ruled line candidate. Specifically, ruled line determination unit 112a first inputs image data to a trained first neural network. Then, based on the output of the first neural network to which the image data has been input, ruled line determination unit 112a determines whether or not each pixel of the image data should be considered a ruled line candidate. Ruled line determination unit 112a may train the first neural network, for example, based on a first dataset in which image data of a form is used as input data and data of pixels indicating ruled lines is used as correct answer data.

[0030] The extraction unit 112 may include, for example, an intersection extraction unit 112b. The intersection extraction unit 112b extracts intersection positions of ruled lines on the image data. The intersection extraction unit 112b may extract the intersection positions of ruled lines on the image data based on the positions of pixels determined to be intersection candidates by the intersection determination unit 112b1, for example.

[0031] Intersection extraction unit 112b may include, for example, intersection determination unit 112b1. Intersection determination unit 112b1 determines for each pixel of image data whether or not it is a candidate for an intersection that forms an intersection of ruled lines on this image data.

[0032] The intersection determination unit 112b1 may use, for example, machine learning techniques to determine whether or not each pixel should be considered a ruled line candidate. Specifically, the intersection determination unit 112b1 first inputs image data to a trained second neural network. Then, based on the output of the second neural network to which the image data has been input, the intersection determination unit 112b1 may determine, for each pixel of the image data, whether or not the pixel should be considered an intersection point candidate that forms an intersection point of a ruled line on the image data. The intersection determination unit 112b1 may train the second neural network, for example, based on a second data set in which image data of a form is used as input data and data of pixels indicating intersections is used as correct answer data.

[0033] The first neural network is a model for determining whether each pixel is a ruled line candidate or not. The second neural network is a model for determining whether each pixel is an intersection candidate or not. The first neural network and the second neural network are, for example, convolutional neural networks ( C onvolutionary N eural N network (CNN).

[0034] [Recognition part] The recognition unit 113 recognizes multiple cells in a table included in a form. The recognition unit 113 recognizes an area containing each character string in the table on the image data as a cell. For each cell, the recognition unit 113 may recognize, for example, the coordinates of the vertices (intersections) of the left, right, top, and bottom areas, the width of the cell, the height of the cell, and / or the character string that constitutes the cell.

[0035] For example, in the case of a cell surrounded by visually recognizable lines, the recognition unit 113 may recognize a rectangular frame made of the lines surrounding each character string. The recognition unit 113 may recognize this recognized rectangular frame as part of the cell.

[0036] The recognition unit 113 may recognize the left and right sides of the table in the horizontal direction (X-axis direction) based on the ruled lines extracted by the extraction unit 112. The recognition unit 113 may also recognize the up and down sides of the table in the vertical direction (Y-axis direction) based on the ruled lines extracted by the extraction unit 112.

[0037] The recognition unit 113 may include, for example, a character recognition unit 113a, which recognizes character strings within cells in the image data.

[0038] The recognition unit 113 may include, for example, an item identification unit 113b. The item identification unit 113b identifies each of the multiple item names included in the table based on each character string recognized by the character recognition unit 113a. Specifically, the item identification unit 113b identifies the position, character string, etc. of each of the multiple item names.

[0039] For example, the item identification unit 113b may identify a cell representing an item name among the cells of the form based on the character string of each cell recognized by the character recognition unit 113a and the item name registration information stored in the storage unit 120. The item identification unit 113b may identify the position of the cell identified in this way as the position of the item name.

[0040] The item identification unit 113b may, for example, search for character strings in each cell using part or all of the item names registered in the item name registration information as search words, and identify cells that contain characters or combinations of characters that match this search word as cells representing the item names.

[0041] "Item name registration information" is information that indicates the name of each item on a pre-registered form. For example, if the form is a receipt issued by a medical institution, the item name registration information is a list of item names to identify each item, such as "Issue Date" which indicates the date the medical receipt was issued, "Initial / Revisit Fee" which indicates the cost of a medical examination by a doctor, or "Surgery" which indicates the cost of surgery.

[0042] For example, the item identification unit 113b may identify an item value corresponding to each identified item name based on each character string recognized by the character recognition unit 113a. Furthermore, the item identification unit 113b may use a first mapping model to identify a range of item values ​​corresponding to the positions of the identified item names, in which the first correlation value is equal to or greater than a predetermined threshold value. Then, the item identification unit 113b may extract each character belonging to this identified range, and identify an item value corresponding to the identified item name based on each extracted character.

[0043] The first mapping model is a model that represents a first correlation value between the position of the item name of each item on the form and the position of the item value corresponding to this item name. The first mapping model may be, for example, a model in which a correlation value between the position of the item name of each item on the form and the position of the item value is mapped on the form for each pattern of the form classified by the pattern classification unit 113f. This correlation value may be, for example, calculated by statistically analyzing multiple pieces of relative position information between the item name and item value of each item, which are learning data. Furthermore, the first mapping model may be, for example, a heat map, which is a visualized graph that represents data on the relative position between the item name and item value of each item, or may be a two-dimensional representation of the correlation level of the relative positions of multiple item values ​​corresponding to the item name.

[0044] The recognition unit 113 may include, for example, a counting unit 113c. As shown in FIG. 4, the counting unit 113c votes for ruled line positions that are candidates for positions at which a composite table made up of multiple tables is divided into multiple tables (hereinafter also referred to as "division positions"). Specifically, for each line segment forming a ruled line, the counting unit 113c first determines whether or not a part of the ruled line frame surrounding each character string of multiple item names (cells) overlaps with this line segment. Then, each time the counting unit 113c determines that a part of each ruled line frame overlaps with this line segment, it counts (votes) a predetermined number. In other words, the counting unit 113c counts a predetermined number each time a part of the ruled line frame of an item name cell coincides with this line segment.

[0045] For example, in the case of a table in which item names are arranged vertically, the counting unit 113c may limit the object to be determined whether or not it overlaps with each line segment forming a ruled line to the left side (the left vertical line) of the ruled frame surrounding each character string of a plurality of item names (cells). Furthermore, in the case of a table in which item names are arranged horizontally, the counting unit 113c may limit the object to be determined whether or not it overlaps with each line segment forming a ruled line to the top side (the upper horizontal line) of the ruled frame surrounding each character string of a plurality of item names (cells). For example, in a table in which item names are arranged vertically, the item names included in the table are generally arranged on the left side. Furthermore, in the case of a table in which item names are arranged horizontally, the item names included in the table are generally arranged on the top side. With this configuration, the counting unit 113c can cast votes on the positions of the ruled lines that are candidates for the division position, taking into account the layout tendencies of tables in which item names are arranged vertically and tables in which item names are arranged horizontally.

[0046] For example, when pixels indicating the ruled line extracted by the extraction unit 112 are adjacent to each other, that is, connected to each other, the recognition unit 113 may form the line segments that form the ruled line by combining these pixels.

[0047] The recognition unit 113 may include, for example, a division position identification unit 113d. As shown in FIG. 4, the division position identification unit 113d identifies a division position based on the result of voting by the counting unit 113c. In the case of a composite table made up of multiple adjacent tables, the division position identification unit 113d identifies a position on the image data at which the composite table is to be divided into multiple tables based on the result of counting by the counting unit 113c. The recognition unit 113 may, for example, divide the composite table into multiple tables at the position identified by the division position identification unit 113d.

[0048] For example, particularly in the case of an atypical format, a composite table made up of multiple adjacent tables may be extracted as a single table. With the above configuration, the recognition unit 113 can divide a composite table extracted as a single table in this way into the multiple tables that make up this composite table, based on the results of voting that takes into account the tendency of the table layout.

[0049] The recognition unit 113 may include, for example, an area identification unit 113e. The area identification unit 113e identifies an area within a table where the item names included in the table are arranged. For example, in the case of a table in which the item names are arranged vertically, the area identification unit 113e may identify the area in the left half of the table as the area where the item names included in the table are arranged. As another example, in the case of a table in which the item names are arranged horizontally, the area identification unit 113e may identify the area in the upper half of the table as the area where the item names included in the table are arranged. As another example, in the case of a table in which the item names are arranged both horizontally and vertically, the area identification unit 113e may identify the outside of the area in the lower right part of the table as the area where the item names included in the table are arranged.

[0050] The recognition unit 113 may include, for example, a pattern classification unit 113f. Based on the image data of the form acquired by the acquisition unit 111, the pattern classification unit 113f classifies the pattern of the form indicated in the image data. For example, the pattern classification unit 113f may extract features of the image data using a convolutional neural network to perform image classification. Furthermore, the pattern classification unit 113f may classify the form by format pattern based on pattern information indicating the format pattern of the form stored in advance in the storage unit 120.

[0051] [Combination identification section] The combination identification unit 114 identifies one or more candidate combinations of parent cells and child cells that are in a parent-child relationship, i.e., in a nested structure, from the multiple cells recognized by the recognition unit 113. Hereinafter, a combination of a parent cell and a child cell will also be referred to as a "parent-child cell." The combination identification unit 114 may also identify one or more candidate combinations of a parent cell, a child cell, and a grandchild cell that are in a parent-grandchild relationship, in other words, in a nested structure. In this combination, the child cell and the grandchild cell have a parent-child relationship in which the child cell is the parent and the grandchild cell is the child, i.e., in a nested structure. Hereinafter, a combination of a parent cell, a child cell, and a grandchild cell will also be referred to as a "parent-grandchild cell." In other words, a parent-grandchild cell has two parent-child cell combinations.

[0052] The combination specifying unit 114 may, for example, combine the plurality of cells recognized by the recognition unit 113 in a round-robin manner and specify the combined cells as candidates for parent-child cells.

[0053] [Selection] The selection unit 115 selects a combination of parent and child cells to be adopted from among candidate combinations based on the positional relationship between the parent and child cells and the arrangement of ruled lines between the parent and child cells. For example, when the evaluation by the validity evaluation unit (described later) is a relative evaluation, the selection unit 115 may select a candidate with a relatively high validity evaluation value from among multiple parent and child cell candidates. This "relatively high" may refer to, for example, a candidate with an evaluation value higher than the average, median, or standard deviation of the validity evaluation values ​​of the parent and child cell candidates. As another example, when the evaluation by the validity evaluation unit is an absolute evaluation, the selection unit 115 may select a candidate with an evaluation value exceeding a predetermined threshold from among multiple parent and child cell candidates.

[0054] According to the above configuration, the selection unit 115 can select parent-child cells based on the positional relationship of the combined cells and the arrangement of ruled lines between the cells. Therefore, even if the table is an unstructured form, the selection unit 115 can select nested item names (cells) included in the table. Therefore, when outputting information on nested items in a table having a nested structure in the image data of the form, the selection unit 115 can handle even an unstructured form table.

[0055] The selection unit 115 may include, for example, a validity evaluation unit 115a. The validity evaluation unit 115a evaluates the validity of a nested structure, i.e., a parent-child cell candidate. The validity evaluation unit 115a may also evaluate the validity of a parent-grandchild cell candidate, for example. The selection unit 115 may select a combination to adopt from the candidates based on the validity evaluated by the validity evaluation unit 115a. With this configuration, the selection unit 115 can select a parent-child cell to adopt by evaluating the validity of the parent-child cell candidate. Therefore, the selection unit 115 can recognize the item names (cells) of a nested structure based on a certain validity, even in an undefined form.

[0056] As the evaluation of the above-mentioned (A) validity, the validity evaluation unit 115a may evaluate the validity of the parent-child cell candidates in accordance with at least one of the following rules 1 to 4, for example. Rule 1: When there are parent-child cell candidates next to each other, the parent cell is more likely to be a candidate if it is positioned in front of the child cell (horizontally to the left). Rule 2: When there are parent-child cell candidates arranged vertically, the parent cell is more likely to be a candidate if it is positioned at the top (vertically). Rule 3: The fewer the number of ruled lines that cross the grid path between parent and child cells (movement at right angles along the horizontal and vertical directions), the more appropriate the parent and child cell candidate is. Rule 4: Determine whether the parent cell and child cell overlap on the Y coordinate (vertical axis), and compare overlapping candidates to non-overlapping candidates to determine their validity.

[0057] The evaluation of validity performed by the validity evaluation unit 115a may be, for example, a relative evaluation between a plurality of candidates, or an absolute evaluation.

[0058] For example, the validity evaluation unit 115a may evaluate the validity of a parent-child cell candidate more highly when the parent cell is positioned to the left of the child cell in the horizontal direction (X-axis direction) in accordance with Rule 1 above, as the positional relationship between the parent cell and the child cell. In this case, the validity evaluation unit 115a may evaluate the validity of the parent-child cell candidate more highly compared to when the parent cell is positioned to the right of the child cell. With this configuration, the validity evaluation unit 115a can evaluate the validity of the parent-child relationship, i.e., the nested structure, based on the layout of the table, which is configured to be read from left to right.

[0059] For example, the validity evaluation unit 115a may evaluate the validity of a parent-child cell candidate higher when the parent cell and the child cell are positioned above the child cell in the vertical direction (Y-axis direction) in accordance with Rule 2 above. In this case, the validity evaluation unit 115a may relatively evaluate the validity of the parent-child cell candidate higher than when the parent cell is positioned below the child cell in the vertical direction. With this configuration, the validity evaluation unit 115a can evaluate the validity of the parent-child relationship, i.e., the nested structure, based on the layout of the table that is configured to be read from top to bottom.

[0060] For example, the validity evaluation unit 115a may lower the evaluation of the validity of a parent-child cell candidate when the number of ruled lines between the parent cell and the child cell exceeds a predetermined number, in accordance with Rule 3 above. In this case, the validity evaluation unit 115a may relatively lower the evaluation of the validity of the parent-child cell candidate compared to when the number of ruled lines between the parent cell and the child cell does not exceed a predetermined number. For example, multiple cells that are unrelated to each other are often not adjacent to each other, or are spaced a certain distance apart. With this configuration, the validity evaluation unit 115a can evaluate the validity of the parent-child relationship, i.e., the nested structure, based on the tendency of the placement of such unrelated cells.

[0061] The validity evaluation unit 115a may evaluate the validity of parent-child cell candidates, for example, in accordance with Rule 4 above, further based on the results of evaluation by the overlap evaluation unit 115b (described later). For example, multiple cells that are placed unrelated to each other are not adjacent to each other, and the frames surrounding each cell often do not overlap. With this configuration, the validity evaluation unit 115a can evaluate the validity of the parent-child relationship, i.e., the nested structure, based on the tendency of the placement of such unrelated cells.

[0062] The selection unit 115 may include, for example, an overlap evaluation unit 115b. The overlap evaluation unit 115b evaluates the overlap state between the ruled frame surrounding the parent cell and the ruled frame surrounding the child cell. Specifically, the overlap evaluation unit 115b may evaluate whether at least a portion of the ruled frame surrounding the parent cell overlaps with at least a portion of the ruled frame surrounding the child cell.

[0063] The overlap evaluation unit 115b may evaluate, for example, whether the intersection coordinates of the ruled lines surrounding the parent cell and the intersection coordinates of the ruled lines surrounding the child cell overlap. Furthermore, the overlap evaluation unit 115b may evaluate, for example, whether the Y coordinate of the intersection coordinates of the parent cell and the Y coordinate of the intersection coordinates of the child cell overlap, i.e., whether they are approximately the same. This "approximately the same (coordinates)" may refer, for example, to whether the respective coordinates are the same value or within a certain range.

[0064] Here, an example of evaluation of the validity of a nested structure using the above rules 1 to 4 in the validity evaluation unit 115a will be described with reference to FIGS.

[0065] In the example of FIG. 5, a combination of (1) "visual acuity" cell, (2) "naked eye" cell, (3a) "right" cell, and (3b) "left" cell will be used for explanation. As shown in FIG. 5, the validity evaluation unit 115a evaluates the validity of the nested structure for each of the parent-descendant cell candidates combined from (1), (2), (3a), and (3b) in accordance with the above rules 1 to 4. Note that in FIG. 5, grid lines are indicated by dashed lines. In addition, in this example, the validity evaluation is performed using a point-adding method. That is, for each candidate, points are added if the rule is met, but points are not subtracted if the rule is not met.

[0066] Regarding the above rule 1, in the case of a candidate combination in which the (1) "visual acuity" cell is the parent cell, the (2) "naked eyes" cell is the child cell, and the (3a) "right" cell is the grandchild cell (hereinafter also referred to as "parent-grandchild cells of (1), (2), and (3a)"), in the parent-child relationship between the child cell and the grandchild cell, the child cell is positioned to the left of the grandchild cell in the horizontal direction (X-axis direction). Therefore, the validity evaluation unit 115a adds a point ("+") to the evaluation of the validity of the nested structure for this candidate.

[0067] Regarding the above rule 2, in the case of parent-child cell candidates (1), (2), and (3a), the parent cell is positioned above the child cell in the vertical direction (Y axis) in the parent-child relationship between the parent cell and the child cell. Therefore, the validity evaluation unit 115a adds a point ("+") to the evaluation of the validity of the nesting structure for this candidate.

[0068] Regarding rule 3 above, for parent-child cell candidates (1), (2), and (3a), in the parent-child relationship between the grandchild cell and the child cell, the number of ruled lines between the parent cell and the child cell, i.e., the number of ruled lines crossing the grid path between the descendant cells (moving at right angles along the X-axis and Y-axis on the grid lines) is less than a predetermined number (in this example, "2"). Therefore, the validity evaluation unit 115a adds points ("+") to the evaluation of the validity of the nested structure for this candidate. This "number of ruled lines crossing the grid path" may be, for example, the number of intersections between the grid lines and the ruled lines surrounding each parent and child cell that are on the path of movement between the parent and child cells along the grid lines. If the number of intersections on this path of movement is less than a predetermined number, the validity evaluation unit 115a may add points to the evaluation of the validity of the nested structure for this parent-child cell candidate.

[0069] Regarding rule 4 above, in the case of the parent-children cell candidates (1), (2), and (3a), in the parent-child relationship between the grandchild and child cells, the Y coordinates of the intersection coordinates of the child cell and the grandchild cell overlap at the two vertices (top right and bottom right) where the Y coordinates of the intersection coordinates of the child cell and the grandchild cell overlap. Therefore, the validity evaluation unit 115a adds a point ("+") to the evaluation of the validity of the nesting structure for this candidate.

[0070] As described above, the validity evaluation unit 115a totals the scores added to the evaluation of the validity of the nested structure for rules 1 to 4. For example, if one "+" is added as one point, the total validity evaluation of the parent-descendant cell candidates (1), (2), and (3a) is four points. For example, if this total exceeds a predetermined threshold (three in this example), the selection unit 115 may select the parent-descendant cell (1), (2), and (3a) as the parent-descendant cell to be adopted. As another example, if this total is relatively high among all the identified candidate combinations of parent-descendant cells, the selection unit 115 may select the parent-descendant cell (1), (2), and (3a) as the parent-descendant cell to be adopted.

[0071] 6, for example, there are cases where some ruled lines cannot be extracted by the extraction unit 112. In such cases, the validity evaluation unit 115a does not need to apply a rule (in this example, rule 2) that uses the ruled lines that could not be extracted when evaluating the validity of the nested structure.

[0072] Continuing the explanation by returning to Fig. 2, the validity evaluation unit 115a may, for example, lower the evaluation of the validity of the parent-child cell candidate when at least one of the parent cell and the child cell is located outside the area identified by the area identification unit 113e in the table, as the evaluation of the validity of the parent-child cell candidate in (i) above.

[0073] Here, with reference to FIG. 7, an example of evaluating the validity of a nested structure based on the table layout will be described. In this example, the identified cells with item names are "blood pressure" and "maximum." As shown in FIG. 7(a), when a table T1 in which item names are arranged vertically is extracted, the area representing the table T1 is divided into a left area and a right area with respect to a line (y1-y1') approximately at the center. The area identifying unit 113e identifies the left area as the area in which the item names of the table T1 are arranged. On the other hand, FIG. 7(b) shows an example in which a graph P1 is erroneously extracted as a table in which item names are arranged vertically. As shown in FIG. 7(b), when the left area is identified with respect to the area representing the graph P1 with respect to a line (y2-y2') approximately at the center, the validity evaluating unit 115a may deduct points from the validity of cells with item names (in this example, "blood pressure" and "maximum") that are at least partially located outside the identified left area, i.e., in the right area, as candidates for a nested structure, i.e., parent-child cells.

[0074] For example, many table layouts tend to be arranged in the left region within the table. With the above configuration, the validity evaluation unit 115a can evaluate the validity of the parent-child relationship, i.e., the nested structure, based on this tendency of table layout. Therefore, for example, if a graph or the like is mistakenly extracted as a table, the validity evaluation unit 115a can reduce the chance of misrecognizing the cells in the graph or the like as a nested structure.

[0075] Returning to Fig. 2, the explanation will be continued. As the evaluation of the above (c) validity, the validity evaluation unit 115a may, for example, refer to the keyword storage unit 121 that stores information indicating one or more keywords that can be placed in a table (hereinafter also referred to as "keyword information"), and if a character string (cell) indicating a keyword does not exist in the table, may lower the evaluation of the validity of a parent-child cell candidate placed in this table. This keyword may, for example, be a character string indicating the title of the table, a character string indicating a column title (item name), a character string indicating a unit, etc.

[0076] Here, referring to FIG. 8, an example of evaluating the validity of a nested structure based on keywords that may be included in a table will be described. In this example, the cells of the identified item names are "blood pressure" and "maximum," and the keyword is the column title "this time." As shown in FIG. 8(a), the keyword "this time" exists among the character strings included in the cells of the extracted table T2. On the other hand, FIG. 8(b) shows an example in which a graph P2 is erroneously extracted as a table. As shown in FIG. 7(b), the validity evaluation unit 115a may deduct points from the validity of the item name cells (in this example, "blood pressure" and "maximum") included in the graph P2 as candidates for a nested structure, i.e., parent-child cells, because the keyword "this time" does not exist in the graph P2.

[0077] According to the above configuration, the validity evaluation unit 115a can evaluate the validity of the parent-child relationship, i.e., the nested structure, based on keywords such as titles and units that may be included in such tables. Therefore, for example, if a graph or the like is mistakenly extracted as a table, the validity evaluation unit 115a can reduce the likelihood of misrecognizing the cells in the graph or the like as a nested structure.

[0078] [Output section] The output unit 116 outputs information indicating the parent cell and information indicating the child cell based on the selected combination. The output unit 116 may, for example, generate output information that associates the character string information of the parent cell with the character string information of the child cell and outputs the output information on the terminal 200 or the like. Then, the output unit 116 may, for example, output the generated output information by causing the communication unit 130 to transmit it to the terminal 200. This output information may, for example, be fixed-length or variable-length text data that lists the character strings of each item name in a parent-child relationship on the form.

[0079] [Storage] The storage unit 120 may store a first neural network and a second neural network. Furthermore, the storage unit 120 may store image data of a form, information indicating each cell, output information, item name registration information, and / or keyword information. The storage unit 120 may include, for example, a keyword storage unit 121. The keyword storage unit 121 stores keyword information. The storage unit 120 may store each piece of information using, for example, a database management system (DBMS), or may store each piece of information using a file system. When a DBMS is used, the storage unit 120 may provide a table for each piece of information, and by associating these tables, the pieces of information may be stored and managed in an associated manner.

[0080] [Communications Department] The communication unit 130 receives image data of a form from the scanner 300 via the network N and transmits output information to the terminal 200.

[0081] <3. Example of operation> An example of the operation of the server device 100 according to this embodiment will be described with reference to Fig. 9. Fig. 9 is a flow diagram showing the flow of processing in the server device 100 to acquire image data of a form and output information read from this image data to the terminal 200. Note that the order of processing shown below is an example and may be changed as appropriate.

[0082] 9, the acquisition unit 111 of the server device 100 acquires image data of a form (S10). This form includes a table having a plurality of cells.

[0083] Ruled line determination unit 112a determines for each pixel of the image data of the form whether it is a ruled line candidate that will form a ruled line on this image data (S11). In parallel with the determination process of ruled line determination unit 112a, intersection determination unit 112b1 determines for each pixel of the image data of the form whether it is an intersection candidate that will form an intersection of a ruled line on this image data (S12). Intersection extraction unit 112b extracts the intersection positions of the ruled lines on the image data based on the positions of pixels determined to be intersection candidates (S13).

[0084] The extraction unit 112 extracts ruled lines from the image data based on the ruled line candidates determined by the ruled line determination unit 112a and the intersection positions extracted by the intersection extraction unit 112b (S14).

[0085] In parallel with the process of determining ruled line candidates, extracting intersection positions, and extracting ruled lines, the character recognition unit 113a recognizes character strings in cells on the image data of the form (S15). Similarly, in parallel with this, the extraction unit 112 extracts tables from the image data of the form (S16).

[0086] The recognition unit 113 recognizes a plurality of cells in the table based on the table extracted by the extraction unit 112 and the character strings recognized by the recognition unit 113 (S17). Specifically, for each cell, the recognition unit 113 recognizes, for example, the vertex coordinates of the cell and the character strings included in the cell.

[0087] The combination identification unit 114 identifies one or more candidate combinations of parent cells and child cells in a parent-child relationship from among the multiple cells recognized by the recognition unit 113 (S18). The selection unit 115 selects a combination to be adopted from the candidate combinations of parent cells and child cells based on the positional relationship between the parent cell and the child cell and the arrangement of ruled lines between the parent cell and the child cell (S19). The output unit 116 outputs information indicating the parent cell and information indicating the child cell to the terminal 200 or the like based on the combination selected by the selection unit 115 (S20).

[0088] <4. Hardware Configuration> 10, an example of a hardware configuration in which the above-described server device 100 is realized by a computer 800 will be described. Note that the functions of each device can also be realized by dividing them into multiple devices.

[0089] As shown in FIG. 10, the computer 800 includes a processor 801, a memory 803, a storage device 805, an input I / F unit 807, a data I / F unit 809, a communication I / F unit 811, and a display device 813.

[0090] The processor 801 controls various processes in the computer 800 by executing a program (e.g., a server program) stored in the memory 803. For example, each functional unit included in the control unit 110 of the server device 100 can be realized as a program that is temporarily stored in the memory 803 and operates mainly on the processor 801.

[0091] The memory 803 is a storage medium such as a RAM (Random Access Memory), and temporarily stores the program code of a program (e.g., a server program) executed by the processor 801 and data required for executing the program.

[0092] The storage device 805 is a non-volatile storage medium such as a hard disk drive (HDD) or flash memory. The storage device 805 stores an operating system and various programs (e.g., server programs) for implementing the above-mentioned configurations. In addition, the storage device 805 can also store tables for registering output information, keyword information, etc., and a DB for managing these tables. Such programs and data are loaded into the memory 803 as needed, and are referenced by the processor 801.

[0093] The input I / F unit 807 is a device for receiving input from a user. Specific examples of the input I / F unit 807 include a keyboard, a mouse, a touch panel, various sensors, and a wearable device. The input I / F unit 807 may be connected to the computer 800 via an interface such as a USB (Universal Serial Bus).

[0094] The data I / F unit 809 is a device for inputting data from outside the computer 800. A specific example of the data I / F unit 809 is a drive device for reading data stored in various storage media. The data I / F unit 809 may be provided outside the computer 800. In this case, the data I / F unit 809 is connected to the computer 800 via an interface such as a USB.

[0095] The communication I / F unit 811 is a device for performing data communication via the Internet N, either wired or wirelessly, with devices external to the computer 800. The communication I / F unit 811 may be provided outside the computer 800. In this case, the communication I / F unit 811 is connected to the computer 800 via an interface such as a USB.

[0096] The display device 813 is a device for displaying various types of information. Specific examples of the display device 813 include a liquid crystal display, an organic EL (Electro-Luminescence) display, and a display of a wearable device. The display device 813 may be provided outside the computer 800. In this case, the display device 813 is connected to the computer 800 via, for example, a display cable. Furthermore, when a touch panel is adopted as the input I / F unit 807, the display device 813 can be configured as an integral part of the input I / F unit 807.

[0097] It should be noted that the present embodiment is an example for explaining the present invention, and is not intended to limit the present invention to only this embodiment. Furthermore, the present invention can be modified in various ways without departing from the gist of the present invention. Furthermore, those skilled in the art can adopt embodiments in which the elements described below are replaced with equivalents, and such embodiments are also within the scope of the present invention.

[0098] [Variations] Although the present invention has been described based on the above embodiment, the following cases are also included in the present invention.

[0099] [Variation 1] At least a part of each component of the server device 100 according to the above embodiment may be included in the terminal 200. For example, the functions of the recognition unit 113, the combination identification unit 114, the selection unit 115, and the output unit 116 of the server device 100 may be included in a control unit (not shown) of the terminal 200.

[0100] [Variation 2] In the above embodiment, as an example of evaluating the validity of (b) and (c) above, an example was described in which the validity of a nested structure was evaluated using the table layout and keywords contained in the table. However, the present invention is not limited to this. For example, the validity evaluation unit 115a may use a second mapping model based on keyword information to identify a range in which keywords corresponding to the positions of the identified item names, whose second correlation values ​​are equal to or greater than a predetermined threshold, may be located. These keywords may be, for example, keywords classified by form pattern. If a character string indicating a keyword does not exist within this identified range, the validity evaluation unit 115a may lower the evaluation of the validity of a parent-child cell candidate placed in this table. Conversely, if a character string indicating a keyword exists within this identified range, the validity evaluation unit 115a may raise the evaluation of the validity of a parent-child cell candidate placed in this table.

[0101] The second mapping model is a model that represents a second correlation value between the position of the item name of each item on the form and the position of the keyword corresponding to this item name. The second mapping model may be, for example, a model in which a correlation value between the position of the item name of each item on the form and the position of the keyword is mapped on the form for each pattern of the form classified by the pattern classification unit 113f. This correlation value may be calculated by statistically analyzing multiple pieces of relative position information between the item name of each item and the keyword, which are learning data. Furthermore, the second mapping model may be, for example, a heat map, which is a visualized graph that represents data on the relative position between the item name of each item and the keyword, or may be a two-dimensional representation of the correlation level of the relative positions of multiple keywords corresponding to the item name. [Explanation of symbols]

[0102] 1...document reading system, 100...server device (information processing device), 110...control unit, 112...extraction unit, 113...recognition unit, 114...combination identification unit, 115...selection unit, 116...output unit, 120...storage unit, 130...communication unit, 200...terminal, 300...scanner, 800...computer, 801...processor, 803...memory, 805...storage device, 807...input I / F unit, 809...data I / F unit, 811...communication I / F unit, 813...display device.

Claims

1. A computer, an acquisition unit that acquires image data of a form including a table made up of a plurality of cells; an extraction unit that extracts ruled lines of the table on the image data; a recognition unit that recognizes a plurality of cells of the table; a combination specifying unit that specifies one or more candidate combinations of parent cells and child cells in a parent-child relationship from the recognized plurality of cells; a selection unit that selects a combination to be adopted from the candidates based on a positional relationship between the parent cell and the child cell and an arrangement state of the ruled lines between the parent cell and the child cell; an output unit that outputs information indicating the parent cell and information indicating the child cell based on the selected combination, the recognition unit recognizes the left and right sides of the table in the horizontal direction based on the extracted ruled lines; the selection unit includes a validity evaluation unit that evaluates validity of the candidates; the validity evaluation unit increases the evaluation of validity of the candidate when the parent cell is located to the left of the child cell in the horizontal direction, as the positional relationship; the selection unit selects the combination to be adopted based on the evaluated validity. program.

2. A computer, an acquisition unit that acquires image data of a form including a table made up of a plurality of cells; an extraction unit that extracts ruled lines of the table on the image data; a recognition unit that recognizes a plurality of cells of the table; a combination specifying unit that specifies one or more candidate combinations of parent cells and child cells in a parent-child relationship from the recognized plurality of cells; a selection unit that selects a combination to be adopted from the candidates based on a positional relationship between the parent cell and the child cell and an arrangement state of the ruled lines between the parent cell and the child cell; an output unit that outputs information indicating the parent cell and information indicating the child cell based on the selected combination, the recognition unit recognizes the top and bottom of the table in the vertical direction based on the extracted ruled lines; the selection unit includes a validity evaluation unit that evaluates validity of the candidates; the validity evaluation unit increases the evaluation of the validity of the candidate when the parent cell is arranged above the child cell in the vertical direction as the positional relationship; the selection unit selects the combination to be adopted based on the evaluated validity. program.

3. A computer, an acquisition unit that acquires image data of a form including a table made up of a plurality of cells; an extraction unit that extracts ruled lines of the table on the image data; a recognition unit that recognizes a plurality of cells of the table; a combination specifying unit that specifies one or more candidate combinations of parent cells and child cells in a parent-child relationship from the recognized plurality of cells; a selection unit that selects a combination to be adopted from the candidates based on a positional relationship between the parent cell and the child cell and an arrangement state of the ruled lines between the parent cell and the child cell; an output unit that outputs information indicating the parent cell and information indicating the child cell based on the selected combination, the selection unit includes a validity evaluation unit that evaluates validity of the candidates; the validity evaluation unit lowers the evaluation of the validity of the candidate when the number of ruled lines arranged between the parent cell and the child cell exceeds a predetermined number as the arrangement state; the selection unit selects the combination to be adopted based on the evaluated validity. program.

4. the selection unit further includes an overlap evaluation unit that evaluates whether or not at least a part of a ruled frame surrounding the parent cell and at least a part of a ruled frame surrounding the child cell overlap with each other; the validity evaluation unit evaluates the validity of the candidate further based on a result of the evaluation by the overlap evaluation unit. The program according to any one of claims 1 to 3.

5. A computer, an acquisition unit that acquires image data of a form including a table made up of a plurality of cells; an extraction unit that extracts ruled lines of the table on the image data; a recognition unit that recognizes a plurality of cells of the table; a combination specifying unit that specifies one or more candidate combinations of parent cells and child cells in a parent-child relationship from the recognized plurality of cells; a selection unit that selects a combination to be adopted from the candidates based on a positional relationship between the parent cell and the child cell and an arrangement state of the ruled lines between the parent cell and the child cell; an output unit that outputs information indicating the parent cell and information indicating the child cell based on the selected combination; an area specifying unit that specifies an area in the table where an item name included in the table is to be placed, the selection unit includes a validity evaluation unit that evaluates validity of the candidates; the validity evaluation unit lowers the evaluation of the validity of the candidate when at least one of the parent cell and the child cell is located outside the specified area in the table; the selection unit selects the combination to be adopted based on the evaluated validity. program.

6. A computer, an acquisition unit that acquires image data of a form including a table made up of a plurality of cells; an extraction unit that extracts ruled lines of the table on the image data; a recognition unit that recognizes a plurality of cells of the table; a combination specifying unit that specifies one or more candidate combinations of parent cells and child cells in a parent-child relationship from the recognized plurality of cells; a selection unit that selects a combination to be adopted from the candidates based on a positional relationship between the parent cell and the child cell and an arrangement state of the ruled lines between the parent cell and the child cell; an output unit that outputs information indicating the parent cell and information indicating the child cell based on the selected combination, the selection unit includes a validity evaluation unit that evaluates validity of the candidates; the validity evaluation unit refers to a keyword storage unit that stores one or more keywords that can be placed in the table, and if a character string indicating the keyword does not exist in the table, lowers the evaluation of the validity of the candidates for the parent cell and the child cell placed in the table; the selection unit selects the combination to be adopted based on the evaluated validity. program.

7. A computer, the extraction unit further functions as a ruled line determination unit and an intersection extraction unit; the ruled line determination unit determines, for each pixel of the image data, whether or not the pixel is a ruled line candidate for forming a ruled line on the image data; the intersection extraction unit extracts intersection positions of ruled lines on the image data; The program according to claim 1 , wherein the extraction unit extracts the ruled line based on the determined ruled line candidate and the extracted intersection position.

8. the ruled line determination unit inputs the image data to a trained first neural network, and determines, based on an output of the first neural network, for each pixel of the image data whether or not the pixel is to be a ruled line candidate for forming a ruled line on the image data; the intersection extraction unit includes an intersection determination unit, the intersection determination unit inputs the image data into a trained second neural network, and determines, based on an output of the second neural network, for each pixel of the image data whether or not the pixel is to be a candidate intersection that forms an intersection of a ruled line on the image data; the intersection extraction unit extracts intersection positions of the ruled lines based on positions of pixels determined to be the intersection candidates; The program according to claim 7.

9. A computer, an acquisition unit that acquires image data of a form including a table made up of a plurality of cells; an extraction unit that extracts ruled lines of the table on the image data; a recognition unit that recognizes a plurality of cells of the table; a combination specifying unit that specifies one or more candidate combinations of parent cells and child cells in a parent-child relationship from the recognized plurality of cells; a selection unit that selects a combination to be adopted from the candidates based on a positional relationship between the parent cell and the child cell and an arrangement state of the ruled lines between the parent cell and the child cell; an output unit that outputs information indicating the parent cell and information indicating the child cell based on the selected combination, the table is a composite table, the composite table being composed of a plurality of adjacent tables, the recognition unit includes a character recognition unit, an item identification unit, a counting unit, and a division position identification unit; the character recognition unit recognizes a character string in a cell on the image data, the item identification unit identifies the position of each of a plurality of item names included in the table based on each of the recognized character strings; the counting unit determines, for each line segment forming the ruled line, whether or not a part of a frame of the ruled line surrounding each character string of the plurality of item names overlaps with the line segment, and if it determines that the line segment overlaps, counts a predetermined number each time; the division position specifying unit specifies positions on the image data at which the composite table is to be divided into the plurality of tables based on a result of the counting; the recognition unit divides the composite table into the plurality of tables at the specified position. program.

10. A memory unit that stores the program according to any one of claims 1 to 9; a control unit that executes the program; An information processing device comprising:

11. An information processing method, in which a computer executes the program according to any one of claims 1 to 9.

Images