Table interaction area identification method and apparatus

Abstract

The invention discloses a method and a device for identifying a table interaction area in a document. The method for identifying the table interaction area comprises the steps of identifying effective cells in the document to be identified, searching for interaction elements in the area where the effective cells are located, determining interaction area information corresponding to the effective cells according to a search result and the like. According to the method and the device, the areas which are not marked and have interactive attributes in the to-be-identified document can be automatically identified as the interactive areas, so that automatic identification and addition of the interactive areas are realized, and the identification efficiency and accuracy can be improved; moreover, under the condition that an editor does not edit the interaction area or the information of the interaction area is lost, the information of the interaction area can be efficiently generated or recovered, so that a user can perform interaction and editing operation on the to-be-identified document through the interaction area without entering an editing mode, and the user experience is improved. Therefore, the use convenience of the electronic document is improved. The method is widely applied to the technical field of computer document processing.

Description

Technical Field

[0001] The present invention relates to the technical field of computer document processing, and in particular to a table interaction area recognition method and device. Background Art

[0002] Tables in electronic documents can enable interactive functions, thereby realizing functions such as interactive forms, providing convenience for users of electronic documents. However, the relevant technology only recognizes and edits tables in documents to be recognized. The editing of interactive forms is achieved by setting form fields in the layout file and pre-entering the form by text entry. This method requires a lot of manual operation, is inefficient and prone to errors. Summary of the invention

[0003] In view of the technical problems that the interactive function of a table in a document currently requires relatively complicated operations to be enabled, which is inconvenient for editors and users, the purpose of the present invention is to provide a method and device for identifying an interactive area of ​​a table in a document.

[0004] In one aspect, an embodiment of the present invention includes a method for identifying a table interaction area, wherein the method for identifying a table interaction area in a document includes the following steps:

[0005] Identify valid cells in the document to be identified;

[0006] Searching for interactive elements in the area where the valid cells in the document to be identified are located; the interactive elements are document elements in the document to be identified that have an interactive attribute indication function;

[0007] According to the search results, the interactive area information corresponding to the valid cell is determined.

[0008] Furthermore, determining the interactive area information corresponding to the valid cell according to the search results includes:

[0009] In case the interactive element is found, determining the validity of the interactive element;

[0010] When the interactive element is valid, the interactive area information is determined according to the interactive element.

[0011] Furthermore, before determining the validity of the interactive element, the method further includes:

[0012] Performing type detection on the interactive element; and determining the validity of the interactive element according to a result of the type detection;

[0013] When the interactive element is valid, determining the interactive area information according to the interactive element includes:

[0014] In the case where it is detected that the interactive element belongs to the first type of interactive element, searching for a first text element in the valid cell; the first text element is a document element of the text type in the document to be identified;

[0015] In the case where the first text element is found in the valid cell, detecting a modification relationship of the interactive element to the first text element;

[0016] The validity of the interactive element is determined according to the modification relationship.

[0017] Further, determining the validity of the interactive element according to the modification relationship includes:

[0018] In the case where the modification relationship indicates that the interactive element modifies the first text element, determining that the interactive element is invalid;

[0019] In a case where the modification relationship indicates that the interactive element does not modify the first text element, the validity of the first-type interactive element is determined according to the position of the first-type interactive element.

[0020] Further, judging the validity of the first-type interactive element according to the position of the first-type interactive element includes:

[0021] Obtaining a cell bounding box, a joint bounding box, and an interactive bounding box; the cell bounding box is a bounding box corresponding to the valid cell, the joint bounding box is a minimum bounding box that includes all the first text elements in the valid cell, and the interactive bounding box is a bounding box corresponding to the interactive element;

[0022] Determining an allowable range of the interaction bounding box according to the relative sizes of the joint bounding box and the cell bounding box;

[0023] When the interaction boundary box is within the allowable range, the interaction element is determined to be valid; otherwise, the interaction element is determined to be invalid.

[0024] Further, the determining the allowable range of the interactive bounding box according to the relative size of the joint bounding box and the cell bounding box includes:

[0025] Acquire a height ratio; the height ratio is a ratio of the height of the joint bounding box to the height of the cell bounding box;

[0026] In the case where the height ratio is greater than a height ratio threshold, determining the allowable range according to the joint bounding box; an upper limit of the allowable range is obtained by adjusting upward the top of the joint bounding box, and a lower limit of the allowable range is obtained by adjusting downward the bottom of the joint bounding box;

[0027] When the height ratio is less than or equal to a height ratio threshold, the allowed range is determined according to the joint bounding box and the cell bounding box; the upper limit of the allowed range is obtained by adjusting upward from the top of the joint bounding box, and the lower limit of the allowed range is obtained by adjusting downward from the bottom of the cell bounding box.

[0028] Further, the determining the interactive area information according to the interactive element includes:

[0029] Determine the left boundary of the interaction bounding box as the left boundary of the interaction area;

[0030] Determine the right boundary of the interaction bounding box as the right boundary of the interaction area;

[0031] An upper boundary of the interaction area and a lower boundary of the interaction area are determined according to the interaction boundary box.

[0032] Further, determining the upper bound and the lower bound of the interaction area according to the interaction bounding box includes:

[0033] determining a first height threshold;

[0034] When the height of the row where the interaction bounding box is located is greater than or equal to the first height threshold, obtaining the upper boundary of the row where the interaction bounding box is located and determining it as the upper boundary of the interaction area, and obtaining the lower boundary of the row where the interaction bounding box is located and determining it as the lower boundary of the interaction area; or

[0035] When the height of the row where the interaction boundary box is located is less than the first height threshold, the upper boundary of the row where the interaction boundary box is located is obtained and raised to be determined as the upper boundary of the interaction area, and the lower boundary of the interaction boundary box is obtained to be determined as the lower boundary of the interaction area.

[0036] Further, determining the first height threshold comprises:

[0037] When there are multiple interactive bounding boxes, the average row height of the rows where all interactive bounding boxes are located is obtained;

[0038] The first height threshold is determined according to the row height average value.

[0039] Furthermore, determining the interactive area information corresponding to the valid cell according to the search results includes:

[0040] In the case that any of the document elements is not found, the interactive element is invalid, or the first text element is not found, searching for surrounding cells of the valid cell; the surrounding cells are cells located in the same table area as the valid cell;

[0041] In the case where the surrounding cells are searched, the interactive area information is determined according to the surrounding cells.

[0042] Furthermore, the searching of surrounding cells for the valid cell includes:

[0043] Obtaining row index information of the valid cell;

[0044] In the table area where the valid cells are located, searching for cells having the same row index information;

[0045] The searched cells are used as the surrounding cells.

[0046] Further, determining the interactive area information according to the surrounding cells includes:

[0047] Searching the surrounding cells for a second text element;

[0048] When the second text element is found, obtaining an average value of the row heights of all rows where the second text elements are located;

[0049] Determine the average of the row heights of the rows where all the second text elements are located as the height of the interactive area;

[0050] Determining the lower boundary of the interaction bounding box as the lower boundary of the interaction area;

[0051] An upper boundary of the interaction area is determined according to the height of the interaction area and the lower boundary of the interaction area.

[0052] Furthermore, the determining the interactive area information according to the surrounding cells further includes:

[0053] If the second text element is not found, obtaining a cell bounding box; the cell bounding box is a bounding box corresponding to the valid cell;

[0054] Lowering the upper limit of the cell boundary box to determine the upper limit of the interaction area;

[0055] The lower limit of the cell boundary box is adjusted upward to determine the lower limit of the interaction area;

[0056] Adjusting the left boundary of the cell boundary box to the right to determine the left boundary of the interaction area;

[0057] The right boundary of the cell boundary box is adjusted to the left to determine the right boundary of the interaction area.

[0058] Furthermore, determining the interactive area information corresponding to the valid cell according to the search results includes:

[0059] When the interactive element is found, performing type detection on the interactive element;

[0060] In the case where it is detected that the interactive element belongs to the second type of interactive element, acquiring an interactive bounding box; the interactive bounding box is a bounding box corresponding to the interactive element;

[0061] The interaction bounding box is determined as an interaction area in the valid cell.

[0062] Furthermore, the table interaction area identification method also includes:

[0063] generating an interactive function description code according to the interactive area information;

[0064] The interactive function description code is added to the document to be identified.

[0065] On the other hand, an embodiment of the present invention further includes a device for identifying a table interaction area in a document, the device for identifying a table interaction area including:

[0066] A cell recognition module is used to recognize valid cells in the document to be recognized;

[0067] A search module, used for searching for interactive elements in the area where the valid cells in the document to be identified are located; the interactive elements are document elements in the document to be identified that have an interactive attribute indication function;

[0068] The interactive area determination module is used to determine the interactive area information corresponding to the valid cell according to the search result.

[0069] The beneficial effects of the present invention are as follows: the table interactive area recognition method and device in the document of the embodiment can automatically recognize the area in the document to be recognized that is not marked but has interactive attributes as the interactive area, thereby realizing automatic recognition and addition of the interactive area, and improving recognition efficiency and accuracy; moreover, it can also efficiently generate or restore the interactive area information when the editor has not edited the interactive area or the interactive area information is lost, so that the user can interact and edit the document to be recognized through the interactive area without entering the editing mode, thereby improving the convenience of using the electronic document. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1A schematic diagram of the steps of a method for identifying a table interaction area in a document in an embodiment;

[0071] Figure 2 is a schematic diagram of a table area and valid cells in a document in an embodiment;

[0072] Figure 3 A schematic diagram of the steps of identifying a table area in a document to be identified in an embodiment;

[0073] Figure 4 A schematic diagram of a table line shape element in an embodiment;

[0074] Figure 5 This is a schematic diagram of the principles of steps P201-P206 in the embodiment;

[0075] Figures 6(a), 6(b), 6(c), 6(d) and 6(e) are schematic diagrams of the principles of steps P3-P4 in the embodiment;

[0076] Figure 7 Schematic diagram of the principle of detecting the concavity and convexity of the shape of the document area according to the intersection-over-union ratio in the embodiment;

[0077] Figure 8 A schematic diagram of a non-full-border table that can be identified in an embodiment;

[0078] FIG9(a) and FIG9(b) are schematic diagrams showing the principle of a first execution method of step P40201 in the embodiment;

[0079] FIG. 10( a ) and FIG. 10( b ) are schematic diagrams showing the principle of a second execution method of step P40201 in the embodiment;

[0080] FIG. 11( a ), FIG. 11( b ) and FIG. 11( c ) are schematic diagrams showing the principle of step P40203 in the embodiment;

[0081] Fig.12 A schematic diagram of the steps of identifying valid cells in a document to be identified in an embodiment;

[0082] 13(a), 13(b) and 13(c) are schematic diagrams of sub-areas divided by table line shape elements in an embodiment;

[0083] Fig.14 This is a schematic diagram of the principle of step P7 in the embodiment;

[0084] Fig.15 It is a schematic flow chart of the first processing process in the embodiment;

[0085] Fig.16 This is an example diagram of the first processing process in the embodiment;

[0086] Fig.17 Schematic diagram of the second process in the embodiment;

[0087] Fig.18 This is an example diagram of the second processing process in the embodiment;

[0088] Fig.19 A schematic diagram of a complex table that can be identified as a valid cell in an embodiment;

[0089] Fig. 20 is a schematic diagram of an embedded table that can be identified as a valid cell in an embodiment;

[0090] Fig.21 is a schematic diagram of a table with non-aligned lines that can be identified as valid cells in an embodiment;

[0091] Fig. 22 A schematic diagram of a modification relationship between a first type of interactive element and a first text element in an embodiment;

[0092] Fig.23 A schematic diagram of setting the allowable range in an embodiment;

[0093] Fig.24 It is a schematic diagram of determining the interactive area information according to the valid first-type interactive element in an embodiment;

[0094] Fig.25 It is a schematic diagram of determining the interactive area information according to the second text element in the surrounding cells in an embodiment;

[0095] Fig.26 is a schematic diagram of determining the interactive area information according to the cell boundary box in an embodiment;

[0096] Fig. 27 It is a structural schematic diagram of a device for identifying valid cells of a table in a document in an embodiment;

[0097] Fig.28 is a schematic diagram of the structure of a computer device in an embodiment;

[0098] Fig.29 Schematic diagram of the working principle of the computer-readable storage medium in the embodiment. DETAILED DESCRIPTION

[0099] For a document to be identified with a table, if the interactive area is not identified, the cells with modification permissions (equivalent to the cells containing the interactive area) in the document to be identified cannot be directly edited, and additional manual settings may be required to edit such cells, thereby affecting the editing efficiency of the document to be identified; if all the cells are set to editable, then although this will cover all the cells and actually realize the identification of the interactive area contained in the table, it will also cause the editing permissions of cells that are not necessarily editable to be mistakenly released, resulting in editing errors. Therefore, in order to identify the interactive area in the table, only some cells can be set as editable on this basis, thereby achieving high editing efficiency and low editing errors. It is necessary to perform the table interactive area recognition method.

[0100] Reference Figure 1 The table interaction area identification method in this embodiment includes the following steps:

[0101] S1. Identify valid cells in the document to be identified;

[0102] S2. Searching for interactive elements in the area where valid cells in the document to be identified are located;

[0103] S3. Determine the interactive area information corresponding to the valid cells according to the search results.

[0104] Steps S1-S3 are to identify the interactive area information in the valid cells in the document to be identified, such as whether the valid cell contains an interactive area, and the specific location of the interactive area if the interactive area is contained.

[0105] In step S1, the valid cells are components of the table area in the document to be identified. Figure 2 is an example of a valid cell, such as Figure 2 As shown, a page of the document to be identified includes a table area, and the table area includes a plurality of valid cells.

[0106] In this embodiment, the document to be recognized is a layout document, and a layout document such as PDF (Portable Document Format) is taken as an example of the document to be recognized for description.

[0107] In this embodiment, the table area can be Figure 2 In addition to the full-border table (a table with a complete border outside) shown in the figure, a non-full-border table can also be used. Figure 2In addition to the "regular" cells divided by table lines (for example, completely containing four closed border lines) as shown, it can also be a part of special-shaped tables such as embedded tables, non-full-border tables (tables with some or all of the external borders missing). Parts of special-shaped tables often appear "unconventional" because they are not divided by table lines (for example, only contain one border line), but because they have the properties of cells (for example, they can accommodate at least part of the information to be displayed in the table, can be distinguished from other parts, and are independently editable), then regardless of whether this part is divided by table lines, this part actually plays the function of a cell, and this part can be called a valid cell.

[0108] When executing step S1, if the document to be identified contains a regular table such as a full-border table, then the table area therein can be directly determined based on the data of the document to be identified, and the valid cells therein can be determined based on the table line divisions in the table area. If the document to be identified does not contain a regular table, so that the table area cannot be directly determined based on the data of the document to be identified, or there are no clear table line divisions in the table area so that the valid cells therein can be determined, then the valid cells are identified by executing steps P1-P4 or steps P5-P8; wherein steps P1-P4 can determine the valid cells contained in the table area during the process of identifying the table area in the document to be identified. In the embodiment provided by the present invention, the valid cells therein can be determined directly based on the data of the document to be identified, and steps P1-P4 or steps P5-P8 can also be executed to identify the valid cells therein, thereby avoiding missing "unconventional" valid cells.

[0109] In this embodiment, refer to Figure 3 ,The first process of identifying valid cells from the document to be identified includes the following steps:

[0110] P1. Get multiple table line shape elements in the document to be identified;

[0111] P2. According to the connectivity relationship between the line shape elements of each table, at least one connected element is determined based on a plurality of line shape elements of the table;

[0112] P3. Determine a document region bounded by at least one connected element in the document to be identified, and detect the concavity and convexity of the shape of the document region;

[0113] P4. Based on the convexity and concavity detection result, determine the table area from the document to be identified and obtain valid cells.

[0114] In step P1, the document to be recognized is parsed to obtain all document elements in the document to be recognized, and then table line shape elements are filtered out from all document elements. The table line shape elements in step P1 refer to some specific document elements in the document to be recognized. After these document elements are rendered and displayed, the display effect can be used to represent the table line. Specifically, the table line shape elements may be document elements of types such as vector line graph element Path, text element Text, image element Image, or other types of document elements. For example, in the document to be recognized, a specific vector line graph element Path can be used to depict a horizontal or vertical straight line to represent the border or the table line dividing cells inside, and such a vector line graph element Path belongs to the table line shape elements in step P1; the editor of the document to be recognized may, due to personal habits or limited by the functions of the document editing software, use multiple text characters "-" (dash) or "——" (double dash) or Chinese character "一", etc. as text elements Text. When these text elements Text are displayed, their display effect is a horizontal straight line, which can achieve the visual effect of a table line. Similarly, when text characters "|" or Arabic numeral character "1", etc. as text elements Text are displayed, their display effect is a vertical straight line, which can also achieve the visual effect of a table line, and such text elements Text belong to the table line shape elements in step P1; there may also be some image elements Image in the document to be recognized, and the content of these image elements Image is the table line. Therefore, when these image elements Image are displayed, they achieve the effect of displaying the table line, and such image elements Image belong to the table line shape elements in step P1.

[0115] Since the table line shape elements obtained in step P1 may specifically be basic types such as vector line graph element Path, text element Text, image element Image, etc., each table line shape element determines the shape and position of the area in the document to be recognized through the coordinates of its circumscribed bounding box BBox. Therefore, there is a certain connectivity relationship between different table line shape elements, and the connectivity relationship between each table line shape element is determined according to the shape and position of each table line shape element. For example, referring to Figure 4, A1, B1, A2, B2, A3 and B3 are all table line shape elements in the document to be identified. Table line shape element A1 represents a horizontal straight line segment, and table line shape element B1 represents another horizontal straight line segment. According to the external bounding boxes BBox of table line shape elements A1 and B1, it can be judged that the two straight line segments represented by table line shape elements A1 and B1 are parallel, and table line shape elements A1 and B1 are not connected; table line shape element A2 represents a horizontal straight line segment, and table line shape element B2 represents a vertical straight line segment. According to the external bounding boxes BBox of table line shape elements A2 and B2, it can be judged that table line shape element A2 is parallel to the two straight line segments represented by table line shape elements B1. The position of table line shape element B2 in the document to be identified is separated, and table line shape element A2 is not connected to table line shape element B2; table line shape element A3 represents a horizontal straight line segment, and table line shape element B3 represents a vertical straight line segment. According to the external bounding boxes BBox of table line shape elements A3 and B3, it can be judged that the positions of table line shape elements A3 and table line shape elements B3 in the document to be identified are intersecting, which is reflected in that the straight lines represented by table line shape elements A3 and table line shape elements B3 are intersecting, and table line shape elements A3 and table line shape elements B3 are connected; based on the same principle, it can be judged that Figure 4 The table line shape element A4 is connected to the table line shape element B4, and the table line shape element A4 is connected to the table line shape element C4, and this connectivity is transitive, that is, although the table line shape element B4 and the table line shape element C4 are not connected themselves, they are respectively connected to the table line shape element A4, so the table line shape element B4 and the table line shape element C4 are also connected.

[0116] In step P2, each table line shape element is identified as at least one connected element according to the connected relationship between each table line shape element. Figure 4 , for table line shape elements such as A1, B1, A2 and B2 that are not connected to any other table line shape elements, such table line shape elements themselves can be regarded as being connected, so that a single such table line shape element can be used as a connected element, that is, table line shape elements A1, B1, A2 and B2 are each a connected element; and table line shape elements A3 and B3 are connected, thus forming a connected element, and similarly, table line shape elements A4, B4 and C4 are connected, thus forming a connected element. Therefore, by executing step P2, Figure 4The individual table line shape elements in can be identified as connected element A1, connected element B1, connected element A2, connected element B2, connected element A1, connected element (A3+B3) and connected element (A4+B4+C4).

[0117] For each connected element obtained in step P2, such as Figure 4 As shown, for connected elements such as A1, B1, A2 and B2, the aspect ratio or width-to-height ratio of their own external bounding boxes is relatively large, that is, connected elements such as A1, B1, A2 and B2 respectively represent straight lines in the document to be identified. Therefore, connected elements such as A1, B1, A2 and B2 will not frame the document area in the document to be identified by themselves, but need to be combined with at least one other connected element to jointly frame the document area in the document to be identified; for connected elements such as (A3+B3) and (A4+B4+C4), the aspect ratio or width-to-height ratio of their own external bounding boxes is relatively small, that is, connected elements such as (A3+B3) and (A4+B4+C4) are respectively composed of multiple table line shape elements. Therefore, connected elements such as (A3+B3) and (A4+B4+C4) can each frame the document area in the document to be identified separately, and can also be combined with at least one other connected element to jointly frame the document area in the document to be identified.

[0118] In step P3, for connected elements that can independently frame the document area in the document to be identified, the convexity and concavity of the shape of the document area independently framed by such connected elements can be detected first; for connected elements that cannot independently frame the document area in the document to be identified, all such connected elements can be combined together in pairs to detect the convexity and concavity of the shape of the document area jointly framed by the combined multiple connected elements; connected elements that cannot independently frame the document area in the document to be identified can also be combined with connected elements that can independently frame the document area in the document to be identified, and the convexity and concavity of the shape of the document area jointly framed by the combined multiple connected elements can be detected.

[0119] In step P3, the shape convexity of the document region framed by the connected elements individually or together is determined to determine whether the document region is a convex figure, or whether the document region is a non-convex figure (also called a concave figure).

[0120] The mathematical definition of a convex figure is: for a figure F and any two points A and B in F, if all points on the line segment AB are always in the figure F, then the figure F is called a convex figure. The rule for determining a convex figure is: for a polygonal figure F, if any side of the polygon F is extended to a straight line, and the remaining sides of the polygon F are on the same side of the straight line, then the polygonal figure F is determined to be a convex polygon. Therefore, when executing step P3, the following processing can be performed on each document area respectively: detect the boundary of the document area, traverse each boundary of the document area, extend one boundary to a straight line, and determine whether the other boundaries are on the same side of the straight line; if there is any boundary extended to a straight line, the other boundaries are on the same side of the straight line, then the document area can be determined to be a convex figure, otherwise, if there is no boundary extended to a straight line, the other boundaries are on the same side of the straight line, then the document area can be determined to be a non-convex figure.

[0121] In step P4, based on the shape concavity and convexity recognition result of the document area in step P3, the table area is determined from the document to be recognized, and valid cells are obtained.

[0122] The principle of executing steps P1-P4 is that the table area in the document to be identified will affect the distribution of connected elements, thereby affecting the concavity and convexity of the shape of the document area, that is, the concavity and convexity of the shape of the document area includes the positional characteristics of the table area in the document to be identified. Therefore, by detecting the concavity and convexity of the shape of the document area, the table area in the document to be identified can be effectively detected, and then the valid cells can be obtained.

[0123] Reference Figure 3 When executing step P4, that is, determining the table area from the document to be identified based on the concavity and convexity detection result, the following steps can be specifically performed:

[0124] P401. In the case where the shape of the document area is a convex figure, the connected elements that frame the document area are identified as table line elements to be determined;

[0125] P402. Filter the pending elements of each table line, and identify the area where the filtered pending elements of the table line are located as the table area.

[0126] In step P401, all connected elements can be set as non-table line pending elements by default. Traverse all document areas determined by step P3. For any document area, if the shape of this document area is a convex figure, then the single connected element that frames this document area alone is identified as a table line pending element, or multiple connected elements that frame the document area together are all identified as table line pending elements; if the shape of this document area is a non-convex figure, then the connected elements that frame this document area are not identified, and the next document area is read to identify the connected elements.

[0127] By executing steps P1-P3 and P401, it is possible that no connected elements are identified as table line pending elements, some connected elements may be identified as table line pending elements, or all connected elements may be identified as table line pending elements. By executing steps P1-P3 and P401, when there is a document area that is framed by connected elements alone or together as a convex figure, it can be determined that such connected elements meet the necessary conditions for forming a table area, so that such connected elements are identified as table line pending elements, and then step P402 is executed to further screen each table line pending element to confirm whether each table line pending element meets the necessary conditions for forming a table area, and the table line pending elements that meet the conditions are screened out and identified as a table in the table area.

[0128] In this embodiment, when executing step P1, that is, the step of obtaining multiple table line shape elements in the document to be identified, the following steps may be specifically performed:

[0129] Acquire multiple document elements from the document to be identified; wherein the document element is at least one of a text document element, a path document element, and a picture document element;

[0130] When a text document element is obtained, content detection is performed on the text document element, and when it is detected that the text document element contains preset characters, the text document element is determined to be a table line shape element;

[0131] When a path-type document element is obtained, the path-type document element is subjected to instruction parsing and bounding box extraction, and when a straight line drawing instruction is parsed but a curve line drawing instruction is not parsed, and the aspect ratio or height-to-width ratio of the extracted bounding box is greater than a first ratio threshold, the path-type document element is determined to be a table line shape element;

[0132] When a picture-type document element is obtained, the picture-type document element is rendered and grayscaled, and then pixel detection is performed on the boundary of the picture-type document element. When continuous pixels with non-second pixel values ​​are detected on any boundary, and the ratio of the continuous length to the length of the boundary is greater than the second ratio threshold, the boundary of the picture-type document element is determined as a table line shape element.

[0133] The above steps are the first execution mode of step P1.

[0134] Among them, from all the document elements obtained from the document to be identified, some or all types of document elements such as text document elements Text, path document elements Path, and image document elements Image are screened out.

[0135] If the above steps of obtaining multiple document elements are executed to filter out the text - type document element Text, for the text - type document element Text, directly detect whether the content such as its text Unicode or character code CharCode contains a preset character in the form of a straight - line segment, such as a dash "-", an em - dash "——", "|", the Chinese character "一", or the Arabic numeral "1". Determine the text - type document element Text with the above - mentioned preset character content as a table - line - shaped element and store it in LineGroup.

[0136] If the above steps of obtaining multiple document elements are executed to filter out the path - type document element Path, perform instruction parsing and bounding - box extraction on the path - type document element Path. For a path - type document element Path, if a BezierTo instruction (curve - drawing instruction) is parsed, it indicates that the path - type document element Path corresponds to a curve, and such a path - type document element Path will not be a table line, so bounding - box extraction can be skipped; if a LineTo instruction (straight - line - drawing instruction) is parsed and the BezierTo instruction is not parsed, it indicates that the path - type document element Path depicts a straight - line - related graphic (such as a rectangle or a line segment), then continue to perform bounding - box extraction on the path - type document element Path to obtain its circumscribed bounding - box coordinates BBox. For the path - type document element Path depicting a horizontal straight line, the aspect ratio BBox.Width / BBox.Height can be calculated based on its circumscribed bounding - box coordinates BBox; for the path - type document element Path depicting a vertical straight line, the aspect ratio BBox.Height / BBox.Width can be calculated based on its circumscribed bounding - box coordinates BBox. The first ratio threshold can be set to 5. When the aspect ratio BBox.Width / BBox.Height is greater than 5 (i.e., BBox.Width>5 * BBox.Height) or the aspect ratio BBox.Height / BBox.Width is greater than 5 (i.e., BBox.Height>5 * BBox.Width), it can be considered that the path - type document element Path is used to depict a sufficiently slender rectangle, that is, the path - type document element Path depicts a straight line. Therefore, determine such a path - type document element Path as a table - line - shaped element.

[0137] For the path - type document element Path depicting a horizontal straight line, other judgment conditions can also be set. For example, when the following conditions are met simultaneously, determine the path - type document element Path as a table - line - shaped element and store it in LineGroup:

[0138] BBox.Width>5 * BBox.Height and BBox.Height<5.25

[0139] |Path.StartPoint.y-Path.EndPoint.y|<0.1

[0140] Among them, Path.StartPoint.y is the Y coordinate of the path-type document element Path before moving, and Path.EndPoint.y is the Y coordinate of the path-type document element Path after moving.

[0141] For a path-type document element Path that depicts a vertical straight line, other judgment conditions may be set. For example, when the following conditions are met at the same time, the path-type document element Path is determined as a table line shape element and stored in LineGroup:

[0142] BBox.Height>5*BBox.Width and BBox.Width<5.25

[0143] |Path.StartPoint.x-Path.EndPoint.x|<0.1

[0144] Among them, Path.StartPoint.x is the X coordinate of the path-like document element Path before moving, and Path.EndPoint.x is the X coordinate of the path-like document element Path after moving.

[0145] If the above-mentioned step of obtaining multiple document elements is executed to filter out a picture-type document element Image, after rendering and grayscale processing of the picture-type document element Image, the pixel value of each pixel in the picture-type document element Image is respectively the first pixel value (0, indicating pure black), the second pixel value (255, indicating pure white) or a pixel value between the first pixel value and the second pixel value; pixel detection is performed on the boundary of the picture-type document element Image, for a picture-type document element Image, if it is detected that there are continuous pixels with non-second pixel values ​​(i.e., non-255) on any boundary thereof, and the ratio of the continuous length to the length of the boundary is greater than the second ratio threshold, then this edge of the picture-type document element Image is determined as a table line shape element. For example, the length of an image document element Image is H, the width is W, Pixel(x,y) represents the pixel value of the yth row and xth column in the image document element Image, and the value range is 0 to 255. The second ratio threshold (indicating the minimum allowed line length, which can be set between 0 and 1) is set to 0.5. If there is a continuous x value range i to j, so that Pixel(x,0) is less than 255 (or less than 200 or a smaller value, indicating that the color of Pixel(x,0) is dark enough), and ji>H*0.5 (indicating that Pixel(x,0) is long enough), then the edge of Pixel(x,0) is determined as a table line shape element, and the coordinates of the edge of Pixel(x,0) (Image.left, Image.top+i, Image.left+linewidth, Image.top+j) are stored in LineGroup to represent the corresponding table line shape element. Among them, linewidth is the line width, which can be a fixed value such as 1 or 0, and Image.left and Image.top are the coordinates of the external bounding box corresponding to this image document element Image.

[0146] By executing the above steps, table line shape elements belonging to various document element types can be identified from the document to be identified and stored in LineGroup, that is, LineGroup is a collection of various table line shape elements.

[0147] In this embodiment, when executing step P2, that is, the step of identifying each table line shape element as at least one connected element according to the connected relationship between each table line shape element, the following steps may be specifically performed:

[0148] P201. Create a binary image;

[0149] P202. Map the document to be identified to a binary image;

[0150] P203. Set the pixel value of the portion of the binary image mapped by the table line shape element to the first pixel value, and set the pixel value of the portion not mapped by the table line shape element to the second pixel value;

[0151] P204. Solve the connected domain of the binary image to determine at least one connected domain in the binary image; the pixel value of the connected domain is the first pixel value;

[0152] P205. Map the binary graph after the connected domain solution back to the document to be identified;

[0153] P206. Identify each table line shape element mapped to each connected domain as a connected element.

[0154] The principle of steps P201-P206 is as follows Figure 5 shown.

[0155] Reference Figure 5 In step P201, a binary image Canvas is created that is proportional to the size of the document to be identified. Specifically, the size of the binary image Canvas can be determined by scaling it proportionally according to a certain ratio Ratio according to the size of the document to be identified. For example, if the size of the document to be identified is 1280×720, it can be scaled by 0.5 times to determine the size of the binary image Canvas to be 640×360. Generally, the smaller the Ratio, the faster the subsequent processing speed, but the accuracy will be affected. The Ratio can be set to around 0.2.

[0156] In step P201, the initial pixel value of each point in the binary image Canvas may be set to the second pixel value 255 (pure white).

[0157] In step P202, the coordinates corresponding to each table line shape element stored in the LineGroup in the document to be recognized are calculated according to the Ratio and mapped to the corresponding points in the binary image Canvas.

[0158] In step P203, the pixel value of the pixel point mapped in the binary image Canvas is set to the first pixel value 0 (pure black), and the pixel value of the part not mapped in the binary image Canvas is still the second pixel value 255. The pixel value settings for each point in the binary image in step P201 and step P203 are not unique. For example, when executing step P201, the initial pixel value of each point in the binary image is set to the first pixel value 0 (pure black), and when executing step P203, the pixel value of the pixel point mapped in the binary image is set to the second pixel value 255 (pure white). When executing steps P201 and P203, the initial pixel value of each point in the binary image and the pixel value of the pixel point mapped can also be set to other values, preferably so that a larger difference (for example, more than 50) is maintained between them.

[0159] In step P204, the binary image Canvas is solved for connected domains. Specifically, a connected domain solving algorithm such as a depth-first traversal algorithm may be used. The connected domain solving algorithm first dilates and then erodes the binary image Canvas, thereby determining a connected domain in which each pixel value in the binary image Canvas is the first pixel value 0.

[0160] In step P205, the coordinates of each connected domain in the binary image Canvas are divided by Ratio and then mapped back to the document to be identified. Each connected domain with a first pixel value of 0 in the binary image Canvas is mapped to the coordinates in the document to be identified, and a corresponding connected element in the document to be identified can be determined.

[0161] In this embodiment, after executing steps P201-P203 and before executing step P204, the binary image Canvas may be closed to repair small gaps in the binary image Canvas caused by loss of mapping accuracy, which is conducive to more accurate calculation of connected domains in the binary image Canvas.

[0162] In this embodiment, by executing steps P201-P205, the binary graph and connected domain solving algorithm can be applied to quickly identify connected elements such as A1, B1, A2, B2, (A3+B3) and (A4+B4+C4) in the document to be identified.

[0163] In this embodiment, when executing step P3, that is, determining a document region bounded by at least one connected element in the document to be identified, the following steps may be specifically performed:

[0164] P301. Get the first circumscribed bounding box of the first connected element;

[0165] P302. When the height-to-width ratio or the width-to-height ratio of the first circumscribed bounding box is less than the first ratio threshold, the document region bounded by the first circumscribed bounding box and the first connected element is determined as the first document region; wherein the first document region refers to the document region bounded solely by the first connected element;

[0166] P303. When the height-to-width ratio or the width-to-height ratio of the first circumscribed bounding box is greater than or equal to the first ratio threshold, obtain a second circumscribed bounding box common to the first connected element and the second connected element, and determine the document area framed by the second circumscribed bounding box, the first connected element and the second connected element as a second document area; wherein the second connected element is another connected element other than the first connected element, and the second document area represents the document area framed by the first connected element and the second connected element.

[0167] In this embodiment, when executing step P401, that is, when the shape of the document area is a convex figure, identifying the connected elements framing the document area as table line to-be-determined elements, the following steps may be specifically performed:

[0168] P40101. When the shape of the first document region is a convex figure, the first connected element corresponding to the first document region is identified as a table line to be determined element;

[0169] P40102. When the shape of the second document area is a convex figure, the first connected element and the second connected element corresponding to the second document area are identified as table line pending elements.

[0170] In step P301, Figure 4 Among the multiple connected elements shown, such as A1, B1, A2, B2, (A3+B3), (A4+B4+C4), any one connected element is selected as the first connected element. The circumscribed bounding box of the first connected element, i.e., the first circumscribed bounding box, is obtained. The first circumscribed bounding box is the smallest rectangular box that can enclose all the table line shape elements contained in the first connected element, and the coordinates of the first circumscribed bounding box can be obtained according to the maximum values ​​of the coordinates of all the table line shape elements contained in the first connected element.

[0171] For example, in step P301, (A3+B3) can be selected as the first connected element, and its first circumscribed bounding box is shown as the dotted box in Figure 6(a). Since the first connected element (A3+B3) contains two mutually perpendicular table line shape elements A3 and B3, the height-to-width ratio or the width-to-height ratio of the first circumscribed bounding box is less than the first ratio threshold, then step P302 is executed. Referring to Figure 6(a), the first circumscribed bounding box and the first connected element (A3+B3) frame document areas such as R1, R2, R3 and R4, that is, the first connected element (A3+B3) alone frames the first document areas such as R1, R2, R3 and R4. Since the first document area is detected, step P40101 is executed when step P401 is executed, and the convexity of the first document areas R1, R2, R3 and R4 is judged. They are all rectangles, so the first document areas R1, R2, R3 and R4 are all convex figures. In step P40101, the first connected element (A3+B3) corresponding to the first document areas R1, R2, R3 and R4 is identified as a table line to be determined element.

[0172] For example, in step P301, (A4+B4+C4) can be selected as the first connected element, and its first circumscribed bounding box is shown as the dotted box in Figure 6(b). Since the first connected element (A4+B4+C4) contains three mutually parallel or perpendicular table line shape elements A4, B4 and C4, the height-to-width ratio or the width-to-height ratio of the first circumscribed bounding box is less than the first ratio threshold, then step P302 is executed. Referring to Figure 6(b), the first circumscribed bounding box and the first connected element (A4+B4+C4) frame document areas such as R5, R6, R7 and R8, that is, the first connected element (A4+B4+C4) alone frames the first document areas such as R5, R6, R7 and R8. Since the first document area is detected, step P40101 is executed when step P401 is executed to judge the convexity of the first document areas R5, R6, R7 and R8, wherein R5 and R8 are both rectangles (convex figures). If the line width of table line shape elements such as B4 and C4 is set to 0, then R6 and R7 are also rectangles (convex figures). The first document areas R5, R6, R7 and R8 are all convex figures. In step P40101, the first connected element (A4+B4+C4) corresponding to the first document areas R5, R6, R7 and R8 is identified as a table line element to be determined; the line width of table line shape elements such as B4 and C4 can also be set to be greater than 0 (for example, 1), then R6 and R7 are rectangles with grooves (non-convex figures). The first document areas R5, R6, R7 and R8 are not all convex figures, and (A4+B4+C4) is not directly identified as a table line element to be determined.

[0173] For example, in step P301, A1 can be selected as the first connected element, and its first circumscribed bounding box is the circumscribed bounding box of the table line shape element A1. Therefore, the aspect ratio of the first circumscribed bounding box is greater than the first ratio threshold. Then, step P303 is executed to select any other connected element, that is, the second connected element. For example, B1 is selected as the second connected element in FIG6(c), and the second circumscribed bounding box common to the first connected element A1 and the second connected element B1 is obtained. As shown in the dotted box in FIG6(c), the second circumscribed bounding box is the minimum rectangular box that can contain the first connected element A1 and the second connected element B1. The document area R9 framed by the second circumscribed bounding box, the first connected element A1 and the second connected element B1 is determined as the second document area. That is, the first connected element A1 and the second connected element B1 jointly frame the second document area R9. Since the second document area is detected, step P40102 is executed when step P401 is executed to judge the convexity of the second document area R9. Since the second document area R9 is a rectangle (convex figure), in step P40102, the first connected element A1 and the second connected element B1 corresponding to the second document area R9 are both identified as table line pending elements.

[0174] For example, in step P301, B1 can be selected as the first connected element, and its first external bounding box is the external bounding box of the table line shape element B1. Therefore, the aspect ratio of the first external bounding box is greater than the first ratio threshold. Then, step P303 is executed to select any other connected element, that is, the second connected element. For example, (A3+B3) is selected as the second connected element in FIG6(d), and the second external bounding box common to the first connected element B1 and the second connected element (A3+B3) is obtained. As shown in the dotted box on the left side of FIG6(d), the second external bounding box is the smallest rectangular box that can contain the first connected element B1 and the second connected element (A3+B3). The document area R10, R11 and R12 framed by the second external bounding box, the first connected element B1 and the second connected element (A3+B3) is determined as the second document area. That is, the first connected element B1 and the second connected element (A3+B3) jointly frame the second document area R10, R11 and R12. Since the second document area is detected, step P40102 is executed when step P401 is executed to judge the convexity and concavity of the second document areas R10, R11 and R12. When the line width of the table line shape elements A3 and B3 is set to 0, the second document areas R10, R11 and R12 are all rectangles (convex figures). In step P40102, the first connected element B1 and the second connected element (A3+B3) corresponding to the second document areas R10, R11 and R12 are both identified as pending elements of the table line.

[0175] For example, in step P301, A2 can be selected as the first connected element, and its first circumscribed bounding box is the circumscribed bounding box of the table line shape element A2. Therefore, the aspect ratio of the first circumscribed bounding box is greater than the first ratio threshold. Then execute step P303, select any other connected element, that is, the second connected element. For example, (A4+B4+C4) is selected as the second connected element in Figure 6(d), and the second circumscribed bounding box of the first connected element A2 and the second connected element (A4+B4+C4) is obtained. As shown in the dotted box on the right side of Figure 6(d), the second circumscribed bounding box is the minimum rectangular box that can contain the first connected element A2 and the second connected element (A4+B4+C4). The document areas R13 and R14 framed by the second circumscribed bounding box, the first connected element A2 and the second connected element (A4+B4+C4) are determined as the second document area. That is, the first connected element A2 and the second connected element (A4+B4+C4) jointly frame the second document regions R13 and R14. Since the second document region is detected, step P40102 is executed when step P401 is executed to determine the convexity of the second document regions R13 and R14. However, regardless of whether the line width of the table line shape element A2 is set to 0, the second document region R13 is not a convex figure. In step P40102, A2 or (A4+B4+C4) is not identified as a table line to be determined element.

[0176] For example, in step P301, A2 can be selected as the first connected element, and its first external bounding box is the external bounding box of the table line shape element A2. Therefore, the aspect ratio of the first external bounding box is greater than the first ratio threshold. Then, step P303 is executed to select any other connected element, that is, the second connected element. For example, B2 is selected as the second connected element in FIG6(e), and the second external bounding box common to the first connected element A2 and the second connected element B2 is obtained. As shown in the dotted box in FIG6(e), the second external bounding box is the minimum rectangular box that can contain the first connected element A2 and the second connected element B2. The document area R15 framed by the second external bounding box, the first connected element A2 and the second connected element B2 is determined as the second document area. That is, the first connected element A2 and the second connected element B2 jointly frame the second document area R15. Since the second document area is detected, step P40102 is executed when step P401 is executed to judge the convexity and concavity of the second document area R15. When the line width of the table line shape elements A2 and B2 is set to 0, the second document area R15 is a rectangle (convex figure). In step P40102, the first connected element A2 and the second connected element B2 corresponding to the second document area R15 are both identified as table line pending elements. Conversely, if the line width of the table line shape elements A2 and B2 is not set to 0 (for example, 1), the first connected element A2 and the second connected element B2 are not identified as table line pending elements.

[0177] It can be known from the above embodiments that by executing steps P301-P303 and P40101-P40102, for the first document regions or second document regions such as (R1+R2+R3+R4), (R5+R6+R7+R8), R9, (R10+R11+R12) and R15 in Figure 6(a) and Figure 6(b), based on their convexity and concavity, it can be judged that the corresponding first connected elements or second connected elements are more likely to belong to a table in the document, thereby identifying them as table line pending elements, and for the second document region (R13+R14), based on its convexity and concavity, it is judged that the corresponding first connected element A2 and the second connected element (A4+B4+C4) are less likely to belong to a table in the document, thereby not identifying these first connected elements A2 and the second connected elements (A4+B4+C4) as table line pending elements. By executing steps P301-P303 and P40101-P40102, a rough screening of table line shape elements can be completed, and the identified table line elements to be determined are stored in Table, where Table is a data table.

[0178] In this embodiment, when executing step P3 to detect the convexity of the shape of the document area, when executing step P3 to obtain the first circumscribed bounding box and the second circumscribed bounding box, the intersection of the first circumscribed bounding box and the projection of the second circumscribed bounding box can be calculated, and the intersection of the first circumscribed bounding box and the projection of the second circumscribed bounding box can be compared with the intersection of the first and second connected elements. The ratio is a threshold value, thereby determining whether the shape of the document area framed by the first connected element and the second connected element is a convex figure. Specifically, the diagonal coordinates line(i) of the first circumscribed bounding box can be obtained. left and line(i) right And the diagonal coordinates of the second circumscribed bounding box line(j) left and line(j) right , and calculate the intersection-over-union (IOU) ratio using the following formula:

[0179] I left =max[line(i) left ,line(j) left ]

[0180] I right = min[line(i) right ,line(j) right ]

[0181] U left = line[line(i) left ,line(j) left ]

[0182] U right =max[line(i) right ,line(j) right ]

[0183]

[0184] Union=U right -U left

[0185]

[0186] Among them, line(i) left and line(i) right is the diagonal coordinate of the first circumscribed bounding box, line(j) left and line(j) right is the diagonal coordinate of the second external bounding box, I left is line(i) left and line(j) left The maximum value in I right is line(i) rightand line(j) right The minimum value in left is line(i) left and line(j) left The minimum value in right is line(i) right and line(j) right Intersect means the length of the intersection of the projections of the first external bounding box and the second external bounding box, Union means the length of the union of the projections of the first external bounding box and the second external bounding box, and IOU is the intersection-union ratio.

[0187] In this embodiment, the size of the IOU is between 0 and 1. The meaning of the IOU is the degree of alignment of the projections between the first circumscribed bounding box of the first connected element and the second circumscribed bounding box of the second connected element. The higher the degree of alignment of the projections, the greater the IOU. Therefore, the IOU threshold can be set to 0.95. If the IOU is greater than the IOU threshold of 0.95, it can be determined that the first circumscribed bounding box is aligned with the second circumscribed bounding box. Figure 7 , where IOU1 is close to 1, that is, the first connected element A1 and the second connected element B1 are aligned (equivalent to the text area jointly framed by the first connected element A1 and the second connected element B1 being a convex figure), so the first connected element A1 and the second connected element B1 can be identified as table line pending elements; and IOU2 is close to 0, that is, the first connected element A2 and the second connected element B2 are not aligned (when the line width is greater than 0, it is equivalent to the text area jointly framed by the first connected element A2 and the second connected element B2 being a non-convex figure), so the first connected element A2 and the second connected element B2 cannot be directly identified as table line pending elements.

[0188] By calculating the intersection-over-union (IOU) ratio to identify the pending elements of the table line, there is no need to judge the convex figure, which is beneficial to reduce the amount of data processing and improve the recognition speed.

[0189] By executing step P401, even if the Figure 8 The complex non-full-border tables such as the "I"-shaped table, "F"-shaped table and "Jing"-shaped table shown can also be accurately identified as table line pending elements, and screened and identified as area tables in the further step P402, thereby having good recognition capabilities for special-shaped tables such as non-full-border tables.

[0190] In this embodiment, when executing step P402, that is, screening each table line pending element and identifying the area where the screened table line pending element is located as the table area, the following steps may be specifically performed:

[0191] P40201. Determine the table characteristics of the pending elements of each table line;

[0192] P40202. When it is determined that the table line pending element does not have table characteristics, the table line pending element is eliminated;

[0193] P40203. Identify the pending elements of the table lines that have not been eliminated as table areas.

[0194] By executing step P401, the table line element to be determined in the obtained Table is likely to belong to a certain table, so continue to execute steps P40201-P40203 for judgment.

[0195] In step P40201, table feature judgment is performed on each table line pending element. If it is determined that the table line pending element does not have the table feature, step P40202 is executed to eliminate the table line pending elements that do not have the table feature. As a result, the table line pending elements that are not eliminated are more likely to belong to a certain table. Step P40203 can be executed to identify the area where the table line pending elements that are not eliminated are located as a table area.

[0196] In this embodiment, when executing step P40201, that is, the step of determining the table characteristics of each table line element to be determined, the following steps may be specifically performed:

[0197] P4020101A. Obtain the third external bounding box of the pending element of the first table line, and the fourth external bounding box of the pending element of the second table line;

[0198] P4020102A. Detect the positional relationship between the third circumscribed bounding box and the fourth circumscribed bounding box;

[0199] P4020103A. When the third circumscribed bounding box intersects with the fourth circumscribed bounding box and the third circumscribed bounding box and the fourth circumscribed bounding box do not contain each other, it is determined that both the first table line pending element and the second table line pending element do not have table features.

[0200] Taking FIG. 9(a) as an example, in step P4020101A, for any two table line pending elements, i.e., the first table line pending element A5 and the second table line pending element B5, their corresponding third external bounding box and fourth external bounding box are shown as two dotted boxes in FIG. 9(a). By executing P4020102A, it can be detected that the positional relationship between the third external bounding box and the fourth external bounding box is "intersecting, and the third external bounding box and the fourth external bounding box do not contain each other", and the styles corresponding to the first table line pending element A5 and the second table line pending element B5 do not have table features, that is, they do not belong to a certain table, so when executing P4020103A, it is determined that the first table line pending element A5 and the second table line pending element B5 do not have table features, and the first table line pending element A5 and the second table line pending element B5 will be eliminated when executing step P40202.

[0201] Taking FIG. 9(b) as an example, in step P4020101A, for any two table line pending elements, i.e., the first table line pending element A6 and the second table line pending element B6, their corresponding third external bounding box and fourth external bounding box are shown as two dotted boxes in FIG. 9(b). By executing P4020102A, it can be detected that the positional relationship between the third external bounding box and the fourth external bounding box is "intersecting, and the fourth external bounding box contains the fourth external bounding box", and the styles corresponding to the first table line pending element A6 and the second table line pending element B6 have table features, i.e., they belong to a certain embedded table, so when executing P4020103A, it is determined that the first table line pending element A6 and the second table line pending element B6 both have table features, and the first table line pending element A6 and the second table line pending element B6 will not be eliminated when executing step P40202, and the area where the first table line pending element A6 and the second table line pending element B6 are located is identified as a table area when executing step P40203.

[0202] In this embodiment, when executing step P40201, that is, the step of determining the table characteristics of each table line element to be determined, the following steps may be specifically performed:

[0203] P4020101B. Perform a virtual line filling operation on any table line element to be determined, and obtain at least one cell bounded by the table line element to be determined and the virtual line;

[0204] P4020102B. Detect empty cells and solid cells in all cells;

[0205] P4020103B. In the document to be recognized, detect the column or row where any empty cell is located;

[0206] P4020104B. When no real cell corresponding to the same table line pending element is detected, it is determined that the table line pending element does not have table features.

[0207] Steps P4020101B-P4020104B are the second execution method of step P40201.

[0208] Taking Figure 10(a) as an example, in step P4020101B, for any table line element A7 in the Table, a virtual line filling operation is performed. Specifically, the circumscribed bounding box of the table line element A7 is obtained, and the virtual line is filled in the position of the circumscribed bounding box, so as to obtain cells R16-R21 bounded by the table line element A7 and the virtual line frame as shown in Figure 10(b).

[0209] In step P4020102B, all cells R16-R21 are traversed. If any cell contains a document element (specifically, it can be a type such as Text, Path or Image), then this cell is marked as a real cell, otherwise this cell is marked as an empty cell.

[0210] If there are no empty cells in cells R16-R21, then it is determined that the table line pending element A7 has table features, and the table line pending element A7 will not be eliminated when step P40202 is executed. When step P40203 is executed, the area where the table line pending element A7 is located is identified as a table area.

[0211] If there are empty cells in cells R16-R21 (for example, R19 is an empty cell), then the column or row where the empty cell is located is detected. For example, cell R18 is in the same row as the empty cell R19, and cells R17, R20 and R21 are in the same column as the empty cell R19. If cells R17, R18, R20 and R21, etc., which are in the same row or column as the empty cell R19, are all empty cells, then it can be determined that the table line pending element A7 does not have table features, and the table line pending element A7 will be eliminated when executing step P40202, and the area where the table line pending element A7 is located will not be identified as a table area when executing step P40203.

[0212] By executing steps P4020101B-P4020104B, when there are too many empty cells in the same row or column, it is determined that the corresponding table line pending element does not play the role of a table, that is, it does not have table features, thereby avoiding misidentification as a table area.

[0213] In this embodiment, when executing step P40203, that is, identifying the table line pending elements that have not been eliminated as table areas, the following steps may be specifically performed:

[0214] P4020301. Detection is performed between the third table line pending element and the fourth table line pending element;

[0215] P4020302. When no document element is detected, obtain the fifth circumscribed bounding box of the third table line element to be determined and the sixth circumscribed bounding box of the fourth table line element to be determined. When the aspect ratio or height-to-width ratio of the fifth circumscribed bounding box is greater than the first ratio threshold, remove the third table line element to be determined. When the aspect ratio or height-to-width ratio of the sixth circumscribed bounding box is greater than the first ratio threshold, remove the fourth table line element to be determined.

[0216] P4020303. In the case of detecting a text document element whose content contains a table annotation keyword, identifying the third table line pending element and the fourth table line pending element as belonging to different non-full-border tables, detecting the distances between the text document element and the third table line pending element and the fourth table line pending element, respectively, and marking the text document element as belonging to the non-full-border table that is closest in distance;

[0217] P4020304. When a text document element is detected whose content contains non-table annotation keywords, the third table line pending element and the fourth table line pending element are identified as belonging to the same non-full-border table.

[0218] Assume that when executing step P40203, it can be assumed that there are table line pending elements such as A8, B8, C8 and D8 that have not been eliminated.

[0219] In step P4020301, the third table line pending element is set as A8, and the fourth table line pending element is set as B8, and the document area between the third table line pending element A8 and the fourth table line pending element B8 is detected. If, as shown in FIG11(a), there is no document element between the third table line pending element A8 and the fourth table line pending element B8, then the aspect ratio or height-to-width ratio of the fifth circumscribed bounding box corresponding to the third table line pending element A8 can be obtained. If the aspect ratio or height-to-width ratio of the fifth circumscribed bounding box is greater than the first ratio threshold, then the third table line pending element A8 represents a line, that is, the third table line pending element A8 is a line alone and cannot constitute a table, so step P4020302 can be executed to remove the third table line pending element A8. Similarly, if the aspect ratio or height-to-width ratio of the sixth circumscribed bounding box corresponding to the fourth table line pending element B8 is greater than the first ratio threshold, then the fourth table line pending element B8 is a single line and cannot constitute a table, so step P4020302 can be executed to remove the fourth table line pending element B8.

[0220] In step P4020303, if, as shown in Figure 11(b), there is a text document element such as "Table 1" representing the table annotation keyword between the third table line pending element A8 and the fourth table line pending element B8, then the third table line pending element A8 and the fourth table line pending element B8 are identified as belonging to different non-full-border tables, and the distance between the text document element "Table 1" and the third table line pending element A8 and the fourth table line pending element B8 is detected. In Figure 11(b), the text document element "Table 1" is closer to the fourth table line pending element B8, and therefore the text document element "Table 1" is marked as belonging to the table where the fourth table line pending element B8 is located.

[0221] In step P4020304, if as shown in Figure 11(c), there is only a text-type document element such as "Hello" that represents a non-table annotation keyword (or a path-type document element or other content that is not related to the table annotation keyword) between the third table line pending element A8 and the fourth table line pending element B8, then the third table line pending element A8 and the fourth table line pending element B8 are identified as the same non-full-border table.

[0222] By executing steps P4020301-P4020304, when the table line elements to be determined that are not eliminated are lines, especially parallel lines, these table line elements to be determined can be accurately divided into corresponding non-full-border tables.

[0223] By executing steps P1-P4, the valid cells to be identified in step S1 can be obtained from the table area determined in the document to be identified; for example, when there are table lines inside the table area, the cells obtained by dividing the table area by these table lines can be used as the valid cells to be identified in step S1; when executing step P4020101B, the cells framed by the table line elements to be determined and the virtual lines can be used as the valid cells to be identified in step S1; when executing steps P4020101B-P4020102B, the detected empty cells or real cells can also be selected as the valid cells to be identified in step S1.

[0224] In this embodiment, refer to Fig.12 ,The second process of identifying valid cells from the table area in the document to be identified includes the following steps:

[0225] P5. Get the table area in the document to be recognized;

[0226] P6. Get multiple table line shape elements located in the table area;

[0227] P7. In the table area, identifying at least one sub-area divided by each table line shape element;

[0228] P8. Screen each sub-region according to the shape of each sub-region and its positional relationship with the table region, and identify the screened sub-regions as valid cells.

[0229] In step P5, the table region in the document to be recognized can be directly determined according to the table region marked by the data of the document to be recognized itself, or the table region in the document to be recognized can be identified from the document to be recognized by using the table valid cell recognition technology in the document.

[0230] In step P6, the document to be recognized is parsed to obtain all document elements in the document to be recognized, and then the table line shape elements are screened out from all document elements. The table line shape elements in step P6 refer to some specific document elements in the document to be recognized. After these document elements are rendered and displayed, the display effect can be used to represent the table line. Specifically, the table line shape elements may be document elements of types such as vector line graph element Path, text element Text, image element Image, or other types of document elements. For example, in the document to be recognized, a specific vector line graph element Path can be used to depict a horizontal or vertical straight line to represent the border or the table line dividing cells inside, and such a vector line graph element Path belongs to the table line shape elements in step S1; the editor of the document to be recognized may, due to personal habits or limited by the functions of the document editing software, use multiple text characters "-" (dash) or "——" (double dash) or Chinese character "一" and other text elements Text. When these text elements Text are displayed, their display effect is a horizontal straight line, which can achieve the visual effect of a table line. Similarly, when text characters "|" or Arabic numeral character "1" and other text elements Text are displayed, their display effect is a vertical straight line, which can also achieve the visual effect of a table line. Such text elements Text belong to the table line shape elements in step P6; there may also be some image elements Image in the document to be recognized, and the content of these image elements Image is the table line. Therefore, when these image elements Image are displayed, they achieve the effect of displaying the table line. Such image elements Image belong to the table line shape elements in step P6.

[0231] Since the table line shape elements obtained in step P6 may be basic types such as vector line graphic elements Path, text elements Text, image elements Image, etc., each table line shape element determines the shape and position of the area in the document to be identified through the coordinates of its external bounding box BBox. Therefore, in step P7, it is possible to identify at least one sub-area into which the table area is divided by the table line shape element. For example, referring to FIG13(a), table line shape elements A1 and B1 divide the table area into sub-areas R1, R2, and R3; referring to FIG13(b), table line shape elements A2 and B2 divide the table area into sub-areas R4, R5, and R6; referring to FIG13(c), table line shape elements A3, B3, and C3 divide the table area into sub-areas R7, R8, and R9.

[0232] In step P8, each sub-region is screened according to its shape and the positional relationship between each sub-region and the table area, and the screened sub-region is identified as a valid cell.

[0233] Specifically, in step P8, the sub-areas in each table area are screened by using "having a specific shape and having a specific positional relationship with the table area" as the screening condition. For any table area, the screening result may be that some or all of the sub-areas are screened out, or no sub-areas are screened out. For any table area, when at least one sub-area is screened out, the table area as a whole is identified as a table, and the sub-areas screened out from the table area are identified as valid cells in the table.

[0234] In this embodiment, when executing step P6, that is, the step of obtaining a plurality of table line shape elements located in the table area, the following steps may be specifically performed:

[0235] Acquire multiple document elements from the document to be identified; wherein the document element is at least one of a text document element, a path document element, and a picture document element;

[0236] When a text document element is obtained, content detection is performed on the text document element, and when it is detected that the text document element contains preset characters, the text document element is determined to be a table line shape element;

[0237] When a path-type document element is obtained, the path-type document element is subjected to instruction parsing and bounding box extraction, and when a straight line drawing instruction is parsed but a curve line drawing instruction is not parsed, and the aspect ratio or height-to-width ratio of the extracted bounding box is greater than a first ratio threshold, the path-type document element is determined to be a table line shape element;

[0238] When an image - type document element is obtained, after rendering and grayscaling the image - type document element, pixel detection is performed on the boundary of the image - type document element. When it is detected that there are consecutive non - second - pixel - value pixels on any boundary, and the proportion of the consecutive length to the length of the boundary where it is located is greater than the second - proportion threshold, this boundary of the image - type document element is determined as a table - line - shaped element.

[0239] In step P6, from all the document elements obtained from the document to be recognized, document elements of some or all of the types such as text - type document elements Text, path - type document elements Path, and image - type document elements Image are screened out.

[0240] If the text - type document element Text is screened out by performing the above steps of obtaining multiple document elements, for the text - type document element Text, directly detect whether the content such as its text Unicode or character code CharCode contains a preset character in the form of a straight - line segment, such as a horizontal bar "-", an em dash "——", "|", the Chinese character "一", or the Arabic numeral "1". Determine the text - type document element Text with the above - mentioned preset character content as a table - line - shaped element and store it in LineGroup.

[0241] If the above steps of obtaining multiple document elements are performed to filter out the path-type document element Path, the path-type document element Path is subjected to instruction parsing and bounding box extraction. For a path-type document element Path, if the BezierTo instruction (curve drawing instruction) is parsed, it indicates that the path-type document element Path corresponds to a curve. Such a path-type document element Path will not be a table line, so bounding box extraction is not required; if the LineTo instruction (straight line drawing instruction) is parsed but the BezierTo instruction is not parsed, it indicates that the path-type document element Path is a graphic related to depicting a straight line (such as a rectangle or a line segment), then the path-type document element Path is subjected to bounding box extraction to obtain its external bounding box coordinates BBox. For a path-type document element Path that depicts a horizontal straight line, the aspect ratio BBox.Width / BBox.Height can be calculated based on its external bounding box coordinates BBox. For a path-like document element Path that depicts a vertical straight line, the aspect ratio BBox.Height / BBox.Width can be calculated based on its circumscribed bounding box coordinates BBox, and the first ratio threshold can be set to 5. When the aspect ratio BBox.Width / BBox.Height is greater than 5 (i.e., BBox.Width>5*BBox.Height) or the aspect ratio BBox.Height / BBox.Width is greater than 5 (i.e., BBox.Height>5*BBox.Width), it can be considered that the path-like document element Path is used to depict a sufficiently slender rectangle, that is, the path-like document element Path depicts a straight line, and such a path-like document element Path is determined as a table line shape element.

[0242] For a path-type document element Path that depicts a horizontal straight line, other judgment conditions may be set. For example, when the following conditions are met at the same time, the path-type document element Path is determined as a table line shape element and stored in LineGroup:

[0243] BBox.Width>5*BBox.Height and BBox.Height<5.25

[0244] |Path.StartPoint.y-Path.EndPoint.y|<0.1

[0245] For a path-type document element Path that depicts a vertical straight line, other judgment conditions may be set. For example, when the following conditions are met at the same time, the path-type document element Path is determined as a table line shape element and stored in LineGroup:

[0246] BBox.Height>5*BBox.Width and BBox.Width<5.25

[0247] |Path.StartPoint.x-Path.EndPoint.x|<0.1

[0248] If the above-mentioned step of obtaining multiple document elements is executed to filter out a picture-type document element Image, after rendering and grayscale processing of the picture-type document element Image, the pixel value of each pixel in the picture-type document element Image is respectively the first pixel value (0, indicating pure black), the second pixel value (255, indicating pure white) or a pixel value between the first pixel value and the second pixel value; pixel detection is performed on the boundary of the picture-type document element Image, for a picture-type document element Image, if it is detected that there are continuous pixels with non-second pixel values ​​(i.e., non-255) on any boundary thereof, and the ratio of the continuous length to the length of the boundary is greater than the second ratio threshold, then this edge of the picture-type document element Image is determined as a table line shape element.

[0249] When a picture-type document element is obtained, for example, the length of a picture-type document element Image is H, and the width is W, Pixel(x,y) represents the pixel value of the y-th row and x-th column in the picture-type document element Image, and the value range is 0 to 255. The second proportional threshold (indicating the minimum allowed line length, which can be set between 0 and 1) is set to 0.5. If there is a continuous x value range i to j, so that Pixel(x,0) is less than 255 (or less than 200 or other smaller values, indicating that the color of Pixel(x,0) is dark enough), and ji>H*0.5 (indicating that Pixel(x,0) is long enough), then the edge of Pixel(x,0) is determined as a table line shape element, and the coordinates of the edge of Pixel(x,0) (Image.left,Image.top+i,Image.left+linewidth,Image.top+j) are stored in LineGroup to represent the corresponding table line shape element. Among them, linewidth is the line width, which can be a fixed value such as 1 or 0, and Image.left and Image.top are the coordinates of the external bounding box corresponding to the Image element of this image document element.

[0250] By executing step P6, table line shape elements belonging to various document element types can be identified from the document to be identified and stored in LineGroup, that is, LineGroup is a collection of various table line shape elements.

[0251] In this embodiment, when executing step P7, that is, the step of identifying at least one sub-region divided by each table line shape element in the table region, the following steps may be specifically performed:

[0252] P701. Create a binary image;

[0253] P702. Map the table area in the document to be identified to a binary image;

[0254] P703. Set the pixel value of the portion of the binary image mapped by the table line shape element as the first pixel value, and set the pixel value of the portion not mapped by the table line shape element as the second pixel value;

[0255] P704. Solve the connected domain of the binary image to determine at least one connected domain in the binary image; the pixel value of the connected domain is the second pixel value;

[0256] P705. Map the binary graph after the connected domain solution back to the document to be identified;

[0257] P706. Identify the document region in the document to be identified that is mapped by each connected domain as a sub-region.

[0258] The principle of steps P701-P706 is as follows Fig.14 shown.

[0259] Reference Fig.14 In step P701, a binary image proportional to the size of the document to be identified is created. Specifically, the size of the binary image can be determined by scaling it proportionally according to a certain ratio Ratio according to the size of the document to be identified. For example, if the size of the document to be identified is 1280×720, it can be scaled by 0.5 times to determine the size of the binary image to be 640×360. Generally, the smaller the Ratio, the faster the subsequent processing speed, but the accuracy will be affected. The Ratio can be set to around 0.2.

[0260] In step P701, the initial pixel value of each point in the binary image may be set to the second pixel value 255 (pure white).

[0261] In step P702, the coordinates corresponding to each table line shape element stored in the LineGroup in the document to be recognized are calculated according to the Ratio and mapped to the corresponding points in the binary image.

[0262] In step P703, the pixel value of the pixel point mapped in the binary image is set to the first pixel value 0 (pure black), and the pixel value of the part not mapped in the binary image is still the second pixel value 255 (pure white). The pixel value settings for each point in the binary image in step P701 and step P703 are not unique. For example, when executing step P701, the initial pixel value of each point in the binary image can be set to the first pixel value 0 (pure black), and when executing step P703, the pixel value of the pixel point mapped in the binary image can be set to the second pixel value 255 (pure white). When executing steps P701 and P703, the initial pixel value of each point in the binary image and the pixel value of the pixel point mapped can also be set to other values, preferably so that a large difference (for example, more than 50) is maintained between them.

[0263] In step P704, the binary graph is solved for connected domains. Specifically, a connected domain solving algorithm such as a depth-first traversal algorithm may be used. The connected domain solving algorithm first dilates and then erodes the binary graph, thereby determining connected domains where each pixel value in the binary graph is the second pixel value 255.

[0264] In step P705, the coordinates of each connected domain in the binary image are divided by Ratio and then mapped back to the document to be identified. Each connected domain in the binary image with a pixel value of the second pixel value 255 is mapped to the coordinates in the document to be identified, and a corresponding sub-region in the corresponding table region in the document to be identified can be determined.

[0265] In this embodiment, after executing steps P701-P703 and before executing step P704, a closing operation may be performed on the binary graph to repair small gaps in the binary graph caused by loss of mapping accuracy, which is conducive to more accurate calculation of connected domains in the binary graph.

[0266] In this embodiment, by executing steps P701 to P705, the binary graph and connected domain solving algorithm can be applied to quickly identify sub-regions such as R1 to R9 in the table region of the document to be identified.

[0267] After identifying sub-regions R1-R9, the information (edge ​​coordinates) of these sub-regions can be stored in Contours. The i-th sub-region can be represented by Contours(i). At the same time, the information of each table region (edge ​​coordinates of all 4 edges) is stored in Canvas.

[0268] In this embodiment, when executing step P8, that is, screening each sub-region according to its shape and the positional relationship with the table area, and identifying the screened sub-region as a valid cell, multiple rounds of first processing procedures can be executed in sequence. Except for the first round of first processing procedures, each round of first processing procedures is based on the execution result of the previous round of first processing procedures, and continues to perform corresponding processing until no valid cells are identified in a certain round of first processing procedures, and the execution of all first processing procedures is terminated, that is, the subsequent first processing procedures are no longer executed.

[0269] In this embodiment, refer to Fig.15 , any round of the first processing process includes the following steps:

[0270] P801A. In the case where the first processing process of this round is the first processing process of the first round, the table border is detected from the table area identified in the document to be identified, and conversely, the table border is detected from the table area updated from the previous first processing process;

[0271] P802A. If at least one edge of any sub-region overlaps with the detected table border and the shape of the sub-region is a convex figure, the sub-region is identified as a valid cell;

[0272] P803A. Delete the sub-area belonging to the valid cells from the table area, thereby obtaining the table area updated by the first processing step of this round.

[0273] Reference Fig.15In step P801A, if the current process is the first round of the first processing, then the table border Canvas of the table area identified from the document to be identified is obtained; if the current process is not the first round of the first processing (for example, the second round, the fifth round of the first processing, etc.), then the table border Canvas updated by the previous round of the first processing is obtained; execute step P802A, traverse all the sub-areas in the table area corresponding to the table border Canvas, for example, for the i-th sub-area Contours(i), determine whether the sub-area Contours(i) has at least one edge line that is consistent with the table border Canvas. If the corresponding edge in as coincides, if not, then the sub-region Contours(i) is not determined as a valid cell, and i=i+1 is set, that is, the i+1th sub-region Contours(i+1) is determined; if there is at least one edge in the sub-region Contours(i) that coincides with the corresponding edge in the table border Canvas, then it is determined whether the shape of the sub-region Contours(i) is a convex figure. If it is a convex figure, then the sub-region Contours(i) can be determined as a valid cell. If it is not a convex figure, then the sub-region Contours(i) is not determined as a valid cell.

[0274] In step P802A, the mathematical definition of a convex figure is: for a figure F and any two points A and B in F, if all points on the line segment AB are always in the figure F, then the figure F is called a convex figure. The rule for determining a convex figure is: for a polygonal figure F, if any side of the polygon F is extended to a straight line, and the remaining sides of the polygon F are on the same side of the straight line, then the polygonal figure F is determined to be a convex polygon. Therefore, when executing step P802A, the sub-region Contours(i) can be processed as follows: detect the boundary of the sub-region Contours(i), traverse each boundary of the sub-region Contours(i), extend one boundary to a straight line, and determine whether the other boundaries are on the same side of the straight line; if there is any boundary that is extended to a straight line and the other boundaries are on the same side of the straight line, then the sub-region Contours(i) can be determined to be a convex figure. On the contrary, if there is no boundary that is extended to a straight line and the other boundaries are on the same side of the straight line, then the sub-region Contours(i) can be determined to be a non-convex figure.

[0275] If in step P802A, sub-area Contours(i) is determined to be a valid cell, then when executing step P803A, sub-area Contours(i) is deleted from the table area corresponding to the table border Canvas, and the remaining area is used as the new table area. The table border Canvas also changes, thereby updating the table border Canvas. Fig.15The updated table border Canvas is called when the next round of the first processing process executes step P801A.

[0276] Fig.15 An example of steps P801A-P803A shown is as follows Fig.16 See Fig.16 A table area of ​​the document to be identified corresponds to the table border Canvas1, and the table line shape elements therein are identified to divide the table area into sub-areas R10, R11, R12, R13 and R14. Steps P801A-P803A are performed on the table area.

[0277] In the first round of the first processing, step P801A is executed to obtain the current table border Canvas1; step P802A is executed to first judge the sub-region R10, and it can be detected that there is an edge in the sub-region R10 (the edge on its left side) that coincides with an edge of the current table border Canvas1 (the edge on its left side), that is, an edge of the current table border Canvas1 is also an edge in the sub-region R10, so the concave-convexity of the sub-region R10 can be further judged, and it can be detected that the shape of the sub-region R10 is a rectangle (convex figure), so that it can be determined that the sub-region R10 belongs to a valid cell in this table area; refer to Fig.16 After deleting sub-area R10 from the table area corresponding to the current table border Canvas1, sub-areas R11, R12, R13 and R14 remain, forming a new table border CanvaP5, thereby completing the update of the table border and executing the first processing process of the second round.

[0278] In the second round of the first processing, execute step P801A to obtain the current table border CanvaP5; execute step P802A to first judge sub-region R11. No edge of sub-region R11 overlaps with the edge in the table border CanvaP5, so sub-region R11 is not identified as a valid cell, and then judge sub-region R12. Based on the same principle, sub-regions R12, R13 and R14 do not have any edge that overlaps with the edge in the table border CanvaP5, so sub-regions R12, R13 and R14 are not identified as valid cells. Since the second round of the first processing did not identify any valid cells, the execution of the first processing is terminated, that is, the second round of the first processing is ended, and the third round of the first processing and other subsequent first processing processes are no longer executed.

[0279] Therefore, for Fig.16After executing two rounds of the first processing process, the table area shown has traversed all sub-areas, and only the sub-area R10 is identified as a valid cell. For sub-areas R11, R12, R13 and R14, since they are not identified as valid cells after all the rounds of the first processing process, sub-areas R11, R12, R13 and R14 can be identified as invalid cells.

[0280] By executing steps P801A-P803A, sub-regions that meet the conditions can be screened out according to the characteristics of the shape of the valid cells and the positional relationship between the valid cells and the table area, so that the valid cells can be efficiently identified.

[0281] In this embodiment, when executing step P8, that is, screening each sub-region according to its shape and the positional relationship with the table area, and identifying the screened sub-region as a valid cell, multiple rounds of the second processing process can be executed in sequence. Except for the first round of the second processing process, each round of the second processing process is based on the execution result of the previous round of the second processing process, and continues to perform corresponding processing until no non-convex sub-region is detected from the table area in a certain round of the second processing process, and the execution of all the second processing processes is terminated, that is, the subsequent second processing processes are no longer executed.

[0282] In this embodiment, refer to Fig.17 , any round of the second processing process includes the following steps:

[0283] P801B. When the second processing process of this round is the second processing process of the first round, detect the convexity of the shape of each sub-region in the table region, obtain the non-convex edge line from the detected non-convex sub-region, and when the second processing process of this round is not the second processing process of the first round, obtain the non-convex edge line updated by the second processing process of the previous round;

[0284] P802B. Merge the sub-regions connected by the obtained non-convex edge lines to obtain a merged sub-region, delete the non-convex edge lines inside the merged sub-region, and detect the concavity and convexity of the shape of the merged sub-region;

[0285] P803B. When the shape of the merged sub-region is a convex figure, the merged sub-region is identified as a valid cell. When the shape of the merged sub-region is not a convex figure, the non-convex edge line of the edge of the merged sub-region is obtained as the non-convex edge line after update in the second processing process of this round.

[0286] Fig.17 An example of steps P801B-P803B shown is as follows Fig.18 shown.

[0287] Fig.18Part (a) is a table area of ​​the document to be recognized, and this table area is divided into sub-areas such as R11, R12, R13 and R14 by various table line shape elements.

[0288] When executing the second processing of the first round, step P801B is used to detect the convexity of sub-regions R11, R12, R13 and R14, where sub-regions R11, R13 and R14 are all rectangular (convex shapes), while sub-region R12 is an irregular shape, and sub-region R12 is a non-convex shape. For a non-convex shape, if one of the edges is extended into a straight line, the other edges cannot be on the same side of the straight line, then the edge in the non-convex shape is called a non-convex edge line. Therefore, Fig.18 In part (a), the boundary between the non-convex sub-region R12 and the sub-region R11 and the boundary between the non-convex sub-region R12 and the sub-region R13 are both non-convex edge lines of the non-convex sub-region R12.

[0289] When executing the second processing of the first round, execute step P802B, for the non-convex edge line "the boundary between the non-convex sub-region R12 and the sub-region R11", merge the two sub-regions R11 and R12 connected by this non-convex edge line to obtain a merged sub-region (R11+R12), delete the non-convex edge line inside the merged sub-region (R11+R12), and form Fig.18 The concavity and convexity of the shape of the combined sub-region (R11+R12) are detected.

[0290] Reference Fig.18 In part (b) of FIG. 1 , the merged sub-region (R11+R12) is a non-convex figure. Therefore, in step P803B of the first round of the second processing, the merged sub-region (R11+R12) is not identified as a valid cell. Instead, the non-convex edge line of the edge of the merged sub-region (R11+R12), that is, the boundary line between the merged sub-region (R11+R12) and the sub-region R13, is obtained as the non-convex edge line after the update of the second processing of this round. The first round of the second processing is completed.

[0291] Since no valid cells are identified after executing steps P801B-P803B in the first round of the second processing, the next round, i.e., the second round of the second processing, is executed.

[0292] In the second round of the second processing, the updated non-convex edge line obtained in step P801B is Fig.18Execute step P802B to merge the subregions connected by the non-convex edge lines [i.e., the merged subregion (R11+R12) and subregion R13] to obtain the merged subregion (R11+R12+R13), delete the non-convex edge lines inside the merged subregion (R11+R12+R13), and form Fig.18 For each sub-region shown in part (c), the concavity and convexity of the shape of the combined sub-region (R11+R12+R13) are detected.

[0293] Reference Fig.18 In part (c) of the figure, the merged sub-region (R11+R12+R13) is a non-convex figure. Therefore, in step P803B of the second round of the second processing, the merged sub-region (R11+R12+R13) is not identified as a valid cell. Instead, the non-convex edge line of the edge of the merged sub-region (R11+R12+R13), that is, the boundary line between the merged sub-region (R11+R12+R13) and the sub-region R14, is obtained as the non-convex edge line after the update of the second round of the second processing. The second round of the second processing is completed.

[0294] Since no valid cells are identified after executing steps P801B-P803B in the second round of the second processing process, the next round, i.e., the third round of the second processing process, is executed.

[0295] In the second processing of the third round, the updated non-convex edge line obtained in step P801B is Fig.18 Execute step P802B to merge the subregions connected by the non-convex edge lines [i.e., the merged subregion (R11+R12+R13) and subregion R14] to obtain the merged subregion (R11+R12+R13+R14), delete the non-convex edge lines inside the merged subregion (R11+R12+R13+R14), and form Fig.18 For each sub-region shown in part (d) of FIG. 1 , the concavity and convexity of the shape of the combined sub-region (R11+R12+R13+R14) are detected.

[0296] Reference Fig.18 In part (d) of FIG. 5 , the merged sub-region (R11+R12+R13+R14) is a convex figure, so in step P803B of the second processing process of the third round, the merged sub-region (R11+R12+R13+R14) is identified as a valid cell.

[0297] In this embodiment, after executing three rounds of the second processing process, the merged sub-region (R11+R12+R13+R14) is identified as a valid cell, that is, Fig.18 Each sub-area in the table area shown is identified as a valid cell, so there are no invalid cells in the table area.

[0298] Reference Fig.18 For non-convex sub-regions in the table area, after executing several rounds of the second processing process to merge the sub-regions, there will eventually be no non-convex sub-regions in the table area, so that a new second processing process cannot be executed, or there may not be non-convex sub-regions in the table area at the beginning, so a new round of the second processing process cannot be executed. Therefore, the end condition of all the second processing processes can be set to "no non-convex sub-regions are detected from the table area". That is, if the situation of "no non-convex sub-regions are detected from the table area" occurs, then a new round of the second processing process will not be executed, thereby terminating the execution of all the second processing processes.

[0299] If there are still sub-regions that are not identified as valid cells after all rounds of the second processing process are executed, then these sub-regions that are not identified as valid cells can be identified as invalid cells.

[0300] In this embodiment, after P802B is executed in any round of the second processing, the table line shape element corresponding to the deleted non-convex edge line can be identified as the table content. Fig.18 In part (d), the dotted lines indicate the locations of the non-convex edge lines deleted in the second processing process. These deleted non-convex edge lines can be identified as the table contents in the valid cell (R11+R12+R13+R14).

[0301] By executing steps P801B-P803B, sub-areas that meet the conditions can be screened out according to the characteristics of the shape of the valid cells and the positional relationship between the valid cells and the table area, so that the valid cells can be identified efficiently.

[0302] In this embodiment, when executing step P8, only steps P801A-P803A or only steps P801B-P803B may be executed, or multiple rounds of the first processing process P801A-P803A may be executed first, and then multiple rounds of the second processing process P801B-P803B may be executed for each sub-area that is not identified as a valid cell, thereby helping to improve the ability to identify valid cells.

[0303] In this embodiment, Fig.19The method for identifying valid cells in a table execution document of a complex form shown in the figure can be performed by executing the first processing process P801A-P803A, firstly identifying the sub-area R21 as one of the valid cells through step P802A, deleting the valid cell R21 from the table area, and then judging the remaining sub-areas (R15+R16+R17+R18+R19+R20), and executing the first processing process P801A-P803A can no longer identify the valid cells therein. However, executing the second processing process P801B-P803B on the remaining sub-areas (R15+R16+R17+R18+R19+R20) can detect that R15, R16, R17, R18, R19 and R20 are valid cells respectively.

[0304] In this embodiment, Fig. 20 The method for identifying valid cells in a table in an embedded table execution document shown can be performed by executing the second processing procedure P801B-P803B, identifying the non-convex sub-region R22 therein through step P801B, and determining that the non-convex edge line is the boundary between the non-convex sub-region R22 and the sub-regions R23, R24, R25, and R26, and then executing step P802B to merge the sub-regions R22, R23, R24, R25, and R26 connected by the non-convex edge line to obtain a merged sub-region (R22+R23+R24+R25+R26), and the merged sub-region (R22+R23+R24+R25+R26) is a rectangle (convex figure), so that the merged sub-region (R22+R23+R24+R25+R26) is identified as a valid cell. Based on the same principle, the second processing procedure P801B-P803B can be executed to Fig.19 The merged sub-region (R70+R28+R29+R30+R31) in the table is identified as a valid cell. Therefore, the table valid cell identification method in this embodiment can accurately identify the valid cells in the embedded table.

[0305] In this embodiment, Fig.21The method for identifying valid cells of a table in a document with a table having non-aligned lines shown in the figure can be performed by executing the second processing process P801B-P803B, identifying the non-convex sub-region R32 and the non-convex edge line L32 therein through step P801B, merging the sub-region R32 connected by the non-convex edge line through step P802B, that is, merging the sub-region R32 with itself, and the obtained merged sub-region is the sub-region R32 itself, and after deleting the non-convex edge line L32 inside the merged sub-region R32, detecting the convexity of the shape of the merged sub-region R32, and the merged sub-region R32 after deleting the non-convex edge line L32 is a rectangle (convex figure), so that the merged sub-region R32 is identified as a valid cell, and on this basis, the non-convex edge line L32 can be identified as the table content in this valid cell, rather than as a table line.

[0306] comprehensive Fig.19 , Fig. 20 and Fig.21 As shown in the example, the table valid cells in this embodiment can identify valid cells in table areas of various forms, such as complex tables, embedded tables, and tables with non-aligned lines, and has a strong valid cell recognition capability.

[0307] When executing step S1, in addition to executing the first process (steps P1-P4) or the second process (steps P5-P8) of identifying valid cells from the document to be identified, valid cells can also be identified in other ways. For example, specific cells such as cells with editable functions enabled in the document to be identified, cells with editable functions disabled, or cells with an area greater than / less than an area threshold (the area threshold can be a fixed value, or determined by multiplying the page area of ​​the document to be identified by a coefficient) can be determined as valid cells in step S1. Among them, by identifying valid cells based on the opening and closing of the editable function, editable / non-editable cells can be screened out to identify the interactive area, thereby facilitating batch maintenance or change of the interactivity of the cells; by identifying valid cells based on the area, valid cells can be screened out based on the amount of interactive information in the interactive area, thereby improving information processing efficiency.

[0308] After executing steps P1-P4 or steps P5-P8 or other processes for identifying valid cells, valid cells in the document to be identified have been identified, and steps S2 and S3 may be continued.

[0309] In steps S2-S3, unless otherwise specified, when a "valid cell" is mentioned, it refers to a specific valid cell, and when an "interactive element" is mentioned, it refers to a specific interactive element.

[0310] In step S2, the interactive element is used as the search target to search for the area in the document to be identified where the valid cell is located. The interactive element refers to a document element in the document to be identified, which has an interactive attribute indication function, that is, the interactive element is a document element that can indicate that the corresponding area has or may have an interactive attribute, thereby triggering the execution of step S3 to further confirm whether the area indicated by it does have an interactive attribute. The area with interactive attributes is an interactive area, which can be used by users to perform interactive operations such as text input, selection box checking, and drop-down forms.

[0311] Specifically, the interactive attribute indication function of the interactive element may be specified by the standard technical specifications of the document to be identified. For example, the standard technical specifications may specify that document elements of certain specific content are used to indicate that certain areas have interactive attributes, so that the document elements of the specific content are interactive elements. The interactive attribute indication function of the interactive element may also be specified when executing steps S1-S3, for example, directly determining that document elements of certain specific content are used to indicate that certain areas have interactive attributes, so that the document elements of the specific content are interactive elements.

[0312] In this embodiment, based on the fact that the interactive operations on the table include specific operations such as "input content" and "selection", the document element that may be used to indicate that some areas have the interactive attribute of "input content" is determined as a first type of interactive element. The first type of interactive element may specifically be a text type or path type document element with the content of "underline (whose display effect is ______)"; the document element that may be used to indicate that some areas have the interactive attribute of "selection" is determined as a second type of interactive element. The second type of interactive element may specifically be a document element with the content of "□" (Unicode is 0x25A1), (Unicode is 0x2611), "○" (Unicode is 0x25CB), (Unicode is 0x25C9) and other text-type document elements.

[0313] When executing step S2, the area where the valid cell itself is located in the document to be identified, such as the part framed by the bounding box corresponding to the valid cell, that is, the cell bounding box CBox, can be searched, or a certain range can be extended on the basis of the cell bounding box CBox to search this range. For example, if the height CBox.height of the cell bounding box CBox is expanded to CBox.height×k1, and the width CBox.width of the cell bounding box CBox is expanded to CBox.width×k2, where k1 and k2 are coefficients greater than 1 (for example, k1=k2=1.1), then the extra range of the expanded cell bounding box CBox relative to the cell bounding box CBox before expansion is the extended range. That is, in the embodiment provided by the present invention, the extended certain range is the extra range of the expanded cell bounding box formed after the height of the cell bounding box CBox is expanded by k1 and the width of the cell bounding box CBox is expanded by k2 on the basis of the cell bounding box CBox, relative to the cell bounding box CBox before expansion.

[0314] In this embodiment, since step S2 uses interactive elements as search targets, the search results include:

[0315] Result 1: Search for interactive elements from the cell bounding box CBox (and its extension);

[0316] Result 2: No interactive elements were found.

[0317] In one embodiment provided by the present invention, step S3, i.e., the step of determining the interactive area information corresponding to the valid cell according to the search result, specifically includes the following steps:

[0318] S301. When an interactive element is found, perform type detection on the interactive element;

[0319] S302. Determine the validity of the interactive element according to the result of the type detection;

[0320] S303. When the interactive element is valid, determine the interactive area information according to the interactive element;

[0321] S304. When no document element is found, the interactive element is invalid, or the first text element is not found, surrounding cells of the valid cell are searched;

[0322] S305. When the surrounding cells are found, determine the interactive area information according to the surrounding cells.

[0323] If the search result of executing step S2 is Result 1, then the process can jump to executing step S301; in step S301, the type of the detected interactive element is further determined. Executing step S301 can further subdivide the search result Result 1 into:

[0324] Result 1-1: The first type of interactive element (underline) is searched from the cell bounding box CBox (and its extension);

[0325] Result 1-2: Search the second type of interactive elements (□, wait).

[0326] For the search result Result 1-1, that is, when the first type of interactive element (underline) is found, it indicates that the first type of interactive element (underline) indicates that there may be an interactive area in this valid cell, and step S302 is further performed to determine.

[0327] In this embodiment, step S302, that is, the step of determining the validity of the interactive element according to the result of the type detection, includes:

[0328] S30201. When it is detected that the interactive element belongs to the first type of interactive element, searching for the first text element in the valid cell;

[0329] S30202. When the first text element is found from the valid cell, the modification relationship of the interactive element to the first text element is detected, and the validity of the interactive element is determined according to the modification relationship;

[0330] S30203. When it is detected that the interactive element belongs to the second type of interactive element, obtain an interactive bounding box;

[0331] S30204. Determine the interaction bounding box as the interaction area in the valid cell.

[0332] Specifically, if the result of executing step S301 is Result 1-1, then the process will jump to execute step S30201, and continue to search for the first text element, i.e., the document element of the text type, from the valid cell, and determine the validity of the first type of interactive element (underline) with the help of the information contained in the first text element in the valid cell. Executing step S30201 can further subdivide the search result Result 1-1 into:

[0333] Result 1-1-1: The first type of interactive element (underline) and the first text element are searched from the cell bounding box CBox (and its extension);

[0334] Result 1-1-2: The first type of interactive element (underline) is searched from the cell bounding box CBox (and its extended range) but the first text element is not searched.

[0335] If the search result of executing step S30201 is Result 1-1-1, then step S30202 may be further executed to determine whether the first type of interactive element (underline) is valid. Specifically, the modification relationship of the interactive element to the first text element may be detected, and whether the first type of interactive element (underline) is valid may be determined based on the modification relationship. In this embodiment, the modification relationship of the first type of interactive element to the first text element indicates whether the first type of interactive element has a modifying effect on the text content expression of the first text element. The modifying effect refers to the influence of the first type of interactive element (underline), as a symbol that itself has no substantial semantics, on the semantic expression of the first text element that itself has substantial semantics.

[0336] For example, when the first type of interactive element (underline) is located directly below the first text element, it can enable the reader to understand that "the text content of the first text element is emphasized" or "the first text element is the content filled in later". Therefore, the first type of interactive element (underline) has a modifying effect on the first text element. At this time, the modification relationship of the interactive element to the first text element is "the interactive element modifies the first text element"; when the first type of interactive element (underline) and the first text element do not overlap or intersect in the vertical projection, the reader's understanding of the text content of the first text element is not affected by the first type of interactive element (underline). Therefore, the first type of interactive element (underline) does not have a modifying effect on the first text element. At this time, the modification relationship of the interactive element to the first text element is "the interactive element does not modify the first text element".

[0337] Specifically, refer to Fig. 22 , for parts (a), (b), (c), etc., the contents of the first text elements are "A", "B", "C", etc., respectively. The bounding box of any first text element is TBox, and the bounding box of the first type of interactive element (underline), i.e., the interactive bounding box UBox, satisfies the following with TBox:

[0338] TBox.left>UBox.right||TBox.right <UBox.left

[0339] Where TBox.left is the left boundary of the bounding box TBox of any first text element, TBox.right is the right boundary of TBox, UBox.left is the left boundary of the interaction bounding box UBox, and UBox.right is the right boundary of the interaction bounding box UBox. Here, "||" represents "or". The first type of interaction element (underlined) that meets this determination condition and the first text elements "A", "B", "C", etc. are as shown in Fig. 22 (a), (b), (c) in. It can be seen that the first type of interaction element (underlined) does not modify the first text element, that is, "there is no underline under the first text elements A, B, C, etc.", so the next judgment can be continued.

[0340] On the contrary, Fig. 22 In parts (d), (e), (f), etc. in, it does not meet the determination condition of TBox.left > UBox.right || TBox.right < UBox.left. Therefore, Fig. 22 In parts (d), (e), (f), etc. in, the first type of interaction element (underlined) modifies the first text element, that is, "there is an underline under the first text elements A, B, C, etc.", so it can be directly determined that the first type of interaction element (underlined) is invalid.

[0341] When meeting the determination condition of TBox.left > UBox.right || TBox.right < UBox.left, that is Fig. 22 In cases where (a), (b), (c), etc. in hold, it can be determined that the first type of interaction element (underlined) may be valid, but it is necessary to further determine whether the first type of interaction element (underlined) is valid according to the specific position of the first type of interaction element (underlined).

[0342] Specifically, referring to Fig.23 , for a valid cell, its corresponding bounding box, that is, the cell bounding box CBox, can be obtained; for all the first text elements included in the valid cell, a bounding box can be obtained, and this bounding box is the smallest bounding box that can enclose all the first text elements, and this bounding box is the union bounding box UnionTBox; while the bounding box corresponding to the first type of interaction element (underlined) is the interaction bounding box UBox.

[0343] In this embodiment, referring to Fig.23, get the height UnionTBox.height of the union bounding box UnionTBox and the height CBox.height of the cell bounding box CBox, and calculate the height ratio formed by these two heights, that is, the ratio UnionTBox.height / CBox.height, as the relative size of the union bounding box UnionTBox and the cell bounding box CBox.

[0344] In this embodiment, refer to Fig.23 , a height ratio threshold can be set (its size can be set to 0.66), and the relative size of the union bounding box UnionTBox and the cell bounding box CBox can be determined by this height ratio threshold.

[0345] A. If UnionTBox.height / CBox.height>height ratio threshold (0.66), then based on the top UnionTBox.top and the bottom UnionTBox.bottom of the union bounding box UnionTBox, they are respectively adjusted up and down by a certain amount as the allowable range. For example, the lower limit of the allowed range is set to the middle position between the bottom UnionTBox.bottom of the union bounding box UnionTBox and the bottom CBox.bottom of the cell bounding box CBox, that is, (UnionTBox.bottom+CBox.bottom) / 2; the upper limit of the allowed range is set to the position from the bottom UnionTBox.bottom of the union bounding box UnionTBox, and the height is UnionTBox.height×the first coefficient (specifically, it can be 1.2), that is, UnionTBox.bottom+UnionTBox.height×the first coefficient (1.2), so that the allowed range is set to ((UnionTBox.bottom+CBox.bottom) / 2, UnionTBox.bottom+UnionTBox.height×1.2).

[0346] B. If UnionTBox.height / CBox.height≤height ratio threshold (0.66), then based on the bottom CBox.bottom of the cell boundary box CBox, adjust it upward by a certain amount as the allowable range. For example, the lower limit of the allowable range is set to the position of CBox.height×the second coefficient (specifically 0.33) from the bottom CBox.bottom of the cell boundary box CBox, that is, CBox.bottom+CBox.height×the second coefficient (0.33); the upper limit of the allowable range can also be UnionTBox.bottom+UnionTBox.height×the first coefficient (1.2), so the allowable range is set to (CBox.bottom+CBox.height×0.33, UnionTBox.bottom+UnionTBox.height×1.2).

[0347] In accordance with Fig.23 After setting the allowed range in the manner shown, it is possible to determine whether the first type of interactive element (underline) is within the allowed range. Specifically, the top coordinates UBox.top and bottom coordinates UBox.bottom of the bounding box corresponding to the first type of interactive element (underline), that is, the interactive bounding box UBox, can be obtained, and the middle center coordinates UBox.yCenter in the height direction of the interactive bounding box UBox can be calculated, that is,

[0348] UBox.yCenter=(UBox.top+UBox.bottom) / 2

[0349] Then determine whether UBox.yCenter is within the allowed range. Specifically, if it is within the allowed range set by method A (such as Fig.23 As shown in the upper part of ), that is, ((UnionTBox.bottom+CBox.bottom) / 2, UnionTBox.bottom+UnionTBox.height×1.2), it is to judge whether (UnionTBox.bottom+CBox.bottom) / 2<UBox.yCenter<UnionTBox.bottom+UnionTBox.height×1.2 is established. If so, it is determined that the first type of interactive element (underline) is valid, otherwise it is determined that the first type of interactive element (underline) is invalid;

[0350] If the allowed range is set by method B (such as Fig.23As shown in the lower part of , that is, ((UnionTBox.bottom+CBox.bottom) / 2, UnionTBox.bottom+UnionTBox.height×1.2), it is to judge whether CBox.bottom+CBox.height×0.33<UBox.yCenter<UnionTBox.bottom+UnionTBox.height×1.2 is established. If so, the first type of interactive element (underline) is determined to be valid, otherwise the first type of interactive element (underline) is determined to be invalid.

[0351] In this embodiment, Fig.23 The principle shown is that when the ratio of UnionTBox.height / CBox.height is relatively large, that is, when the union bounding box UnionTBox is larger than the cell bounding box CBox, it indicates that the first text element in the valid cell may be a text element with substantial content such as Chinese characters and letters, and there is a high possibility that there is an interactive area in the valid cell. Fig.23 As shown in the upper part of , at this time, according to method A, a wider allowable range can be set, so that the first type of interactive element (underline) has a greater possibility of being valid, which is conducive to detecting a larger range of interactive areas; on the contrary, when the ratio of UnionTBox.height / CBox.height is relatively small, that is, the union bounding box UnionTBox is smaller than the cell bounding box CBox, it indicates that the first text element in the valid cell may be a symbolic text element with no substantial content, such as a period, superscript or subscript symbol, and the possibility of the existence of an interactive area in the valid cell is small. Fig.23 As shown in the lower part of , at this time, a narrower allowed range can be set according to method A, so that the first type of interactive element (underline) is more likely to be invalid, thereby saving the continued detection of this valid cell and improving the detection efficiency of the interactive area.

[0352] pass Fig. 22 and Fig.23 The operation shown can determine whether the first type of interactive element (underline) is valid, so executing step S30202 can further subdivide the search result Result 1-1-1 into:

[0353] Result 1-1-1-1: It is determined that the first type of interactive element (underline) is valid according to the first text element;

[0354] Result 1-1-1-2: It is determined that the first type of interactive element (underline) is invalid according to the first text element.

[0355] If the search result is Result 1-1-1-1, then the process can jump to step S303. When executing step S303, if Fig.24 As shown, the interactive area information, such as the upper boundary, lower boundary, left boundary and right boundary of the interactive area, is determined according to the valid first type interactive element (underline).

[0356] Reference Fig.24 Part (a) includes three valid first-type interactive elements, namely underline 1, underline 2 and underline 3.

[0357] In this embodiment, according to the data of the document to be identified, the line height curLine.Height_1 of the line where underline 1 is located in the document to be identified, the line height curLine.Height_2 of the line where underline 2 is located in the document to be identified, and the line height curLine.Height_3 of the line where underline 3 is located in the document to be identified can be obtained respectively, and their sizes can be determined respectively.

[0358] In this embodiment, a first height threshold may be set to measure the heights of the rows where underline 1, underline 2, and underline 3 are located, namely, curLine.Height_1, curLine.Height_2, and curLine.Height_3. In this embodiment, the average height of the rows where all valid first-type interactive elements (underline) are located in the valid cells may be calculated, for example Fig.24 The average value of curLine.Height_1, curLine.Height_2 and curLine.Height_3 in part (a) is used to obtain the line height average value AvgLineHeight = (curLine.Height_1 + curLine.Height_2 + curLine.Height_3) / 3, and the first height threshold is determined based on the line height average value AvgLineHeight.

[0359] Specifically, a fixed value std(LineHeight) may be subtracted from the average line height value AvgLineHeight to obtain the first height threshold, that is, the first height threshold in this embodiment is AvgLineHeight-std(LineHeight).

[0360] After obtaining the first height threshold, Fig.24 As shown in part (b):

[0361] For underline 1, the height curLine.Height_1 of the row where it is located is greater than the first height threshold AvgLineHeight, that is, curLine.Height_1>AvgLineHeight-std(LineHeight), so it can be determined that the row height of underline 1 is normal. Then, for interactive area 1 corresponding to underline 1, the height IntersectBox.Height_1 of interactive area 1 can be set to be the same as the row height curLine.Height_1 of the row where it is located, that is, IntersectBox.Height_1=curLine.Height_1; at this time, the lower bound IntersectBox.bottom_1 of the boundary box of interactive area 1 is the same as the lower bound curLine.bottom_1 of the row where underline 1 is located, and the upper bound IntersectBox.top_1 of the boundary box of interactive area 1 is the same as the upper bound curLine.top_1 of the row where underline 1 is located, that is, IntersectBox.bottom_1=curLine.bottom_1, IntersectBox.top_1=curLine.top_1;

[0362] For underline 2, the height curLine.Height_2 of the row where it is located is less than the first height threshold AvgLineHeight-std(LineHeight), that is, curLine.Height_2<AvgLineHeight-std(LineHeight), so it can be determined that the row height of underline 2 is too low. Then, for interactive area 2 corresponding to underline 2, the height IntersectBox.Height_2 of interactive area 2 can be set to the average row height AvgLineHeight, that is, IntersectBox.Height_2=AvgLineHeight; at this time, the lower bound IntersectBox.bottom_2 of the boundary box of interactive area 2 is the same as the lower bound curLine.bottom_2 of the row where underline 2 is located, that is, IntersectBox.bottom_2=curLine.bottom_2, and the upper bound IntersectBox.top_2 of the boundary box of interactive area 2 satisfies IntersectBox.top_2=curLine.bottom_2+AvgLineHeight;

[0363] For the underline 3, the height curLine.Height_3 of the row where the underline 3 is located is greater than the first height threshold AvgLineHeight, that is, curLine.Height_3>AvgLineHeight-std(LineHeight), so it can be determined that the row height of the underline 3 is normal. Then, for the interactive area 3 corresponding to the underline 3, the height IntersectBox.Height_3 of the interactive area 3 can be set to be the same as the row height curLine.Height_3 of the row where the underline 3 is located, that is, IntersectBox.Height_3=curLine.Height_3; at this time, the lower bound IntersectBox.bottom_3 of the boundary box of the interactive area 3 is the same as the lower bound curLine.bottom_3 of the row where the underline 3 is located, and the upper bound IntersectBox.top_3 of the boundary box of the interactive area 3 is the same as the upper bound curLine.top_3 of the row where the underline 3 is located, that is, IntersectBox.bottom_3=curLine.bottom_3, and IntersectBox.top_2=curLine.top_2.

[0364] In this embodiment, for underline 2, the height of its corresponding interactive area 2 is set to AvgLineHeight, which is greater than the height of the line where underline 2 is located, curLineHeight_2. Since underline 2 is adjacent to underline 1, the upper boundary of the boundary box of interactive area 2, IntersectBox.top_2, may enter the line where underline 1 is located, that is, IntersectBox.top_2>curLine.bottom_1. By limiting IntersectBox.top_2 to be equal to curLine.bottom_1, it is possible to avoid the intersection of interactive area 2 and interactive area 1, so that interactive area 2 and interactive area 1 are two independent interactive areas.

[0365] In this embodiment, the bounding box width of each interactive area can be set to be the same as the bounding box width of the corresponding first type interactive element (underline). Fig.24In part (b), the left boundary of the interaction area 1 bounding box IntersectBox.left_1 is the same as the left boundary of the underline 1 bounding box curLine.left_1, and the right boundary of the interaction area 1 bounding box IntersectBox.right_1 is the same as the right boundary of the underline 1 bounding box curLine.right_1, that is, IntersectBox.left_1 = curLine.left_1, IntersectBox.right_1 = curLine.right_1.

[0366] This was implemented, according to Fig.24 The principle of determining the interactive area in the manner shown is: setting the left and right boundaries of the corresponding interactive area according to the left and right boundaries of the boundary box of the valid first-type interactive element (underline), so that the size of the interactive area can be set to be equivalent to the range of the first-type interactive element (underline); and when the height of a row where a first-type interactive element (underline) is located is smaller than the first height threshold, it indicates that the row where the first-type interactive element (underline) is located may only contain first text elements with flattened content such as punctuation marks, and the height of the interactive area corresponding to the first-type interactive element (underline) is adjusted to the average row height, so that the height of the interactive area can be adapted to all first text elements (including text and symbols, etc.) in the valid cell, thereby avoiding setting the interactive area too small, thereby enabling various text element interactive operations to be performed on the interactive area, and improving the usability of the identified interactive area.

[0367] In this embodiment, when the search result is Result 2 (no interactive element is found when executing step S2), Result 1-1-2 [the first type of interactive element (underline) is found when executing step S301, but the first text element is not found when executing step S30201], or Result 1-1-1-2 [the first text element is found when executing step S30201, but the first type of interactive element (underline) is determined to be invalid based on the first text element when executing step S30202], it indicates that the valid cell itself does not have an interactive element or the first text element to provide information, thereby identifying the interactive area. In this case, when executing step S3, step S303 is skipped and step S304 is executed to find relevant information from the surrounding cells of the valid cell in the same table area to identify the interactive area in the valid cell.

[0368] In step S304, specifically, refer to Fig.25In part (a), for a valid cell, cells with the same row index information in the same table area can be searched according to the row index information of the valid cell, and the searched cells are used as peripheral cells. By using the row index information, the searched peripheral cells are located in the same row as the valid cell.

[0369] For example, Fig.25 As shown in part (a) of FIG. 1 , multiple surrounding cells such as surrounding cell 1, surrounding cell 2, surrounding cell 3, surrounding cell 4, surrounding cell 5 and surrounding cell 6 can be searched. Among them, only the row height of the combination of surrounding cell 3 and surrounding cell 4 is the same as the row height of the valid cell. In this embodiment, the surrounding cells or their combinations with the same row height are preferably selected, that is, using Fig.25 The surrounding cells 3 and 4 shown in part (a) of the figure; if no second text element is found from the surrounding cells with the same row height, surrounding cells 1 and 2 with different row heights are used.

[0370] If step S304 is executed as Fig.25 As shown in part (a) of FIG. 1 , the surrounding cells are searched out, and then step S305 is executed. In step S305, refer to Fig.25 In part (b), suppose that multiple second text elements with contents such as "PQ", "RS" and "TU" are searched from surrounding cells 3 and 4, and the line heights of the lines where their bounding boxes are located are Line_height_1, Line_height_2 and Line_height_3 respectively, then their line height average Average = (Line_height_1+Line_height_2+Line_height_3) / 3 can be calculated as the height of the interactive area in the valid cell. Specifically, refer to Fig.25 In part (b) of the flowchart, execute step S3 to identify the interactive area from the valid cell, whose left boundary is the same as the left boundary of the bounding box of the first type of interactive element in the valid cell, the right boundary of the interactive area is the same as the right boundary of the bounding box of the first type of interactive element, the lower boundary of the interactive area is the same as the lower boundary of the bounding box of the first type of interactive element, and the height of the interactive area is the average row height Average. After the lower boundary and height of the interactive area are determined, the upper boundary of the interactive area is naturally determined, thereby identifying the interactive area in the valid cell.

[0371] This was implemented, according to Fig.25The principle of determining the interactive area in the manner shown is that: when the valid cell itself lacks information provided by the first text element, surrounding cells with the second text element are searched from the surrounding cells of the valid cell to obtain information for determining the height of the interactive area; since the second text element is a text element existing in the table area itself, the height of the interactive area is determined according to the average value of the heights of the rows where the second text elements are located, so that the height of the interactive area can be adapted to all the second text elements (including text and symbols, etc.) existing in the table area where the valid cell is located, thereby setting a suitable interactive area, being able to perform various text element interactive operations on the interactive area, and improving the usability of the identified interactive area.

[0372] In step S305, if Fig.25 In part (a), if no second text element exists in any of the surrounding cells, that is, no surrounding cell can provide height information of the interaction area through the second text element, then the interaction area can be determined based on the cell bounding box CBox of the valid cell itself.

[0373] Specifically, refer to Fig.26 , get the cell bounding box CBox of the valid cell, you can get the height CBox.height and width CBox.width of the cell bounding box CBox, as well as the upper bound CBox.top, lower bound CBox.bottom, left bound CBox.left and right bound CBox.right of the cell bounding box CBox. Fig.26 , based on the upper boundary CBox.top of the cell boundary box, adjust CBox.height down by the third coefficient (specifically 0.1) to obtain the upper boundary IntersectBox.top of the interaction area; based on the lower boundary CBox.bottom of the cell boundary box, adjust CBox.height up by the fourth coefficient (specifically 0.1) to obtain the lower boundary IntersectBox.bottom of the interaction area; based on the left boundary CBox.left of the cell boundary box, adjust CBox.width to the right by the fifth coefficient (specifically 0.1) to obtain the left boundary IntersectBox.left of the interaction area; based on the right boundary CBox.right of the cell boundary box, adjust CBox.width to the left by the sixth coefficient (specifically 0.1) to obtain the right boundary IntersectBox.right of the interaction area; therefore, the following conditions are satisfied:

[0374] IntersectBox.top = CBox.top - CBox.height × third coefficient (0.1);

[0375] IntersectBox.bottom = CBox.bottom + CBox.height × fourth coefficient (0.1);

[0376] IntersectBox.left = CBox.left + CBox.width × fourth coefficient (0.1);

[0377] IntersectBox.right=CBox.right-CBox.width×fifth coefficient (0.1).

[0378] According to the upper bound IntersectBox.top, lower bound IntersectBox.bottom, left bound IntersectBox.left and right bound IntersectBox.right of the interactive area, the interactive area in the valid cell can be determined.

[0379] In this embodiment, if the search result of executing step S2 is Result 1-2 (searching from the cell boundary box CBox to □, ○ and If a second type of interactive element is present, the interactive bounding box of the second type of interactive element can be directly obtained, and the position corresponding to the interactive bounding box is determined as the interactive area in the valid cell, that is, the bounding box of the interactive area is the same as the interactive bounding box of the second type of interactive element.

[0380] In this embodiment, after detecting the interactive area in the valid cell by executing steps S1-S3, the following steps may also be executed:

[0381] S4. Generate an interactive function description code based on the interactive area information;

[0382] S5. Add the interactive function description code to the document to be identified.

[0383] In step S4, the interactive area information includes the position information of the interactive area (such as the upper boundary, lower boundary, left boundary and right boundary of the boundary box of the interactive area) and the functional information, wherein the functional information indicates the interactive function of the interactive area. For example, in this embodiment, if the interactive area is identified by the first type of interactive element (such as an underline), the corresponding interactive function may be "text input" or the like; if the interactive area is identified by the content "□", If the interactive area is identified by the second type of interactive element with the content of "○", If the second type of interactive element is identified, the corresponding interactive function can be a "button" etc.

[0384] In step S5, if the document to be identified is a layout document in a format such as pdf, it can be added in an incremental update manner. Specifically, the interactive function description code can be directly added based on the original code of the document to be identified.

[0385] In this embodiment, steps S1-S5 can be executed when the document to be identified has been opened by the reader, so that the interactive area information in the document to be identified can be detected online, and the interactive area can be visualized and interactive operations can be performed; steps S1-S5 can also be executed when the document to be identified has not been opened by the reader, so that the documents to be identified whose interactive areas have not been identified can be identified in batches, thereby improving the recognition efficiency of the interactive area.

[0386] In this embodiment, steps S1-S5 can be executed by the original editor of the document to be identified, by an intermediate editor of the document to be identified, or by a reader of the document to be identified.

[0387] By executing steps S1-S3 or S1-S5, the unmarked area with interactive attributes in the document to be identified can be automatically identified as the interactive area, thereby realizing the automatic identification and addition of the interactive area, and being able to efficiently generate or restore the interactive area information when the editor does not edit the interactive area or the interactive area information is lost, thereby improving the convenience of using the electronic document. At the same time, it prevents the area from being too large, so that the text elements that should not be modified and edited also release the editing authority, resulting in editing errors.

[0388] For example, after using the table interactive area recognition method in this embodiment to identify the interactive area contained in the document to be identified, an interactive function description code is added to the document to be identified. When displaying the table area, the identified interactive area can be rendered using different colors and other parameters according to the interactive function description code, thereby reminding the user that this is an interactive area; since the interactive function description code is added to the document to be identified, when the identified document to be identified is opened using tools such as PDF reading software, the reading software will display words such as "Form field found" and "The document contains interactive form fields", thereby reminding the user that there is an interactive area in the electronic document.

[0389] In this embodiment, the table interaction area recognition device can be run to execute the table interaction area recognition method. Specifically, corresponding to the above method embodiment, as shown in FIG. Fig. 27 As shown, the table interaction area identification device includes:

[0390] A cell recognition module is used to recognize valid cells in the document to be recognized;

[0391] A search module, used for searching for interactive elements in the area where the valid cells in the document to be identified are located; the interactive elements are document elements in the document to be identified that have an interactive attribute indication function;

[0392] The interactive area determination module is used to determine the interactive area information corresponding to the valid cell according to the search result.

[0393] When the table interactive area recognition device in the running document is used, the cell recognition module can execute step S1, the search module can execute step S2, and the interactive area determination module can execute step S3, thereby achieving the technical effect of the table interactive area recognition method in the document.

[0394] In this embodiment, a computer program for executing each step in the method for identifying a table interactive area in a document is obtained through computer programming, and such a computer program is stored in Fig.28 The memory of the computer device shown, or Fig.29 The computer program stored in the memory or computer-readable storage medium may be Fig.28 The processor in Fig.29 The processor in the document reads it out and obtains instructions for executing each step in the method for identifying a table interaction area in a document through operations such as compilation. The processor executes these instructions to execute each step S1-S4 in the method for identifying a table interaction area in a document.

[0395] because Fig.28 The computer device and Fig.29 The computer-readable storage media shown can enable the processor to execute the table interaction area recognition method in the document, so Fig.28 The computer device and Fig.29 The computer-readable storage media shown can achieve the same technical effect as the method for identifying a table interaction area in a document in the embodiment.

[0396] It should be noted that, unless otherwise specified, the singular forms of "a" and "the" used in this disclosure are also intended to include plural forms, unless the context clearly indicates other meanings. In addition, unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as those generally understood by those skilled in the art. The terms used in the present embodiment specification are only for describing specific embodiments, not for limiting the present invention. The term "and / or" used in this embodiment includes any combination of one or more related listed items.

[0397] It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided by the present embodiment is only intended to better illustrate embodiments of the present invention, and unless otherwise required, the scope of the present invention will not be limited.

[0398] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed dedicated integrated circuit for this purpose.

[0399] In addition, the operations of the process described in this embodiment may be performed in any suitable order, unless otherwise indicated in this embodiment or otherwise clearly contradicted by the context. The process described in this embodiment (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed on one or more processors in common, by hardware or a combination thereof. A computer program includes a plurality of instructions that may be executed by one or more processors.

[0400] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or part thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the above steps in conjunction with a microprocessor or other data processor, the invention of this embodiment includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0401] The above are only preferred embodiments of the present invention. The present invention is not limited to the above embodiments. As long as the technical effects of the present invention are achieved by the same means, any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical solutions and / or implementation methods may have various modifications and changes.

Claims

1. A table interaction area recognition method, characterized in that: include: Identify valid cells in the document to be identified; Searching for interactive elements in the area where the valid cells in the document to be identified are located; The interactive element is a document element in the document to be identified that has an interactive attribute indication function; According to the search results, the interactive area information corresponding to the valid cell is determined.

2. The table interactive area recognition method according to claim 1, characterized in that: Determining the interactive area information corresponding to the valid cell according to the search result includes: In case the interactive element is found, determining the validity of the interactive element; When the interactive element is valid, the interactive area information is determined according to the interactive element.

3. The table interaction area recognition method according to claim 2, characterized in that: Before determining the validity of the interactive element, the method further comprises: Performing type detection on the interactive element; and determining the validity of the interactive element according to a result of the type detection; When the interactive element is valid, determining the interactive area information according to the interactive element includes: In the case where it is detected that the interactive element belongs to the first type of interactive element, searching for a first text element in the valid cell; the first text element is a document element of the text type in the document to be identified; In the case where the first text element is found in the valid cell, detecting a modification relationship of the interactive element to the first text element; The validity of the interactive element is determined according to the modification relationship.

4. The table interaction area recognition method according to claim 3, characterized in that: Determining the validity of the interactive element according to the modification relationship includes: In the case where the modification relationship indicates that the interactive element modifies the first text element, determining that the interactive element is invalid; In a case where the modification relationship indicates that the interactive element does not modify the first text element, the validity of the first-type interactive element is determined according to the position of the first-type interactive element.

5. The table interactive area recognition method according to claim 4, characterized in that: The determining the validity of the first-type interactive element according to the position of the first-type interactive element includes: Obtaining a cell bounding box, a joint bounding box, and an interactive bounding box; the cell bounding box is a bounding box corresponding to the valid cell, the joint bounding box is a minimum bounding box that includes all the first text elements in the valid cell, and the interactive bounding box is a bounding box corresponding to the interactive element; Determining an allowable range of the interaction bounding box according to the relative sizes of the joint bounding box and the cell bounding box; When the interaction boundary box is within the allowable range, the interaction element is determined to be valid; otherwise, the interaction element is determined to be invalid.

6. The table interaction area recognition method according to claim 5, characterized in that: The determining the allowable range of the interactive bounding box according to the relative size of the joint bounding box and the cell bounding box includes: Acquire a height ratio; the height ratio is a ratio of the height of the joint bounding box to the height of the cell bounding box; In the case where the height ratio is greater than a height ratio threshold, determining the allowable range according to the joint bounding box; an upper limit of the allowable range is obtained by adjusting upward the top of the joint bounding box, and a lower limit of the allowable range is obtained by adjusting downward the bottom of the joint bounding box; When the height ratio is less than or equal to a height ratio threshold, the allowed range is determined according to the joint bounding box and the cell bounding box; the upper limit of the allowed range is obtained by adjusting upward from the top of the joint bounding box, and the lower limit of the allowed range is obtained by adjusting downward from the bottom of the cell bounding box.

7. The table interactive area recognition method according to claim 2, characterized in that: The determining the interactive area information according to the interactive element includes: Determine the left boundary of the interaction bounding box as the left boundary of the interaction area; Determine the right boundary of the interaction bounding box as the right boundary of the interaction area; An upper boundary of the interaction area and a lower boundary of the interaction area are determined according to the interaction boundary box.

8. The table interaction area recognition method according to claim 7, characterized in that: The determining, according to the interaction boundary box, an upper boundary of the interaction area and a lower boundary of the interaction area includes: determining a first height threshold; When the height of the row where the interaction bounding box is located is greater than or equal to the first height threshold, obtaining the upper boundary of the row where the interaction bounding box is located and determining it as the upper boundary of the interaction area, and obtaining the lower boundary of the row where the interaction bounding box is located and determining it as the lower boundary of the interaction area; or When the height of the row where the interaction boundary box is located is less than the first height threshold, the upper boundary of the row where the interaction boundary box is located is obtained and raised to be determined as the upper boundary of the interaction area, and the lower boundary of the interaction boundary box is obtained to be determined as the lower boundary of the interaction area.

9. The table interactive area recognition method according to claim 8, characterized in that: The determining of the first height threshold comprises: When there are multiple interactive bounding boxes, the average row height of the rows where all interactive bounding boxes are located is obtained; The first height threshold is determined according to the row height average value.

10. The table interactive area recognition method according to claim 2, characterized in that: Determining the interactive area information corresponding to the valid cell according to the search result includes: In the case that any of the document elements is not found, the interactive element is invalid, or the first text element is not found, searching for surrounding cells of the valid cell; the surrounding cells are cells located in the same table area as the valid cell; In the case where the surrounding cells are searched, the interactive area information is determined according to the surrounding cells.

11. The table interactive area recognition method according to claim 10, characterized in that: The searching of surrounding cells for the valid cell includes: Obtaining row index information of the valid cell; In the table area where the valid cells are located, searching for cells having the same row index information; The searched cells are used as the surrounding cells.

12. The table interactive area recognition method according to claim 10, characterized in that: The determining the interactive area information according to the surrounding cells includes: Searching the surrounding cells for a second text element; When the second text element is found, obtaining an average value of the row heights of all rows where the second text elements are located; Determine the average of the row heights of the rows where all the second text elements are located as the height of the interactive area; Determining the lower boundary of the interaction bounding box as the lower boundary of the interaction area; An upper boundary of the interaction area is determined according to the height of the interaction area and the lower boundary of the interaction area.

13. The table interactive area recognition method according to claim 12, characterized in that: The determining the interactive area information according to the surrounding cells further includes: If the second text element is not found, obtaining a cell bounding box; the cell bounding box is a bounding box corresponding to the valid cell; Lowering the upper limit of the cell boundary box to determine the upper limit of the interaction area; The lower limit of the cell boundary box is adjusted upward to determine the lower limit of the interaction area; Adjusting the left boundary of the cell boundary box to the right to determine the left boundary of the interaction area; The right boundary of the cell boundary box is adjusted to the left to determine the right boundary of the interaction area.

14. The table interactive area recognition method according to claim 2 or 3, characterized in that: Determining the interactive area information corresponding to the valid cell according to the search result includes: When the interactive element is found, performing type detection on the interactive element; In the case where it is detected that the interactive element belongs to the second type of interactive element, acquiring an interactive bounding box; the interactive bounding box is a bounding box corresponding to the interactive element; The interaction bounding box is determined as an interaction area in the valid cell.

15. The table interactive area recognition method according to claim 2 or 3, characterized in that: The table interaction area identification method further includes: generating an interaction function description code according to the interaction area information; The interactive function description code is added to the document to be identified.

16. A table interactive area recognition device, characterized in that: include: A cell recognition module is used to recognize valid cells in the document to be recognized; A search module, used for searching for interactive elements in the area where the valid cells in the document to be identified are located; The interactive element is a document element in the document to be identified that has an interactive attribute indication function; The interactive area determination module is used to determine the interactive area information corresponding to the valid cell according to the search result.

Images