Image table structure identification method and device, computer device and storage medium

Abstract

The present application relates to the technical field of computer vision, and provides an image table structure recognition method and device, computer equipment and storage medium, the method comprises the following steps: detecting each text block included in a table structure in a table image, constructing a dense map corresponding to the table structure according to each text block; determining non-adjacent text blocks according to the polygonal contour between the text boxes corresponding to each text block, and clipping the edges of the non-adjacent text blocks in the dense map to obtain a rough edge-clipped dense map; inputting the text subgraph and text content corresponding to the two text blocks connected by an edge in the rough edge-clipped dense map into a trained model to predict the actual positional relationship of the two text blocks connected by the edge; adjusting the rough edge-clipped dense map according to the actual positional relationship to obtain a table structure dense map; and restoring the table structure dense map to obtain a table recognition result. The present method can improve the universality and accuracy of table structure recognition.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to an image table structure recognition method and device, a computer device and a storage medium. BACKGROUND

[0002] Tables are an important form of information presentation. By organizing complex data into a standard structure, information retrieval, comparison and analysis are facilitated, and tables have become an indispensable element in people's daily information-based office work. However, in actual application, for the convenience of transmission and reading, tables are usually transmitted in the form of PDF or image, which makes it impossible for computers to directly understand table information, and manual extraction and processing are required. Due to the huge number and complex styles of tables in the information age, manual processing is tedious and time-consuming, and table understanding in PDF or image carriers has become a problem to be solved.

[0003] At present, with the rapid development of science and technology, table understanding technology has ushered in a spring of development, and has become a research hotspot in the academic and industrial circles. According to the current research progress, table understanding technology can be divided into table detection and table recognition. Table detection refers to locating the table region from the image, which is the premise of table recognition. Table recognition can be further divided into table structure recognition and table restoration. Table structure recognition is the key to table understanding, and the association between cells in the table needs to be obtained. Table restoration is a post-processing process of table recognition, which organizes the position information and table structure information of the structure table detection and the text recognition result into an editable table. Among them, table structure recognition is a research hotspot in the academic circle, mainly including methods based on table lines and methods based on cell segmentation and merging. However, due to the complex and varied styles of tables, including full-line tables, non-full-line tables and non-line tables, these methods cannot be applied to all scenarios, and the lack of universality leads to reduced accuracy of table structure recognition. SUMMARY

[0004] Therefore, it is necessary to provide an image table structure recognition method, device, computer device and storage medium capable of improving universality and accuracy in view of the above technical problems.

[0005] The present application provides an image table structure recognition method, comprising:

[0006] detecting each text block included in a table structure in a table image, and constructing a dense map corresponding to the table structure according to each text block;

[0007] determining non-adjacent text blocks according to the polygonal outlines between the text boxes corresponding to each text block, and clipping the edges of the non-adjacent text blocks in the dense map to obtain a roughly clipped dense map;

[0008] inputting the text subgraph and the text content corresponding to two text blocks of an edge in the rough edge-cutting dense graph into the trained model to predict an actual position relationship of the two text blocks of the edge;

[0009] adjusting the rough edge-cutting dense graph according to the actual position relationship to obtain a table structure dense graph;

[0010] restoring a table recognition result based on the table structure dense graph.

[0011] In one of the embodiments, the rough edge-cutting dense graph is obtained by determining non-adjacent text blocks according to polygon outlines between text boxes corresponding to the text blocks and cutting edges between the non-adjacent text blocks in the dense graph, and the method comprises the following steps.

[0012] combining each of the text blocks two by two to obtain a text block group, and regarding a text block outside the text block group as an out-of-group text block of the text block group;

[0013] determining a minimum circumscribed polygon of a text box corresponding to each text block in the text block group, and calculating an intersection area of the minimum circumscribed polygon and a text box corresponding to each out-of-group text block;

[0014] when a preset multiple of an area of the text box corresponding to the out-of-group text block is less than the intersection area, determining that each text block in the text block group is non-adjacent;

[0015] cutting edges between non-adjacent text blocks in the dense graph to obtain a rough edge-cutting dense graph.

[0016] In one of the embodiments, the actual position relationship of the two text blocks of the edge in the rough edge-cutting dense graph is predicted by inputting a text subgraph and text content corresponding to two text blocks of an edge in the rough edge-cutting dense graph into a trained model, and the method comprises the following steps.

[0017] for two text blocks of an edge in the rough edge-cutting dense graph, obtaining corresponding text content and cutting a text box image and a minimum circumscribed rectangle image corresponding to the text content from the table image as a text subgraph;

[0018] extracting shallow semantic features of the text subgraph and the text content, and fusing each of the shallow semantic features to obtain a multi-modal semantic feature;

[0019] extracting an image high-dimensional semantic feature of the multi-modal semantic feature by using a deep semantic feature model, and outputting an actual position relationship of the two text blocks of the edge according to the image high-dimensional semantic feature.

[0020] In one of the embodiments, the shallow semantic features of the text sub-graph and the text content are extracted, the information of the shallow semantic features is fused to obtain the multi-modal semantic features, including:

[0021] When the text sub-graph is the text box image, the image semantic information of the text sub-graph is extracted by using the shallow CNN convolution, and a ReShape operation is performed to obtain the shallow semantic features of the text sub-graph;

[0022] When the text sub-graph is the minimum bounding rectangle image, the image semantic information of the text sub-graph is extracted by using the shallow CNN convolution, and a ReShape operation is performed after the image semantic information is pooled to obtain the shallow semantic features of the text sub-graph;

[0023] The text content is converted into a high-dimensional vector by using a word vector model, and a ReShape operation is performed after the context semantic information is extracted based on the high-dimensional vector to obtain the shallow semantic features of the text content;

[0024] The shallow semantic features of the text sub-graph and the text content are fused to obtain the multi-modal semantic features.

[0025] In one of the embodiments, the table structure dense graph is obtained by adjusting the rough edge-cutting dense graph according to the actual position relationship, including:

[0026] When it is determined according to the actual position relationship that the two text blocks connected by the edge are not adjacent, the edge is cut from the rough edge-cutting dense graph;

[0027] When it is determined according to the actual position relationship that the two text blocks connected by the edge belong to the same cell, the two text blocks connected by the edge are merged from the rough edge-cutting dense graph;

[0028] The position of the rough edge-cutting dense graph after cutting and merging is adjusted to obtain the table structure dense graph.

[0029] In one of the embodiments, the text blocks included in the table structure in the table image are detected, and the dense graph corresponding to the table structure is constructed according to the text blocks, including:

[0030] Text detection is performed on the table structure in the table image to obtain text boxes and text content;

[0031] The text boxes and the corresponding text content are combined to form text blocks;

[0032] Each of the text blocks is connected to each other as a graph node to obtain a dense graph.

[0033] In one of the embodiments, before extracting the shallow semantic features of the text subgraph, the method further comprises:

[0034] scaling the minimum edge of the text subgraph to a preset size by equal ratio;

[0035] when the maximum edge of the text subgraph after equal ratio scaling is greater than or equal to the product of the preset size and a threshold, performing centering clipping on the text subgraph;

[0036] when the maximum edge of the text subgraph after equal ratio scaling is less than the product of the preset size and a threshold, performing blank pixel supplementing on the text subgraph.

[0037] In one of the embodiments, an image table structure recognition device is provided, comprising:

[0038] a detection construction module, configured to detect each text block included in a table structure in a table image, and construct a dense graph corresponding to the table structure according to each text block;

[0039] a rough edge clipping module, configured to determine non-adjacent text blocks according to the polygonal contour between the text boxes corresponding to each text block, and clip the edges of the non-adjacent text blocks in the dense graph to obtain a rough edge clipping dense graph;

[0040] a prediction module, configured to input the text subgraph and text content corresponding to two text blocks connected by an edge in the rough edge clipping dense graph into a trained model to predict the actual positional relationship of the two text blocks connected by the edge;

[0041] a fine edge clipping module, configured to adjust the rough edge clipping dense graph according to the actual positional relationship to obtain a table structure dense graph;

[0042] a table restoration module, configured to restore a table recognition result based on the table structure dense graph.

[0043] The present application also provides a computer device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor implements the steps of the image table structure recognition method of any one of the above embodiments when executing the computer program.

[0044] The present application also provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the image table structure recognition method of any one of the above embodiments when executed by a processor.

[0045] The image table structure recognition method, device, computer device and storage medium, by detecting each text block included in the table structure in the table image to construct a corresponding dense map, then determining the polygon contour between the text boxes corresponding to each text block to cut edges of non-adjacent text blocks in the dense map to obtain a rough edge-cutting dense map, further inputting the text subgraph and text content corresponding to the two text blocks connected by edges in the rough edge-cutting dense map into the trained model to predict the actual position relationship of the two text blocks connected by edges, and then adjusting the rough edge-cutting dense map according to the actual position relationship to obtain a table structure dense map and restoring the table recognition result. The method combines image processing technology and deep learning technology, starts from the essential characteristics of the image table, ignores the external style information of the table, is not limited to the table style, has a wide application scenario, makes the table structure recognition more universal and accurate. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 An application environment diagram of the image table structure recognition method in an embodiment.

[0047] Figure 2 A flowchart of the image table structure recognition method in an embodiment.

[0048] Figure 3 A schematic diagram of a text detection result in an embodiment.

[0049] Figure 4 A structure schematic diagram of a dense map in an embodiment.

[0050] Figure 5 A structure schematic diagram of a rough edge-cutting dense map in an embodiment.

[0051] Figure 6 A whole network structure schematic diagram of a trained model in an embodiment.

[0052] Figure 7 A structure schematic diagram of a fine edge-cutting dense map in an embodiment.

[0053] Figure 8 A structure schematic diagram of a table structure dense map in an embodiment.

[0054] Figure 9 A schematic diagram of a table recognition result in an embodiment.

[0055] Figure 10 A structure block diagram of an image table structure recognition device in an embodiment. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and not used to limit the present application.

[0057] The image table structure recognition method provided by the present application can be applied in an application environment as shown in Figure 1 The application environment involves a terminal 102 and a server 104. The terminal 102 communicates with the server 104 through a network. The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers. When the terminal 102 receives an image table structure recognition instruction, the terminal 102 can implement the above-mentioned image table structure recognition method alone. Alternatively, the terminal 102 can send the image table structure recognition instruction to the server 104 in communication, and the server 104 implements the above-mentioned image table structure recognition method.

[0058] Taking the server 104 as an example, specifically, the server 104 detects each text block included in the table structure in the table image, and constructs a dense graph corresponding to the table structure according to each text block; the server 104 determines non-adjacent text blocks according to the polygonal contour between the text boxes corresponding to each text block, and clips the edges of the non-adjacent text blocks in the dense graph to obtain a roughly clipped dense graph; the server 104 inputs the text subgraph and the text content corresponding to the two text blocks connected by the edge in the roughly clipped dense graph into the trained model to predict the actual position relationship of the two text blocks connected by the edge; the server 104 adjusts the roughly clipped dense graph according to the actual position relationship to obtain a table structure dense graph; and the server 104 restores the table structure dense graph to obtain a table recognition result.

[0059] In one embodiment, as shown in Figure 2 An image table structure recognition method is provided, which is taken as an example to illustrate the method applied to a server and includes the following steps:

[0060] Step S201, detecting each text block included in the table structure in the table image, and constructing a dense graph corresponding to the table structure according to each text block.

[0061] The table image refers to an image including a table structure, the text block refers to text in the table structure, and includes text content and a corresponding text box. The dense graph is a data structure graph with nodes and edges connecting the nodes. The dense graph in the present embodiment is constructed based on the text blocks of the table structure in the table image.

[0062] Specifically, when receiving the image table structure recognition instruction, first, the corresponding table image is acquired, text detection is performed on the table structure in the table image to obtain a text block included by the table structure. Then, the text block is regarded as a node, and a corresponding dense graph is constructed.

[0063] In one embodiment, the step S201 includes: performing text detection on the table structure in the table image to obtain a text box and text content; grouping the text box and the corresponding text content into a text block; and connecting edges between each two of the text blocks as nodes to obtain a dense graph.

[0064] Specifically, a text detection technique is used on the input table image to detect the included table structure to obtain a series of text boxes, and a text recognition technique is used on each text box to obtain text content. The corresponding text box bbox and text content text are grouped into a text block, which can be recorded as t=(bbox,text). All text blocks in the table structure can be recorded as T=(t1,t2,……,tn). As shown in FIG. 2, Figure 3 a schematic diagram of a text detection result is provided, Figure 3 which includes text content such as department, responsibility, sales department, and the like. The rectangular box surrounding the text content in the figure is the text box.

[0065] Then, a dense graph corresponding to the table structure is constructed based on the obtained text box. That is, for the text block set T, each text block t in the set T is regarded as a node, and edges are connected between any two text blocks t, that is, a dense graph G=<T,E> of the table structure is obtained, E is a set of edges between the text blocks, which can be recorded as E={e1,e2,……,en}. As shown in FIG. 3, Figure 4 a structural schematic diagram of a dense graph is provided. Figure 4 As shown in FIG. 4, Figure 3 a dense graph corresponding to the table structure shown in FIG. 1 is provided.

[0066] In step S202, non-adjacent text blocks are determined according to the polygonal contours between the text boxes corresponding to the text blocks, and the non-adjacent text blocks are cut in the dense graph to obtain a roughly cut dense graph.

[0067] The non-adjacent text blocks refer to two text blocks that are not adjacent in the actual table structure. For example, Figure 3 “department” and “responsibility” in FIG. 1 are adjacent text boxes, but “department” and “technical department” are non-adjacent text blocks. Cutting edges refers to removing the edges between the two text blocks in the dense graph, that is, deleting the edges between the two text blocks.

[0068] Specifically, in the embodiment, the construction of the dense graph is to connect all the text blocks in pairs to obtain, and the purpose of the connection is mainly to represent whether the text blocks are adjacent. However, in fact, not all the text blocks in the table structure are adjacent to each other. Therefore, in order to accurately obtain the adjacent relationship of the text blocks in the table structure, it is necessary to further remove the connection between the non-adjacent nodes in the dense graph. In the embodiment, whether the nodes in the dense graph are adjacent is determined based on the polygon outline of the text box corresponding to each text block. For any two text blocks, if there are other text boxes in the polygon outline of the two text boxes, it means that the two text blocks are not adjacent, and if there are no other text boxes in the polygon outline, it means that the two text blocks are adjacent. In addition, in order to accurately determine whether two text blocks are adjacent based on the text boxes in the polygon outline, the polygon outline is preferably a minimum enclosing polygon, which can ensure that the area of the polygon outline is as small as possible to prevent the judgment from being wrong due to too much information. Finally, for the two adjacent text blocks determined, the connection between the two text blocks is retained, and for the two non-adjacent text blocks determined, the connection between the two text blocks is removed, and the dense graph obtained after all the edge cutting operations is the rough edge-cutting dense graph.

[0069] In one embodiment, step S202 comprises: combining each text block in pairs to obtain a text block group, and regarding the text blocks outside the text block group as out-group text blocks of the text block group; determining the minimum enclosing polygon of the text box corresponding to each text block in the text block group, and calculating the intersection area of the minimum enclosing polygon and the text box corresponding to each out-group text block; when the preset multiple of the area of the text box corresponding to the out-group text block is less than the intersection area, determining that each text block in the text block group is not adjacent; removing the connection between the non-adjacent text blocks in the dense graph to obtain a rough edge-cutting dense graph.

[0070] Specifically, first, for all edges e in the edge set E, find the corresponding two text blocks from the dense graph G, assume to be t1=(bbox1, text1) and t2=(bbox2, text2), that is, a text block group obtained by two-by-two combination, and for the text block group t1 and t2, any one text block t3, t4, …, tn outside the group is called an out-of-group text block. Other text block groups are the same, and will not be repeated here. Second, find the minimum enclosing polygon of the two text boxes corresponding to each text block in the text block group. The determination of the minimum enclosing polygon can use any existing method, such as using the related method in OpenCV. That is, for the text block group t1 and t2, the minimum enclosing polygon C corresponding to bbox1 and bbox2 can be found by using the related method in OpenCV. Then, the intersection area of the minimum enclosing polygon C and the area of the text box corresponding to each out-of-group text block is calculated, and then based on the size relationship between the area of the text box corresponding to each out-of-group text block and the intersection area, it is judged whether the two text blocks in the text block group are adjacent. That is, when there is any out-of-group text block in the out-of-group text blocks corresponding to any text block group that satisfies the following area condition, it means that the two text blocks in the text block group are not adjacent, and the edge e corresponding to the two text blocks in the dense graph is removed. For example, taking the text block group t1 and t2 as an example, when the text block t4 satisfies the following area condition, the edge of t1 and t2 is removed. The area condition is as follows:

[0071]

[0072] Where area is an area calculation formula, and a is a threshold hyperparameter (a preset multiple of the embodiment), and the value range is 0~1, that is, if the intersection area of the area of the text box bboxi corresponding to the out-of-group text block and the minimum enclosing polygon C exceeds a times the area of the text box bboxi, it means that the two text blocks corresponding to the minimum enclosing polygon C are not adjacent, and the edge does not exist, and the edge is removed in the dense graph. As shown in Figure 5 A structure diagram of a rough edge-cutting dense graph is shown in Figure 5 The dense graph is Figure 4 The dense graph based on the rough edge-cutting dense graph based on the above area method.

[0073] In step S203, the text subgraph and text content corresponding to the two text blocks of the edge in the rough edge-cutting dense graph are input into the trained model to predict the actual position relationship of the two text blocks of the edge.

[0074] The two text blocks connected by an edge in the dense graph refer to two nodes connected by an edge in the dense graph, the text subgraph refers to a subgraph related to the text block intercepted from the table image, and the text content is the literal content corresponding to the text block. The actual position relationship refers to the actual position relationship of the two text blocks in the table structure, including adjacent, non-adjacent, and belonging to the same cell. Among them, based on the table structure shown in FIG. 7, it can be known that Figure 3 The "responsible for sales and promotion" and "promote company products" in the table structure shown in FIG. 7 are actually the content of the same cell in the table, although they are two nodes in the dense graph. Figure 5

[0075] Specifically, after obtaining the rough edge-cutting dense graph, in order to further accurately obtain the actual dense graph corresponding to the table structure, the embodiment further utilizes the trained model to predict the actual position relationship of each node in the rough edge-cutting dense graph. The model can be a neural network model, and in this embodiment, the input is the text subgraph and the text content of the text block. The rough edge-cutting dense graph is a dense graph that has been cut once, so in order to save resources and improve efficiency, the second prediction can only be performed on the text blocks that are still connected by edges. That is, the text subgraph and the text content of the two text blocks connected by an edge are input into the trained model for prediction, and the model outputs the actual position relationship. The actual position relationship can be a label, a character, or other marks. Taking the label as an example, if during the training of the model, the two text blocks corresponding to the edge are not adjacent in the actual table structure, a label 0 is set to represent that they belong to the same cell, and a label 1 is set to represent that they belong to adjacent cells in the table structure. Then, the output of the model is the labels 0, 1, and 2, and the actual position relationship of the two text blocks is determined according to the labels 0, 1, and 2.

[0076] In one embodiment, step S203 includes: for the two text blocks connected by an edge in the rough edge-cutting dense graph, obtaining the corresponding text content and intercepting the corresponding text box image and minimum bounding rectangle image from the table image as the text subgraph; extracting the shallow semantic features of the text subgraph and the text content, and fusing the shallow semantic features to obtain the multi-modal semantic features; extracting the image high-dimensional semantic features of the multi-modal semantic features by using the deep semantic feature model, and outputting the actual position relationship of the two text blocks connected by an edge according to the image high-dimensional semantic features.

[0077] The text box image refers to the area image corresponding to the text box in the table image, the minimum bounding rectangle image refers to the area image corresponding to the minimum bounding rectangle of the two text boxes in the table image, and the area image refers to part of the table image.

[0078] ​Specifically, for two connected text blocks, first, the two text contents corresponding to the two text blocks are obtained. Then, the corresponding region images of the two text boxes are cut from the original table image as text subgraphs, and the minimum bounding rectangle of the two text boxes is found using the method in OpenCV to cut the region image corresponding to the minimum bounding rectangle from the original table image as a text subgraph. The obtained text content and text subgraph are used as the input of the model. Assuming that t1=(bbox1, text1) and t2=(bbox2, text2) are taken as examples, the input of the model includes text1, text2, the region image S1 corresponding to bbox1, the region image S2 corresponding to bbox2, and the region image S3 corresponding to the minimum bounding rectangle of bbox1 and bbox2, S1, S2 and S3 are the text subgraphs obtained in this embodiment. Then, the shallow semantic features of the text subgraphs and the text contents are extracted, and the information of each shallow semantic feature is fused to obtain a multi-modal semantic feature. Finally, the image high-dimensional semantic feature of the multi-modal semantic feature is extracted by using the deep semantic feature model, and the actual position relationship of the two text blocks is predicted and output according to the image high-dimensional semantic feature, and the output label is 0, 1 or 2.

[0079] In one embodiment, after obtaining the text subgraph, before extracting the shallow semantic feature of the text subgraph, a subgraph preprocessing step is further included: the minimum side of the text subgraph is equal to the preset size; when the maximum side of the equalized text subgraph is greater than or equal to the product of the preset size and the threshold, the text subgraph is centered and cropped; when the maximum side of the equalized text subgraph is less than the product of the preset size and the threshold, the text subgraph is supplemented with blank pixels.

[0080] Specifically, the preprocessing of the text subgraph is set according to the actual application situation, such as the size requirement of the model input image or the required size according to the actual business situation. Therefore, the preset size th can be set according to the actual situation. In this embodiment, after obtaining the text subgraph, first, the minimum side min(w, h) is equalized to the preset size th, that is, the minimum side is th, and the maximum side is max(w, h) x th / min(w, h). If the maximum side is greater than or equal to the product of the preset size th and the threshold, the threshold is set to 8 in this embodiment. That is, if the maximum side exceeds 8 x th, the text subgraph is centered and cropped to obtain the final text subgraph. If the maximum side is less than 8 x th, blank pixels are supplemented on both sides of the text subgraph to obtain the final text subgraph. That is, if the maximum side is wide, the corresponding size of the text subgraph is (8 x th, th), and if the maximum side is high, the corresponding size of the text subgraph is (th, 8 x th).

[0081] In one embodiment, shallow semantic features of the text subgraph and the text content are extracted, and information fusion is performed on the shallow semantic features to obtain a multi-modal semantic feature, including: when the text subgraph is a text box image, image semantic information of the text subgraph is extracted by using a shallow CNN convolution, and a ReShape operation is performed to obtain a shallow semantic feature of the text subgraph; when the text subgraph is a minimum bounding rectangle image, image semantic information of the text subgraph is extracted by using a shallow CNN convolution, and a ReShape operation is performed after the image semantic information is pooled to obtain a shallow semantic feature of the text subgraph; a word vector model is used to convert the text content into a high-dimensional vector, a ReShape operation is performed after context semantic information is extracted based on the high-dimensional vector to obtain a shallow semantic feature of the text content; and the shallow semantic features of the text subgraph and the text content are fused to obtain a multi-modal semantic feature.

[0082] Specifically, in order to accurately obtain shallow semantic features of different types of data, different ways are used to extract shallow semantic features for images and texts in this embodiment, and different ways are also used to extract shallow semantic features for text box images and minimum bounding rectangle images.

[0083] For the text box image, the image semantic information is extracted by shallow CNN convolution and then directly reshaped. For the minimum bounding rectangle image, the image semantic information is extracted by shallow CNN convolution, then pooled and reshaped, so as to obtain the shallow semantic features of different text subgraphs. Among them, based on the difference in text layout in the table, the text line is usually divided into horizontal and vertical. Therefore, in order to avoid the influence of direction, the horizontal shallow CNN convolution can be used for the text box image of the horizontal text line, and the vertical shallow CNN convolution can be used for the text box image of the vertical text line. The difference between horizontal and vertical convolution lies in the size of the convolution kernel. For example, for a text subgraph with a size of (8xth, th), that is, the text line direction is horizontal, the horizontal shallow CNN convolution is used to extract the image semantic information, wherein the convolution setting of the embodiment is preferably (256, 3, 7), that is, the number of convolution kernels is 256, and the size of the convolution kernel is (7, 3). After the convolution operation, the output dimension is (th, 8xth, 256). For a text subgraph with a size of (th, 8xth), that is, the text line direction is vertical, the vertical shallow CNN convolution is used to extract the image semantic information, wherein the convolution setting of the vertical shallow CNN convolution is opposite to the horizontal direction, that is, (256, 7, 3), that is, the number of convolution kernels is still 256, but the size of the convolution kernel is (3, 7). Therefore, after the convolution operation, the output dimension is (8xth, th, 256), and then the output result is reshaped to obtain the output dimension of (th, 8xth, 256). For the text box image, that is, the text subgraphs S1 and S2, the th of the embodiment is preferably set to 32, so that the final output dimension is (32, 256, 256). For the minimum bounding rectangle image, that is, the text subgraph S3, the th is preferably set to 64, and then the high-dimensional feature vector with a dimension of (64, 512, 512) is obtained by shallow semantic feature extraction. Then, the final output dimension is (32, 256, 256) by using the pooling operation. The maxpool operation is preferably performed by using a 2x2 pooling kernel.

[0084] For the text content text, an existing word vector model is adopted, such as a word vector model provided by Google (Char Embed), to convert the text content text into a high-dimensional vector, and then a ReShape operation is performed based on the high-dimensional vector to extract context semantic information, so as to obtain the shallow semantic features of the text content text. In this embodiment, BiLSTM is preferably used for learning the context semantic information. In addition, the dimension of the word vector model is 256, and therefore the maximum length of the text is set to 64, and if the length of the text is less than 64, 0 is supplemented, and finally the obtained high-dimensional feature vector is (64, 256). At the same time, if the BiLSTM model is used, since the number of hidden layer units is 2048, after the context semantic information is extracted, the output dimension is (64, 2048). After the ReShape operation, the obtained dimension is (32, 16, 256).

[0085] Then, the shallow semantic features corresponding to each input are fused in the second dimension based on the first dimension and the third dimension, to obtain multi-modal semantic features. For the dimensions of the shallow semantic features corresponding to S1, S2, S3, text1 and text2 (32, 256, 256), (32, 256, 256), (32, 256, 256), (32, 16, 256) and (32, 16, 256), the output dimension of the multi-modal semantic features is (32, 800, 256).

[0086] Finally, the multi-modal semantic features are input into a deep semantic feature model in the backend to extract image high-dimensional semantic features, the network input dimension is (32, 256, 256), and the network output dimension is (3). The deep semantic feature model in the backend can adopt any existing neural network model, such as VGG series (VGG16, VGG19), Inception series (Inception-v1, Inception-v2, Inception-v3, Inception-v4) and the like.

[0087] The overall network structure of the model trained in this embodiment is shown in Figure 6 The input of the network structure shown in Figure 6 The input of the network structure shown in Figure 6 When the model shown in

[0088] Step S204, adjusting the rough edge-cutting dense graph according to the actual position relationship to obtain a table structure dense graph.

[0089] Specifically, after the model outputs the actual position relationship of the two text blocks of each edge, an edge-cutting or merging adjustment operation is performed in the rough edge-cutting dense graph based on the actual position relationship, so as to obtain a table structure dense graph corresponding to the table structure.

[0090] In one embodiment, step S204 includes: when it is determined according to the actual position relationship that the two text blocks of the edge are not actually adjacent, cutting the edge from the rough edge-cutting dense graph; when it is determined according to the actual position relationship that the two text blocks of the edge belong to the same cell, merging the two text blocks of the edge in the rough edge-cutting dense graph; and performing position adjustment on the rough edge-cutting dense graph after the cutting and merging to obtain a table structure dense graph.

[0091] Specifically, after the model predicts the actual position relationship, if the model output result is 0, it indicates that the two text blocks are not adjacent, and the edge is removed in the rough edge-cutting dense graph to complete the edge-cutting. If the output result is 1, it indicates that the two text blocks belong to the same cell, and the two text block nodes corresponding to the edge are merged into one node, and the edge is migrated to the new node. If the output result is 2, it indicates that the two text blocks are adjacent, and no processing is performed. Thus, the rough edge-cutting dense graph after the cutting and merging can be called a fine edge-cutting dense graph. Then, the fine edge-cutting dense graph obtained is further adjusted to obtain a table structure dense graph. As shown in Figure 7 a fine edge-cutting dense graph is provided, Figure 7 as shown in Figure 5 the fine edge-cutting dense graph of the rough edge-cutting dense graph shown in Figure 8 a table structure dense graph is provided, Figure 8 the dense graph shown in Figure 7 is a dense graph after position adaptive adjustment according to the adjacent relationship.

[0092] Step S205, restoring a table recognition result based on the table structure dense graph.

[0093] Specifically, after obtaining the table structure dense graph, based on the table structure information and the text detection and recognition result, a table restoration technology is used to convert the image table into an editable table, such as an editable Word table. As shown in Figure 9 a table recognition result schematic diagram is provided, Figure 9 as shown in Figure 8 the table recognition result after restoration of the table dense graph shown in

[0094] The image table structure identification method considers that there are various table styles in actual scenarios, such as full-line tables, non-full-line tables, and non-line tables, and each table display form is complex and changeable. Starting from the alignment consistency of table rows and columns, the traditional image processing technology and the deep learning technology are combined, and the multi-modal semantic features such as image features and text features are fused to obtain the table structure information by using the edge clipping strategy from top to bottom. The method starts from the essential characteristics of the image table, ignores the external style information of the table, is widely applicable, and makes the table structure identification more universal and accurate.

[0095] It should be understood that, although Figure 2 The steps in the flowchart of the method are displayed in sequence according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps in the method can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps.

[0096] In one embodiment, as shown in Figure 10 An image table structure identification apparatus is provided, comprising:

[0097] The detection and construction module 1001 is configured to detect each text block included in the table structure in the table image, and construct a dense graph corresponding to the table structure according to each text block.

[0098] The rough edge clipping module 1002 is configured to determine non-adjacent text blocks according to the polygonal contour between the text boxes corresponding to each text block, and clip the edges of the non-adjacent text blocks in the dense graph to obtain a rough edge clipping dense graph.

[0099] The prediction module 1003 is configured to input the text subgraph and text content corresponding to the two text blocks connected by an edge in the rough edge clipping dense graph into a trained model to predict the actual positional relationship of the two text blocks connected by the edge.

[0100] The fine edge clipping module 1004 is configured to adjust the rough edge clipping dense graph according to the actual positional relationship to obtain a table structure dense graph.

[0101] The table restoration module 1005 is configured to restore a table recognition result based on the table structure dense graph.

[0102] In an embodiment, the rough edge clipping module 1002 is further configured to combine each text block to form a text block group, and a text block outside the text block group is regarded as an out-group text block; determine a minimum bounding polygon of a text box corresponding to each text block in the text block group, and calculate an intersection area between the minimum bounding polygon and a text box corresponding to each out-group text block; when a preset multiple of an area of the text box corresponding to the out-group text block is smaller than the intersection area, determine that each text block in the text block group is not adjacent; and remove edges between the non-adjacent text blocks in the dense graph to obtain a rough edge clipping dense graph.

[0103] In an embodiment, the prediction module 1003 is further configured to, for two text blocks connected by an edge in the rough edge clipping dense graph, obtain corresponding text content and cut out a text box image and a minimum bounding rectangle image corresponding to the text box image from the table image as a text sub-image; extract shallow semantic features of the text sub-image and the text content, and fuse the shallow semantic features to obtain a multi-modal semantic feature; extract an image high-dimensional semantic feature of the multi-modal semantic feature by using a deep semantic feature model, and output an actual positional relationship of the two text blocks connected by the edge according to the image high-dimensional semantic feature.

[0104] In an embodiment, the prediction module 1003 is further configured to, when the text sub-image is the text box image, perform a ReShape operation on image semantic information of the text sub-image extracted by using a shallow CNN convolution to obtain a shallow semantic feature of the text sub-image; when the text sub-image is the minimum bounding rectangle image, perform a ReShape operation on image semantic information of the text sub-image extracted by using a shallow CNN convolution after pooling the image semantic information to obtain a shallow semantic feature of the text sub-image; convert the text content into a high-dimensional vector by using a word vector model, perform a ReShape operation on context semantic information extracted based on the high-dimensional vector to obtain a shallow semantic feature of the text content; and fuse the shallow semantic features of the text sub-image and the text content to obtain the multi-modal semantic feature.

[0105] In an embodiment, the fine edge clipping module 1004 is further configured to, when it is determined according to the actual positional relationship that the two text blocks connected by the edge are actually not adjacent, clip the edge from the rough edge clipping dense graph; when it is determined according to the actual positional relationship that the two text blocks connected by the edge belong to a same cell, merge the two text blocks connected by the edge from the rough edge clipping dense graph; and perform position adjustment on the rough edge clipping dense graph after clipping and merging to obtain a table structure dense graph.

[0106] In an embodiment, the detection and construction module 1001 is further configured to perform text detection on a table structure in a table image to obtain a text box and text content; combine the text box and the corresponding text content to form a text block; and connect each text block by an edge to form a dense graph.

[0107] In an embodiment, the prediction module 1003 is further configured to scale the minimum side of the text sub-graph to a preset size, and when the maximum side of the scaled text sub-graph is greater than or equal to the product of the preset size and a threshold, the text sub-graph is cropped in the center, and when the maximum side of the scaled text sub-graph is less than the product of the preset size and the threshold, the text sub-graph is supplemented with blank pixels.

[0108] The specific limitations of the image table structure recognition device can refer to the limitations of the image table structure recognition method described above, and will not be repeated here. Each module in the above image table structure recognition device can be realized by software, hardware and their combinations. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to call and execute the operations corresponding to the above modules by the processor. Based on such understanding, the present application realizes all or part of the processes in the above embodiment method, and can also be completed by instructing related hardware through a computer program. The computer program can be stored in a computer readable storage medium, and when the processor executes the computer program, the steps of the above image table structure recognition method embodiments can be realized. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form, etc.

[0109] In an embodiment, a computer device is provided, which can be a server, comprising a processor, a memory and a network interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store data. The network interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement an image table structure recognition method. Illustratively, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present application. One or more modules can be a series of computer program instruction segments that can complete a specific function, which are used to describe the execution process of the computer program in the computer device.

[0110] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like, which is a control center of the computer device and connects all parts of the computer device through various interfaces and lines.

[0111] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the computer device by running or executing the computer program and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc.; and the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0112] Those skilled in the art can understand that the computer device structure shown in the embodiment is only part of the structure related to the application scheme, and does not constitute a limitation on the computer device to which the application is applied. The specific computer device can include more or fewer components, or combine certain components, or have a different component arrangement.

[0113] In one embodiment, a computer device is provided, including a memory and a processor, the memory stores a computer program, and the processor realizes the image table structure identification method described in any of the above embodiments when executing the computer program.

[0114] In one embodiment, a computer readable storage medium is provided, and the computer program is stored in the computer readable storage medium, and the computer program is executed by a processor to implement the image table structure recognition method described in any of the above embodiments.

[0115] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the computer program can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory. The volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0116] The technical features of the above embodiments can be combined in any manner. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0117] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. An image table structure recognition method, characterized by, The method comprises the following steps: detecting each text block included in the table structure of the table image, and constructing a dense graph corresponding to the table structure according to each text block; combining each two text blocks to form a text block group, and identifying an out-group text block; calculating the intersection area of the minimum circumscribed polygon of each text block in the group and the text box of the out-group text block, judging the adjacency of each text block in the group according to a preset multiple condition, and cutting the edges between the non-adjacent text blocks in the dense graph to obtain a roughly cut dense graph; for each pair of text blocks connected by an edge in the roughly cut dense graph, obtaining the corresponding text content and text subgraph, fusing the shallow semantic features of the text content and the text subgraph to obtain a multi-modal semantic feature, then extracting the image high-dimensional semantic feature thereof, and further outputting the actual position relationship between the two text blocks; cutting the edges between the actually non-adjacent text blocks according to the actual position relationship, merging the text blocks belonging to the same cell, and adjusting the roughly cut dense graph after the cutting and merging to obtain a table structure dense graph; based on the table structure dense graph, the table recognition result is recovered.

2. The method of claim 1, wherein, combining each two text blocks to form a text block group, and identifying an out-group text block; calculating the intersection area of the minimum circumscribed polygon of each text block in the group and the text box of the out-group text block, judging the adjacency of each text block in the group according to a preset multiple condition, and cutting the edges between the non-adjacent text blocks in the dense graph to obtain a roughly cut dense graph, comprising: combining each two text blocks to form a text block group, and identifying an out-group text block; calculating the intersection area of the minimum circumscribed polygon of each text block in the group and the text box of the out-group text block, judging the adjacency of each text block in the group according to a preset multiple condition, and cutting the edges between the non-adjacent text blocks in the dense graph to obtain a roughly cut dense graph, comprising: combining each two text blocks to form a text block group, and identifying an out-group text block; calculating the intersection area of the minimum circumscribed polygon of each text block in the group and the text box of the out-group text block, judging the adjacency of each text block in the group according to a preset multiple condition, and cutting the edges between the non-adjacent text blocks in the dense graph to obtain a roughly cut dense graph, comprising: combining each two text blocks to form a text block group, and identifying an out-group text block; calculating the intersection area of the minimum circumscribed polygon of each text block in the group and the text box of the out-group text block, judging the adjacency of each text block in the group according to a preset multiple condition, and cutting the edges between the non-adjacent text blocks in the dense graph to obtain a roughly cut dense graph, comprising: for each pair of text blocks connected by an edge in the roughly cut dense graph, obtaining the corresponding text content and text subgraph, fusing the shallow semantic features of the text content and the text subgraph to obtain a multi-modal semantic feature, then extracting the image high-dimensional semantic feature thereof, and further outputting the actual position relationship between the two text blocks, comprising:

3. The method of claim 1, wherein, for each pair of text blocks connected by an edge in the roughly cut dense graph, obtaining the corresponding text content and text subgraph, fusing the shallow semantic features of the text content and the text subgraph to obtain a multi-modal semantic feature, then extracting the image high-dimensional semantic feature thereof, and further outputting the actual position relationship between the two text blocks, comprising: for each pair of text blocks connected by an edge in the roughly cut dense graph, obtaining the corresponding text content and text subgraph, fusing the shallow semantic features of the text content and the text subgraph to obtain a multi-modal semantic feature, then extracting the image high-dimensional semantic feature thereof, and further outputting the actual position relationship between the two text blocks, comprising: extracting the shallow semantic features of the text subgraph and the text content, fusing each shallow semantic feature to obtain a multi-modal semantic feature, comprising: ​ 4. The method of claim 3, wherein, ​ When the text subgraph is the text box image, image semantic information of the text subgraph is extracted by using shallow layer CNN convolution, and a ReShape operation is performed to obtain shallow layer semantic features of the text subgraph; When the text subgraph is the minimum bounding rectangle image, image semantic information of the text subgraph is extracted by using shallow layer CNN convolution, the image semantic information is pooled, and a ReShape operation is performed to obtain shallow layer semantic features of the text subgraph; The text content is converted into a high-dimensional vector by using a word vector model, context semantic information is extracted based on the high-dimensional vector, and a ReShape operation is performed to obtain shallow layer semantic features of the text content; The shallow layer semantic features of the text subgraph and the text content are fused to obtain multi-modal semantic features.

5. The method of claim 1, wherein, The actual position relationship is used to perform edge cutting on text blocks that are actually not adjacent, to perform merging on text blocks belonging to the same cell, and to adjust the rough edge cutting dense graph after the edge cutting and the merging to obtain a table structure dense graph, including: When it is determined according to the actual position relationship that the two text blocks of the edge are actually not adjacent, the edge is cut from the rough edge cutting dense graph; When it is determined according to the actual position relationship that the two text blocks of the edge belong to the same cell, the two text blocks of the edge are merged from the rough edge cutting dense graph; The position of the rough edge cutting dense graph after the edge cutting and the merging is adjusted to obtain a table structure dense graph.

6. The method of claim 1, wherein, The text blocks included in the table structure in the table image are detected, and a dense graph corresponding to the table structure is constructed according to the text blocks, including: Text detection is performed on the table structure in the table image to obtain text boxes and text content; The text boxes and the corresponding text content are combined to form text blocks; The text blocks are connected by edges two by two as graph nodes to obtain a dense graph.

7. The method of claim 3, wherein, Before the shallow layer semantic features of the text subgraph are extracted, the method further includes: The minimum side of the text subgraph is scaled to a preset size by equal ratio; When the maximum side of the text subgraph after the equal ratio scaling is greater than or equal to the product of the preset size and a threshold value, the text subgraph is centered and cropped; When the maximum side of the text subgraph after the equal ratio scaling is less than the product of the preset size and the threshold value, the text subgraph is supplemented with blank pixels.

8. An image table structure recognizing apparatus characterized by comprising: The method includes: A detection and construction module is configured to detect text blocks included in a table structure in a table image, and construct a dense graph corresponding to the table structure according to the text blocks; A rough edge cutting module is configured to combine the text blocks two by two to form text block groups, and identify text blocks outside the groups; by calculating the intersection area of the minimum bounding polygon of the text blocks inside the groups and the text box of the text blocks outside the groups, the adjacent nature of the text blocks inside the groups is determined according to a preset multiple condition, the text blocks that are not adjacent in the dense graph are cut, and a rough edge cutting dense graph is obtained; The prediction module is configured to, for each pair of text blocks connected by an edge in the rough edge-cutting dense graph, obtain corresponding text content and a text subgraph, fuse shallow semantic features of the text content and the text subgraph to obtain a multi-modal semantic feature, extract an image high-dimensional semantic feature, and then output an actual position relationship between the two text blocks. The fine edge-cutting module is configured to perform edge-cutting processing on text blocks that are actually not adjacent according to the actual position relationship, perform merging processing on text blocks belonging to the same cell, and then adjust the rough edge-cutting dense graph after the edge-cutting and merging processing to obtain a table structure dense graph. The table restoration module is configured to restore a table recognition result based on the table structure dense graph.

9. A computer device comprising a processor and a memory, the memory storing a computer program, characterized in that, The processor is configured to implement the image table structure recognition method in any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is configured to implement the image table structure recognition method in any one of claims 1-7 when executed by the processor.

Images