Table extraction method, device, computer equipment, and storage medium

Abstract

The present application relates to a table extraction method, device, computer equipment, storage medium and computer program product. The method comprises: obtaining a table image, inputting the table image into a table extraction model, wherein the table extraction model comprises a text extraction network block, a cell extraction network block and a structure extraction block, performing text extraction on the table image through the text extraction network block to obtain text information in cells in the table area of ​​the table image, performing cell extraction on the table image through the cell extraction network block to obtain cell information of cells in the table area of ​​the table image, generating a table structure corresponding to the table area according to the cell information of the cells through the structure extraction block, and merging the text information with the cells according to the table structure to obtain an electronic table. The present method can be used to perform end-to-end table data extraction on the table image using the table extraction model to generate a structured electronic table, thereby improving the versatility of the table extraction method.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a table extraction method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of computer technology, the demand for using computers to process and count tabular data has also greatly increased. For paper forms, the table data of the paper forms needs to be entered into the computer before the computer can perform subsequent processing. In order to improve the efficiency of entering table data, table extraction technology has emerged.

[0003] In traditional technology, a form template is customized in advance, the scanned results of a paper form are matched with multiple form templates, the form template corresponding to the paper form is determined, the scanned results of the paper form are parsed according to the determined form template, the table data of the paper form is extracted, and the table data is added to the corresponding form template to obtain an electronic form corresponding to the paper form.

[0004] However, paper forms have various formats and complex layouts. It is difficult to determine the form template corresponding to the paper form using the table extraction method in traditional technology, which leads to poor versatility of the table extraction method. Summary of the invention

[0005] Based on this, it is necessary to provide a table extraction method, device, computer equipment, storage medium and computer program product with high versatility to address the above technical problems.

[0006] In a first aspect, the present application provides a table extraction method. The method comprises:

[0007] Get the table image;

[0008] Inputting the table image into a table extraction model, wherein the table extraction model includes a text extraction network block, a cell extraction network block, and a structure extraction block;

[0009] Performing text extraction on the table image through the text extraction network block to obtain text information in cells in the table area of ​​the table image;

[0010] Extract cells from the table image by using the cell extraction network block to obtain cell information of cells in the table area of ​​the table image;

[0011] Generate a table structure corresponding to the table area according to the cell information of the cell by the structure extraction block;

[0012] According to the table structure, the text information is merged with the cells to obtain a spreadsheet.

[0013] In one of the embodiments, the table area includes a plurality of cells, and the cell information of each of the cells includes cell coordinates;

[0014] The step of generating a table structure corresponding to the table area according to the cell information of the cell by the structure extraction block includes:

[0015] The structure extraction block performs clustering processing on the multiple cells according to the cell coordinates of the multiple cells in the table area to determine multiple cell sets, and aligns the cells in each cell set according to the cell coordinates of the cells in each cell set to generate the table structure.

[0016] In one embodiment, the cell coordinates include a first cell coordinate, and the first cell coordinates include a first coordinate in a first direction and a second coordinate in a second direction;

[0017] The step of clustering the plurality of cells according to the cell coordinates of the plurality of cells in the table area by the structure extraction block to determine a plurality of cell sets, aligning the cells in each of the cell sets according to the cell coordinates of the cells in each of the cell sets to generate the table structure includes:

[0018] sorting the first coordinates of the plurality of cells in the table area by the structure extraction block, clustering the plurality of cells according to the sorted first coordinates, and determining a plurality of first cell sets in the first direction;

[0019] Generate a first reference coordinate corresponding to the first cell set according to the first coordinates of the cells in each of the first cell sets, and use the first reference coordinates to align the cells belonging to the first cell set in the first direction;

[0020] sorting the second coordinates of the plurality of cells in the table area by the structure extraction block, clustering the plurality of cells according to the sorted second coordinates, and determining a plurality of second cell sets in the second direction;

[0021] Generate a second reference coordinate corresponding to the second cell set according to the second coordinate of each cell in the second cell set, and use the second reference coordinate to align the cells belonging to the second cell set in the second direction;

[0022] According to the first reference coordinates, the second reference coordinates, and the cell coordinates corresponding to each of the cells after the alignment process, the structural information of each of the cells is determined to generate the table structure.

[0023] In one embodiment, the cell coordinates further include a second cell coordinate; the first cell coordinates and the second cell coordinates are coordinates at diagonally opposite corners of the cell;

[0024] The step of determining the structural information of each cell according to the first reference coordinate, the second reference coordinate, and the cell coordinates corresponding to each cell after the alignment process, and generating the table structure includes:

[0025] Generate a plurality of reference coordinates according to the first reference coordinates and the second reference coordinates;

[0026] Matching the first cell coordinate of each of the cells after the alignment process with the plurality of reference coordinates to determine a first structure reference coordinate corresponding to the first cell coordinate;

[0027] Matching the second cell coordinates of each of the cells after the alignment process with the plurality of reference coordinates to determine a second structure reference coordinate corresponding to the second cell coordinates;

[0028] The structural information of each cell is determined according to the first structural reference coordinates and the second structural reference coordinates corresponding to each cell after the alignment process, and the table structure corresponding to the table area is generated according to the structural information of each cell.

[0029] In one of the embodiments, generating a table structure corresponding to the table area according to the structural information of each cell further includes:

[0030] Divide the table image into regions according to the cell information of the cells after the alignment process by the structure extraction block, and determine the position information of the table region;

[0031] A table structure corresponding to the table area is generated according to the structure information of each cell and the position information of the table area.

[0032] In one embodiment, the text information includes text content and location information of the text content;

[0033] The step of merging the text information with the cells according to the table structure to obtain an electronic table includes:

[0034] Matching the position information of the text content with the cell information of the cell to determine the cell corresponding to the text content;

[0035] The text content is merged with the corresponding cells according to the table structure to obtain the electronic table.

[0036] In one embodiment, the table extraction model further includes an angle detection network block; the method further includes:

[0037] Performing text detection on the table image through the angle detection network block to obtain a plurality of text boxes and the coordinates of each of the text boxes;

[0038] Determining the angle of each text box according to the coordinates of each text box;

[0039] sorting the angles of the text boxes, dividing the plurality of text boxes according to the sorted angles of the text boxes to obtain a plurality of text box sets, and determining a target text box set from the plurality of text box sets according to the number of the text boxes in each of the text box sets;

[0040] According to the angles of the text boxes in the target text box set, the table image is angle-corrected to obtain the table image after angle correction;

[0041] The step of performing text extraction on the table image by using the text extraction network block to obtain text information in cells in a table area of ​​the table image, and performing cell extraction on the table image by using the cell extraction network block to obtain cell information of cells in the table area of ​​the table image includes:

[0042] Performing text extraction on the angle-corrected table image through the text extraction network block to obtain text information in cells in the table area of ​​the angle-corrected table image;

[0043] The cell extraction network block is used to extract cells from the angle-corrected table image to obtain cell information of the angle-corrected cells.

[0044] In one embodiment, the text detection is performed on the table image by the angle detection network block to obtain a plurality of text boxes and the coordinates of each of the text boxes, including:

[0045] Performing text detection on the table image through the angle detection network block to obtain a plurality of original text boxes and the coordinates of each of the original text boxes;

[0046] Determine the area of ​​each original text box according to the coordinates of each original text box;

[0047] The original text boxes whose areas are less than or equal to the area threshold are taken as the text boxes, and the coordinates of each of the text boxes are obtained.

[0048] In one embodiment, after determining the angle of the text box according to the coordinates of each text box, the method further comprises:

[0049] Determine the aspect ratio of each text box according to the coordinates of each text box;

[0050] Delete text boxes whose aspect ratio is less than the aspect ratio threshold.

[0051] In one embodiment, the table extraction model further includes a direction classification network block; the method further includes:

[0052] Determining the direction of the table image by the direction classification network block, and when the direction is an inverted direction, rotating the table image to obtain the rotated table image;

[0053] The step of performing text extraction on the table image by using the text extraction network block to obtain text information in cells in a table area of ​​the table image, and performing cell extraction on the table image by using the cell extraction network block to obtain cell information of cells in the table area of ​​the table image includes:

[0054] Performing text extraction on the rotated table image by the text extraction network block to obtain text information in cells in the table area of ​​the rotated table image;

[0055] The cell extraction network block performs cell extraction on the table image after the rotation processing to obtain the cell information of the cell after the rotation processing.

[0056] In a second aspect, the present application also provides a table extraction device. The device comprises:

[0057] A table image input module, used for acquiring a table image, and inputting the table image into a table extraction model, wherein the table extraction model includes a text extraction network block, a cell extraction network block and a structure extraction block;

[0058] A text information extraction module, used to extract text from the table image through the text extraction network block to obtain text information in cells in the table area of ​​the table image;

[0059] A cell extraction module, used for performing cell extraction on the table image through the cell extraction network block to obtain cell information of cells in the table area of ​​the table image;

[0060] A table structure generating module, used for generating a table structure corresponding to the table area according to the cell information of the cell through the structure extracting block;

[0061] The electronic table generation module is used to merge the text information with the cells according to the table structure to obtain an electronic table.

[0062] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the table extraction method described in any one of the embodiments of the first aspect is implemented.

[0063] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the table extraction method described in any one of the embodiments of the first aspect is implemented.

[0064] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the table extraction method described in any one of the embodiments of the first aspect is implemented.

[0065] The above table extraction method, device, computer equipment, storage medium and computer program product, by inputting the acquired table image into the table extraction model, respectively extracting text and extracting cells from the table image through the text extraction network block and the cell extraction network block in the table extraction model, and generating a table structure based on the cell information of the cell obtained by the cell extraction through the structure extraction block in the table extraction model, can fuse the text information in the cell obtained by the text extraction and the cell information of the cell obtained by the cell extraction according to the table structure through the table extraction model to generate a structured electronic table, without pre-customizing the table template, using the table extraction model to perform end-to-end table data extraction on the table image, thereby directly and quickly extracting structured table data from a table image with various layouts and unstructured, so that the table extraction method has high versatility. In addition, since the training data of the text extraction network block and the cell extraction network block in the table extraction model provided in the present application contain a large number of table images of different layouts, the table data obtained after the table image is subjected to table extraction by the table extraction model of the present application has high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A schematic diagram of a flow chart of a table extraction method in one embodiment;

[0067] Figure 2 A schematic diagram of a flow chart of a table structure generation step in one embodiment;

[0068] Figure 2a A schematic diagram of a cell before cell alignment in one embodiment;

[0069] Figure 2b A schematic diagram of cells after cell alignment in one embodiment;

[0070] Figure 3 A schematic diagram of a flow chart of a table structure generation step in one embodiment;

[0071] Figure 4 A schematic diagram of a table image after a table area is determined in one embodiment;

[0072] Figure 5 A schematic diagram of a process of performing angle correction on a table image in one embodiment;

[0073] Figure 5a is a schematic diagram of a step of determining a main direction of a text box in one embodiment;

[0074] Figure 6 It is a schematic diagram of a process of rotating a table image in one embodiment;

[0075] Figure 7 A schematic diagram of a flow chart of a table extraction method in one embodiment;

[0076] Figure 7a A schematic diagram of a process of an angle correction step of a table image in one embodiment;

[0077] Figure 7b is a schematic diagram of a training image with noise in one embodiment;

[0078] Figure 7c is a schematic diagram of a training image carrying a seal in one embodiment;

[0079] Figure 7d is a schematic diagram of training images with different background colors in one embodiment;

[0080] Figure 7e A schematic diagram of the experimental structure comparison of a cell detection model in one embodiment;

[0081] Figure 7f A schematic diagram of a flow chart of a table extraction method in one embodiment;

[0082] Figure 8is a structural block diagram of a table extraction device in one embodiment;

[0083] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0084] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0085] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0086] The table extraction method provided in the embodiment of the present application can be applied to a computer device. The computer device can be a terminal, or a server, or a system composed of a terminal and a server. A trained table extraction model is pre-deployed on the computer device. The computer device obtains a table image, and inputs the table image into the table extraction model, and the table extraction model includes a text extraction network block, a cell extraction network block, and a structure extraction block. The computer device performs text extraction on the table image through the text extraction network block to obtain the text information in the cells in the table area of ​​the table image. The computer device performs cell extraction on the table image through the cell extraction network block to obtain the cell information of the cells in the table area of ​​the table image. The computer device generates a table structure corresponding to the table area according to the cell information of the cells through the structure extraction block, and merges the text information with the cells according to the table structure to obtain a spreadsheet.

[0087] The terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. IoT devices may be smart TVs, smart car devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server may be implemented as an independent server or a server cluster consisting of multiple servers.

[0088] In one embodiment, Figure 1As shown, a table extraction method is provided, which is described by taking the method applied to a computer device as an example, and includes the following steps:

[0089] Step S102, obtaining a table image.

[0090] The table image may be an image obtained by photographing or scanning a paper table file by the user, or may be an image pre-stored in a database. The table image may be in any of a variety of file formats such as jpg (Joint Photo graphic Experts Group), png (Portable Network Graphic Format), pdf (Portable Document Format), tif (Tag Image File Format), etc.

[0091] Specifically, the computer device obtains the table image in response to the table extraction request. The table extraction request can be manually triggered by the user as needed, for example, the user clicks the corresponding table extraction button on the page to trigger the table extraction request; it can also be automatically triggered by the computer device, for example, when the computer device detects the existence of the table image, it automatically triggers the table extraction request, obtains the table image, and performs table extraction on the table image.

[0092] Step S104, inputting the table image into the table extraction model.

[0093] Among them, the table extraction model is an end-to-end model, including a text extraction network block, a cell extraction network block, and a structure extraction block.

[0094] Specifically, a trained table extraction model is pre-deployed in the computer device, and the trained table extraction model includes a text extraction network block, a cell extraction network block, and a structure extraction block. The computer device inputs the acquired table image into the table extraction model. In one example, when the computer device acquires multiple table images, the multiple table images can be split into multiple single table images, and each table image is input into the table extraction model in turn.

[0095] In one embodiment, a table extraction model is constructed by using a multi-GPU (Graphics Processing Unit) distributed reasoning method. By inputting a table image into a multi-GPU table extraction model to extract the table image, a high end-to-end recognition speed is achieved, while having a high recognition accuracy, and being able to meet high concurrency requirements and real-time response requirements.

[0096] Step S106, extracting text from the table image through a text extraction network block to obtain text information in cells in the table area of ​​the table image.

[0097] The cell can be used to represent the smallest unit in the table area. The table area includes at least one cell. The text information in the cell can include but is not limited to the text content in the cell and the position information of the text content.

[0098] Specifically, the computer device detects cells in the table image through the text extraction network block, determines the table area in the table image where cells exist, and extracts text from each cell in the table area. When text exists in the cell, the computer device obtains the text content in the cell and the location information of the text content through the text extraction network block.

[0099] In one example, the text extraction network block can be implemented using a deep learning-based text detection model, such as any of the text detection models such as the CTPN model (Connectionist Text Proposal Network), the RRPN model (Rotation Region Proposal Networks), and the RetinaNet model (a single-stage target detection model).

[0100] Step S108, extracting cells from the table image through the cell extraction network block to obtain cell information of cells in the table area of ​​the table image.

[0101] The cell information may include but is not limited to the coordinates of the cell, the width of the cell, and the height of the cell. In one example, the cell information may be expressed as (x, y, w, h), where x is the horizontal coordinate of the cell, y is the vertical coordinate of the cell, w is the width of the cell, and h is the height of the cell.

[0102] Specifically, the computer device performs cell detection on the table area through the cell extraction network block to determine the table area in the table image. The computer device performs the following operations on each cell in the table area through the cell extraction network block: detects and obtains the coordinates of the two vertices on the diagonal line of each cell, determines the cell coordinates, the width and the height of the cell according to the coordinates of the two vertices corresponding to the cell, and generates cell information corresponding to the cell.

[0103] In one example, a computer device performs cell detection on a table image through a cell extraction network block to obtain the upper left vertex coordinates (1,3) and the lower right vertex coordinates (2,4) of the cell, and determines that the width of the cell is 1 and the height of the cell is 1 based on the upper left vertex coordinates and the lower right vertex coordinates of the cell, and uses the upper left vertex coordinates as the cell coordinates (1,3) of the cell to generate cell information corresponding to the cell (1,3,1,1).

[0104] In one example, the cell extraction block can be implemented using a deep learning-based target detection model, such as the YOLO model (You Only Look Once, a single-stage target detection model), Faster R-CNN (Faster Region-Convolutional Neural Networks, fast regional convolutional neural network model), FSAF model (Feature Selective Anchor-Free, anchor-free feature selection model), etc.

[0105] Step S110 , generating a table structure corresponding to the table area according to the cell information of the cell through the structure extraction block.

[0106] The table structure may include, but is not limited to, a first parameter in a first direction and a second parameter in a second direction corresponding to each cell.

[0107] The first direction may be any one of a vertical direction and a horizontal direction, and the first parameter may be used to characterize the position information of the cell in the first direction. For example, when the first direction is a horizontal direction, the first parameter is the position information of the cell in the horizontal direction.

[0108] The second direction may be a direction perpendicular to the first direction, and the second parameter may be used to characterize the position information of the cell in the second direction. For example, when the first direction is a vertical direction, the second direction may be a horizontal direction, and the second parameter is the position information of the cell in the horizontal direction.

[0109] Specifically, the computer device performs clustering processing on multiple cells according to the cell information of the cells through the structure extraction block to obtain multiple cell sets. The cell coordinates of each cell in the cell set are calculated to obtain the first parameter in the first direction and the second parameter in the second direction corresponding to each cell. Through the structure extraction block, the cells are aligned in the first direction according to the first parameter corresponding to each cell, and the cells are aligned in the second direction according to the second parameter corresponding to each cell, to obtain the cell information of each cell after the alignment processing. Through the structure extraction block, the table image is divided into regions according to the cell information of the aligned cells, the position information of the table area and the position information of the non-table area in the table image are determined, the structure information of each cell is determined according to the first parameter in the first direction and the second parameter in the second direction corresponding to each cell after the alignment processing, and the table structure corresponding to the table area is generated according to the structure information of each cell and the position information of the table area.

[0110] Step S112, merging the text information with the cells according to the table structure to obtain a spreadsheet.

[0111] The electronic spreadsheet is used to represent a table that can be recognized by a computer device and perform data processing. In one example, the electronic spreadsheet can be a visual table that is used to display table data and perform calculations on the table data.

[0112] Specifically, the computer device matches the position information of the text content with the coordinates of the cell through the table extraction model, determines the cell corresponding to the text content, adds the text content to the corresponding cell, and obtains the electronic table. Further, after the computer device obtains the electronic table, the computer device can analyze and process the data in the electronic table according to the obtained electronic table.

[0113] The above table extraction method, by inputting the acquired table image into the table extraction model, respectively extracting text and extracting cells from the table image through the text extraction network block and the cell extraction network block in the table extraction model, and generating a table structure based on the cell information of the cell obtained by the cell extraction through the structure extraction block in the table extraction model, can fuse the text information in the cell obtained by the text extraction and the cell information of the cell obtained by the cell extraction according to the table structure through the table extraction model to generate a structured electronic table, without pre-customizing the table template, and using the table extraction model to perform end-to-end table data extraction on the table image, thereby directly and quickly extracting structured table data from a table image with various layouts and unstructured, so that the table extraction method has high versatility. In addition, since the training data of the text extraction network block and the cell extraction network block in the table extraction model provided in the present application contain a large number of table images of different layouts, the table data obtained after the table image is subjected to table extraction by the table extraction model of the present application has high accuracy.

[0114] In one embodiment, the table area includes a plurality of cells, and the cell information of each cell includes cell coordinates. Step S110, generating a table structure corresponding to the table area according to the cell information of the cell by the structure extraction block, including: clustering the plurality of cells according to the cell coordinates of the plurality of cells in the table area by the structure extraction block, determining a plurality of cell sets, aligning the cells in each cell set according to the cell coordinates of the cells in each cell set, and generating a table structure.

[0115] Specifically, the computer device performs calculation processing on the cell coordinates of each cell in the table area through the structure extraction block to determine the coordinate difference of the cell coordinates between the multiple cells. The structure extraction block performs clustering processing on the multiple cells according to the coordinate difference between the multiple cells to obtain multiple cell sets. The structure extraction block performs calculation processing on the cell coordinates of each cell in the cell set to determine the coordinate parameters corresponding to the cell set, and the coordinate parameters corresponding to the cell set are used to update the cell coordinates of each cell in the cell set, align each cell in the table area, and generate a table structure.

[0116] In this embodiment, a plurality of cells in a table area are clustered by a structure extraction block to determine a plurality of cell sets, and the cells in each cell set are aligned according to the cell coordinates of the cells in each cell set to generate a table structure. The cells obtained by the cell extraction network block can be aligned to generate a corresponding table structure, thereby improving the accuracy of the generated table structure and thus improving the accuracy of the subsequent fusion of text information with cells according to the table structure.

[0117] In one embodiment, the cell coordinates include a first cell coordinate, the first cell coordinates include a first coordinate in a first direction, and a second coordinate in a second direction. Figure 2 As shown, a structure extraction block is used to cluster multiple cells according to the cell coordinates of multiple cells in the table area to determine multiple cell sets, and the cells in each cell set are aligned according to the cell coordinates of the cells in each cell set to generate a table structure, including:

[0118] Step S202: sorting the first coordinates of a plurality of cells in the table area through the structure extraction block, clustering the plurality of cells according to the sorted first coordinates, and determining a plurality of first cell sets in a first direction.

[0119] The first cell coordinates may be used to represent the coordinates of any vertex of the cell. In one example, the first cell coordinates of the cell may be the coordinates of the vertex at the upper left corner of the cell.

[0120] The first coordinate can be used to represent the coordinate of the first cell coordinate in a first direction. In one example, the first direction is a horizontal direction. The first coordinate is the coordinate of the first cell coordinate in the horizontal direction.

[0121] Specifically, the computer device extracts the first coordinate corresponding to each cell from the first cell coordinate of each cell through the structure extraction block. The first coordinates of multiple cells are sorted through the structure extraction block, and starting from the first cell after sorting, the nth cell (n is a positive integer) whose difference with the first coordinate of the first cell meets the coordinate difference threshold is determined, and the first cell to the nth cell are clustered to obtain a first cell set in the first direction. The n+1th cell is clustered according to the above content until the clustering of the sorted multiple cells is completed.

[0122] Step S204: generating first reference coordinates corresponding to the first cell set according to the first coordinates of the cells in each first cell set, and aligning the cells belonging to the first cell set in the first direction using the first reference coordinates.

[0123] Specifically, the computer device operates the first coordinates of multiple cells in each first cell set through the structure extraction block to generate a first reference coordinate corresponding to each first cell set. The structure extraction block updates the first coordinates of the cells in each first cell set to the first reference coordinate, so that the multiple cells are aligned in the first direction.

[0124] In one example, the computer device can obtain the average value of the first coordinates of multiple cells in the first cell set through the structure extraction block, and use the average value of the first coordinates as the first reference coordinate. In another example, the median of the first coordinates of multiple cells in the first cell set can be obtained, and the median of the first coordinates can be used as the first reference coordinate. In another example, the mode value of the first coordinates of multiple cells in the first cell set can be obtained, and the mode value of the first coordinates can be used as the first reference coordinate.

[0125] Step S206, sorting the second coordinates of the plurality of cells in the table area by the structure extraction block, clustering the plurality of cells according to the sorted second coordinates, and determining a plurality of second cell sets in the second direction.

[0126] The second coordinate can be used to represent the coordinate of the first cell coordinate in the second direction. In one example, if the first direction is a horizontal direction, the second direction is a vertical direction, and the second coordinate is the coordinate of the first cell coordinate in the vertical direction; or, if the first direction is a vertical direction, the second direction is a horizontal direction, and the second coordinate is the coordinate of the first cell coordinate in the horizontal direction.

[0127] Specifically, the computer device extracts the second coordinates corresponding to each cell from the first cell coordinates of each cell through the structure extraction block. The second coordinates of multiple cells in the table area are sorted, and starting from the first sorted cell, the mth cell (m is a positive integer) whose difference with the second coordinate of the first cell meets the coordinate difference threshold is determined, and the first cell to the mth cell are clustered to obtain a second cell set in the second direction. The m+1th cell is clustered according to the above content until the clustering of the sorted multiple cells is completed.

[0128] Step S208, generating second reference coordinates corresponding to the second cell set according to the second coordinates of the cells in each second cell set, and aligning the cells belonging to the second cell set in the second direction using the second reference coordinates.

[0129] Specifically, the computer device operates the second coordinates of the multiple cells in each second cell set through the structure extraction block to generate a second reference coordinate corresponding to each second cell set. The second coordinates of the cells in each second cell set are updated to the second reference coordinates through the structure extraction block, so that the multiple cells are aligned in the first direction.

[0130] Figure 2a The cells before the alignment process are shown as an example, referring to Figure 2a It can be seen that the cell information of the cell obtained by the cell extraction network block is biased, resulting in a deviation in the position of the cell. Figure 2b The example shows the steps S202 to S208. Figure 2a The cells in the table are aligned to obtain the cells. Figure 2b It can be seen that the cell information of the cell is more accurate after the alignment process, which conforms to the rules of the structured table.

[0131] Step S210, determining the structural information of each cell according to the first reference coordinates, the second reference coordinates, and the cell coordinates corresponding to each cell after the alignment process, and generating a table structure.

[0132] The structural information of the cell may include, but is not limited to, first position information in a first direction and second position information in a second direction.

[0133] Specifically, the computer device uses a structure extraction block to sort multiple first reference coordinates from small to large, generates an identifier corresponding to each first reference coordinate in the order of the sorted first reference coordinates, sorts multiple second reference coordinates from small to large, and generates an identifier corresponding to each second reference coordinate in the order of the sorted second reference coordinates. Determine the first distance between each first reference coordinate and the first coordinate of each cell after the alignment process, obtain the target first reference coordinate whose first distance to the cell is less than the coordinate distance threshold, and use the identifier corresponding to the target first reference coordinate as the first position information of the cell in the first direction. Determine the second distance between each second reference coordinate and the second coordinate of each cell after the alignment process, obtain the target second reference coordinate whose second distance to the cell is less than the coordinate distance threshold, and use the identifier corresponding to the target second reference coordinate as the second position information of the cell in the second direction.

[0134] In this embodiment, by aligning cells in the first direction and the second direction and generating a table structure based on the coordinates of the aligned cells, the accuracy of the generated table structure can be improved, and it also helps to improve the accuracy of subsequent fusion of text information and cells.

[0135] In one embodiment, the cell coordinates further include a second cell coordinate, and the first cell coordinate and the second cell coordinate are coordinates at diagonally opposite corners of the cell. Figure 3 As shown, step S210, according to the first reference coordinates, the second reference coordinates, and the cell coordinates corresponding to each cell after the alignment process, determines the structural information of each cell and generates a table structure, and also includes:

[0136] Step S302: Generate multiple reference coordinates according to the first reference coordinate and the second reference coordinate.

[0137] Specifically, the computer device performs the following operations on each first reference coordinate through the structure extraction block to generate multiple reference coordinates: the first reference coordinate is combined with each second reference coordinate to generate multiple reference coordinates corresponding to the first reference coordinate. The multiple reference coordinates are sorted based on the first reference coordinate of each reference coordinate to determine the first order of the sorted reference coordinates; the multiple reference coordinates are sorted based on the second reference coordinate of each reference coordinate to determine the second order of the sorted reference coordinates, and the reference coordinate identifier is generated according to the first order and the second order of the reference coordinates.

[0138] In an example, taking the first reference coordinate of each reference coordinate as a reference, multiple reference coordinates are sorted from small to large, and the first order of the reference coordinates is 2; taking the second reference coordinate of each reference coordinate as a reference, multiple reference coordinates are sorted from small to large, and the second order of the reference coordinates is 3, then the generated reference coordinates are identified as (2,3).

[0139] Step S304 , matching the first cell coordinate of each cell after the alignment process with a plurality of reference coordinates, and determining a first structure reference coordinate corresponding to the first cell coordinate.

[0140] The first structure reference coordinates may be used to represent reference coordinates whose distance from the first cell coordinates is less than or equal to a distance threshold.

[0141] Specifically, the computer device matches the first cell coordinate of each cell after the alignment process with multiple reference coordinates through the structure extraction block, obtains the distance between the first cell coordinate of each cell after the alignment process and each reference coordinate, determines the first target reference coordinate whose distance to each first cell coordinate is less than or equal to the distance threshold, and uses the identifier of the first target reference coordinate as the first structure reference coordinate corresponding to each cell.

[0142] Step S306: Match the second cell coordinates of each cell after the alignment process with a plurality of reference coordinates to determine a second structure reference coordinate corresponding to the second cell coordinates.

[0143] The second structure reference coordinates may be used to represent reference coordinates whose distance from the second cell coordinates is less than or equal to a distance threshold.

[0144] Specifically, the computer device matches the second cell coordinates of each cell after the alignment process with multiple reference coordinates through the structure extraction block, obtains the distance between the second cell coordinates of each cell after the alignment process and each reference coordinate, determines a second target reference coordinate whose distance to each second cell coordinate is less than or equal to a distance threshold, and uses the identifier of the second target reference coordinate as the second structure reference coordinate corresponding to each cell.

[0145] Step S308, determining the structural information of each cell according to the first structural reference coordinates and the second structural reference coordinates corresponding to each cell after the alignment process, and generating a table structure corresponding to the table area according to the structural information of each cell.

[0146] Specifically, the computer device performs calculation processing on the first structure reference coordinate and the second structure reference coordinate corresponding to each cell through the structure extraction block to determine the first starting value and the first ending value of each cell in the first direction, and the second starting value and the second ending value of each cell in the second direction. According to the first starting value and the first ending value corresponding to the cell, the position information of the cell in the first direction is determined. According to the second starting value and the second ending value corresponding to the cell, the position information of the cell in the second direction is determined. According to the position information of each cell in the first direction and the position information in the second direction, a table structure corresponding to the table area is generated.

[0147] In one example, the first cell coordinates may be the coordinates of the upper left vertex of the cell, and the second cell coordinates may be the coordinates of the lower right vertex of the cell. The first direction is the horizontal direction, and the second direction is the vertical direction. The computer device determines through the structure extraction block that the first structure reference coordinate corresponding to the upper left vertex of the cell is identified as (0,0), and the second structure reference coordinate corresponding to the lower right vertex coordinate is identified as (2,3), and determines that the first starting value of the cell in the horizontal direction is 0, and the first ending value is 2, then the position information of the cell in the horizontal direction is: first column to second column; determine that the second starting value of the cell in the vertical direction is 0, and the second ending value is 3, then the position information of the cell in the vertical direction is: first row to third row.

[0148] In this embodiment, the first cell coordinates are matched with the reference coordinates through the structure extraction block to determine the first structure reference coordinates of the cell; the second cell is matched with the reference coordinates to determine the second structure reference coordinates of the cell, and the structural information of the cell is determined based on the first structure reference coordinates and the second structure reference coordinates corresponding to the cell, and a table structure corresponding to the table area is generated, which can improve the efficiency of the computer device in determining the cell structure information through the structure extraction block.

[0149] In one embodiment, step S308 determines the structural information of each cell according to the first structural reference coordinates and the second structural reference coordinates corresponding to each cell after the alignment process, and generates a table structure corresponding to the table area according to the structural information of each cell, including: dividing the table image into regions according to the cell information of the cells after the alignment process by a structure extraction block to determine the position information of the table area; generating a table structure corresponding to the table area according to the structural information of each cell and the position information of the table area.

[0150] The position information of the table area may be used to represent the position information of the outer boundary line of the table area.

[0151] Specifically, the computer device sets the gradient value of each pixel in the area where the cell is located to a first threshold through the structure extraction block, and sets the gradient value of each pixel in the non-table area of ​​the table image except the table area to a second threshold. The gradient value of each pixel in the table image is traversed in sequence, and the coordinates of the pixel points with different gradient values ​​in the first direction and the coordinates of the pixel points with different gradient values ​​in the second direction are determined. According to the coordinates of the determined multiple pixels, the coordinates of the table area in the table image are generated to obtain the position information of the table area. According to the structure information of each cell and the position information of the table area, a table structure corresponding to the table area is generated.

[0152] In one example, the first threshold may be 1, and the second threshold may be 0. The computer device determines, through the structure extraction block, the pixel points at which the gradient value changes from 1 to 0 or from 0 to 1 in the horizontal direction, and the pixel points at which the gradient value changes from 1 to 0 or from 0 to 1 in the vertical direction, and determines the position information of the table area according to the determined pixel points.

[0153] In one example, after the computer device determines the table area in the table image through the structure extraction block, the computer device can merge multiple parallel non-table areas to determine the position information of each independent non-table area after the merger. Figure 4 As shown, a table image is provided after determining the table area and the non-table area.

[0154] In this embodiment, the location information of the table area is determined through the structure extraction block, and a table structure corresponding to the table area is generated. The table image can be divided into areas, and the location information of the table area and the location information of the non-table area can be determined, thereby improving the efficiency of subsequent computer equipment in fusing text information and cells according to the location information of the table area.

[0155] In one embodiment, the text information includes text content and location information of the text content. Step S112, according to the table structure, merges the text information with the cells to obtain a spreadsheet, including: matching the location information of the text content with the cell information of the cells to determine the cells corresponding to the text content; and merging the text content with the corresponding cells according to the table structure to obtain a spreadsheet.

[0156] Specifically, the computer device matches the position information of the text content with the cell information of the cell through the table extraction model, determines the difference between the position information of the text content and the cell, and takes the cell whose difference with the position information of the text content is less than a preset information difference threshold as the cell corresponding to the text content. The text content and the corresponding cell are merged according to the table structure to obtain an electronic table.

[0157] In this embodiment, the cell corresponding to the text content is determined by matching the position information of the text content with the cell information of the cell; the text content and the corresponding cell are merged according to the table structure to obtain an electronic table, which can improve the efficiency of generating the electronic table.

[0158] In one embodiment, the table extraction model also includes an angle detection network block, such as Figure 5 As shown, the table extraction method further includes:

[0159] Step S502, performing text detection on the table image through the angle detection network block to obtain multiple text boxes and the coordinates of each text box.

[0160] Step S504, determining the angle of each text box according to the coordinates of each text box.

[0161] The angle detection network block is used to detect the angle of text in the table image and determine the tilt angle of the table image according to the angle of the text. In one example, the angle detection network block can be implemented using a text detection model based on deep learning, such as any of the PSENet model (Progressive Scale Expansion Network), RRPN model, DMPNet model (Deep Matching Prior Network), and other text detection models.

[0162] Specifically, after the table image is input into the table extraction model, the table image is firstly subjected to text detection through the angle detection network block to obtain the text box corresponding to each text content in the table image and the vertex coordinates of each text box. According to the vertex coordinates of each text box, the length of each text box in the first text direction and the angle between the first text direction and the vertical direction, as well as the length in the second text direction and the angle between the second text direction and the vertical direction are determined. The length of the text box in the first text direction and the length in the second text direction are compared. When the length of the text box in the first text direction is greater than the length of the text box in the second text direction, the angle between the first text direction and the vertical direction of the text box is taken as the angle of the text box; when the length of the text box in the first text direction is less than the length of the text box in the second text direction, the angle between the second text direction and the vertical direction of the text box is taken as the angle of the text box.

[0163] In one example, the coordinates output by the angle detection network block include the vertex coordinates of the four vertices of the text box, and the midpoint of each side of each text box is calculated based on the vertex coordinates of the four vertices of each text box. Connect the midpoints of the two sides of the text box in the first text direction to obtain a first line segment parallel to the second text direction, obtain the length of the first line segment, and the angle between the first line segment and the vertical direction, and use the length of the first line segment as the length of the text box in the second text direction, and use the angle between the first line segment and the vertical direction as the angle of the text box in the second text direction. Connect the midpoints of the two sides of the text box in the second text direction to obtain a second line segment parallel to the first text direction, obtain the length of the second line segment, and the angle between the second line segment and the vertical direction, and use the length of the second line segment as the length of the text box in the first text direction, and use the angle between the second line segment and the vertical direction as the angle of the text box in the first text direction.

[0164] Step S506, sorting the angles of the text boxes, dividing the multiple text boxes according to the sorted angles of the text boxes to obtain multiple text box sets, and determining a target text box set from the multiple text box sets according to the number of text boxes in each text box set.

[0165] Specifically, the computer device uses the angle detection network block to sort multiple text boxes in the order of the angles of the text boxes from small to large. Starting from the first text box after sorting, determine the i-th text box (i is a positive integer) whose angle difference with the first text box meets the angle difference threshold, and cluster the first text box to the i-th text box to obtain a text box set. Continue to cluster the i+1th text box according to the above content until the clustering of the sorted multiple text boxes is completed. Determine the number of text boxes in each text box set, and use the text box set with the largest number of text boxes as the target text box set.

[0166] Step S508, performing angle correction on the table image according to the angles of the text boxes in the target text box set to obtain an angle-corrected table image.

[0167] The angle correction is used to rotate the table image according to the tilt angle of the table image to obtain a table image with a tilt angle of zero with respect to the horizontal direction / vertical direction.

[0168] Specifically, the computer device calculates the average angle of the text boxes in the target text box set through the angle detection network block, and uses the average angle of the text boxes as the angle in the main direction of the text boxes. The table image is angle-corrected according to the angle in the main direction of the text boxes, and the angle in the main direction of the text boxes is set to zero, so as to obtain a table image with a zero tilt angle with respect to the vertical direction. In an example, Figure 5a , which shows a schematic diagram of determining the main direction of a text box according to the angle of each text box in the target text box set.

[0169] Step S510, performing text extraction on the angle-corrected table image through a text extraction network block to obtain text information in cells in the table area of ​​the angle-corrected table image.

[0170] Step S512, performing cell extraction on the angle-corrected table image through a cell extraction network block to obtain cell information of cells in the table area of ​​the angle-corrected table image.

[0171] Specifically, the computer device performs text extraction on the angle-corrected table image through the text extraction network block to obtain text information in the cells in the table area of ​​the angle-corrected table image. The computer device performs cell extraction on the angle-corrected table image through the cell extraction network block to obtain cell information of the cells in the table area of ​​the angle-corrected table image. The specific implementation methods of text extraction and cell extraction can refer to the above embodiments and will not be elaborated here.

[0172] In this embodiment, since the table image may have a certain tilt angle, it is easy to cause certain errors in text extraction and cell extraction. Therefore, the angle detection network block is used to correct the angle of any table image with a non-zero tilt angle, which helps to improve the accuracy of the obtained text information and cell information.

[0173] In one embodiment, step S502, performing text detection on the table image through the angle detection network block to obtain multiple text boxes and the coordinates of each text box, including: performing text detection on the table image through the angle detection network block to obtain multiple original text boxes and the coordinates of each original text box; determining the area of ​​each original text box according to the coordinates of each original text box; taking the original text box whose area is less than or equal to the area threshold as the text box, and obtaining the coordinates of each text box.

[0174] Specifically, the computer device stores an area threshold in advance. The computer device performs text detection on the table image through the angle detection network block to obtain multiple original text boxes and the coordinates of each original text box. According to the coordinates of each original text box, the length of each original text box in the first text direction and the length of each original text box in the second text direction are determined, and the length of each original text box in the first text direction and the length in the second text direction are processed to determine the area of ​​each original text box. Delete the original text boxes whose areas are greater than the area threshold, take the original text boxes whose areas are less than or equal to the area threshold as text boxes, and obtain the coordinates of the text boxes. The specific implementation method of the process of determining the length of the original text box in the first text direction and the process of determining the length of the original text box in the second text direction can refer to the above embodiments, which will not be elaborated here.

[0175] In this embodiment, since there are multiple lines of text in the text box, it is easy for the angle of the text box determined by the angle detection network block to have errors. Therefore, by treating the original text box with an area less than or equal to the area threshold as a text box for subsequent processing, the situation where there are multiple lines of text in a text box can be avoided, thereby improving the accuracy of the text box angle determined by the angle detection model.

[0176] In one embodiment, after determining the angle of each text box according to the coordinates of each text box in step S504, the method further includes: determining the aspect ratio of the text box according to the coordinates of each text box; and deleting text boxes whose aspect ratio is less than an aspect ratio threshold.

[0177] Specifically, the computer device performs calculation processing on the length of each text box in the first direction and the length of the text box in the second direction through the angle detection network block to determine the aspect ratio of each text box. According to the aspect ratio of each text box, the average aspect ratio of the text box is determined, and the average aspect ratio of the text box is used as the aspect ratio threshold, and the text box with an aspect ratio less than the aspect ratio threshold is deleted.

[0178] In this embodiment, since short text boxes with an aspect ratio smaller than the average aspect ratio may have a negative impact on the angle correction process of the table image, the accuracy of angle correction of the table image through the angle detection model can be improved by deleting text boxes with an aspect ratio smaller than the aspect ratio threshold.

[0179] In one embodiment, the table extraction model also includes a direction classification network block, such as Figure 6 As shown, the table extraction method also includes:

[0180] Step S602, determine the direction of the table image through the direction classification network block, and when the direction is an inverted direction, rotate the table image to obtain a rotated table image.

[0181] Among them, the direction classification network block can be a multi-classification network for detecting the direction category of the table image. In one example, the direction classification network is a binary classification network, which has the ability to recognize a table image of 0° (i.e., the direction of the table image is in an upright direction) and a table image of 180° (i.e., the direction of the table image is in an inverted direction). In one example, the direction classification network block can be implemented using a classification model based on deep learning, such as any of the image classification models such as the Inception model (a sparse network model), the DBN model (Deep Belief Network), and the ResNet model (Residual Neural Network).

[0182] Specifically, the computer device identifies the header position of the table area in the table image through the direction classification network block, and determines the direction classification of the table image according to the header position. When the computer device determines that the header position in the table image is in an upright direction through the direction classification network block, the direction of the table image is an upright direction, and the table image is not rotated; when the computer device determines that the header position in the table image is in an inverted direction through the direction classification network block, the direction of the table image is an inverted direction, and the table image is flipped 180 degrees to obtain a rotated table image.

[0183] Step S604, performing text extraction on the rotated table image through a text extraction network block to obtain text information in cells in the table area of ​​the rotated table image.

[0184] Step S606, extracting cells from the rotated table image through a cell extraction network block to obtain cell information of cells in the table area of ​​the rotated table image.

[0185] Specifically, the computer device performs text extraction on the rotated table image through the text extraction network block to obtain text information in the cells in the table area of ​​the rotated table image. The computer device performs cell extraction on the rotated table image through the cell extraction network block to obtain cell information of the cells in the table area of ​​the rotated table image. The specific implementation method of performing text extraction and cell extraction on the rotated table image can be implemented by referring to the method provided in the above embodiment, which will not be elaborated here.

[0186] In this embodiment, due to the direction of the table image, there may be two directions: upright and inverted. When the direction of the table image is inverted, it is easy to cause certain errors in the table data obtained after the table image is processed by the text extraction network and the cell extraction network block. Therefore, the direction of the table image is determined by the direction classification network block, and the table image is rotated according to the direction of the table image to obtain the rotated table image. The direction of the table image can be set to an upright direction, thereby improving the accuracy of the text information and cell information obtained after the subsequent text extraction network block and the cell extraction network block process the rotated table image.

[0187] In one embodiment, Figure 7 As shown, a table extraction method is provided, including:

[0188] Step S702, obtaining a table image, and inputting the table image into a table extraction model.

[0189] Specifically, the computer device obtains a table image, and when there are multiple table images, each table image is input into the table extraction model in turn.

[0190] Step S704, performing text detection on the table image through the angle detection network block, determining the angle of the text box, and performing angle correction on the table image according to the angle of the text box to obtain the table image after angle correction.

[0191] Specifically, the computer device can use an angle detection network block based on the PSENet model to perform text detection on the table image, determine multiple original text boxes in the table image and the coordinates of each original text box. Perform calculation processing on the coordinates of each original text box to obtain the first length of each original text box in the first text direction and the second length of the text box on the second text box. Multiply the first length by the second length to obtain the area of ​​each original text box, delete the original text box with an area greater than the area threshold, and use the original text box with an area less than or equal to the area threshold as the text box to obtain the coordinates of the text box. Determine the angle of the text box and the aspect ratio of the text box according to the coordinates of the text box, and delete the text box with an aspect ratio less than the average aspect ratio. Sort the angles of the deleted text boxes, divide the multiple text boxes according to the angles of the sorted text boxes, obtain multiple text box sets, and the number of text boxes in each text box set, and use the text box set with the largest number of text boxes as the target text box set. The angle of each text box in the target text box set is processed to determine the average value of the text box angles, and the table image is angle-corrected according to the average value of the text box angles to obtain the table image after angle correction. The specific implementation of the text box angle determination process can be implemented with reference to the method provided in the above embodiment, and will not be elaborated in detail here. In an example, Figure 7a As shown, a schematic diagram of the process of performing angle correction on a table image through an angle detection network block based on a PSENet model is provided.

[0192] Step S706, determining the direction of the table image after angle correction through the direction classification network block, and when the direction of the table image is an inverted direction, rotating the table image to obtain a rotated table image.

[0193] Specifically, the computer device can use a direction classification network block based on the ResNet model to determine the direction of the table image after angle correction. When the direction of the table image after angle correction is inverted, the table image is rotated to obtain a rotated table image. The specific implementation method of rotating the table image after angle correction can be implemented with reference to the method provided in the above embodiment, which is not elaborated in detail here. In one example, the direction classification network block based on the ResNet model can use ResNet18 as the backbone network, use several forward table images and several reverse table images as training data, train the untrained ResNet model, and obtain a trained ResNet model.

[0194] Step S708, performing text extraction on the rotated table image through a text extraction network block to obtain text information in cells in the table area of ​​the table image.

[0195] Specifically, the computer device can use a text extraction network block based on the RetinaNet model to extract text from the rotated table image to obtain text information in cells in the table area of ​​the table image. The specific implementation of text extraction can be implemented with reference to the method provided in the above embodiment, and will not be elaborated here.

[0196] In one example, by repeatedly using the anchor-based RetinaNet model to extract text information, it is found that the anchor-based RetinaNet model has better text information extraction effect on long fields by comparing the text information extraction results of the RetinaNet model. Therefore, the RetinaNet model is used as the text extraction network block to extract text from the rotated table image. The following is an exemplary explanation of the training method of the RetinaNet model: First, ResNet50 is used as the backbone network to generate an untrained RetinaNet model, and several table images with text boxes in cells are input into the untrained RetinaNet model as training data, the aspect ratio of the training data is retained, and the size of the training data is adjusted to 1024*1024. Set the anchor value of the RetinaNet model, use the P2-P5 layers of FPN to train the untrained RetinaNet model, use the clustering kmeans tool to identify the text boxes in the training data, and adjust the anchor value according to the recognition results of the text boxes until the text boxes recognized by the RetinaNet model can cover most of the text boxes, thus obtaining the trained RetinaNet model.

[0197] Step S710, performing cell extraction on the rotated table image through a cell extraction network block to obtain cell information of cells in the table area of ​​the table image.

[0198] Specifically, the computer device can use the cell extraction block based on the FSAF model to extract cells from the rotated table image, determine the coordinates of each cell, and obtain the cell information of the cell in the table area of ​​the table image according to the coordinates of each cell. The specific implementation of cell extraction can be implemented with reference to the method provided in the above embodiment, which will not be elaborated in detail here.

[0199] In one example, the cell extraction block can use table images of different formats, different background colors, and different sizes as training data for training, including special formats, such as dotted table lines in the table area, table lines that are not closed on both sides in the table area, stamps, noise interference, dense tables, table binarization, black stains, etc. Figure 7b As shown in , a training data with noise is provided. Figure 7cAs shown in , a training data with a seal is provided. Figure 7d As shown, training data with different background colors is provided.

[0200] In one example, both the FSAF model and the RetinaNet model can be used for cell detection. Figure 7e As shown in the figure, the FSAF model and the RetinaNet model are used to detect cells in table images. According to the detection results obtained from multiple experiments, it is found that the cells recognized by the FSAF model have fewer missing frames than those recognized by the RetinaNet model, and the cells recognized by the FSAF model are neater than those recognized by the RetinaNet model. Therefore, the FSAF model is used as the cell extraction network block to extract cells from table images.

[0201] Step S712: clustering the cells according to the cell information of the cells through the structure extraction block to obtain multiple cell sets, and aligning the cells in each cell set.

[0202] Step S714, the table image is divided into regions according to the cell information of the aligned cells by the structure extraction block, and the position information of the table region is determined.

[0203] Step S716, the structure extraction block determines the structure information of each cell according to the cell coordinates of the aligned cells, and generates a table structure corresponding to the table area according to the structure information of each cell after the alignment and the position information of the table area.

[0204] Step S718, according to the table structure, merge the text information with the cells to obtain a spreadsheet.

[0205] Specifically, the computer device clusters the cells according to the cell information of the cells through the structure extraction block to obtain multiple cell sets, determine the reference coordinates of each cell set, and align the cells in each cell set according to the reference coordinates of each cell set. Determine the position information of the table area according to the cell information of the cells after the alignment process. Determine the structural reference coordinates corresponding to each cell according to the cell information of the cells after the alignment process, determine the structural information of each cell, and generate a table structure corresponding to the table area according to the structural information of each cell after the alignment process and the position information of the table area. The computer device merges the text information with the cells according to the table structure through the table extraction model to obtain an electronic table. The specific implementation methods of table structure generation and electronic table generation can be implemented with reference to the methods provided in the above embodiments, which will not be elaborated in detail here.

[0206] In an example, Figure 7f As shown, a schematic diagram of a process of extracting a table from a table image through a table extraction model is provided.

[0207] In this embodiment, by inputting a table image into a table extraction model, using an angle detection network block to perform angle correction on the table image, using a direction classification network block to rotate the angle-corrected table image, using a text extraction network block to perform text extraction on the rotated table image, using a cell extraction network block to perform cell extraction on the rotated table image, using a structure extraction block to process the cells in the table image, generating a table structure corresponding to the table area, and merging the text information with the cells according to the table structure to obtain an electronic table. The table extraction model can be used to implement an end-to-end table recognition process, avoiding interference with cell extraction caused by factors such as seals, noise, and background colors, and extracting structured table data from unstructured table images with various formats, thereby enhancing the robustness of the table image and improving the versatility of the table extraction method. In addition, after the table image is subjected to table extraction by the table extraction model of the present application, the obtained table data has a high accuracy.

[0208] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0209] Based on the same inventive concept, the embodiment of the present application also provides a table extraction device for implementing the table extraction method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more table extraction device embodiments provided below can refer to the limitations on the table extraction method above, and will not be repeated here.

[0210] In one embodiment, Figure 8 As shown, a table extraction device 800 is provided, comprising: a table image input module 802, a text information extraction module 804, a cell extraction module 806, a table structure generation module 808 and an electronic table generation module 810, wherein:

[0211] The table image input module 802 is used to obtain a table image and input the table image into a table extraction model. The table extraction model includes a text extraction network block, a cell extraction network block and a structure extraction block.

[0212] The text information extraction module 804 is used to extract text from the table image through a text extraction network block to obtain text information in cells in the table area of ​​the table image.

[0213] The cell extraction module 806 is used to extract cells from the table image through the cell extraction network block to obtain cell information of cells in the table area of ​​the table image.

[0214] The table structure generating module 808 is used to generate a table structure corresponding to the table area according to the cell information of the cell through the structure extracting block.

[0215] The electronic table generation module 810 is used to merge the text information with the cells according to the table structure to obtain the electronic table.

[0216] In one embodiment, the table area includes multiple cells, and the cell information of each cell includes cell coordinates. The table structure generation module 808 is also used to: perform clustering processing on the multiple cells according to the cell coordinates of the multiple cells in the table area through the structure extraction block, determine multiple cell sets, and align the cells in each cell set according to the cell coordinates of the cells in each cell set to generate a table structure.

[0217] In one embodiment, the cell coordinates include first cell coordinates, and the first cell coordinates include first coordinates in a first direction and second coordinates in a second direction. The table structure generation module 808 includes: a first cell set generation unit, which is used to sort the first coordinates of multiple cells in the table area through a structure extraction block, cluster multiple cells according to the sorted first coordinates, and determine multiple first cell sets in the first direction; a first direction alignment unit, which is used to generate a first reference coordinate corresponding to the first cell set according to the first coordinates of the cells in each first cell set, and use the first reference coordinates to align the cells belonging to the first cell set in the first direction; a second cell set generation unit, which is used to sort the second coordinates of multiple cells in the table area through a structure extraction block, cluster multiple cells according to the sorted second coordinates, and determine multiple second cell sets in the second direction; a second direction alignment unit, which is used to generate a second reference coordinate corresponding to the second cell set according to the second coordinates of the cells in each second cell set, and use the second reference coordinates to align the cells belonging to the second cell set in the second direction; a table structure generation unit, which is used to determine the structural information of each cell according to the first reference coordinates, the second reference coordinates, and the cell coordinates corresponding to each cell after the alignment process, and generate a table structure.

[0218] In one embodiment, the cell coordinates also include a second cell coordinate; the first cell coordinate and the second cell coordinate are coordinates at diagonally opposite corners of the cell. The table structure generation unit includes: a reference coordinate generation subunit, used to generate multiple reference coordinates based on the first reference coordinate and the second reference coordinate; a structure reference coordinate determination subunit, used to match the first cell coordinate of each cell after the alignment process with the multiple reference coordinates to determine the first structure reference coordinate corresponding to the first cell coordinate; match the second cell coordinate of each cell after the alignment process with the multiple reference coordinates to determine the second structure reference coordinate corresponding to the second cell coordinate; a table structure generation subunit, used to determine the structure information of each cell based on the first structure reference coordinate and the second structure reference coordinate corresponding to each cell after the alignment process, and generate a table structure corresponding to the table area based on the structure information of each cell.

[0219] In one embodiment, the table structure generation module 808 is also used to: divide the table area into regions according to the cell information of the aligned cells through the structure extraction block, and determine the position information of the table area; and generate a table structure corresponding to the table area according to the structure information of each cell and the position information of the table area.

[0220] In one embodiment, the text information includes text content and position information of the text content. The electronic table generation module 810 is also used to: match the position information of the text content with the cell information of the cell to determine the cell corresponding to the text content; merge the text content with the corresponding cell according to the table structure to obtain an electronic table.

[0221] In one embodiment, the table extraction model also includes an angle detection network block. The table extraction device 800 also includes: a text detection module, which is used to perform text detection on the table image through the angle detection network block to obtain multiple text boxes and the coordinates of each text box; an angle determination module, which is used to determine the angle of the text box according to the coordinates of each text box; a target text box set determination module, which is used to sort the angles of the text boxes, divide the multiple text boxes according to the angles of the sorted text boxes, obtain multiple text box sets, and determine the target text box set from the multiple text box sets according to the number of text boxes in each text box set; an angle correction module, which is used to perform angle correction on the table image according to the angles of the text boxes in the target text box set to obtain an angle-corrected table image; a text information extraction module 804, which is also used to perform text extraction on the angle-corrected table image through the text extraction network block to obtain text information in the cells in the table area of ​​the angle-corrected table image; a cell extraction module 806, which is also used to perform cell extraction on the angle-corrected table image through the cell extraction network block to obtain cell information of the cells in the table area of ​​the angle-corrected table image.

[0222] In one embodiment, the text detection module is also used to: perform text detection on the table image through the angle detection network block to obtain multiple original text boxes and the coordinates of each original text box; determine the area of ​​each original text box according to the coordinates of each original text box; take the original text box whose area is less than or equal to the area threshold as the text box, and obtain the coordinates of each text box.

[0223] In one embodiment, the angle determination module is further used to: determine the aspect ratio of each text box according to the coordinates of each text box; and delete text boxes whose aspect ratio is less than an aspect ratio threshold.

[0224] In one embodiment, the table extraction model also includes a direction classification network block, and the table extraction device 800 also includes: a direction classification module, which is used to determine the direction of the table image through the direction classification network block, and when the direction is an inverted direction, the table image is rotated to obtain the rotated table image; a text information extraction module 804, which is also used to perform text extraction on the rotated table image through the text extraction network block to obtain the text information in the cells in the table area of ​​the rotated table image; a cell extraction module 806, which is also used to perform cell extraction on the rotated table image through the cell extraction network block to obtain the cell information of the cells in the table area of ​​the rotated table image.

[0225] Each module in the above table extraction device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0226] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a table extraction method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0227] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0228] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0229] In one embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0230] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0231] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0232] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0233] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0234] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A table extraction method, characterized in that: The method comprises: Get the table image; Inputting the table image into a table extraction model, wherein the table extraction model includes a text extraction network block, a cell extraction network block, and a structure extraction block; Performing text extraction on the table image through the text extraction network block to obtain text information in cells in the table area of ​​the table image; Extract cells from the table image by using the cell extraction network block to obtain cell information of cells in the table area of ​​the table image; Generate a table structure corresponding to the table area according to the cell information of the cell by the structure extraction block; According to the table structure, the text information is merged with the cells to obtain an electronic table; The table area includes a plurality of cells, the cell information of each cell includes cell coordinates, the cell coordinates include a first cell coordinate and a second cell coordinate, the first cell coordinate and the second cell coordinate are coordinates at diagonally opposite corners of the cell, the first cell coordinate includes a first coordinate in a first direction, and a second coordinate in a second direction; The step of generating a table structure corresponding to the table area according to the cell information of the cell by the structure extraction block includes: sorting the first coordinates of the plurality of cells in the table area by the structure extraction block, clustering the plurality of cells according to the sorted first coordinates, and determining a plurality of first cell sets in the first direction; Obtaining an average, a median or a mode of first coordinates of cells in each of the first cell sets as first reference coordinates corresponding to the first cell set, and aligning the cells belonging to the first cell set in the first direction using the first reference coordinates; sorting the second coordinates of the plurality of cells in the table area by the structure extraction block, clustering the plurality of cells according to the sorted second coordinates, and determining a plurality of second cell sets in the second direction; Obtaining the average, median or mode of the second coordinates of each cell in the second cell set as the second reference coordinates corresponding to the second cell set, and aligning the cells belonging to the second cell set in the second direction using the second reference coordinates; Generate a plurality of reference coordinates by combining the first reference coordinates and the second reference coordinates, and obtain a first sequence of each of the reference coordinates by sorting them based on the first reference coordinates, obtain a second sequence of each of the reference coordinates by sorting them based on the second reference coordinates, and generate an identifier of each of the reference coordinates according to the first sequence and the second sequence; Matching the first cell coordinate of each of the cells after the alignment process with the multiple reference coordinates, obtaining the distance between the first cell coordinate of each of the cells after the alignment process and each of the reference coordinates, determining a first target reference coordinate whose distance to each of the first cell coordinates is less than or equal to a distance threshold, and using the identifier of the first target reference coordinate as a first structure reference coordinate corresponding to the first cell coordinate; Matching the second cell coordinate of each of the cells after the alignment process with the plurality of reference coordinates, obtaining the distance between the second cell coordinate of each of the cells after the alignment process and each of the reference coordinates, determining a second target reference coordinate whose distance to each of the second cell coordinates is less than or equal to the distance threshold, and using the identifier of the second target reference coordinate as the second structure reference coordinate corresponding to the second cell coordinate; Determine a first starting value and a first ending value of each cell in the first direction, and a second starting value and a second ending value of each cell in the second direction according to the first structure reference coordinates and the second structure reference coordinates corresponding to each cell after the alignment process; Determine the first position information of each cell in the first direction according to the first starting value and the first ending value of each cell in the first direction, determine the second position information of each cell in the second direction according to the second starting value and the second ending value of each cell in the second direction, and use the first position information and the second position information as structural information; A table structure corresponding to the table area is generated according to the structural information of each cell.

2. The method according to claim 1, characterized in that The step of generating a table structure corresponding to the table area according to the structural information of each cell further includes: Divide the table image into regions according to the cell information of the cells after the alignment process by the structure extraction block, and determine the position information of the table region; A table structure corresponding to the table area is generated according to the structure information of each cell and the position information of the table area.

3. The method according to any one of claims 1 to 2, characterized in that: The text information includes text content and location information of the text content; The step of merging the text information with the cells according to the table structure to obtain an electronic table includes: Matching the position information of the text content with the cell information of the cell to determine the cell corresponding to the text content; The text content is merged with the corresponding cells according to the table structure to obtain the electronic table.

4. The method according to claim 1, characterized in that: The table extraction model also includes an angle detection network block; the method also includes: Performing text detection on the table image through the angle detection network block to obtain a plurality of text boxes and the coordinates of each of the text boxes; Determining the angle of each text box according to the coordinates of each text box; sorting the angles of the text boxes, dividing the plurality of text boxes according to the sorted angles of the text boxes to obtain a plurality of text box sets, and determining a target text box set from the plurality of text box sets according to the number of the text boxes in each of the text box sets; According to the angles of the text boxes in the target text box set, the table image is angle-corrected to obtain the table image after angle correction; The step of performing text extraction on the table image by using the text extraction network block to obtain text information in cells in the table area of ​​the table image, and performing cell extraction on the table image by using the cell extraction network block to obtain cell information of cells in the table area of ​​the table image includes: Performing text extraction on the angle-corrected table image through the text extraction network block to obtain text information in cells in the table area of ​​the angle-corrected table image; The cell extraction network block is used to extract cells from the angle-corrected table image to obtain cell information of the angle-corrected cells.

5. The method according to claim 4, characterized in that The step of performing text detection on the table image through the angle detection network block to obtain a plurality of text boxes and the coordinates of each of the text boxes includes: Performing text detection on the table image through the angle detection network block to obtain a plurality of original text boxes and the coordinates of each of the original text boxes; Determine the area of ​​each original text box according to the coordinates of each original text box; The original text boxes whose areas are less than or equal to the area threshold are taken as the text boxes, and the coordinates of each of the text boxes are obtained.

6. The method according to claim 4, characterized in that After determining the angle of the text box according to the coordinates of each text box, the method further comprises: Determine the aspect ratio of each text box according to the coordinates of each text box; Delete text boxes whose aspect ratio is less than the aspect ratio threshold.

7. The method according to claim 1, characterized in that The table extraction model also includes a direction classification network block; the method also includes: Determining the direction of the table image by the direction classification network block, and when the direction is an inverted direction, rotating the table image to obtain the rotated table image; The step of performing text extraction on the table image by using the text extraction network block to obtain text information in cells in the table area of ​​the table image, and performing cell extraction on the table image by using the cell extraction network block to obtain cell information of cells in the table area of ​​the table image includes: Performing text extraction on the rotated table image by the text extraction network block to obtain text information in cells in the table area of ​​the rotated table image; The cell extraction network block performs cell extraction on the table image after the rotation processing to obtain the cell information of the cell after the rotation processing.

8. A table extraction device, characterized in that: The device comprises: A table image input module, used for acquiring a table image, and inputting the table image into a table extraction model, wherein the table extraction model includes a text extraction network block, a cell extraction network block and a structure extraction block; A text information extraction module, used to extract text from the table image through the text extraction network block to obtain text information in cells in the table area of ​​the table image; A cell extraction module, used for performing cell extraction on the table image through the cell extraction network block to obtain cell information of cells in the table area of ​​the table image; A table structure generating module, used for generating a table structure corresponding to the table area according to the cell information of the cell through the structure extracting block; An electronic table generation module, used for merging the text information with the cells according to the table structure to obtain an electronic table; The table area includes a plurality of cells, the cell information of each cell includes cell coordinates, the cell coordinates include a first cell coordinate and a second cell coordinate, the first cell coordinate and the second cell coordinate are coordinates at diagonally opposite corners of the cell, the first cell coordinate includes a first coordinate in a first direction, and a second coordinate in a second direction; The table structure generating module comprises: A first cell set generating unit, configured to sort the first coordinates of the plurality of cells in the table area by using the structure extraction block, cluster the plurality of cells according to the sorted first coordinates, and determine a plurality of first cell sets in the first direction; a first direction alignment unit, configured to obtain an average value, a median, or a mode of first coordinates of cells in each of the first cell sets as first reference coordinates corresponding to the first cell set, and to align cells belonging to the first cell set in the first direction using the first reference coordinates; A second cell set generating unit, configured to sort the second coordinates of the plurality of cells in the table area by using the structure extraction block, cluster the plurality of cells according to the sorted second coordinates, and determine a plurality of second cell sets in the second direction; a second direction alignment unit, configured to obtain an average value, a median value, or a mode value of the second coordinates of each cell in the second cell set as a second reference coordinate corresponding to the second cell set, and align the cells belonging to the second cell set in the second direction using the second reference coordinate; Table structure generation unit, including: a reference coordinate generating subunit, configured to generate a plurality of reference coordinates by combining the first reference coordinates and the second reference coordinates, and to obtain a first sequence of each of the reference coordinates by sorting the first reference coordinates as a reference, and to obtain a second sequence of each of the reference coordinates by sorting the second reference coordinates as a reference, and to generate an identifier of each of the reference coordinates according to the first sequence and the second sequence; a structural reference coordinate determination subunit, used for matching the first cell coordinate of each of the cells after the alignment process with the multiple reference coordinates, obtaining the distance between the first cell coordinate of each of the cells after the alignment process and each of the reference coordinates, determining a first target reference coordinate whose distance to each of the first cell coordinates is less than or equal to a distance threshold, and using the identifier of the first target reference coordinate as the first structural reference coordinate corresponding to the first cell coordinate; matching the second cell coordinate of each of the cells after the alignment process with the multiple reference coordinates, obtaining the distance between the second cell coordinate of each of the cells after the alignment process and each of the reference coordinates, determining a second target reference coordinate whose distance to each of the second cell coordinates is less than or equal to the distance threshold, and using the identifier of the second target reference coordinate as the second structural reference coordinate corresponding to the second cell coordinate; A table structure generating subunit is used to determine the first starting value and the first ending value of each cell in the first direction, and the second starting value and the second ending value of each cell in the second direction according to the first structure reference coordinates and the second structure reference coordinates corresponding to each cell after alignment processing; determine the first position information of each cell in the first direction according to the first starting value and the first ending value of each cell in the first direction, determine the second position information of each cell in the second direction according to the second starting value and the second ending value of each cell in the second direction, and use the first position information and the second position information as structure information; generate a table structure corresponding to the table area according to the structure information of each cell.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images