Table structure extraction method, device, electronic device and storage medium

Abstract

The embodiments of this specification provide a table structure extraction method, device, electronic device and storage medium. The method includes: using a multi-scale image super-resolution network to pre-process the original image to obtain feature maps with different scale features, and using a predetermined cascade table detection network to detect the feature maps to obtain the position of the target box in each feature map and the confidence score; based on the corresponding position and confidence score of the target box, performing a cropping operation on the target box in each feature map to obtain a table image corresponding to each feature map; using the table image as input, using a predetermined table internal structure extraction algorithm to perform a text removal operation on the table image, and processing the table image after the text is removed to obtain the internal contour of the table, and restoring the table structure based on the internal contour of the table. The present disclosure can reduce the cost of table structure extraction and significantly improve the table recognition precision, recognition accuracy and recognition efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a table structure extraction method, device, electronic device, and storage medium. Background Art

[0002] In real life, document images often contain many tables, and many tables also exist in the form of images. In order to restore tables in image form into a data format that can be structured and stored, such as Excel tables, it is first necessary to extract the table structure in the document image. Therefore, the recognition accuracy and rate of the table structure extraction algorithm are of great significance for restoring tables into structured data.

[0003] Traditional methods for extracting table structures, such as extracting table data from certificate files, typically rely on manual entry. While this method can accurately extract table structures with simple structures and data, it can be quite challenging when tables are complex and densely populated with cells. However, traditional methods are prone to misidentification and missed recognition, resulting in low recognition accuracy, poor precision, and low efficiency.

[0004] In view of the above problems in the prior art, it is necessary to provide a table structure extraction method that can reduce labor costs, ensure the efficiency of table structure extraction, and take into account recognition precision and accuracy. Summary of the Invention

[0005] In view of this, the embodiments of the present disclosure provide a table structure extraction method, device, electronic device and storage medium to solve the problems of high labor cost, low algorithm recognition accuracy, poor recognition precision and low recognition efficiency in table structure extraction in the prior art.

[0006] According to a first aspect of an embodiment of the present disclosure, a table structure extraction method is provided, comprising: obtaining an original image, preprocessing the original image using a predetermined multi-scale image super-resolution network to obtain a feature map with different scale features, so as to reconstruct the original image into a clear image; using the feature map as input, detecting the feature map using a predetermined cascade table detection network to obtain the position and confidence score of a target box in each feature map, wherein the target box contains a table; performing a cropping operation on the target box in each feature map based on the corresponding position and confidence score of the target box to obtain a table image corresponding to each feature map; using the table image as input, removing text in the table image using a predetermined table internal structure extraction algorithm, processing the table image after the text is removed to obtain an internal contour of the table, and performing a recovery operation based on the internal contour of the table to obtain a table structure corresponding to the table.

[0007] According to a second aspect of an embodiment of the present disclosure, a table structure extraction device is provided, comprising: a processing module configured to obtain an original image, and pre-process the original image using a predetermined multi-scale image super-resolution network to obtain a feature map with different scale features, so as to reconstruct the original image into a clear image; a detection module configured to take the feature map as input, and detect the feature map using a predetermined cascade table detection network to obtain the position and confidence score of a target box in each feature map, wherein the target box contains a table; a cropping module configured to perform a cropping operation on the target box in each feature map based on the corresponding position and confidence score of the target box, and obtain a table image corresponding to each feature map; an extraction module configured to take the table image as input, and use a predetermined table internal structure extraction algorithm to perform a text removal operation on the table image, and process the table image after the text is removed to obtain the internal contour of the table, and perform a recovery operation based on the internal contour of the table to obtain the table structure corresponding to the table.

[0008] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0009] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above method are implemented.

[0010] At least one of the above technical solutions adopted in the embodiments of the present disclosure can achieve the following beneficial effects:

[0011] By obtaining an original image, pre-processing the original image using a predetermined multi-scale image super-resolution network to obtain feature maps with different scale features, so as to reconstruct the original image into a clear image; taking the feature map as input, detecting the feature map using a predetermined cascade table detection network to obtain the position and confidence score of the target box in each feature map, wherein the target box contains a table; based on the corresponding position and confidence score of the target box, performing a cropping operation on the target box in each feature map respectively to obtain a table image corresponding to each feature map; taking the table image as input, using a predetermined table internal structure extraction algorithm, performing a text removal operation on the table image, and processing the table image after the text removal to obtain the internal contour of the table, performing a restoration operation based on the internal contour of the table to obtain the table structure corresponding to the table. The present disclosure can not only reduce the labor cost of table structure extraction, but also improve recognition accuracy, recognition precision and recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0013] Figure 1 It is a flowchart of the table structure extraction method provided by the embodiment of the present disclosure;

[0014] Figure 2 is a structural diagram of a table structure extraction device provided by an embodiment of the present disclosure;

[0015] Figure 3 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present disclosure with unnecessary detail.

[0017] As mentioned above, in real life, a large number of documents are saved in the form of images, and document images usually contain a large number of tables. Therefore, tables are actually stored in the form of images. In actual work, it is necessary to restore the table in the form of images into a data format that can be structured for storage, such as Excel tables. An important part of table restoration is to extract the table structure in the document image, that is, to identify the tables and the specific locations of the cells in the table from the document image. The following takes the extraction of the table structure in the certificate file as an example to explain in detail the problems existing in the traditional table structure extraction method, which can specifically include the following:

[0018] Among the traditional extraction methods, one is to extract by manual entry, and the other is to use a table frame line extraction method based on traditional vision to restore the table structure. For the first method, manual entry is performed manually. For tables with relatively simple structures and data, this traditional method can also extract the table structure more accurately. However, when the table structure is relatively complex and the cells are dense, the traditional extraction method is prone to misidentification and missed recognition problems. For the second method, a table frame line extraction method based on traditional vision is used to restore the table structure. This method has a certain extraction accuracy for images with clear cell lines, but for images with poor clarity of cell lines, it is impossible to accurately identify the cells therein. Therefore, no matter which of the above methods is used, it cannot meet the needs of table extraction in scenarios with relatively complex table structures, and cannot accurately identify the cells in the table structure, resulting in low recognition accuracy, poor recognition precision, low recognition efficiency and other problems.

[0019] Therefore, in view of the above-mentioned problems of the prior art, in order to reduce labor costs and improve recognition accuracy, the present disclosure proposes a table structure extraction algorithm based on OpenCV and neural networks. The extraction algorithm process may include the steps of preprocessing, table detection, table structure extraction, table text recognition and table information reconstruction. The technical solution of the present disclosure is described in detail below in combination with specific embodiments.

[0020] Figure 1 It is a flowchart of the table structure extraction method provided by the embodiment of the present disclosure. Figure 1 The table structure extraction method can be executed by the server. Figure 1 As shown, the table structure extraction method may specifically include:

[0021] S101, obtaining an original image, and preprocessing the original image using a predetermined multi-scale image super-resolution network to obtain a feature map with different scale features, so as to reconstruct the original image into a clear image;

[0022] S102, taking the feature map as input, and detecting the feature map using a predetermined cascade table detection network to obtain the position and confidence score of a target box in each feature map, where the target box contains a table;

[0023] S103, based on the position and confidence score corresponding to the target box, performing a cropping operation on the target box in each feature map to obtain a table image corresponding to each feature map;

[0024] S104, taking the table image as input, using a predetermined table internal structure extraction algorithm to perform a text removal operation on the table image, and processing the table image after the text removal to obtain the table internal contour, performing a recovery operation based on the table internal contour to obtain the table structure corresponding to the table.

[0025] Specifically, the original image refers to an image containing a table, and of course it can also be an image that does not contain any table. For images containing tables, this solution can be used to efficiently and accurately extract the table therein. For images that do not contain any tables, the cascade table detection network of this solution cannot obtain the table. In practical applications, the original image can be a certificate file in the form of a text document converted into a PDF or picture format, that is, the original image targeted by this solution can be considered as a certificate file in the form of a picture. It should be understood that although the following embodiments of the present disclosure are described with certificate files as the processing object, the embodiments of the present disclosure are not limited to the extraction of table structures in certificate files, and any image or picture can be the processing object of the present disclosure.

[0026] Furthermore, the target box can also be called a candidate box or a table box, which means a box containing a table extracted from the feature map using the region generation network. The target box contains the coordinate positions of the four vertices of the table. In the embodiment of the present disclosure, the feature map obtained by preprocessing using the multi-scale image super-resolution network incorporates multi-scale information. The feature map can be considered as a matrix, and the scale can be considered as the proportion of different images (i.e., the area size). The larger the feature scale, the larger the receptive field and the richer the feature information.

[0027] According to the technical solution provided by the embodiment of the present disclosure, the present disclosure obtains an original image, uses a predetermined multi-scale image super-resolution network to pre-process the original image to obtain a feature map with different scale features, so as to reconstruct the original image into a clear image; takes the feature map as input, uses a predetermined cascade table detection network to detect the feature map, obtains the position and confidence score of the target box in each feature map, and the target box contains a table; based on the corresponding position and confidence score of the target box, performs a cropping operation on the target box in each feature map, and obtains a table image corresponding to each feature map; takes the table image as input, uses a predetermined table internal structure extraction algorithm, performs a text removal operation on the table image, and processes the table image after the text is removed to obtain the internal contour of the table, performs a restoration operation based on the internal contour of the table, and obtains the table structure corresponding to the table. The present disclosure can not only reduce the labor cost of table structure extraction, but also improve recognition accuracy, improve recognition precision difference and recognition efficiency.

[0028] In some embodiments, a predetermined multi-scale image super-resolution network is used to pre-process the original image to obtain feature maps with different scale features, including: using the backbone network in the multi-scale image super-resolution network to perform a convolution operation on the original image to obtain a first feature map, inputting the first feature map into multiple first residual networks respectively, the multiple first residual networks respectively corresponding to different scale feature extraction branches, using the scale feature extraction branches to extract second feature maps containing different scale information; performing a weighted fusion operation on the multiple second feature maps to obtain a third feature map, and using the second residual network to extract different scale features in the third feature map to obtain a fourth feature map, and using the scale feature extraction branch to process the fourth feature map to obtain multiple feature maps with different scale features.

[0029] Specifically, the table structure extraction algorithm based on OpenCV and neural networks provided in the embodiments of the present disclosure may include the following steps: pre-processing (i.e., pre-processing), table detection, table structure extraction, table text recognition, and table information reconstruction. In the pre-processing stage, since the recognition accuracy of table images in real scenes is very poor due to the presence of blur, in order to avoid the influence of blurred images on table structure recognition, the present disclosure proposes a multi-scale image super-resolution algorithm to complete the mapping of blurred images to clear images, that is, mapping the original blurred image to a clear image.

[0030] Furthermore, image super-resolution is essentially to improve the pixel density based on the input image. Traditional upsampling algorithms (Bilinear, Bicubic, Nearst-neighbor, etc.) complete pixel filling through fixed rules, so that the output high-resolution image contains a lot of unreal high-frequency information. The optimal value selection method of these pixels must be non-human intervention and have a self-learning mechanism. Therefore, many deep learning algorithms can achieve good results in the field of super-resolution. However, these previous algorithms ignore the fact that image super-resolution is a pathological problem, that is, there are multiple reasonable predictions for a given low-resolution input. It models image super-resolution as a deterministic process, which leads to unsatisfactory results. Therefore, based on the previous image super-resolution algorithm, the present disclosure constructs multiple feature extraction branches, extracts multi-scale information in the encoding stage, and performs multi-scale reconstruction in the decoding stage, so that the output has a higher divergence. In conjunction with specific embodiments, the process of pre-processing using a multi-scale image super-resolution algorithm to obtain a feature map of multi-scale features is described in detail, which may specifically include the following contents:

[0031] The multi-scale image super-resolution network mainly consists of a backbone network, a residual network, and an attention network. The original image is first processed by convolution in the backbone network to obtain a feature map (i.e., the first feature map). The backbone network here is a Stem network, which contains many convolutional layers. The 7x7 convolutional layer in the original Stem network is replaced with three consecutive 3x3 convolutional layers, so that they have the same receptive field size and similar computational cost as the original Stem network design. At the same time, in the shortcut branch, the downsampling in the 1x1 convolutional layer is replaced with average pooling with a stride of 2 to avoid information loss.

[0032] After using the backbone network to extract feature maps, the residual blocks ResBlock corresponding to the feature extraction branches of different scales are used to extract features of different receptive fields. The more modules the ResBlock stack has, the larger the scale of the extracted features, the larger the receptive field, and the richer the information. For example, in practical applications, three scale feature extraction branches can be used to extract multi-scale information respectively. The ResBlocks corresponding to these three scale feature extraction branches are ResBlock / x2, ResBlock / x3, and ResBlock / x4, respectively.

[0033] Furthermore, after extracting multi-scale features in the encoding stage, a split-attention module is used to perform weighted fusion of the multi-scale features. The split-attention module adaptively learns a weight for each scale feature and then performs weighted fusion. That is, a weight is assigned to the feature map output by each scale feature extraction branch, the weight is multiplied by the feature map, and then weighted fusion is performed to obtain a weighted fused feature map with multi-scale information.

[0034] Furthermore, after using the Split-Attention module to perform weighted fusion of multi-scale features, two consecutive residual networks (Res2Blocks) are used to further extract multi-scale features. That is, Res2Block is introduced in the intermediate calculation stage of the network, and multiple receptive fields are introduced inside the convolution kernel, thereby improving the network's ability to capture multi-scale features. Finally, feature maps with different scales are output, that is, images with different receptive fields are obtained, thereby reconstructing blurred images into clear images.

[0035] It should be noted that the activation function in the middle of the multi-scale image super-resolution network disclosed in this invention is the Mish function, and its calculation formula is: y = x*tanh(ln(1+exp(x))). The function has the characteristics of smoothness, non-monotonicity, upper unbounded, and lower bound. Compared with other commonly used functions (such as ReLU, Swish, etc.), it can improve the nonlinear transformation ability of the model.

[0036] In some embodiments, the cascade table detection network includes a feature extraction network, a multi-level information fusion network, a region generation network and a prediction network; the feature map is detected using a predetermined cascade table detection network to obtain the position of the target box in each feature map and the confidence score, including: using the feature extraction network to process the feature map to obtain multiple fifth feature maps, the feature extraction network includes a deformable convolution network, the deformable convolution network is used to learn the offset value, and the offset value is used for convolution calculation; using the multi-level information fusion network to pass the low-level information in the fifth feature map to the high-level information, and performing information fusion on the fifth feature map to obtain a sixth feature map; using the region generation network to extract the target box in the sixth feature map, and mapping the extracted target box to the sixth feature map; using the prediction network to correct the position of the target box, and performing a confidence score on the target box, and using the corrected target box position and confidence score as the output of the cascade table detection network.

[0037] Specifically, in the table detection stage, since the original Cascade Mask R-CNN network is used for table detection, there are many cases of missed detection and false detection. Therefore, in order to adapt to the table detection task, the original Cascade Mask R-CNN is improved from three aspects: Backbone module, Neck module and loss function. A cascade table detection network is constructed, which can accurately and efficiently extract tables in images. The following is a detailed description of the process of obtaining a table frame by using the cascade table detection network for table detection in conjunction with specific embodiments, which may include the following:

[0038] In the network structure of the cascaded table detection network, the input feature map first passes through the feature extraction network Backbone for feature extraction, and then performs multi-level information fusion in the multi-level information fusion network Neck; it then passes through the region generation network RPN to extract proposals (i.e., target boxes), and then uses ROI to map the extracted table boxes to the original Neck output, and these target boxes are sent to the cascaded prediction network Head for position adjustment to obtain the final output result.

[0039] Furthermore, since table images in real scenes exist in various scales and at various tilt angles, in order to enhance the ability of convolutional neural networks to process these complex data, deformable convolution (DCN) is added to the feature extraction network Backbone. DCN belongs to the middle layer of the feature extraction network Backbone. Before performing standard convolution calculations, DCN first learns a 2D offset map by inputting the feature map, and then applies the learned offset value to the corresponding position of the current convolution kernel, and uses the offset value to perform the normal convolution calculation process. By adding DCN, the model accuracy can be significantly improved.

[0040] Furthermore, in the multi-level information fusion network Neck, since high-level feature maps focus on target semantic information, while low-level feature maps focus on target position information, the original Cascade Mask R-CNN network uses FPN in the Neck stage for top-down information fusion. While this strengthens the target semantic information at the bottom layer, it does not enhance the target position information at the top layer. Therefore, this paper utilizes the PA-FPN structure to transmit low-level information to the upper layers, while reducing the number of convolutional layers required for information to flow between the upper and lower layers.

[0041] Furthermore, the prediction network Head is composed of a convolutional layer and a fully connected layer. The prediction network Head is used to correct the coordinates of the table box, adjust the position of the table box, score the table box to obtain a confidence score, and obtain the final output result, that is, the confidence score C1 of the target box and the precise position B1 of the target box. It should be emphasized that the loss function of the regression branch of the original CascadeMask R-CNN network is Smooth L1 Loss, but because the detection and evaluation method uses IoU, and the actual regression coordinate box uses 4 coordinate points, the two methods are not equivalent, that is, the box with the same Loss has a non-unique IoU. At the same time, because Smooth L1 Loss is not invariant to scale, the present disclosure uses DIoU Loss instead of Smooth L1 Loss, and regards the box composed of the 4 points of the table box as a whole for regression, so as to better guide the training process. The calculation process can be calculated using the following formula:

[0042]

[0043] Among them, b and bgt represent the center points of the prediction box B and the target box Bgt respectively, ρ represents the Euclidean distance, and c represents the diagonal distance of the minimum outer rectangle of B and Bgt.

[0044] In some embodiments, the predetermined table internal structure extraction algorithm is a table internal structure extraction algorithm based on OpenCV, and the predetermined table internal structure extraction algorithm is used to perform a removal operation on the text in the table image, including: performing grayscale and threshold operations on the table image in sequence to obtain a binarized image, inverting the binarized image, denoising the binarized image obtained after the inversion processing using a non-local average denoising algorithm, and removing the text in the table image obtained after the denoising processing through a corrosion algorithm and an expansion algorithm.

[0045] Specifically, in the table structure extraction stage, in order to extract the cell outlines inside the table, the embodiment of the present disclosure proposes a table internal structure extraction algorithm based on OpenCV to obtain the cell outlines and position information inside the table. Through this information, the complete table structure can be restored. The following is a detailed description of the process of using OpenCV to extract the internal structure of the table in conjunction with a specific embodiment, which may include the following:

[0046] First, the table is cropped based on the position coordinates of the table frame. Then, all table images are grayscaled and thresholded. The principle of thresholding is to convert the pixel values ​​corresponding to the pixels in the table image according to a preset pixel value threshold to obtain a binary image (that is, a table image containing only black and white). The binary image is then inverted, that is, the blank areas of the table are turned black, and the lines and text in the table are turned white.

[0047] Furthermore, after the inversion process is complete, some noise may remain in the table image, which can affect subsequent recognition. Therefore, the non-local mean denoising algorithm is used here to remove the noise points in the table image. Finally, the erosion and dilation algorithms are used to remove the text in the table image. The more times the erosion and dilation process is repeated, the more effective the text removal effect is. However, multiple erosion and dilation processes can also affect the table lines, so the value needs to be adjusted according to the specific dataset. Here, a value of 2 is used.

[0048] In some embodiments, the table image after text removal is processed to obtain the internal contour of the table, and a recovery operation is performed based on the internal contour of the table to obtain a table structure corresponding to the table, including: using a horizontal filter and a vertical filter to process the table image after text removal to obtain all horizontal lines and vertical lines, superimposing the horizontal lines and the vertical lines to obtain the overall contour of the table; using a multi-layer contour search algorithm to obtain the cells in the overall contour of the table, and merging the lines between adjacent cells to obtain the internal contour of the table, and restoring the table structure based on the internal contour of the table and the cell information.

[0049] Specifically, after removing the text from the table image to obtain a binary image (i.e., the table image after text removal), a preset horizontal filter is applied over the image to obtain all horizontal lines in the image. Similarly, a preset vertical filter is applied over the image to obtain all vertical lines in the image. By simply adding these two types of lines, the overall outline of the table can be obtained.

[0050] Furthermore, because tables in real-world scenarios can be tilted, they need to be corrected. By calculating the angles between all lines on a horizontal image and the horizontal line, the overall tilt angle of the table is determined. The table image is then rotated by the corresponding angle until it is horizontal. This angular rotation yields a horizontal table image outline. A multi-layer contour search algorithm is then used to identify all cells in the table. Because the same line may serve as the edge of multiple cells, these edges are fused together to obtain the standard internal outline of the table.

[0051] Furthermore, based on the obtained internal outline of the table and the information corresponding to all cells, the specific table structure can be restored in Excel using the obtained table cell information.

[0052] In some embodiments, after obtaining the table structure corresponding to the table, the method also includes: obtaining the original image, using a table text recognition algorithm to identify the table in the original image, extracting the position information and text information of the table, and embedding the text information into the table structure based on the coordinate value in the position information, so as to reconstruct the table information based on the table structure.

[0053] Specifically, the above examples provide a detailed introduction to the first three stages of the table structure extraction algorithm disclosed herein. In practical applications, to obtain the final table—that is, a table containing both the table structure and textual information—it is necessary to perform text recognition on the original image separately, extracting the textual information, location information, and material information from the table in the original image. Based on the positional coordinates of the textual information corresponding to the table, the text is embedded into the table structure, thereby reconstructing the table information.

[0054] According to the technical solution provided by the embodiment of the present disclosure, the present disclosure proposes a table structure extraction algorithm based on OpenCV and neural network. The algorithm introduces receptive field features of different sizes in the encoding stage and the decoding stage, so that multi-scale information can be extracted. The algorithm can be used to complete the mapping of blurred images to clear images. In order to adapt to the table detection task, the original Cascade Mask R-CNN is improved from three aspects: Backbone, Neck and loss function, and a cascade table detection network is constructed, which can accurately and efficiently extract tables in pictures; finally, the table internal structure extraction algorithm based on OpenCV is used to directly obtain the cell outline and position information inside the table, and the complete table structure can be restored through this information. Based on the above processing, the labor cost of table structure extraction can be reduced, and the table recognition accuracy, recognition accuracy and recognition efficiency can be significantly improved.

[0055] The following are embodiments of the apparatus disclosed herein, which can be used to implement the method embodiments disclosed herein. For details not disclosed in the apparatus embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0056] Figure 2 Schematic diagram of the structure of the table structure extraction device provided by the embodiment of the present disclosure. Figure 2 As shown, the table structure extraction device includes:

[0057] The processing module 201 is configured to obtain an original image, and pre-process the original image using a predetermined multi-scale image super-resolution network to obtain a feature map with different scale features, so as to reconstruct the original image into a clear image;

[0058] A detection module 202 is configured to take the feature map as input and detect the feature map using a predetermined cascade table detection network to obtain the location and confidence score of a target box in each feature map, where the target box contains a table;

[0059] A cropping module 203 is configured to perform a cropping operation on the target box in each feature map based on the position and confidence score of the target box, thereby obtaining a table image corresponding to each feature map.

[0060] The extraction module 204 is configured to take the table image as input, use a predetermined table internal structure extraction algorithm to perform a text removal operation on the table image, and process the table image after the text is removed to obtain the internal contour of the table, and perform a recovery operation based on the internal contour of the table to obtain the table structure corresponding to the table.

[0061] In some embodiments, Figure 2The processing module 201 uses the backbone network in the multi-scale image super-resolution network to perform a convolution operation on the original image to obtain a first feature map, and inputs the first feature map into multiple first residual networks respectively. The multiple first residual networks correspond to different scale feature extraction branches, and the scale feature extraction branches are used to extract second feature maps containing different scale information; a weighted fusion operation is performed on the multiple second feature maps to obtain a third feature map, and the second residual network is used to extract different scale features in the third feature map to obtain a fourth feature map, and the scale feature extraction branch is used to process the fourth feature map to obtain multiple feature maps with different scale features.

[0062] In some embodiments, the cascaded table detection network includes a feature extraction network, a multi-level information fusion network, a region generation network, and a prediction network; Figure 2 The detection module 202 uses a feature extraction network to process the feature map to obtain multiple fifth feature maps, where the feature extraction network includes a deformable convolution network, which is used to learn offset values ​​and perform convolution calculations using the offset values; uses a multi-level information fusion network to transfer low-level information in the fifth feature map to high-level information, and performs information fusion on the fifth feature map to obtain a sixth feature map; uses a region generation network to extract the target box in the sixth feature map, and maps the extracted target box to the sixth feature map; uses a prediction network to correct the position of the target box, and performs a confidence score on the target box, and uses the corrected target box position and confidence score as the output of the cascade table detection network.

[0063] In some embodiments, the predetermined table internal structure extraction algorithm is an OpenCV-based table internal structure extraction algorithm. Figure 2 The extraction module 204 performs grayscale and threshold operations on the table image in sequence to obtain a binary image, inverts the binary image, denoises the binary image obtained after the inversion process using a non-local average denoising algorithm, and removes text from the table image obtained after the denoising process using an erosion algorithm and a dilation algorithm.

[0064] In some embodiments, Figure 2 The extraction module 204 uses a horizontal filter and a vertical filter to process the table image after the text is removed, obtains all horizontal lines and vertical lines, and superimposes the horizontal lines and vertical lines to obtain the overall outline of the table; uses a multi-layer outline search algorithm to obtain the cells in the overall outline of the table, and merges the lines between adjacent cells to obtain the internal outline of the table, and restores the table structure based on the internal outline of the table and the cell information.

[0065] In some embodiments, Figure 2After obtaining the table structure corresponding to the table, the reconstruction module 205 obtains the original image, uses the table text recognition algorithm to recognize the table in the original image, extracts the position information and text information of the table, and embeds the text information into the table structure according to the coordinate value in the position information, so as to reconstruct the table information based on the table structure.

[0066] In some embodiments, the original image is an original image generated according to the certificate file, and the original image is a blurred image.

[0067] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0068] Figure 3 Schematic diagram of the structure of the electronic device 3 provided by the embodiment of the present disclosure. Figure 3 As shown, the electronic device 3 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, the steps of the above-mentioned method embodiments are implemented. Alternatively, when the processor 301 executes the computer program 303, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0069] For example, the computer program 303 may be divided into one or more modules / units, which are stored in the memory 302 and executed by the processor 301 to implement the present disclosure. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program 303 in the electronic device 3.

[0070] The electronic device 3 may be a desktop computer, a notebook, a PDA, a cloud server or other electronic device. The electronic device 3 may include but is not limited to a processor 301 and a memory 302. Those skilled in the art will understand that Figure 3 It is only an example of electronic device 3 and does not constitute a limitation of electronic device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0071] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0072] The memory 302 can be an internal storage unit of the electronic device 3, such as a hard drive or memory of the electronic device 3. The memory 302 can also be an external storage device of the electronic device 3, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped on the electronic device 3. Furthermore, the memory 302 can include both an internal storage unit of the electronic device 3 and an external storage device. The memory 302 is used to store computer programs and other programs and data required by the electronic device. The memory 302 can also be used to temporarily store data that has been output or is about to be output.

[0073] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0074] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0075] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0076] In the embodiments provided in the present disclosure, it should be understood that the disclosed apparatus / computer equipment and methods can be implemented in other ways. For example, the apparatus / computer equipment embodiments described above are merely schematic. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of the apparatus or unit, which may be electrical, mechanical or other forms.

[0077] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0078] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0079] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0080] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present disclosure, and should all be included in the scope of protection of the present disclosure.

Claims

1. A table structure extraction method, characterized in that: include: Acquire an original image, and preprocess the original image using a predetermined multi-scale image super-resolution network to obtain a feature map with different scale features, so as to reconstruct the original image into a clear image; Taking the feature map as input, detecting the feature map using a predetermined cascade table detection network to obtain a position and a confidence score of a target box in each feature map, wherein the target box contains a table; Based on the position and confidence score corresponding to the target frame, a cropping operation is performed on the target frame in each of the feature maps to obtain a table image corresponding to each of the feature maps; Taking the table image as input, using a predetermined table internal structure extraction algorithm to perform a text removal operation on the table image, processing the table image after the text removal to obtain a table internal outline, and performing a restoration operation based on the table internal outline to obtain a table structure corresponding to the table; The cascade table detection network includes a feature extraction network, a multi-level information fusion network, a region generation network, and a prediction network; the feature map is detected using the predetermined cascade table detection network to obtain the position of the target box in each feature map and the confidence score, including: Processing the feature map using the feature extraction network to obtain a plurality of fifth feature maps, wherein the feature extraction network includes a deformable convolutional network, the deformable convolutional network is used to learn offset values, and the offset values ​​are used to perform convolution calculations; Using the multi-level information fusion network, the low-level information in the fifth feature map is transferred to the high-level information, and the fifth feature map is subjected to information fusion to obtain a sixth feature map; Extracting the target frame from the sixth feature map using the region generation network, and mapping the extracted target frame to the sixth feature map; Correcting the position of the target frame using the prediction network, performing a confidence score on the target frame, and using the corrected position of the target frame and the confidence score as outputs of the cascade table detection network; The predetermined table internal structure extraction algorithm is a table internal structure extraction algorithm based on OpenCV. The predetermined table internal structure extraction algorithm is used to perform a text removal operation on the table image, including: performing grayscale and thresholding operations on the table image in sequence to obtain a binary image, inverting the binary image, denoising the binary image obtained after the inversion process using a non-local mean denoising algorithm, and removing text from the table image obtained after the denoising process using an erosion algorithm and a dilation algorithm; The step of processing the table image after removing text to obtain an internal outline of the table, and performing a restoration operation based on the internal outline of the table to obtain a table structure corresponding to the table includes: Using a horizontal filter and a vertical filter respectively to process the table image after removing the text, to obtain all horizontal lines and vertical lines, and superimposing the horizontal lines and vertical lines to obtain the overall outline of the table; A multi-layer contour search algorithm is used to obtain cells in the overall contour of the table, and lines between adjacent cells are merged to obtain the internal contour of the table. The table structure is restored based on the internal contour of the table and the information of the cells.

2. The method according to claim 1, characterized in that The preprocessing of the original image using a predetermined multi-scale image super-resolution network to obtain a feature map with different scale features includes: Using a backbone network in the multi-scale image super-resolution network, a convolution operation is performed on the original image to obtain a first feature map, and the first feature map is respectively input into a plurality of first residual networks, wherein the plurality of first residual networks respectively correspond to different scale feature extraction branches, and the scale feature extraction branches are used to extract second feature maps containing information of different scales; A weighted fusion operation is performed on the plurality of second feature maps to obtain a third feature map, and different scale features are extracted from the third feature map using a second residual network to obtain a fourth feature map. The fourth feature map is processed using the scale feature extraction branch to obtain a plurality of feature maps having different scale features.

3. The method according to claim 1, characterized in that After obtaining the table structure corresponding to the table, the method further includes: Acquire the original image, use a table text recognition algorithm to recognize the table in the original image, extract the position information and text information of the table, and embed the text information into the table structure according to the coordinate value in the position information, so as to reconstruct the table information based on the table structure.

4. The method according to any one of claims 1 to 3, characterized in that The original image is an original image generated according to the certificate file, and the original image is a blurred image.

5. A table structure extraction device, characterized in that: include: a processing module configured to obtain an original image, and preprocess the original image using a predetermined multi-scale image super-resolution network to obtain a feature map with different scale features, so as to reconstruct the original image into a clear image; a detection module configured to take the feature map as input, detect the feature map using a predetermined cascade table detection network, and obtain a position and a confidence score of a target box in each feature map, wherein the target box contains a table; a cropping module configured to perform a cropping operation on the target box in each of the feature maps based on a position corresponding to the target box and a confidence score, to obtain a table image corresponding to each of the feature maps; an extraction module configured to take the table image as input, perform a text removal operation on the table image using a predetermined table internal structure extraction algorithm, process the table image after text removal to obtain a table internal contour, and perform a restoration operation based on the table internal contour to obtain a table structure corresponding to the table; Among them, the cascade table detection network includes a feature extraction network, a multi-level information fusion network, a region generation network and a prediction network; the detection module is used to use the feature extraction network to process the feature map to obtain multiple fifth feature maps, and the feature extraction network includes a deformable convolution network, and the deformable convolution network is used to learn offset values ​​and perform convolution calculations using the offset values; the multi-level information fusion network is used to transfer the low-level information in the fifth feature map to the high-level information, and the fifth feature map is fused to obtain a sixth feature map; the region generation network is used to extract the target box in the sixth feature map, and the extracted target box is mapped to the sixth feature map; the prediction network is used to correct the position of the target box, and the target box is scored with a confidence score, and the corrected position of the target box and the confidence score are used as the output of the cascade table detection network; The predetermined table internal structure extraction algorithm is a table internal structure extraction algorithm based on OpenCV, the extraction module is used to sequentially perform grayscale and thresholding operations on the table image to obtain a binary image, perform color inversion processing on the binary image, perform denoising processing on the binary image obtained after the color inversion processing using a non-local mean denoising algorithm, and remove text in the table image obtained after the denoising processing using an erosion algorithm and a dilation algorithm; The extraction module is also used to process the table image after removing the text using a horizontal filter and a vertical filter respectively to obtain all horizontal lines and vertical lines, and superimpose the horizontal lines and vertical lines to obtain the overall outline of the table; use a multi-layer contour search algorithm to obtain the cells in the overall outline of the table, and merge the lines between adjacent cells to obtain the internal outline of the table; and restore the table structure based on the internal outline of the table and the information of the cells.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images