A method for constructing a picture table line extraction model and a picture table extraction method

Abstract

The present invention relates to the field of graphic extraction. A method for constructing a model for extracting lines from image tables is provided. First, training data is selected; an image containing a table is analyzed, each table line is labeled with a line segment of a preset width, and pixels on the line segment are marked; unlabeled portions are considered background. Second, a dataset is constructed using the training data; any dataset is randomly split into a training set and a test set; then, a U-Net model, a U-Net++ model, or an integrated model of U-Net and U-Net++ is constructed, and the model is trained using the training set. Training is terminated until the model prediction performance reaches a preset value after testing on the test set, and the model for extracting lines from image tables is constructed. This reduces the difficulty of constructing the training set, combines the advantages of the U-Net and U-Net++ models, and enables more accurate extraction of graphic tables.

Description

Technical Field

[0001] The present invention belongs to the field of graphic extraction, and in particular relates to a method for constructing a picture table line extraction model and a picture table extraction method. Background Art

[0002] Currently existing methods for identifying image tables:

[0003] (1) Traditional method: Based on the image processing method of OpenCV, the horizontal and vertical lines in the image are extracted using erosion and dilation operations. The horizontal and vertical lines are then superimposed to form a table structure. The coordinates of each intersection in the table can be obtained. The outline of the formed table is extracted and the outer frame of the table is obtained using polygon fitting. The outlines of all connected areas in the table are extracted to determine the position and size of each cell in the table. Finally, the structure of the table is determined based on the outer frame and cells of the table and the corresponding information is extracted.

[0004] (2) Deep learning method: Based on image semantic segmentation, a corresponding table line detection model is trained to automatically detect the table line area in the image. Then, the table lines are extracted based on the detected table line area to reconstruct the table structure. Image semantic segmentation is to segment the image into blocks with certain semantic meanings through a certain method, and identify the semantic category of each block, realizing the semantic reasoning process from the bottom to the top, and finally obtaining a segmented image with pixel-by-pixel semantic annotations.

[0005] Traditional semantic segmentation methods based on candidate regions first extract free-form regions from an image and describe their features. They then perform region-based classification and finally convert the region-based predictions into pixel-level predictions, labeling pixels using the region containing the pixel with the highest classification score. This method requires generating a large number of candidate regions, consuming significant time and memory. Using this method to label line segments in an image can cause pixels in the line segments and their edges to overlap, resulting in poor labeling results and excessive complexity and tedium.

[0006] Commonly used semantic segmentation models:

[0007] U-Net model: The main execution process of the U-Net model is similar to other semantic segmentation models, and it can adapt to very small training sets. The structure of the U-Net model is as follows Figure 1 As shown, it includes two parts, the first part is the feature extraction part on the left, and the second part is the upsampling part on the right, which is the encoder-decoder architecture.

[0008] U-Net++ model: U-Net++ model is as follows Figure 2As shown in the figure, this is an improvement to the U-Net model. It alleviates the problem of unknown network depth by effectively integrating U-Net model networks of different depths. During model training, pruning operations can also be used to reduce the number of parameters of the U-Net++ model.

[0009] Existing methods for labeling the pixels that make up table lines in images are overly cumbersome and complex. Furthermore, testing of the U-Net and U-Net++ models reveals that U-Net++ performs better than the U-Net model at recognizing thick lines, recognizing them more completely. However, the U-Net model performs better with thinner lines. Summary of the Invention

[0010] The purpose of the present invention is to address the problem that the test results of the U-Net and U-Net++ models in the prior art each have their own advantages and disadvantages, and the marking method of table lines in images is too cumbersome and complicated. A method for constructing a picture table line extraction model and a picture table extraction method are proposed, thereby reducing the difficulty of marking the pixels of the table lines. At the same time, the advantages of the U-Net and U-Net++ models are combined to more accurately extract the graphic table lines.

[0011] To achieve the above-mentioned object of the invention, the present invention provides a method for constructing a line extraction model for an image table, comprising the following steps:

[0012] Step 1: Select training data; analyze the image containing the table, mark each table line with a line segment of preset width, and mark the pixels on the line segment; the unmarked part is regarded as the background;

[0013] Step 2: Use the training data to construct a data set; for any data set, randomly split it into a training set and a test set;

[0014] Step 3: Build a U-Net model, a U-Net++ model, or an integrated model of U-Net and U-Net++, and train the model using the training set. Stop training when the model prediction performance reaches the preset value after testing on the test set, and the image table line extraction model is built.

[0015] Specifically, in step 1, when the table line to be marked is a horizontal line or a vertical line, it is marked using a straight line segment of preset width; when marking, the pixel points at the beginning and end of the line segment are used to complete the pixel point marking of the entire line segment.

[0016] Furthermore, in step 1, the preset width is a single pixel width.

[0017] Furthermore, the thick table line is marked with a plurality of straight line segments, and the width of the thick table line is greater than a preset width of the straight line segment.

[0018] Furthermore, in step 3, when the constructed model is an integrated model of U-Net and U-Net++, the outputs of the U-Net model and the U-Net++ model are directly added or weightedly added as the final output result of the model.

[0019] Furthermore, the model adopts a weighted binary cross entropy loss function, which is formulated as follows:

[0020]

[0021] Where, L weighted : Loss function that distinguishes the weights of positive and negative samples; N: total number of samples; β: weight of positive samples; α: weight of negative samples; y i : represents the label of sample i; p i : Indicates the probability that sample i is predicted to be positive.

[0022] The present invention also provides a method for extracting a picture table, comprising the following steps:

[0023] Step 1: Analyze the layout of the image and extract the table area;

[0024] Step 2: constructing a picture table extraction model constructed by the picture table line extraction model construction method, applying the model to extract table lines from each table area, determining the category of the pixels on the table lines in the area, and thus determining the table lines;

[0025] Step 3: Optimize the table lines;

[0026] Step 4: Extract text information;

[0027] Step 5: Generate an Excel table based on the optimized table lines and text information.

[0028] Specifically, in step 3, optimizing the table lines includes:

[0029] Preset the horizontal line angle deviation threshold; remove the horizontal lines that exceed the horizontal line angle deviation threshold;

[0030] Preset vertical line angle deviation threshold; remove vertical lines that exceed the vertical line angle deviation threshold.

[0031] Furthermore, in step 3, optimizing the table lines includes:

[0032] Set the shortest horizontal distance threshold between two line segments on the same horizontal line; when the distance between two line segments on the same horizontal line is less than the shortest horizontal distance threshold, connect the two line segments;

[0033] Set the shortest vertical distance threshold between two line segments on the same vertical line; when the distance between two line segments on the same vertical line is less than the shortest vertical distance threshold, the two line segments are connected.

[0034] Furthermore, in step 3, optimizing the table lines includes: merging line segments with a distance of less than 15 pixels in the same direction.

[0035] The beneficial effect of the present invention is that the present invention uses line segments to mark and classify each table line. When the marked table line is a horizontal line or a vertical line, the beginning and end of the straight line segment are used for marking, which greatly reduces the difficulty of marking the pixel points. In the area where the line segments are thicker, multiple line segments are used for marking, so that as many pixels in the area as possible are marked as pixels on the line segments, rather than being marked as background pixels, thereby reducing the error of the training data during training and improving the accuracy of the model. Based on the deep learning theory, an integrated model of U-Net and U-Net++ is established, and the output probabilities of U-Net and U-Net++ are directly added or weightedly added as the output result of the model superposition. The advantages of the U-Net model and the U-Net++ model can be combined into one, so that the training model has a better recognition effect on both thick and thin line ends. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a structural diagram of the U-Net model in the background technology of the present invention.

[0037] Figure 2 This is a structural diagram of the U-Net++ model in the background technology of the present invention.

[0038] Figure 3 This figure compares the recognition results of the U-Net++ model with those of the U-Net model for an image with dense table lines in Example 1 of the present invention.

[0039] Figure 4 The recognition results of the U-Net model, the U-Net++ model, and the superposition of the recognition results of the two models in Example 1 of the present invention are shown.

[0040] Figure 5 This is a diagram of the model probability output results in Example 3 of the present invention.

[0041] Figure 6 This is a graph showing the table line extraction results in Example 3 of the present invention.

[0042] Figure 7 This is a result diagram of removing excessively offset table lines in Example 3 of the present invention.

[0043] Figure 8 This is a diagram showing the result of the dotted line connection in Example 3 of the present invention.

[0044] Figure 9 This is the result graph of merging adjacent line segments in Example 3 of the present invention.

[0045] Figure 10 This is a graph showing the result of completing the intersection points of the table lines in Example 3 of the present invention.

[0046] Figure 11 This is a graph showing the results of extending or shortening the table lines in Example 3 of the present invention. DETAILED DESCRIPTION

[0047] The technical solution of the present invention is described in detail below with reference to embodiments.

[0048] Example 1

[0049] This example builds an image table extraction model, including the following steps:

[0050] Step 1: Select training data; build a dataset.

[0051] Traditional semantic segmentation methods based on candidate regions first extract free-form regions from an image and describe their features. They then perform region-based classification and finally convert the region-based predictions into pixel-level predictions, labeling pixels using the region containing the pixel with the highest classification score. This method requires generating a large number of candidate regions, consuming significant time and memory. Using this method to label line segments in an image can cause pixels in the line segments and their edges to overlap, resulting in poor labeling results and excessive complexity and tedium.

[0052] This example analyzes an image containing a table, annotating each line with a line segment of a preset width. The pixels along the line segment are also marked, while the unmarked areas are considered background. Using line segments for labeling replaces the original image segmentation model's concept of distinguishing different blocks, effectively reducing the complexity and tediousness of pixel labeling.

[0053] Preferably, when the table line to be annotated is a horizontal or vertical line, it is annotated using a straight line segment; when marking, the pixels at the beginning and end of the line segment are used to complete the pixel labeling of the entire line segment. During training, increasing the weight of the pixels on the line segment makes the model more biased towards the recognition of pixels on the line segment, which can further improve the accuracy of the model.

[0054] As can be seen from the use of line segments instead of blocks for marking, in this case, the thinner the line segments on the image, the more correctly marked pixels in the marking information, and the better the model training effect. If the model table line is too thick, forming a rectangular block, using a line segment as the marker for this block will cause all pixels in this block except those on the line segment to be marked as non-line segment areas, increasing the error during training. The model is prone to classifying pixels in the line segment area as background pixels. Although the training weight of pixels marked as line segments can be increased, the recognition effect on thicker line segment areas is still not very good.

[0055] To address this issue, this example uses multiple straight line segments to mark thick table lines. Thick table lines are wider than the preset width of straight line segments. This ensures that as many pixels within the area as possible are marked as line segments rather than background pixels, reducing errors in the training data and further improving model accuracy.

[0056] Theoretically, the annotation effect is best when as many annotation lines as the number of pixels occupied by the table line width is used. Therefore, the straight line segment width can be preset to a single pixel width.

[0057] Step 2: Use the training data to construct a dataset; for any dataset, randomly split it into a training set and a test set.

[0058] Step 3: Build a U-Net model, a U-Net++ model, or an integrated model of U-Net and U-Net++, and train the model using the training set. Stop training when the model prediction performance reaches the preset value after testing on the test set, and the image table line extraction model is built.

[0059] The differences between the U-Net model and the U-Net++ model are as follows:

[0060] U-Net++ is better at recognizing thick lines than the U-Net model. It recognizes thick line areas in the image more completely. In the recognition results of the U-Net model, many pixels in the thick line area cannot be classified as pixels on the line segment.

[0061] The U-Net model handles thinner lines better, while the U-Net++ model classifies the pixels around the thinner lines as pixels on the line segments, which increases the error of the recognition results but has little effect on the results of line segment recognition.

[0062] For dense lines, such as Figure 3As shown, the U-Net model's recognition performance is not very good. Areas with dense table lines are misidentified as background, resulting in many lines not being recognized. In contrast, the U-Net++ model has a higher recognition completeness rate. If there are multiple types of line segments, the recognition results of multiple models can be integrated to enhance recognition. You can choose to build different models based on your specific needs to improve recognition accuracy.

[0063] When building an integrated model of U-Net and U-Net++, if the two model networks are integrated to form a single model network and then trained using training data, the training speed will be slow and the trained U-Net model and U-Net model file will not be utilized. However, directly adding or weighted adding the output probabilities of U-Net and U-Net++ as the final output of the model can effectively combine the advantages of the two models in recognizing thick and thin line segments, effectively enhancing the recognition effect. Figure 4 shown.

[0064] Example 2

[0065] This example is based on the model constructed in Example 1 and describes the selection of its loss function as follows:

[0066] (1) Both U-Net and U-Net++ models are image semantic segmentation models, which are pixel-level classification tasks. The most commonly used loss function for image semantic segmentation tasks is the cross entropy loss function, which is calculated as:

[0067] rowloss=-∑y true log(y pred )

[0068] From this cross entropy loss function, we can see that when y true is 0, that is, the loss of pixels not marked as horizontal lines in the input is 0, only when y true When it is 1, that is, when the pixels in the input are marked as horizontal lines, there will be a loss value. Therefore, if this cross entropy loss function is used for calculation, the loss of most pixels will be discarded, resulting in poor model training results.

[0069] (2) In order to solve the above problem, the binary cross entropy loss function, namely BCE Loss, is introduced. The loss function formula is:

[0070] bceloss=-∑[y true logy pred +(1-y true )log(1-y pred )]

[0071] From this formula we can see that when y true When it is 0, the loss value is -log(1-y pred ), when y true When it is 1, the loss value remains unchanged. In this case, the loss of all pixels is included, and then the loss of all pixels is averaged, so in essence, every pixel in the image is learned equally.

[0072] However, in the current task, the proportion of pixels marked as horizontal or vertical lines in the total pixels in the image is too low, and the category distribution in the image is unbalanced. This may cause the training process to be dominated by the class with a large number of pixels, that is, the pixels not marked as horizontal or vertical lines dominate. The model will mainly learn the features of the samples of the large number of categories, and the learned model will be more biased towards predicting pixels as this category.

[0073] (3) To address the above problem, each value in the output probability distribution vector is weighted, hoping that the model can pay more attention to the smaller number of labeled pixels to alleviate the problem of class imbalance in the image, so that the model can learn the features of more labeled pixels. Therefore, the binary cross entropy loss function is changed to a weighted binary cross entropy loss function, and the formula is:

[0074]

[0075] L weighted : Loss function that distinguishes the weights of positive and negative samples; N: total number of samples; β: weight of positive samples, which defaults to 1, but can also be slightly higher or lower than 1; α: weight of negative samples, which defaults to 1, but can also be slightly higher or lower than 1; y i : represents the label of sample i, the positive class is 1 and the negative class is 0; p i : Indicates the probability that sample i is predicted to be positive.

[0076] In this formula, the weighted cross entropy loss function simply adds a weight parameter to each category on top of the BCE loss to weight positive samples. When β > 1, it prioritizes the loss of labeled pixels, improving the recognition of pixels on horizontal or vertical lines. Finally, the total loss is calculated by taking the average of the weighted losses for each category.

[0077] Example 3

[0078] This example provides a method for extracting information from images and tables. The deep learning-based image and table information extraction process includes the following steps:

[0079] Step 1: Analyze the image layout and extract the table area.

[0080] For a picture, it may contain one or more tables, or it may not contain a table, so it is necessary to first find the table area in the picture and then extract the table lines and reconstruct the table.

[0081] Step 2: Construct any one of the image table extraction models in Example 1 and apply it to each table area for table extraction, determine the category of each pixel in the area, and determine the table lines. The specific model selection has been described in detail in Example 1 and will not be repeated here.

[0082] This example uses an integrated model based on U-Net and U-Net++, with the output probabilities of U-Net and U-Net++ directly added or weighted added as the final output of the model. Whether it is the U-Net model or the U-Net++ model, their output is the classification probability of each pixel being classified as a horizontal line or a vertical line. The data results of the model are saved in a graph, with one layer for the probability of classification as a horizontal line and one layer for the probability of classification as a vertical line, as shown in the figure. Figure 5 As shown, the table extraction diagram is as follows Figure 6 shown.

[0083] Step 3: Optimize the table lines; this may include the following methods:

[0084] 1. Remove messy lines: preset the horizontal line angle offset threshold; remove horizontal lines that exceed the horizontal line angle offset threshold; preset the vertical line angle offset threshold; remove vertical lines that exceed the vertical line angle offset threshold. Figure 7 This is the result of removing line segments that differ from most horizontal or vertical lines by more than 2°.

[0085] 2. Connect the intermittent lines and determine the connected domain: set the shortest horizontal distance threshold between two line segments on the same horizontal line; when the distance between two line segments on the same horizontal line is less than the shortest horizontal distance threshold, connect the two line segments; set the shortest vertical distance threshold between two line segments on the same vertical line; when the distance between two line segments on the same vertical line is less than the shortest vertical distance threshold, connect the two line segments. Figure 8 shown.

[0086] 3. In order to better process the table line data, it needs to be converted back to its original size.

[0087] 4. Merge overlapping or close segments. In this example, segments with a distance of less than 15 pixels in the same direction are considered overlapping or close segments. Figure 9 shown.

[0088] 5. Process the area at the intersection of each horizontal and vertical line to determine whether there is a situation where they should intersect but do not, and extend the line segment so that the horizontal and vertical lines completely intersect, and fill in the intersection points of the table lines. Figure 10 shown.

[0089] 6. The table line is extended or shortened. After processing, Figure 11 shown.

[0090] 7. Surround the outermost edge of the table with a rectangle and remove the overlapping line segments to complete the adjustment of the table lines. This will ensure that there will be no problems with the cell correspondence when the data is saved in the Excel file.

[0091] Step 4: Extract text information; In order to extract the information in each cell in the table, it is necessary to identify the text information in the cell based on OCR technology.

[0092] Step 5: Generate an Excel table based on the optimized table lines and text information. So far, the extraction and storage of image table information has been completed.

Claims

1. A method for constructing a line extraction model for an image table, characterized in that: The following steps are involved: Step 1: Select training data; analyze the image containing the table, mark each table line with a line segment of preset width, and mark the pixels on the line segment; the unmarked part is regarded as the background; Step 2: Use the training data to construct a data set; for any data set, randomly split it into a training set and a test set; Step 3: Build a U-Net model, a U-Net++ model, or an integrated model of U-Net and U-Net++, and train the model using the training set. Stop training when the model prediction performance reaches the preset value after testing on the test set. The image table line extraction model is built. In step 1, when the table line to be marked is a horizontal line or a vertical line, a straight line segment of a preset width is used to mark it; when marking, the pixel points at the beginning and end of the line segment are used to complete the pixel marking of the entire line segment; In step 1, the preset width is a single pixel width; A thick table line is marked with a plurality of straight line segments, wherein the width of the thick table line is greater than a preset width of the straight line segment; In step 3, when the constructed model is an integrated model of U-Net and U-Net++, the outputs of the U-Net model and the U-Net++ model are directly added or weightedly added as the final output result of the model; The model uses a weighted binary cross entropy loss function, the formula is: Where, L weighted : Loss function that distinguishes the weights of positive and negative samples; N: total sample size; β: weight of positive samples; α: weight of negative samples; y i : represents the label of sample i; p i : Indicates the probability that sample i is predicted to be positive.

2. A method for extracting a picture table, characterized in that: The following steps are involved: Step 1: Analyze the layout of the image and extract the table area; Step 2: constructing an image table extraction model constructed by the method of claim 1, applying the model to extract table lines from each table area, determining the categories of pixels on the table lines in the area, and thereby determining the table lines; Step 3: Optimize the table lines; Step 4: Extract text information; Step 5: Generate an Excel table based on the optimized table lines and text information.

3. A method for extracting a picture table according to claim 2, characterized in that: In step 3, optimizing the table lines includes: Preset the horizontal line angle deviation threshold; remove the horizontal lines that exceed the horizontal line angle deviation threshold; Preset vertical line angle deviation threshold; remove vertical lines that exceed the vertical line angle deviation threshold.

4. A method for extracting a picture table according to claim 3, characterized in that: In step 3, optimizing the table lines includes: Set the shortest horizontal distance threshold between two line segments on the same horizontal line; when the distance between two line segments on the same horizontal line is less than the shortest horizontal distance threshold, connect the two line segments; Set the shortest vertical distance threshold between two line segments on the same vertical line; when the distance between two line segments on the same vertical line is less than the shortest vertical distance threshold, the two line segments are connected.

5. A method for extracting a picture table according to claim 4, characterized in that: In step 3, optimizing the table lines includes merging line segments whose distance in the same direction is less than 15 pixels.

Images