Table positioning method, device and related equipment

Abstract

The present application discloses a table positioning method, including obtaining sample data containing table information; performing positive and negative sample balancing processing on the sample data to obtain processed sample data; using the processed sample data to train a table positioning model; using the table positioning model to perform table positioning on a document to be processed to obtain each table prediction frame; using the Confluence algorithm based on intersection-over-union ratio to screen all the table prediction frames to obtain the optimal table prediction frame. The technical solution provided by the present application can effectively improve the positioning accuracy of the document table. The present application also discloses a table positioning device, equipment and computer-readable storage medium, all of which have the above-mentioned beneficial effects.

Description

Technical Field

[0001] The present application relates to the technical field of document processing, and in particular to a table positioning method, and also to a table positioning device, equipment and computer-readable storage medium. Background Art

[0002] With the massive growth of PDF documents in the financial industry, traditional manual processing methods require a lot of time and effort and can no longer meet daily work needs. In order to improve the review efficiency of financial documents, financial companies have begun to embark on the road of intelligence, especially the electronicization and intelligence of disclosure-type financial documents has become a trend. Among them, table parsing is an important branch of the intelligence of disclosure-type financial documents. The process of table parsing mainly includes table positioning, table restoration and content extraction, among which table positioning is the crucial first step. However, due to the variety of table designs and types, and rich features, the existing single-mode recognition method cannot effectively guarantee the positioning accuracy of the table.

[0003] Therefore, how to effectively improve the positioning accuracy of document tables is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0004] The purpose of the present application is to provide a table positioning method, which can effectively improve the positioning accuracy of the document table; another purpose of the present application is to provide a table positioning device, equipment and computer-readable storage medium, all of which have the above-mentioned beneficial effects.

[0005] In a first aspect, the present application provides a table positioning method, comprising:

[0006] Get sample data containing table information;

[0007] Performing positive and negative sample balancing processing on the sample data to obtain processed sample data;

[0008] Using the processed sample data to train and obtain a table positioning model;

[0009] Using the table positioning model to locate the table in the document to be processed, and obtaining prediction boxes of each table;

[0010] The optimal table prediction box is obtained by screening all the table prediction boxes using the Confluence algorithm based on intersection-over-union ratio.

[0011] Preferably, performing positive and negative sample balancing processing on the sample data to obtain processed sample data includes:

[0012] Generate each preset anchor frame according to the sample data, and calculate the intersection-over-union ratio between each preset anchor frame and the real frame;

[0013] The first intersection-over-union ratio threshold is calculated according to the positive sample selection formula, and the positive sample selection formula is:

[0014] IoU1 = m + ln(1-v);

[0015] Wherein, the IoU1 is the first intersection-over-union ratio threshold, m represents the mean of the intersection-over-union ratio between the real box and the preset anchor box, and v represents the variance of the intersection-over-union ratio between the real box and the preset anchor box;

[0016] Taking the preset anchor frame whose IoU ratio exceeds the first IoU ratio threshold as a positive sample;

[0017] The preset anchor frames whose IoU ratio does not exceed the first IoU ratio threshold are taken as negative samples.

[0018] Preferably, the table positioning method further includes:

[0019] Determine each prediction box according to the sample data;

[0020] A preset number of predicted boxes that are closest to the center point of the true box are used as the positive samples.

[0021] Preferably, after taking the preset anchor frame whose IoU ratio does not exceed the first IoU ratio threshold as a negative sample, the method further includes:

[0022] Among all the negative samples, the negative samples whose IoU ratio exceeds a second IoU ratio threshold are deleted.

[0023] Preferably, the method of using the Confluence algorithm based on intersection-over-union ratio to select and obtain the optimal table prediction frame from all the table prediction frames includes:

[0024] Generate a first prediction frame set based on all the table prediction frames;

[0025] Select any one table prediction box in the first prediction box set as a first table prediction box, calculate an intersection-and-union ratio between the first table prediction box and other table prediction boxes, and delete other table prediction boxes and the first table prediction box whose intersection-and-union ratios are zero, to obtain a second prediction box set;

[0026] Calculating a confidence weighted intersection-and-union ratio of the first table prediction frame and each table prediction frame in the second prediction frame set, and taking a table prediction frame whose confidence weighted intersection-and-union ratio exceeds a first threshold as the optimal table prediction frame;

[0027] Calculating an intersection-and-union ratio between the optimal table prediction frame and other table prediction frames in the second prediction frame set, and deleting other table prediction frames whose intersection-and-union ratio exceeds a second threshold, to obtain a third prediction frame set;

[0028] The first table prediction box is deleted from the first prediction box set, and the step of selecting any table prediction box in the first prediction box set as the first table prediction box is returned until the first prediction box set is empty, and all the optimal table prediction boxes are obtained.

[0029] Preferably, the step of training the processed sample data to obtain a table positioning model includes:

[0030] Combine the Involution algorithm and the single-stage feature detector to build the initial network model;

[0031] The processed sample data is used to perform model training on the initial network model to obtain a table positioning model with a preset evaluation index lower than a preset threshold; wherein the preset evaluation index is specifically the alignment accuracy between the real frame and the predicted frame.

[0032] Preferably, before performing positive and negative sample balancing processing on the sample data to obtain the processed sample data, the method further comprises:

[0033] The tables in the sample data are copied and deleted according to the table row and column layout to obtain sample data after data enhancement.

[0034] In a second aspect, the present application further discloses a table positioning device, comprising:

[0035] A sample acquisition module is used to obtain sample data containing table information;

[0036] A sample balancing module, used to perform positive and negative sample balancing processing on the sample data to obtain processed sample data;

[0037] A model training module, used for training and obtaining a table positioning model using the processed sample data;

[0038] A table positioning module, used to use the table positioning model to perform table positioning on the document to be processed and obtain prediction frames of each table;

[0039] The prediction box screening module is used to screen the optimal table prediction box from all the table prediction boxes using the Confluence algorithm based on intersection-over-union ratio.

[0040] In a third aspect, the present application further discloses a table positioning device, comprising:

[0041] Memory for storing computer programs;

[0042] A processor is used to implement the steps of any one of the table locating methods described above when executing the computer program.

[0043] In a fourth aspect, the present application further discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of the table locating methods described above are implemented.

[0044] The present application provides a table positioning method, comprising obtaining sample data containing table information; performing positive and negative sample balancing processing on the sample data to obtain processed sample data; using the processed sample data to train a table positioning model; using the table positioning model to perform table positioning on a to-be-processed document to obtain prediction frames of each table; and using a Confluence algorithm based on intersection-over-union ratio to screen all the table prediction frames to obtain the optimal table prediction frame.

[0045] By applying the technical solution provided in the present application, after obtaining a large amount of sample data, the sample data is first subjected to positive and negative sample balance processing to obtain high-quality sample data with positive and negative sample balance, and then these high-quality sample data are used for model training to obtain a table positioning model with high precision. Therefore, in the process of table positioning, the table positioning model can be directly used to locate each table prediction box from the document to be processed. Finally, the Confluence algorithm based on intersection and union ratio is used to filter out the optimal table prediction box from all table prediction boxes, thereby achieving the optimal result from the model prediction results, and then realizing table positioning based on the neural network model, which effectively improves the positioning accuracy of the document table.

[0046] The table positioning device, equipment and computer-readable storage medium provided in the present application all have the above-mentioned beneficial effects, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the prior art and the embodiments of the present application, the drawings required for use in the description of the prior art and the embodiments of the present application are briefly introduced below. Of course, the drawings described below in relation to the embodiments of the present application are only part of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work, and the obtained other drawings also belong to the protection scope of the present application.

[0048] Figure 1 A flowchart of a table positioning method provided in this application;

[0049] Figure 2A framework diagram of a table positioning model provided in this application;

[0050] Figure 3 A schematic diagram of the structure of the DardNet network in a table positioning model provided in this application;

[0051] Figure 4 A schematic diagram of the structure of the FPN+ network in a table positioning model provided in this application;

[0052] Figure 5 A schematic diagram of a table data enhancement method provided in this application;

[0053] Figure 6 A schematic diagram of the structure of a table positioning device provided in this application;

[0054] Figure 7 A schematic diagram of the structure of a table positioning device provided in this application. DETAILED DESCRIPTION

[0055] The core of the present application is to provide a table positioning method, which can effectively improve the positioning accuracy of the document table; another core of the present application is to provide a table positioning device, equipment and computer-readable storage medium, which also have the above-mentioned beneficial effects.

[0056] In order to describe the technical solutions in the embodiments of the present application more clearly and completely, the technical solutions in the embodiments of the present application will be introduced below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0057] An embodiment of the present application provides a table positioning method.

[0058] Please refer to Figure 1 , Figure 1 A flowchart of a table positioning method provided by the present application is shown below. The table positioning method may include:

[0059] S101: Obtain sample data including table information.

[0060] This step aims to achieve the acquisition of sample data. Since the technical solution provided by this application aims to achieve table positioning, the sample data is sample data containing table information. It can be understood that the sample data is used to implement model training to train a neural network model for achieving table positioning. Therefore, in order to achieve model training, the sample data should include positive sample data and negative sample data.

[0061] Of course, there is no unique way to obtain sample data, which can be achieved through a large amount of document collection, and this application does not limit this. It can be imagined that the more sample data there is, the more helpful it is to achieve more accurate model training, thereby obtaining a more accurate neural network model and further improving the table positioning accuracy.

[0062] S102: Perform positive and negative sample balancing processing on the sample data to obtain processed sample data.

[0063] This step aims to achieve balanced processing of positive and negative samples to effectively ensure the balance of positive and negative samples in the sample data, thereby effectively improving the accuracy of the model. It can be understood that the balanced processing of positive and negative samples here specifically refers to achieving a state of balanced quantity of positive sample data and negative sample data in the sample data. For example, when the number of positive sample data is large and the number of negative sample data is small, the number of negative sample data can be increased; when the number of positive sample data is small and the number of negative sample data is large, the number of positive sample data can be increased.

[0064] Among them, the specific implementation method of the positive and negative sample balance processing is not unique, and the present application does not limit this as long as the positive and negative samples are balanced. For example, the ATSS (Adaptive Training Sample Selection, an adaptive training sample selection method) algorithm can be used for implementation.

[0065] S103: Using the processed sample data to train and obtain a table positioning model;

[0066] This step is intended to implement model training to obtain a table positioning model, which is used to implement table prediction and further obtain the table prediction frame. Specifically, on the basis of obtaining sample data with balanced processing of positive and negative samples, these high-quality sample data can be used for model training to obtain a table positioning model that meets the requirements. It can be understood that the specific type of the table positioning model does not affect the implementation of the present technical solution, as long as table prediction can be implemented. For example, it can be a neural network model based on the Involution algorithm, etc., and the present application does not limit this. In addition, the training process of the table positioning model can refer to the existing technology, and the present application will not repeat it here.

[0067] S104: Using the table positioning model to locate the table in the document to be processed, and obtaining prediction boxes of each table;

[0068] This step aims to realize table positioning using the table positioning model, so as to predict various table prediction boxes from the document to be processed. Among them, the document to be processed is the document that needs to be positioned for the table. Its specific type does not affect the implementation of the present technical solution, and this application does not limit this. For example, it can be a PDF document, a WORD document, etc. Of course, the source of the document to be processed is not unique. It can be directly input by the user, or it can be downloaded from the network according to the user's instructions. This application also does not limit this. Specifically, after obtaining the document to be processed, it can be input into the table positioning model, and the model output is the prediction boxes of each table in the document to be processed.

[0069] S105: Filter all table prediction frames to obtain the optimal table prediction frame using the Confluence algorithm based on intersection-over-union ratio.

[0070] This step aims to implement table prediction box screening, so as to screen out the optimal table prediction box from all table prediction boxes output by the table positioning model, thereby achieving the best result from the model prediction results, which can effectively ensure the accuracy of the table positioning results. It can be understood that for a document to be processed, the number of tables contained therein is determined by the document content, that is, the number of tables actually contained in the document to be processed is not unique. On this basis, when screening the optimal table prediction box, the number of optimal table prediction boxes obtained is also not unique.

[0071] Among them, the screening process of the optimal table prediction box can be implemented using the Confluence algorithm based on intersection-over-union ratio. Specifically, the Confluence algorithm proposes to use Manhattan distance to measure the overlap degree of two sets of bounding boxes. The formula is defined as:

[0072]

[0073]

[0074]

[0075] Among them, p (i,j) Indicates the degree of overlap between two sets of bounding boxes (bounding box i and bounding box j), MH represents the Manhattan distance, i u represents the coordinates of the upper left corner of the bounding box i (x1, y1), j v represents the coordinates of the upper left corner of the bounding box j (p1, q1), i m represents the coordinates of the lower right corner of the bounding box i (x2, y2), j n Represents the coordinates of the lower right corner of the bounding box j (p2, q2).

[0076] However, due to the normalized bounding box in the Confluence algorithm i,j After filtering of <2, there are still outliers, which may affect the selection of the optimal bounding box in the group. All coordinates of the bounding box in Confluence are normalized to the maximum and minimum, and they are located between (0,1). The Confluence algorithm believes that p i,j < 2 bounding box pairs are intersecting, however, in the special case, there exists p i,j Based on this, since there is no overlap in the disclosure document table, the Confluence+ algorithm chooses to introduce IoU (intersection over union) to replace Manhattan distance to solve the above outlier problem of the Confluence algorithm, that is, the Confluence algorithm based on intersection over union is used for model training.

[0077] It can be understood that the advantages of the Confluence algorithm based on intersection-over-union ratio are as follows: on the one hand, it can effectively ensure that the bounding box pairs in the filtered clusters are intersecting with each other, which improves the consistency of the bounding boxes, while removing the normalization operation and streamlining the post-processing algorithm; on the other hand, the algorithm no longer relies solely on the confidence score, but uses the confidence-weighted IoU value to select the optimal bounding box, which can effectively reduce false positives and improve detection accuracy.

[0078] It should be noted that the above S101 to S103 are the process of obtaining the table positioning model, which only needs to be executed once. In the actual application process, the trained table positioning model can be directly called to process the document to be processed, wherein the trained table positioning model can be pre-stored in the corresponding storage space for direct calling. Of course, in order to effectively ensure the accuracy of the model and the accuracy of the table positioning results, the model can also be optimized regularly / irregularly.

[0079] It can be seen that the table positioning method provided in the present application, after obtaining a large amount of sample data, first performs positive and negative sample balance processing on the sample data to obtain high-quality sample data with positive and negative sample balance, and then uses these high-quality sample data to train the model to obtain a table positioning model with high precision. Therefore, in the table positioning process, the table positioning model can be directly used to locate each table prediction box from the document to be processed. Finally, the Confluence algorithm based on intersection and union ratio is used to screen all the table prediction boxes to obtain the optimal table prediction box, thereby realizing the optimal result from the model prediction results, and then realizing the table positioning based on the neural network model, which effectively improves the positioning accuracy of the document table.

[0080] In one embodiment of the present application, the above-mentioned performing positive and negative sample equalization processing on the sample data to obtain the processed sample data may include: generating each preset anchor frame according to the sample data, and calculating the intersection and union ratio of each preset anchor frame and the real frame; calculating the first intersection and union ratio threshold according to the positive sample selection formula, and the positive sample selection formula is:

[0081] IoU1 = m + ln(1-v);

[0082] Among them, IoU1 is the first intersection-over-union ratio threshold, m represents the mean of the intersection-over-union ratio between the real box and the preset anchor box, and v represents the variance of the intersection-over-union ratio between the real box and the preset anchor box;

[0083] The preset anchor frames whose IoU ratio exceeds the first IoU ratio threshold are taken as positive samples; the preset anchor frames whose IoU ratio does not exceed the first IoU ratio threshold are taken as negative samples.

[0084] The embodiment of the present application provides a method for implementing positive and negative sample balance processing. As described above, the ATSS algorithm can be used to implement positive and negative sample balance processing. The ATSS algorithm can calculate the sum of the mean and the standard deviation based on the IoU values ​​of the preset anchor box and the real box on different feature layers as the IoU threshold for selecting positive samples, and select the preset anchor box with an IoU greater than the threshold as the positive sample, and set the remaining preset anchor boxes as negative samples; then set the IoU threshold for negative samples, and change the predicted anchor box with an IoU greater than the threshold to an ignored sample. It can be seen that the ATSS algorithm does not need to specify a threshold for judgment, but selects positive samples by dynamically and automatically generating a threshold, which realizes the function of adaptively balancing samples to a certain extent.

[0085] However, when the IoU value between the real box and the preset anchor box is extremely unbalanced, the sum of the mean and the standard deviation will be too large, resulting in no preset anchor box meeting the conditions. Based on this, in order to further increase the number of positive samples, the ATSS algorithm can be optimized, and the above improved positive sample selection formula is proposed. This formula is smoother than the formula before optimization, and can better handle the situation of uneven distribution of IoU values, thereby better selecting positive samples on different feature layers.

[0086] Therefore, after determining each preset anchor frame based on the sample data and calculating the IoU between each preset anchor frame and the real frame, the first intersection-over-union threshold can be calculated based on the above formula. The first intersection-over-union threshold is the IoU threshold used to select positive sample data. Furthermore, when the IoU between the preset anchor frame and the real frame exceeds the first intersection-over-union threshold, the preset anchor frame can be used as a positive sample. When the IoU between the preset anchor frame and the real frame does not exceed the first intersection-over-union threshold, the preset anchor frame can be used as a negative sample.

[0087] In one embodiment of the present application, the table positioning method may further include: determining each prediction box according to the sample data; and taking a preset number of prediction boxes that are closest to the center point of the real box as positive samples.

[0088] After the sample data is processed by the previous embodiment for positive and negative sample balance, the number of negative sample data is still much larger than the positive sample data. Therefore, the positive sample data can be expanded to obtain more positive sample data. Specifically, each prediction box in the sample data can be determined first, and then the distance between the center point of each prediction box and the center point of the real box can be calculated. Finally, a certain number of prediction boxes with the closest distance are used as positive samples to further increase the number of positive samples. It can be understood that the closer the center point distance is, the greater the possibility that the prediction box is a real box.

[0089] In one embodiment of the present application, after taking the preset anchor frames whose IoU ratio does not exceed the first IoU ratio threshold as negative samples, the method may further include: deleting the negative samples whose IoU ratio exceeds the second IoU ratio threshold from all negative samples.

[0090] In addition to expanding the positive sample data to obtain more positive sample data, the negative sample data can also be screened with reference to the above-mentioned ATSS algorithm to reduce the number of negative samples to achieve a balance between positive and negative samples. Specifically, after taking the preset anchor frame whose IoU does not exceed the first IoU threshold as a negative sample, a second IoU threshold can be further set among all negative samples, that is, the IoU threshold for screening negative samples, and then the negative samples whose IoU exceeds the second IoU threshold are deleted, that is, the negative samples whose IoU exceeds the second IoU threshold are modified to be ignored samples, so as to achieve the purpose of reducing the number of negative samples and improving the quality of negative samples.

[0091] In addition, the specific values ​​of the first intersection-and-union ratio threshold and the second intersection-and-union ratio threshold do not affect the implementation of the present technical solution and can be set by technical personnel according to actual conditions, and this application does not limit this.

[0092] In one embodiment of the present application, the above-mentioned use of the Confluence algorithm based on the intersection-and-union ratio to screen and obtain the optimal table prediction frame from all table prediction frames may include: generating a first prediction frame set based on all table prediction frames; selecting any table prediction frame in the first prediction frame set as the first table prediction frame, calculating the intersection-and-union ratio of the first table prediction frame with other table prediction frames, and deleting other table prediction frames and the first table prediction frame with an intersection-and-union ratio of zero to obtain a second prediction frame set; calculating the confidence-weighted intersection-and-union ratio of the first table prediction frame with each table prediction frame in the second prediction frame set, and taking the table prediction frame whose confidence-weighted intersection-and-union ratio exceeds a first threshold as the optimal table prediction frame; calculating the intersection-and-union ratio of the optimal table prediction frame with other table prediction frames in the second prediction frame set, and deleting other table prediction frames whose intersection-and-union ratio exceeds a second threshold to obtain a third prediction frame set; deleting the first table prediction frame from the first prediction frame set, and returning to the step of selecting any table prediction frame in the first prediction frame set as the first table prediction frame until the first prediction frame set is empty, and all optimal table prediction frames are obtained.

[0093] The embodiment of the present application provides a method for selecting an optimal table prediction frame. Specifically, after obtaining each table prediction frame in the document to be processed based on the table positioning model, a prediction frame set is generated based on these table prediction frames, that is, the first prediction frame set mentioned above. Assume that the first prediction frame set is B = {b1, b2, ..., b n}, traverse the first prediction box set and select any table prediction box b from it i As the first table prediction box, and for each group (b i ,b j )(j ranges from 1 to n and j≠i) calculate IoU i,j value, IoU i,j The table prediction box with a value of 0 is deleted, and the first table prediction box b is i After the elimination, the predicted box set after the initial screening is obtained, that is, the second preset box set mentioned above; further, assuming that the second preset box set is B', for each predicted box in the second preset box set B', calculate its difference with the predicted box b in the first table i The confidence weighted intersection-and-union ratio, the confidence weighted intersection-and-union ratio calculation formula is:

[0094]

[0095]

[0096] Among them, (b i ,b j ) represents two sets of prediction boxes, c represents the confidence score, WP (i,j) represents the confidence weighted intersection-over-union ratio.

[0097] Thus, after obtaining each confidence weighted intersection-and-union ratio, it is compared with a preset threshold, that is, the above-mentioned first threshold. If it exceeds the first threshold, the corresponding table prediction frame (the table prediction frame within the second preset frame set B', not the first table prediction frame) can be regarded as the "optimal" table prediction frame. On this basis, the intersection-and-union ratio of the optimal table prediction frame and the other table prediction frames in the second prediction frame set B' is calculated, and the other table prediction frames whose intersection-and-union ratio exceeds the second threshold are deleted to remove other table prediction frames that are highly overlapped with the optimal table prediction frame, and obtain the third prediction frame set. Further, the first table prediction frame is deleted from the first prediction frame set, and any table prediction frame is reselected from the first prediction frame set where the first table prediction frame is deleted as the new first table prediction frame, and the above steps are repeated until the first prediction frame set is traversed, that is, when the first prediction frame set is empty, all optimal table prediction frames are obtained.

[0098] Similarly, the specific values ​​of the first threshold and the second threshold do not affect the implementation of the present technical solution and can be set by technical personnel according to actual conditions, and this application does not impose any limitations on this.

[0099] In one embodiment of the present application, the above-mentioned training of the table positioning model using the processed sample data may include: constructing an initial network model by combining the Involution algorithm and the single-stage feature detector; performing model training on the initial network model using the processed sample data to obtain a table positioning model with a preset evaluation index lower than a preset threshold; wherein the preset evaluation index is specifically the alignment accuracy between the real box and the predicted box.

[0100] The embodiment of the present application provides a specific type of table location model, namely, a table location model based on the Involution algorithm and a single-stage feature detector. Specifically, the initial network model is first constructed using the Involution algorithm and the single-stage feature detector, and then the initial network model is trained using the high-quality sample data after the above sample equalization processing, and combined with the pre-set evaluation index, a table location model that meets the requirements is obtained, that is, a table location model whose pre-set evaluation index is lower than the pre-set threshold.

[0101] On this basis, the present application embodiment provides a table positioning model with a specific framework structure. Figure 2 , Figure 2 This is a framework diagram of a table positioning model provided in this application. The specific contents of the network structure are as follows:

[0102] 1. Backbone part (backbone network):

[0103] In the table detection task, in addition to the basic features of lines and text, there are also important features such as the relationship between rows and columns, indentation, cell background color, etc. However, the traditional convolution has spatial invariance, which deprives the convolution kernel of the ability to learn different visual information at different spatial positions. In order to fully extract the spatial layout features of the table, the involution operator (a neural network operator) can be introduced into the backbone network, and the DardNet network is proposed. Figure 3 As shown, Figure 3 A schematic diagram of the structure of the DardNet network in a table positioning model provided in this application. The structure of the DardNet network consists of two parts, namely DardNnet Conv (DardNet convolution module) and Involution Residual (Involution residual module). DardnNet Conv is composed of a 7*7involution operator, 2 3*3 convolutions, a BatchNorm (regularization layer) and ReLU (activation function). Involution Residual is composed of 5 residual modules, among which the residual module is composed of 2 7*7involution operators and 1 1*1 convolution.

[0104] Since Involution generates the Involution kernel by convolving a single feature point pixel, it learns the spatial layout features of the table. The process of Involution generating the Involution convolution kernel is related to self-attention, which enables the interaction between pixels of interest and surrounding pixels and can capture long-distance relationships. In addition, the property of Involution to share convolution kernel parameters between channels makes it possible to use large convolution kernels, thereby improving the receptive field of the model, learning more contextual information, and obtaining the overall layout information of the table.

[0105] 2. Neck part (the part between the backbone network and the prediction layer):

[0106] Since the tables in disclosure documents are medium-to-large objects, the feature maps downsampled by 32 times are sufficient for table detection. Therefore, we can choose to use the feature map pyramid structure network FPN+ of a single-stage feature detector, with the input being the feature map downsampled by 32 times and the output being a 13*13 detection layer, thereby simplifying the network structure and accelerating the model convergence speed. Figure 4 As shown, Figure 4A structural diagram of the FPN+ network in a table positioning model provided in this application. In order to improve the speed while ensuring the detection effect, FPNs+ adopts an expanded residual module. The network mainly includes two blocks: Project layer (projection layer) and Dialated Residual block (void residual module). The Project layer is composed of a 1x1 and a 3x3 convolutional layer; the Dialated Residual block is composed of 5 residual blocks with different expansion rates.

[0107] Furthermore, for the table positioning model, its loss function is mainly composed of the bounding box regression loss function, the classification loss function and the confidence loss function. Among them, the bounding box regression loss function can choose GIoULoss, because the coordinate L2 norm is sensitive to the scale of the bounding box. For example, when the coordinate L2 norm of the large-scale prediction box is equal to the coordinate L2 norm of the small-scale prediction box, the IoU corresponding to the two may be very different. GIoU can not only reflect the actual distance when the prediction box has no intersection with the real box, but also optimize the problem of zero gradient of the IoU loss function. The formula is as follows:

[0108]

[0109] Among them, A and B represent two boxes respectively, C represents the union of box A and box B, and A∪B represents the intersection of box A and box B.

[0110] Then, the bounding box regression loss function based on GIoU is:

[0111] L GIoU =λ coord (1-GIoU);

[0112] λ coord =2-t w t h ;

[0113] Among them, L GIoU represents the bounding box regression loss function based on GIoU, GIoU represents an overlap measurement indicator, t w and t h Respectively represent the width and height of the feature map predicted by the model, λ coord Represents the weight coefficient of the bounding box regression loss function.

[0114] Due to λ coord =2-t w t h , then, when t w *t hThe smaller it is, the larger the weight coefficient is, and the larger the regression loss function of the corresponding small target box is. This weight coefficient can increase the proportion of small target boxes by increasing the regression loss value of small target boxes.

[0115] However, in the table detection task of disclosure documents, the main task is to detect medium and large target boxes. In order to increase the weight ratio of large target boxes and reduce the weight ratio of small target boxes, a new weight coefficient can be used Then, the final bounding box regression loss function is:

[0116]

[0117] In addition, the above-mentioned pre-set evaluation indicators are set to effectively ensure the accuracy of the model, and are used to train a table positioning model whose evaluation indicators meet the preset conditions, wherein the preset evaluation indicators can specifically be the alignment accuracy of the real box and the prediction box. It can be understood that, unlike the target detection task, table detection requires precise boundary segmentation, so the tabular data is more concerned about the degree of fit of the prediction box boundary. On this basis, in order to more accurately measure the degree of fit of the table boundary, you can choose to introduce the EoB evaluation indicator (a model evaluation indicator). The principle is to calculate the accuracy of the alignment between the target box and the prediction box. The calculation formula is as follows:

[0118]

[0119] Among them, i and j represent the predicted box and the true box respectively. Indicates the coordinates of the upper left corner of the real box, Represents the coordinates of the lower right corner of the real box, Indicates the coordinate of the upper left corner of the prediction box, Indicates the coordinates of the lower right corner of the prediction box.

[0120] In one embodiment of the present application, the above-mentioned positive and negative sample equalization processing of the sample data, before obtaining the processed sample data, may also include: copying and deleting the table in the sample data according to the table row and column layout to obtain the sample data after data enhancement.

[0121] It is understandable that in target detection tasks, large data sets can train networks with stronger generalization capabilities, obtain models with higher recognition accuracy, and are better suited for application scenarios. Due to the high cost of manual labeling, related technologies usually use data enhancement to expand the data set to improve the accuracy of the model. Standard data enhancement methods include rotation, scaling, and increasing contrast. However, for tabular data sets, standard data enhancement does not take into account the layout of rows and columns in the table, cannot generate representative data, and may even lead to a decrease in model performance. Therefore, in order to address the problem that standard data enhancement techniques are ineffective for tabular data sets, a table-based data enhancement method can be used, such as Figure 5 As shown, Figure 5 This is a schematic diagram of a table data enhancement method provided in the present application. The method can copy and delete rows and columns of a table, which can not only change the local structure of the table but also maintain the overall layout of the table, thereby obtaining a large amount of effective augmented data.

[0122] Of course, the data enhancement method for the above-mentioned copying and deletion operations is only one implementation form provided in the embodiment of the present application, and is not the only one. Other methods can also be used to implement it, as long as the sample data can be expanded. The present application does not limit this.

[0123] This application also provides a table positioning device, please refer to Figure 6 , Figure 6 This is a schematic diagram of the structure of a table positioning device provided in the present application. The table positioning device may include:

[0124] The sample acquisition module 1 is used to acquire sample data including table information;

[0125] The sample balancing module 2 is used to perform positive and negative sample balancing processing on the sample data to obtain the processed sample data;

[0126] Model training module 3, used for training and obtaining a table positioning model using the processed sample data;

[0127] Table positioning module 4, used for positioning the table of the document to be processed by using the table positioning model, and obtaining the prediction box of each table;

[0128] The prediction box screening module 5 is used to screen all table prediction boxes to obtain the optimal table prediction box by using the Confluence algorithm based on intersection-over-union ratio.

[0129] It can be seen that the table positioning device provided in the embodiment of the present application, after obtaining a large amount of sample data, first performs positive and negative sample balance processing on the sample data to obtain high-quality sample data with positive and negative sample balance, and then uses these high-quality sample data to train the model to obtain a table positioning model with high precision. Therefore, in the table positioning process, the table positioning model can be directly used to locate each table prediction box from the document to be processed. Finally, the Confluence algorithm based on intersection and union ratio is used to filter out the optimal table prediction box from all table prediction boxes, thereby realizing the optimal result from the model prediction results, and then realizing table positioning based on the neural network model, which effectively improves the positioning accuracy of the document table.

[0130] In one embodiment of the present application, the sample balancing module 2 can be specifically used to generate each preset anchor frame according to the sample data, and calculate the intersection and union ratio of each preset anchor frame and the real frame; calculate the first intersection and union ratio threshold according to the positive sample selection formula, and the positive sample selection formula is:

[0131] IoU1 = m + ln(1-v);

[0132] Among them, IoU1 is the first intersection-over-union ratio threshold, m represents the mean of the intersection-over-union ratio between the real box and the preset anchor box, and v represents the variance of the intersection-over-union ratio between the real box and the preset anchor box;

[0133] The preset anchor frames whose IoU ratio exceeds the first IoU ratio threshold are taken as positive samples; the preset anchor frames whose IoU ratio does not exceed the first IoU ratio threshold are taken as negative samples.

[0134] In one embodiment of the present application, the sample equalization module 2 can also be used to determine each prediction box according to the sample data; a preset number of prediction boxes that are closest to the center point of the real box are used as positive samples.

[0135] In one embodiment of the present application, the sample balancing module 2 may also be used to delete negative samples whose IoU ratio exceeds a second IoU ratio threshold from among all negative samples after taking the preset anchor frames whose IoU ratio does not exceed the first IoU ratio threshold as negative samples.

[0136] In one embodiment of the present application, the above-mentioned prediction box screening module 5 can be specifically used to generate a first prediction box set based on all table prediction boxes; select any table prediction box in the first prediction box set as the first table prediction box, calculate the intersection and union ratio of the first table prediction box and other table prediction boxes, and delete other table prediction boxes and the first table prediction box with an intersection and union ratio of zero to obtain a second prediction box set; calculate the confidence-weighted intersection and union ratio of the first table prediction box and each table prediction box in the second prediction box set, and use the table prediction box whose confidence-weighted intersection and union ratio exceeds the first threshold as the optimal table prediction box; calculate the intersection and union ratio of the optimal table prediction box and other table prediction boxes in the second prediction box set, and delete other table prediction boxes whose intersection and union ratio exceeds the second threshold to obtain a third prediction box set; delete the first table prediction box from the first prediction box set, and return to the step of selecting any table prediction box in the first prediction box set as the first table prediction box, until the first prediction box set is empty, and all optimal table prediction boxes are obtained.

[0137] In one embodiment of the present application, the above-mentioned model training module 3 can be specifically used to construct an initial network model in combination with the Involution algorithm and a single-stage feature detector; the initial network model is trained using the processed sample data to obtain a table positioning model with a preset evaluation index lower than a preset threshold; wherein the preset evaluation index is specifically the alignment accuracy between the real box and the predicted box.

[0138] In one embodiment of the present application, the table locating device may further include a sample enhancement module, which is used to copy and delete the table in the sample data according to the table row and column layout before performing the positive and negative sample equalization processing on the sample data to obtain the processed sample data, so as to obtain the sample data after data enhancement.

[0139] For an introduction to the device provided in this application, please refer to the above method embodiment, and this application will not go into details here.

[0140] This application also provides a table positioning device, please refer to Figure 7 , Figure 7 This is a structural schematic diagram of a table positioning device provided by the present application, and the table positioning device may include:

[0141] Memory for storing computer programs;

[0142] A processor, when used to execute a computer program, can implement the steps of any one of the above table locating methods.

[0143] like Figure 7, which is a schematic diagram of the structure of a table locating device, the table locating device may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 communicate with each other through the communication bus 13.

[0144] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.

[0145] The processor 10 may call a program stored in the memory 11 . Specifically, the processor 10 may execute operations in the embodiment of the table locating method.

[0146] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0147] Get sample data containing table information;

[0148] Perform positive and negative sample balancing on the sample data to obtain processed sample data;

[0149] Using the processed sample data to train and obtain a table positioning model;

[0150] Use the table positioning model to locate the table in the document to be processed and obtain the prediction box of each table;

[0151] The Confluence algorithm based on intersection-and-union ratio is used to screen all table prediction boxes to obtain the optimal table prediction box.

[0152] In a possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0153] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0154] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0155] Of course, it should be noted that Figure 7 The structure shown does not constitute a limitation on the table positioning device in the embodiment of the present application. In actual applications, the table positioning device may include Figure 7 More or fewer components than shown, or combinations of certain components.

[0156] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned table positioning methods can be implemented.

[0157] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0158] For an introduction to the computer-readable storage medium provided in this application, please refer to the above method embodiment, and this application will not go into details here.

[0159] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0160] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0161] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0162] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples herein, and the description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A table positioning method, characterized in that: include: Get sample data containing table information; Performing positive and negative sample balancing processing on the sample data to obtain processed sample data; Using the processed sample data to train and obtain a table positioning model; Using the table positioning model to locate the table in the document to be processed, and obtaining prediction boxes of each table; Using the Confluence algorithm based on intersection-and-union ratio to screen all the table prediction frames to obtain the optimal table prediction frame; The performing positive and negative sample equalization processing on the sample data to obtain the processed sample data includes: generating each preset anchor frame according to the sample data, and calculating the intersection and union ratio of each preset anchor frame and the real frame; calculating the first intersection and union ratio threshold according to the positive sample selection formula, and the positive sample selection formula is: IoU 1=m+ln(1-v); wherein, IoU1 is the first intersection and union ratio threshold, m represents the mean of the intersection and union ratio of the real frame and the preset anchor frame, and v represents the intersection and union ratio variance of the real frame and the preset anchor frame; taking the preset anchor frame whose intersection and union ratio exceeds the first intersection and union ratio threshold as a positive sample; taking the preset anchor frame whose intersection and union ratio does not exceed the first intersection and union ratio threshold as a negative sample; The method uses the Confluence algorithm based on intersection-and-union ratio to screen and obtain the optimal table prediction frame from all the table prediction frames, including: generating a first prediction frame set based on all the table prediction frames; selecting any table prediction frame from the first prediction frame set as the first table prediction frame, calculating the intersection-and-union ratio between the first table prediction frame and other table prediction frames, and deleting other table prediction frames with the intersection-and-union ratio of zero and the first table prediction frame to obtain a second prediction frame set; calculating the confidence-weighted intersection-and-union ratio between the first table prediction frame and each table prediction frame in the second prediction frame set, and taking the table prediction frame whose confidence-weighted intersection-and-union ratio exceeds a first threshold as the optimal table prediction frame; calculating the intersection-and-union ratio between the optimal table prediction frame and other table prediction frames in the second prediction frame set, and deleting other table prediction frames whose intersection-and-union ratio exceeds a second threshold to obtain a third prediction frame set; deleting the first table prediction frame from the first prediction frame set, and returning to the step of selecting any table prediction frame from the first prediction frame set as the first table prediction frame, until the first prediction frame set is empty, and all the optimal table prediction frames are obtained.

2. The table positioning method according to claim 1, characterized in that: Also includes: Determine each prediction box according to the sample data; A preset number of predicted boxes that are closest to the center point of the true box are used as the positive samples.

3. The table positioning method according to claim 1, characterized in that: After taking the preset anchor frame whose IoU ratio does not exceed the first IoU threshold as a negative sample, the method further includes: Among all the negative samples, the negative samples whose IoU ratio exceeds a second IoU ratio threshold are deleted.

4. The table positioning method according to claim 1, characterized in that: The step of training the processed sample data to obtain a table positioning model includes: Combine the Involution algorithm and the single-stage feature detector to build the initial network model; The processed sample data is used to perform model training on the initial network model to obtain a table positioning model with a preset evaluation index lower than a preset threshold; wherein the preset evaluation index is specifically the alignment accuracy between the real box and the predicted box.

5. The table positioning method according to claim 1, characterized in that: Before performing positive and negative sample balancing processing on the sample data to obtain the processed sample data, the method further includes: The tables in the sample data are copied and deleted according to the table row and column layout to obtain sample data after data enhancement.

6. A table positioning device, characterized in that: include: A sample acquisition module is used to obtain sample data containing table information; A sample balancing module, used to perform positive and negative sample balancing processing on the sample data to obtain processed sample data; A model training module, used for training and obtaining a table positioning model using the processed sample data; A table positioning module, used to use the table positioning model to perform table positioning on the document to be processed and obtain prediction frames of each table; A prediction box screening module, used for screening all the table prediction boxes to obtain the optimal table prediction box by using the Confluence algorithm based on intersection-over-union ratio; The sample balancing module is specifically used to generate each preset anchor frame according to the sample data, and calculate the intersection and union ratio of each preset anchor frame and the real frame; calculate the first intersection and union ratio threshold according to the positive sample selection formula, and the positive sample selection formula is: IoU 1=m+ln(1-v); wherein, IoU 1 is the first intersection and union ratio threshold, m represents the mean of the intersection and union ratio of the real frame and the preset anchor frame, and v represents the variance of the intersection and union ratio of the real frame and the preset anchor frame; the preset anchor frame whose intersection and union ratio exceeds the first intersection and union ratio threshold is taken as a positive sample; the preset anchor frame whose intersection and union ratio does not exceed the first intersection and union ratio threshold is taken as a negative sample; The prediction box screening module is specifically used to generate a first prediction box set based on all the table prediction boxes; select any one table prediction box in the first prediction box set as the first table prediction box, calculate the intersection and union ratio of the first table prediction box and other table prediction boxes, and delete the other table prediction boxes and the first table prediction box whose intersection and union ratio is zero to obtain a second prediction box set; calculate the confidence-weighted intersection and union ratio of the first table prediction box and each table prediction box in the second prediction box set, and use the table prediction box whose confidence-weighted intersection and union ratio exceeds a first threshold as the optimal table prediction box; calculate the intersection and union ratio of the optimal table prediction box and other table prediction boxes in the second prediction box set, and delete the other table prediction boxes whose intersection and union ratio exceeds a second threshold to obtain a third prediction box set; delete the first table prediction box from the first prediction box set, and return to the step of selecting any one table prediction box in the first prediction box set as the first table prediction box, until the first prediction box set is empty, and all the optimal table prediction boxes are obtained.

7. A table positioning device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the table locating method as claimed in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the table locating method according to any one of claims 1 to 5 are implemented.

Images