A method and device for detecting document forms

Abstract

The present invention discloses a document table detection method and device, comprising: extracting abstract semantic features of a table image to be detected based on a rectangular convolution kernel in a Table Residual Block, so that the extracted semantic features are more complete; fusing the abstract semantic features based on Bottom-up Path Augmentation and Advanced Image Format (ASFF); effectively fusing multi-scale features to obtain multi-scale fusion features, wherein the ASFF adaptive fusion method is used to increase the flexibility of the fusion; performing K-means clustering on the aspect ratios of table frames in a training set, determining anchor types based on the clustering results, applying the anchor types to the multi-scale fusion features in a head, obtaining each target table frame in the table image to be abstracted, and improving the accuracy of the target table frame.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for detecting a document table. Background Art

[0002] Intelligent document analysis is a key topic in today's intelligent office environment. Scanned or photographed documents can be analyzed using relevant technologies to identify chapter distribution and text content, enabling the digitization of document images. Tables in documents, such as bank and securities firm announcements, financial statements, and hospital medication lists, contain a wealth of information. The digitization of these documents, once paper documents, can significantly improve office efficiency.

[0003] The first step in digitizing document tables is table detection. Traditional table detection algorithms are generally based on OpenCV methods. They first perform preprocessing through grayscale and binarization, then apply edge detection operators to locate row and column line segments. The detected line segments are often discontinuous, so morphological methods are required to connect the required line segments to detect the preliminary outline of the table. Then, through various filtering and optimization methods, the final table is made clearer. During the detection process, edge detection relies heavily on human-designed detection operators, such as the Roberts Cross operator, the Prewitt operator, the Sobel operator, and the Marr-Hildreth operator. Traditional methods can roughly detect the position of tables in simple document images. However, these edge detection operators are significantly less effective in documents with wrinkles, tilted images, images with only row segments but no column segments, or high levels of noise, compromising the accuracy of the final table detection. Summary of the Invention

[0004] In view of this, the present invention provides a document table detection method and device to solve the problem that traditional methods can roughly detect the position of tables in document images in simple scenarios; however, the effectiveness of these edge detection operators will be greatly reduced for documents with wrinkles, tilted positions, only row segments but no column segments, or high noise, which in turn affects the accuracy of the final table detection. The specific solution is as follows:

[0005] A method for detecting a document table, comprising:

[0006] Extracting abstract semantic features of the table image to be detected based on the Table Residual Block in Backbone, wherein the Table Residual Block includes: short-circuit connections and rectangular convolution kernels;

[0007] Passing the abstract semantic features to FPN, fusing the abstract semantic features based on Bottom-up Path Augmentation and ASFF in the FPN to obtain multi-scale fusion features;

[0008] Perform K-means clustering on the aspect ratio of the table boxes in the training set, and determine the anchor types based on the clustering results;

[0009] In the Head, the anchors of the type are applied to the multi-scale fusion features to obtain each target table frame in the table image to be abstracted.

[0010] In the above method, optionally, the rectangular convolution kernel includes: a first rectangular convolution kernel and a second rectangular convolution kernel, and extracting abstract semantic features of the table image to be detected based on the Table Residual Block in Backbone, including:

[0011] Extracting a row feature matrix of the table image to be detected based on the first rectangular convolution kernel;

[0012] Extracting a column feature matrix of the table image to be detected based on the second rectangular convolution kernel;

[0013] Extracting a graph feature matrix of the table image to be detected based on the short-circuit connection;

[0014] Based on the row feature matrix, the column feature matrix and the graph feature matrix, abstract semantic features are obtained by accumulation.

[0015] The above method may optionally further include:

[0016] The number of 1×1 convolution operations in stage 3 of the Backbone is adjusted from 7 to 5.

[0017] The above method optionally fuses the abstract semantic features based on Bottom-up Path Augmentation and ASFF in the FPN to obtain multi-scale fusion features, including:

[0018] Based on Bottom-up Path Augmentation, the shallow features in the abstract semantic features are converted into shallow feature maps through downsampling;

[0019] Converting shallow features in the abstract semantic features into a first feature map based on upsampling;

[0020] Determine respective second feature maps based on the shallow feature map and the first feature map;

[0021] Based on ASFF, adaptive weights are assigned to the respective second feature maps, and multi-feature fusion is performed based on the adaptive weights and the respective second feature maps to obtain multi-scale fusion features.

[0022] The above method optionally applies the type of anchors to the multi-scale fusion features in the Head to obtain each target table frame in the table image to be abstracted, including:

[0023] For each anchor, traverse the multi-scale fusion features to determine the table box position and confidence that meets the current anchors

[0024] Compare each confidence level with a preset confidence threshold, and filter out target table frames that meet the preset confidence threshold requirements.

[0025] A document form detection device, comprising:

[0026] An extraction module is used to extract abstract semantic features of the table image to be detected based on the Table Residual Block in Backbone, wherein the Table Residual Block includes: a short-circuit connection and a rectangular convolution kernel;

[0027] A fusion module is used to pass the abstract semantic features to the FPN, and fuse the abstract semantic features based on the Bottom-up Path Augmentation and ASFF in the FPN to obtain a multi-scale fusion feature;

[0028] Clustering module, used to perform K-means clustering on the aspect ratio of table boxes in the training set and determine the anchor types based on the clustering results;

[0029] The determination module is used to apply the anchors of the type to the multi-scale fusion features in the Head to obtain each target table frame in the table image to be abstracted.

[0030] In the above device, optionally, the rectangular convolution kernel includes: a first rectangular convolution kernel and a second rectangular convolution kernel, and the extraction module includes:

[0031] A first extraction unit is configured to extract a row feature matrix of the table image to be detected based on the first rectangular convolution kernel;

[0032] A second extraction unit, configured to extract a column feature matrix of the table image to be detected based on the second rectangular convolution kernel;

[0033] a third extraction unit, configured to extract a graph feature matrix of the table image to be detected based on the short-circuit connection;

[0034] An accumulation unit is used to accumulate based on the row feature matrix, the column feature matrix and the graph feature matrix to obtain abstract semantic features.

[0035] The above device may optionally further include:

[0036] The adjustment unit is used to adjust the number of 1×1 convolution operations in the stage 3 of the Backbone from 7 to 5.

[0037] In the above device, optionally, the fusion module includes:

[0038] A first conversion unit is configured to convert shallow features in the abstract semantic features into shallow feature maps by downsampling based on Bottom-up Path Augmentation;

[0039] A second conversion unit, configured to convert shallow features in the abstract semantic features into a first feature map based on upsampling;

[0040] A first determining unit, configured to determine respective second feature maps based on the shallow feature map and the first feature map;

[0041] A fusion unit is used to assign adaptive weights to each of the second feature maps based on ASFF, and perform multi-feature fusion based on the adaptive weights and each of the second feature maps to obtain a multi-scale fusion feature.

[0042] In the above device, optionally, the determining module includes:

[0043] The second determination unit is used to traverse the multi-scale fusion features for each anchor to determine the table box position and confidence that meet the current anchors

[0044] The comparison and screening unit is used to compare each confidence level with a preset confidence threshold, and screen out target table frames that meet the preset confidence threshold requirement.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] The present invention discloses a document table detection method and device, comprising: extracting abstract semantic features of a table image to be detected based on a Table Residual Block, wherein the Table Residual Block includes a short-circuit connection and a rectangular convolution kernel; fusing the abstract semantic features based on Bottom-up Path Augmentation and ASFF to obtain multi-scale fusion features; performing K-means clustering on the aspect ratios of table frames in a training set, determining anchor types based on the clustering results, applying the anchor types in the Head to the multi-scale fusion features, and obtaining each target table frame in the table image to be abstracted. The above process extracts abstract semantic features based on a rectangular convolution kernel, resulting in more comprehensive extracted semantic features; effectively fuses multi-scale features based on the ASFF adaptive fusion method, increasing the flexibility of fusion; and determining anchors based on the aspect ratio of the table frame, thereby improving the accuracy of the target table frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 A flow chart of a document table detection method disclosed in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of a Table Residual Block disclosed in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of a Bottom-up Path Augmentation disclosed in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of a Bottom-up Path Augmentation process disclosed in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of an ASFF disclosed in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of a document table detection model disclosed in an embodiment of the present application;

[0054] Figure 7This is a structural block diagram of a document table detection device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0056] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

[0057] The present invention discloses a method and device for detecting document tables, which are applied to the process of detecting document tables in table images. In an embodiment of the present invention, a target detection algorithm based on deep learning utilizes the powerful feature learning ability of a deep neural network. Through training with a large amount of image data, the model parameters can be well learned and effective features can be extracted. In actual scenarios, it can cope with more and more complex document images and achieve good results in image tasks such as target detection. At present, the commonly used target detection network algorithms are generally applied to general scenarios in nature. The effect in document table detection tasks is not very good, the table detection is not complete, and smaller tables will be missed. Therefore, based on the commonly used target detection algorithms, the present invention makes targeted improvements to the network model according to the characteristics of the table in the document, so that the general target detection algorithm can achieve better results in specific scenario services and meet actual requirements.

[0058] In the prior art, the general model processing flow for table detection is feature extraction, multi-scale feature fusion, and target table frame detection. The models commonly used in the industry at this stage are generally based on general scenarios in nature. In specific task scenarios, the generalization performance is poor, so targeted improvements need to be made to each stage. The models commonly used in the industry are generally used for general scenarios. In the feature extraction stage, they do not fully utilize the rich horizontal and vertical line segments of the table. The present invention improves the convolution kernel in the feature extraction stage so that it can fully extract the table line features. In terms of feature fusion, the commonly used models generally use fixed weight fusion, which lacks flexibility. Taking into account the differences in features of different scales, the present invention uses adaptive weights to fuse and improve the effectiveness of the fused features. In the anchor design stage, in general scenarios, it is generally designed based on empirical values, without taking into account the high aspect ratio characteristics of the table. The present invention designs anchors in a targeted manner based on the aspect ratios of all tables in the training set to improve the effect of the final model prediction.

[0059] In the embodiment of the present invention, the target detection model based on deep learning is the commonly used target detection model YOLOv3 model, which is targeted for document table detection tasks. Most commonly used target detection models adopt Backbone+FPN+Head networking, where Backbone is used to extract abstract semantic features, FPN is used to achieve multi-scale feature fusion, and Head is used for feature post-processing, target positioning and classification. The execution process of commonly used detection methods is as follows: Figure 1 As shown, the steps include:

[0060] S101, extracting abstract semantic features of the table image to be detected based on the Table Residual Block in Backbone, wherein the Table Residual Block includes: a short-circuit connection and a rectangular convolution kernel;

[0061] In the embodiment of the present invention, the shuffleNetv2 lightweight network model is selected to obtain abstract semantic features. ShuffleNetV2 is an existing lightweight network model. The present invention selects the front part of the network and improves it according to the table features as the backbone of the network of the invention. A module similar to ResNet is added to the front end of the backbone network, and two branches are added next to the conventional convolution channel. One branch uses a 1×5 convolution kernel to detect row features, and the other branch uses a 5×1 convolution kernel to detect column features; then the features of the three branches are fused and enter the subsequent stage of the backbone.

[0062] The backbone used in this invention is a model based on shuffleNet v2 with targeted modifications. The network structure is shown in Table 1. After the first layer of Conv1, the present invention adds a Table Residual Block module, specifically as follows: Figure 2 As shown, there are three branches here. The middle one is a short-circuited branch, and the feature map of Conv1 is directly passed over. The left branch is 1×5conv+BN+ReLu+1×5conv+BN, and the right branch is 5×1conv+BN+ReLu+5×1conv+BN. Finally, the results of the three branches are added together (that is, the feature matrices are added) to achieve the fusion of the three feature maps and obtain the abstract semantic features. Among them, the 1×5 convolution kernel (the first rectangular convolution) can effectively extract the row segments in the table on the low-level high-resolution feature map, and the 5×1 convolution kernel (the second rectangular convolution) can effectively extract the column segments in the table. These two branches are used to enhance the table features in the feature map, providing effective assistance for subsequent deep feature extraction. Optionally, the first rectangular convolution can also be a 1×3 convolution kernel, and the second rectangular convolution can be a 3×1 convolution kernel.

[0063] Table 1

[0064]

[0065] On the other hand, in order to achieve the lightweight table detection model, the present invention reduces the number of repetitions of the 1×1 convolution operation in the stage 3 stage of the original shuffleNet v2 by two, from 7 to 5; and removes other operations after the conv5 layer. The final model downsamples the input image four times, from the input 224×224 to 7×7.

[0066] S102, passing the abstract semantic features to FPN, and fusing the abstract semantic features based on Bottom-up Path Augmentation and ASFF in the FPN to obtain multi-scale fusion features;

[0067] In this embodiment of the present invention, the abstract semantic features are passed to the FPN. During the Backbone extraction of these abstract semantic features, shallow features contain a large number of features such as edge shapes, which are crucial for the subsequent object detection task in the Head stage. Therefore, to preserve more shallow features, the present invention introduces Bottom-up Path Augmentation.

[0068] like Figure 3As shown in the figure, in the commonly used FPN model (D2, D3, D4 and D5 to P3, P4 and P5), because of the bottom-up process, the shallow features need to pass through dozens of network layers to reach the top layer. Therefore, after passing through multiple network layers, the shallow feature information is lost more seriously. The bottom-up path augmentation structure added by the present invention has less than 10 layers. In this way, the shallow features are horizontally connected to the top layers. Then from The information is transferred to the top layer along the Bottom-up Path Augmentation, passing through less than 10 layers, which can better preserve the shallow feature information.

[0069] Bottom-up Path Augmentation structure such as Figure 4 As shown, the feature map D i+1 After a convolution with a stride of 2 and a size of 3×3, the feature map size is reduced to half of its original size and then combined with the feature map P of the corresponding scale. i Perform the add operation, and then pass the result through the ReLu activation function to obtain the feature map P i+1 .therefore, and They are feature maps obtained by preliminary fusion of three scales through addition.

[0070] In this embodiment of the present invention, each block diagram represents a feature map. The numbers in the lower right corner increase from small to large, indicating increasingly abstract features and deeper layers. The D series represents features extracted in the Backbone stage, and the P series represents features fused from upsampled features from D5 and features from the D series on the left, using matrix addition.

[0071] In this embodiment of the present invention, initial feature fusion is performed only at a single scale. To fuse features at different scales, the present invention introduces Adaptive Spatial Feature Fusion (ASFF) on the initial fused feature map P. The concept of ASFF is that the original FPN add method is now enhanced with an additional learnable coefficient. This parameter is automatically learned, enabling adaptive fusion effects similar to fully connected parameters.

[0072] ASFF fusion methods such as Figure 5As shown in the figure, it performs fusion of features of different scales at each scale layer, resulting in the fusion results ASFF-1, ASFF-2, and ASFF-3. To facilitate feature fusion, the output feature size and the number of output channels are changed through upsampling or downsampling before the features of each layer are fused with the features of the target layer. For example, the resolution of the level 1 feature map is increased through upsampling to be consistent with the level 2 or level 3 feature map; the resolution of level 3 is reduced through downsampling to be consistent with level 1 or level 2; and the resolution of level 2 is upsampled to be consistent with level 3 and downsampled to be consistent with level 1.

[0073]

[0074] The adaptive feature fusion method is shown in Formula 1. Represents the l-th layer fusion feature output, such as ASFF-l, and Represents the weight coefficients of level1, level2 and level3 layers when the feature is fused at layer l. The weight coefficients are shared on all channels of the layer. For example, in layer l feature fusion, level1 has c channel layers at this layer, then the weight coefficients are shared on c channel layers. Represents the features of the first-level feature map level1 after sampling at layer l, Represents the features of the second-level feature map level2 after sampling at the second layer, Represents the features of the third-level feature map level3 after sampling at the third layer. The three scale features have the same number of channels. Weighted linear superposition is performed on the corresponding channels to obtain the fused multi-scale fusion features.

[0075] The adaptive weight in the formula is obtained by the feature map output by another branch in the module. The hyperparameter n represents the number of channels on this branch. A 1×1 convolution kernel is used to compress the number of channels to n. The feature maps obtained at the three scaling layers are p1, p2, and p3. p1, p2, and p3 are then concatenated along the channel direction to obtain a feature map q with 3n channels. A 1×1 convolution kernel is then used to reduce the number of channels in q to 3. Global average pooling is then performed on the feature maps of the three channels to obtain a three-dimensional vector v. To ensure that the sum of the weights α, β, and γ is 1, the three-dimensional vector v is normalized using the softmax function, resulting in the adaptive weight coefficients α, β, and γ.

[0076] Compared with the traditional direct addition of multi-scale features, this adaptive feature fusion method allows the network model to continuously update the model parameters of the corresponding stage during training, so that the model parameters are updated in the direction of reducing losses and improving the final effect.

[0077] Furthermore, in the fusion process, Bottom-up Path Augmentation and ASFF are combined. On the one hand, features of different scales can more effectively fuse low-level semantic information. On the other hand, an adaptive mechanism is added to increase the flexibility of the model in multi-scale feature fusion, making the fused feature map richer and more effective.

[0078] S103, performing K-means clustering on the aspect ratios of the table frames in the training set, and determining the types of anchors based on the clustering results;

[0079] In an embodiment of the present invention, after preliminary feature extraction and multi-scale feature fusion, target detection is performed in the network model head stage. This stage uses anchors on the fused feature maps at each scale. Conventional target detection targets general natural scenes and performs generally well in document table detection tasks. To enhance the relevance of table detection tasks, in an embodiment of the present invention, before model training, all table frames in the training set are listed, their aspect ratios R are calculated, and the k-means clustering algorithm is used to cluster the aspect ratios R of the table frames into k clusters, where k represents the type and is an experimental value. A range of values is set, and then an optimal value is determined based on experiments as k, i.e., the type of anchor. Similarly, clustering is used to select the size of the anchors. These k-class aspect ratios are applied to the feature maps at each scale, and the corresponding anchors are calculated based on the receptive fields of the corresponding feature maps, which serve as the anchors set for the model. Targeted design of hyperparameters such as anchors for specific tasks can effectively improve the model's final performance.

[0080] S104 , applying the type of anchors to the multi-scale fusion features in the Head to obtain each target table frame in the table image to be abstracted.

[0081] In an embodiment of the present invention, the multi-scale fusion features output by the FPN stage are predicted based on the previously set anchors to obtain the position and confidence of the table box. Among them, the determination of position and category confidence is the last step of target detection, that is, the Head stage, and the category and position of the detection box are obtained based on the input multi-scale feature ASFF. Simply put, the input feature ASFF is subjected to processes such as reshape, fully connected layer and softmax to obtain two matrices. One matrix is the category matrix, and the row vector represents the category confidence of the detection box, that is, the probability that each detection box belongs to each category, also called confidence. The other matrix is the position matrix, and the row vector represents the position of the four corners of each detection box.

[0082] Optionally, a preset confidence threshold is set in advance, wherein the preset confidence threshold can be set based on experience or specific circumstances, and is not specifically defined in detail in the embodiment of the present invention. Each confidence is compared with the preset confidence threshold to screen out the target table box that meets the preset confidence threshold requirements.

[0083] The present invention discloses a method for detecting document tables, comprising: extracting abstract semantic features of a table image to be detected based on a Table Residual Block, wherein the Table Residual Block includes a short-circuit connection and a rectangular convolution kernel; fusing the abstract semantic features based on Bottom-up Path Augmentation and Automatic Format Function (ASFF) to obtain multi-scale fusion features; performing K-means clustering on the aspect ratios of table frames in a training set, determining anchor types based on the clustering results, and applying the anchor types in a head to the multi-scale fusion features to obtain each target table frame in the table image to be abstracted. The above process extracts abstract semantic features based on a rectangular convolution kernel, resulting in more comprehensive extracted semantic features; effectively fuses multi-scale features based on the ASFF adaptive fusion method, increasing the flexibility of the fusion; and determining anchors based on the aspect ratio of the table frame, thereby improving the accuracy of the target table frame.

[0084] The present invention is an embodiment, and the overall processing flow based on the above method is as follows Figure 6As shown in Figure 1, TableDetNet is an end-to-end detection model consisting of three stages: Backbone, FPN, and Head. Backbone extracts abstract semantic features from the original image. FPN obtains abstract semantic features from the Backbone stage. Bottom-up path augmentation fusion is performed based on D2, D3, D4, and D5 to P3, P4, and P5, and P3, P4, P5, α, β, γ, N2, N3, N4, and N5 to obtain ASFF-1, ASFF-2, and ASFF-3. The specific fusion process is shown in S102 and will not be repeated here. The Head stage performs detection and prediction on the multi-scale fused features of the FPN stage at different scales to obtain Loss / Results. Based on the Loss / Results, the table box, confidence, and category are obtained. When training the network, the loss is also calculated based on the ground truth annotation results; however, no loss calculation is required during prediction.

[0085] The document table detection model proposed in this paper is an end-to-end object detection model. Therefore, during training, the original document images and corresponding table annotation files in the training set are directly fed into the network for training. Because the model modifies its backbone, it cannot directly use model parameters trained on large datasets by major artificial intelligence companies or research institutions. Therefore, in practical scenarios, a large dataset from a document table analysis competition can be selected to perform end-to-end fine-tuning on the network model, ensuring that the model performs well on its test set. After this stage, the network model parameters are saved and used as pre-trained model parameters.

[0086] In actual engineering environments, the document images to be predicted differ from the competition document images. To ensure accurate predictions for actual document images, fine-tuning training is required based on the pre-trained model from the previous step. To prevent significant differences in data distribution from causing drastic changes in model parameters during fine-tuning training, the two datasets are combined in a certain ratio as the training set and fed into the model for training. After a period of fine-tuning training, the model's performance meets actual engineering requirements.

[0087] Based on the above-mentioned method for detecting a document table, an embodiment of the present invention further provides a device for detecting a document table. The structural block diagram of the device is shown in FIG. Figure 7 As shown, including:

[0088] Extraction module 201 , fusion module 202 , clustering module 203 and determination module 204 .

[0089] in,

[0090] The extraction module 201 is used to extract abstract semantic features of the table image to be detected based on the Table Residual Block in Backbone, wherein the Table Residual Block includes: a short-circuit connection and a rectangular convolution kernel;

[0091] The fusion module 202 is configured to transfer the abstract semantic features to the FPN, and fuse the abstract semantic features based on the Bottom-up Path Augmentation and ASFF in the FPN to obtain a multi-scale fusion feature;

[0092] The clustering module 203 is used to perform K-means clustering on the aspect ratios of the table frames in the training set and determine the types of anchors based on the clustering results;

[0093] The determining module 204 is configured to apply the type of anchors to the multi-scale fusion features in the Head to obtain target table frames in the table image to be abstracted.

[0094] The present invention discloses a device for detecting document tables. The device comprises: extracting abstract semantic features of a table image to be detected based on a Table Residual Block, wherein the Table Residual Block comprises a short-circuit connection and a rectangular convolution kernel; fusing the abstract semantic features based on Bottom-up Path Augmentation and ASFF to obtain multi-scale fusion features; performing K-means clustering on the aspect ratios of table frames in a training set, determining the types of anchors based on the clustering results, applying the anchors of the types to the multi-scale fusion features in the Head, and obtaining each target table frame in the table image to be abstracted. The above process extracts abstract semantic features based on a rectangular convolution kernel, and the extracted semantic features are more complete. Based on the ASFF adaptive fusion method, the multi-scale features are effectively fused, increasing the flexibility of the fusion. The anchors are determined based on the aspect ratio of the table frame, thereby improving the accuracy of the target table frame.

[0095] In the embodiment of the present invention, the rectangular convolution kernel includes: a first rectangular convolution kernel and a second rectangular convolution kernel, and the extraction module 201 includes:

[0096] A first extraction unit 205 , a second extraction unit 206 , a third extraction unit 207 and an accumulation unit 208 .

[0097] in,

[0098] The first extraction unit 205 is configured to extract a row feature matrix of the table image to be detected based on the first rectangular convolution kernel;

[0099] The second extraction unit 206 is configured to extract a column feature matrix of the table image to be detected based on the second rectangular convolution kernel;

[0100] The third extraction unit 207 is configured to extract a graphic feature matrix of the table image to be detected based on the short-circuit connection;

[0101] The accumulation unit 208 is configured to perform accumulation based on the row feature matrix, the column feature matrix, and the graph feature matrix to obtain abstract semantic features.

[0102] In the embodiment of the present invention, the extraction module 201 further includes: an adjustment unit 209.

[0103] in,

[0104] The adjustment unit 209 is used to adjust the number of 1×1 convolution operations in the stage 3 of the Backbone from 7 to 5.

[0105] In the embodiment of the present invention, the fusion module 202 includes:

[0106] A first conversion unit 210 , a second conversion unit 211 , a first determination unit 212 and a fusion unit 213 .

[0107] in,

[0108] The first conversion unit 210 is configured to convert shallow features in the abstract semantic features into shallow feature maps by downsampling based on Bottom-up Path Augmentation;

[0109] The second conversion unit 211 is configured to convert the shallow features in the abstract semantic features into a first feature map based on upsampling;

[0110] The first determining unit 212 is configured to determine each second feature map based on the shallow feature map and the first feature map;

[0111] The fusion unit 213 is configured to assign adaptive weights to the respective second feature maps based on ASFF, and perform multi-feature fusion based on the adaptive weights and the respective second feature maps to obtain multi-scale fusion features.

[0112] In this embodiment of the present invention, the determining module 204 includes:

[0113] a second determining unit 214 and a comparing and screening unit 215 .

[0114] in,

[0115] The second determining unit 214 is used to traverse the multi-scale fusion features for each anchor to determine the table box position and confidence that meets the current anchors.

[0116] The comparison and screening unit 215 is used to compare each confidence level with a preset confidence threshold, and screen out target table frames that meet the preset confidence threshold requirement.

[0117] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For device embodiments, since they are generally similar to method embodiments, their description is relatively simple, and for relevant details, reference can be made to the description of the method embodiments.

[0118] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0119] For the convenience of description, the above device is described as being divided into various units according to their functions. Of course, when implementing the present invention, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0120] From the above description of the embodiments, it can be seen that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus the necessary general-purpose hardware platform. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0121] The above is a detailed introduction to the document table detection method and device provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for general technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for detecting a document table, characterized in that: include: Extracting abstract semantic features of the table image to be detected based on the Table Residual Block in Backbone, wherein the Table Residual Block includes: short-circuit connections and rectangular convolution kernels; The abstract semantic features are passed to FPN, and the abstract semantic features are fused based on Bottom-up Path Augmentation and ASFF in the FPN to obtain multi-scale fusion features; Perform K-means clustering on the aspect ratio of the table boxes in the training set, and determine the anchor types based on the clustering results; Applying the type of anchors to the multi-scale fusion features in the Head to obtain each target table frame in the table image to be abstracted; The Table Residual Block is connected after the first convolution layer of the Backbone. The rectangular convolution kernel includes: a first rectangular convolution kernel and a second rectangular convolution kernel. The abstract semantic features of the table image to be detected are extracted based on the Table Residual Block in the Backbone, including: Extracting a row feature matrix of the table image to be detected based on the first rectangular convolution kernel; Extracting a column feature matrix of the table image to be detected based on the second rectangular convolution kernel; Extracting a graph feature matrix of the table image to be detected based on the short-circuit connection; Based on the row feature matrix, the column feature matrix and the graph feature matrix, abstract semantic features are obtained by accumulation.

2. The method according to claim 1, characterized in that Also includes: The number of 1×1 convolution operations in stage 3 of the Backbone is adjusted from 7 to 5.

3. The method according to claim 1, characterized in that The abstract semantic features are fused based on the Bottom-up Path Augmentation and ASFF in the FPN to obtain multi-scale fusion features, including: Based on Bottom-up Path Augmentation, the shallow features in the abstract semantic features are converted into shallow feature maps through downsampling; Converting shallow features in the abstract semantic features into a first feature map based on upsampling; Determine respective second feature maps based on the shallow feature map and the first feature map; Based on ASFF, adaptive weights are assigned to the respective second feature maps, and multi-feature fusion is performed based on the adaptive weights and the respective second feature maps to obtain multi-scale fusion features.

4. The method according to claim 1, wherein In the Head, the anchors of the above type are applied to the multi-scale fusion features to obtain the target table frames in the table image to be abstracted, including: For each anchor, traverse the multi-scale fusion features to determine the table box position and confidence that meets the current anchors Compare each confidence level with a preset confidence threshold, and filter out target table frames that meet the preset confidence threshold requirements.

5. A document table detection device, characterized in that: include: An extraction module is configured to extract abstract semantic features of the table image to be detected based on the Table Residual Block in Backbone, wherein the Table Residual Block includes a short-circuit connection and a rectangular convolution kernel, and the rectangular convolution kernel includes a first rectangular convolution kernel and a second rectangular convolution kernel, and the Table Residual Block is connected after the first convolution layer of the Backbone; A fusion module is used to pass the abstract semantic features to FPN, and fuse the abstract semantic features based on Bottom-up Path Augmentation and ASFF in the FPN to obtain multi-scale fusion features; Clustering module, used to perform K-means clustering on the aspect ratio of table boxes in the training set and determine the anchor types based on the clustering results; A determination module, configured to apply the type of anchors to the multi-scale fusion features in the Head to obtain target table frames in the table image to be abstracted; The extraction module includes: A first extraction unit is configured to extract a row feature matrix of the table image to be detected based on the first rectangular convolution kernel; A second extraction unit, configured to extract a column feature matrix of the table image to be detected based on the second rectangular convolution kernel; a third extraction unit, configured to extract a graph feature matrix of the table image to be detected based on the short-circuit connection; An accumulation unit is used to accumulate based on the row feature matrix, the column feature matrix and the graph feature matrix to obtain abstract semantic features.

6. The device according to claim 5, characterized in that Also includes: The adjustment unit is used to adjust the number of 1×1 convolution operations in the stage 3 of the Backbone from 7 to 5.

7. The device according to claim 5, characterized in that The fusion module includes: A first conversion unit is configured to convert shallow features in the abstract semantic features into shallow feature maps by downsampling based on Bottom-up Path Augmentation; A second conversion unit, configured to convert shallow features in the abstract semantic features into a first feature map based on upsampling; A first determining unit, configured to determine respective second feature maps based on the shallow feature map and the first feature map; A fusion unit is used to assign adaptive weights to each of the second feature maps based on ASFF, and perform multi-feature fusion based on the adaptive weights and each of the second feature maps to obtain a multi-scale fusion feature.

8. The device according to claim 5, characterized in that The determination module includes: The second determination unit is used to traverse the multi-scale fusion features for each anchor to determine the table box position and confidence that meet the current anchors The comparison and screening unit is used to compare each confidence level with a preset confidence threshold, and screen out target table frames that meet the preset confidence threshold requirement.

Images