Table detection and content extraction method and system based on image processing

Abstract

The invention relates to a table detection and content extraction method and system based on image processing. The method comprises the following steps: constructing a table detection model and a table structure recognition model; using the public data set to train the table detection model and the table structure identification model, and using the self-made data set to finely adjust the table detection model and the table structure identification model; converting the input file into a picture format and preprocessing the input file; inputting the preprocessed file picture into a table detection model for feature extraction, and obtaining frame information and category information of a table in the picture according to a feature graph of the input picture; extracting the table in the picture according to the obtained frame information and category information of the table; inputting the table picture extracted by the table detection model into a table structure detection model, and detecting and identifying a table unit structure; and identifying the text information in the table picture by using an optical character recognition (OCR) technology, and matching the obtained table unit structure with the text information content to generate a target table.

Description

Technical Field

[0001] The present invention relates to the technical field of document data mining and extraction, and in particular to a table detection and content extraction method and system based on image processing. Background Art

[0002] Existing methods have the following shortcomings in the task of table recognition and extraction: First, traditional document processing-based methods, such as pdfminer and pdfplumber, essentially read PDF document information and can only process text-type PDFs. They have a good effect on text processing, but are generally not very effective in recovering the structure of tables; second, the existing table detection and content extraction process, from original documents to formatted output, is cumbersome, with low extraction efficiency and low accuracy; third, deep learning-based methods often need to build models of different architectures according to different application scenarios and use different training data to train models to adapt to specific application scenarios, and the model generalization ability is generally average.

[0003] Prior art, such as the Chinese patent with patent number "CN115761773A", discloses a method and system for recognizing tables in images based on deep learning. The method includes: step S1: recognizing and locating the table in the image through a deep neural network to obtain the relative position information of the table frame; step S2: intercepting the table image based on the relative position information, and recognizing the table structure through a deep learning network; step S3: extracting the information recorded in each cell through OCR technology based on the acquired table structure; step S4: generating an editable table document based on the recognized table structure and the information recorded in the table. The present invention can recognize any image with a table, and return the structure of the table, the information of each cell and the text information with a high accuracy, so that the image table data can be digitized.

[0004] The problems existing in the above prior art are:

[0005] 1. The performance of deep learning networks depends on a large amount of training data. If the training data is not diverse enough, it may not cover all possible table types and image scenarios, resulting in poor performance when processing some special table images.

[0006] 2. The generalization ability of the model may be limited. When faced with images that are significantly different from the distribution of the training data, the model may not be able to accurately recognize the table, or the accuracy of the recognition results may decrease.

[0007] 3. In practical applications, when processing a large number of image tables, there may be limitations on computing resources and time. Especially for high-resolution and large-size images, the recognition process may be time-consuming, affecting work efficiency. Summary of the invention

[0008] In order to solve the above technical problems, the present invention proposes a table detection and content extraction method and system based on image processing.

[0009] The technical solution of the present invention is as follows:

[0010] The present invention proposes a table detection and content extraction method based on image processing, comprising the following steps:

[0011] Step S1, constructing a table detection model and a table structure recognition model, wherein the table detection model and the table structure recognition model include a plurality of encoding modules and decoding modules;

[0012] Step S2, using the public data set to train the table detection model and the table structure recognition model, and using the self-made data set to fine-tune the table detection model and the table structure recognition model;

[0013] Step S3, converting the input file into a picture format and performing preprocessing;

[0014] Step S4, inputting the preprocessed file image into the table detection model for feature extraction, obtaining the border information and category information of the table in the image according to the feature map of the input image; extracting the table in the image according to the obtained table border information and category information;

[0015] Step S5, inputting the table image extracted by the table detection model into the table structure detection model to detect and identify the table unit structure;

[0016] Step S6, using optical character recognition (OCR) technology to recognize text information in the table image, matching the obtained table unit structure with the text information content, and generating a target table.

[0017] As a preferred embodiment, the self-made dataset includes images and image labels, wherein the image labels include category labels for indicating the category information of the table or table cell in the image; position labels for indicating the position information of the table or table cell in the image; and mask labels for indicating which parts of the image need to be paid attention to in the cell detection task of the table detection model;

[0018] The self-made dataset is constructed using pre-trained models to assist annotation and automatic generation. The specific construction steps are as follows:

[0019] Obtain an XML file corresponding to the PDF file from a public website, obtain the table sequence in the XML, and extract its style features as a template;

[0020] Use a table generator based on HTML syntax to generate a table sequence. For the generated table sequence, randomly sample some text or pictures to fill the cells.

[0021] Apply multiple styles to the filled table, including cells, lines, backgrounds, texts, and colors, to obtain tables with multiple styles; in order to make the generated styles closer to the real table styles while maintaining diversity, first obtain style templates, focusing on at least two types of styles, the table style with pictures inserted in cells and the table style with missing borders, and then apply different styles to different types of tables; and convert the table sequence into a table picture;

[0022] Use the pre-trained model to assist in labeling the table structure. Input the table image as a sample into the pre-trained table structure recognition model and text detection model. The correct prediction is used as a positive sample. Use the prediction result to add a label to each sample.

[0023] The final dataset is constructed by merging positive samples and a small number of negative samples, where negative samples are samples that do not contain tables in the images.

[0024] As a preferred implementation, during the fine-tuning stage, a third output head is added to the table structure recognition model to supervise the learning of the table structure recognition model and improve the detection capability of table cell structure recognition.

[0025] As a preferred implementation, the table detection model and the table structure recognition model use the real-time target detection architecture RT-DETR; an adapter layer is inserted into the real-time target detection architecture RT-DETR for table detection and content extraction tasks.

[0026] As a preferred implementation, the adapter layer is used in the model fine-tuning stage. The adapter layer contains new trainable parameters, which are updated during the fine-tuning process of the model. The adapter layer is a convolutional layer or a linear layer, which is inserted into the encoding network to transform the features.

[0027] As a preferred embodiment, the feature extraction using the real-time target detection architecture RT-DETR includes the following steps:

[0028] Use the feature extraction network to extract image features from the input image and the table image extracted by the table detection model to obtain a multi-scale feature set of the target image, namely, a feature map pyramid;

[0029] The highest feature map of the target image is flattened into a vector sequence as the input of the encoder to complete the intrinsic interaction calculation of the features;

[0030] The context and channel fusion CCFM module is used to concatenate and output the fused multi-scale features.

[0031] As a preferred implementation, after the step of splicing and outputting the fused multi-scale features using the context and channel fusion CCFM module, the method further includes:

[0032] The Hungarian algorithm is used to match the output content with the label content, so that a one-to-one correspondence is established between the predicted detection box and the detection table and content.

[0033] On the other hand, the present invention also provides a table detection and content extraction system based on image processing, comprising:

[0034] A model building module is used to build a table detection model and a table structure recognition model. The table detection model and the table structure recognition model include a number of encoding modules and decoding modules.

[0035] Model training and fine-tuning module: Use public data sets to train table detection models and table structure recognition models, and use self-made data sets to fine-tune table detection models and table structure recognition models;

[0036] Preprocessing module, converts input files into image format and performs preprocessing;

[0037] The table recognition module inputs the preprocessed file image into the table detection model for feature extraction, obtains the border information and category information of the table in the image according to the feature map of the input image; extracts the table in the image according to the obtained table border information and category information;

[0038] The table structure recognition module inputs the table image extracted by the table detection model into the table structure detection model to detect and recognize the table unit structure;

[0039] The text matching module uses optical character recognition (OCR) technology to identify text information in table images, matches the obtained table cell structure with the text information content, and generates a target table.

[0040] In yet another aspect, the present invention further provides an electronic device having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for table detection and content extraction based on image processing as described in any embodiment of the present invention is implemented.

[0041] On the other hand, the present invention also provides a computer-readable medium for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement a table detection and content extraction method based on image processing as described in any embodiment of the present invention.

[0042] The present invention has the following beneficial effects:

[0043] 1. The recognition accuracy of tables without table lines or tables lacking table lines is improved; the recognition accuracy of tables with large cells (for example, pictures inserted in cells) is high; rotated table pictures can be recognized.

[0044] 2. With an end-to-end architecture, for target detection and recognition tasks, no prior knowledge and artificially designed rules such as box size, confidence threshold, and non-maximum suppression (NMS) threshold are required.

[0045] 3. The model used in this method is first pre-trained on a large number of multi-type documents to greatly improve the generalization ability of the model, and then only a small number of samples are used for fine-tuning. By adopting the Adapter fine-tuning strategy, the accuracy of table recognition is improved.

[0046] 4. Such as Figure 2 As shown in the figure, in this method, the Adapter layer is inserted into the RT-DETR architecture to form a new encoding network structure; the Adapter layer is inserted into the existing encoding network as an independent module, without large-scale modification of the original network structure, and specific functional modules can be easily added, which improves the adaptability and scalability of the model; the Adapter layer usually has fewer parameters, and compared with retraining the entire encoding network, it can greatly reduce the number of parameters that need to be adjusted during the fine-tuning process. It can improve the generalization ability of the model, reduce the computational workload and storage requirements of the model, and further improve the performance of the model.

[0047] 5. Using the RT-DETR architecture for feature extraction can achieve instant recognition, not only for tables in documents, but also for tables in videos, with high processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 It is a schematic diagram of a process for implementing the first embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the structure of the Adapter layer in an embodiment of the present invention;

[0051] Figure 3 Table rotated in the embodiment of the present invention represents a table rotated 90° counterclockwise;

[0052] Figure 4 Table in the embodiment of the present invention represents a table in a normal direction;

[0053] Figure 5 The frameless table detection result in the embodiment of the present invention;

[0054] Figure 6 The table structure recognition result of the frameless table in the embodiment of the present invention;

[0055] Figure 7 This is a cell after being processed in the embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] It should be understood that the step numbers used in this document are only for convenience of description and are not intended to limit the order in which the steps are executed.

[0058] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0059] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0060] The term "and / or" means and includes any and all possible combinations of one or more of the associated listed items.

[0061] Embodiment 1:

[0062] In order to make the purpose, technical solutions and advantages of the present invention more clear, the following will be combined with the specific embodiments of the present application and refer to the attached Figure 1 , clearly and completely describe the technical solution of the present invention.

[0063] To solve the problems in the prior art, the present invention provides a table detection and content extraction method based on image processing, comprising the following steps:

[0064] Step S1, constructing a table detection model and a table structure recognition model, wherein the table detection model and the table structure recognition model include a plurality of encoding modules and decoding modules;

[0065] Construct a table detection model and a table structure recognition model, which include an encoding module and a decoding module. In actual operation, the feature map of the input image and the segmented table image are respectively input into the table detection module and the table structure recognition module for encoding and vectorization to obtain the recognition latent vector, which is then input into the table decoding module for decoding, and the decoded vectors are output through different output heads.

[0066] Step S2, using the public data set to train the table detection model and the table structure recognition model, and using the self-made data set to fine-tune the table detection model and the table structure recognition model;

[0067] The table detection model and table structure recognition model are trained on public datasets such as PubTables-1M, PubTabNet, and TableBank. The documents in this dataset come from a wide range of sources and are highly diverse, which greatly improves the model's ability to detect and recognize tables in documents. In the fine-tuning stage, the Adapter fine-tuning technology is used to fine-tune on a self-made dataset. Figure 2 As shown in the figure, at least one Adapter layer is inserted into the encoding module of the original RT-DETR network as a new trainable parameter, where the Adapter layer contains several convolutional layers or linear layers. During the fine-tuning process, only the parameters of the Adapter layer are updated while the parameters of other layers of the model remain unchanged. The Adapter fine-tuning technology greatly improves the efficiency of fine-tuning while ensuring good generalization ability.

[0068] Step S201, in this embodiment, the table detection model is obtained by pre-training and fine-tuning. First, a public data set such as PubTables-1M, PubTabNet, TableBank, etc. is used to train the table detection model to obtain a pre-trained model. The production of the fine-tuning data set uses the pre-trained model. First, the detection file is obtained, the file page is extracted and saved as a picture format, and then the image sample is predicted using the pre-trained table detection model. The sample predicted correctly by the model is used as a positive sample, and the sample is annotated using the prediction result, and the label and position of the table are annotated. At the same time, several pictures that do not contain tables are selected as negative samples, and the positive samples and some randomly selected negative samples are mixed to obtain a self-made data set, and finally the pre-trained model is further trained. In the actual fine-tuning process, data enhancement technology is used. For each input sample, corrosion, expansion, blurring, sharpening, cropping, scaling and other operations are applied to generate disturbed samples, and then multiple samples are merged into a batch and input into the model for training. According to the actual situation, the fine-tuning process may be repeated once or multiple times until the expected accuracy is achieved.

[0069] Step S203, the table structure recognition model can be obtained by pre-training with sample images, and the training of the table structure recognition model and the table detection model also follow the training method of pre-training plus fine-tuning. The pre-training process is completely consistent with the table detection model; in this embodiment, the fine-tuning of the table structure recognition model uses a self-made data set, which contains table images and corresponding annotations; the annotations include category information, location information and mask information; the production of the data set uses the pre-training model and text detection method, wherein the pre-training model is used to generate annotation information of the location and type of the table structure; the text detection method uses DBnet, DBnet++ and other models to detect the text in each of the table structures, and generate a pixel-level mask of the text area in each of the table structures, that is, for each pixel in the table structure, the pixels belonging to the text area are marked as 1, and the pixels belonging to the background area are marked as 0.

[0070] Step S204, in order to improve the detection ability of the table structure recognition model for cells, in the process of constructing the self-made data set, data samples containing noise are added at the same time, such as table image samples with missing table lines and cells containing pictures, and negative samples, that is, image samples that do not contain tables in the image; the construction of noise data adopts an automatic generation method. The first step is to use crawler technology to obtain XML format files corresponding to PDF files from public websites such as PubMed or USPTO, obtain the table sequence in XML, and extract its style features as a template; the second step is to use a table generator based on HTML syntax to generate a table sequence. For the generated table sequence, some text or pictures are randomly sampled to fill the cells; the third step is to apply multiple styles to the filled table, including single Cells, lines, backgrounds, texts, colors, etc., to obtain tables of various styles; in order to make the generated styles closer to the table styles in the real world while maintaining diversity, first obtain the style templates in the first step, focusing on at least two types of styles, the table style with pictures inserted in cells, and the table style with missing borders, and then apply different styles to different types of tables; the fourth step is to convert the table sequence into an image; the fifth step is to annotate the table structure using the pre-trained model assisted annotation method, and use the table images obtained in the fourth step as samples to input them into the pre-trained table structure recognition model and text detection model respectively. The correct ones predicted by the model are used as positive samples, and the prediction results are used to add labels to each sample. The sixth step is to construct the data set, and the positive samples and a small number of negative samples are merged to construct the final data set.

[0071] Step S3, converting the input file into a picture format and performing preprocessing;

[0072] Convert the input file to a preset grayscale image, and perform enhancement and denoising operations on the grayscale image, such as cropping, scaling, blurring, etc.

[0073] Step S4, inputting the preprocessed file image into the table detection model for feature extraction, obtaining the border information and category information of the table in the image according to the feature map of the input image; extracting the table in the image according to the obtained table border information and category information;

[0074] Step S401: Image feature extraction

[0075] Perform feature extraction on the target image after preprocessing. Specifically, you can select ResNet18, ResNet50, ResNet101 and other networks as the backbone network for feature extraction to obtain a multi-scale feature set of the target image, the so-called feature map pyramid.

[0076] To reduce computational overhead, after obtaining the feature map pyramid, only the highest feature map S5 is aligned and flattened into a sequence in vector form as the input of the encoder to complete the computation of the feature's intra-scale interaction. Compared with the shallower S3 and S4 features, the S5 feature has deeper, more advanced, and richer semantic features, and the self-attention mechanism (AIFI-Attention-based Intra-scale Feature Interaction) can pay more attention to the semantics of the features. Then the CCFM module (cross-scale feature fusion module) is used to complete the cross-scale feature fusion, and finally the fused multi-scale features are spliced ​​together and handed over to the subsequent network for processing. The use of the above intra-scale and cross-scale-based feature interactions and feature fusions enables the model to distinguish which parts of the input image belong to the table.

[0077] Step S402: Detect the object and select the matching box

[0078] Most object detectors such as FasterRCNN assign labels based on the local spatial position relationship (such as IoU) between pixels (or anchors) and ground truth, but this will lead to many repeated detections and waste computing resources. This type of deep learning model requires prior knowledge and manually designed rules such as box size, confidence threshold, and non-maximum suppression (NMS) threshold, thereby introducing bias based on researchers' experience.

[0079] The model used in this embodiment is fully data-driven and end-to-end. The Hungarian matching algorithm is used to solve the bipartite graph matching problem from a global perspective, and a one-to-one correspondence is established between the model output and the label (i.e., the predicted detection box and the table).

[0080] Step S403, detecting the table in the image according to the feature map of the target image, and obtaining the border information and category information of the table in the image.

[0081] In this embodiment, the features of the input image can be input into the model encoding module for encoding and vectorization to obtain a recognition latent vector, and the recognition latent vector can be input into the decoding module for decoding and output. The final output includes two types of output. The first output is label information, that is, the category of the table in the target image; the second output is the border position information of the table.

[0082] The border position information refers to a minimum circumscribed rectangular area enclosed by a table, which is determined by four coordinate points [x1, y1, x2, y2], corresponding to the coordinates of the upper left corner and the lower right corner of the rectangle respectively. The coordinates are calibrated based on the coordinates (0,0) of the upper left corner of the original page. x represents the pixel offset in the horizontal direction, and y represents the pixel offset in the vertical direction. The category refers to the category to which the detected table belongs. In this embodiment, two table label categories are included, Table and Table rotated. The difference is that the Table label represents a table in the normal direction, that is, no rotation occurs, while the Table rotated label represents a table rotated 90° counterclockwise, such as Figure 3 , Figure 4 shown.

[0083] Step S404, segment the target image using the identified table border and category information to obtain the target table image. For all detected tables, segment the target table image using the border information. For rotated tables, rotate the image 90 degrees clockwise at the same time; the target table image finally obtained only contains the table, and all rotated tables are corrected.

[0084] Step S405, if there are multiple tables in the identified page image at the same time, the extracted table file is named according to the input file name, page number and the position in the page image, which is convenient for manual rapid backtracking and review.

[0085] Step S5, inputting the table image extracted by the table detection model into the table structure detection model to detect and identify the table unit structure;

[0086] Step S501, the table structure recognition model may include an encoding module and a decoding module. The encoding module may be composed of one or more encoders, and the decoding module may be composed of one or more decoders. The number of encoders and decoders can be determined according to actual needs and training processes. In practical applications, the features of the input image can be input into the table structure encoding module for encoding and vectorization to obtain the recognition latent vector, and the recognition latent vector is input into the decoding module for decoding and output. The final output includes three types of outputs. The first output is label information, that is, the category of the table structure in the target image, the second output is the border position information of the table structure, and the third output is the mask information of the text in the table structure.

[0087] Step S502, in this embodiment, in order to improve the detection ability of the model for samples with missing table lines, a certain number of table image samples, such as 500, are collected at the same time, and the table in each sample is missing some table lines or has no table lines at all. Specifically, firstly, a table containing at least one cell is randomly generated using HTML language, and then the cells in the table are filled with text or pictures, and then the lines in the table are randomly deleted, and the deletion ratio of the table lines is between 0% and 100%, and finally a variety of tables with missing table body lines are obtained.

[0088] Step S503. In this embodiment, an encoding module and a decoding module are used in the pre-training stage of the table structure recognition model, wherein the decoder is connected to two output heads, the first output head is a feed-forward neural network, which outputs predicted table structure category information, and the first output head contains five categories of labels, Column, Row, Column header, Projected rowheader, and Spanning cell; the second output head is a feed-forward neural network, which outputs predicted table structure position information, and the position information can be the vertex position of the minimum circumscribed rectangle constituting the table structure border, that is, the border position information can be represented by an array consisting of 4 values, such as [x1, y1, x2, y2], wherein x1, y1 represent the coordinates of the upper left vertex of the border, and x2, y2 represent the coordinates of the lower right vertex of the border.

[0089] Step S504, in this embodiment, in order to further improve the model's ability to recognize table structures, in the fine-tuning stage of the table structure recognition model, the output of the table structure recognition model includes three output heads, the first output head and the second output head are consistent with the model structure of the training stage, the first output head outputs label information, and the second output head outputs the border position information of the table structure; the difference is that the third output head outputs the mask information of the table structure, and the mask information includes the mask of the text area in the table structure. The third output head may include one to multiple deconvolution layers and a 1*1 convolution layer; specifically, firstly obtain the output of the encoder as the first feature map, obtain the second feature map after dimensional transformation, use the second feature map as the input of the third output head, then upsample through no less than 1 deconvolution layer to obtain a feature map with a width and height of W and H respectively, then change the number of channels of the feature map to C through a 1*1 convolution layer to obtain a third feature map, and then perform matrix operations on the output of the decoder and the third feature map to obtain the mask prediction.

[0090] Step S505 In this embodiment, in the inference stage of the table structure recognition model, the third output head may not be included, that is, only the outputs of the first output head and the second output head are used as subsequent inputs.

[0091] Step S506, specifically, the recognition result of the table structure is as follows Figure 5 As shown, in Figure 5 There are four types of table structures, labeled Column header, Spanning cell, Column, and Row. The table header contains a Column header, which consists of three Spanning cells. The content part contains seven rows and ten columns, and the corresponding borders are surrounded by lines of corresponding colors.

[0092] Step S6, using optical character recognition (OCR) technology to recognize text information in the table image, matching the obtained table unit structure with the text information content, and generating a target table.

[0093] Step S601, Optical Character Recognition (OCR), is a process of detecting character blocks using a text detection method in an image, and then translating the shapes into computer text using a character recognition method. Specifically in this embodiment, optical character recognition can be performed on the input image, and the recognition information obtained can include text content in the table and content location information.

[0094] Step S602, generating formatted output, such as csv, html, based on the table structure identification information and the text identification information.

[0095] In this embodiment, the table structure recognition model is used to identify a table structure sequence, which includes the row and column information, header information, and merged cell information of the table. First, preprocessing is required to convert this information into a cell sequence. Specifically, according to the results output by the table detection model, the category and position of the cell are obtained by judging the intersection information of the rows and columns, and finally a cell sequence is obtained. For example, the intersection area between the Column header and the column is the cell corresponding to each column, and the cell position information, i.e., the cell border coordinates, can also be generated. The cell information is as follows: Figure 6 As shown. Next, the degree of overlap between the cell position and the position of the text content can be calculated, and the cell sequence and the corresponding text content sequence can be matched according to the degree of overlap to obtain the cells containing the content, and then the cells containing the content are sorted and integrated according to the table structure recognition results to generate a formatted output. In this embodiment, the model can be used based on the image features of the input image. First, the table is detected by the table detection model, and the table structure is identified by the table structure recognition model using the identified table. That is, the two models are used to extract features from the input image in turn, and finally the target table structure sequence is obtained; at the same time, the table image is input into the OCR model to detect the text, and the table structure recognition result and the OCR recognition result are integrated. The two methods are combined, such as Figure 7 As shown, the verification of cell position recognition and the complementarity of cell content recognition are achieved, and finally table recognition is realized and recognition accuracy is guaranteed.

[0096] Embodiment 2:

[0097] A model building module is used to build a table detection model and a table structure recognition model. The table detection model and the table structure recognition model include a number of encoding modules and decoding modules.

[0098] Model training and fine-tuning module: Use public data sets to train table detection models and table structure recognition models, and use self-made data sets to fine-tune table detection models and table structure recognition models;

[0099] Preprocessing module, converts input files into image format and performs preprocessing;

[0100] The table recognition module inputs the preprocessed file image into the table detection model for feature extraction, obtains the border information and category information of the table in the image according to the feature map of the input image; extracts the table in the image according to the obtained table border information and category information;

[0101] The table structure recognition module inputs the table image extracted by the table detection model into the table structure detection model to detect and recognize the table unit structure;

[0102] The text matching module uses optical character recognition (OCR) technology to identify text information in table images, matches the obtained table cell structure with the text information content, and generates a target table.

[0103] Embodiment three:

[0104] This embodiment provides an electronic device having a computer program stored thereon. When the computer program is executed by a processor, the method for table detection and content extraction based on image processing as described in any embodiment of the present invention is implemented.

[0105] Embodiment 4:

[0106] This embodiment provides a computer-readable medium for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement a table detection and content extraction method based on image processing as described in any embodiment of the present invention.

[0107] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0108] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0109] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0110] In several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk, and other media that can store program codes.

[0111] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A table detection and content extraction method based on image processing, characterized in that: The following steps are involved: Step S1, constructing a table detection model and a table structure recognition model, wherein the table detection model and the table structure recognition model include a plurality of encoding modules and decoding modules; Step S2, using the public data set to train the table detection model and the table structure recognition model, and using the self-made data set to fine-tune the table detection model and the table structure recognition model; Step S3, converting the input file into a picture format and performing preprocessing; Step S4, inputting the preprocessed file image into the table detection model for feature extraction, and obtaining the border information and category information of the table in the image according to the feature map of the input image; Extract the table in the image according to the obtained table border information and category information; Step S5, inputting the table image extracted by the table detection model into the table structure detection model to detect and identify the table unit structure; Step S6, using optical character recognition (OCR) technology to recognize text information in the table image, matching the obtained table unit structure with the text information content, and generating a target table.

2. The table detection and content extraction method based on image processing according to claim 1, characterized in that: The self-made dataset includes images and image labels, wherein the image labels include category labels, which are used to indicate the category information of the table or table cell in the image; position labels, which are used to indicate the position information of the table or table cell in the image; and mask labels, which are used to indicate which parts of the image need to be paid attention to in the cell detection task of the table detection model; The self-made dataset is constructed using pre-trained models to assist annotation and automatic generation. The specific construction steps are as follows: Obtain an XML file corresponding to the PDF file from a public website, obtain the table sequence in the XML, and extract its style features as a template; Use a table generator based on HTML syntax to generate a table sequence. For the generated table sequence, randomly sample some text or pictures to fill the cells. Apply multiple styles to the filled table, including cells, lines, backgrounds, texts, and colors, to obtain tables with multiple styles; in order to make the generated styles closer to the real table styles while maintaining diversity, first obtain style templates, focusing on at least two types of styles, the table style with pictures inserted in cells and the table style with missing borders, and then apply different styles to different types of tables; and convert the table sequence into a table picture; Use the pre-trained model to assist in labeling the table structure. Input the table image as a sample into the pre-trained table structure recognition model and text detection model. The correct prediction is used as a positive sample. Use the prediction result to add a label to each sample. The final dataset is constructed by merging positive samples and a small number of negative samples, where negative samples are samples that do not contain tables in the images.

3. The table detection and content extraction method based on image processing according to claim 1, characterized in that: During the fine-tuning stage, a third output head is added to the table structure recognition model to supervise the learning of the table structure recognition model and improve the detection capability of table cell structure recognition.

4. The table detection and content extraction method based on image processing according to claim 1, characterized in that: The table detection model and the table structure recognition model use the real-time target detection architecture RT-DETR; an adapter layer is inserted into the real-time target detection architecture RT-DETR for table detection and content extraction tasks.

5. The table detection and content extraction method based on image processing according to claim 4 is characterized in that: The adapter layer is used in the model fine-tuning stage. The adapter layer contains new trainable parameters, which are updated during the model fine-tuning process. The adapter layer is a convolutional layer or a linear layer, which is inserted into the encoding network to transform the features.

6. The table detection and content extraction method based on image processing according to claim 4 is characterized in that: The feature extraction using the real-time target detection architecture RT-DETR includes the following steps: Use the feature extraction network to extract image features from the input image and the table image extracted by the table detection model to obtain a multi-scale feature set of the target image, namely, a feature map pyramid; The highest feature map of the target image is flattened into a vector sequence as the input of the encoder to complete the intrinsic interaction calculation of the features; The context and channel fusion CCFM module is used to concatenate and output the fused multi-scale features.

7. The table detection and content extraction method based on image processing according to claim 6, characterized in that: After the step of splicing and outputting the fused multi-scale features using the context and channel fusion CCFM module, the method further includes: The Hungarian algorithm is used to match the output content with the label content, so that a one-to-one correspondence is established between the predicted detection box and the detection table and content.

8. A table detection and content extraction system based on image processing, characterized in that: include: A model building module is used to build a table detection model and a table structure recognition model. The table detection model and the table structure recognition model include a number of encoding modules and decoding modules. Model training and fine-tuning module: Use public data sets to train table detection models and table structure recognition models, and use self-made data sets to fine-tune table detection models and table structure recognition models; The preprocessing module converts the input file into image format and performs preprocessing; The table recognition module inputs the preprocessed file image into the table detection model for feature extraction, and obtains the border information and category information of the table in the image based on the feature map of the input image; Extract the table in the image according to the obtained table border information and category information; The table structure recognition module inputs the table image extracted by the table detection model into the table structure detection model to detect and recognize the table unit structure; The text matching module uses optical character recognition (OCR) technology to identify text information in table images, matches the obtained table cell structure with the text information content, and generates a target table.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the table detection and content extraction method based on image processing as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for table detection and content extraction based on image processing as described in any one of claims 1 to 7 is implemented.

Images