Picture type archive Chinese table identification method based on deep learning
Through deep learning methods, combined with PA-FPN model and convolutional recursive neural network, the problem of Chinese table recognition with dense text arrangement in cells and large differences in text row length is solved, and high-accurate text recognition and table content are achieved recover.
Patent Information
- Application Number
- CN202411054787.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-29
AI Technical Summary
The prior art is difficult to accurately identify archive Chinese tables with dense text arrangements in cells and extremely different text row lengths in different cells, resulting in reduced text recognition accuracy and incomplete recovery of table content.
Design a Chinese table recognition method for image-type archives based on deep learning, obtain the table structure sequence through the table detection model, use the PA-FPN model to detect the four-point coordinates of text rows in the cell, and crop the text row pictures for convolutional recursive neural network recognition , combine the aggregation rules to generate a table file represented by HTML language.
It improves the accuracy and accuracy of text recognition, and effectively improves the accuracy of table content recovery, especially when text rows in cells are arranged in densely and lengths are large.
Smart Images

Figure CN119049074A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of table recognition, and in particular to a method for recognizing Chinese tables in picture-type archives based on deep learning. Background Art
[0002] With the advancement of industrial informatization, a large amount of data has been accumulated in various fields such as finance, archive management, and smart medical care. These data are stored in a specific format, providing opportunities for analysis and utilization for relevant personnel. However, in order to prevent the table information from being edited and modified at will, the table is generally generated and stored in the form of pictures or PDF files, which causes the table to lose its original structural information that is easy for computers to understand. Therefore, in order to enable relevant personnel to more conveniently query and analyze data through computer programs, unstructured table images need to be converted into machine-readable structured representations.
[0003] Table content extraction, also known as table recognition, has the main goal of converting unstructured table images into machine-readable structured representations. In the process of table recognition, the table image needs to be analyzed, processed and parsed to extract information such as text, numbers and table structure. Traditional methods for recognizing archival tables are mostly simple or based on heuristic rules. Table recognition templates are designed for various fields, and table lines or text blocks are extracted based on the layout structure of the table, and then analyzed using rule methods. However, this method often has poor generalization ability, is not universal, and has difficulty processing complex tables. With the rapid development of deep learning, general table recognition technology based on deep learning can automatically learn table features and accurately recognize table images in various fields, including complex table image recognition.
[0004] The Chinese patent "CN115457580A Digital Archive Table Conversion Method and System" discloses a deep learning-based archive table conversion method. The method first uses a table recognition network to identify the table structure; then uses a text detection algorithm to detect single-line text on the archive table image, and obtains text detection coordinates and recognized text data; secondly, based on the intersection and union ratio and vertex distance between the text detection coordinates and the cell coordinates in the table structure, single-line to multi-line text is merged, and text belonging to the same cell is spliced; finally, the merged text is sorted, and a character string with preset writing rules is constructed in combination with the overall structure of the table and the corresponding cell recognition results in the table to generate editable table data.
[0005] The paper "Ye J, Qi X, He Y, et al. PingAn-VC Group's Solution for ICDAR2021 Competition on Scientific Literature Parsing Task B: Table Recognition to HTML[J]. 2021. DOI: 10.48550 / arXiv.2105.01848." proposed a general table recognition model based on deep learning, which mainly consists of two branches, one of which is the core TableMASTER component, which is used for cell text position regression and table structure feature prediction. The other branch uses OCR (optical character recognition) technology to obtain the text area and its content in the cell. Finally, the cell text position features, cell text content features and table structure features are logically added to obtain the final table structure recognition result.
[0006] The paper "PALIWAL SS, VISHWANATH D, RAHUL R, et al. TableNet: Deep Learning Model for End-to-end Table Detection and Tabular Data Extraction from Scanned Document Images [C]. 2019 International Conference on Document Analysis and Recognition, 2019: 128-133." proposed a one-time end-to-end reasoning based on image segmentation and autoencoder to realize table recognition. TableNet mainly consists of an encoder and two decoders. The encoder is used to extract the structural features of the table and the column, and then the two decoders are responsible for decoding the table features to form the table structure prediction map and decoding the column features to form the column prediction map. However, although the model network assembly is flexible, the end-to-end prediction will lead to error propagation, difficulty in adjusting parameters, and poor recognition ability for borderless tables.
[0007] Therefore, we adopt non-end-to-end table recognition based on deep learning as a solution for archival Chinese table recognition. Although this method can effectively improve the recognition ability of complex tables, it still has the following defects: (1) The cell contains multiple lines of text with small line spacing and dense arrangement. The existing method cannot accurately detect the minimum circumscribed rectangular outline of the text line, which will directly lead to a significant reduction in text recognition accuracy. (2) There are extreme differences in the length of text lines in the table, and most of the existing methods are trained and evaluated on benchmark datasets. There will be a large number of missed detections and false detections in the detection of text lines in archival table cells, which is directly related to the integrity of the table content recovery. Summary of the invention
[0008] In view of the above-mentioned problems, the purpose of the present invention is to provide a method for recognizing Chinese tables in picture-type archives based on deep learning. Taking the archive table picture as the object, the table structure sequence is obtained through the table detection model, and then the text line detection model is used to accurately obtain the text line area in the cell, and then the text recognition model is used to obtain the text line character string sequence, and finally a series of rules are aggregated to output the final table file represented by HTML language.
[0009] Therefore, the technical problem solved by the present invention is: how to design a Chinese table recognition method for archives where multiple lines of text are densely arranged in cells and there are extreme differences in the lengths of text lines in different cells.
[0010] The technical solution of the present invention to solve the above technical problems is as follows: A method for recognizing Chinese tables in picture-based archives based on deep learning, comprising:
[0011] Recognize the table structure of the Chinese table image in the archive, and obtain the table structure sequence expressed in HTML language and the four-point coordinates of the cell according to the recognition result;
[0012] Design a PA-FPN model to detect the four-point coordinates of the text line in the cell, and take the minimum circumscribed rectangular outline for each text line in the cell;
[0013] After cropping the original table image according to the four-point coordinates of the text line, performing text recognition on each cropped text line image through a convolutional recurrent neural network to obtain a text string;
[0014] According to the aggregation rules, the table structure sequence, cell coordinates, text row coordinates and their contents are aggregated and output to generate a complete table file represented by HTML language.
[0015] Furthermore, the table structure identification includes:
[0016] The input table image first passes through the feature extraction network to convert the input image into a feature sequence; the output feature sequence after encoding is then positionally encoded and input into the decoder; in the decoding stage, the feature sequence first passes through a transformer and is decoupled into two branches; each branch consists of two transformers, which are used to obtain cell coordinates and predict the table structure sequence represented in HTML language respectively.
[0017] Furthermore, the PA-FPN model includes:
[0018] The input table image first passes through the resnet18 feature extraction network; in the feature extraction network, the network part with the same output size is divided into one level, and the output result of the last layer of each level is selected as the feature map to obtain a feature map with 4 layers; in order to reduce the number of parameters and reduce the computational complexity, the design uses 1x1 convolution to reduce the number of channels of each feature map to 128; the obtained 4-layer feature map passes through two cascaded feature pyramid networks; the obtained 2 enhanced 4-layer feature maps first add the feature layers of the same scale element by element, and perform upsampling and connection operations to obtain a feature map with 512 channels of the same input image size; the feature map is then subjected to 1x1 convolution to obtain a feature matrix with 6 channels; 2 channels in the feature matrix respectively save the segmentation results of the text area and the text kernel, and the other 4 channels are used to save the 1×4 dimensional vector corresponding to each pixel of the input image, recorded as the similarity vector; finally, with the help of the learnable PixelAggregation algorithm, the pixels in the text area are merged into the kernel to reconstruct a complete and separated text instance, and finally the four-point coordinates of the text row in the cell are output.
[0019] Furthermore, the text recognition includes:
[0020] First, the original table image is cropped according to the four-point coordinates of the text line to obtain a single-line text image; then, the single-line text image is subjected to text recognition through a pre-trained convolutional recurrent neural network to obtain a corresponding text string.
[0021] Furthermore, the aggregation rules include:
[0022] Through the three submodules of text line detection, text recognition and table structure recognition, the minimum text line circumscribed rectangular outline in the cell and its corresponding text content, table structure sequence and its cell box are obtained; in order to generate a complete table HTML sequence, the text line box needs to be assigned to the corresponding cell. The specific rules are as follows:
[0023] Step 1: Calculate the center points of all text line boxes obtained by the PA-FPN model. If the center point is in the cell coordinates regressed by the table structure recognition module, pair the text line with the cell.
[0024] Step 2: For text line boxes whose center points are not in the cell, apply the IOU rule, that is, Where A represents the text line box and B represents the cell box; the IOU value between the text line box and all cell boxes is calculated and assigned to the cell with the maximum IOU value;
[0025] Step 3: Fill the text content corresponding to the text line box into the corresponding cell, and finally obtain a complete HTML sequence representing the table structure and its cell content.
[0026] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.
[0027] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.
[0028] A computer program product comprises a computer program, which implements the steps of the above method when executed by a processor.
[0029] Compared with the prior art, the beneficial effects of the present invention are as follows: the table recognition scheme provided by the present invention first determines the logical structure of the table and the four-point coordinates of the cell through the table structure recognition model. Then the PA-FPN model is used to detect the minimum circumscribed rectangular outline of the text lines in all cells. The model has fast speed and good performance, and can successfully identify closely arranged adjacent text lines and text lines of different lengths, thereby effectively improving the accuracy and precision of text recognition. Secondly, the text recognition model CRNN is used to perform text recognition on each cropped text line image to obtain a text string sequence. Finally, all the above results are aggregated according to the rules to output a table file expressed in HTML language. In addition, the PA-FPN model trained with a self-made text line detection data set effectively improves the accuracy of text line detection in Chinese table cells in archives, and effectively improves the accuracy of table content recovery. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments.
[0031] Figure 1 is a schematic diagram of a Chinese table image to be detected;
[0032] Figure 2A schematic diagram of the main steps of an embodiment of the table recognition method of the present invention is provided;
[0033] Figure 3 A schematic diagram of a table structure recognition model TableMASTER network architecture provided for an embodiment of the table recognition method of the present invention;
[0034] Figure 4 A schematic diagram of a feature extraction network structure in a table structure recognition model TableMASTER provided in an embodiment of a table recognition method of the present invention;
[0035] Figure 5 A schematic diagram of a multi-channel global context attention module in a table structure recognition model TableMASTER provided in an embodiment of a table recognition method of the present invention;
[0036] Figure 6 A table structure category label provided for an embodiment of the table recognition method of the present invention;
[0037] Figure 7 A schematic diagram of the network structure of a PA-FPN model for text line detection provided in an embodiment of the table recognition method of the present invention;
[0038] Figure 8 A schematic diagram of the CRNN network structure of a text recognition model provided for an embodiment of the table recognition method of the present invention;
[0039] Fig. 9 A flow chart of aggregation rules provided for an embodiment of the table recognition method of the present invention. DETAILED DESCRIPTION
[0040] The present invention provides a method for recognizing Chinese tables in picture-based archives based on deep learning, comprising:
[0041] Recognize the table structure of the Chinese table image in the archive, and obtain the table structure sequence expressed in HTML language and the four-point coordinates of the cell according to the recognition result;
[0042] Aiming at the problem that multiple lines of text are densely arranged in Chinese tables in archives and the lengths of text lines in different cells have extreme differences, a Chinese text line detection dataset is made by ourselves, and a PA-FPN model is designed to accurately detect the four-point coordinates of the text line in the cell and take the minimum circumscribed rectangle outline for the text line in each cell.
[0043] After cropping the original table image according to the four-point coordinates of the text line, performing text recognition on each cropped text line image through a convolutional recurrent neural network to obtain a text string;
[0044] The table structure sequence, cell coordinates, text row coordinates and text recognition content are aggregated and outputted using aggregation rules to generate a complete table file represented by HTML language.
[0045] As a deep learning-based Chinese table recognition method for image-based archives,
[0046] The PA-FPN model includes:
[0047] The input table image first passes through the resnet18 feature extraction network. In the feature extraction network, the network part with the same output size is divided into one stage, and the output result of the last layer of each stage is selected as the feature map to obtain a feature map with 4 layers. In order to reduce the number of parameters and reduce the computational complexity, the design uses 1x1 convolution to reduce the number of channels of each feature map to 128. However, the features extracted by the lightweight feature extraction network usually have a small receptive field and weak expression ability. Therefore, in order to enhance the expression ability of the feature map, the obtained 4-layer feature map passes through two cascaded feature pyramid networks. The obtained 2 enhanced 4-layer feature maps first add the feature layers of the same scale element by element, and perform upsampling and connection operations to obtain a feature map with 512 channels of the same input image size. The feature map is then subjected to 1x1 convolution to obtain a feature matrix with 6 channels. The 2 channels in the feature matrix respectively save the segmentation results of the text area and the text kernel, and the other 4 channels are used to save the 1×4 dimensional vector corresponding to each pixel of the input image, which is recorded as the similarity vector. Finally, with the help of a learnable Pixel Aggregation (PA) algorithm, the pixels in the text area are merged into the kernel to reconstruct a complete and separated text instance, and finally the four-point coordinates of the text line in the cell are output.
[0048] In order to make the above-mentioned objectives, technical solutions and advantages of the present invention more clear and understandable, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0049] The difficulty of recognizing Chinese table images in archives lies in the dense arrangement of text lines in cells and the extreme differences in the length of text lines in different cells. Using an end-to-end table recognition model may result in error accumulation, but the lack of interpretability of the model greatly limits the ability to analyze and improve the model. At the same time, such models require a lot of computing resources, which is not feasible for devices with limited computing power. In addition, the existing public datasets cannot meet the training of end-to-end Chinese table recognition models. Therefore, the table recognition task is decomposed into three subtasks: text line detection, content recognition, and table structure recognition. However, due to the following factors, this table recognition method is generally only applicable to the case where there is only a single line of text in the cell and the length of the text line in each cell is similar. These factors include: (1) complex row and column configuration, and cells span multiple rows and columns; (2) large blank areas appear in the table; (3) cells contain multiple lines of text, with small line spacing and dense arrangement; (4) there are extreme differences in the length of text lines in the table. Figure 1 is an example of a Chinese table image to be detected, such as Figure 1 As shown, the table image presents all the above features. However, the present invention can realize accurate detection of text lines through the text line detection PA-FPN model trained by the self-made text line detection dataset, effectively improving the accuracy of table content recovery. Figure 1 Take this as an example to illustrate.
[0050] Figure 2 1 is a schematic diagram of the main steps of an embodiment of the table recognition method of the present invention, which can output the recognized table structure and its content into a file marked with HTML language. The method includes:
[0051] The table recognition method mainly consists of two branches. One branch uses the TableMASTER network, which can simultaneously perform cell coordinate regression and table structure sequence prediction. The other branch first passes through the PA-FPN model and then separates into two processing lines. One of them is used to obtain the text row coordinates and match them with the cell coordinates regressed by the TableMASTER model, and assign the text rows to the corresponding cells to obtain the matching results. The other one cuts out the specific text row image according to the text row coordinates, and recognizes the text string sequence through the CRNN model. Finally, the text content is filled into the tag sequence of the corresponding cell, and a complete structured table marked by the HTML language can be obtained.
[0052] It should be noted that in an embodiment of the present invention, the image to be detected is a digital picture, which may be one or more pictures. When it is multiple pictures, each table can be identified separately in a custom order, and the identification steps are the same, so a table will be used as an example for explanation later.
[0053] Step 1: Identify the table structure of the Chinese table image, and obtain the table structure sequence expressed in HTML language and the four-point coordinates of the cell according to the recognition result.
[0054] The network architecture of the TableMASTER model for table structure recognition is as follows Figure 3 As shown in the figure. The input image first passes through the feature extraction network to convert the input image into a feature sequence. Compared with the ordinary resnet network, this feature extraction network connects a multi-channel attention module (MAGC) after each residual block. The sequence features output after encoding are then positionally encoded and input into the decoder. In the decoding stage, the feature sequence first passes through a regular transformer and is decoupled into two branches. Each branch consists of two regular transformers, which are used to regress cell coordinates and table structure sequence prediction respectively.
[0055] In the table structure recognition network, the feature extraction network plays a key role, which encodes the input image into a feature sequence that the transformer can decode. This model uses the encoding part of the Master network as the feature extraction network, such as Figure 4 As shown in Figure 1, the input image first passes through two 3×3 convolutions with 64 and 128 channels, and then passes through four basic blocks, each of which contains a residual block, a multi-channel global context attention module, and a 3×3 convolution. Finally, a feature sequence representing the input image is obtained.
[0056] The structure of the multi-channel global context attention module is as follows Figure 5 As shown, we can formulate it as:
[0057]
[0058] Among them, x represents the input, w k 、w v1 and w v2 represents the linear transformation matrix, h represents the number of channels, Concat(·) represents the connection operation, g i represents the global context of the i-th channel, α j is the jth pixel x j The corresponding weights, H and W represent the height and width of the feature matrix respectively. MAGC(x) represents the context modeling operation on the input image x, and LN represents layer normalization. Finally, the output y is obtained after the feature transformation operation.
[0059] Specifically, the TableMASTER model inputs an image with a dimension of 480*480*3 in the feature extraction stage, and outputs a dimension of 7*7*500. The output sequence features are then reshaped to 49*500 and input to the decoding stage. 500 is the sequence length of the model, and 49 is the dimension of the sequence features at each position. In addition, the category labels of the table are divided into 38, such as Figure 6 , plus the start and end labels, there are a total of 41 category labels used in the model. There are two of them, one for an empty cell and one for a non-empty cell. We decode the features of each position and assign it the most likely label category.
[0060] Step 2: Take the minimum circumscribed rectangle outline of the text line in the cell of the Chinese table and obtain the coordinates of the four points of the text line.
[0061] The text line detection algorithm is a technology that uses a text detection network to locate the text area in an image and represents the text area through a polygonal bounding box. The current text line detection algorithms are mainly divided into regression-based text line detection algorithms and segmentation-based text line detection algorithms. Although the former can directly regress the bounding box of the text instance in the image, it is difficult to segment closely adjacent text instances. The latter performs pixel-level prediction on the image to determine whether the pixel belongs to text, thereby obtaining the text line area. Existing mainstream algorithms such as PSENet, DBNet, PANet, etc., although they can effectively process neighboring text instances, are all trained and evaluated on benchmark datasets. They perform poorly when performing text line detection on archival Chinese table images with dense text lines in cells and extreme differences in the length of text lines in different cells, resulting in a large number of missed detections and false detections.
[0062] In order to improve the efficiency and accuracy of text detection in images, a segmentation-based text detection PA-FPN model is designed, inspired by the PANet model. In this model, we use lightweight convolutional layers and a simpler feature pyramid network (FPN) module. However, the features extracted by lightweight convolutional layers usually have a small receptive field and weaker expressive power. Therefore, we introduce a strategy of cascading multiple FPNs to fuse features at different levels, thereby enhancing the expressive power of features. Through this design, we can seek a more ideal balance between detection accuracy and detection efficiency.
[0063] The network structure of the PA-FPN model is as follows Figure 7As shown in the figure, the input table image first passes through the resnet18 network. In the feature extraction network, we divide the network parts with the same output size into one stage, and select the features of the last layer of each stage as the output. In order to reduce the number of parameters and reduce the computational complexity, the design uses 1x1 convolution to reduce the number of channels of the four feature maps to 128. Subsequently, these four feature maps use two cascaded FPN modules to fuse low-level and high-level information with less computational overhead and lower complexity, enhance features of different scales, and make features of different sizes deeper and more expressive. Next, the feature fusion module (FFM) is used to perform F n1 、F n2 The feature maps of the same scale in are added element by element, and upsampled and concatenated to obtain a feature map F with 512 channels of the same input image size. f Finally, a 1×1 convolution is used to generate an output matrix output with 6 channels, of which 2 channels store the segmentation results of the text region and the text kernel respectively, and the other 4 channels are used to store the 1×4 dimensional vector corresponding to each pixel of the input image, recorded as the similarity vector. Finally, with the help of the learnable Pixel Aggregation (PA) algorithm, the pixels in the text region are merged into the kernel, and finally a complete and separated text instance can be reconstructed.
[0064] The PA-FPN model is a text line detector that can detect dense oriented text well. It is a new powerful segmentation network. It has two advantages: (1) As a segmentation-based method, the PA-FPN model can locate oriented text of any length; (2) The model can successfully identify neighboring text instances through the PA algorithm.
[0065] The PA algorithm borrows the idea of clustering to reconstruct a complete text instance from the kernel. The specific post-processing steps are as follows:
[0066] (1) Extract output[0] from the predicted output matrix output as the score map (score_map), and then use a predefined threshold (set to 0.5) to binarize the score map to obtain the text area matrix;
[0067] (2) Binarize output[1] using a predefined threshold (set to 0.5) and multiply it element-wise with the text region matrix to obtain the kernel matrix;
[0068] (3) Take output[2:6] as the similarity matrix sv;
[0069] (4) Through connected region analysis, all connected kernel regions are extracted from the kernel matrix, and then kernels with too small areas are removed according to the threshold to obtain the numbers and positions of all kernels;
[0070] (5) For each kernel, the mean of all pixels in the kernel region in each dimension is calculated according to the similarity matrix sv to obtain a one-dimensional vector [k 1 ,k 2 ,k 3 ,k 4 ];
[0071] (6) Enqueue all pixels in the kernel;
[0072] (7) Traverse the positions of all pixels in the queue and expand from the four directions of up, down, left, and right. For each expanded point, extract its feature vector from the similarity matrix sv. If the point is in the text area and is consistent with the one-dimensional vector [k 1 ,k 2 ,k 3 ,k 4 ] is less than a predefined threshold, the point is expanded and enqueued;
[0073] (8) After pixel expansion and removal of text areas that are too small, the final text area map is obtained.
[0074] Therefore, the PA-FPN model can accurately and quickly detect text instances in Chinese tables with dense text lines in cells and extreme differences in the length of text lines in different cells. Accurate detection of text lines can effectively improve the accuracy of text recognition and provide an important basis for the recovery of text content in tables.
[0075] Step 3: Use a convolutional recursive neural network to recognize the text within each minimum circumscribed rectangle to obtain a text string.
[0076] It should be noted that Convolutional Recurrent Neural Network (CRNN) is a popular text recognition model and is widely used to solve image-based sequence recognition problems. CRNN has the following advantages:
[0077] (1) It can achieve end-to-end training, simplifying the design and training process of the entire model;
[0078] (2) Unlike traditional text recognition methods based on character segmentation, CRNN does not need to perform character segmentation on samples and can directly recognize text sequences of any length, making it more suitable for processing continuous text content;
[0079] (3) CRNN exhibits the characteristics of fast speed and good performance in text recognition tasks, and can achieve faster inference speed while maintaining high accuracy.
[0080] (4) Since the CRNN structure is simple and efficient with relatively few parameters, the model takes up little space when actually deployed and is suitable for use in resource-constrained scenarios.
[0081] Its network structure is as follows Figure 8 As shown in the figure, it is mainly composed of three parts: convolutional layer, recurrent layer and transcription layer. Taking the identification of "silty silt" as an example, the input image is first scaled to a size of 1×w×32, and the height is fixed at 32. Then the standard CNN model VGG16 (remove the fully connected layer) is used to extract features from the image. Considering that most text images are small in height and long in width, the window size of the last two pooling layers is changed from 2×2 to 1×2 to ensure that no information in the width direction is lost. Finally, 512 feature maps are obtained, each with a height of 1 and a width of w / 4. Next, the feature vector sequence required by the recurrent layer is extracted based on the feature map. Since each column of the feature map corresponds to a rectangular area on the input image, each feature vector (such as Figure 8 The red box) can be generated from left to right on the feature map by column. Each column contains 512-dimensional features, recorded as a feature vector. These w / 4 feature vectors together constitute a feature sequence. Then, this feature sequence is passed through 2 BiLSTMs and softmaxed to obtain a posterior probability matrix of size n×(w / 4), which is used as the input of the transcription layer, where n is the number of character categories. The transcription stage finds the label sequence combination with the highest probability based on the prediction of each frame. Since RNN is used for temporal classification, this leads to the problem that the text length in the input image and the predicted output of RNN cannot be aligned. Therefore, CTC (Connectionist Temporal Classification) Loss is used to solve the problem of indefinite length sequence alignment and realize end-to-end joint training of CNN and RNN.
[0082] Transcription is the process of converting the per-frame predictions made by the recurrent layer into a label sequence that is aligned with the length of the text in the input image. We use dictionary-based transcription to subject the prediction sequence to dictionary constraints and output the label sequence with the highest probability.
[0083] Specifically, the feature vector sequence is defined as x = (x 1 ,x 2 ,…,x t ,…x T ), where T is the length of the sequence, and each column x in x tThe dimension is 512. After two BiLSTMs, softmax is performed to output a posterior probability matrix, which is defined as y = (y 1 ,y 2 ,…,y t ,…y T ), each column of y t Is a collection The probability distribution on Contains all characters in the dictionary. "blank" means there are no characters in the current frame. Also defined Transformation, that is, first delete the repeated characters and then remove "-" (replace blank with "-"), and map the selected label sequence π with the highest probability to the original sequence l. For example, the label sequence "silt-quality-silt---soil" is transformed to obtain the original sequence "silt-quality-silt-soil". Therefore, given the posterior probability matrix, CTC will output the conditional probability of the original sequence l as follows:
[0084]
[0085] in Indicates all the After transformation, it is the path π of l. For any path π, there is the following definition:
[0086]
[0087] That is, the probability product of the predicted output characters in T time steps in the path π. The goal of training is to maximize the probability value of this formula, so CTC Loss is defined as the negative log-likelihood function of this formula.
[0088] Step 4: According to the aggregation rules, the table structure sequence, cell coordinates, text row coordinates and their contents are aggregated and output to generate a complete table file represented by HTML language.
[0089] In order to merge the table structure sequence, cell coordinates, text line coordinates and their contents to obtain a complete table file, we designed some rules to aggregate the results. The aggregation steps are as follows: Fig. 9 The detailed description is as follows:
[0090] (1) First, we apply the center point rule and calculate the center point based on the four-point coordinates of all text line boxes obtained by the PA-FPN model. If the center point is in the cell coordinates regressed by the table structure recognition model, we pair the text line with the cell.
[0091] (2) For text line boxes whose center points are not within the cell, the IOU rule is applied, that is, Where A represents the text line box and B represents the cell box. Calculate the IOU value between the text line box and all cell boxes and assign it to the cell with the maximum IOU value;
[0092] (3) Fill the text content corresponding to the text line box into the corresponding cell, and finally obtain a complete HTML sequence representing the table structure and its cell content.
[0093] The above description is only a preferred specific implementation of the present invention, but the protection scope of the present invention is not limited thereto.
[0094] Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention.
[0095] It should be understood that in order to simplify the present invention and help those skilled in the art understand the various aspects of the present invention, in the above description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes described in a single embodiment or described with reference to a single figure. However, the present invention should not be interpreted as the features included in the exemplary embodiments are all necessary technical features of the patent claims.
[0096] It should be understood that the modules, units, components, etc. included in the device of an embodiment of the present invention can be adaptively changed to be set in a device different from the embodiment. The different modules, units, or components included in the device of the embodiment can be combined into one module, unit, or component, or they can be divided into multiple sub-modules, sub-units, or sub-components.
Claims
1. A method for recognizing Chinese tables in image-based archives based on deep learning. It is characterized in that include: Recognize the table structure of the Chinese table image in the archive, and obtain the table structure sequence expressed in HTML language and the four-point coordinates of the cell according to the recognition result; Design the PA-FPN model to detect the four-point coordinates of the text line in the cell, and take the minimum circumscribed rectangle outline for the text line in each cell; After cropping the original table image according to the four-point coordinates of the text line, the convolutional recurrent neural network is used to perform text recognition on each cropped text line image to obtain the text string; According to the aggregation rules, the table structure sequence, cell coordinates, text row coordinates and their contents are aggregated and output to generate a complete table file represented by HTML language.
2. A method for recognizing Chinese tables in picture-based archives based on deep learning as claimed in claim 1, It is characterized in that The table structure identification includes: The input table image first passes through the feature extraction network to convert the input image into a feature sequence; the output feature sequence after encoding is then positionally encoded and input into the decoder; in the decoding stage, the feature sequence first passes through a transformer and is decoupled into two branches; each branch consists of two transformers, which are used to obtain cell coordinates and predict the table structure sequence represented in HTML language respectively.
3. A method for recognizing Chinese tables in picture-based archives based on deep learning as claimed in claim 1, It is characterized in that The PA-FPN model includes: The input table image first passes through the resnet18 feature extraction network; in the feature extraction network, the network part with the same output size is divided into one level, and the output result of the last layer of each level is selected as the feature map to obtain a feature map with 4 layers; the design uses 1x1 convolution to reduce the number of channels of each feature map to 128; the obtained 4-layer feature map passes through two cascaded feature pyramid networks; the obtained 2 enhanced 4-layer feature maps first add the feature layers of the same scale element by element, and perform upsampling and connection operations to obtain a feature map with 512 channels of the same input image size; the feature map is then subjected to 1x1 convolution to obtain a feature matrix with 6 channels; 2 channels in the feature matrix respectively save the segmentation results of the text area and the text kernel, and the other 4 channels are used to save the 1×4 dimensional vector corresponding to each pixel of the input image, recorded as the similarity vector; through the learnable PixelAggregation algorithm, the pixels in the text area are merged into the kernel to reconstruct a complete and separated text instance, and finally the four-point coordinates of the text row in the cell are output.
4. A method for recognizing Chinese tables in picture-based archives based on deep learning as claimed in claim 1, It is characterized in that The text recognition includes: First, the original table image is cropped according to the four-point coordinates of the text line to obtain a single-line text image; then, the single-line text image is subjected to text recognition through a pre-trained convolutional recurrent neural network to obtain a corresponding text string.
5. A method for recognizing Chinese tables in picture-based archives based on deep learning as claimed in claim 1, It is characterized in that The aggregation rules include: Through the three submodules of text line detection, text recognition and table structure recognition, the minimum text line circumscribed rectangular outline in the cell and its corresponding text content, table structure sequence and its cell box are obtained; in order to generate a complete table HTML sequence, the text line box needs to be assigned to the corresponding cell. The specific rules are as follows: Step 1: Calculate the center points of all text line boxes obtained by the PA-FPN model. If the center point is in the cell coordinates regressed by the table structure recognition module, pair the text line with the cell. Step 2: For text line boxes whose center points are not in the cell, apply the IOU rule, that is, Where A represents the text line box and B represents the cell box; the IOU value between the text line box and all cell boxes is calculated and assigned to the cell with the maximum IOU value; Step 3: Fill the text content corresponding to the text line box into the corresponding cell, and finally obtain a complete HTML sequence representing the table structure and its cell content.
6. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.
7. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.
8. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.
Citation Information
Patent Citations
Digital archive table conversion method and system
CN115457580A
Table recognition method and device, electronic equipment and readable storage medium
CN115273112A
Unstructured text table recognition method and system
CN115424282A
E-commerce table image recognition method and device based on deep learning
CN117710999A
General table identification method and device
CN117854090A
Cited By
Lightweight table structure identification method
CN119919952A
File digital processing method, device, equipment, medium and product
CN120071376A
End-to-end general table content identification and splitting method and device and computer readable storage medium
CN120148056A
Table restoration method and device
CN121236784A
