Table recognition methods, devices, electronic devices and storage media

Abstract

This invention discloses a table recognition method, apparatus, electronic device, and storage medium. The method includes: extracting features from a target table image to obtain image features of the target table; using a table structure recognition model to recognize the table structure based on the image features, obtaining a table structure recognition sequence; using a table cell recognition model to recognize cells based on the image features and the table structure recognition sequence, obtaining a cell recognition sequence; performing optical character recognition on the target table image to obtain recognition information; and generating a target table based on the table structure recognition sequence, cell recognition sequence, and recognition information. This invention can use a model to convert an image into a sequence, thereby recognizing the table structure and cells. Image-to-sequence recognition can resist background interference, improve table recognition accuracy, and achieve table recognition without table lines. By integrating the model recognition results and optical character recognition results, it achieves complementary verification of cell position recognition and cell content recognition.

Description

Technical Field

[0001] The embodiments of the present invention relate to image recognition technology, and more particularly to a table recognition method, apparatus, electronic device and storage medium. Background Technology

[0002] Traditional table recognition techniques can be divided into two categories: top-down and bottom-up. Top-up methods commonly use OpenCV (a cross-platform computer vision and machine learning software library) to detect and extract tables. Bottom-up methods have various approaches, such as using pdfplumber (a library for processing PDF information, capable of extracting tables and visualizing them) or Camelot (a PDF table data extraction tool). In developing this invention, the inventors discovered that traditional methods have the following drawbacks in table recognition tasks: First, background interference can negatively impact table recognition accuracy; second, traditional methods are less suitable for situations lacking table lines, failing to recognize the table. Summary of the Invention

[0003] This invention provides a table recognition method, apparatus, electronic device, and storage medium that can achieve table recognition without table lines and improve table recognition accuracy.

[0004] In a first aspect, embodiments of the present invention provide a table recognition method, including:

[0005] Feature extraction is performed on the target table image to obtain the image features of the target table;

[0006] The table structure recognition model is used to identify the table structure of the target table based on the image features of the target table, thereby obtaining a table structure recognition sequence;

[0007] The cell recognition model is used to identify cells in the target table based on the image features of the target table and the table structure recognition sequence, thereby obtaining a cell recognition sequence;

[0008] Optical character recognition is performed on the target table image to obtain recognition information;

[0009] The target table is generated based on the table structure recognition sequence, the cell recognition sequence, and the recognition information.

[0010] In a second aspect, embodiments of the present invention provide a table recognition device, comprising:

[0011] The feature extraction module is used to extract features from the target table image to obtain the image features of the target table;

[0012] The table structure recognition module is used to identify the table structure of the target table based on the image features of the target table using a table structure recognition model, and to obtain a table structure recognition sequence.

[0013] The table cell recognition module is used to identify cells in the target table based on the image features of the target table and the table structure recognition sequence using a table cell recognition model, thereby obtaining a cell recognition sequence.

[0014] The OCR recognition module is used to perform optical character recognition on the target table image to obtain recognition information;

[0015] The table generation module is used to generate the target table based on the table structure recognition sequence, the cell recognition sequence, and the recognition information.

[0016] Thirdly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the table recognition method as described in any embodiment of the present invention.

[0017] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the table recognition method as described in any of the embodiments of the present invention.

[0018] In this embodiment of the invention, feature extraction is performed on the target table image to obtain the image features of the target table; a table structure recognition model is used to identify the table structure of the target table based on the image features of the target table, resulting in a table structure recognition sequence; a table cell recognition model is used to identify the cells in the target table based on the image features of the target table and the table structure recognition sequence, resulting in a cell recognition sequence; optical character recognition is performed on the target table image to obtain recognition information; and the target table is generated based on the table structure recognition sequence, the cell recognition sequence, and the recognition information. In other words, this invention can use a model based on the image features of a table image to identify the structure and cells of a table represented by a sequence. That is, the model converts the image into a sequence to identify the table. Image-to-sequence recognition can resist background interference, improve the accuracy of table recognition, and even without table lines, the model can predict the cells in the table using image features and table structure. Integrating the model recognition results and the optical character recognition results, the two methods complement each other, achieving cell position verification and cell content recognition, ultimately realizing table recognition while ensuring recognition accuracy. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a table recognition method provided in an embodiment of the present invention;

[0021] Figure 2a This is an example image of a table with table lines provided in an embodiment of the present invention;

[0022] Figure 2b This is an example image of a table without table lines provided in an embodiment of the present invention;

[0023] Figure 3 This is a flowchart illustrating a training method for a table structure recognition model provided in an embodiment of the present invention.

[0024] Figure 4 This is a flowchart illustrating a training method for a table cell recognition model provided in an embodiment of the present invention.

[0025] Figure 5 This is another flowchart illustrating the table recognition method provided in this embodiment of the invention;

[0026] Figure 6 This is an example diagram of the table recognition method provided in an embodiment of the present invention;

[0027] Figure 7 This is a schematic diagram of the structure of the table recognition device provided in an embodiment of the present invention;

[0028] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] Figure 1 This is a flowchart illustrating a table recognition method provided in an embodiment of the present invention. This method can be executed by a table recognition device provided in this embodiment, which can be implemented using software and / or hardware. In a specific embodiment, the device can be integrated into an electronic device, such as a computer or server. The following embodiments will illustrate this using the integration of the device into an electronic device as an example. (Reference) Figure 1 The method may specifically include the following steps:

[0032] Step 101: Extract features from the target table image to obtain the image features of the target table.

[0033] Specifically, the target table image can be an image containing a table; that is, the target table image is a table image. The table contained in the target table image can be a table with table lines, or a table without table lines or with missing table lines. The table contained in the target table image is the target table to be identified. For example, Figure 2a The image shown is a table image containing table lines. Figure 2b The image shown is a table image containing a table without table lines.

[0034] After obtaining the target table image, it can be preprocessed for use in subsequent models. During preprocessing, the image size can be converted to a preset size. Next, feature extraction can be performed on the preprocessed image. Specifically, the target table image can be input into a feature extraction network to extract its features. The feature extraction network can be a backbone network, and various types are possible. Here, MobileNetV3 can be selected to improve the accuracy and efficiency of image feature extraction.

[0035] In this embodiment, the image features are subsequently input into the table structure recognition model and the table cell recognition model for corresponding recognition tasks. Both the table structure recognition model and the table cell recognition model can be transformer models, meaning they both include an encoding module and a decoding module. Therefore, after obtaining the image features, a feedforward neural network can be used to align the dimensions of the image features with the input dimensions required by the model's encoding module. In the multi-head self-attention part of the encoding module, the loop can be repeated twice, with each self-attention module containing eight heads.

[0036] Step 102: Use the table structure recognition model to identify the table structure of the target table based on the image features of the target table, and obtain the table structure recognition sequence.

[0037] The table structure recognition sequence refers to the sequence used to label the recognition structure of the target table; that is, it is a recognition sequence obtained during the process of using the table structure recognition model. The table structure recognition model can be pre-trained using sample images. It can include a table structure encoding module and a table structure decoding module. The encoding module can consist of one or more encoders, and the decoding module can consist of one or more decoders. The number of encoders and decoders can be determined based on actual needs and the training process. In practical applications, the image features of the target table can be input into the table structure encoding module for encoding and vectorization to obtain the first latent recognition vector. This first latent recognition vector is then input into the table structure decoding module for decoding and recognition to obtain the table structure recognition sequence.

[0038] Step 103: Use the table cell recognition model to identify cells in the target table based on the image features and table structure recognition sequence of the target table, and obtain the cell recognition sequence.

[0039] The cell recognition sequence refers to the sequence used to mark the specific location of a cell within the target table; that is, it is a location recognition sequence obtained during the use of the table cell recognition model. The table cell recognition model can be pre-trained using sample images and includes a table cell encoding module and a table cell decoding module. The table cell encoding module can consist of one or more encoders, and the table cell decoding module can consist of one or more decoders. The number of encoders and decoders can be determined based on actual needs and the training process. In practical applications, the image features of the target table can be input into the table cell encoding module for encoding and vectorization to obtain a second latent vector. This second latent vector is then input into the table cell decoding module for decoding and recognition to obtain the table cell recognition sequence.

[0040] The two models used in this embodiment (table structure recognition model and table cell recognition model) can have the same structure, but they are used to perform different recognition tasks. The table structure recognition model is used to perform the table structure recognition task, while the table cell recognition model is used to perform the cell position recognition task, i.e., the cell border regression task. In practical use, the table structure recognition task is performed first. The recognition result of the table structure recognition task (table structure recognition sequence) and the features used to generate the recognition result (image features of the target table) are simultaneously input into the table cell recognition model to perform the cell border regression task. This can improve the accuracy of cell recognition to a certain extent.

[0041] Step 104: Perform optical character recognition on the target table image to obtain recognition information.

[0042] Optical Character Recognition (OCR) is a process that uses devices to examine characters in an image, determine their shape by detecting dark and light patterns, and then translate the shape into computer text using character recognition methods. Specifically, in this embodiment, optical character recognition can be performed on a target table image, and the resulting recognition information can include cell content and the location information of the content.

[0043] Step 105: Generate the target table based on the table structure recognition sequence, cell recognition sequence, and recognition information.

[0044] Since the cell recognition sequence obtained using the table cell recognition model also represents the cell's position information, the overlap between the cell recognition sequence and the content's position information can be calculated. Based on the overlap, the cell recognition sequence is bound to the corresponding cell content to obtain cells containing content. Then, these cells containing content are sorted and integrated according to the table structure recognition sequence to generate the target table. For example, if the table cell recognition model identifies three cells (i.e., three cell border position information), the overlap between the content position information recognized by OCR and the border position information of these three cells can be calculated. The content position information with the highest overlap with these three cells is found from the OCR-recognized content position information. The content in the found position information is bound to the corresponding cell to obtain three cells containing content. These three cells containing content are then sorted and integrated according to the table structure recognition sequence to generate the final table.

[0045] In this embodiment, the model can be used to identify the structure and cells of a table represented by a sequence by utilizing the image features of the table image. That is, the model is used to convert the image into a sequence to identify the table. Image-to-sequence recognition can resist background interference and improve the accuracy of table recognition. Even without table lines, the model can predict the cells in the table using image features and table structure. The model recognition result and the OCR recognition result are integrated. The combination of the two methods achieves the complementarity of cell position recognition verification and cell content recognition, and finally realizes table recognition while ensuring recognition accuracy.

[0046] The training process of the table structure recognition model is described below, such as... Figure 3 As shown, the specific steps may include the following:

[0047] Step 201: Extract features from the sample images to obtain the image features of the sample table.

[0048] For example, the sample images come from a sample set, which may include a large number of sample images. Each sample image may contain a table, meaning the sample image is also a table image (i.e., a sample table image), and the table contained within the sample image is the sample table. Preprocessing can be performed on each sample image. Since sample images vary in size and are not suitable for subsequent tasks such as feature extraction and sequence prediction, their dimensions can be converted to a preset size during the preprocessing stage. After preprocessing, the sample images can be input into a feature extraction network to extract features, thereby obtaining the image features of the sample images. After obtaining the image features, a feedforward neural network is also needed to align the dimensions of the image features of the sample images with the input dimensions required by the model's encoding module.

[0049] Step 202: Obtain the table structure label sequence and merged cell label sequence of the sample table.

[0050] Among them, the table structure label sequence is a sequence used to label the true structure of the sample table, and the merged cell label sequence is a sequence used to label the true merged cells that exist in the sample table. That is, these two sequences are pre-labeled true values ​​used for model training. These two label sequences can be obtained through manual or machine annotation.

[0051] The table structure tag sequence of the sample table can be obtained as follows: Create a Hyper Text Markup Language (HTML) structure tag sequence for the sample table; convert the HTML structure tag sequence into a table structure tag sequence based on a tag vocabulary, which includes preset numerical identifiers corresponding to each character symbol in the HTML structure tag sequence. The HTML structure tag sequence is a tag sequence using HTML language to describe the table structure, including tags (or character symbols, elements, sequences, etc.) such as ``, ... <thread> 、 <colspan>and <rowspan>Here, is short for table datacell, representing a cell in a table, which is the label that actually stores the table data; is short for table row, representing a row in a table. The number of labels in a table indicates the number of rows of data. <thread>This indicates the header of the table; <colspan>This indicates the number of columns the cell spans horizontally. <colspan>It's used to merge columns, for example...<colspan=2> This indicates a cell that has been merged from two columns; <rowspan>This indicates the number of rows the cell spans vertically. <rowspan>It is used to merge rows. For example, <rowspan=3> represents a cell that merges three rows.

[0052] Exemplarily, there is a table with two rows and three columns. Among them, the first column and the second column of the first row of the table are merged. Then, the HTML structure tag sequence of the table can be as follows:

[0053]

[0054]

[0055]

[0056]

[0057] In addition, when recording the HTML structure tag sequence, two special characters, "sos" and "eos", can also be used. Among them, "sos" represents the starting position of the HTML structure tag sequence, and "eos" represents the ending position of the HTML structure tag sequence. For the convenience of subsequent model training, the HTML structure tag sequence can be converted into a table structure tag sequence by using a tag vocabulary. The tag vocabulary includes the preset digital identifiers corresponding to each character symbol in the HTML structure tag sequence, that is, the table structure represented by character symbols is numericalized by using the tag vocabulary. Specifically, the preset digital identifier corresponding to each character symbol in the HTML structure tag sequence can be queried in the tag vocabulary, and the obtained preset digital identifiers are sorted in order to obtain the corresponding table structure tag sequence.

[0058] Exemplarily, the tag vocabulary can be as follows: :1, :2, :3, :4, :5, :6, colspan=”2”:7, >:8, <td:9. Then, the table structure tag sequence converted from the HTML structure tag sequence in the above example can be as follows: 5, 1, 9, 7, 8, 4, 3, 4, 2, 1, 3, 4, 3, 4, 3, 4, 2, 6.

[0059] The character symbols in the HTML structure tag sequence can be divided into two categories according to their specific meanings: character symbols for merging cells and character symbols for non-merging cells. Character symbols for merging cells, such as <colspan> 、 <rowspan>Character symbols in non-merged cells, such as , , . <thread>The merged cell label sequence can be obtained as follows: Modify the preset numeric identifier corresponding to the character symbol of the merged cell in the table structure label sequence to the first numeric identifier, and modify the preset numeric identifier corresponding to the character symbol of the non-merged cell in the table structure label sequence to the second numeric identifier, so as to obtain the merged cell label sequence, where the first numeric identifier and the second numeric identifier are different.

[0060] For example, if the first numeric identifier can be 1 and the second numeric identifier can be 0, then the merged cell tag sequence obtained by converting the table structure tag sequence 5, 1, 9, 7, 8, 4, 3, 4, 2, 1, 3, 4, 3, 4, 3, 4, 2, 6 can be 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0. In summary, an HTML structure tag sequence is a string of characters, a table structure tag sequence is a string of numbers consisting of multiple different numbers (such as 5, 1, 9, 7, 8, 4), and a merged cell tag sequence is a string of numbers consisting of 0s and 1s.

[0061] Step 203: Input the image features of the sample table into the table structure encoding module for encoding and vectorization to obtain the first predicted latent vector.

[0062] The table structure encoding module can consist of multiple encoders. The input of the first encoder is the image features of the sample table, the input of each subsequent encoder is the output of the previous encoder, and the first predicted latent vector output by the last encoder enters the table structure decoding module.

[0063] Step 204: Input the first predicted latent vector and the table structure label sequence into the table structure decoding module for decoding and prediction to obtain the table structure prediction sequence and the merged cell prediction sequence.

[0064] The table structure decoding module can also include multiple decoders. Each decoder can include the following components: masked multi-head attention, a feed-forward neural network (FFN), unmasked multi-head attention, and linear layers. When the first predicted latent vector enters the decoder, the table structure prediction sequence can be encoded to generate 8-head Q (query), K (key), and V (value) matrices. These matrices are then input into the masked multi-head attention. After the current self-attention ends, the matrix is ​​processed through residual connections and layer normalization before being input into the next unmasked multi-head attention. It is worth noting that in this input process, the Q, K, and V matrices are still required, but they are different from the previous matrices. Here, the Q matrix is ​​the output of the previous masked multi-head attention, while the K and V matrices are the K and V matrices output by the encoder. This set of Q, K, and V matrices is then input into the unmasked multi-head attention. After this step, we perform residual connections and layer normalization on the output value, feed it into the feedforward neural network, and finally perform another residual connection and a layer of neural network to obtain the table structure prediction sequence.

[0065] The predicted table structure sequence obtained here is not very satisfactory. To improve the prediction performance, a special constraint network can be added when predicting the table structure to constrain merged cells, the start position of cells, and the end position of cells. Specifically, the output of the decoder of the table structure decoding module can be input into a multilayer perceptron to generate a predicted sequence of merged cells.

[0066] Step 205: Determine the table structure training loss based on the table structure prediction sequence and the table structure label sequence, and determine the merged cell training loss based on the merged cell prediction sequence and the merged cell label sequence.

[0067] The table structure prediction sequence is a sequence of predicted structures obtained during training to label the sample table, and the merged cell prediction sequence is a sequence of predicted merged cells obtained during training to label the sample table. In other words, these two sequences are predicted values ​​obtained during training. In this embodiment, the training loss for the corresponding task can be determined based on the true values ​​and the corresponding predicted values. Specifically, the table structure training loss is determined based on the table structure prediction sequence (predicted values) and the table structure label sequence (true values), and the merged cell training loss is determined based on the merged cell prediction sequence (predicted values) and the merged cell label sequence (true values).

[0068] Step 206: Adjust the parameters of the table structure encoding module and the table structure decoding module based on the table structure training loss and the merged cell training loss to obtain the table structure recognition model.

[0069] In this embodiment, when training the table structure recognition model, two labels are set: a table structure label sequence and a merged cell label sequence. The table structure label sequence is used to constrain the prediction results of the table structure. The table structure label sequence acts on the constraint network, which refines the features of the table structure prediction. This makes the table structure recognition model more sensitive to information such as merged rows and columns, the start position of cells, and the end position, thereby improving the accuracy of the model in recognizing table structures.

[0070] The training process of the table cell recognition model is described below, such as... Figure 4 As shown, the specific steps may include the following:

[0071] Step 301: Extract features from the sample images to obtain the image features of the sample table.

[0072] The sample images also require preprocessing. During preprocessing, the dimensions of these sample images can be converted to a preset size. Simultaneously, to facilitate subsequent cell border regression training, the border position information of the cells within the sample tables in the sample images can be normalized during preprocessing, converting this information into values ​​between 0 and 1. The cell border position information can be represented by the coordinates of the top-left and bottom-right corners of the cell; that is, the cell border position information can be represented by an array of four values, such as [x1, y1, x2, y2], where x1 and y1 represent the coordinates of the top-left corner of the cell, and x2 and y2 represent the coordinates of the bottom-right corner of the cell.

[0073] After preprocessing, the sample image can be input into a feature extraction network to extract features, thereby obtaining the image features of the sample image. After obtaining the image features of the sample image, it is also necessary to align the dimensions of the image features of the sample image with the input dimensions required by the model's encoding module using a feedforward neural network.

[0074] Step 302: Obtain the cell specific location label sequence and the cell global location label sequence of the sample table.

[0075] The cell-specific location label sequence is a sequence used to mark the actual location of a cell in the sample table, while the cell-global location label sequence is a sequence used to highlight the actual location of a cell in the table structure label sequence; that is, these two sequences are pre-labeled true values ​​used for model training, and these two label sequences can be obtained through manual or machine annotation.

[0076] The cell specific location label sequence can be obtained as follows: initialize the specific location label sequence according to the length of the table structure label sequence; assign values ​​to the initialized specific location label sequence corresponding to the cell position in the table structure label sequence according to the border position information of the cell in the sample table, and obtain the cell specific location label sequence.

[0077] The initialization of the specific position label sequence has the same length as the table structure label sequence, but each value in the table structure label sequence corresponds to an array in the initialization of the specific position label sequence. For example, if the table structure label sequence is 5, 1, 9, 7, 8, 4, 3, 4, 2, 1, 3, 4, 3, 4, 3, 4, 2, 6, then the initialization of the specific position label sequence could be [0, 0, 0, 0], ... [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], In this example, the length of the table structure label sequence is 18, so the initialization of the specific position label sequence can be a sequence composed of 18 arrays.

[0078] For example, according to the previous definitions, it can be known that a cell representing a table has a corresponding preset digital identifier of 3 (:3). When there are merged cells, <td also represents a cell, and the corresponding preset digital identifier of <td is 9 (<td:9). Then, the arrays corresponding to the positions of 3 and 9 in the table structure tag sequence can be assigned values according to the border position information of the cells in the sample table, or the arrays corresponding to the positions of and <td in the HTML structure tag sequence can be assigned values according to the border position information of the cells in the sample table. The arrays corresponding to other positions still retain the initialized values. When the table structure tag sequence is 5, 1, 9, 7, 8, 4, 3, 4, 2, 1, 3, 4, 3, 4, 3, 4, 2, 6, the specific cell position tag sequence can be [0, 0, 0, 0], [0, 0, 0, 0], [0.01, 0.01, 0.05, 0.1], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0.01, 0.02, 0.1, 0.2], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0.14, 0.05, 0.3, 0.4], [0, 0, 0, 0], [0.18, 0.03, 0.3, 0.4], [0, 0, 0, 0], [0.23, 0.09, 0.3, 0.4], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], that is, there are 5 cells in this example.

[0079] Since the table structure tag sequence includes a sequence representing cells and a sequence representing non-cells, the cell global position tag sequence can be obtained in the following way: replace the sequence representing cells in the table structure tag sequence with a third digital identifier, and replace the sequence representing non-cells in the table structure tag sequence with a fourth digital identifier, thereby obtaining the cell global position tag sequence; the third digital identifier and the fourth digital identifier are not the same. For example, the third digital identifier can be 1, and the fourth digital identifier can be 0.

[0080] In the above example, the sequence representing cells in the table structure tag sequence is 3 and 9 for example, then 3 and 9 in the table structure tag sequence can be replaced with 1, and other values are replaced with 0; or the sequence representing cells in the HTML structure tag sequence is and <td for example, then and <td in the HTML structure tag sequence can be replaced with 1, and other character symbols are replaced with 0. Then, the obtained cell global position tag sequence can be 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0. Generally speaking, the specific cell position tag sequence is a sequence composed of arrays, and the cell global position tag sequence is a numerical sequence composed of the numerical values 0 and 1.

[0081] Step 303: Input the image features of the sample table into the table cell encoding module for encoding and vectorization to obtain the second predicted latent vector.

[0082] The table cell encoding module can consist of multiple encoders. The input of the first encoder is the image features of the sample table, the input of each subsequent encoder is the output of the previous encoder, and the first predicted latent vector output by the last encoder is fed into the table structure decoding module.

[0083] Step 304: Input the second predicted latent vector and the table structure label sequence into the table cell decoding module for decoding and prediction to obtain the intermediate sequence of cell specific position prediction.

[0084] The table cell decoding module can also include multiple decoders, each of which can include the following components: masked multi-head attention, a feed-forward neural network (FFN), unmasked multi-head attention, and a linear layer. In the cell specific location prediction task, unlike the previous table structure prediction task, when performing the second unmasked multi-head attention, the Q matrix is ​​still the output of the previous masked multi-head attention, but the K and V matrices are the output of the previous sequence prediction decoder. This means that when performing cell bounding box regression, some features of the table structure are included, and these features have been processed by the sequence prediction decoder. This computational process directly leads to a difference in inference compared to previous tasks.

[0085] Step 305: Multiply the intermediate sequence of cell specific location prediction and the cell global location label sequence to obtain the target sequence of cell specific location prediction.

[0086] The intermediate sequence for predicting the specific cell location includes prediction sequences for the cell's location and prediction sequences for locations other than the cell's location. To reduce computation, accelerate model training, and improve training performance, the intermediate sequence for predicting the specific cell location can be multiplied by the global cell location label sequence to obtain the target sequence for predicting the specific cell location. The target sequence for predicting the specific cell location only includes the prediction sequence for the cell's location. The prediction sequences for locations other than the cell's location will become an array of zeros after multiplication. This avoids the prediction sequences for locations other than the cell's location from participating in subsequent loss calculations.

[0087] Step 306: Determine the training loss for the specific cell location based on the predicted target sequence and the label sequence for the specific cell location.

[0088] That is, the training loss for a specific cell location is determined based on the predicted value (the predicted target sequence for the specific cell location) and the actual value (the label sequence for the specific cell location).

[0089] Step 307: Adjust the parameters of the table cell encoding module and the table cell decoding module based on the training loss of the specific cell location to obtain the table cell recognition model.

[0090] In this embodiment, when training the table cell recognition model, two labels are set: a cell specific location label sequence and a cell global location label sequence. The cell specific location label sequence is used to constrain the prediction results of the cell border regression task, while the cell global location label sequence is used to obtain better features, optimize loss calculation, and improve training effect when training cell border regression.

[0091] In this embodiment, both the table structure recognition model and the table cell recognition model can be Transformer models. Using the method of this embodiment to train table recognition using Transformer models can greatly shorten the training time compared to traditional model training methods. Experiments show that using Transformer models for table recognition training in this embodiment can reduce the training time by 30-40% compared to traditional methods, and the trained model can support the recognition of tables without table lines and tables with missing table lines, with high recognition accuracy.

[0092] The following examples, using the two models trained above, further illustrate the table recognition method provided in this embodiment of the invention. Figure 5 As shown, the method may include the following steps:

[0093] Step 401: Extract features from the target table image to obtain the image features of the target table.

[0094] The target table image is a table image, and the table contained in the target table image is the target table to be identified.

[0095] Step 402: Input the image features of the target table into the table structure encoding module for encoding and vectorization to obtain the first latent recognition vector.

[0096] In this embodiment, the table structure recognition model is a transformer model, that is, the table structure recognition model includes a table structure encoding module and a table structure decoding module. The table structure encoding module consists of one or more encoders, and the table structure decoding module can consist of one or more decoders.

[0097] Step 403: The first latent vector is input into the table structure decoding module for decoding and recognition to obtain the table structure recognition sequence.

[0098] Table structure recognition sequence refers to the sequence used to represent the target table structure, which is identified by the trained table structure recognition model. The table structure recognition sequence is a series of numerical sequences.

[0099] Step 404: Input the image features of the target table into the table cell encoding module for encoding and vectorization to obtain the second latent recognition vector.

[0100] In this embodiment, the table cell recognition model is a transformer model, that is, the table cell recognition model includes a table cell encoding module and a table cell decoding module. The table cell encoding module consists of one or more encoders, and the table cell decoding module can consist of one or more decoders.

[0101] Step 405: Input the second latent vector and the table structure recognition sequence into the table cell decoding module for decoding and recognition to obtain the intermediate sequence for recognizing the specific location of the cell.

[0102] Step 406: Determine the global cell position recognition sequence of the target table based on the table structure recognition sequence.

[0103] Through the model training process, it can be deduced that the table structure recognition sequence includes sequences representing cells and sequences representing non-cells. The sequences representing cells in the table structure recognition sequence can be replaced with the third numerical identifier, and the sequences representing non-cells in the table structure recognition sequence can be replaced with the fourth numerical identifier, thereby obtaining the global cell position recognition sequence.

[0104] Step 407: Multiply the global cell location recognition sequence and the intermediate cell location recognition sequence to obtain the cell location recognition target sequence, and use the cell location recognition target sequence as the cell recognition sequence.

[0105] The intermediate sequence for cell-specific location recognition includes recognition sequences for both the cell's location and non-cell locations. Since the ultimate focus is on obtaining the recognition sequence for the cell's location, the global cell location recognition sequence and the intermediate sequence for cell-specific location recognition can be multiplied. The resulting target sequence for cell-specific location recognition only includes the recognition sequence for the cell's location; the recognition sequences for non-cell locations, after multiplication, will become an array of zeros. This target sequence for cell-specific location recognition is used as the cell recognition sequence, meaning the cell recognition sequence primarily represents the specific location information of the identified cells.

[0106] Step 408: Based on the overlap between the cell recognition sequence and the content location information in the recognition information, bind the cell recognition sequence and the cell content in the recognition information to obtain a cell containing content.

[0107] Since the data obtained by model recognition mainly consists of two parts: the structure of the table (table structure recognition sequence) and the specific location of the cells in the table (cell recognition sequence), but the specific content of the cells is not solved by model recognition, while OCR recognition can identify the specific content of the cells and the location information of the content. Therefore, in this embodiment, the results of model recognition and OCR recognition can be combined to generate the target table.

[0108] Specifically, the overlap between the location information obtained by the two methods (model recognition and OCR recognition) can be calculated. The location information with the highest overlap with the location information obtained by the model recognition is then identified from the location information obtained by the OCR recognition. This allows the location information obtained by the two methods to be bound together. After binding, the content of the cell recognized by the model and the cell recognized by the OCR are linked according to the cell binding information, resulting in a cell containing content. During this process, for cells with empty content, the corresponding cell content recognized by the OCR will also be empty.

[0109] For example, such as Figure 6 As shown, Figure 6 In this diagram, A, B, C, D, E, and F represent the cell content recognized by the OCR, each containing corresponding location information (i.e., the location of the content). 1, 2, 3, and 4 represent the cells recognized by the model (i.e., cell location information). We can find the location information with the highest overlap with the location information of these four cells from the location information of these six contents, bind the found location information to the cell, and then bind the cell to the corresponding content based on the binding information. For example, if the location of content A has the highest overlap with the location of cell 1, then content A can be bound to cell 1; the location of content B has the highest overlap with the location of cell 2, then content B can be bound to cell 2; the location of content E has the highest overlap with the location of cell 3, then content E can be bound to cell 3; and the locations of content C, D, and F have the highest overlap with the location of cell 4, then content C, D, and F can be bound to cell 4. After binding, we can obtain cells containing the content. If a cell is empty, the content recognized by the OCR at the corresponding location will also be empty.

[0110] Step 409: Sort and integrate the cells containing content according to the table structure recognition sequence to generate the target table.

[0111] When sorting cells, the order of cells and their corresponding bound content needs to be adjusted simultaneously to avoid content misalignment or confusion. Since the table structure recognition sequence output by the model is a numerical sequence, it can be converted back to an HTML structure recognition sequence based on the tag word list. The table structure can then be reconstructed from the HTML structure recognition sequence. Based on the table structure, the cells containing content are sorted and integrated to generate the target table.

[0112] In this embodiment, feature extraction is performed on the target table image to obtain the image features of the target table; a table structure recognition model is used to identify the table structure of the target table based on the image features of the target table, resulting in a table structure recognition sequence; a table cell recognition model is used to identify the cells in the target table based on the image features of the target table and the table structure recognition sequence, resulting in a cell recognition sequence; optical character recognition is performed on the target table image to obtain recognition information; and the target table is generated based on the table structure recognition sequence, cell recognition sequence, and recognition information. In other words, this invention can use a model based on the image features of a table image to identify the structure and cells of a table represented by a sequence. That is, the model converts the image into a sequence to identify the table. Image-to-sequence recognition can resist background interference and improve the accuracy of table recognition. Even without table lines, the model can predict the cells in the table using image features and table structure. Integrating the model recognition results and optical character recognition results, the two methods complement each other, achieving cell position verification and cell content recognition, ultimately realizing table recognition while ensuring recognition accuracy.

[0113] In addition, the model training process is constrained by setting table structure label sequences, merged cell label sequences, cell specific position label sequences, and cell global position label sequences. These constraints are only applied during model training, and will not participate in the final table generation process during model use. These constraints can help the model better focus on merged rows and columns, table start and end positions, optimize the features the model focuses on, and improve model training efficiency and results.

[0114] Furthermore, prioritize the table structure recognition task. Input the recognition results (table structure recognition sequence) and the features used to generate the recognition results (image features of the target table) into the table cell recognition model to perform cell border regression. This can improve the accuracy of cell recognition to some extent.

[0115] Figure 7 This is a structural diagram of a table recognition device provided in an embodiment of the present invention. This device is suitable for executing the table recognition method provided in an embodiment of the present invention. Figure 7 As shown, the device may specifically include:

[0116] The feature extraction module 501 is used to extract features from the target table image to obtain the image features of the target table;

[0117] The table structure recognition module 502 is used to recognize the table structure of the target table based on the image features of the target table using a table structure recognition model, and obtain a table structure recognition sequence.

[0118] The table cell recognition module 503 is used to recognize cells in the target table based on the image features of the target table and the table structure recognition sequence using a table cell recognition model, and to obtain a cell recognition sequence.

[0119] OCR recognition module 504 is used to perform optical character recognition on the target table image to obtain recognition information;

[0120] The table generation module 505 is used to generate the target table based on the table structure recognition sequence, the cell recognition sequence, and the recognition information.

[0121] In one embodiment, the device includes a first training module, the first training module being used to:

[0122] Feature extraction is performed on the sample images to obtain the image features of the sample table;

[0123] Obtain the table structure label sequence and merged cell label sequence of the sample table. The table structure label sequence is a sequence used to mark the actual structure of the sample table, and the merged cell label sequence is a sequence used to mark the actual merged cells that exist in the sample table.

[0124] The table structure recognition model is obtained by training a model based on the image features of the sample table, the table structure label sequence, and the merged cell label sequence.

[0125] In one embodiment, the first training module obtains the table structure label sequence of the sample table, including:

[0126] Create a sequence of Hypertext Markup Language (HTML) structural tags for the sample table;

[0127] The HTML structure tag sequence is converted into the table structure tag sequence according to the tag vocabulary, wherein the tag vocabulary includes preset numerical identifiers corresponding to each character symbol in the HTML structure tag sequence.

[0128] In one embodiment, each character symbol in the HTML structure tag sequence includes character symbols for merged cells and character symbols for non-merged cells. The first training module obtains the merged cell tag sequence of the sample table, including:

[0129] The preset numeric identifier corresponding to the character symbol of the merged cell in the table structure label sequence is modified to a first numeric identifier, and the preset numeric identifier corresponding to the character symbol of the non-merged cell in the table structure label sequence is modified to a second numeric identifier, thus obtaining the merged cell label sequence, wherein the first numeric identifier and the second numeric identifier are different.

[0130] In one embodiment, the table structure recognition model includes a table structure encoding module and a table structure decoding module. A first training module trains the model based on the image features of the sample table, the table structure label sequence, and the merged cell label sequence to obtain the table structure recognition model, including:

[0131] The image features of the sample table are input into the table structure encoding module for encoding and vectorization to obtain the first predicted latent vector;

[0132] The first predicted latent vector and the table structure label sequence are input into the table structure decoding module for decoding and prediction to obtain the table structure prediction sequence and the merged cell prediction sequence.

[0133] The table structure training loss is determined based on the table structure prediction sequence and the table structure label sequence, and the merged cell training loss is determined based on the merged cell prediction sequence and the merged cell label sequence.

[0134] The parameters of the table structure encoding module and the table structure decoding module are adjusted based on the table structure training loss and the merged cell training loss to obtain the table structure recognition model.

[0135] In one embodiment, the table structure recognition module 502 is specifically used for:

[0136] The image features of the target table are input into the table structure encoding module for encoding and vectorization to obtain the first latent recognition vector;

[0137] The first latent vector is input into the table structure decoding module for decoding and recognition to obtain the table structure recognition sequence.

[0138] In one embodiment, the device further includes a second training module, the second training module being used for:

[0139] Obtain the cell specific location label sequence and the cell global location label sequence of the sample table. The cell specific location label sequence is used to mark the actual specific location of the cell in the sample table, and the cell global location label sequence is used to highlight the actual location of the cell in the table structure label sequence.

[0140] The table cell recognition model is obtained by training a model based on the image features of the sample table, the table structure label sequence, the cell specific location label sequence, and the cell global location label sequence.

[0141] In one embodiment, the second training module obtains the specific location label sequence of the cells in the sample table, including:

[0142] Initialize the specific position label sequence according to the length of the table structure label sequence;

[0143] Based on the border position information of the cells in the sample table, the initial specific position label sequence corresponding to the cell position in the table structure label sequence is assigned a value to obtain the specific position label sequence of the cell.

[0144] In one embodiment, the table structure label sequence includes a sequence representing cells and a sequence representing non-cells. The second training module obtains the global cell position label sequence of the sample table, including:

[0145] The sequence representing cells in the table structure label sequence is replaced with a third numerical identifier, and the sequence representing non-cells in the table structure label sequence is replaced with a fourth numerical identifier, thereby obtaining the global cell position label sequence, wherein the third numerical identifier and the fourth numerical identifier are different.

[0146] In one embodiment, the table cell recognition model includes a table cell encoding module and a table cell decoding module. A second training module trains the model based on the image features of the sample table, the table structure label sequence, the cell specific location label sequence, and the cell global location label sequence to obtain the table cell recognition model, including:

[0147] The image features of the sample table are input into the table cell encoding module for encoding and vectorization to obtain the second predicted latent vector;

[0148] The second predicted latent vector and the table structure label sequence are input into the table cell decoding module for decoding and prediction to obtain the intermediate sequence of cell specific position prediction.

[0149] Multiply the intermediate sequence of cell location prediction by the cell global location label sequence to obtain the target sequence of cell location prediction;

[0150] The training loss for the specific cell location is determined based on the predicted target sequence and the label sequence for the specific cell location.

[0151] The parameters of the table cell encoding module and the table cell decoding module are adjusted based on the training loss of the specific location of the cell, thereby obtaining the table cell recognition model.

[0152] In one embodiment, the table cell recognition module 503 is specifically used for:

[0153] The image features of the target table are input into the table cell encoding module for encoding and vectorization to obtain the second latent recognition vector;

[0154] The second latent recognition vector and the table structure recognition sequence are input into the table cell decoding module for decoding and recognition to obtain the intermediate sequence for the specific location recognition of the cell.

[0155] The global cell position identification sequence of the target table is determined based on the table structure identification sequence;

[0156] Multiply the global cell location recognition sequence and the intermediate cell location recognition sequence to obtain the cell location recognition target sequence, and use the cell location recognition target sequence as the cell recognition sequence.

[0157] In one embodiment, the identification information includes cell content and the location information of the content, and the table generation module 505 is specifically used for:

[0158] Based on the degree of overlap between the cell recognition sequence and the location information of the content, the cell recognition sequence and the corresponding cell content are bound together to obtain a cell containing content;

[0159] The cells containing content are sorted and integrated according to the table structure identification sequence to generate the target table.

[0160] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0161] The apparatus of this invention can extract features from a target table image to obtain image features of the target table; use a table structure recognition model to identify the table structure of the target table based on the image features of the target table, obtaining a table structure recognition sequence; use a table cell recognition model to identify cells in the target table based on the image features of the target table and the table structure recognition sequence, obtaining a cell recognition sequence; perform optical character recognition on the target table image to obtain recognition information; and generate the target table based on the table structure recognition sequence, cell recognition sequence, and recognition information. In other words, this invention can use a model based on the image features of a table image to identify the structure and cells of a table represented by a sequence. That is, it uses the model to convert the image into a sequence to identify the table. Image-to-sequence recognition can resist background interference and improve the accuracy of table recognition. Even without table lines, the model can predict the cells in the table using image features and table structure. Integrating the model recognition results and optical character recognition results, the two methods complement each other, achieving table recognition while ensuring recognition accuracy.

[0162] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the table recognition method provided in any of the above embodiments.

[0163] This invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the table recognition method provided in any of the above embodiments.

[0164] The following is for reference. Figure 8 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing an electronic device according to embodiments of the present invention. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0165] like Figure 8 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the computer system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0166] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0167] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this invention.

[0168] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0170] The modules and / or units described in the embodiments of the present invention can be implemented in software or hardware. The described modules and / or units can also be housed in a processor; for example, a processor can be described as including a feature extraction module, a table structure recognition module, a table cell recognition module, an OCR recognition module, and a table generation module. The names of these modules do not necessarily constitute a limitation on the module itself.

[0171] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: extracting features from a target table image to obtain image features of the target table; recognizing the table structure of the target table based on the image features of the target table using a table structure recognition model to obtain a table structure recognition sequence; recognizing cells in the target table based on the image features of the target table and the table structure recognition sequence using a table cell recognition model to obtain a cell recognition sequence; performing optical character recognition on the target table image to obtain recognition information; and generating the target table based on the structure recognition sequence, the cell recognition sequence, and the recognition information.

[0172] According to the technical solution of the present invention, the structure and cells of a table represented by a sequence can be identified by using a model based on the image features of a table image. That is, the model is used to convert the image into a sequence to identify the table. Image-to-sequence recognition can resist background interference and improve the accuracy of table recognition. Even without table lines, the model can predict the cells in the table using image features and table structure. The model recognition results and optical character recognition results are integrated. The combination of the two methods achieves the complementarity of cell position recognition verification and cell content recognition, and finally realizes table recognition while ensuring recognition accuracy.

[0173] In addition, the model training process is constrained by setting table structure label sequences, merged cell label sequences, cell specific position label sequences, and cell global position label sequences. These constraints are only applied during model training, and will not participate in the final table generation process during model use. These constraints can help the model better focus on merged rows and columns, table start and end positions, optimize the features the model focuses on, and improve model training efficiency and results.

[0174] Furthermore, prioritize the table structure recognition task. Input the recognition results (table structure recognition sequence) and the features used to generate the recognition results (image features of the target table) into the table cell recognition model to perform cell border regression. This can improve the accuracy of cell recognition to some extent.

[0175] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0176] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.< / thread> < / rowspan> < / colspan> < / rowspan> < / rowspan> < / colspan> < / colspan> < / thread> < / rowspan> < / colspan> < / thread>

Claims

1. A table recognition method, characterized in that, include: Feature extraction is performed on the target table image to obtain the image features of the target table; The table structure recognition model is used to identify the table structure of the target table based on the image features of the target table, thereby obtaining a table structure recognition sequence; The cell recognition model is used to identify cells in the target table based on the image features of the target table and the table structure recognition sequence, thereby obtaining a cell recognition sequence; Optical character recognition is performed on the target table image to obtain recognition information; The target table is generated based on the table structure identification sequence, the cell identification sequence, and the identification information; The table structure recognition model is trained in the following way: feature extraction is performed on the sample image to obtain the image features of the sample table; Obtain the table structure label sequence and merged cell label sequence of the sample table. The table structure label sequence is a sequence used to mark the actual structure of the sample table, and the merged cell label sequence is a sequence used to mark the actual merged cells that exist in the sample table. The table structure recognition model is obtained by training a model based on the image features of the sample table, the table structure label sequence, and the merged cell label sequence. The table cell recognition model is trained as follows: A sequence of specific cell location labels and a sequence of global cell location labels are obtained from the sample table. The specific cell location label sequence is used to mark the actual location of the cell in the sample table, and the global cell location label sequence is used to highlight the actual location of the cell in the table structure label sequence. The model is trained based on the image features of the sample table, the table structure label sequence, the specific cell location label sequence, and the global cell location label sequence to obtain the table cell recognition model.

2. The method according to claim 1, characterized in that, The step of obtaining the table structure label sequence of the sample table includes: Create a sequence of Hypertext Markup Language (HTML) structural tags for the sample table; The HTML structure tag sequence is converted into the table structure tag sequence according to the tag vocabulary, wherein the tag vocabulary includes preset numerical identifiers corresponding to each character symbol in the HTML structure tag sequence.

3. The method according to claim 2, characterized in that, The character symbols in the HTML structure tag sequence include character symbols for merged cells and character symbols for non-merged cells. Obtaining the merged cell tag sequence of the sample table includes: The preset numeric identifier corresponding to the character symbol of the merged cell in the table structure label sequence is modified to a first numeric identifier, and the preset numeric identifier corresponding to the character symbol of the non-merged cell in the table structure label sequence is modified to a second numeric identifier, thus obtaining the merged cell label sequence, wherein the first numeric identifier and the second numeric identifier are different.

4. The method according to any one of claims 1 to 3, characterized in that, The table structure recognition model includes a table structure encoding module and a table structure decoding module. The model is trained based on the image features of the sample table, the table structure label sequence, and the merged cell label sequence to obtain the table structure recognition model, including: The image features of the sample table are input into the table structure encoding module for encoding and vectorization to obtain the first predicted latent vector; The first predicted latent vector and the table structure label sequence are input into the table structure decoding module for decoding and prediction to obtain the table structure prediction sequence and the merged cell prediction sequence. The table structure training loss is determined based on the table structure prediction sequence and the table structure label sequence, and the merged cell training loss is determined based on the merged cell prediction sequence and the merged cell label sequence. The parameters of the table structure encoding module and the table structure decoding module are adjusted based on the table structure training loss and the merged cell training loss to obtain the table structure recognition model.

5. The method according to claim 4, characterized in that, The step of using a table structure recognition model to identify the table structure of the target table based on the image features of the target table, and obtaining a table structure recognition sequence, includes: The image features of the target table are input into the table structure encoding module for encoding and vectorization to obtain the first latent recognition vector; The first latent vector is input into the table structure decoding module for decoding and recognition to obtain the table structure recognition sequence.

6. The method according to claim 1, characterized in that, The step of obtaining the specific location label sequence of the cells in the sample table includes: Initialize the specific position label sequence according to the length of the table structure label sequence; Based on the border position information of the cells in the sample table, the initial specific position label sequence corresponding to the cell position in the table structure label sequence is assigned a value to obtain the specific position label sequence of the cell.

7. The method according to claim 1, characterized in that, The table structure label sequence includes a sequence representing cells and a sequence representing non-cells. Obtaining the global cell position label sequence of the sample table includes: The sequence representing cells in the table structure label sequence is replaced with a third numerical identifier, and the sequence representing non-cells in the table structure label sequence is replaced with a fourth numerical identifier, thereby obtaining the global position label sequence of the cells. The third numerical identifier and the fourth numerical identifier are different.

8. The method according to any one of claims 1, 6, and 7, characterized in that, The table cell recognition model includes a table cell encoding module and a table cell decoding module. The model is trained based on the image features of the sample table, the table structure label sequence, the cell specific location label sequence, and the cell global location label sequence to obtain the table cell recognition model, including: The image features of the sample table are input into the table cell encoding module for encoding and vectorization to obtain the second predicted latent vector; The second predicted latent vector and the table structure label sequence are input into the table cell decoding module for decoding and prediction to obtain the intermediate sequence of cell specific position prediction. Multiply the intermediate sequence of cell location prediction by the cell global location label sequence to obtain the target sequence of cell location prediction; The training loss for the specific cell location is determined based on the predicted target sequence and the label sequence for the specific cell location. The parameters of the table cell encoding module and the table cell decoding module are adjusted based on the training loss of the specific location of the cell, thereby obtaining the table cell recognition model.

9. The method according to claim 8, characterized in that, The method of using a table cell recognition model to identify cells in the target table based on the image features of the target table and the table structure recognition sequence, and obtaining a cell recognition sequence, includes: The image features of the target table are input into the table cell encoding module for encoding and vectorization to obtain the second latent recognition vector; The second latent recognition vector and the table structure recognition sequence are input into the table cell decoding module for decoding and recognition to obtain the intermediate sequence for the specific location recognition of the cell. The global cell position identification sequence of the target table is determined based on the table structure identification sequence; Multiply the global cell location recognition sequence and the intermediate cell location recognition sequence to obtain the cell location recognition target sequence, and use the cell location recognition target sequence as the cell recognition sequence.

10. The method according to claim 1, characterized in that, The identification information includes cell content and content location information. Generating the target table based on the table structure identification sequence, the cell identification sequence, and the identification information includes: Based on the degree of overlap between the cell recognition sequence and the location information of the content, the cell recognition sequence and the corresponding cell content are bound together to obtain a cell containing content; The cells containing content are sorted and integrated according to the table structure identification sequence to generate the target table.

11. A form recognition device, characterized in that, include: The feature extraction module is used to extract features from the target table image to obtain the image features of the target table; The table structure recognition module is used to identify the table structure of the target table based on the image features of the target table using a table structure recognition model, and to obtain a table structure recognition sequence. The table cell recognition module is used to identify cells in the target table based on the image features of the target table and the table structure recognition sequence using a table cell recognition model, thereby obtaining a cell recognition sequence. The OCR recognition module is used to perform optical character recognition on the target table image to obtain recognition information; The table generation module is used to generate the target table based on the table structure recognition sequence, the cell recognition sequence, and the recognition information; The first training module is used to extract features from the sample images to obtain the image features of the sample table. Obtain the table structure label sequence and merged cell label sequence of the sample table. The table structure label sequence is a sequence used to mark the actual structure of the sample table, and the merged cell label sequence is a sequence used to mark the actual merged cells that exist in the sample table. The table structure recognition model is obtained by training a model based on the image features of the sample table, the table structure label sequence, and the merged cell label sequence. The second training module is used to obtain the cell specific location label sequence and the cell global location label sequence of the sample table. The cell specific location label sequence is used to mark the actual specific location of the cell in the sample table, and the cell global location label sequence is used to highlight the actual location of the cell in the table structure label sequence. The model is trained based on the image features of the sample table, the table structure label sequence, the cell specific location label sequence, and the cell global location label sequence to obtain the table cell recognition model.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the table recognition method as described in any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the table recognition method as described in any one of claims 1 to 10.

Images