Table structure identification method and device, equipment and storage medium

Abstract

The application relates to the technical field of artificial intelligence, and provides a table structure identification method and device, equipment and a storage medium, the method comprises the following steps: obtaining a to-be-identified picture, the content of the to-be-identified picture comprising a frameless line table; performing feature extraction on the to-be-identified picture to obtain a first feature vector; obtaining a first label sequence corresponding to the to-be-identified picture, the first label sequence comprising table structure labels that have been predicted for the to-be-identified picture for t-1 times; performing semantic feature extraction on the first label sequence to obtain a second feature vector; obtaining a tth predicted table structure label and a tth coordinate of the to-be-identified picture according to the first feature vector and the second feature vector; and obtaining table structure information of the frameless line table according to the tth predicted table structure label and the tth coordinate. According to the application, the accuracy of frameless line table structure identification can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and storage medium for table structure recognition. Background Technology

[0002] Tables, as the most common and important form of data presentation, contain a wealth of information. With the massive increase in the number of tabular documents, the work of finding and verifying information within these documents has become increasingly arduous, and the error rate of manually retrieving data from documents is high. Therefore, the need to automatically digitize tabular documents using computer technology has become increasingly common. However, when a completed table is saved as a PDF or image file, its structural information cannot be directly obtained. In recent years, with the development of deep learning, more and more technicians have begun to study table digitization technology, that is, to restore the table structure information from PDFs or images. Currently, borderless table recognition technology has a low accuracy rate in recognizing table structure information. Summary of the Invention

[0003] Therefore, it is necessary to provide a table structure recognition method, apparatus, device, and storage medium to address the aforementioned technical problems. This method can extract image features of the table from the image to be recognized and obtain the next table structure label and corresponding cell coordinates of the frameless table based on the predicted table structure labels, thereby obtaining the table structure information of the frameless table and improving the accuracy of frameless table structure recognition.

[0004] Firstly, this application provides a table structure identification method, the method comprising:

[0005] Obtain the image to be identified, wherein the content of the image to be identified includes a frameless table;

[0006] Feature extraction is performed on the image to be identified to obtain a first feature vector;

[0007] Obtain the first tag sequence corresponding to the image to be identified. The first tag sequence includes the first t-1 predicted table structure tags of the image to be identified. Each table structure tag in the first t-1 table structure tags is predicted based on at least one table structure tag in the first tag sequence before each table structure tag. The table structure tag is a language tag in Hypertext Markup Language used to represent the frameless table structure.

[0008] Semantic features are extracted from the first label sequence to obtain a second feature vector;

[0009] Based on the first feature vector and the second feature vector, the second label sequence corresponding to the image to be identified and the t-th coordinate of the image to be identified are obtained. The second label sequence includes the first t-1 predicted table structure labels and the t-th predicted table structure label.

[0010] Based on the t-th predicted table structure label and the t-th coordinate, the table structure information of the frameless table is obtained.

[0011] In conjunction with the first aspect, in some embodiments, obtaining the second label sequence corresponding to the image to be identified and the t-th coordinate of the image to be identified based on the first feature vector and the second feature vector includes:

[0012] The input vector is obtained based on the first feature vector and the second feature vector;

[0013] The input vector is input into the first classification network to obtain the second label sequence corresponding to the image to be identified;

[0014] The input vector is fed into the second classification network to obtain the t-th coordinate of the image to be identified.

[0015] In conjunction with the first aspect, in some embodiments, obtaining the input vector based on the first feature vector and the second feature vector includes:

[0016] The first feature vector and the second feature vector are input into the decoding layer for decoding processing to obtain the decoded vector;

[0017] The first feature vector and the decoded vector are fused together to obtain the input vector.

[0018] In conjunction with the first aspect, in some embodiments, the step of extracting features from the image to be identified to obtain a first feature vector includes:

[0019] Obtain the three-channel image vector of the image to be identified, wherein the three-channel image vector is used to represent the grayscale value of each pixel in each color channel of the image to be identified;

[0020] The three-channel image vectors are input into the encoding layer for encoding processing to obtain the feature encoding vectors;

[0021] The feature encoding vector is positionally encoded to obtain the first position encoding vector corresponding to the feature encoding vector;

[0022] The first feature vector is obtained based on the feature encoding vector and the first position encoding vector.

[0023] In conjunction with the first aspect, in some embodiments, the feature encoding vector is a multi-dimensional vector, and the step of performing positional encoding on the feature encoding vector to obtain a first positional encoding vector corresponding to the feature encoding vector includes:

[0024] The last two dimensions of the feature encoding vector are flattened to obtain the rearranged image feature vector;

[0025] The rearranged image feature vector is transposed to obtain the target tensor;

[0026] Based on the target tensor, determine the first position encoding vector corresponding to the feature encoding vector.

[0027] In conjunction with the first aspect, in some embodiments, the step of extracting semantic features from the first label sequence to obtain a second feature vector includes:

[0028] The first tag sequence is input into the word embedding layer for word vectorization processing to obtain the word vectors corresponding to the first tag sequence;

[0029] The word vector is positionally encoded to obtain a second positionally encoded vector corresponding to the word vector.

[0030] The second feature vector is obtained based on the word vector and the second position encoding vector.

[0031] In conjunction with the first aspect, in some embodiments, obtaining the second feature vector based on the word vector and the second positional encoding vector includes:

[0032] The word vector and the second position encoding vector are added together to obtain the second feature vector.

[0033] Secondly, this application provides a table structure recognition device, the device comprising:

[0034] The first acquisition unit is used to acquire an image to be identified, wherein the content of the image to be identified includes a frameless table.

[0035] The first processing unit is used to extract features from the image to be identified and obtain a first feature vector.

[0036] The second acquisition unit is used to acquire the first tag sequence corresponding to the image to be identified. The first tag sequence includes the first t-1 predicted table structure tags of the image to be identified. Each table structure tag in the first t-1 table structure tags is predicted based on at least one table structure tag in the first tag sequence before each table structure tag. The table structure tag is a language tag in Hypertext Markup Language used to represent the frameless table structure.

[0037] The second processing unit is used to extract semantic features from the first label sequence to obtain a second feature vector.

[0038] The third acquisition unit is used to obtain the second label sequence corresponding to the image to be identified and the t-th coordinate of the image to be identified based on the first feature vector and the second feature vector. The second label sequence includes the first t-1 predicted table structure labels and the t-th predicted table structure label.

[0039] The fourth acquisition unit is used to obtain the table structure information of the frameless table based on the t-th predicted table structure label and the t-th coordinate.

[0040] In conjunction with the second aspect, in some embodiments, the fourth acquisition unit is specifically used for:

[0041] The input vector is obtained based on the first feature vector and the second feature vector;

[0042] This is used to input the input vector into a first classification network to obtain a second label sequence corresponding to the image to be identified;

[0043] This is used to input the input vector into the second classification network to obtain the coordinates of the t-th cell of the image to be identified.

[0044] In conjunction with the second aspect, in some embodiments, the fourth acquisition unit is specifically used for:

[0045] The first feature vector and the second feature vector are input into the decoding layer for decoding processing to obtain the decoded vector;

[0046] The first feature vector and the decoded vector are fused together to obtain the input vector.

[0047] In conjunction with the second aspect, in some embodiments, the first processing unit is specifically used for:

[0048] Obtain the three-channel image vector of the image to be identified, wherein the three-channel image vector is used to represent the grayscale value of each pixel in each color channel of the image to be identified;

[0049] The three-channel image vectors are input into the encoding layer for encoding processing to obtain the feature encoding vectors;

[0050] The feature encoding vector is positionally encoded to obtain the first position encoding vector corresponding to the feature encoding vector;

[0051] The first feature vector is obtained based on the feature encoding vector and the first position encoding vector.

[0052] In conjunction with the second aspect, in some embodiments, the first processing unit is specifically used for:

[0053] The last two dimensions of the feature encoding vector are flattened to obtain the rearranged image feature vector;

[0054] The rearranged image feature vector is transposed to obtain the target tensor;

[0055] Based on the target tensor, determine the first position encoding vector corresponding to the feature encoding vector.

[0056] In conjunction with the second aspect, in some embodiments, the second processing unit is specifically used for:

[0057] The first tag sequence is input into the word embedding layer for word vectorization processing to obtain the word vectors corresponding to the first tag sequence;

[0058] The word vector is positionally encoded to obtain a second positionally encoded vector corresponding to the word vector.

[0059] The second feature vector is obtained based on the word vector and the second position encoding vector.

[0060] In conjunction with the second aspect, in some embodiments, the second processing unit is specifically used for:

[0061] The word vector and the second position encoding vector are added together to obtain the second feature vector.

[0062] Thirdly, this application provides a table structure identification device, including a processor, a memory, and an input / output interface, wherein the processor, memory, and input / output interface are interconnected, wherein the input / output interface is used to receive and send data, the memory is used to store program code, and the processor is used to call the program code to execute the method described in the first aspect and any possible implementation thereof.

[0063] Fourthly, this application provides a computer-readable storage medium storing a computer program; when the computer program is run on one or more processors, it causes the terminal device to perform the method described in the first aspect and any possible implementation thereof.

[0064] In this embodiment, a first feature vector can be extracted from the image to be identified containing a frameless table to obtain a first label sequence corresponding to the image to be identified, which has been predicted. The first label sequence includes the first t-1 predicted table structure labels of the image to be identified. Semantic feature extraction is performed on the first label sequence to obtain a second feature vector. The t-th table structure label and the t-th coordinate of the frameless table are predicted based on the first feature vector and the second feature vector. The table structure information of the frameless table is obtained based on the predicted t-th table structure label and the t-th coordinate value. This application can extract the image features of the table from the image to be identified and obtain the next table structure label and the corresponding cell coordinate of the frameless table based on the predicted table structure labels, thereby obtaining the table structure information of the frameless table and improving the accuracy of frameless table structure recognition. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below.

[0066] Figure 1 A schematic diagram of the architecture of a table structure recognition method provided in an embodiment of this application;

[0067] Figure 2 A flowchart illustrating a table structure recognition method provided in an embodiment of this application;

[0068] Figure 3 An example diagram illustrating the flattening process of a feature encoding vector provided in this application embodiment;

[0069] Figure 4 This is a schematic diagram of a table structure recognition device provided in an embodiment of this application;

[0070] Figure 5 This is a schematic diagram of a table structure recognition device provided in an embodiment of this application. Detailed Implementation

[0071] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0072] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0073] It should be understood that the use of terms such as "first" and "second" in this document is for illustrative purposes only and to distinguish the objects being described. There is no order to these terms, nor do they indicate any particular limitation on the number of devices in the embodiments of this application, and they do not constitute any limitation on the embodiments of this application.

[0074] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0075] Foundational technologies in artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0076] This application provides a table structure identification method. To more clearly describe the solution of this application, some of the accompanying drawings involved in this application will be further introduced below.

[0077] Please see Figure 1 , Figure 1 This is a schematic diagram of the architecture of a table structure recognition method provided in an embodiment of this application. Figure 1As shown, the architecture of this table structure recognition method includes an encoding stage 101 and a decoding stage 102. The core of the encoding stage 101 consists of the RES encoding layer and positional encoding. The RES encoding module is a feature extraction module derived from a modified Deep Residual Network (ResNet). In the encoding stage 101, the input table image is processed by the RES encoding layer and positional encoding, outputting image feature values. In the decoding stage 102, the first t-1 predicted table structure labels are processed by word embedding and positional encoding, combined with the image feature values ​​output from the encoding stage 101, and then decoded through two TRM decoding layers. The decoded vector is then combined with the image feature values ​​output from the encoding stage 101, resulting in two branches. The first branch consists of a TRM decoding layer, a linear layer, and a Softmax activation function; the second branch consists of a TRM decoding layer, a linear layer, and a Sigmoid activation function. The TRM decoding layer is derived from the decoding layer network of the Transformer model, and its core structure consists of a multi-head self-attention layer, a multi-head attention layer, and a feedforward network layer. The first branch is used to obtain the table structure labels obtained from the first t predictions, and the second branch is used to obtain the t-th coordinate value. The operation of the above decoding stage 102 is repeated until the table structure labels and coordinate values ​​of each table cell of the table image are output, that is, the table structure information of the image.

[0078] Please see Figure 2 , Figure 2 This is a flowchart illustrating a table structure recognition method provided in an embodiment of this application. Figure 2 As shown, the table structure identification method in this application embodiment may include, but is not limited to, the following steps:

[0079] S201, Obtain the image to be recognized, the content of which includes a frameless table.

[0080] In this embodiment of the application, the image to be identified may be in Portable Document Format (PDF) or other image formats, and the cells in the frameless table may be partially or entirely without borders.

[0081] S202, perform feature extraction on the above image to be identified to obtain the first feature vector.

[0082] In this embodiment of the application, feature extraction is performed on the above-mentioned image to be identified to obtain a first feature vector, including but not limited to the following steps: obtaining a three-channel image vector of the above-mentioned image to be identified, wherein the three-channel image vector is used to represent the grayscale value of each pixel of the above-mentioned image to be identified in each color channel; inputting the above-mentioned three-channel image vector into an encoding layer for encoding processing to obtain a feature encoding vector; performing position encoding on the above-mentioned feature encoding vector to obtain a first position encoding vector corresponding to the above-mentioned feature encoding vector; and obtaining the above-mentioned first feature vector based on the above-mentioned feature encoding vector and the above-mentioned first position encoding vector.

[0083] In one possible implementation, the Shape of the aforementioned three-channel image vector is (3, 480, 480). The Shape can be understood as the size of a multi-dimensional matrix. When the parameters in the Shape include three values, the parameters correspond to the number of channels, the height of the image, and the width of the image, respectively. A three-channel image vector with a Shape of (3, 480, 480) represents a three-dimensional matrix with 3 channels and a height and width of 480. This means that the matrix has 480 smaller two-dimensional matrices. In each smaller two-dimensional matrix, there are 480 rows, and each row has 3 columns. The three pixel values ​​in each row correspond to the gray values ​​of the B channel, G channel, and R channel at the current position. The gray value of each channel is between 0 and 255.

[0084] In one possible implementation, the aforementioned three-channel image vector is input into the RES encoding layer for encoding processing to obtain a feature encoding vector with a shape of (512, 60, 60). The RES encoding layer is a feature extraction module derived from the block / layer parameters of a Deep Residual Network (ResNet). This feature extraction module supports input three-channel image vectors with a shape of (3, 480, 480). The aforementioned feature encoding vector with a shape of (512, 60, 60) is a three-dimensional matrix with 512 channels and a height and width of 60.

[0085] In one possible implementation, the feature encoding vector is a multi-dimensional vector. Position encoding is performed on the feature encoding vector to obtain the first position encoding vector corresponding to the feature encoding vector, including but not limited to the following steps: flattening the last two dimensions of the feature encoding vector to obtain a rearranged image feature vector; transposing the rearranged image feature vector to obtain a target tensor; and determining the first position encoding vector corresponding to the feature encoding vector based on the target tensor.

[0086] The process of flattening the last two dimensions of the above feature encoding vector to obtain the rearranged image feature vector can be found in [link to documentation]. Figure 3 , Figure 3 This is an example diagram illustrating the flattening process of a feature encoding vector, as provided in an embodiment of this application. Figure 3 The multidimensional vector shown has a size of n1*n2*n3, corresponding to a feature encoding vector size of 512*60*60. Flattening the last two dimensions of the feature encoding vector involves... Figure 3 The process of cutting the multidimensional vector shown is as follows: cutting it into sections of size n2*n3, n1*n2, or n1*n3 to obtain several cross-sections. The last two dimensions of the feature encoding vector are then flattened, i.e., according to... Figure 3 The image shows a cross-section composed of the last two dimensions. Each cross-section is a 60*60 matrix, resulting in a rearranged image feature vector with a shape of (512, 3600).

[0087] Transpose the rearranged image feature vector to obtain the target tensor. Transposing the rearranged feature vector involves swapping the values ​​of two dimensions in the rearranged image feature vector with shape (512, 3600) to obtain the target tensor with shape (3600, 512). The tensor can be understood as a two-dimensional matrix.

[0088] Based on the target tensor, determine the first position encoding vector corresponding to the feature encoding vector. Create a tensor PE with the same size and dimensions as the target tensor. The shape of tensor PE is (3600, 512). PE is a two-dimensional matrix, and the elements in tensor PE are calculated using the following formula:

[0089]

[0090]

[0091] Where pos represents the index of the pixel on the first axis of the target tensor (that is, the position of the pixel in the feature encoding vector); d model The dimension of the feature encoding vector is 512; i represents the nth dimension in the 512-dimensional feature encoding vector (i.e., 512 elements). Therefore, the above formula means adding a sin variable at even positions and a cos variable at odd positions to fill the entire PE matrix and obtain the first position encoding.

[0092] In one optional implementation, the feature encoding vector and the first position encoding vector are added together to obtain the first feature vector.

[0093] S203, obtain the first label sequence corresponding to the above image to be identified.

[0094] In this embodiment, the first tag sequence includes the first t-1 predicted table structure tags of the image to be identified. Each of the first t-1 table structure tags is predicted based on at least one table structure tag preceding each of the previous table structure tags in the first tag sequence. For example, the second table structure tag is predicted based on the previous table structure tag, the third table structure tag is predicted based on the previous two table structure tags, and so on. The table structure tags are language tags in Hyper Text Markup Language (HTML) used to represent the frameless table structure. Semantically, the table structure tags divide the table into a header, body, and footer. Please refer to Table 1 for details. Table 1 is a table structure tag classification diagram provided in this embodiment.

[0095] Table 1. Structural Label Classification Diagram

[0096] 1. 14. rowspan = "6" 27.colspan = "10" 2. 15.rowspan = "7" 28. 3. 16.rowspan = "8" 29. 4. 17.rowspan = "9" 30. 5. 18.rowspan = "10" 31. 6. 19.colspan = "2" 32. <sep>< / sep> 7. 20.colspan = "3" 33. 8.<td 21.colspan = "4" 34. 9.> 22.colspan = "5" 35. 10. rowspan = "2" 23.colspan = "6" 36. 11.rowspan = "3" 24.colspan = "7" 37. 12.rowspan = "4" 25.colspan = "8" 38. <sep>< / sep> 13.rowspan = "5" 26.colspan = "9" 39.

[0097] As shown in Table 1, the tag in table cell 1 is used to identify table header elements, the tag in table cell 3 is used to identify table body elements, the tag in table cell 5 is used to identify a row in the table, and the tag in table cell 7 is used to identify table cells. It is understandable that tags similar to those in table cell 7, such as those in table cells 30 and 31, are also used to identify table cells. The `rowspan` tag in table cell 10 is used to identify the number of rows a column spans in the table, and the `colspan` tag in table cell 19 is used to identify the number of columns a row spans in the table. It can be seen that table structure tags can be a single symbol, as in the tag in table cell 1, or multiple symbols, as in the tags in table cell 30. Only specific table structure tags, such as ``, used to identify table cells, can represent table cells in this borderless table. In the first tag sequence above, the first table structure tag is `SOS`, and the last table structure tag is `EOS`. The length of the first tag sequence is fixed at 500. If the actual output length of the first tag sequence is less than 500, it is padded to 500 with a reasonable value (this value should be greater than the maximum index value of the table structure tag).

[0098] S204, Semantic feature extraction is performed on the first label sequence to obtain the second feature vector.

[0099] In this embodiment of the application, semantic feature extraction of the first label sequence to obtain the second feature vector includes, but is not limited to, the following steps: inputting the first label sequence into a word embedding layer for word vectorization processing to obtain word vectors corresponding to the first label sequence; performing position encoding on the word vectors to obtain second position encoding vectors corresponding to the word vectors; and obtaining the second feature vector based on the word vectors and the second position encoding vectors.

[0100] In one possible implementation, the first tag sequence is input into the word embedding layer and processed into word vectors to obtain word vectors corresponding to the first tag sequence, wherein the dimension of the word vectors is 512.

[0101] In one possible implementation, the word vectors are positionally encoded to obtain a second positionally encoded vector corresponding to the word vectors. The calculation formula for positionally encoding the word vectors is as follows:

[0102]

[0103]

[0104] Where pos represents the absolute position of the table structure label in the first label sequence; d model The dimension of the word vector is represented by d. model The value is 512; i represents the nth dimension (512 dimensions, i.e., 512 elements) in the word vector. The second position encoding vector corresponding to the word vector is obtained according to the above formula.

[0105] In one possible implementation, the word vector and the second position encoding vector are added together to obtain the second feature vector.

[0106] S205, based on the first feature vector and the second feature vector, obtain the second label sequence corresponding to the image to be identified and the t-th coordinate of the image to be identified.

[0107] In this embodiment, the second label sequence includes the first t-1 predicted table structure labels and the t-th predicted table structure label. Optionally, the second label sequence can be predicted from top to bottom and left to right according to the frameless table. Obtaining the second label sequence corresponding to the image to be identified and the t-th cell coordinate of the image to be identified based on the first feature vector and the second feature vector includes, but is not limited to, the following steps: obtaining an input vector based on the first feature vector and the second feature vector; inputting the input vector into a first classification network to obtain the second label sequence corresponding to the image to be identified; and inputting the input vector into a second classification network to obtain the t-th coordinate of the image to be identified.

[0108] In one possible implementation, obtaining the input vector based on the first feature vector and the second feature vector includes, but is not limited to, the following steps: inputting the first feature vector and the second feature vector into a decoding layer for decoding processing to obtain a decoded vector; and fusing the first feature vector and the decoded vector to obtain the input vector. The first feature vector and the second feature vector are then processed through two TRM decoding layers. The TRM decoding layers are modified from the decoding layer network of the Transformer model, and their core structure consists of a multi-head self-attention layer, a multi-head attention layer, and a feedforward network layer.

[0109] In one possible implementation, the first classification network is used to predict a second label sequence for the image to be identified based on the input vector. The second label sequence includes the first t-1 predicted table structure labels and the t-th predicted table structure label. The first classification network includes a TRM decoding layer, a linear layer, and a Sofmax activation function. The output dimension of the linear layer predicting table structure labels is equal to the number of table structure label categories. Then, the probability value of each candidate table structure label at the t-th position of the sequence is obtained through the Sofmax activation function, thus obtaining the t-th predicted table structure label. For example, if the dictionary includes all possible table structure labels, the Sofmax activation function will output the probability of all table structure labels in the dictionary at the t-th position of the sequence. The table structure label with the highest probability value is the model's prediction.

[0110] In one possible implementation, the second classification network is used to predict the t-th coordinate of the image to be identified based on the input vector. The second classification network includes a TRM decoding layer, a linear layer with a sigmoid activation function, and the linear layer output dimension for predicting cell coordinates is 4. Its output value is normalized by the sigmoid activation function, which facilitates data processing and accelerates network learning. The four coordinate values ​​are the center point coordinates (x, y) and the width and height (w, h) of the cell. If the t-th table structure label predicted by the first classification network is not used to identify a table cell, then the predicted t-th coordinate is (0, 0, 0, 0). If the table structure label predicted by the first classification network is used to identify a table cell, then the predicted t-th coordinate is the coordinate corresponding to that cell.

[0111] S206. Based on the table structure label obtained from the t-th prediction and the t-th coordinate, obtain the table structure information of the frameless table.

[0112] In this embodiment of the application, the table structure label and the t-th coordinate obtained by the first branch network and the second branch network are used to identify the table cell. The table structure label used to identify the table cell will have a corresponding specific coordinate value. Based on all the table structure labels that identify the table cell and their corresponding coordinate values, the table cell of the frameless table and its corresponding cell coordinate value can be obtained, that is, the table structure information of the frameless table can be obtained.

[0113] Alternatively, based on the table structure labels and their corresponding cell coordinates, and combined with a text line detection and recognition model, the text line coordinates and text content information can be output, thus converting the table from an image format to an Excel format.

[0114] In this embodiment, the image features of the table can be extracted from the image to be identified, which contains a frameless table. Based on the image features and the predicted table structure labels, the next table structure label and the corresponding cell coordinates of the frameless table can be predicted. Based on all the predicted table structure labels and their corresponding coordinate values, the table structure information of the frameless table can be obtained, thereby improving the accuracy of the recognition of the frameless table structure.

[0115] Please see Figure 4 The diagram below illustrates the structure of a table structure recognition device as provided in this application. Figure 4 As shown, the table structure recognition device 400 may include:

[0116] The first acquisition unit 401 is used to acquire the image to be identified, the content of which includes a frameless table.

[0117] The first processing unit 402 is used to extract features from the above-mentioned image to be identified and obtain a first feature vector.

[0118] The second acquisition unit 403 is used to acquire the first tag sequence corresponding to the image to be identified. The first tag sequence includes the first t-1 predicted table structure tags of the image to be identified. Each table structure tag in the first t-1 table structure tags is predicted based on at least one table structure tag in the first tag sequence that precedes each table structure tag. The table structure tag is a language tag in Hypertext Markup Language used to represent the frameless table structure.

[0119] The second processing unit 404 is used to extract semantic features from the first label sequence to obtain a second feature vector.

[0120] The third acquisition unit 405 is used to obtain the second label sequence corresponding to the image to be identified and the t-th coordinate of the image to be identified based on the first feature vector and the second feature vector. The second label sequence includes the first t-1 predicted table structure labels and the t-th predicted table structure label.

[0121] The fourth acquisition unit 406 is used to obtain the table structure information of the frameless table based on the table structure label obtained by the t-th prediction and the t-th coordinate.

[0122] In one possible design, the fourth acquisition unit 406 is specifically used for:

[0123] Based on the first feature vector and the second feature vector, the input vector is obtained;

[0124] This is used to input the above input vector into the first classification network to obtain the second label sequence corresponding to the above image to be identified;

[0125] This is used to input the above input vector into the second classification network to obtain the coordinates of the t-th cell of the image to be identified.

[0126] In one possible design, the fourth acquisition unit 406 is specifically used for:

[0127] The first feature vector and the second feature vector are input into the decoding layer for decoding to obtain the decoded vector.

[0128] The first feature vector and the decoded vector are fused together to obtain the input vector.

[0129] In one possible design, the first processing unit 402 is specifically used for:

[0130] Obtain the three-channel image vector of the image to be identified. The three-channel image vector is used to represent the grayscale value of each pixel in each color channel of the image to be identified.

[0131] The three-channel image vectors are input into the encoding layer for encoding processing to obtain the feature encoding vector;

[0132] The above feature encoding vector is positionally encoded to obtain the first position encoding vector corresponding to the above feature encoding vector;

[0133] Based on the aforementioned feature encoding vector and the aforementioned first position encoding vector, the aforementioned first feature vector is obtained.

[0134] In one possible design, the first processing unit 402 is specifically used for:

[0135] Flatten the last two dimensions of the above feature encoding vector to obtain the rearranged image feature vector;

[0136] The target tensor is obtained by transposing the feature vectors of the rearranged image described above.

[0137] Based on the target tensor, the first position encoding vector corresponding to the feature encoding vector is determined.

[0138] In one possible design, the second processing unit 404 is specifically used for:

[0139] The first label sequence is input into the word embedding layer for word vectorization to obtain the word vectors corresponding to the first label sequence.

[0140] The word vectors above are positionally encoded to obtain the second positionally encoded vectors corresponding to the word vectors above.

[0141] Based on the word vectors and the second position encoding vectors, the second feature vector is obtained.

[0142] In one possible design, the second processing unit 404 is specifically used for:

[0143] The word vector and the second position encoding vector are added together to obtain the second feature vector.

[0144] in, Figure 4 For a detailed description of the device embodiment shown, please refer to the foregoing. Figure 2 The specific details of the method embodiments shown are not elaborated here.

[0145] Please see Figure 5 , Figure 5 This is a schematic diagram of a table structure recognition device 500 provided in an embodiment of this application. The table structure recognition device 500 includes a processor 501, a memory 502, an input / output interface 503, and a communication bus 504. The processor 501 is connected to the memory 502 and the input / output interface 503. For example, the processor 501 can be connected to the memory 502 and the input / output interface 503 through the communication bus 504.

[0146] Processor 501 is configured to support the execution of the base table structure recognition device. Figure 2The corresponding function in the table structure identification method. The processor 501 can be a Central Processing Unit (CPU), a Network Processor (NP), a hardware chip, or any combination thereof. The aforementioned hardware chip can be an Application-Specific Integrated Circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The aforementioned PLD can be a Complex Programmable Logic Device (CPLD), a Field-Programmable Gate Array (FPGA), a Generic Array Logic (GAL), or any combination thereof.

[0147] Memory 502 is used to store program code, etc. Memory 502 may include volatile memory (VM), such as random access memory (RAM); memory 502 may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory 502 may also include combinations of the above types of memory.

[0148] The input / output interface 503 is used for inputting or outputting data.

[0149] Processor 501 can call this program code to perform the following operations:

[0150] Obtain the image to be recognized, the content of which includes a table without borders;

[0151] Feature extraction is performed on the above image to be identified to obtain the first feature vector;

[0152] Obtain the first label sequence corresponding to the above image to be identified. The first label sequence includes the first t-1 predicted table structure labels of the above image to be identified. Each table structure label in the first t-1 table structure labels is predicted based on at least one table structure label in the first label sequence that precedes each of the above table structure labels. The table structure labels are language labels in Hypertext Markup Language used to represent the above frameless table structure.

[0153] Semantic features are extracted from the first label sequence to obtain the second feature vector;

[0154] Based on the first feature vector and the second feature vector, the second label sequence corresponding to the image to be identified and the t-th coordinate of the image to be identified are obtained. The second label sequence includes the first t-1 predicted table structure labels and the t-th predicted table structure label.

[0155] Based on the table structure label obtained from the t-th prediction and the t-th coordinate, the table structure information of the frameless table is obtained.

[0156] It should be noted that the implementation of each operation can also be referred to accordingly. Figure 2 The corresponding description of the method embodiment shown; the processor 501 can also cooperate with the input / output interface 503 to perform other operations in the above method embodiment.

[0157] This application also provides a computer storage medium that can store multiple instructions, which are adapted to be loaded and executed by a processor as described above. Figure 2 The method steps of the illustrated embodiment can be found in the following documentation for detailed execution. Figure 2 The specific details of the illustrated embodiments will not be elaborated here.

[0158] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0159] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A table structure identification method characterized by comprising: The table structure recognition method includes an encoding stage and a decoding stage; the encoding stage includes the following steps: Obtain the image to be identified, wherein the content of the image to be identified includes a frameless table; Obtain the three-channel image vector of the image to be identified, wherein the three-channel image vector is used to represent the grayscale value of each pixel in each color channel of the image to be identified; The three-channel image vectors are input into the encoding layer for encoding processing to obtain the feature encoding vectors; The feature encoding vector is positionally encoded to obtain the first position encoding vector corresponding to the feature encoding vector; Based on the feature encoding vector and the first position encoding vector, a first feature vector is obtained; The decoding stage includes the following steps: Obtain the first tag sequence corresponding to the image to be identified. The first tag sequence includes the first t-1 predicted table structure tags of the image to be identified. Each table structure tag in the first t-1 table structure tags is predicted based on at least one table structure tag in the first tag sequence before each table structure tag. The table structure tag is a language tag in Hypertext Markup Language used to represent the frameless table structure. Semantic features are extracted from the first label sequence to obtain a second feature vector; The first feature vector and the second feature vector are input into the decoding layer for decoding processing to obtain the decoded vector; The first feature vector and the decoded vector are fused to obtain the input vector; The second label sequence corresponding to the image to be identified and the t-th coordinate of the image to be identified are obtained based on the input vector. The second label sequence includes the first t-1 predicted table structure labels and the t-th predicted table structure label. If the t-th predicted table structure label is not used to identify a table cell, then the t-th coordinate is (0,0,0,0); if the t-th predicted table structure label is used to identify a table cell, then the t-th coordinate is the coordinate corresponding to the table cell. The steps included in the decoding stage are executed repeatedly until the table structure labels of each table structure of the image to be recognized and the coordinates of each table cell are output, thereby obtaining the table structure information of the frameless table.

2. The method of claim 1, wherein, The step of obtaining the second label sequence corresponding to the image to be identified and the t-th coordinate of the image to be identified based on the input vector includes: The input vector is input into the first classification network to obtain the second label sequence corresponding to the image to be identified; The input vector is fed into the second classification network to obtain the t-th coordinate of the image to be identified.

3. The method of claim 1, wherein, The feature encoding vector is a multi-dimensional vector. The step of performing positional encoding on the feature encoding vector to obtain the first positional encoding vector corresponding to the feature encoding vector includes: The last two dimensions of the feature encoding vector are flattened to obtain the rearranged image feature vector; The rearranged image feature vector is transposed to obtain the target tensor; Based on the target tensor, determine the first position encoding vector corresponding to the feature encoding vector.

4. The method according to any one of claims 1 to 3, characterized in that, The step of extracting semantic features from the first label sequence to obtain the second feature vector includes: The first tag sequence is input into the word embedding layer for word vectorization processing to obtain the word vectors corresponding to the first tag sequence; The word vector is positionally encoded to obtain a second positionally encoded vector corresponding to the word vector. The second feature vector is obtained based on the word vector and the second position encoding vector.

5. The method of claim 4, wherein, The step of obtaining the second feature vector based on the word vector and the second positional encoding vector includes: The word vector and the second position encoding vector are added together to obtain the second feature vector.

6. A table structure identification apparatus characterized by comprising: The apparatus includes a unit for performing the method as described in any one of claims 1-5, the apparatus comprising: The first acquisition unit is used to acquire an image to be identified, wherein the content of the image to be identified includes a frameless table. The first processing unit is used to extract features from the image to be identified and obtain a first feature vector. The second acquisition unit is used to acquire the first tag sequence corresponding to the image to be identified. The first tag sequence includes the first t-1 predicted table structure tags of the image to be identified. Each table structure tag in the first t-1 table structure tags is predicted based on at least one table structure tag in the first tag sequence before each table structure tag. The table structure tag is a language tag in Hypertext Markup Language used to represent the frameless table structure. The second processing unit is used to extract semantic features from the first label sequence to obtain a second feature vector. The third acquisition unit is used to obtain a second label sequence corresponding to the image to be identified and the t-th coordinate of the image to be identified based on the first feature vector and the second feature vector. The second label sequence includes the first t-1 predicted table structure labels and the t-th predicted table structure label. If the t-th predicted table structure label is not used to identify a table cell, then the t-th coordinate is (0,0,0,0); if the t-th predicted table structure label is used to identify a table cell, then the t-th coordinate is the coordinate corresponding to the table cell. The fourth acquisition unit is used to obtain the table structure information of the frameless table based on the table structure labels and corresponding coordinates of each table cell in the frameless table.

7. A table structure identification device characterized by comprising: The system includes a processor, a memory, and an input / output interface, which are interconnected. The input / output interface is used to input or output data, the memory is used to store program code, and the processor is used to call the program code to execute the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program; when the computer program is run on one or more processors, it performs the method as described in any one of claims 1-5.

Images