Table image recognition method, device, equipment and readable storage medium

Abstract

The present invention relates to the field of image processing technology, and specifically discloses a table image recognition method, device, equipment and readable storage medium, the method comprising: obtaining a table image to be recognized, performing OCR recognition on the table image to be recognized, and obtaining single-line text information of each cell frame in the table to be recognized; performing convolutional neural network analysis and recognition on the table image to be recognized, and obtaining multiple feature maps of the table image to be recognized; obtaining a pre-trained table image recognition model, inputting the single-line text information and the feature map into the table image recognition model, and obtaining a calculation result output by the table image recognition model, the calculation result including output word fragments and the association relationship between the output word fragments, the association relationship including a self-attention score and a relative position vector, wherein the output word fragment is obtained after the table image recognition model performs word segmentation on the single-line text information; generating a table structure corresponding to the table image to be recognized according to the output word fragments and the association relationship between the output word fragments.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a table image recognition method, device, equipment and readable storage medium. Background Art

[0002] As a means of organizing and collating data, tables are used by people from all walks of life. Usually, in order to prevent the table content from being arbitrarily changed and causing damage to the original data, the table is saved in the form of a picture. When the table data is needed, it can be obtained accordingly by identifying the table picture.

[0003] In the prior art, OCR detection is used to obtain the text content of table images, and rows, columns and merged cells are deduced based on the spatial arrangement information of the text. However, this table recognition mechanism has the following two problems: the text in two adjacent cells with a small spatial row spacing will be incorrectly spliced ​​together, or multiple lines of text in the same cell with a large row spacing will be divided into different cells, which leads to low accuracy in table structure recognition.

[0004] There is currently no effective solution to the technical problem that in the prior art, table recognition is performed based on the spatial arrangement information of text content, resulting in low accuracy in table structure recognition. Summary of the invention

[0005] The purpose of the present invention is to provide a table image recognition method, device, equipment and readable storage medium, which can solve the technical problem in the prior art that table recognition based on the spatial arrangement information of text content leads to low table structure recognition accuracy.

[0006] One aspect of the present invention provides a table image recognition method, the method comprising: obtaining a table image to be recognized, performing OCR recognition on the table image to be recognized, and obtaining single-line text information of each cell box in the table to be recognized; performing convolutional neural network analysis and recognition on the table image to be recognized, and obtaining multiple feature maps of the table image to be recognized; obtaining a pre-trained table image recognition model, inputting the single-line text information and the feature map into the table image recognition model, and obtaining a calculation result output by the table image recognition model, the calculation result including output word fragments and the association relationship between the output word fragments, the association relationship including a self-attention score and a relative position vector, wherein the output word fragment is obtained after the table image recognition model performs word segmentation on the single-line text information; according to the output word fragments and the association relationship between the output word fragments, a table structure corresponding to the table image to be recognized is generated.

[0007] Optionally, the step of generating a table structure corresponding to the table image to be identified based on the output word fragments and the association relationship between the output word fragments includes: judging the association relationship between the output word fragments, and determining the positional relationship between the output word fragment and the output word fragment previous to the output word fragment based on the judgment result, the positional relationship including that the output word fragment and the output word fragment previous to the output word fragment are in the same frame or in different frames, that the unit frame to which the output word fragment belongs and the unit frame to which the output word fragment previous to the output word fragment belongs are in the same row or in different rows, and that the unit frame to which the output word fragment previous to the output word fragment belongs are in a merged unit frame relationship or a standard unit frame relationship; generating a table structure corresponding to the table image to be identified based on the positional relationship between the output word fragment and the output word fragment previous to the output word fragment.

[0008] Optionally, the step of determining the positional relationship between the output word fragment and the output word fragment before the output word fragment according to the judgment result includes: determining whether the output word fragment and the output word fragment before the output word fragment are in the same frame or in a different frame relationship according to the self-attention scores of the output word fragment and the output word fragment before the output word fragment; determining whether the unit frame to which the output word fragment before the output word fragment belongs is a merged unit frame relationship or a standard unit frame relationship according to the self-attention scores of the output word fragment and the output word fragment before the output word fragment; determining whether the unit frame to which the output word fragment belongs and the unit frame to which the output word fragment before the output word fragment belongs are in the same row or in a different row according to the self-attention scores of the output word fragment and the output word fragment before the output word fragment and the relative position vector in the x-direction.

[0009] Optionally, the step of determining whether an output word fragment and the output word fragment preceding the output word fragment are in the same frame or in a different frame relationship based on the self-attention score of the output word fragment and the output word fragment preceding the output word fragment includes: when the self-attention score of the output word fragment and the output word fragment preceding the output word fragment is greater than or equal to a first preset threshold, determining that the output word fragment and the output word fragment preceding the output word fragment are in the same frame relationship; when the self-attention score of the output word fragment and the output word fragment preceding the output word fragment is less than the first preset threshold, determining that the output word fragment and the output word fragment preceding the output word fragment are in a different frame relationship.

[0010] Optionally, the step of determining whether the unit frame to which the previous output word fragment of the output word fragment belongs is a merged unit frame relationship or a standard unit frame relationship based on the self-attention score of the output word fragment and the previous output word fragment of the output word fragment includes: when the self-attention score of the output word fragment and the previous output word fragment of the output word fragment is greater than or equal to a second preset threshold and less than a first preset threshold, then determining that the unit frame to which the previous output word fragment of the output word fragment belongs is a merged unit frame relationship, wherein the first preset threshold is greater than the second preset threshold; when the self-attention score of the output word fragment and the previous output word fragment of the output word fragment is less than the second preset threshold, then determining that the unit frame to which the previous output word fragment of the output word fragment belongs is a standard unit frame relationship.

[0011] Optionally, the step of determining whether a unit box to which an output word fragment belongs and a unit box to which an output word fragment belongs before the output word fragment are in the same row or in a different row according to the self-attention scores of the output word fragment and the output word fragment before the output word fragment and the relative position vector in the x-direction includes: when the self-attention scores of the output word fragment and the output word fragment before the output word fragment are less than a second preset threshold and the relative position vector in the x-direction is due west, it is determined that the unit box to which the output word fragment belongs and the unit box to which the output word fragment belongs before the output word fragment are in the same row; when the self-attention scores of the output word fragment and the output word fragment before the output word fragment are less than a second preset threshold and the relative position vector in the x-direction is due east, it is determined that the unit box to which the output word fragment belongs and the unit box to which the output word fragment belongs before the output word fragment are in a different row.

[0012] Optionally, the step of outputting the calculation result of the table image recognition model by the following method includes: in the encoder: performing word segmentation operation on the single-line text information according to the preset BERT algorithm to generate multiple output word fragments; sorting the output word fragments and feature maps according to the original unit frame layout of the table image to be recognized, and obtaining an initial text sequence and an initial feature map sequence; inputting the initial text sequence and the initial feature map sequence into the embedding layer to generate an initial text sequence and an initial feature map sequence represented by vectors respectively; splicing the initial text sequence and the initial feature map sequence represented by vectors to generate an image text sequence; inputting the image text sequence into the conversion layer, performing self-attention and relative position calculation on each output word fragment in the image text sequence, and obtaining the association relationship between the output word fragments in the image text sequence, the association relationship including the self-attention score and the one-dimensional relative position vector between the output word fragments; outputting the association relationship between the image text sequence and the output word fragments to the decoder; in the decoder: receiving the association relationship between the image text sequence and the output word fragments, and obtaining a preset start mark, embedding the preset start mark in the image text sequence One-dimensional position, calculate the association relationship between the preset starting mark and each output word fragment after the one-dimensional position of the preset starting mark in the image text sequence, and output the output word fragment with the highest self-attention score and / or the smallest modulus of the one-dimensional relative position vector with the preset starting mark; input the association relationship between the image text sequence and the output word fragment to the decoder again, and concatenate the preset starting mark and the previous output word fragment in sequence to form a first text sequence for input, and embed the one-dimensional position of the previous output word fragment in the image text sequence, calculate the association relationship between the previous output word fragment in the first text sequence and each output word fragment after the one-dimensional position of the previous output word fragment in the image text sequence, output the output word fragment with the highest self-attention score and / or the smallest modulus of the one-dimensional relative position vector with the previous output word fragment and the association relationship between the output word fragment and the previous output word fragment, repeat this step until the last output word fragment and the association relationship between the output word fragment and the previous output word fragment are output; each output word fragment of the decoder and the association relationship between each output word fragment and the corresponding previous output word fragment are used as the output of the table image recognition model.

[0013] Another aspect of the present invention provides a table image recognition device, which includes: a first acquisition module, used to acquire a table image to be recognized, perform OCR recognition on the table image to be recognized, and obtain single-line text information of each cell box in the table to be recognized; a second acquisition module, used to perform convolutional neural network analysis and recognition on the table image to be recognized, and obtain multiple feature maps of the table image to be recognized; a recognition module, used to acquire a pre-trained table image recognition model, input the single-line text information and the feature map into the table image recognition model, and obtain the calculation result output by the table image recognition model, the calculation result includes output word fragments and the association relationship between the output word fragments, the association relationship includes a self-attention score and a relative position vector, wherein the output word fragment is obtained after the table image recognition model performs word segmentation on the single-line text information; a generation module, used to generate a table structure corresponding to the table image to be recognized based on the output word fragments and the association relationship between the output word fragments.

[0014] Another aspect of the present invention provides a computer device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the table image recognition method of any of the above-mentioned embodiments when executing the computer program.

[0015] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the table image recognition method of any of the above embodiments is implemented. Further, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of a blockchain node, etc.

[0016] The table image recognition method, device, equipment and readable storage medium provided by the present invention, after obtaining the table image to be recognized, perform OCR recognition on the table image to be recognized, obtain the single-line text information of each cell frame in the table to be recognized, and perform convolutional neural network analysis and recognition on the table image to be recognized, obtain multiple feature maps of the table image to be recognized, and by collecting two types of table information, single-line text information and feature map, the basic data of the table image is made more comprehensive and diverse; then, the basic data is calculated based on the pre-trained table image recognition model to obtain output word fragments and the association relationship between the output word fragments, wherein the association relationship between the output word fragments includes two types of self-attention scores and relative position vectors. The self-attention score can reflect the contextual semantic relationship or belonging relationship between the output word fragments, while the relative position vector reflects the spatial layout relationship between the output word fragments. The combination of the two dimensional information can more comprehensively reflect the relevance of the text content in the table to be recognized. Accordingly, the table image to be recognized is constructed according to the output word fragments and the association relationship between the output word fragments, which can better restore the text content of each cell frame of the table to be recognized. The recognition accuracy is bound to be higher than that of inferring the table structure only based on the spatial layout information of the text content. Based on the present application, the technical problem of low table structure recognition accuracy caused by performing table recognition based on the spatial arrangement information of the text content in the prior art is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0018] Figure 1 An optional flow chart of the table image recognition method provided in the first embodiment of the present invention is shown;

[0019] Figure 2 An optional application schematic diagram of the table image recognition method based on OCR recognition and convolutional neural network recognition provided in the first embodiment of the present invention is shown;

[0020] Figure 3 An optional structural block diagram of a table image recognition model provided in the first embodiment of the present invention is shown;

[0021] Figure 4 An optional structural block diagram of an encoder in a table image recognition model provided in Embodiment 1 of the present invention is shown;

[0022] Figure 5An optional structural block diagram of a table image recognition device provided in Embodiment 2 of the present invention is shown; and

[0023] Figure 6 A block diagram of an optional computer device suitable for implementing a table image recognition method provided in Embodiment 3 of the present invention is shown. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0025] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0026] Embodiment 1

[0027] Embodiment 1 of the present invention provides a table image recognition method, specifically, Figure 1 An optional flow chart of the table image recognition method is shown. Figure 1 As shown, the table image recognition method may include steps S1 to S4, wherein:

[0028] Step S1, obtaining a picture of a table to be identified, performing OCR recognition on the picture of the table to be identified, and obtaining single-line text information of each cell frame in the table to be identified.

[0029] After obtaining the image of the table to be identified, the image of the table to be identified is identified based on the OCR (Optical Character Recognition) technology to obtain the single-line text content of each cell box in the table to be identified. Preferably, a detection frame corresponding to each single-line text content can be identified, wherein the number of single-line text content is determined by the specific text content of the actual cell box, and the detection frame is the minimum rectangular bounding box enveloping the single-line text information in a cell box, which is used to indicate the specific position of the single-line text content.

[0030] Step S2, performing convolutional neural network analysis and recognition on the table image to be recognized, and obtaining multiple feature maps of the table image to be recognized.

[0031] Based on the convolutional neural network, the image dimension of the table image to be identified is recognized to obtain multiple feature maps, which contain local area text content (for example, several single-line text information). The multiple feature maps and each single-line text information are used as the basic data of the table image to be identified to realize the recognition of the table structure.

[0032] specific Figure 2 FIG. 1 shows an optional application schematic diagram of the table image recognition method based on OCR recognition and convolutional neural network recognition in the first embodiment of the present invention, such as Figure 2 As shown, after obtaining the table image to be recognized, a convolutional neural network is used to obtain multiple feature maps (V1, V2, V3 and V4), which are sorted according to the original layout of the table image to be recognized to obtain a feature map sequence; and OCR recognition is used to obtain text recognition content, Line1 is the first line of text recognition content, including three word fragments T1, T2 and T3, Line2 is the second line of text recognition content, including four word fragments T4, T5, T6 and T7, and T2 and T4 are occluded through parser calculation so that the table image recognition model can make predictions and obtain the specific content of the occluded position, thereby improving the recognition accuracy of the table image recognition model. Figure 2 This is only an illustrative description and is not intended to limit the present invention.

[0033] It should be noted here that the purpose of setting the above steps S1 and S2 is to obtain the table image to be identified, and then identify the single-line text information of each cell in the table based on OCR and multiple feature maps of the table image to be identified based on the convolutional neural network. There is no special processing timing requirement. That is to say, in the specific implementation, the above step S1 can be executed first, and then the above step S2, or the above step S2 can be executed first, and then the above step S1, or both steps S1 and S2 can be executed at the same time.

[0034] This embodiment recognizes the table structure by respectively acquiring two dimensional information, text content and image features, to determine the relevant information between text contents and between text contents and images, thereby providing complete basic data for the recognition of table structures and improving the recognition accuracy. At the same time, there is no need to extract the border lines of the cell frames of the table images. For various types of table images (for example, table images with frame lines and table images without frame lines), valid data can be extracted for complete recognition, and the application is strong.

[0035] Step S3, obtain a pre-trained table image recognition model, input the single-line text information and the feature map into the table image recognition model, and obtain the calculation results output by the table image recognition model, the calculation results include output word fragments and the association between the output word fragments, the association relationship includes a self-attention score and a relative position vector, wherein the output word fragment is obtained after the table image recognition model segments the single-line text information.

[0036] After obtaining the basic data of the table to be recognized, the pre-trained table image recognition model is used to process the basic data. During the recognition process, after the single-line text information is segmented, multiple output word fragments are obtained, the self-attention scores and relative position relationships between the output word fragments are calculated, and the output word fragments and the association relationships between the output word fragments are output.

[0037] In this embodiment, the self-attention score represents the contextual semantic relationship or belonging relationship of the text content, and the relative position represents the spatial layout relationship of the text content in the table. Combining the above two dimensional information can clearly reflect the correlation between the text contents and improve the recognition accuracy of the table image.

[0038] In an optional embodiment of the present invention, a solution for outputting calculation results of a table image recognition model is also provided. Figure 3 A structural block diagram of a table image recognition model is shown, such as Figure 3 As shown, the table image recognition model includes an encoder and a decoder. In the encoder, the single-line text information and feature map are preprocessed (word segmentation, sorting) to obtain the initial text sequence and the initial feature map series. The initial text sequence and the initial feature map sequence are sequentially input into the embedding layer and the conversion layer for corresponding calculations to obtain the output result of the encoder. The decoder receives the output result of the encoder as input, and uses its own output result as another input. It is calculated in turn through the masked multi-head self-attention layer, the encoding and decoding layer, and the softmax layer, and the calculation result is output. In specific implementation, the table image recognition model outputs the calculation result through the following scheme:

[0039] The following steps A1 to A6 are performed in the encoder:

[0040] Step A1, perform word segmentation operation on a single line of text information according to a preset BERT algorithm to generate multiple output word fragments.

[0041] Use the preset BERT algorithm to perform word segmentation on a single line of text information to obtain multiple output word fragments. For example, if the single line of text information is "Beijing welcomes you", after word segmentation, three output word fragments "Beijing", "Welcome" and "You" will be obtained.

[0042] Step A2, sorting the output word fragments and feature maps according to the original unit frame layout of the table image to be recognized, to obtain an initial text sequence and an initial feature map sequence.

[0043] All output word fragments are sorted according to the original cell frame layout of the table structure to be identified, and each feature map is numbered from left to right row by row according to its position in the table image to be identified to form an initial feature map sequence; the three conditions of the cell frame position to which the output word fragment belongs, the row position to which the output word fragment belongs, and the position of the output word fragment in the single-line text content are compared in turn for sorting, and the smaller values ​​of each position are sorted first, where the output word fragment position is expressed in the form of ? -? -?, where ? is an Arabic numeral. For example, 1-1-1 is the first output word fragment in the first row of the first cell. There are multiple output word fragments, A(1-1-4), B(1-2-1), C(1-1-1), and D(1-3-1). After comparing the positions of the above output word fragments, the sequence generated by sorting is CABD. And add before the starting position of the generated sequence respectively. <cls>Add after mark and end position <sep>Mark, generate the initial text sequence, determine whether the number of output word fragments in the initial text sequence is the same as the preset text sequence length, if it is less than the preset text sequence length, then <sep>Use after marking <padding>If the length of the text sequence is greater than the preset length, the text sequence will be segmented.

[0044] specific Figure 4 An optional structural block diagram of an encoder in a table image recognition model is shown, and Figure 3 The local encoder structure in the table image recognition model corresponds to, Figure 4 As shown, the initial text sequence and the initial feature map sequence are processed by the embedding layer and the conversion layer of the encoder in turn to generate a corresponding image text sequence and a corresponding association relationship. Specifically, the following steps A3 to A6 are performed in the encoder to calculate the initial text sequence and the initial feature map sequence:

[0045] In step A3, the initial text sequence and the initial feature map sequence are input into the embedding layer to generate the initial text sequence and the initial feature map sequence represented by vectors respectively.

[0046] Obtain the one-dimensional position information, two-dimensional position information and segmentation information corresponding to each feature map, and input the corresponding embedding layer for linear mapping to obtain the corresponding vector representation of the same dimension, wherein the one-dimensional position information is the position of the feature map in the feature map sequence, the two-dimensional position coordinates are the coordinates of the upper left corner vertex, the coordinates of the lower right corner vertex of the feature map and the width and height of the feature map, and the segmentation information is the image category. For example, in this embodiment, the segmentation vector of the image can be represented as 1, and each feature map vector and the corresponding one-dimensional position vector, two-dimensional position vector, and segmentation vector are added and combined to obtain an initial feature map sequence represented by a vector; each output word fragment, the corresponding one-dimensional position vector, and the ... The one-dimensional position information, two-dimensional position information and segmentation information are input into the corresponding embedding layer for linear mapping to obtain a vector representation with the same dimension as the feature map, where the one-dimensional position information is the position of the output word fragment in the text sequence, and the two-dimensional position information is the upper left corner vertex coordinates, the lower right corner vertex coordinates of the detection box corresponding to the output word fragment, and the width and height of the feature map, where one detection box may correspond to multiple output word fragments, and the segmentation information is the text category. For example, in this embodiment, the segmentation vector of the text can be expressed as 2, and each output word fragment vector and the corresponding one-dimensional position vector, two-dimensional position vector, and segmentation vector are added and combined to obtain the initial text sequence represented by a vector.

[0047] Step A4, concatenating the initial text sequence represented by vectors and the initial feature map sequence to generate an image text sequence.

[0048] The initial text sequence represented by the vector and the initial feature map sequence are concatenated to generate the image text sequence. For each word fragment vector a in the image text sequence i , each a i There are three matrices: query matrix (a matrix for querying other word fragments), key matrix (a matrix used to be queried by other word fragments), and value matrix (a matrix used to represent the value of the extracted information). i Multiplying the above three matrices respectively gives q i , k i and v i matrix.

[0049] Step A5, input the image text sequence into the conversion layer, perform self-attention and relative position calculation on each output word fragment in the image text sequence, and obtain the association relationship between the output word fragments in the image text sequence. The association relationship includes the self-attention score and one-dimensional relative position vector between the output word fragments.

[0050] The image-text sequence is input into the conversion layer, and the q of each output word fragment is converted based on the spatial-aware self-attention mechanism. i The matrix k for all output word fragments i The matrix is ​​calculated by dot multiplication to obtain the association relationship between each output word fragment and other output word fragments in the image text sequence. The association relationship refers to the self-attention score and relative position vector of each output word fragment and other output word fragments, where the relative position vector includes the one-dimensional relative position vector between the output word fragments, the relative position vector in the x direction, and the relative position vector in the y direction.

[0051] Step A6, outputting the association relationship between the image text sequence and the output word fragment to the decoder.

[0052] The association between different output word fragments and the image text sequence (k corresponding to each output word fragment) i and v i matrix) as the output of the encoder.

[0053] In order to improve the accuracy of the table image recognition model, Figure 4 The encoder structure also sets up a pre-training module to train the occluded area in the single-line text information obtained after OCR recognition (see Figure 2 ) to make predictions, such as Figure 4 As shown in the figure, the pre-training module includes a masked visual language module, a text-image alignment module, and a text-image matching module. The masked text content is predicted based on the masked visual language module, and the obscured text content is inferred by combining the vector information of the image text sequence; the masked area of ​​the image is predicted based on the text-image alignment module, and the word-level binary classification is performed using the text part output of the model to predict whether each word is covered. Based on the text-image matching module, a part of the document image is randomly replaced or discarded to construct a negative sample of image-text mismatch, and whether the image and text match is predicted by document-level binary classification.

[0054] The following steps A7-A9 are performed in the decoder:

[0055] Step A7, receiving the association relationship between the image text sequence and the output word fragments, and obtaining a preset starting mark, embedding the one-dimensional position of the preset starting mark in the image text sequence, calculating the association relationship between the preset starting mark and each output word fragment after the one-dimensional position of the preset starting mark in the image text sequence, and outputting the output word fragment with the highest self-attention score with the preset starting mark and / or the smallest modulus of the one-dimensional relative position vector.

[0056] When the decoder is in the starting state and has not yet output anything, the preset starting mark is obtained and input into the masked multi-head self-attention layer, and the one-dimensional position of the preset starting mark in the image text sequence is embedded, and the q of the preset starting mark is calculated. i matrix, inputs the association between the image text sequence and the output word fragment into the encoding and decoding layer, and at the same time i The matrix is ​​input into the encoding and decoding layer, and the q i The k of other output word fragments after the preset one-dimensional position of the starting mark in the matrix and image text sequence i Interact and calculate multiple self-attention weight matrices, and then use multiple self-attention weight matrices to compare with other output word fragments v i The matrix is ​​calculated to obtain a decoding vector, which indicates how much attention should be placed on the output word fragments at different positions in the image text sequence when predicting the output word fragments related to the preset initial mark. The decoding vector is normalized using the softmax layer to obtain the influence probability (self-attention score) of each output word fragment on the preset starting mark, and the output word fragment vector with the largest probability value is found. The corresponding output word fragment is searched in the preset word list and output. Since the decoder has not yet output anything, there is no association between the output word fragments.

[0057] Step A8, the association relationship between the image text sequence and the output word fragment is input into the decoder again, and the preset start mark and the previous output word fragment are sequentially spliced ​​to form a first text sequence for input, and the one-dimensional position of the previous output word fragment in the image text sequence is embedded, and the association relationship between the previous output word fragment in the first text sequence and each output word fragment after the one-dimensional position of the previous output word fragment in the image text sequence is calculated, and the output word fragment with the highest self-attention score and / or the smallest one-dimensional relative position vector modulus with the previous output word fragment and the association relationship between the output word fragment and the previous output word fragment are output, and this step is repeated until the last word fragment and the association relationship between the output word fragment and the previous output word fragment are output.

[0058] The preset start flag is sequentially concatenated with the previous output word fragment, input into the masked multi-head self-attention layer, and the one-dimensional position of the previous output word fragment in the image text sequence is embedded to calculate the q of the previous output word fragment. i matrix, the association between the image text sequence and the output word fragment is input into the encoding and decoding layer again, and the q i The matrix is ​​input into the encoding and decoding layer, and the q i The matrix and the k of other output word fragments after the one-dimensional position of the previous output word fragment in the image text sequence i The matrix interacts and calculates multiple self-attention weight matrices, and then the multiple self-attention weight matrices are compared with the v of other output word fragments. i The matrix is ​​calculated to obtain a decoding vector, which indicates how much attention should be placed on the output word fragments at different positions in the image text sequence when predicting the output word fragment related to the previous output word fragment. The decoding vector is normalized by the softmax layer to obtain the influence probability (self-attention score) of each output word fragment on the previous output word fragment, find the word fragment vector with the largest probability value, and search for the corresponding output word fragment in the preset word table and output it. At the same time, the association relationship between the output word fragment and the previous output word fragment is also used as another output of the decoder. The association relationship refers to the self-attention score and relative position vector of the output word fragment and the previous output word fragment. Repeat this step until the last output word fragment and the association relationship between the output word fragment and the previous output word fragment are output, indicating that the recognition of the table structure to be recognized is completed.

[0059] Step A9: taking each output word segment of the decoder and the association between each output word segment and the corresponding previous output word segment as the output of the table image recognition model.

[0060] Each output word segment of the decoder and the association between each output word segment and the corresponding previous output word segment are used as the output of the table image recognition model.

[0061] Step S4, generating a table structure corresponding to the table image to be recognized according to the output word fragments and the association relationship between the output word fragments.

[0062] The output word fragments and the association relationship between the output word fragments can reflect the specific contextual semantic relationship, belonging relationship and spatial layout information of the output word fragments in the table. The relationship of the text content is determined by multiple dimensional information, which improves the recognition accuracy of the table structure. According to the association relationship, the actual table structure can be determined and restored to match the actual table form.

[0063] In an optional embodiment of the present invention, there is also provided a preferred solution for generating a table structure corresponding to the table image to be identified according to the output word fragments and the association relationship between the output word fragments in the above step S4. Specifically, step S4 may include the following steps S41 to S42:

[0064] Step S41, judge the association relationship between the output word fragments, and determine the positional relationship between the output word fragment and the previous output word fragment of the output word fragment based on the judgment result, the positional relationship includes that the output word fragment and the previous output word fragment of the output word fragment are in the same frame or in different frames, the unit frame to which the output word fragment belongs and the unit frame to which the previous output word fragment belongs are in the same line or in different lines, and the unit frame to which the previous output word fragment belongs is a merged unit frame or a standard unit frame.

[0065] Since the image text sequence is sorted according to the original layout of the table image to be recognized, and the output is the word fragment with the highest self-attention score and the smallest one-dimensional relative position vector modulus with the input word fragment, it can be concluded that the decoder outputs the word fragment according to the original layout of the table image to be recognized. According to the association relationship between the output word fragments, the position relationship between a certain output word fragment and the previous output word fragment of the output word fragment can be determined, where the position relationship is divided into the following three situations: the output word fragment and the previous output word fragment of the output word fragment are in the same frame relationship or different frame relationship, the unit frame to which the output word fragment belongs and the unit frame to which the previous output word fragment belongs are in the same line relationship or different line relationship, and the unit frame to which the previous output word fragment of the output word fragment belongs is a merged unit frame relationship or a standard unit frame relationship.

[0066] In an optional embodiment of the present invention, there is also provided a preferred solution for judging the association relationship between the output word fragments in the above step S41, and determining the position relationship between the output word fragment and the previous output word fragment of the output word fragment according to the judgment result. Specifically, step S41 may include the following steps B1 to B3:

[0067] Step B1, determining whether the output word segment and the output word segment before the output word segment are in the same frame or in a different frame relationship based on the self-attention scores of the output word segment and the output word segment before the output word segment.

[0068] The self-attention score reflects a certain contextual semantic relationship of the text content. Through the self-attention score of the output word fragment and the previous output word fragment of the output word fragment, it can be determined whether the output word fragment and the previous output word fragment of the output word fragment are in the same frame or in a different frame relationship.

[0069] In an optional embodiment of the present invention, there is also provided a preferred solution for determining in the above step B1 whether the output word segment and the output word segment before the output word segment are in the same frame relationship or in a different frame relationship according to the self-attention score of the output word segment and the output word segment before the output word segment. Specifically, step B1 may include the following steps B11 to B12:

[0070] Step B11, when the self-attention score of the output word segment and the output word segment before the output word segment is greater than or equal to a first preset threshold, it is determined that the output word segment and the output word segment before the output word segment are in the same frame relationship;

[0071] Step B12: When the self-attention score of the output word segment and the previous output word segment of the output word segment is less than a first preset threshold, it is determined that the output word segment and the previous output word segment of the output word segment are in a different-frame relationship.

[0072] This embodiment sets a first preset threshold as an indicator for measuring whether there is a contextual semantic relationship between word fragments, thereby determining whether the corresponding word fragments are in the same frame or in a different frame. When the self-attention score of the output word fragment and the previous output word fragment of the output word fragment is greater than or equal to the first preset threshold, it indicates that the two adjacent word fragments have a close contextual semantic relationship, indicating that the word fragment pair is adjacent text content of the same unit frame, which is a same frame relationship; when the self-attention score of the output word fragment and the previous output word fragment of the output word fragment is less than the first preset threshold, it indicates that the corresponding word fragment pair does not have a close contextual semantic relationship, indicating that the word fragment pair does not belong to the text content of the same unit frame, which is a different frame relationship.

[0073] Step B2, determining whether the unit frame to which the previous output word segment of the output word segment belongs is a merged unit frame relationship or a standard unit frame relationship according to the self-attention scores of the output word segment and the previous output word segment of the output word segment.

[0074] The self-attention score can also reflect the relationship between different text contents. Through the self-attention score of the output word fragment and the previous output word fragment of the output word fragment, it can be determined whether the previous output word fragment of the output word fragment is a merged unit frame relationship or a standard unit frame relationship.

[0075] In an optional embodiment of the present invention, there is also provided a preferred solution for determining in step B2 above whether the unit frame to which the previous output word segment of the output word segment belongs is a merged unit frame relationship or a standard unit frame relationship according to the self-attention score of the output word segment and the previous output word segment of the output word segment. Specifically, step B2 may include the following steps B21 to B22:

[0076] Step B21, when the self-attention score of the output word fragment and the previous output word fragment of the output word fragment is greater than or equal to the second preset threshold and less than the first preset threshold, it is determined that the unit frame to which the previous output word fragment of the output word fragment belongs is a merged unit frame relationship, wherein the first preset threshold is greater than the second preset threshold.

[0077] A second preset threshold is set as an indicator to measure whether there is a relationship between word segments. When the self-attention score of the output word segment and the previous output word segment of the output word segment is greater than or equal to the second preset threshold and less than the first preset threshold, wherein the first preset threshold is greater than the second preset threshold, it indicates that the output word segment and the previous output word segment of the output word segment are in a different frame relationship, and the two word segments have a relationship of belonging or a parent-child relationship, that is, the previous output word segment contains the output word segment, then it is determined that the cell to which the previous output word segment of the output word segment belongs is a merged cell relationship. For example, the previous output word segment is "gender" and the next output word segment is "male". There may be other word segments in the table image that have the same self-attention score as the previous output word segment (that is, the corresponding word segment is "female"), but because the one-dimensional position vector modulus of the "male" word segment and the previous output word segment in the image text sequence is the smallest, the "male" word segment is output. Since the previous output word fragment includes the next output word fragment, the cell frame to which the previous word fragment belongs is a merged cell frame relationship. Usually in a table image, if a cell frame has a parent-child relationship with multiple cell frames, the cell frame is set to a merged cell frame for distinction.

[0078] Step B22, when the self-attention score between the output word segment and the previous output word segment of the output word segment is less than a second preset threshold, it is determined that the unit frame to which the previous output word segment of the output word segment belongs is a standard unit frame relationship.

[0079] When the self-attention score of the output word segment and the previous word segment of the output word segment is less than the second preset threshold, it means that the output word segment has no contextual semantic relationship with the previous output word segment, does not belong to adjacent word segments of the same unit frame, but is a word segment between different unit frames, and the above two word segments have no belonging relationship, so it is determined that the unit frame to which the previous word segment of the output word segment belongs is a standard unit frame relationship, and the standard unit frame refers to the unmodified initial unit frame form. For example, the previous output word segment is "goodbye", the output word segment is "fruit", "goodbye" has no belonging relationship with "fruit", so the unit frame to which "goodbye" belongs is a standard unit frame relationship.

[0080] Step B3, determining whether the unit box to which the output word segment belongs and the unit box to which the previous output word segment belongs are in the same row or in a different row according to the self-attention scores and the relative position vector in the x direction of the output word segment and the previous output word segment of the output word segment.

[0081] Since each word fragment in the image text sequence is arranged according to the original layout information of the table image to be recognized, the output of the table image recognition model is also output in the order in the image text sequence. Therefore, if the output word fragment and the previous output word fragment of the output word fragment are in a different frame relationship, it means that the unit frames to which the output word fragment and the previous output word fragment belong are adjacent unit frames, but the adjacent unit frames include two types of relationships: the same row relationship and the different row relationship. The above two relationships can be determined by the relative position vector between the word fragments. The relative position vector includes a one-dimensional relative position vector between the word fragments, an x-direction relative position vector, and a y-direction relative position vector. Specifically, it needs to be determined according to the x-direction relative position vector of the output word fragment and the previous output word fragment of the output word fragment (the x-direction of the vector formed by the output word fragment as the starting point and the previous output word fragment as the end point). The x-direction is the horizontal axis direction of the two-dimensional coordinate system constructed in the table image to be recognized, and training is performed in the table image recognition model. Among them, adjacent cell frames in the same row refer to adjacent cell frames at any position in the same row, and adjacent cell frames in different rows refer to the cell frame to which the previous output word fragment belongs is the last cell frame in a row of the table, and the cell frame to which the next output word fragment belongs is the first cell frame in the next adjacent row.

[0082] In an optional embodiment of the present invention, there is also provided a preferred solution for determining in step B3 above whether the unit box to which the output word segment belongs and the unit box to which the previous output word segment belongs are in the same row or in a different row according to the self-attention score and the relative position vector in the x direction of the output word segment and the previous output word segment of the output word segment. Specifically, step B3 may include the following steps B31 to B32:

[0083] Step B31, when the self-attention scores of the output word segment and the previous output word segment of the output word segment are less than the second preset threshold and the relative position vector in the x direction is in the due west direction, it is determined that the unit box to which the output word segment belongs and the unit box to which the previous output word segment belongs are in a row relationship;

[0084] Step B32, when the self-attention score of the output word fragment and the previous output word fragment of the output word fragment is less than the second preset threshold and the relative position vector in the x direction is due east, it is determined that the unit box to which the output word fragment belongs and the unit box to which the previous output word fragment belongs are in a different row relationship.

[0085] When the self-attention score of an output word fragment and the output word fragment before the output word fragment is less than the second preset threshold and the relative position vector in the x direction is due west, it means that the above two word fragments have no contextual semantic relationship or belonging relationship, are in a different frame relationship, and the due west direction is a word fragment layout relationship of adjacent cell frames in the same row, and it is determined that the cell frame to which the output word fragment belongs and the cell frame to which the output word fragment before the output word fragment belongs are in the same row; when the self-attention score of the output word fragment and the output word fragment before the output word fragment is less than the second preset threshold and the relative position vector in the x direction is due east, and the due east direction is a word fragment layout relationship of cross-row cells, it is determined that the cell frame to which the output word fragment belongs and the cell frame to which the output word fragment before the output word fragment belongs are in different rows.

[0086] Step S42: Generate a table structure corresponding to the table image to be recognized according to the positional relationship between the output word segment and the previous output word segment of the output word segment.

[0087] After determining the positional relationship between the output word fragment and the previous output word fragment of the output word fragment, the output word fragment pair is combined with the table structure in the manner of constructing the HTML format table structure to restore the table structure. The restored table structure can be opened with a browser and used, wherein the browser may be Google Chrome, Sohu Browser, etc., without any limitation. Usually, the construction method of the HTML format table structure is: to indicate the start mark of each row of cells, to indicate the end mark of each row of cells, to indicate the start mark of each cell, to indicate the end mark of each cell, rowspan is a cross-row mark, colspan is a cross-column mark, and the output word fragment is encoded in the above manner to construct a complete table structure. For example, ABSTSRC is a row of cell box text content, wherein "ABST" is the text content of the first cell box in the row, and "SRC" is the text content of the second cell box in the row. For another example, Baidu represents two rows of merged cell boxes, containing the text content "Baidu".

[0088] The table image recognition method provided by the present invention, after obtaining the table image to be recognized, performs OCR recognition on the table image to be recognized, obtains the single-line text information of each cell frame in the table to be recognized, and performs convolutional neural network analysis and recognition on the table image to be recognized, obtains multiple feature maps of the table image to be recognized, and by collecting two dimensional information of single-line text information and feature map, the basic data of the table image is made more comprehensive and diverse; then, the basic data is calculated based on the pre-trained table image recognition model to obtain output word fragments and the association relationship between the output word fragments, wherein the association relationship between the output word fragments includes two dimensional information of self-attention score and relative position vector, and the self-attention score is used to obtain the output word fragments and the association relationship between the output word fragments. The attention score can reflect the contextual semantic relationship or belonging relationship between the output word fragments, while the relative position vector reflects the spatial layout relationship between the output word fragments. The combination of the two dimensional information can more comprehensively reflect the relevance of the text content in the table to be identified. Accordingly, the table image to be identified is constructed according to the output word fragments and the association relationship between the output word fragments, which can better restore the text content of each cell frame of the table to be identified. The recognition accuracy is bound to be higher than that of inferring the table structure only based on the spatial layout information of the text content. Based on the present application, the technical problem of low table structure recognition accuracy caused by performing table recognition based on the spatial arrangement information of the text content in the prior art is solved.

[0089] Embodiment 2

[0090] Embodiment 2 of the present invention further provides a table image recognition device, which corresponds to the table image recognition method provided in the above embodiment 1. The corresponding technical features and technical effects are not described in detail in this embodiment. For relevant parts, please refer to the above embodiment 1. Specifically, Figure 5 FIG. 2 shows a structural block diagram of a table image recognition device provided by Embodiment 2 of the present invention. Figure 5 As shown, the table image recognition device 500 includes a first acquisition module 501, a second acquisition module 502, a recognition module 503 and a generation module 504, wherein:

[0091] The first acquisition module 501 is used to acquire a table image to be recognized, perform OCR recognition on the table image to be recognized, and obtain single-line text information of each cell frame in the table to be recognized;

[0092] The second acquisition module 502 is connected to the first acquisition module 501 and is used to perform convolutional neural network analysis and recognition on the table image to be recognized, and obtain multiple feature maps of the table image to be recognized;

[0093] The recognition module 503 is connected to the second acquisition module 502 and is used to obtain a pre-trained table image recognition model, input the single-line text information and the feature map into the table image recognition model, and obtain the calculation result output by the table image recognition model, wherein the calculation result includes the output word fragments and the association relationship between the output word fragments, and the association relationship includes the self-attention score and the relative position vector, wherein the output word fragment is obtained after the table image recognition model performs word segmentation on the single-line text information;

[0094] The generation module 504 is connected to the recognition module 503 and is used to generate a table structure corresponding to the table image to be recognized according to the output word fragments and the association relationship between the output word fragments.

[0095] Optionally, the generation module includes: a judgment submodule, used to judge the association relationship between output word fragments, and determine the positional relationship between the output word fragment and the output word fragment previous to the output word fragment according to the judgment result, the positional relationship including that the output word fragment and the output word fragment previous to the output word fragment are in the same frame or in different frames, the unit frame to which the output word fragment belongs and the unit frame to which the output word fragment previous to the output word fragment belongs are in the same row or in different rows, and the unit frame to which the output word fragment previous to the output word fragment belongs are in a merged unit frame relationship or a standard unit frame relationship; a generation submodule, used to generate a table structure corresponding to the table image to be identified according to the positional relationship between the output word fragment and the output word fragment previous to the output word fragment.

[0096] Optionally, the judgment submodule includes: a first determination unit, used to determine whether the output word fragment and the output word fragment before the output word fragment are in the same frame relationship or different frame relationship according to the self-attention score of the output word fragment and the output word fragment before the output word fragment; a second determination unit, used to determine whether the unit frame to which the output word fragment before the output word fragment belongs is a merged unit frame relationship or a standard unit frame relationship according to the self-attention score of the output word fragment and the output word fragment before the output word fragment; a third determination unit, used to determine whether the unit frame to which the output word fragment belongs and the unit frame to which the output word fragment before the output word fragment belongs are in the same row relationship or in a different row relationship according to the self-attention score and the relative position vector in the x-direction of the output word fragment and the output word fragment before the output word fragment.

[0097] Optionally, when the first determination unit executes the step of determining whether an output word fragment and the output word fragment preceding the output word fragment are in the same frame or in a different frame relationship based on the self-attention score of the output word fragment and the output word fragment preceding the output word fragment, it is specifically used to: when the self-attention score of the output word fragment and the output word fragment preceding the output word fragment is greater than or equal to a first preset threshold, it is determined that the output word fragment and the output word fragment preceding the output word fragment are in the same frame relationship; when the self-attention score of the output word fragment and the output word fragment preceding the output word fragment is less than the first preset threshold, it is determined that the output word fragment and the output word fragment preceding the output word fragment are in a different frame relationship.

[0098] Optionally, when the second determination unit executes the step of determining whether the unit frame to which the previous output word fragment of the output word fragment belongs is a merged unit frame relationship or a standard unit frame relationship based on the self-attention score of the output word fragment and the previous output word fragment of the output word fragment, it is specifically used to: when the self-attention score of the output word fragment and the previous output word fragment of the output word fragment is greater than or equal to a second preset threshold and less than a first preset threshold, determine that the unit frame to which the previous output word fragment of the output word fragment belongs is a merged unit frame relationship, wherein the first preset threshold is greater than the second preset threshold; when the self-attention score of the output word fragment and the previous output word fragment of the output word fragment is less than the second preset threshold, determine that the unit frame to which the previous output word fragment of the output word fragment belongs is a standard unit frame relationship.

[0099] Optionally, when the third determination unit executes the step of determining whether the cell box to which the output word fragment belongs and the cell box to which the previous output word fragment belongs are in the same row or in a different row according to the self-attention scores of the output word fragment and the previous output word fragment of the output word fragment and the relative position vector in the x-direction, it is specifically used to: when the self-attention scores of the output word fragment and the previous output word fragment of the output word fragment are less than a second preset threshold and the relative position vector in the x-direction is due west, then it is determined that the cell box to which the output word fragment belongs and the cell box to which the previous output word fragment belongs are in the same row; when the self-attention scores of the output word fragment and the previous output word fragment of the output word fragment are less than the second preset threshold and the relative position vector in the x-direction is due east, then it is determined that the cell box to which the output word fragment belongs and the cell box to which the previous output word fragment belongs are in a different row.

[0100] Optionally, the table image recognition model includes an encoder and a decoder, and the table image recognition model outputs the calculation result through the following method, including: in the encoder: performing word segmentation operation on a single line of text information according to a preset BERT algorithm to generate multiple output word fragments; sorting the output word fragments and feature maps according to the original unit frame layout of the table image to be recognized, respectively, to obtain an initial text sequence and an initial feature map sequence; inputting the initial text sequence and the initial feature map sequence into the embedding layer, respectively generating an initial text sequence and an initial feature map sequence represented by vectors; splicing the initial text sequence and the initial feature map sequence represented by vectors to generate an image text sequence; inputting the image text sequence into the conversion layer, performing self-attention and relative position calculation on each output word fragment in the image text sequence, and obtaining the association relationship between the output word fragments in the image text sequence, the association relationship including the self-attention score and the one-dimensional relative position vector between the output word fragments; outputting the association relationship between the image text sequence and the output word fragments to the decoder; in the decoder: receiving the association relationship between the image text sequence and the output word fragments, and obtaining a preset start mark, and embedding the preset start mark At a one-dimensional position in an image text sequence, calculate the association relationship between a preset start mark and each output word fragment after the one-dimensional position of the preset start mark in the image text sequence, and output the output word fragment with the highest self-attention score with the preset start mark and / or the smallest modulus of the one-dimensional relative position vector; input the association relationship between the image text sequence and the output word fragment into the decoder again, and sequentially splice the preset start mark and the previous output word fragment to form a first text sequence for input, and embed the one-dimensional position of the previous output word fragment in the image text sequence, calculate the association relationship between the previous output word fragment in the first text sequence and each output word fragment after the one-dimensional position of the previous output word fragment in the image text sequence, output the output word fragment with the highest self-attention score with the previous output word fragment and / or the smallest modulus of the one-dimensional relative position vector and the association relationship between the output word fragment and the previous output word fragment, repeat this step until the last output word fragment and the association relationship between the output word fragment and the previous output word fragment are output; use the output word fragments of the decoder and the association relationship between each output word fragment and the corresponding previous output word fragment as the output of the table image recognition model.

[0101] Embodiment 3

[0102] Figure 6 The block diagram of a computer device suitable for implementing the table image recognition method provided by the third embodiment of the present invention is shown. In this embodiment, the computer device 600 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a rack server, a blade server, a tower server or a cabinet server (including an independent server, or a server cluster composed of multiple servers) that executes a program. Figure 6 As shown, the computer device 600 of this embodiment includes at least but is not limited to: a memory 601, a processor 602, and a network interface 603 that can be interconnected through a system bus. It should be noted that Figure 6 Only computer device 600 is shown with components 601 - 603 , but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0103] In this embodiment, the memory 603 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 601 can be an internal storage unit of the computer device 600, such as a hard disk or memory of the computer device 600. In other embodiments, the memory 601 can also be an external storage device of the computer device 600, such as a plug-in hard disk equipped on the computer device 600, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 601 can also include both the internal storage unit of the computer device 600 and its external storage device. In this embodiment, the memory 601 is generally used to store the operating system and various application software installed on the computer device 600, such as the program code of the table image recognition method, etc.

[0104] In some embodiments, the processor 602 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 602 is generally used to control the overall operation of the computer device 600. For example, the processor 602 performs control and processing related to data interaction or communication with the computer device 600. In this embodiment, the processor 602 is used to run the program code of the steps of the table image recognition method stored in the memory 601.

[0105] In this embodiment, the table image recognition method stored in the memory 601 can also be divided into one or more program modules and executed by one or more processors (processor 602 in this embodiment) to complete the present invention.

[0106] The network interface 603 may include a wireless network interface or a wired network interface, and the network interface 603 is generally used to establish a communication link between the computer device 600 and other computer devices. For example, the network interface 603 is used to connect the computer device 600 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 600 and the external terminal. The network can be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0107] Embodiment 4

[0108] This embodiment also provides a computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc., on which a computer program is stored, and when the computer program is executed by a processor, the steps of the table image recognition method are implemented.

[0109] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0110] It should be noted that the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0111] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0112] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.< / padding> < / sep> < / sep> < / cls>

Claims

1. A table image recognition method, characterized in that: The method comprises: Obtain an image of a table to be identified, perform OCR on the image of the table to be identified, and obtain single-line text information of each cell frame in the table to be identified; Performing convolutional neural network analysis and recognition on the to-be-recognized table image to obtain multiple feature maps of the to-be-recognized table image; Obtain a pre-trained table image recognition model, input the single-line text information and the feature map into the table image recognition model, and obtain a calculation result output by the table image recognition model, wherein the calculation result includes output word fragments and associations between output word fragments, and the associations include self-attention scores and relative position vectors, wherein the output word fragments are obtained after the table image recognition model performs word segmentation on the single-line text information; According to the output word segments and the association relationship between the output word segments, a table structure corresponding to the table image to be identified is generated.

2. The method according to claim 1, characterized in that The step of generating a table structure corresponding to the table image to be identified according to the output word segments and the association relationship between the output word segments includes: The association relationship between the output word segments is judged, and the position relationship between the output word segment and the output word segment before the output word segment is determined according to the judgment result, wherein the position relationship includes that the output word segment and the output word segment before the output word segment are in the same frame relationship or in different frames relationship, the unit frame to which the output word segment belongs and the unit frame to which the output word segment before the output word segment belongs are in the same line relationship or in different lines relationship, and the unit frame to which the output word segment before the output word segment belongs are in a merged unit frame relationship or a standard unit frame relationship; A table structure corresponding to the table image to be recognized is generated according to the positional relationship between the output word segment and the previous output word segment of the output word segment.

3. The method according to claim 2, characterized in that The relative position vector includes an x-direction relative position vector between word segments, wherein the x-direction is the horizontal axis direction of a two-dimensional coordinate system constructed with the table image to be identified, and the determining the positional relationship between the output word segment and the previous output word segment of the output word segment according to the judgment result includes: Determine, according to the self-attention scores of the output word segment and the output word segment before the output word segment, whether the output word segment and the output word segment before the output word segment are in the same frame relationship or in a different frame relationship; Determine, according to the self-attention scores of the output word segment and the previous output word segment of the output word segment, whether the cell frame to which the previous output word segment of the output word segment belongs is a merged cell frame relationship or a standard cell frame relationship; According to the self-attention scores of the output word fragment and the previous output word fragment of the output word fragment and the relative position vector in the x direction, it is determined whether the unit box to which the output word fragment belongs and the unit box to which the previous output word fragment of the output word fragment belongs are in the same row or in a different row.

4. The method according to claim 3, characterized in that The step of determining whether the output word segment and the output word segment before the output word segment are in the same frame or in a different frame relationship according to the self-attention scores of the output word segment and the output word segment before the output word segment comprises: When the self-attention score of the output word segment and the output word segment before the output word segment is greater than or equal to a first preset threshold, it is determined that the output word segment and the output word segment before the output word segment are in the same frame relationship; When the self-attention score of an output word segment and the output word segment before the output word segment is less than a first preset threshold, it is determined that the output word segment and the output word segment before the output word segment are in a different-frame relationship.

5. The method according to claim 4, characterized in that The step of determining, based on the self-attention scores of the output word segment and the previous output word segment of the output word segment, that the unit frame to which the previous output word segment belongs is a merged unit frame relationship or a standard unit frame relationship comprises: When the self-attention score of the output word segment and the previous output word segment of the output word segment is greater than or equal to the second preset threshold and less than the first preset threshold, it is determined that the unit frame to which the previous output word segment of the output word segment belongs is a merged unit frame relationship, wherein the first preset threshold is greater than the second preset threshold; When the self-attention score between an output word segment and a previous output word segment of the output word segment is less than a second preset threshold, it is determined that the unit frame to which the previous output word segment of the output word segment belongs is a standard unit frame relationship.

6. The method according to claim 5, characterized in that The step of determining whether a cell box to which the output word segment belongs and a cell box to which the previous output word segment belongs are in the same row or in a different row according to the self-attention scores of the output word segment and the previous output word segment of the output word segment and the relative position vector in the x direction comprises: When the self-attention score of the output word segment and the previous output word segment of the output word segment is less than the second preset threshold and the relative position vector in the x direction is in the due west direction, it is determined that the unit box to which the output word segment belongs and the unit box to which the previous output word segment belongs are in the same row relationship; When the self-attention scores of an output word segment and the previous output word segment of the output word segment are less than a second preset threshold and the relative position vector in the x direction is due east, it is determined that the unit box to which the output word segment belongs and the unit box to which the previous output word segment belongs are in a different row relationship.

7. The method according to claim 1, characterized in that The table image recognition model includes an encoder and a decoder, and the table image recognition model outputs the calculation result by the following method, including: In the encoder: Perform word segmentation on the single line of text information according to a preset BERT algorithm to generate multiple output word fragments; The output word fragments and feature maps are sorted according to the original cell frame layout of the table image to be recognized, and an initial text sequence and an initial feature map sequence are obtained; Inputting the initial text sequence and the initial feature map sequence into an embedding layer to generate an initial text sequence and an initial feature map sequence represented by vectors respectively; splicing the initial text sequence represented by the vector and the initial feature map sequence to generate an image text sequence; Input the image text sequence into the conversion layer, perform self-attention and relative position calculation on each output word segment in the image text sequence, and obtain the association relationship between the output word segments in the image text sequence, wherein the association relationship includes the self-attention score and the one-dimensional relative position vector between the output word segments; Outputting the association relationship between the image text sequence and the output word fragment to the decoder; In the decoder: Receive the association relationship between the image text sequence and the output word fragment, and obtain a preset starting mark, embed the one-dimensional position of the preset starting mark in the image text sequence, calculate the association relationship between the preset starting mark and each output word fragment after the one-dimensional position of the preset starting mark in the image text sequence, and output the output word fragment with the highest self-attention score and / or the smallest modulus of the one-dimensional relative position vector with the preset starting mark; The association relationship between the image text sequence and the output word fragment is input to the decoder again, and the preset start mark and the previous output word fragment are sequentially spliced ​​to form a first text sequence for input, and the one-dimensional position of the previous output word fragment in the image text sequence is embedded, and the association relationship between the previous output word fragment in the first text sequence and each output word fragment after the one-dimensional position of the previous output word fragment in the image text sequence is calculated, and the output word fragment with the highest self-attention score and / or the smallest one-dimensional relative position vector modulus with the previous output word fragment and the association relationship between the output word fragment and the previous output word fragment are output, and this step is repeated until the last output word fragment and the association relationship between the output word fragment and the previous output word fragment are output; Each output word segment of the decoder and the association between each output word segment and the corresponding previous output word segment are used as the output of the table image recognition model.

8. A table image recognition device, characterized in that: The device comprises: The first acquisition module is used to acquire a table image to be identified, perform OCR recognition on the table image to be identified, and obtain single-line text information of each cell frame in the table to be identified; A second acquisition module is used to perform convolutional neural network analysis and recognition on the table image to be identified, and obtain multiple feature maps of the table image to be identified; A recognition module, used to obtain a pre-trained table image recognition model, input the single-line text information and the feature map into the table image recognition model, and obtain a calculation result output by the table image recognition model, wherein the calculation result includes output word fragments and associations between output word fragments, and the associations include self-attention scores and relative position vectors, wherein the output word fragments are obtained after the table image recognition model performs word segmentation on the single-line text information; A generation module is used to generate a table structure corresponding to the table image to be identified based on the output word fragments and the association relationship between the output word fragments.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images