Cell relationship prediction method, prediction model training method and table parsing method

Abstract

The present application discloses a cell relationship prediction method, a prediction model training method and a table parsing method. The prediction method includes obtaining a table structure diagram of a document to be predicted; inputting the table structure diagram of the document to be predicted into a cell relationship prediction model, and outputting the cell relationship between word tokens in the table structure diagram of the document to be predicted through the cell relationship prediction model. The cell relationship between word tokens includes a same cell relationship matrix, a next cell relationship matrix in the same row, and a next cell relationship matrix in the same column. The embodiment of the present application divides the cell relationship into a same cell relationship, a next cell relationship in the same row, and a next cell relationship in the same column, and then performs relationship prediction on the table structure diagram of the document to be predicted through a trained cell relationship prediction model, and outputs the accurate cell relationship between each word token, so as to better parse the table structure, especially for complex tables, and parse to an accurate table structure.

Description

Technical Field

[0001] The present application relates to the technical field of document understanding, and in particular to a cell relationship prediction method, a prediction model training method and a table parsing method. Background Art

[0002] In recent years, with the increasing degree of digitalization of information, the amount of data in the form of document images has shown a trend of massive growth. Extracting information from document images to form structured data can be used to better build indexes, facilitate search, and can be used in scientific research, engineering, statistics, strategy formulation, market research, etc. to provide quantitative data.

[0003] As the most concise way to summarize data records, tables are the basis of data analysis. Automatically identifying these document image table data and restoring the table content in the image into structured data can undoubtedly improve data collection efficiency. The commonly used table structure parsing methods currently mainly adopt the parsing method of the target detection model and the parsing method of the sequence generation model. Among them, the target detection model first detects the position of rows, columns and merged cells in the document, and then obtains the table structure through post-processing; the sequence generation model uses the decoder to directly decode the table structure represented by Html or Markdown. Although these two methods can parse the table structure, the parsing accuracy of the two methods for complex tables containing merged cells is low. This is because complex tables contain merged cells, and merged cells can have row and column relationships with multiple rows or columns. This will affect the accuracy of the target detection model and the sequence generation model in recognizing the row and column relationship between cells, and then affect the accuracy of the table structure parsing.

[0004] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0005] The technical problem to be solved by the present application is to provide a cell relationship prediction method, a prediction model training method and a table parsing method in view of the deficiencies of the prior art.

[0006] In order to solve the above technical problems, the first aspect of the present application provides a cell relationship prediction method, wherein the cell relationship prediction method specifically includes:

[0007] Get the table structure diagram of the document to be predicted;

[0008] The table structure diagram of the document to be predicted is input into a cell relationship prediction model, and the cell relationship between word tokens in the table structure diagram of the document to be predicted is output through the cell relationship prediction model, wherein the cell relationship between the word tokens includes a same cell relationship matrix, a next cell relationship matrix in the same row, and a next cell relationship matrix in the same column.

[0009] In a specific embodiment, the same-cell relationship matrix includes the same-cell relationships between two word tokens in the table structure diagram of the document to be predicted, the next-cell-in-the-same-row relationship matrix includes the next-cell-in-the-same-row relationship between two word tokens in the table structure diagram of the document to be predicted, and the next-cell-in-the-same-column relationship matrix includes the next-cell-in-the-same-column relationship between two word tokens in the table structure diagram of the document to be predicted.

[0010] In a specific embodiment, the step of inputting the table structure diagram of the document to be predicted into a cell relationship prediction model, and outputting the cell relationship between word tokens in the table structure diagram of the document to be predicted through the cell relationship prediction model specifically includes:

[0011] Based on the table structure diagram of the document to be predicted, determining the representation vector of each word Token through the encoding module in the cell relationship prediction model;

[0012] The representation vectors of each word Token are combined in pairs and input into the relationship prediction module in the cell relationship prediction model, and the cell relationship between the word Tokens in the table structure diagram of the document to be predicted is output through the relationship prediction module.

[0013] In a specific embodiment, the step of determining the representation vector of each word token based on the table structure diagram of the document to be predicted through the encoding module in the cell relationship prediction model specifically includes:

[0014] Obtaining an image vector of the table structure diagram of the document to be predicted;

[0015] Identify each word token and the word position vector of each word token in the table structure diagram of the document to be predicted, and determine the text vector based on each word token and the word position vector of each word token;

[0016] Based on the image vector and the text vector, the representation vector of each word Token is determined through the encoding module in the cell relationship prediction model.

[0017] In a specific embodiment, obtaining the image vector of the table structure diagram of the document to be predicted specifically includes:

[0018] Divide the table structure diagram of the document to be predicted into a plurality of image blocks, and obtain a position vector of each image block in the table structure diagram of the document to be predicted;

[0019] Input each image block into a linear layer, and output a graph embedding of each image block through the linear layer;

[0020] The position vector of each image block is input into the image position embedding layer, and the image position embedding of each image block is output through the image position embedding layer;

[0021] Merge the image embedding and image position embedding of each image block to obtain the pixel features of each image block;

[0022] The vector formed by the pixel features of each image block is used as the image vector of the document table structure diagram to be predicted.

[0023] In a specific embodiment, determining the text vector based on each word token and the word position vector of each word token specifically includes:

[0024] Get the word embedding of each word token and the word position embedding of the word position vector of each word token, and merge the word embedding and word position embedding of each word token to obtain the text features of each word token;

[0025] The vector composed of the text features of each word token is used as the text vector.

[0026] In a specific embodiment, the step of combining the representation vectors of each word token in pairs and inputting them into a relationship prediction module in a cell relationship prediction model, and outputting the cell relationship between the word tokens in the table structure diagram of the document to be predicted by the relationship prediction module specifically includes:

[0027] Merge the representation vectors of each word token in pairs to obtain several combined representation vectors;

[0028] Inputting a plurality of combined representation vectors into a first relation classification unit, a second relation classification unit and a third relation classification unit in the relation prediction module respectively;

[0029] The first relationship classification unit outputs the same cell relationship matrix, the second relationship classification unit outputs the next cell relationship matrix in the same row, and the third relationship classification unit outputs the next cell relationship matrix in the same column to obtain the cell relationship between the word tokens in the table structure diagram of the document to be predicted.

[0030] In a second aspect of the present application, a device for constructing a cell relationship prediction model is provided, wherein the device is used to train the cell relationship prediction model used in the cell relationship prediction method as described above, wherein the cell relationship prediction model training method specifically includes:

[0031] Obtaining a training document table structure diagram and the labeled cell relationships between word tokens therein, wherein the labeled cell relationships between word tokens include a labeled same cell relationship matrix, a labeled next cell relationship matrix in the same row, and a labeled next cell relationship matrix in the same column;

[0032] Input the training document table structure diagram into an initial cell relationship prediction model, and output the predicted cell relationship between word tokens in the training document table structure diagram through the initial cell relationship prediction model, wherein the predicted cell relationship includes a predicted same cell relationship matrix, a predicted next cell relationship matrix in the same row, and a predicted next cell relationship matrix in the same column;

[0033] The cell relationship prediction model is trained based on the labeled cell relationship between the word tokens and the predicted cell relationship between the word tokens to obtain a cell relationship prediction model.

[0034] In a specific embodiment, the labeling process of labeling the cell relationship between the word tokens specifically includes:

[0035] Read each word token and the row and column information of each word token in the training document table structure diagram;

[0036] The cell relationships between word tokens are annotated according to the row and column information of each word token to obtain the annotated cell relationships between word tokens corresponding to the training document table structure diagram. When two word tokens are located in the same cell, the two word tokens are annotated with a same-cell relationship; when two word tokens do not belong to the same-cell relationship, one word token is located after the other word token in the row direction, and there is no non-empty cell between the two word tokens in the row direction, the two word tokens are annotated with a next-cell relationship in the same row; when two word tokens do not belong to the same-cell relationship, one word token is located after the other word token in the column direction, and there is no non-empty cell between the two word tokens in the column direction, the two word tokens are annotated with a next-cell relationship in the same column.

[0037] In a specific embodiment, the training of the cell relationship prediction model based on the labeled cell relationship between the word tokens and the predicted cell relationship between the word tokens to obtain the cell relationship prediction model specifically includes:

[0038] Calculate a first loss function term based on the labeled same-cell relationship matrix and the predicted same-cell relationship matrix;

[0039] Calculate the second loss function term based on the labeled next cell relationship matrix and the predicted next cell relationship matrix;

[0040] Calculate the third loss function term based on the labeled next cell relationship matrix in the same column and the predicted next cell relationship matrix in the same column;

[0041] A total loss function term is calculated based on the first loss function term, the second loss function term and the third loss function term, and the initial cell relationship prediction model is trained based on the total loss function term to obtain a cell relationship prediction model.

[0042] The third aspect of the present application provides a table parsing method, using the cell relationship prediction method as described above, the table parsing method specifically includes:

[0043] Determine the target cell relationship between word tokens in the table structure diagram of the document to be parsed by the cell relationship prediction method;

[0044] The table structure is decoded based on the target cell relationship between the word tokens to obtain the document table structure diagram of the document to be parsed.

[0045] In a specific embodiment, the table structure decoding based on the target cell relationship between the word tokens to obtain the document table structure diagram to be parsed specifically includes:

[0046] According to the target same-cell relationship matrix in the target cell relationship between word tokens, the word tokens with the same-cell relationship are clustered into the same cell to obtain a number of clustered cells;

[0047] Determine, according to the target in-line next cell relationship matrix in the target cell relationship between the word tokens, the cluster cells having the in-line next cell relationship among the plurality of cluster cells, so as to restore the in-line next cell relationship between the cluster cells;

[0048] According to the target next cell relationship matrix in the target cell relationship between word tokens, the cluster cells having the next cell relationship in the same column among the plurality of cluster cells are determined to restore the next cell relationship in the same column between the cluster cells and obtain the document table structure of the document table structure diagram to be parsed.

[0049] In a specific embodiment, clustering the word tokens having same-cell relationships into the same cell according to the target same-cell relationship matrix in the target cell relationships between the word tokens specifically includes:

[0050] Treat each word token as a cell community;

[0051] Traverse each word token, determine the number of first word tokens in each cell community that have a same-cell relationship with the word token according to the target same-cell relationship matrix, determine the target cell community corresponding to the word token according to the number of second word tokens in each cell community and the number of first word tokens corresponding to them, and move the word token from its original cell community to the target cell community;

[0052] Re-execute the step of traversing each word token until the word tokens in all cell communities are not moved, so as to obtain several clustered cells.

[0053] In a specific embodiment, determining the cluster cells having the same-row next cell relationship among the plurality of cluster cells according to the target same-row next cell relationship matrix in the target cell relationship between the word tokens to restore the same-row next cell relationship between the cluster cells specifically includes:

[0054] Traverse each cluster cell in the plurality of cluster cells, and calculate the number of third word tokens that have a next cell relationship in the same row between the word token in the cluster cell and the word token in each other cluster cell according to the target next cell relationship matrix; determine the next cell relationship in the same row between each cluster cell and other cluster cells based on the fourth token number of the cluster cell, the third word token number, and the fifth word token number of other cluster cells corresponding to the word token number;

[0055] According to the in-line next cell relationships between each cluster cell and other cluster cells, cluster cells having in-line next cell relationships among the plurality of cluster cells are determined to restore the in-line next cell relationships between the cluster cells.

[0056] In a specific embodiment, determining the cluster cells having the same column next cell relationship among the plurality of cluster cells according to the target same column next cell relationship matrix in the target cell relationship between the word tokens to restore the same column next cell relationship between the cluster cells specifically includes:

[0057] Traverse each clustering cell in the plurality of clustering cells, and calculate the number of sixth word tokens having a same-column next-cell relationship between the word token in the clustering cell and the word token in each other clustering cell according to the target same-column next-cell relationship matrix; determine the same-column next-cell relationship between each clustering cell and other clustering cells based on the number of seventh word tokens in the clustering cell, the number of sixth word tokens, and the number of eighth word tokens in other clustering cells corresponding to the number of word tokens;

[0058] According to the next cell in the same column relationship between each clustering cell and other clustering cells, clustering cells having the next cell in the same column relationship among the plurality of clustering cells are determined to restore the next cell in the same column relationship between the clustering cells.

[0059] A fourth aspect of the present application provides a cell relationship prediction device, wherein the cell relationship prediction device specifically includes:

[0060] The first acquisition module is used to obtain the table structure diagram of the document to be predicted;

[0061] The first control module is used to input the table structure diagram of the document to be predicted into a cell relationship prediction model, and output the cell relationship between word tokens in the table structure diagram of the document to be predicted through the cell relationship prediction model, wherein the cell relationship includes a same cell relationship matrix, a next cell relationship matrix in the same row, and a next cell relationship matrix in the same column.

[0062] In a fifth aspect, the present application provides a cell relationship prediction model training device, which is used to train the cell relationship prediction model used in the cell relationship prediction method described above, and the cell relationship prediction model training device specifically includes:

[0063] The second acquisition module is used to obtain the training document table structure diagram and the annotated cell relationship between the word tokens therein, wherein the annotated cell relationship between the word tokens includes an annotated same cell relationship matrix, an annotated next cell relationship matrix in the same row, and an annotated next cell relationship matrix in the same column;

[0064] A second control module is used to input the training document table structure diagram into an initial cell relationship prediction model, and output the predicted cell relationship between word tokens in the training document table structure diagram through the initial cell relationship prediction model, wherein the predicted cell relationship includes a predicted same cell relationship matrix, a predicted next cell relationship matrix in the same row, and a predicted next cell relationship matrix in the same column;

[0065] The training module is used to train the cell relationship prediction model based on the labeled cell relationship between the word tokens and the predicted cell relationship between the word tokens to obtain the cell relationship prediction model.

[0066] A sixth aspect of the present application provides a table parsing device, which uses the above-mentioned cell relationship prediction method and specifically includes:

[0067] A determination module, used to determine the target cell relationship between word tokens in the table structure diagram of the document to be parsed by using the cell relationship prediction method;

[0068] The decoding module is used to decode the table structure based on the target cell relationship between the word tokens to obtain the document table structure of the document table structure diagram to be parsed.

[0069] The seventh aspect of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the cell relationship prediction method as described above, and / or to implement the steps in the cell relationship prediction method as described above, and / or to implement the steps in the table parsing method as described above.

[0070] An eighth aspect of the present application provides a terminal device, comprising: a processor and a memory;

[0071] The memory stores a computer-readable program executable by the processor;

[0072] When the processor executes the computer-readable program, it implements the steps in the cell relationship prediction method described above, and / or implements the steps in the cell relationship prediction method described above, and / or implements the steps in the table parsing method described above.

[0073] Beneficial effects:

[0074] (1) The cell relationship prediction model of the present application predicts the same cell relationship matrix, the next cell relationship matrix in the same row, and the next cell relationship matrix in the same column to obtain the cell relationship between word tokens in the table structure diagram of the document to be predicted. The cell relationship between two word tokens can be well learned, so that the table structure can be better parsed, especially for complex tables, and the accurate table structure can be parsed.

[0075] (2) This application determines the representation vector of each word token by combining the image vector and the document vector, so that the cell relationship prediction model can use the image information of the table structure diagram of the document to be predicted and the document information of the table, thereby learning multimodal information and improving the accuracy of the cell relationship predicted by the cell relationship prediction model.

[0076] (3) This application uses the cell relationship between word tokens as the output item of the cell relationship prediction model, and restores the position of the same cell and the relationship between cells through the cell relationship between word tokens, so that the table structure can be parsed in a single step, which simplifies the table structure parsing process and improves the parsing efficiency of the table structure parsing. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0078] Figure 1 A flowchart of a cell relationship prediction method provided in an embodiment of the present application.

[0079] Figure 2 Schematic diagram of the process of cell relationship prediction method.

[0080] Figure 3 Schematic diagram of the process of cell relationship prediction model.

[0081] Figure 4 A flowchart of a cell relationship prediction model training method provided in an embodiment of the present application.

[0082] Figure 5 This is an example diagram of the table structure diagram of the document to be predicted provided in an embodiment of the present application.

[0083] Figure 6 for Figure 5 The provided table structure diagram of the document to be predicted is a schematic diagram after the word Token is annotated.

[0084] Figure 7 This is a schematic diagram after the relationship between cells is marked.

[0085] Figure 8 An example diagram for labeling the next cell relationship of the same row.

[0086] Fig. 9 Another example diagram for labeling peer-next-cell relationships.

[0087] Fig.10 An example diagram for labeling the relationship between the next cell in the same column.

[0088] Fig.11 Another example diagram for labeling the relationship between the next cell in the same column.

[0089] Fig.12 A flowchart of a table parsing method provided in an embodiment of the present application.

[0090] Fig.13 This is a structural schematic diagram of the cell relationship prediction model provided in an embodiment of the present application.

[0091] Fig.14 This is a structural schematic diagram of the cell relationship prediction model training device provided in an embodiment of the present application.

[0092] Fig.15 A schematic diagram of the structure of a table parsing device provided in an embodiment of the present application.

[0093] Fig.16 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0094] The embodiments of the present application provide a cell relationship prediction method, a prediction model training method, and a table parsing method. In order to make the purpose, technical solution, and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0095] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0096] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0097] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not mean the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0098] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0099] This embodiment provides a cell relationship prediction method, such as Figure 1 As shown, the method includes:

[0100] S10, obtaining a table structure diagram of the document to be predicted.

[0101] Specifically, the table structure diagram of the document to be predicted is a table image containing a table, wherein the table structure diagram of the document to be predicted may include one or more tables. For ease of understanding and explanation, the embodiment of the present application is explained as an example in which the table structure diagram of the document to be predicted includes one table.

[0102] S20, inputting the table structure diagram of the document to be predicted into a cell relationship prediction model, and outputting the cell relationship between word tokens in the table structure diagram of the document to be predicted through the cell relationship prediction model.

[0103] Specifically, the cell relationships between word tokens include same-cell relationship, next-cell-in-the-row relationship, and next-cell-in-the-column relationship. The same-cell relationship is used to indicate that the word tokens are in the same cell, the next-cell-in-the-row relationship indicates that the cells where the word tokens are located are adjacent in the row direction, and the next-cell-in-the-column relationship indicates that the cells where the word tokens are located are adjacent in the column direction.

[0104] Further, the cell relationships between word tokens in the table structure diagram of the document to be predicted may include a same-cell relationship matrix, a next-cell relationship matrix in the same row, and a next-cell relationship matrix in the same column. The same-cell relationship matrix includes the same-cell relationships between two word tokens in the table structure diagram of the document to be predicted. The next-cell relationship matrix in the same row includes the next-cell relationships in the same row between two word tokens in the table structure diagram of the document to be predicted. The next-cell relationship matrix in the same column includes the next-cell relationships in the same column between two word tokens in the table structure diagram of the document to be predicted. Among them, in the same-cell relationship matrix, the next-cell relationship matrix in the same row, and the next-cell relationship matrix in the same column, each matrix can use 1 to indicate that the two word tokens have the relationship corresponding to the matrix, and use 0 to indicate that the two word tokens do not have the relationship corresponding to the matrix. Of course, in practical applications, other methods can also be used to express it, and no specific restrictions are made here.

[0105] It should be noted that in actual applications, in addition to the same-cell relationship, the next-cell relationship in the same row, and the next-cell relationship in the same column, the cell relationship between word tokens can be expressed in other ways, for example, the same-cell relationship, the previous-cell relationship in the same row, and the previous-cell relationship in the same column, or the same-cell relationship, the next-cell relationship in the same row, and the previous-cell relationship in the same column, or the same-cell relationship, the previous-cell relationship in the same row, and the next-cell relationship in the same column, etc.

[0106] The cell relationship prediction model is a trained neural network model, which is used to predict the cell relationship between word tokens in the table structure diagram of the document to be predicted. The input item of the cell relationship prediction model can be the table structure diagram of the document to be predicted. The cell relationship in the table structure diagram of the document to be predicted is learned by the cell relationship prediction model, and the same cell relationship matrix, the next cell relationship matrix in the same row, and the next cell relationship matrix in the same column are output to obtain the cell relationship between word tokens in the table structure diagram of the document to be predicted.

[0107] In one implementation, if Figure 2 The step of inputting the table structure diagram of the document to be predicted into a cell relationship prediction model and outputting the cell relationship between word tokens in the table structure diagram of the document to be predicted through the cell relationship prediction model specifically includes:

[0108] S21, based on the table structure diagram of the document to be predicted, determining the representation vector of each word Token through the encoding module in the cell relationship prediction model;

[0109] S22, combining the representation vectors of each word Token in pairs and inputting them into a relationship prediction module in the cell relationship prediction model, and outputting the cell relationship between the word Tokens in the table structure diagram of the document to be predicted through the relationship prediction module.

[0110] Specifically, in step S21, the encoding module is used to learn the word tokens in the table structure diagram of the document to be predicted to determine the representation vector of each word token, wherein the representation vector of each word token includes the knowledge information of each word token, so that the cell relationship between each word token can be determined based on the representation vector of each word token.

[0111] In one implementation, the step of determining the representation vector of each word token based on the table structure diagram of the document to be predicted by using the encoding module in the cell relationship prediction model specifically includes:

[0112] S211, obtaining an image vector of the table structure diagram of the document to be predicted;

[0113] S212, identifying each word token and the word position vector of each word token in the table structure diagram of the document to be predicted, and determining a text vector based on each word token and the word position vector of each word token;

[0114] S213. Based on the image vector and the text vector, determine the representation vector of each word Token through the encoding module in the cell relationship prediction model.

[0115] Specifically, in step S211, the image vector is obtained by encoding the table structure diagram of the document to be predicted, and carries the image information of the table structure diagram of the document to be predicted. When obtaining the image vector, since each word token is an image block in the table structure diagram of the document to be predicted, in order to better fuse the text information corresponding to the word token with its corresponding image information, the table structure diagram of the document to be predicted can be divided into several image blocks, and then the image vector corresponding to the table structure diagram of the document to be predicted is determined by feature encoding each image block. Of course, in practical applications, the table structure diagram of the document to be predicted can also be directly encoded to obtain an image vector, etc. Here, the method for obtaining the image vector is not specifically limited, and only a specific implementation method is given as an illustration.

[0116] For example, Figure 3 As shown, the step of obtaining the image vector of the table structure diagram of the document to be predicted specifically includes:

[0117] Divide the table structure diagram of the document to be predicted into a plurality of image blocks, and obtain a position vector of each image block in the table structure diagram of the document to be predicted;

[0118] Input each image block into a linear layer, and output a graph embedding of each image block through the linear layer;

[0119] The position vector of each image block is input into the image position embedding layer, and the image position embedding of each image block is output through the image position embedding layer;

[0120] Merge the image embedding and image position embedding of each image block to obtain the pixel features of each image block;

[0121] The vector formed by the pixel features of each image block is used as the image vector of the document table structure diagram to be predicted.

[0122] Specifically, a number of image blocks are obtained by segmenting the table structure diagram of the document to be predicted, and each of the image blocks is a Patch, and each Patch corresponds to a graph position vector. The graph position vector is used to reflect the position information of the image block (i.e., Patch) in the table structure diagram of the document to be predicted, wherein the position information includes the x-axis coordinate and y-axis coordinate of the upper left point of the image block, and the x-axis coordinate and y-axis coordinate of the lower right point, that is, the image position vector can be expressed as (x1, y1, x2, y2), x1 is the x-axis coordinate of the upper left point, y1 is the y-axis coordinate of the upper left point, x2 is the x-axis coordinate of the lower right point, and y2 is the y-axis coordinate of the lower right point. Of course, in practical applications, the position information can also be expressed in other ways, such as the x-axis coordinate and y-axis coordinate of the lower left point, and the x-axis coordinate and y-axis coordinate of the upper right point, or the x-axis coordinate and y-axis coordinate of the center point, and the width and height of the image block.

[0123] Image embedding is obtained by encoding the image block, and is used to reflect the content information of the image block. Image position embedding is obtained by encoding the image position vector, and is used to reflect the position information of the image block. Pixel features are obtained by merging image embedding and image position embedding, so that the pixel features include the content information and position information of the image block. That is, after obtaining the image embedding and image position embedding, the image embedding and the image position embedding are added to obtain the pixel features, and then the vector composed of the pixel features is used as the image vector, where the image vector includes Pixel features , , Represents the dimension of pixel features, Indicates the number of image blocks.

[0124] It should be noted that the image vector acquisition process can be obtained through a functional module independent of the cell relationship prediction model, or through a functional module in the cell relationship prediction model. In an embodiment of the present application, the image vector acquisition process is obtained through a functional module in the cell relationship prediction model. That is, the cell relationship prediction model may include a graph encoding module, the graph encoding module includes a linear layer and a graph position embedding layer, the linear layer is used to encode the image block, and the graph position embedding layer is used to encode the graph position vector. That is to say, when acquiring the image vector, after the document table structure diagram to be predicted is divided into several image blocks and the graph position vector of each image block is obtained, the image block is passed through a linear layer to determine the graph embedding of the image block, and then the graph position vector corresponding to the image block is passed through the graph position embedding layer to determine the graph position embedding of the image block; then the graph embedding and the graph position embedding are added to obtain the pixel features of the image block; finally, the vector composed of all pixel features is used as the image vector.

[0125] Further, in step S212, the word Token is a word in the table in the table structure diagram of the document to be predicted, and the word position vector of the word Token is used to reflect the position information of the word Token in the table structure diagram of the document to be predicted. Among them, the expression form of the word position vector of the Token is the same as the expression form of the image position vector of the above-mentioned image block, which will not be repeated here, and the specific description can refer to the description of the image position vector of the above-mentioned image block. Each word Token and the word position vector of each word Token can determine the table structure diagram of the document to be predicted through OCR recognition and a word segmenter (for example, the Tokenizer of the Bert model, etc.), that is, first identify the text part in the table structure diagram of the document to be predicted, and then segment the identified text information to obtain each word Token and the text box corresponding to each word Token, and determine the position information of the word Token through the text box.

[0126] After obtaining each word token and the word position vector of each word token, the word token and the word position vector of the token are encoded respectively to obtain text features. In one implementation, Figure 3 As shown, the determination of the text vector based on each word token and the word position vector of each word token specifically includes:

[0127] Input each word token into the word embedding layer, and output the word embedding of each word token through the word embedding layer;

[0128] The word position vector of each word token is input into the word position embedding layer, and the word position embedding of each word token is output through the word position embedding layer;

[0129] Merge the word embedding and word position embedding of each word token to obtain the text features of each word token;

[0130] The vector composed of the text features of each word token is used as the text vector.

[0131] Specifically, word embedding is obtained by encoding word tokens and is used to reflect the content information of word tokens. Word position embedding is obtained by encoding word position vectors and is used to reflect the position information of word tokens. Text features are obtained by combining word embedding and word position embedding, so that text features include the content information and position information of word tokens. That is to say, after obtaining word embedding and word position embedding, the word embedding and word position embedding are added to obtain text features, and then the vector composed of the text features is used as a text vector, where the text vector includes Text features , , Represents the dimension of text features, Indicates the number of word tokens.

[0132] It should be noted that the process of obtaining the text vector can be obtained through a functional module independent of the cell relationship prediction model, or it can be obtained through a functional module in the cell relationship prediction model. In an embodiment of the present application, the process of obtaining the text vector is obtained through a functional module in the cell relationship prediction model. That is, the cell relationship prediction model may include a text encoding module, and the text encoding module may include a word embedding layer and a word position embedding layer, the word embedding layer is used to encode the word token, and the word position embedding layer is used to encode the word position vector. That is to say, when obtaining the text vector, after obtaining each word token and the word position vector of each token, each token is passed through the word embedding layer to determine the word embedding, and then the word position vector of the word token is passed through the word position embedding layer to determine the word position embedding; then the word embedding and the word position embedding are added to obtain the text features of the word token; finally, the vector composed of all text features is used as the text vector.

[0133] Further, in step S213, the encoding module is used to determine the representation vector of each word token, wherein the representation vector of the word token is obtained by cross-learning the image vector and the text vector. That is, after obtaining the image vector and the text vector, the image vector and the text vector are concatenated to obtain a concatenated vector, and the vector length of the concatenated vector is + , the feature dimension of each vector feature in the concatenated vector is ; Then the concatenated vector is input into the encoding module, and the encoding module outputs the representation vector of each word Token. Among them, the encoding module can use a Transformer encoder, which includes several encoding layers (for example, 12 layers). The concatenated vector is operated through each encoding layer in the Transformer encoder in turn, and the representation vector output by the last encoding layer is taken as the representation vector of each Token: ,in , represents the dimension of the representation vector, Indicates the number of word tokens.

[0134] In addition, in practical applications, other methods can also be used to determine the representation vector of each word token, for example, by performing feature extraction on the table structure of the document to be predicted to determine the representation vector of the word token, or by determining the representation vector of the word token based on the document vector, etc.

[0135] The embodiment of the present application first divides the table structure of the document to be predicted into several image blocks, then obtains the pixel features of each image block to determine the image vector, and then determines the document vector based on each word token and its word position vector, and finally combines the image vector and the document vector to determine the representation vector of each word token, so that the cell relationship prediction model can utilize the image information of the table structure of the document to be predicted and the document information of the table, so as to learn multimodal information and improve the accuracy of the cell relationship predicted by the cell relationship prediction model.

[0136] Further, in step S22, the relationship prediction module is used to predict the cell relationship between word tokens in the table structure diagram of the document to be predicted, wherein the input item of the relationship prediction model is the pairwise combination of the representation vectors of each word token, and the output item is the same cell relationship matrix, the next cell relationship matrix in the same row, and the next cell relationship matrix in the same column. That is to say, when predicting the cell relationship between word tokens through the relationship prediction module, the relationship prediction module needs to perform three classification prediction tasks. For this purpose, the relationship prediction module can include three relationship classification units, each relationship classification unit corresponds to a cell relationship, and then three types of cell relationship predictions are performed through the three relationship classification units to obtain the predicted cell relationship.

[0137] Based on this, the representation vector based on each word token, through the relationship prediction module in the cell relationship prediction model, determines the predicted cell relationship corresponding to the table structure diagram of the document to be predicted, specifically including:

[0138] Concatenate the representation vectors of each word token in pairs to obtain several combined representation vectors;

[0139] Inputting a plurality of combined representation vectors into a first relation classification unit, a second relation classification unit and a third relation classification unit in the relation prediction module respectively;

[0140] The first relationship classification unit outputs the same cell relationship matrix, the second relationship classification unit outputs the next cell relationship matrix in the same row, and the third relationship classification unit outputs the next cell relationship matrix in the same column to obtain the predicted cell relationship corresponding to the table structure diagram of the document to be predicted.

[0141] Specifically, the combined representation vector is obtained by connecting the representation vectors corresponding to the two word tokens, where the combined representation vector can be expressed as , , For the The representation vector of the word Token, For the The representation vector of word tokens, and when the number of word tokens is When , the number of combined representation vectors is .

[0142] After obtaining a number of combined representation vectors, the combined representation vectors are respectively input into the first relationship classification unit, the second relationship classification unit and the third relationship classification unit in the relationship prediction module, and the predicted cell relationship corresponding to the document table structure diagram to be predicted is determined by the first relationship classification unit, the second relationship classification unit and the third relationship classification unit. Among them, the first relationship classification unit, the second relationship classification unit and the third relationship classification unit can adopt the same model structure, or different model structures, for example, the first relationship classification unit, the second relationship classification unit and the third relationship classification unit all include one MLP layer, or the first relationship classification unit, the second relationship classification unit and the third relationship classification unit all include multiple MLP layers, and the number of layers of the MLP layers included in the first relationship classification unit, the second relationship classification unit and the third relationship classification unit can be different. In addition, it should be noted that since the first relationship classification unit, the second relationship classification unit and the third relationship classification unit are respectively used to predict different cell relationships, the model parameters of the first relationship classification unit, the second relationship classification unit and the third relationship classification unit are different.

[0143] In a typical implementation, the first relationship classification unit, the second relationship classification unit, and the third relationship classification unit each include an MLP layer, and the MLP layer may include a linear layer Linear1, a binary linear layer Linear2, and an activation function layer Sigmoid, the linear layer is connected to the binary linear layer, the binary linear layer is connected to the activation function layer, the linear layer may have a Tanh activation function, the activation function layer is configured with a Sigmoid function, and the relationship probability is obtained through the Sigmoid function ,Relationship probability Used to indicate the word token and The relationship probability between the word tokens is determined when the relationship probability is greater than the preset threshold. word token and There is a relationship between the word tokens, where the relationship corresponds to the relationship classification unit to which the MLP layer belongs. Accordingly, the execution process of the first relationship classification unit, the second relationship classification unit, and the third relationship classification unit can be expressed as:

[0144] Linear1=Dense(in_dim=d,out_dim=d)

[0145] Linear2=Dense(in_dim=d,out_dim=1)

[0146] =Tanh(Linear1( ))

[0147] = Sigmoid(Linear2( ));

[0148] in, represents the combined representation vector, Express the output item of the linear layer Linear1, Represents the relationship probability, Dense(in_dim=d, out_dim=d) represents the operation of the linear layer Linear1, where in_dim=d means that the input dimension of Linear1 is d, and out_dim=d means that the output dimension is d, Dense(in_dim=d, out_dim=1) represents the operation of the binary linear layer Linear2, where in_dim=d means that the input dimension of Linear1 is d, and out_dim=1 means that the output dimension is 1.

[0149] Of course, in practical applications, the first relation classification unit, the second relation classification unit, and the third relation classification unit may also adopt other schemes, for example, a binary classification scheme of MLP (such as GlobalPointer: processing nested and non-nested NER in a unified manner, etc.) to directly predict the cell relationship between two word tokens.

[0150] From the above, it can be seen that in a typical implementation of the embodiment of the present application, Fig. 9 As shown, the cell relationship prediction model may include a graph encoding module, a text encoding module, an encoding module and a relationship prediction module, the graph encoding module and the text encoding module are both connected to the encoding module, and the encoding module is connected to the relationship prediction model. The graph encoding module may include a linear layer and a graph position embedding layer, the text encoding module may include a word embedding layer and a word position embedding layer, the encoding module adopts a Transformer encoder, the relationship prediction module includes a first relationship classification unit (i.e., same cell relationship MLP), a second relationship classification unit (i.e., next cell relationship MLP in the same row) and a third relationship classification unit (i.e., next cell relationship MLP in the same column), the image vector output by the graph encoding module and the text vector output by the text encoding module are input items of the Transformer encoder, the representation vectors of each token output by the Transformer encoder are combined in pairs as the input of the first relationship classification unit, the second relationship classification unit and the third relationship classification unit, and the cell relationship between the word tokens in the document table structure diagram to be predicted is output through the first relationship classification unit, the second relationship classification unit and the third relationship classification unit.

[0151] It should be noted that the above implementation method is a typical implementation method given in this application. In practical applications, the cell relationship prediction model can also adopt other structures. For example, the cell relationship prediction model includes an encoding module and a prediction module, or the cell relationship prediction model includes the above-mentioned graph encoding module, text encoding module, encoding module and relationship prediction module, and also includes a segmentation module and a recognition module. The recognition module includes an OCR model and a word segmenter, wherein the segmentation model can divide the table structure diagram of the document to be predicted into several image blocks, and obtain the image position vector of each image block; the recognition module is used to identify each word token and the word position vector of each word token in the table structure diagram of the document to be predicted, etc.

[0152] In summary, an embodiment of the present application provides a cell relationship prediction method, the method comprising obtaining a table structure diagram of a document to be predicted; inputting the table structure diagram of the document to be predicted into a cell relationship prediction model, and outputting the cell relationships between word tokens in the table structure diagram of the document to be predicted through the cell relationship prediction model, the cell relationships between word tokens including the same cell relationship matrix, the next cell relationship matrix of the same row, and the next cell relationship matrix of the same column. An embodiment of the present application divides the cell relationships into the same cell relationship, the next cell relationship of the same row, and the next cell relationship of the same column, and then performs relationship prediction on the table structure diagram of the document to be predicted through a trained cell relationship prediction model, and outputs the accurate cell relationship between each word token, so as to better parse the table structure, especially for complex tables, the accurate table structure can also be parsed.

[0153] Based on the above cell relationship prediction method, the present application embodiment provides a cell relationship prediction model training method, such as Figure 4 As shown, the cell relationship prediction model training method specifically includes:

[0154] N10. Obtain the training document table structure diagram and the labeled cell relationships between the word tokens therein;

[0155] N20, inputting the training document table structure diagram into an initial cell relationship prediction model, and outputting predicted cell relationships between word tokens in the training document table structure diagram through the initial cell relationship prediction model, wherein the predicted cell relationships include a predicted same-cell relationship matrix, a predicted next-cell relationship matrix in the same row, and a predicted next-cell relationship matrix in the same column;

[0156] N30. Train the cell relationship prediction model based on the labeled cell relationship between the word tokens and the predicted cell relationship between the word tokens to obtain a cell relationship prediction model.

[0157] Specifically, in step N10, the training document table structure diagram is a sample document table structure diagram in a training sample set for training a cell relationship prediction model. The training sample set may be collected through a network or other means, and includes a plurality of training document table structure diagrams, wherein the plurality of training document table structure diagrams may include one or more tables. For ease of understanding and explanation, the embodiment of the present application takes the training document table structure diagram including one table as an example for explanation. Of course, in addition to being collected directly through a network or other means, the training sample set may be an initial sample set first collected through a network or other means, and then obtained by enhancing the initial sample set. For example, the initial sample set may be first obtained, and then randomly rotated and flipped, randomly subjected to brightness transformation, and mesh elastic deformation, etc.

[0158] The labeled cell relationship between word tokens in the training document table structure diagram is obtained by labeling the cell relationship between each word token in the table in the document table structure to be predicted, and is used as the supervision information of the cell relationship prediction model to supervise the training process of the cell relationship prediction model by labeling the cell relationship. The labeled cell relationship between the word tokens includes a labeling same cell relationship matrix, a labeling next cell relationship matrix in the same row, and a labeling next cell relationship matrix in the same column, wherein the labeling same cell relationship matrix stores the same unit labeling relationship between two tokens, the labeling next cell relationship matrix in the same row stores the labeling next cell relationship between two tokens in the same row, and the labeling next cell relationship matrix in the same column stores the labeling next cell relationship between two word tokens in the same column. In addition, in the labeling same cell relationship matrix, the labeling next cell relationship matrix in the same row, and the labeling matrix of the next cell relationship in the same column, each matrix can use 1 to indicate that the two word tokens have the corresponding relationship of the matrix, and use 0 to indicate that the two word tokens do not have the corresponding relationship of the matrix. Of course, in practical applications, other methods can also be used to represent, and no specific restrictions are made here.

[0159] The embodiment of the present application labels cell relationships including labeling the same cell relationship matrix, labeling the next cell relationship matrix in the same row, and labeling the next cell relationship matrix in the same column, and uses the labeled cell relationships as supervisory information to train the cell relationship prediction model, so that the cell relationship prediction model can well learn the positional relationship between cells, thereby better parsing the table structure, especially for complex tables, and can also parse the accurate table structure.

[0160] The labeling process of labeling the cell relationship between the word tokens specifically includes:

[0161] Obtain each word Token in the training document table structure and the row and column information of each word Token;

[0162] Mark the cell relationships between word Tokens according to the row and column information of each word Token, so as to obtain the marked cell relationships between word Tokens corresponding to the training document table structure. Among them, when two word Tokens are in the same cell, mark the same cell relationship between the two word Tokens; when two word Tokens do not belong to the same cell relationship, and one word Token is behind the other word Token in the row direction and there is no non-empty cell between the two word Tokens in the row direction, mark the relationship of the next cell in the same row between the two word Tokens; when two word Tokens do not belong to the same cell relationship, and one word Token is behind the other word Token in the column direction and there is no non-empty cell between the two word Tokens in the column direction, mark the relationship of the next cell in the same column between the two word Tokens.

[0163] Specifically, the word Token is a word in the table of the to-be-predicted document table structure. For example, for the to-be-predicted document table structure shown as Figure 5 shown, a series of word Tokens can be obtained as shown in Figure 6 shown: "chi", "yard", "male", etc. The row and column information of the word Token is used to represent the row and column to which the word Token belongs in the table. For example, in the to-be-predicted document table structure shown as Figure 6 shown, "chi" is a word Token in the to-be-predicted document table structure, and the row and column information of this word Token is the 1st and 2nd rows, and the 1st column.

[0164] It can be understood that the Token is a tokenizer such as the tokenizer in bert, which tokenizes the text to obtain a series of words. However, some tokenizers will split an English word in a natural language into many segments: name is split into na and me. Therefore, in one embodiment, when obtaining the word Token and the row and column information of each word Token, the cells in the training document table structure and the row and column information of the cells can be obtained first, and then the relationships between the cells and the cells are marked based on the row and column information of each cell. Then, each cell text is tokenized to obtain the word Tokens included in each cell, and the marked cell relationships between the word Tokens required for training are obtained through the marked cells and the relationships between the cells.

[0165] After obtaining each word token and the row and column information of each word token, the cell relationship between each word token can be determined according to the row and column information of each word token. For example, when the row and column information of two word tokens or multiple word tokens are the same, it means that the two word tokens or multiple word tokens are located in the same cell. When the cells where the word tokens are located have a row-adjacent relationship, it means that the cells where the word tokens are located are adjacent in the row direction. When the cells where the word tokens are located have a column-adjacent relationship, it means that the cells where the word tokens are located are adjacent in the column direction. In addition, because when two word tokens have a cell relationship, the two word tokens do not have a next cell relationship in the same row and / or a next cell relationship in the same column, when marking the cell relationship between each word token according to the row and column information of each word token, the same cell relationship between the tokens can be marked first, and then the next cell relationship in the same row and the next cell relationship in the same column can be marked.

[0166] Exemplarily, the process of marking the cell relationship between two word tokens according to the row and column information of each word token can be as follows: first, determine whether the two word tokens are located in the same cell according to the row and column information of each word token, and when the two word tokens are located in the same cell, configure the same cell relationship between the two word tokens; then, determine whether the cell to which one of the two word tokens belongs is located after the cell to which the other word token belongs in the row direction according to the row and column information of each word token (that is, the cells to which the two tokens belong are not the same cell, and are adjacent in the row direction), and when the cell to which one of the two word tokens belongs is located after the cell to which the other word token belongs in the row direction, en belongs to the cell, and when there is no non-empty cell between the two word tokens in the row direction, a labeled next cell relationship in the same row is configured between the two word tokens, and according to the row and column information of each word token, it is judged that the cell to which one of the two word tokens belongs is located after the cell to which the other word token belongs in the column direction (that is, the cells to which the two tokens belong are not the same cell, and are adjacent in the column direction), and when the cell to which one of the two word tokens belongs is located after the cell to which the other word token belongs in the column direction, and there is no non-empty cell between the two word tokens in the column direction, a labeled next cell relationship in the same column is configured between the two word tokens.

[0167] The following uses specific examples to illustrate the relationship between marking the same cell, marking the next cell in the same row, and marking the next cell in the same column.

[0168] 1. Label the relationship between cells

[0169] The row and column information of "chi" and "yard" are both in the 1st and 2nd rows, 1st column, and the row and column information of "gender" are both in the 1st row, 2nd and 3rd columns. "Chi" and "yard" have the same cell relationship, and "gender" has the same cell relationship. As Figure 7 shown, the same cell relationship is marked for "chi" and "yard", and the same cell relationship is marked for "gender".

[0170] II. Marking the relationship of the next cell in the same row

[0171] The relationship of the next cell in the same row will be constructed between non-empty cells in the same row from left to right. As Figure 8 shown, "chi" and "yard" are in the cells of the 1st and 2nd rows, 1st column; "gender" is in the cells of the 1st row, 2nd and 3rd columns; "male" is in the cell of the 2nd row, 2nd column. Therefore, pairwise relationships of the next cell in the same row are formed between "chi", "yard" and "gender", "male"; if the next cell is empty, then it is linked to the subsequent non-empty cell in the same row until the end. As Fig. 9 shown, there is a relationship of the next cell in the same row between "yard" and "female".

[0172] II. Marking the relationship of the next cell in the same column

[0173] The relationship of the next cell in the same column will be constructed between non-empty cells in the same column from top to bottom. As Fig.10 shown, "gender" is in the cells of the 1st row, 2nd and 3rd columns; "male" is in the cell of the 2nd row, 2nd column; "female" is in the cell of the 2nd row, 3rd column. Therefore, pairwise relationships of the next cell in the same column are formed between "gender" and "male", "female"; if the next cell is empty, then it is linked to the subsequent non-empty cell in the same column until the end. As Fig.11 shown, there is a relationship of the next cell in the same column between "female" and "6".

[0174] Furthermore, in step N20, the model parameters of the initial cell relationship prediction model are different from those of the cell relationship prediction model. Among them, the model parameters of the initial cell relationship prediction model are the initial model parameters, and the model parameters of the cell relationship prediction model are the model parameters after training the initial cell relationship prediction model. The model structure and processing process of the initial cell relationship prediction model are the same as those of the above-mentioned cell relationship prediction model, and can be specifically referred to the above description, which will not be elaborated here.

[0175] Further, in step N30, when the cell relationship prediction model is trained based on the labeled cell relationship between word tokens and the predicted cell relationship between the word tokens, a total loss function term is constructed based on the labeled cell relationship and the predicted cell relationship, and then the initial model parameters of the initial cell relationship prediction model are optimized based on the total loss function term to obtain the cell relationship prediction model.

[0176] Exemplarily, the training of the cell relationship prediction model based on the labeled cell relationship between the word tokens and the predicted cell relationship between the word tokens to obtain the cell relationship prediction model specifically includes:

[0177] S31, calculating a first loss function term based on the labeled same-cell relationship matrix and the same-cell relationship matrix;

[0178] S32, calculating a second loss function term based on the labeled next cell relationship matrix in the same row and the next cell relationship matrix in the same row;

[0179] S33, calculating a third loss function term based on the labeled next cell relationship matrix in the same column and the next cell relationship matrix in the same column;

[0180] S34. Calculate a total loss function term based on the first loss function term, the second loss function term and the third loss function term, and train the cell relationship prediction model based on the total loss function term to obtain a cell relationship prediction model.

[0181] Specifically, the first loss function term is used to reflect the error of the same cell relationship, the second loss function term is used to reflect the error of the next cell relationship in the same row, and the third loss function term is used for the error of the next cell relationship in the same column, wherein the first loss function term, the second loss function term and the third loss function term can be constructed in the same way or in different ways. For example, the first loss function term, the second loss function term and the third loss function term are all constructed using the cross entropy loss function or the GlobalPointer loss. After obtaining the first loss function term, the second loss function term and the third loss function term, the sum of the first loss function term, the second loss function term and the third loss function term can be used as the total loss function term, or the first loss function term, the second loss function term and the third loss function term can be weighted to determine the total loss function term, etc.

[0182] In summary, the embodiment of the present application provides a method for training a cell relationship prediction model, the method specifically comprising: obtaining a training document table structure and the cell annotation relationship between word tokens therein, wherein the cell annotation relationship between the word tokens includes a same cell annotation relationship matrix, a next cell annotation relationship matrix in the same row, and a next cell annotation relationship matrix in the same column; inputting the training document table structure into an initial cell relationship prediction model, and outputting the cell prediction relationship between the word tokens in the training document table structure through the initial cell relationship prediction model, wherein the cell prediction relationship includes a predicted same cell relationship matrix, a predicted next cell relationship matrix in the same row, and a predicted next cell relationship matrix in the same column; training the cell relationship prediction model based on the cell annotation relationship between the word tokens and the cell prediction relationship between the word tokens to obtain a cell relationship prediction model. The embodiment of the present application constructs a cell relationship including a same cell relationship matrix, a next cell relationship matrix in the same row, and a next cell relationship matrix in the same column, and trains the cell relationship prediction model with the cell relationship as supervision, so that the cell relationship prediction model can well learn the positional relationship of each word token, so that the table structure can be better parsed, especially for complex tables, and the accurate table structure can also be parsed.

[0183] Based on the above cell relationship prediction method, this embodiment provides a table parsing method, using the above cell relationship prediction method, such as Fig.12 As shown, the table parsing method specifically includes:

[0184] B10, determining the target cell relationship between the word tokens in the table structure diagram of the document to be parsed by the cell relationship prediction method;

[0185] B20. Decode the table structure based on the target cell relationship between the word tokens to obtain the document table structure diagram of the document to be parsed.

[0186] Specifically, the target cell relationship between word tokens is obtained by using the above-mentioned cell relationship prediction method to perform cell relationship prediction on the table structure diagram of the document to be parsed. Accordingly, the target cell relationship between word tokens includes a target same-cell relationship matrix, a target next-cell relationship matrix in the same row, and a target next-cell relationship matrix in the same column.

[0187] Furthermore, after obtaining the target same-cell relationship matrix, the target next-cell relationship matrix in the same row, and the target next-cell relationship matrix in the same column, the cell relationship between each pair of word tokens is determined through the target same-cell relationship matrix, the target next-cell relationship matrix in the same row, and the target next-cell relationship matrix in the same column. That is, the word tokens located in the same cell are determined through the target same-cell relationship matrix, the word tokens having the same-cell next-cell relationship are determined through the target next-cell relationship matrix in the same row, and the word tokens having the same-column next-cell relationship matrix are determined through the target next-cell relationship matrix in the same column. Then, the table structure is restored according to the cell relationship between each word token to obtain the document table structure of the document table structure diagram to be parsed.

[0188] It should be understood that the embodiment of the present application predicts the cell relationship between word tokens based on the cell relationship prediction method, and then infers the relationship between cells to obtain the document table structure of the document table structure diagram to be parsed. If the word token is regarded as the lower layer and the cell is regarded as the upper layer, when training the initial cell relationship prediction model, the relationship between cells is first marked, and then the cell text is segmented to obtain the word token, and the marked cell relationship between the word tokens required for training is marked by the marked cells and the relationship between cells. The initial cell relationship prediction model is trained with the marked cell relationship between word tokens to obtain a cell relationship prediction model. After using the cell relationship prediction model to obtain the cell relationship between word tokens in the document table structure diagram to be predicted, the relationship between the cells where the word tokens are located is inferred based on the three relationships between the predicted word tokens, thereby achieving the purpose of table parsing.

[0189] Exemplarily, the table structure decoding based on the target cell relationship between the word tokens to obtain the document table structure diagram to be parsed specifically includes:

[0190] B21. Cluster the word tokens with the same cell relationship into the same cell according to the target same cell relationship matrix in the target cell relationship between the word tokens, so as to obtain a plurality of clustered cells;

[0191] B22, according to the target in-line next cell relationship matrix in the target cell relationship between the word tokens, determining the cluster cells having the in-line next cell relationship among the plurality of cluster cells, so as to restore the in-line next cell relationship between the cluster cells;

[0192] B23. According to the target next cell relationship matrix in the target cell relationship between word tokens, the cluster cells having the next cell relationship in the same column among the plurality of cluster cells are determined to restore the next cell relationship in the same column between the cluster cells and obtain the document table structure of the document table structure diagram to be parsed.

[0193] Specifically, in step B21, a plurality of clustering cells are obtained by clustering word tokens according to cell relationships, wherein the plurality of clustering cells may include one word token or multiple word tokens, and when a clustering cell includes multiple word tokens, the multiple word tokens have a same-cell relationship. When determining the next cell relationship in the same row and the next cell relationship in the same column later, each of the plurality of clustering cells will be regarded as a cell.

[0194] Exemplarily, clustering word tokens having same-cell relationships into the same cell according to the target same-cell relationship matrix in the target cell relationships between word tokens specifically includes:

[0195] Treat each word token as a cell community;

[0196] Traverse each word token, determine the number of first word tokens in each cell community that have a same-cell relationship with the word token according to the target same-cell relationship matrix, determine the target cell community corresponding to the word token according to the number of second word tokens in each cell community and the number of first word tokens corresponding to them, and move the word token from its original cell community to the target cell community;

[0197] Re-execute the step of traversing each word token until the word tokens in all cell communities are not moved, so as to obtain several clustered cells.

[0198] Specifically, when each word token is separated into a cell community in the initial stage, the cell community can be expressed as ,in ,in is the number of word tokens. Then for each word token (denoted as word ), determine each cell community based on the target cell relationship matrix Chinese and words Words with same cell relationship , among which the word With words The same cell relationship specifies that the word in the target same cell relationship matrix With words The cell relationship of the target cell meets the requirements of the same cell relationship. For example, the relationship probability between the word tokens stored in the target cell relationship matrix, then the word tokens in the target cell relationship matrix With words The probability of association between them is greater than the preset threshold.

[0199] After obtaining the cell community Chinese and words After all the word tokens with the same cell relationship are obtained, read the number of word tokens obtained to get the number of first word tokens , then according to and cell communities The number of second word tokens of the included word tokens Determining words With the cell community The first probability of having the same cell relationship, and finally judging the word based on the first probability Is it a cell community? A member of, where the first probability = , and when the first probability is greater than a preset probability threshold (for example, 0.5, etc.), the decision word For the cell community A member of, on the contrary, a judgement Not for cell communities A member of.

[0200] When the word For the cell community When you are a member of Move to Cell Community A member of which, when the word Determined to be multiple cell communities When a member of The largest cell community as word The corresponding target cell community, and Move to the target cell community, and originally contain the word The words in the cell community (referred to as the original cell community) will be removed. When all words After all traversals are completed, the moved cell community is used as the cell community, and each word token is traversed again to update the cell community until the word tokens in all cell communities are not moved, and the clustering of word tokens with the same cell relationship is completed to obtain several clustered cells.

[0201] Example: There is only a "same cell relationship" between the word Token "chi" and the word Token "ma". The first probability between the word Token "chi" and the word Token "ma" is 1.0 > 0.5. Then, the Token "chi" and the Token "ma" will be clustered into a single clustering cell.

[0202] It should be noted that in actual applications, other methods can also be used to determine the clustering cells. For example, a group of word Tokens with the same cell relationship can be selected from all the word Tokens to form a clustering cell first, and then another group of word Tokens with the same cell relationship can be selected from the remaining Tokens to form a clustering cell, and so on to obtain several clustering cells.

[0203] Furthermore, in step B22, when restoring the next cell relationship in the same row between the clustering cells, the aggregated cells can be regarded as independent cells. Then, the next cell relationship in the same row between the aggregated cells can be determined by the next cell relationship in the same row between the word Tokens in the aggregated cells in the target next cell relationship matrix in the same row, and then the next cell relationship in the same row between the clustering cells can be restored according to the determined next cell relationship in the same row.

[0204] Exemplarily, determining the clustering cells with the next cell relationship in the same row among the several clustering cells according to the target next cell relationship matrix in the target cell relationship between the word Tokens to restore the next cell relationship in the same row between the clustering cells specifically includes:

[0205] Traverse each clustering cell among the several clustering cells. According to the target next cell relationship matrix, calculate the number of third word Tokens with the next cell relationship in the same row between the word Tokens in the clustering cell and the word Tokens in each other clustering cell; based on the fourth Token number of the clustering cell, the number of third word Tokens, and the fifth Token number of the other clustering cell corresponding to the Token number, determine the next cell relationship in the same row between each clustering cell and the other clustering cells;

[0206] According to the next cell relationship in the same row between each clustering cell and the other clustering cells, determine the clustering cells with the next cell relationship in the same row among the several clustering cells to restore the next cell relationship in the same row between the clustering cells.

[0207] Specifically, the number of third-word tokens is the number of word tokens in the cluster cell that have a next-cell relationship with the word tokens in each other cluster cell, and the number of fourth-word tokens and the number of fifth-word tokens are the number of word tokens in the two cluster cells corresponding to the number of third-word tokens. Each word token and cluster cell in The number of peer-next-cell relationships between each word token in is , clustering cells The number of word tokens in , clustering cells The number of word tokens in , then the number of tokens for the third word is , the number of tokens for the fourth word is , the number of tokens for the fifth word is .

[0208] After obtaining the number of third-word tokens, the number of fourth-word tokens, and the number of fifth-word tokens, determine the next cell relationship in the same row between the two cluster cells corresponding to the number of third-word tokens according to the number of third-word tokens, the number of fourth-word tokens, and the number of fifth-word tokens. Among them, the method for determining the next cell relationship in the same row between the two cluster cells corresponding to the number of third-word tokens can be: first calculate the product of the number of fourth-word tokens and the number of fifth-word tokens, and then calculate the ratio of the number of third-word tokens to the calculated product to obtain the second probability that the two cluster cells corresponding to the number of third-word tokens have a next cell relationship in the same row, and then determine whether there is a next cell relationship in the same row between the two cluster cells corresponding to the number of third-word tokens according to the second probability, wherein when the second probability is greater than a preset probability threshold (such as 0.5, etc.), it is determined that there is a next cell relationship in the same row, otherwise, it is determined that there is no next cell relationship in the same row. For example, as in the above example, the number of third-word tokens is , the number of tokens for the fourth word is , the number of tokens for the fifth word is , then the second probability = If the second probability > 0.5, the cell can be considered a cluster With clustered cells There is a peer-next-cell relationship between them.

[0209] For example, the clustering cell "size" and the clustering cell "gender", where the clustering cell "size" has two word tokens, "size" and "code"; the clustering cell "gender" also has two word tokens, "sex" and "gender". According to the target in-line next cell relationship matrix, there is a "in-line next cell relationship" between the word tokens in the clustering cell "size" and the clustering cell "gender", such as "size" -> "sex", "size" -> "gender", and "code" -> "sex". Then the second probability between the clustering cell "size" and the clustering cell "gender" = 3 / (2*2) = 0.75. Since 0.75>0.5, there is a "in-line next cell relationship" between the clustering cell "size" and the clustering cell "gender".

[0210] Further, in step B23, when restoring the next cell relationship in the same column between clustered cells, the clustered cells can be treated as independent cells, and then the next cell relationship in the same column between the word tokens in the clustered cells in the target next cell relationship matrix is ​​used to determine the next cell relationship in the same column between the clustered cells, and then the next cell relationship in the same column between the clustered cells is restored according to the determined next cell relationship in the same column.

[0211] Exemplarily, the determining, according to the target same-column next cell relationship matrix in the target cell relationship between the word tokens, the clustering cells having the same-column next cell relationship among the plurality of clustering cells to restore the same-column next cell relationship between the clustering cells specifically includes:

[0212] Traverse each clustering cell in the plurality of clustering cells, and calculate the number of sixth word tokens having a same-column next-cell relationship between the word token in the clustering cell and the word token in each other clustering cell according to the target same-column next-cell relationship matrix; determine the same-column next-cell relationship between each clustering cell and other clustering cells based on the number of seventh word tokens in the clustering cell, the number of sixth word tokens, and the number of eighth word tokens in other clustering cells corresponding to the number of word tokens;

[0213] According to the next cell in the same column relationship between each clustering cell and other clustering cells, clustering cells having the next cell in the same column relationship among the plurality of clustering cells are determined to restore the next cell in the same column relationship between the clustering cells.

[0214] Specifically, the number of the sixth word token is the number of word tokens in the cluster cell that have a next-cell relationship with the word tokens in each other cluster cell, and the number of the seventh word token and the number of the eighth word token are the number of word tokens in the two cluster cells corresponding to the number of the sixth word token. Each word token and cluster cell in The number of next cell relationships in the same column between each word token in is , clustering cells The number of word tokens in , clustering cells The number of word tokens in , then the number of tokens for the sixth word is , the number of tokens for the seventh word is , the number of tokens for the eighth word is .

[0215] After obtaining the number of tokens for the sixth word, the number of tokens for the seventh word, and the number of tokens for the eighth word, determine the next cell relationship in the same column between the two cluster cells corresponding to the number of tokens for the sixth word according to the number of tokens for the seventh word, and the number of tokens for the eighth word. Among them, the method for determining the next cell relationship in the same column between the two cluster cells corresponding to the number of tokens for the sixth word can be: first calculate the product of the number of tokens for the seventh word and the number of tokens for the eighth word, and then calculate the ratio of the number of tokens for the sixth word to the calculated product to obtain the third probability that the two cluster cells corresponding to the number of tokens for the sixth word have the next cell relationship in the same column, and then determine whether the two cluster cells corresponding to the number of tokens for the sixth word have the next cell relationship in the same column according to the third probability, wherein when the third probability is greater than a preset probability threshold (such as 0.5, etc.), it is determined that there is a next cell relationship in the same column, otherwise, it is determined that there is no next cell relationship in the same column. For example, as in the above example, the number of tokens for the sixth word is , the number of tokens for the seventh word is , the number of tokens for the eighth word is , then the third probability = If the second probability > 0.5, the cell can be considered a cluster With clustered cells There is a peer-next-cell relationship between them.

[0216] For example: clustering cell "gender" and clustering cell "male", among which clustering cell "gender" also has 2 word tokens, "sex" and "gender", and clustering cell "male" has 1 word token, "male"; according to the target same column next cell relationship matrix, it can be seen that the word tokens in clustering cell "gender" and clustering cell "male" have "sex" -> "male", "gender" -> "male" and have "same column next cell relationship", then the third probability between clustering cell "gender" and clustering cell "male" = 2 / (2*1) = 1, because 1>0.5, there is a "same column next cell relationship" between clustering cell "gender" and clustering cell "male".

[0217] Based on the above cell relationship prediction method, the present application embodiment provides a cell relationship prediction device, such as Fig.13 As shown, the cell relationship prediction device specifically includes:

[0218] The first acquisition module 101 is used to acquire a table structure diagram of a document to be predicted;

[0219] The first control module 102 is used to input the table structure diagram of the document to be predicted into a cell relationship prediction model, and output the cell relationship between word tokens in the table structure diagram of the document to be predicted through the cell relationship prediction model, wherein the cell relationship includes a same cell relationship matrix, a next cell relationship matrix in the same row, and a next cell relationship matrix in the same column.

[0220] Based on the above-mentioned cell relationship prediction model training method, the present application embodiment provides a cell relationship prediction model training device, such as Fig.14 As shown, the cell relationship prediction model used in the cell relationship prediction method as described above is trained, and the cell relationship prediction model training device specifically includes:

[0221] The second acquisition module 201 is used to acquire the table structure of the training document and the cell labeling relationship between the word tokens therein, wherein the cell labeling relationship between the word tokens includes a same cell labeling relationship matrix, a next cell labeling relationship matrix in the same row, and a next cell labeling relationship matrix in the same column;

[0222] The second control module 202 is used to input the training document table structure into an initial cell relationship prediction model, and output the cell prediction relationship between word tokens in the training document table structure through the initial cell relationship prediction model, wherein the cell prediction relationship includes a prediction matrix of the same cell relationship, a prediction matrix of the next cell relationship in the same row, and a prediction matrix of the next cell relationship in the same column;

[0223] The training module 203 is used to train the cell relationship prediction model based on the cell labeling relationship between the word tokens and the cell prediction relationship between the word tokens to obtain the cell relationship prediction model.

[0224] Based on the above table parsing method, the present application embodiment provides a table parsing device, such as Fig.15 As shown, the table parsing device using the cell relationship prediction method as described above specifically includes:

[0225] A determination module 301 is used to determine the target cell relationship between word tokens in the table structure diagram of the document to be parsed by using the cell relationship prediction method;

[0226] The decoding module 302 is used to perform table structure decoding based on the target cell relationship between the word tokens to obtain the document table structure of the document table structure diagram to be parsed.

[0227] This embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the method described in the above embodiment.

[0228] The present application also provides a terminal device, such as Fig.16 As shown, it includes at least one processor 20; a display screen 21; and a memory 22, and may also include a communications interface 23 and a bus 24. The processor 20, the display screen 21, the memory 22, and the communications interface 23 may communicate with each other through the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setting mode. The communications interface 23 may transmit information. The processor 20 may call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0229] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0230] The memory 22 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 22, that is, implementing the methods in the above embodiments.

[0231] The memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, may also be a transient storage medium.

[0232] In addition, the specific process of loading and executing the multiple instruction processors in the above-mentioned storage medium and the terminal device has been described in detail in the above-mentioned method, and will not be described one by one here.

[0233] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A cell relationship prediction method, characterized in that: The cell relationship prediction method specifically includes: Get the table structure diagram of the document to be predicted; Input the table structure diagram of the document to be predicted into a cell relationship prediction model, and output the cell relationship between the word tokens in the table structure diagram of the document to be predicted through the cell relationship prediction model, wherein the cell relationship between the word tokens includes a same cell relationship matrix, a next cell relationship matrix in the same row, and a next cell relationship matrix in the same column; The step of inputting the table structure diagram of the document to be predicted into a cell relationship prediction model and outputting the cell relationship between word tokens in the table structure diagram of the document to be predicted through the cell relationship prediction model specifically includes: Based on the table structure diagram of the document to be predicted, determining the representation vector of each word Token through the encoding module in the cell relationship prediction model; The representation vectors of each word Token are combined in pairs and input into the relationship prediction module in the cell relationship prediction model, and the cell relationship between the word Tokens in the table structure diagram of the document to be predicted is output through the relationship prediction module.

2. The cell relationship prediction method according to claim 1, characterized in that: The same-cell relationship matrix includes the same-cell relationships between two word tokens in the table structure diagram of the document to be predicted, the next-cell-in-the-same-row relationship matrix includes the next-cell-in-the-same-row relationship between two word tokens in the table structure diagram of the document to be predicted, and the next-cell-in-the-same-column relationship matrix includes the next-cell-in-the-same-column relationship between two word tokens in the table structure diagram of the document to be predicted.

3. The cell relationship prediction method according to claim 1, characterized in that: The step of determining the representation vector of each word Token based on the table structure diagram of the document to be predicted through the encoding module in the cell relationship prediction model specifically includes: Obtaining an image vector of the table structure diagram of the document to be predicted; Identify each word token and the word position vector of each word token in the table structure diagram of the document to be predicted, and determine the text vector based on each word token and the word position vector of each word token; Based on the image vector and the text vector, the representation vector of each word Token is determined through the encoding module in the cell relationship prediction model.

4. The cell relationship prediction method according to claim 3, characterized in that: The step of obtaining the image vector of the table structure diagram of the document to be predicted specifically includes: Divide the table structure diagram of the document to be predicted into a plurality of image blocks, and obtain a position vector of each image block in the table structure diagram of the document to be predicted; Input each image block into a linear layer, and output a graph embedding of each image block through the linear layer; The position vector of each image block is input into the image position embedding layer, and the image position embedding of each image block is output through the image position embedding layer; Merge the image embedding and image position embedding of each image block to obtain the pixel features of each image block; The vector formed by the pixel features of each image block is used as the image vector of the document table structure diagram to be predicted.

5. The cell relationship prediction method according to claim 3, characterized in that: The determining of the text vector based on each word token and the word position vector of each word token specifically includes: Get the word embedding of each word token and the word position embedding of the word position vector of each word token, and merge the word embedding and word position embedding of each word token to obtain the text features of each word token; The vector composed of the text features of each word token is used as the text vector.

6. The cell relationship prediction method according to claim 1, characterized in that: The step of combining the representation vectors of each word token in pairs and inputting them into a relationship prediction module in the cell relationship prediction model, and outputting the cell relationship between the word tokens in the table structure diagram of the document to be predicted through the relationship prediction module specifically includes: Merge the representation vectors of each word token in pairs to obtain several combined representation vectors; Inputting a plurality of combined representation vectors into a first relation classification unit, a second relation classification unit and a third relation classification unit in the relation prediction module respectively; The first relationship classification unit outputs the same cell relationship matrix, the second relationship classification unit outputs the next cell relationship matrix in the same row, and the third relationship classification unit outputs the next cell relationship matrix in the same column to obtain the cell relationship between the word tokens in the table structure diagram of the document to be predicted.

7. A cell relationship prediction model training method, characterized in that: Used for training the cell relationship prediction model used in the cell relationship prediction method according to any one of claims 1 to 6, wherein the cell relationship prediction model training method specifically comprises: Obtaining a training document table structure diagram and the labeled cell relationships between word tokens therein, wherein the labeled cell relationships between word tokens include a labeled same cell relationship matrix, a labeled next cell relationship matrix in the same row, and a labeled next cell relationship matrix in the same column; Input the training document table structure diagram into an initial cell relationship prediction model, and output the predicted cell relationship between word tokens in the training document table structure diagram through the initial cell relationship prediction model, wherein the predicted cell relationship includes a predicted same cell relationship matrix, a predicted next cell relationship matrix in the same row, and a predicted next cell relationship matrix in the same column; The cell relationship prediction model is trained based on the labeled cell relationship between the word tokens and the predicted cell relationship between the word tokens to obtain a cell relationship prediction model.

8. The cell relationship prediction model training method according to claim 7, characterized in that: The labeling process of labeling the cell relationship between the word tokens specifically includes: Read each word token and the row and column information of each word token in the training document table structure diagram; The cell relationships between word tokens are annotated according to the row and column information of each word token to obtain the annotated cell relationships between word tokens corresponding to the training document table structure diagram. When two word tokens are located in the same cell, the two word tokens are annotated with a same-cell relationship; when two word tokens do not belong to the same-cell relationship, one word token is located after the other word token in the row direction, and there is no non-empty cell between the two word tokens in the row direction, the two word tokens are annotated with a next-cell relationship in the same row; when two word tokens do not belong to the same-cell relationship, one word token is located after the other word token in the column direction, and there is no non-empty cell between the two word tokens in the column direction, the two word tokens are annotated with a next-cell relationship in the same column.

9. The cell relationship prediction model training method according to claim 7, characterized in that: The step of training the cell relationship prediction model based on the labeled cell relationship between the word tokens and the predicted cell relationship between the word tokens to obtain the cell relationship prediction model specifically includes: Calculate a first loss function term based on the labeled same-cell relationship matrix and the predicted same-cell relationship matrix; Calculate the second loss function term based on the labeled next cell relationship matrix and the predicted next cell relationship matrix; Calculate the third loss function term based on the labeled next cell relationship matrix in the same column and the predicted next cell relationship matrix in the same column; A total loss function term is calculated based on the first loss function term, the second loss function term and the third loss function term, and the initial cell relationship prediction model is trained based on the total loss function term to obtain a cell relationship prediction model.

10. A table parsing method, characterized in that: Using the cell relationship prediction method as described in any one of claims 1 to 6, the table parsing method specifically includes: Determine the target cell relationship between word tokens in the table structure diagram of the document to be parsed by the cell relationship prediction method; The table structure is decoded based on the target cell relationship between the word tokens to obtain the document table structure diagram of the document to be parsed.

11. The table parsing method according to claim 10, characterized in that: The table structure decoding based on the target cell relationship between the word tokens to obtain the document table structure diagram to be parsed specifically includes: According to the target same-cell relationship matrix in the target cell relationship between word tokens, the word tokens with the same-cell relationship are clustered into the same cell to obtain a number of clustered cells; Determine, according to the target in-line next cell relationship matrix in the target cell relationship between the word tokens, the cluster cells having the in-line next cell relationship among the plurality of cluster cells, so as to restore the in-line next cell relationship between the cluster cells; According to the target next cell relationship matrix in the target cell relationship between word tokens, the cluster cells having the next cell relationship in the same column among the plurality of cluster cells are determined to restore the next cell relationship in the same column between the cluster cells and obtain the document table structure of the document table structure diagram to be parsed.

12. The table parsing method according to claim 11, characterized in that: The clustering of word tokens having same-cell relationships into the same cell according to the target same-cell relationship matrix in the target-cell relationship between word tokens specifically includes: Treat each word token as a cell community; Traverse each word token, determine the number of first word tokens in each cell community that have a same-cell relationship with the word token according to the target same-cell relationship matrix, determine the target cell community corresponding to the word token according to the number of second word tokens in each cell community and the number of first word tokens corresponding to them, and move the word token from its original cell community to the target cell community; Re-execute the step of traversing each word token until the word tokens in all cell communities are not moved, so as to obtain several clustered cells.

13. The table parsing method according to claim 11, characterized in that: The step of determining the cluster cells having the same-row next-cell relationship among the plurality of cluster cells according to the target-row next-cell relationship matrix in the target-cell relationship between the word tokens to restore the same-row next-cell relationship between the cluster cells specifically includes: Traverse each cluster cell in the plurality of cluster cells, and calculate the number of third word tokens that have a next cell relationship in the same row between the word token in the cluster cell and the word token in each other cluster cell according to the target next cell relationship matrix; determine the next cell relationship in the same row between each cluster cell and other cluster cells based on the fourth token number of the cluster cell, the third word token number, and the fifth word token number of other cluster cells corresponding to the word token number; According to the in-line next cell relationships between each cluster cell and other cluster cells, cluster cells having in-line next cell relationships among the plurality of cluster cells are determined to restore the in-line next cell relationships between the cluster cells.

14. The table parsing method according to claim 11, characterized in that: The step of determining the cluster cells having the same column next cell relationship among the plurality of cluster cells according to the target same column next cell relationship matrix in the target cell relationship between the word tokens to restore the same column next cell relationship between the cluster cells specifically includes: Traverse each clustering cell in the plurality of clustering cells, and calculate the number of sixth word tokens having a same-column next-cell relationship between the word token in the clustering cell and the word token in each other clustering cell according to the target same-column next-cell relationship matrix; determine the same-column next-cell relationship between each clustering cell and other clustering cells based on the number of seventh word tokens in the clustering cell, the number of sixth word tokens, and the number of eighth word tokens in other clustering cells corresponding to the number of word tokens; According to the next cell in the same column relationship between each clustering cell and other clustering cells, clustering cells having the next cell in the same column relationship among the plurality of clustering cells are determined to restore the next cell in the same column relationship between the clustering cells.

15. A cell relationship prediction device, characterized in that: The cell relationship prediction device specifically comprises: The first acquisition module is used to obtain the table structure diagram of the document to be predicted; A first control module is used to input the table structure diagram of the document to be predicted into a cell relationship prediction model, and output the cell relationship between the word tokens in the table structure diagram of the document to be predicted through the cell relationship prediction model, wherein the cell relationship includes a same cell relationship matrix, a next cell relationship matrix in the same row, and a next cell relationship matrix in the same column; The step of inputting the table structure diagram of the document to be predicted into a cell relationship prediction model and outputting the cell relationship between word tokens in the table structure diagram of the document to be predicted through the cell relationship prediction model specifically includes: Based on the table structure diagram of the document to be predicted, determining the representation vector of each word Token through the encoding module in the cell relationship prediction model; The representation vectors of each word Token are combined in pairs and input into the relationship prediction module in the cell relationship prediction model, and the cell relationship between the word Tokens in the table structure diagram of the document to be predicted is output through the relationship prediction module.

16. A cell relationship prediction model training device, characterized in that: Used for training the cell relationship prediction model used in the cell relationship prediction method according to any one of claims 1 to 6, the cell relationship prediction model training device specifically comprises: The second acquisition module is used to obtain the training document table structure diagram and the annotated cell relationship between the word tokens therein, wherein the annotated cell relationship between the word tokens includes an annotated same cell relationship matrix, an annotated next cell relationship matrix in the same row, and an annotated next cell relationship matrix in the same column; A second control module is used to input the training document table structure diagram into an initial cell relationship prediction model, and output the predicted cell relationship between word tokens in the training document table structure diagram through the initial cell relationship prediction model, wherein the predicted cell relationship includes a predicted same cell relationship matrix, a predicted next cell relationship matrix in the same row, and a predicted next cell relationship matrix in the same column; The training module is used to train the cell relationship prediction model based on the labeled cell relationship between the word tokens and the predicted cell relationship between the word tokens to obtain the cell relationship prediction model.

17. A table parsing device, characterized in that: The table parsing device using the cell relationship prediction method according to any one of claims 1 to 6 specifically comprises: A determination module, used to determine the target cell relationship between word tokens in the table structure diagram of the document to be parsed by using the cell relationship prediction method; The decoding module is used to decode the table structure based on the target cell relationship between the word tokens to obtain the document table structure of the document table structure diagram to be parsed.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the cell relationship prediction method as described in any one of claims 1-6, and / or to implement the steps in the cell relationship prediction model training method as described in claims 7-9, and / or to implement the steps in the table parsing method as described in any one of claims 10-14.

19. A terminal device, characterized in that: include: Processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, it implements the steps in the cell relationship prediction method as described in any one of claims 1-6, and / or implements the steps in the cell relationship prediction method as described in any one of claims 7-9, and / or implements the steps in the table parsing method as described in any one of claims 10-14.

Images