Methods, computing devices, and computer-readable storage media for processing tables

Abstract

The present disclosure provides a method, a computing device and a computer readable storage medium for processing a table. The method comprises: cropping one or more table subgraphs from a picture using a target detection model, wherein each table subgraph comprises a table; performing optical character recognition on each table subgraph to detect a plurality of word blocks in the table subgraph, wherein each word block contains one or more characters; predicting, using a deep neural network model, a row probability that any two word blocks in the plurality of word blocks are in the same row and a column probability that any two word blocks in the plurality of word blocks are in the same column; and structurally reorganizing the plurality of word blocks based on the row probability that any two word blocks in the plurality of word blocks are in the same row and the column probability that any two word blocks in the plurality of word blocks are in the same column to reconstruct the table into a structured table.

Description

Technical Field

[0001] The present invention generally relates to the field of machine learning, and more particularly, to a method for processing a table, a computing device, and a computer-readable storage medium. Background Art

[0002] An academic paper is a scientific record of new scientific research results, innovative insights, and knowledge in experimental, theoretical, or predictive research on a particular academic topic, or a scientific summary of new advances achieved through the application of known principles to practical applications. Scientific research often requires reading vast quantities of academic papers to extract, summarize, and organize knowledge. In particular, experimental results in academic papers are often recorded in tabular form, making systematic extraction and reconstruction of this tabular information crucial for acquiring and summarizing knowledge.

[0003] Therefore, table structure recognition (TSR) is one of the most challenging tasks in information extraction from academic papers. It attempts to represent structured tables in a unified format so that the table information can be automatically extracted and applied by computers.

[0004] Currently, there are two main solutions for obtaining table text from academic papers in PDF or image format: one is to upload the papers in batches to online software or similar small software, and the software will automatically process it and return structured text, in which the table part is converted into a string of text; the other is to call an open source API interface, such as installing Python modules PyPDF2, pdfplumber, etc. to automatically extract table content from PDF papers.

[0005] In addition, current solutions that only handle TSR tasks, such as GraphTSR, can handle a single TSR task based on some rules, such as recognizing table lines or machine learning (graph convolutional network (GCN)).

[0006] However, in the above-mentioned solutions that use online software or call open source API interfaces, there is usually a problem of inconsistency between the text order and the table, which leads to confusion between the content of different columns or rows in the table. Therefore, these solutions can often only recognize the text in the table, but cannot reconstruct the cells and cannot achieve the complete table structuring process. As for solutions that only handle TSR tasks, on the one hand, calling the table structure recognition algorithm alone usually only supports word blocks as input and cannot use original academic papers (such as images or PDF formats) as input. On the other hand, the current solutions have poor recognition accuracy for complex tables. On the other hand, because the TSR algorithm does not introduce a pre-trained language model that incorporates prior knowledge, it cannot yet support semantic analysis of text in tables, resulting in insufficient TSR accuracy. Summary of the Invention

[0007] To address at least one of the aforementioned issues, the present invention provides a solution for extracting structured tables from images. This solution constructs multi-dimensional features for the table area in the image and uses a deep neural network model to predict the row and / or column probabilities of the word blocks in the table, thereby reconstructing the table in the image into a structured table. The present invention provides a comprehensive technical solution for reconstructing structured tables from original images.

[0008] According to one aspect of the present invention, a method for processing a table is provided. The method comprises: using an object detection model to extract one or more table sub-images from an image, wherein each table sub-image includes a table; performing optical character recognition on each table sub-image to detect multiple character blocks in the table sub-image, wherein each character block includes one or more characters; using a deep neural network model to predict the row probability that any two of the multiple character blocks are in the same row and the column probability that any two of the multiple character blocks are in the same column; and structurally reorganizing the multiple character blocks based on the row probability that any two of the multiple character blocks are in the same row and the column probability that any two of the multiple character blocks are in the same column to reconstruct the table into a structured table.

[0009] According to another aspect of the present invention, a computing device is provided. The computing device includes: at least one processor; and at least one memory, the at least one memory being coupled to the at least one processor and storing instructions for execution by the at least one processor, wherein when the instructions are executed by the at least one processor, the computing device performs the steps according to the above method.

[0010] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program code is stored. When the computer program code is executed, the method described above is executed.

[0011] In some embodiments, the deep neural network model comprises an input layer, a BioBERT network layer, a first fusion vector layer, a GCN network layer, a second fusion vector layer, a fully connected network layer, and an output layer, wherein predicting, by the deep neural network model, a row probability that any two of the plurality of word blocks are in a same row and a column probability that the any two of the plurality of word blocks are in a same column comprises: at the input layer, determining input data of the deep neural network model for a first word block and a second word block of the two word blocks to be predicted, wherein the input data comprises a first text ID of the first word block, a first position vector, a second text ID of the second word block, a second position vector, a relative position vector between the first word block and the second word block, and an adjacency matrix and a weight matrix of the table subgraph; at the BioBERT network layer, determining a first feature vector of the first word block and a second feature vector of the second word block based on the first text ID of the first word block and the second text ID of the second word block, respectively; at the first fusion vector layer, concatenating the first position vector and the first feature vector of the first word block to generate a first fusion vector of the first word block, and concatenating the second position vector and the second feature vector of the second word block to generate a second fusion vector of the second word block; at the GCN network layer, determining a first convolutional output vector of the first word block and a second convolutional output vector of the second word block based on the first fusion vector of the first word block, the second fusion vector of the second word block, and the adjacency matrix and the weight matrix of the table subgraph, respectively; at the second fusion vector layer, concatenating the relative position vector between the first word block and the second word block, the first fusion vector and the first convolutional output vector of the first word block, and the second fusion vector and the second convolutional output vector of the second word block to determine a fusion feature vector of the first word block and the second word block; at the fully connected network layer, predicting, by a first fully connected network, a row probability that the first word block and the second word block are in a same row based on the fusion feature vector of the first word block and the second word block, and predicting, by a second fully connected network, a column probability that the first word block and the second word block are in a same column based on the fusion feature vector; and at the output layer, outputting the row probability and the column probability of the first word block and the second word block.

[0012] In some embodiments, determining the input data of the deep neural network model for the first block and the second block of the two blocks to be predicted includes: converting the text of the first block and the second block into the first text ID and the second text ID, respectively; obtaining the first position vector of the first block and the second position vector of the second block based on the position information of the first block and the second block, respectively; determining the relative position vector between the first block and the second block based on the first position vector of the first block and the second position vector of the second block; determining the adjacency matrix of the table subgraph based on the distance between the multiple blocks of the table subgraph; and determining the weight matrix of the table subgraph based on the adjacency matrix.

[0013] In some embodiments, obtaining the first position vector of the first character block and the second position vector of the second character block based on the position information of the first character block and the second character block respectively includes: determining the normalized coordinate information of the first character block based on the position information of the first character block, determining the normalized center position of the first character block and the normalized width and normalized height of the first character block based on the normalized coordinate information of the first character block, and determining the first position vector of the first character block based on the normalized coordinate information, normalized center position, normalized width and normalized height of the first character block, and determining the normalized coordinate information of the second character block based on the position information of the second character block, determining the normalized center position of the second character block and the normalized width and normalized height of the second character block based on the normalized coordinate information of the second character block, and determining the second position vector of the second character block based on the normalized coordinate information, normalized center position, normalized width and normalized height of the second character block.

[0014] In some embodiments, structurally reorganizing the plurality of character blocks based on row probabilities that any two character blocks among the plurality of character blocks are in the same row and column probabilities that any two character blocks among the plurality of character blocks are in the same column to reconstruct the table into a structured table includes: for each character block among the plurality of character blocks, determining the number of rows and columns of the structured table based on the row probability that the character block is in the same row and the column probability that the character block is in the same column as other character blocks among the plurality of character blocks and the positional relationship between the character block and the other character blocks; determining the boundaries of each candidate cell in the structured table based on the position information of each character block among the plurality of character blocks and the number of rows and columns of the structured table; determining whether two adjacent candidate cells in the structured table should be merged based on the row probability and the column probability between the character blocks contained in the two adjacent candidate cells; merging the two adjacent candidate cells into one cell in response to determining that the two adjacent candidate cells should be merged; determining the two adjacent candidate cells as two separate cells in response to determining that the two adjacent candidate cells should not be merged; and merging the character blocks contained in each cell based on the position information of the plurality of character blocks to reconstruct the structured table.

[0015] In some embodiments, determining the number of rows and columns of the structured table includes: for each target block among the multiple blocks, determining a candidate right block set, a candidate left block set, a candidate upper block set, and a candidate lower block set of the target block; determining the right block, left block, upper block, and lower block of the target block based on the position information of each block in the candidate right block set, candidate left block set, candidate upper block set, and candidate lower block set of the target block and the position information of the target block; determining the right block, left block, upper block, and lower block of the target block based on the position information of each block in the multiple blocks The right block determines the rightmost block set of the table subgraph; for each rightmost block in the rightmost block set, determines the number of its left blocks; determines the number of columns of the structured table based on the number of left blocks of each rightmost block in the rightmost block set; determines the bottommost block set of the table subgraph based on the bottom block of each block in the multiple blocks; for each bottommost block in the bottommost block set, determines the number of its upper blocks; and determines the number of rows of the structured table based on the number of upper blocks of each bottommost block in the bottommost block set.

[0016] In some embodiments, determining the boundaries of each candidate cell in the structured table includes: constructing a row feature vector of the character block based on the upper boundary coordinates, lower boundary coordinates and center position vertical coordinates of each character block; clustering the multiple character blocks based on the row feature vectors of the multiple character blocks and the number of rows of the structured table to determine multiple row categories of the multiple character blocks, wherein the number of the row categories is equal to the number of rows of the structured table; for each of the multiple row categories, determining the average center vertical coordinate of the character blocks included in the row category; sorting the multiple row categories according to the size of the average center vertical coordinate of the character blocks included in each row category; determining the average vertical coordinate of each row category after sorting; and determining the vertical coordinate of the row boundary between the two adjacent row categories based on the average vertical coordinates of the two adjacent row categories after sorting.

[0017] In some embodiments, determining the boundary of each candidate cell in the structured table includes: constructing a column feature vector of the character block based on the left boundary coordinate, right boundary coordinate and center position horizontal coordinate of each character block; clustering the multiple character blocks based on the column feature vectors of the multiple character blocks and the number of columns of the structured table to determine multiple column categories of the multiple character blocks, wherein the number of column categories is equal to the number of columns of the structured table; for each column category in the multiple column categories, determining the average center horizontal coordinate of the character blocks included in the column category; sorting the multiple column categories according to the size of the average center horizontal coordinate of the character blocks included in each column category; determining the average horizontal coordinate of each column category after sorting; and determining the horizontal coordinate of the column boundary between the two adjacent column categories based on the average horizontal coordinates of the two adjacent column categories after sorting.

[0018] In some embodiments, the two adjacent candidate cells are located in the same column and include a first candidate cell located in the i-th row and a second candidate cell located in the i+1-th row, and determining whether the two adjacent candidate cells should be merged includes: determining a third block set excluding the first candidate cell in the i-th row; determining a fourth block set excluding the second candidate cell in the i+1-th row; determining a row merge value between the first candidate cell and the second candidate cell based on the row probability between any block in the i-th row and any block in the i+1-th row, the number of blocks included in the first candidate cell, the number of blocks included in the second candidate cell, the number of blocks included in the third block set, and the number of blocks included in the fourth block set; determining whether the row merge value is greater than a predetermined threshold; in response to determining that the row merge value is greater than the predetermined threshold, determining that the first candidate cell and the second candidate cell should be merged; and in response to determining that the row merge value is less than or equal to the predetermined threshold, determining that the first candidate cell and the second candidate cell should not be merged.

[0019] In some embodiments, the two adjacent candidate cells are located in the same row and include a first candidate cell located in the jth column and a second candidate cell located in the j+1th column, and determining whether the two adjacent candidate cells should be merged includes: determining a fifth block set in the jth column excluding the first candidate cell; determining a sixth block set in the j+1th column excluding the second candidate cell; determining a column merge value between the first candidate cell and the second candidate cell based on the column probability between any block in the jth column and any block in the j+1th column, the number of blocks included in the first candidate cell, the number of blocks included in the second candidate cell, the number of blocks included in the fifth block set, and the number of blocks included in the sixth block set; determining whether the column merge value is greater than a predetermined threshold; in response to determining that the column merge value is greater than the predetermined threshold, determining that the first candidate cell and the second candidate cell should be merged; and in response to determining that the column merge value is less than or equal to the predetermined threshold, determining that the first candidate cell and the second candidate cell should not be merged.

[0020] In some embodiments, the method further includes: obtaining a training data set for the deep neural network model, the training data set including a plurality of training data, each training data including information of a plurality of training blocks contained in a training table corresponding to the training data, the information of each training block including the characters contained in the training block, the coordinate position of the training block, and the row and column information of the training block in the training table; determining, at the input layer of the deep neural network model, the training input data of the deep neural network model for the first training block and the second training block of the two training blocks in the training table, wherein the training input data includes the text ID of the first training block and position vector, the text ID and position vector of the second training block, the relative position vector between the first training block and the second training block, and the adjacency matrix and weight matrix of the training table; in the BioBERT network layer of the deep neural network model, the feature vector of the first training block and the feature vector of the second training block are respectively determined based on the text ID of the first training block and the text ID of the second training block; in the first fusion vector layer of the deep neural network model, the position vector and the feature vector of the first training block are spliced ​​to generate a fusion vector of the first training block, and the position vector and the feature vector of the second training block are fused to generate a fusion vector of the second training block; in the GCN network layer of the deep neural network model, based on the fusion vector of the first training block, the fusion vector of the second training block, and the adjacency matrix and weight matrix of the training table, respectively determine the convolution output vector of the first training block and the convolution output vector of the second training block; in the second fusion vector layer of the deep neural network model, the relative position vector between the first training block and the second training block, the fusion vector and convolution output vector of the first training block, and the fusion vector and convolution output vector of the second training block are spliced ​​to determine the fusion feature of the first training block and the second training block vector; in the fully connected network layer of the deep neural network model, determining the row probability that the first training block and the second training block are in the same row based on the fused feature vector of the first training block and the second training block and the first fully connected network, and determining the column probability that the first training block and the second training block are in the same column based on the fused feature vector of the first training block and the second training block and the second fully connected network; performing softmax regression on the row probability that the first training block and the second training block are in the same row and the column probability that they are in the same column, and determining the row loss value and column loss value of the first training block and the second training block using the cross entropy loss function;and determining a loss value of the deep neural network model based on the row loss values ​​and column loss values ​​of the first training word block and the second training word block, and updating a parameter matrix of the deep neural network model based on the loss value. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention will be better understood and other objects, details, features and advantages of the present invention will become more apparent through the following description of specific embodiments of the present invention given with reference to the accompanying drawings.

[0022] Figure 1 A schematic diagram of a system for implementing a method for processing a table according to an embodiment of the present invention is shown.

[0023] Figure 2 A flow chart illustrating a method for processing a form according to some embodiments of the present invention is shown.

[0024] Figure 3 A structural diagram of the Mask R-CNN model according to an embodiment of the present invention is shown.

[0025] Figure 4 A schematic diagram of a table subgraph captured according to an embodiment of the present invention is shown.

[0026] Figure 5 A schematic structural diagram of a deep neural network model according to an embodiment of the present invention is shown.

[0027] Figure 6 The flowchart shows the steps of determining the row probability that two word blocks are in the same row and the column probability that two word blocks are in the same column according to an embodiment of the present invention.

[0028] Figure 7 A flowchart of sub-steps for determining input data for a deep neural network model according to an embodiment of the present invention is shown.

[0029] Figure 8 A flowchart illustrating steps for rebuilding a structured table according to an embodiment of the present invention is shown.

[0030] Figure 9 A flowchart showing the sub-steps of determining the number of rows and columns of a structured table according to an embodiment of the present invention is shown.

[0031] Figure 10 A flowchart of a process of determining row boundaries between candidate cells according to an embodiment of the present invention is shown.

[0032] Figure 11 A flowchart of a process of determining column boundaries between candidate cells according to an embodiment of the present invention is shown.

[0033] Figure 12 A flowchart of a process for determining whether two adjacent candidate cells should be merged is shown according to an embodiment of the present invention.

[0034] Figure 13 A flowchart of a process for determining whether two adjacent candidate cells should be merged into columns according to an embodiment of the present invention is shown.

[0035] Figure 14 A flowchart showing the steps of training a deep neural network model according to an embodiment of the present invention is shown.

[0036] Figure 15 A block diagram of a computing device suitable for implementing embodiments of the present invention is shown. DETAILED DESCRIPTION

[0037] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to make the present invention more thorough and complete and to fully convey the scope of the present invention to those skilled in the art.

[0038] In the following description, for the purpose of illustrating various invention embodiments, certain specific details are set forth to provide a thorough understanding of the various invention embodiments. However, those skilled in the relevant art will recognize that the embodiments may be practiced without one or more of these specific details. In other cases, well-known devices, structures, and techniques associated with the present application may not be shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0039] Unless the context requires otherwise, throughout the specification and claims, the word "comprise" and variations such as "include" and "have" should be construed in an open, inclusive sense, that is, should be interpreted to mean "including, but not limited to."

[0040] Reference throughout this specification to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of "in one embodiment" or "some embodiments" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any manner in one or more embodiments.

[0041] In addition, the terms "first", "second", "third", "fourth", etc. used in the specification and claims are only used to distinguish objects for the sake of clarity of description, and do not limit the size or other order of the objects they describe.

[0042] Figure 1 FIG. 1 is a schematic diagram showing a system 1 for implementing a method for processing a table according to an embodiment of the present invention. Figure 1 As shown in , the system 1 includes a computing device 10, a server 20 and a network 30. The computing device 10 and the server 20 can exchange data through the network 30. Here, the server 20 can be, for example, a server of a service provider dedicated to providing table reconstruction services, and the computing device 10 is connected to the server 20 to perform corresponding operations based on commands from the server 20. The computing device 10 may include at least one processor 110 and at least one memory 120 coupled to the at least one processor 110, the memory 120 storing instructions 130 executable by the at least one processor 110, and the instructions 130, when executed by the at least one processor 110, perform at least a part of the method 200 described below. Note that in this article, the computing device 10 can be part of the server 20 or can be independent of the server 20. The specific structure of the computing device 10 or the server 20 can be, for example, combined as follows Figure 15 As stated.

[0043] Figure 2 1 shows a flow chart of a method 200 for processing a table according to some embodiments of the present invention. The method 200 may be performed by, for example Figure 1 The following takes the execution in the computing device 10 as an example, combined with Figures 1 to 14 Method 200 is described.

[0044] like Figure 2 As shown in , the method 200 includes step 210, wherein the computing device 10 may use the object detection model to extract one or more table sub-images from the image to be detected, wherein each table sub-image includes a table.

[0045] An object detection model is a machine learning model that detects specific objects in an image. Depending on the application requirements, you can use either a two-stage object detection model (such as R-CNN (Region-based Convolutional Neural Networks), Fast R-CNN, Faster R-CNN, Mask R-CNN, etc.) or a one-stage object detection model (such as YOLO (You Only Look Once) and SSD (Single Shot Multibox Detector)).

[0046] For a single-stage target detection model, the model can be trained only for target objects of the desired table category, and the trained model is used only for detecting target objects of the table category.

[0047] For a two-stage target detection model, the model can be trained for target objects of multiple categories (including the table category), and the trained model can detect target objects of multiple categories.

[0048] In step 210, the computing device 10 can use the trained target detection model to crop one or more target regions from the picture to be detected, each target region containing a table. Of course, it is also possible that no target region is detected. In this case, it can be determined that there is no table in the picture, and the processing of the picture is skipped.

[0049] In this paper, the training and use of the target detection model are described by taking the Mask R-CNN model as an example. Figure 3 The structure of the Mask R-CNN model 300 according to an embodiment of the present application is shown. However, those skilled in the art can understand that the target detection model described in this paper is not limited to the Mask R-CNN model, but can use various target detection models described above or other conventional target detection algorithms.

[0050] Specifically, for example, the PubLayNet dataset can be used as the training data set of the target detection model, where the PubLayNet dataset is derived from PubMed and contains 360,000 document image layout analysis, all of which are academic paper related images and contain relevant annotation information. The annotation information mainly includes the position information and category of each target region, and the category includes, for example, the Text category, the Title category, the Table category, the Figure category and the List category, etc.

[0051] As shown in Figure 3 The model 300 can include a convolutional layer 310, an RPN (Region Proposal Network) layer 320, an ROI (Region of Interest) alignment layer 330, a target region prediction layer 340, a mask layer 350 and an output layer 360.

[0052] The convolution layer 310 can be implemented using, for example, a residual network (ResNet) to extract features from the input training image to generate a feature map of the image. For example, the convolution layer 310 can be implemented using a 101-layer residual network (ResNet-50) or a feature pyramid network (FPN).

[0053] The RPN layer 320 may, for example, use a fully convolutional network to detect the feature map from the convolutional layer 310 to generate a predicted target object region.

[0054] The ROI alignment layer 330 may align each region of interest of the feature map output by the convolution layer 310 based on the predicted target object region output by the RPN layer 320 to generate a ROI region.

[0055] The target region prediction layer 340 is used to classify and regress the regional features of the ROI region generated by the ROI alignment layer 330 to determine a predicted target region.

[0056] The mask layer 350 is used to perform object detection and segmentation on the ROI region generated by the ROI alignment layer 330 to determine a binary mask of the ROI region.

[0057] The output layer 360 determines the loss value of each ROI region based on the predicted target region output by the target region prediction layer 340 and the binary mask output by the mask layer 350 and iteratively updates the parameter matrix of the entire model 300 .

[0058] After model 300 is trained, in step 210, various types of ROI regions and their coordinate positions in the image can be determined from the image to be detected in the above manner, and the table-type ROI regions are identified as table sub-images and sequentially cut out according to the coordinate positions. Each table sub-image identified in this manner contains a table.

[0059] In some cases, the file to be detected is not in a picture (e.g., jpg) format. In this case, in step 210 or before step 210, the file to be detected may be converted into a picture format. For example, a common format for academic papers is PDF. In this case, the file in PDF format may be converted into a picture format by various file format conversion software (e.g., the pdf2image module in Python), wherein, when the file in PDF format contains multiple pages, each page may be converted into a separate picture.

[0060] continue Figure 2In method 200, in step 220, the computing device 10 may perform optical character recognition (OCR) on each table sub-image intercepted in step 210 to detect multiple character blocks in the table sub-image, where each character block includes one or more characters.

[0061] In step 220, various known or future developed OCR solutions can be used to detect the character blocks in the table sub-image. For example, in one embodiment, Google's open source OCR project Tesseract can be used to recognize the text in each table sub-image. Specifically, Tesseract can be run to perform optical character recognition on each table sub-image intercepted in step 210, and all characters therein can be recognized as multiple character blocks. In this case, the recognition result includes the characters contained in each character block and the coordinate position information of the character block in the image. Figure 4 FIG. 4 is a schematic diagram of a table subgraph 400 intercepted according to an embodiment of the present invention. Figure 4 As shown in , after performing OCR on the table sub-image, multiple word blocks in the table sub-image can be detected (in Figure 4 appears as a box around a letter or number).

[0062] Here, various existing or future developed OCR tools can be used to perform OCR operations on the table sub-image to detect the word blocks therein. For example, Figure 4 The table subgraph 400 shown in FIG is the result of an OCR operation using Google's open source OCR project Tesseract. Note that different OCR tools have different recognition strategies, and thus the resulting results may be presented in different forms, but this does not affect the scope of protection of the present invention.

[0063] Next, in step 230 , the computing device 10 may use a deep neural network model to predict the row probability that any two character blocks in the multiple character blocks in the table subgraph detected in step 220 are in the same row and the column probability that any two character blocks are in the same column.

[0064] Figure 5 FIG. 5 shows a schematic diagram of the structure of a deep neural network model 500 according to an embodiment of the present invention. Figure 5As shown in , the deep neural network model 500 may include an input layer 510, a BioBERT network layer 520, a first fusion vector layer 530, a GCN (Graph Convolutional Network) network layer 540, a second fusion vector layer 550, a fully connected network layer 560, and an output layer 570. The input layer 510 is used to determine or input the input data of the deep neural network model 500, which includes the text ID of each block in the table subgraph 400, the position vector, the relative position vector between two blocks, the adjacency matrix and the weight matrix between all blocks in the table subgraph. The BioBERT network layer 520 is used to take the text ID of each block as input and encode the characters in each block using the BioBERT framework to obtain the feature vector of the block. The first fusion vector layer 530 is used to fuse the position vector and feature vector of each block to obtain the fusion vector of the block. The GCN network layer 540 determines the convolution output vector of each block based on the fusion vector of each block, the adjacency matrix of the table subgraph, and the weight matrix. The second fusion vector layer 550 is used to fuse the fusion vectors, convolution output vectors, and relative position vectors of the two blocks to be predicted to determine the fusion feature vectors of the two blocks. The fully connected network layer 560 includes a first fully connected network 562 and a second fully connected network 564 for row prediction and column prediction, respectively. Based on the fusion feature vectors of the two blocks, the first and second fully connected networks 562 and 564 can predict the row probability and column probability of the two blocks being in the same row and column respectively.

[0065] The following combination Figure 6 and Figure 7 The operations of each layer of the deep neural network model 500 are described in detail. Figure 6 FIG2 is a flowchart showing step 230 of determining the row probability and column probability of two blocks being in the same row and the column probability of two blocks being in the same column according to an embodiment of the present invention. Note that the following description focuses only on the data processing process related to the two blocks A and B (also referred to as the first block and the second block) to be predicted in a prediction process, while omitting the data processing process related to other blocks ( Figure 5 The data processing process related to (represented by X).

[0066] like Figure 6 As shown in , step 230 may include sub-step 231, wherein at the input layer 510, the computing device 10 may determine input data of the deep neural network model 500 for the two character blocks A and B to be predicted. The input data may include a first text ID and a first position vector of the first character block A, a second text ID and a second position vector of the second character block B, a relative position vector between the first character block A and the second character block B, and an adjacency matrix and a weight matrix of the table subgraph 400.

[0067] Figure 7 A flowchart of sub-step 231 for determining input data of the deep neural network model 500 according to an embodiment of the present invention is shown.

[0068] like Figure 7 As shown in , sub-step 231 may include sub-step 2311, in which computing device 10 may convert the text of first block A and second block B into a first text ID and a second text ID, respectively. Sub-step 2311 may employ different conversion methods depending on the network structure of deep neural network model 500. In this context, since the next layer after input layer 510 is BioBERT network layer 520, sub-step 2311 may utilize a method compatible with BioBERT to convert the text in the blocks.

[0069] In sub-step 2312 , the computing device 10 may obtain a first position vector n(A) of the first word block A and a second position vector n(B) of the second word block B based on the position information of the first word block A and the second word block B, respectively.

[0070] In some embodiments herein, the position vector of each block may include three pieces of information: 1) the normalized coordinate information of the block in the image; 2) the normalized center position of the block; and 3) the normalized width and normalized height of the block.

[0071] Specifically, the computing device 10 can determine the normalized coordinate information of a character block based on the position information of the character block. For example, assuming that the position information of a character block can be expressed as absolute coordinates (x1, x2, y1, y2), where x2>x1 and y2>y1, it represents a rectangle composed of four vertex coordinates (x1, y1), (x1, y2), (x2, y1) and (x2, y2). The width and height of the image are W and H. Then the normalized coordinate information of the character block can be expressed as relative coordinates (x1', x2', y1', y2'), where x1'=x1 / W, x2'=x2 / W, y1'=y1 / H, y2'=y2 / H.

[0072] The computing device 10 can then determine the normalized center position ((x1'+x2') / 2, (y1'+y2') / 2), normalized width (x2'-x1') and normalized height (y2'-y1') of the character block based on the normalized coordinate information (x1', x2', y1', y2').

[0073] Based on the normalized coordinate information, normalized center position, normalized width, and normalized height of the character block, the computing device 10 may determine a position vector for the character block. For example, the position vector for the character block may be an 8-dimensional position vector formed by concatenating the normalized coordinate information of the character block in four dimensions (x1', x2', y1', y2'), the normalized center position in two dimensions ((x1'+x2') / 2, (y1'+y2') / 2), the normalized width in one dimension (x2'-x1'), and the normalized height in one dimension (y2'-y1').

[0074] For Figure 5 For the first character block A and the second character block B shown, the computing device 10 can respectively determine the position vector of the first character block A (referred to as the first position vector n(A)) and the position vector of the second character block B (referred to as the second position vector n(B)) in the same manner as described above.

[0075] continue Figure 7 In sub-step 2313, the computing device 10 may determine the relative position vector between the first word block A and the second word block B based on the first position vector n(A) of the first word block A and the second position vector n(B) of the second word block B.

[0076] Specifically, the computing device 10 may calculate the normalized center position (x A1 '+x A2 ') / 2,(y A1 '+y A2 ') / 2) and the normalized center position of the second block B ((x B1 '+x B2 ') / 2,(y B1 '+y B2 ') / 2) determine the relative position vector between the first block A and the second block B, the relative position vector w(AB) can be expressed as ((x B1 '+x B2 ') / 2-(x A1 '+x A2 ') / 2,(y B1 '+y B2 ') / 2-(y A1 '+y A2 ') / 2).

[0077] In sub-step 2314 , computing device 10 may determine an adjacency matrix for table sub-graph 400 based on distances between all word blocks of table sub-graph 400 .

[0078] Specifically, each block in table subgraph 400 is considered a node. For any node, the spatial distances between it and other nodes are calculated. An undirected edge is then constructed between each of the several (e.g., 10) nodes with the closest spatial distances to it, thereby generating an undirected graph for all blocks in table subgraph 400. Based on this undirected graph, an adjacency matrix can be determined for all blocks in table subgraph 400. In this adjacency matrix, the value of the element corresponding to two nodes with an undirected edge is 1, and the value of the element corresponding to two nodes with an undirected edge is 0. For example, for a table subgraph 400 containing Nt blocks, its adjacency matrix is ​​an Nt*Nt matrix.

[0079] Next, in sub-step 2315 , computing device 10 may determine a weight matrix for table subgraph 400 based on the adjacency matrix.

[0080] In some embodiments of the present invention, each weight value in the weight matrix may be determined by dividing the value of an element in the adjacency matrix by the sum of all elements in the row where the element is located.

[0081] continue Figure 6 In sub-step 232 of step 230, in the BioBERT network layer 520, the computing device 10 may determine a feature vector of the first character block A (hereinafter referred to as the first feature vector m(A)) and a feature vector of the second character block B (hereinafter referred to as the second feature vector m(B)) based on the first text ID of the first character block A and the second text ID of the second character block B, respectively.

[0082] Specifically, the computing device 10 may use the text ID of each character block as input and use the BioBERT framework to encode the characters in each character block to obtain a feature vector of the character block.

[0083] BERT is a context-based word representation model that is pre-trained based on a labeled language model using a bidirectional transformer. It uses a masked language model to predict random masked words in a sequence, making it possible to learn bidirectional representations. The BioBERT model has almost the same structure as the BERT model, except that it is pre-trained using a biomedical literature database and can be used for text mining in the biomedical field. In this article, the BioBERT network layer 520 uses a well-trained BioBERT model in the prior art, so its structure and training method will not be described in detail in this article.

[0084] Depending on the actual application requirements, different BioBERT model parameters can be selected. In one example, the size of the hidden layer of the selected BioBERT model is 768, so the size of the feature vector (embedding) generated for the input (word or sentence) of the BioBERT network layer 520 is 768 dimensions.

[0085] Next, in sub-step 233, in the first fusion vector layer 530, the computing device 10 may concatenate the first position vector n(A) and the first feature vector m(A) of the first word block A to generate a first fusion vector u(A) of the first word block A, and concatenate the second position vector n(B) and the second feature vector m(B) of the second word block B to generate a second fusion vector u(B) of the second word block B.

[0086] In one example, for the aforementioned 8-dimensional position vector and 768-dimensional feature vector, the dimension of the generated fusion vector is 776 dimensions.

[0087] In sub-step 234, in GCN network layer 540, computing device 10 may determine a first convolution output vector v(A) for first block A and a second convolution output vector v(B) for second block B, respectively, based on the first fused vector u(A) for first block A, the second fused vector u(B) for second block B, and the adjacency matrix and weight matrix of table subgraph 400. Here, GCN network layer 540 uses a trained GCN model from the prior art, and therefore its structure and training methods are not described in detail herein.

[0088] Different GCN model parameters can be selected depending on actual application requirements. In one example, in the matrix operation of the GCN network layer 540, the dimension of the selected weight matrix is ​​776*223, so that it can convert the 776-dimensional first fusion vector u(A) into the 223-dimensional first convolution output vector v(A), and convert the 776-dimensional second fusion vector u(B) into the 223-dimensional second convolution output vector v(B).

[0089] Through the GCN network layer 540 , information of other blocks except the two blocks A and B to be predicted in the table subgraph 400 is filtered out, thereby obtaining the convolution output vectors of the two blocks A and B to be predicted.

[0090] continue Figure 6In sub-step 235, in the second fusion vector layer 550, the computing device 10 may concatenate the relative position vector w(AB) between the first word block A and the second word block B, the first fusion vector u(A) and the first convolution output vector v(A) of the first word block A, and the second fusion vector u(B) and the second convolution output vector v(B) of the second word block B to determine the fusion feature vector of the first word block A and the second word block B.

[0091] In one example, for the above-mentioned 2-dimensional relative position vector w(AB), the 776-dimensional first fusion vector u(A) and the second fusion vector u(B), and the 223-dimensional first convolution output vector v(A) and the second convolution output vector v(B), the dimension of the obtained fusion feature vector is 2000 dimensions.

[0092] In sub-step 236, in the fully connected network layer 560, the computing device 10 can predict the row probability that the first word block A and the second word block B are in the same row based on the fused feature vector of the first word block A and the second word block B and the first fully connected network 562, and predict the column probability that the first word block A and the second word block B are in the same column based on the fused feature vector and the second fully connected network 564.

[0093] The first fully connected network 562 may be a fully connected network composed of multiple fully connected layers and activation functions. It can respectively determine the probability that the first block A and the second block B belong to the same row and the probability that they do not belong to the same row, and predict the probability that the first block A and the second block B belong to the same row as the row probability that the first block A and the second block B are in the same row. For example, assume that after the first block A and the second block B pass through the first fully connected network 562, a two-dimensional probability vector [0.2, 0.8] is output. This indicates that the probability that the first block A and the second block B belong to the same row is 0.2, and the probability that they do not belong to the same row is 0.8. Therefore, the row probability that the first block A and the second block B are in the same row is determined to be 0.2.

[0094] Similarly, the second fully-connected network 564 can also be a fully-connected network composed of multiple fully-connected layers and activation functions, which can respectively determine the probability values ​​of the first block A and the second block B belonging to the same column and the probability values ​​of not belonging to the same column, and predict the probability value of the first block A and the second block B belonging to the same column as the column probability of the first block A and the second block B being in the same column. For example, assume that the first block A and the second block B output a two-dimensional probability vector [0.66, 0.33] after passing through the second fully-connected network 564. This means that the probability value of the first block A and the second block B belonging to the same column is 0.66, and the probability value of not belonging to the same column is 0.33. Therefore, the column probability of the first block A and the second block B being in the same column is determined to be 0.66.

[0095] Finally, in sub-step 237 , at the output layer 570 , the computing device 10 may output the row probabilities and column probabilities of the first word block A and the second word block B for subsequent reconstruction of the structured table.

[0096] For the multiple blocks of the table sub-graph 400 detected in step 220, the following steps may be repeated for any two blocks: Figure 6 The operation of step 230 is to determine the row probability that any two word blocks are in the same row and the column probability that any two word blocks are in the same column.

[0097] continue Figure 2 In step 240, the computing device 10 may structurally reorganize the multiple word blocks in the table subgraph 400 based on the row probability that any two word blocks in the multiple word blocks are in the same row and the column probability that any two word blocks in the multiple word blocks are in the same column determined in step 230 to reconstruct the table in the table subgraph 400 into a structured table.

[0098] Figure 8 A flowchart of step 240 for rebuilding a structured table according to an embodiment of the present invention is shown.

[0099] like Figure 8 As shown in , step 240 may include sub-step 241, in which the computing device 10 may determine, for each of the multiple word blocks in the table subgraph 400, the number of rows and columns of the structured table based on the row probability of the word block being in the same row and the column probability of the word block being in the same column as other word blocks in the multiple word blocks in the table subgraph 400 and the positional relationship between the word block and the other word blocks.

[0100] Figure 9 The flowchart of the sub-step 241 of determining the number of rows and columns of the structured table according to an embodiment of the present invention is shown.

[0101] like Figure 9 As shown in , sub-step 241 may include sub-step 2411, in which the computing device 10 may determine, for each target word block in the multiple word blocks of the table subgraph 200, a candidate right word block set, a candidate left word block set, a candidate upper word block set, and a candidate lower word block set for the target word block.

[0102] Specifically, in some embodiments, computing device 10 may traverse all other blocks in table subgraph 400 except the block (also referred to as the target block), and select each block whose left boundary is larger than the right boundary of the target block from all other blocks as a candidate right block in the set of candidate right blocks for the target block. Furthermore, computing device 10 may filter out from these candidate right blocks the candidate right blocks whose row probability of being in the same row as the target block is greater than a predetermined threshold (e.g., 0.5) as the final set of candidate right blocks.

[0103] Similarly, the computing device 10 may also determine a candidate left block set, a candidate upper block set, and a candidate lower block set of the target block.

[0104] In sub-step 2412, the computing device 10 may determine the right block, left block, upper block and lower block of the target block based on the position information of each block in the candidate right block set, candidate left block set, candidate upper block set and candidate lower block set of the target block and the position information of the target block.

[0105] Here, the right block, left block, upper block and lower block of the target block refer to the right block, left block, upper block and lower block immediately adjacent to the target block.

[0106] Specifically, in some embodiments, the computing device 10 may select, from the set of candidate right blocks of the target block, a candidate right block with the smallest horizontal coordinate (i.e., furthest to the left) at its center position as the right block of the target block. For example, when block A is the target block, its right block may be labeled right(A).

[0107] Similarly, the computing device 10 may also determine the left block (left(A)), the upper block (up(A)), and the lower block (down(A)) of the target block.

[0108] In sub-step 2413 , the computing device 10 may determine a rightmost word block set of the table sub-graph 400 based on the right word block of each word block in the plurality of word blocks of the table sub-graph 400 .

[0109] Specifically, taking block A as an example, in some embodiments, it is determined whether the right block of block A, right(A), is empty. If right(A) is empty, block A is treated as a rightmost block in the rightmost block set of table subgraph 400.

[0110] Next, in sub-step 2414, the computing device 10 may determine the number of left blocks for each rightmost block in the rightmost block set of the table subgraph 400, and in sub-step 2415, determine the number of columns of the structured table based on the number of left blocks of each rightmost block in the rightmost block set.

[0111] Here, assume that the rightmost block set determined in sub-step 2413 is {x1, x2, ..., xn}, where xi (1≤i≤n, where n is a positive integer greater than 1) represents a rightmost block in the rightmost block set. For the rightmost block xi, recursively search for its left block left(xi) among all blocks in table subgraph 400, and determine the predicted column number number(xi) for the rightmost block xi.

[0112] Specifically, for the rightmost character block xi, it can be determined whether it has a left character block left(xi). If there is no left character block left(xi), number(xi) = 1; if there is a left character block left(xi), number(xi) is added by 1, and it is continued to be determined whether the left character block left(xi) still has a left character block left(left(xi)).

[0113] In this way, the predicted column number {number(x1), number(x2), ..., number(xn)} of each rightmost block in the rightmost block set {x1, x2, ..., xn} can be determined.

[0114] In one example, the number of columns ncol of the structured table can be determined as:

[0115] ncol=median(number(x1),number(x2),...,number(xn))

[0116] Among them, median() represents the median operation.

[0117] On the other hand, similar to sub-step 2413 , in sub-step 2416 , the computing device 10 may determine a lowermost character block set of the table sub-graph 400 based on the lower character block of each of the plurality of character blocks of the table sub-graph 400 .

[0118] Specifically, taking block A as an example, in some embodiments, it is determined whether the next block down(A) of block A is empty. If the next block down(A) of block A is empty, block A is used as a bottom block in the bottom block set of table subgraph 400.

[0119] In sub-step 2417, similar to sub-step 2414, the computing device 10 may determine the number of upper blocks for each lowermost block in the lowermost block set of the table sub-graph 400, and in sub-step 2418, similar to sub-step 2415, determine the number of rows of the structured table based on the number of upper blocks of each lowermost block in the lowermost block set.

[0120] Here, assume that the bottom block set determined in sub-step 2416 is {y1, y2, ..., ym}, where yi (1≤i≤m, where m is a positive integer greater than 1) represents a bottom block in the bottom block set. For bottom block yi, a recursive search is performed among all blocks in table subgraph 400 to find its upper block up(yi), and the predicted row number number(yi) for bottom block yi is determined.

[0121] Specifically, for the bottom character block yi, it can be determined whether there is an upper character block up(yi). If there is no upper character block up(yi), number(yi) = 1; if there is an upper character block up(yi), add 1 to number(yi), and continue to determine whether there is still an upper character block up(up(yi)) for the upper character block up(yi).

[0122] In this way, the predicted row number {number(y1), number(y2), ..., number(ym)} of each bottom block in the bottom block set {y1, y2, ..., ym} can be determined.

[0123] In one example, the number of rows nrow of the structured table can be determined as:

[0124] nrow=median(number(y1),number(y2),...,number(ym))

[0125] Among them, median() represents the median operation.

[0126] Note that although Figure 9 Sub-steps 2416 to 2418 are shown as being performed after sub-steps 2413 to 2415, but those skilled in the art will appreciate that the order of the sub-steps shown in the figure is merely schematic, and sub-steps 2416 to 2418 may be performed before sub-steps 2413 to 2415 or in parallel with sub-steps 2413 to 2415.

[0127] continue Figure 8 In sub-step 242, the computing device 10 can determine the boundaries of each candidate cell in the structured table based on the position information of each of the multiple blocks of the table sub-graph 400 and the number of rows and columns of the structured table determined in sub-step 241. As is well known to those skilled in the art, a cell is the intersection of rows and columns in a table. It is the smallest unit that makes up a table and can be split or merged. The input and modification of single data are all performed in cells. When using tools such as MS Word, Excel or WPS to generate a table, operations are usually performed in units of cells. The text data in each cell is used to represent independent semantics. Therefore, when reconstructing the structured table, the range / boundary of each cell should also be determined to obtain an accurate semantic representation of the table.

[0128] Determining the boundaries between cells in a structured table may include determining row boundaries and column boundaries between cells. Depending on the specific style and reconstruction requirements of the table, only row boundaries, only column boundaries, or both row and column boundaries may be determined. Since cells may be split or merged when generating a table, the range of cells determined by row boundaries and column boundaries between cells is only the smallest granularity of cells in the structured table, also referred to as candidate cells in the following description.

[0129] Figure 10 A flowchart of a process of determining row boundaries between candidate cells according to an embodiment of the present invention is shown.

[0130] like Figure 10 As shown in , determining the row boundary between candidate cells may include sub-step 2421, where the computing device 10 may construct a row feature vector of each character block based on the upper boundary coordinates, lower boundary coordinates and center position vertical coordinates of the character block.

[0131] As mentioned above, assuming that the normalized coordinate information of a character block is represented as (x1', x2', y1', y2'), its normalized center position is represented as ((x1'+x2') / 2, (y1'+y2') / 2). In this case, the row feature vector constructed for it can be expressed as (y1', y2', (y1'+y2') / 2).

[0132] Next, in sub-step 2422 , the computing device 10 may cluster the multiple word blocks of the table subgraph 400 based on their row feature vectors and the number of rows nrow of the structured table to determine multiple row categories of the word blocks.

[0133] Here, a clustering algorithm such as Kmean can be used to cluster these character blocks (referred to as row clustering). During row clustering, the number of row categories is set to nrow, the number of rows in the structured table. After performing row clustering on the multiple character blocks in table subgraph 400, nrow row categories are generated, each containing one or more character blocks.

[0134] In sub-step 2423 , the computing device 10 may determine, for each of the nrow row categories, the average central vertical coordinate of the character blocks included in the row category.

[0135] Here, for example, the average central ordinate of the character blocks included in the row category may be determined based on the central position ordinate in the row feature vector of each character block determined in sub-step 2421 and the character blocks included in the row category.

[0136] In sub-step 2424, the computing device 10 may sort the nrow row categories according to the average center vertical coordinate of the character blocks included in each row category determined in sub-step 2423. The sequence numbers of the sorted row categories may be represented as 1, 2, 3, ... nrow, for example.

[0137] Next, in sub-step 2425, computing device 10 may determine the average vertical coordinate of each sorted row category. For example, for any row category i (1≤i≤nrow), the average vertical coordinate of the center positions of all word blocks in the row category may be averaged to determine the average vertical coordinate of the row category.

[0138] In sub-step 2426 , the computing device 10 may determine the vertical coordinate of the row boundary between the two adjacent row categories based on the average vertical coordinate of the two adjacent row categories after sorting.

[0139] In one embodiment, the vertical coordinate y_axis of the row boundary between two adjacent row categories can be expressed as:

[0140] y_axis(line j )=(y j +y j+1 ) / 2

[0141] Among them, line j Indicates the row boundary between the jth row category and the j+1th row category, y j Indicates the average ordinate of the j-th row category, y j+1 Represents the average ordinate of the j+1th row category, where 1≤j≤nrow-1.

[0142] In this way, the vertical coordinate of the row boundary between two adjacent row categories can be determined, and the row boundary is the row boundary between the candidate cells.

[0143] Figure 11 1 shows a flow chart of a process for determining column boundaries between candidate cells according to an embodiment of the present invention. The method for determining column boundaries can be similar to the method for determining row boundaries described above, so reference is made to Figure 10 Described in a similar way.

[0144] like Figure 11 As shown in , determining the column boundaries between candidate cells may include sub-step 2421', where the computing device 10 may construct a column feature vector of each character block based on the left boundary coordinate, right boundary coordinate and center position horizontal coordinate of the character block.

[0145] As mentioned above, assuming that the normalized coordinate information of a character block is represented as (x1', x2', y1', y2'), and its normalized center position is represented as ((x1'+x2') / 2, (y1'+y2') / 2), in this case, the column feature vector constructed for it can be expressed as (x1', x2', (x1'+x2') / 2).

[0146] Next, in sub-step 2422 ′, the computing device 10 may cluster the multiple word blocks of the table subgraph 400 based on their column feature vectors and the number of columns ncol of the structured table to determine multiple column categories of the word blocks.

[0147] Here, a clustering algorithm such as Kmean can be used to cluster these word blocks (referred to as column clustering). During column clustering, the number of column categories is set to the number of columns in the structured table, ncol. After performing column clustering on the multiple word blocks in table subgraph 400, ncol column categories are generated, each containing one or more word blocks.

[0148] In sub-step 2423 ′, the computing device 10 may determine, for each of the ncol column categories, an average central horizontal coordinate of the character blocks included in the column category.

[0149] Here, for example, the average central horizontal coordinate of the character blocks included in the column category can be determined based on the central position horizontal coordinate in the column feature vector of each character block determined in sub-step 2421 ′ and the character blocks included in the column category.

[0150] In sub-step 2424', the computing device 10 may sort the ncol column categories according to the average central horizontal coordinate of the character blocks included in each column category determined in sub-step 2423. The sequence numbers of the sorted column categories may be represented as 1, 2, 3, ... ncol, for example.

[0151] Next, in sub-step 2425′, the computing device 10 may determine the average horizontal coordinate of each sorted column category. For example, for any column category i (1≤i≤ncol), the average horizontal coordinate of the center positions of all the blocks in that column category may be averaged to determine the average horizontal coordinate of that column category.

[0152] In sub-step 2426 ′, the computing device 10 may determine the abscissa of the column boundary between the two adjacent column categories based on the average abscissa of the two adjacent column categories after sorting.

[0153] In one embodiment, the horizontal coordinate x_axis of the column boundary between two adjacent column categories can be expressed as:

[0154] x_axis(line i)=(x i +x i+1 ) / 2

[0155] Among them, line i Indicates the column boundary between the i-th column category and the i+1-th column category, x i Represents the average horizontal coordinate of the i-th column category, x i+1 Represents the average abscissa of the x+1th column category, where 1≤i≤ncol-1.

[0156] In this way, the horizontal coordinate of the column dividing line between two adjacent column categories can be determined, and the column dividing line is the column dividing line between the candidate cells.

[0157] After determining the row boundaries and / or column boundaries between the candidate cells, the range of each candidate cell is determined, and the computing device 10 can match each character block with the corresponding candidate cell based on the normalized center position of the character block.

[0158] continue Figure 8 In sub-step 243 , the computing device 10 may determine whether the two adjacent candidate cells in the structured table should be merged based on the row probability and the column probability between the word blocks contained in the two adjacent candidate cells.

[0159] According to whether the two adjacent candidate cells are in the same row or the same column, it can be determined whether the two adjacent candidate cells should be merged in rows or in columns.

[0160] Figure 12 A flowchart of a process for determining whether two adjacent candidate cells should be merged is shown according to an embodiment of the present invention.

[0161] Assume that the two adjacent candidate cells include a first candidate cell M located in the i-th row and a second candidate cell N located in the i+1-th row, and assume that the two adjacent candidate cells are located in the same column.

[0162] like Figure 12 As shown in , the process for determining whether two adjacent candidate cells should be merged into a row may include sub-step 2431, in which the computing device 10 may determine a set of character blocks other than the first candidate cell M in the i-th row (hereinafter also referred to as the third character block set in order to distinguish it from other character block sets).

[0163] In sub-step 2432 , the computing device 10 may determine a block set (hereinafter also referred to as a fourth block set to distinguish it from other block sets) excluding the second candidate cell N in the (i+1)th row.

[0164] In sub-step 2433, the computing device 10 can determine the row merge value between the first candidate cell M and the second candidate cell N based on the row probability between any word block in the i-th row and any word block in the i+1-th row, the number of word blocks contained in the first candidate cell M, the number of word blocks contained in the second candidate cell N, the number of word blocks contained in the third word block set, and the number of word blocks contained in the fourth word block set.

[0165] Specifically, in one embodiment, assuming that the first candidate cell M contains m blocks {X1, X2, ..., Xm}, the second candidate cell N contains n blocks {Y1, Y2, ..., Yn}, the third block set contains a blocks {U1, U2, ..., Ua}, and the fourth block set contains b blocks {V1, V2, ..., Vb}, let prob_row(A, B) represent the row probability that any two blocks A and B predicted in step 230 are in the same row. Then, the row merge value score_row between the first candidate cell M and the second candidate cell N can be determined as follows:

[0166]

[0167] In sub-step 2434, the computing device 10 may determine whether the row merge value score_row is greater than a predetermined threshold, such as 0.5.

[0168] If it is determined that the row merge value is greater than the predetermined threshold, in sub-step 2435, the computing device 10 may determine that the first candidate cell M and the second candidate cell N should be merged; conversely, if it is determined that the row merge value is less than or equal to the predetermined threshold, in sub-step 2436, the computing device 10 may determine that the first candidate cell M and the second candidate cell N should not be merged.

[0169] Furthermore, the computing device 10 repeats the above process to further determine whether the merged cell can continue to be merged in a row. In addition, the computing device 10 traverses all adjacent candidate cells in the same manner to obtain the final row-merged cell.

[0170] Figure 13 FIG2 is a flowchart of a process for determining whether two adjacent candidate cells should be merged into columns according to an embodiment of the present invention. The method for determining whether to merge columns can be similar to the method for determining whether to merge rows described above, so reference is made to FIG2. Figure 12 Described in a similar way.

[0171] Assume that the two adjacent candidate cells include a first candidate cell M located in the j-th column and a second candidate cell N located in the j+1-th column, and assume that the two adjacent candidate cells are located in the same row.

[0172] like Figure 13 As shown in , the process for determining whether two adjacent candidate cells should be merged into columns may include sub-step 2431', in which the computing device 10 may determine a set of character blocks other than the first candidate cell M in the j-th column (hereinafter also referred to as the fifth character block set in order to distinguish it from other character block sets).

[0173] In sub-step 2432 ′, the computing device 10 may determine a word block set excluding the second candidate cell N in the j+1 th column (hereinafter also referred to as a sixth word block set to distinguish it from other word block sets).

[0174] In sub-step 2433', the computing device 10 may determine the column merge value between the first candidate cell M and the second candidate cell N based on the column probability between any word block in the j-th column and any word block in the j+1-th column, the number of word blocks contained in the first candidate cell M, the number of word blocks contained in the second candidate cell N, the number of word blocks contained in the fifth word block set, and the number of word blocks contained in the sixth word block set.

[0175] Specifically, in one embodiment, assuming that the first candidate cell M contains m blocks {X1, X2, ..., Xm}, the second candidate cell N contains n blocks {Y1, Y2, ..., Yn}, the fifth block set contains c blocks {U1, U2, ..., Uc}, and the sixth block set contains d blocks {V1, V2, ..., Vd}, let prob_col(A, B) represent the column probability that any two blocks A and B predicted in step 230 are in the same column. Then, the column merge value score_col between the first candidate cell M and the second candidate cell N can be determined as follows:

[0176]

[0177] In sub-step 2434', the computing device 10 may determine whether the column merge value score_col is greater than a predetermined threshold, such as 0.5.

[0178] If it is determined that the column merge value is greater than the predetermined threshold, in sub-step 2435', the computing device 10 may determine that the first candidate cell M and the second candidate cell N should be merged; conversely, if it is determined that the column merge value is less than or equal to the predetermined threshold, in sub-step 2436', the computing device 10 may determine that the first candidate cell M and the second candidate cell N should not be merged.

[0179] Furthermore, the computing device 10 repeats the above process to further determine whether the merged cell can continue to be merged into a column. In addition, the computing device 10 traverses all candidate cells adjacent to each other in the same manner to obtain the final cell to be merged into a column.

[0180] continue Figure 8 When it is determined in sub-step 243 that two adjacent candidate cells should be merged, in sub-step 244, the computing device 10 may merge the two adjacent candidate cells into one cell. Conversely, when it is determined in sub-step 243 that the two adjacent candidate cells should not be merged, in sub-step 245, the computing device 10 may determine the two adjacent candidate cells as two separate cells.

[0181] After performing the above operations on the candidate cells, actual cells of the structured table are obtained. Such cells are not candidate cells divided only by row boundaries and column boundaries, but actual cells that may contain more candidate cells.

[0182] Finally, in sub-step 246 , the computing device 10 may merge the character blocks contained in each cell based on the position information of the multiple character blocks to reconstruct the structured table.

[0183] At this point, the method 200 of the present invention uses the target detection model to capture the table sub-image in the image, detects multiple character blocks in the table sub-image through optical character recognition, and uses the deep neural network model to predict the row probability that any two character blocks are in the same row and the column probability that they are in the same column so as to perform structural reorganization of these character blocks to reconstruct the structured table.

[0184] In the above method, when predicting row probabilities and column probabilities, the trained deep neural network model 500 is used to perform the above predictions. In some embodiments of the present invention, the method 200 may further include a step 250 of training the deep neural network model 500.

[0185] From the perspective of the entire model, the relationship between the input and output of the deep neural network model 500 can be expressed as:

[0186] X out =W*X in +b

[0187] Where W is the weight function of the deep neural network model 500, and b is the bias function of the deep neural network model 500. The purpose of training the deep neural network model 500 is to continuously update its weight function W and bias function b (collectively referred to as parameter matrices) to convergence values. Here, the initial value of the parameter matrix can be set arbitrarily or based on experience.

[0188] Figure 14 FIG2 shows a flow chart of step 250 of training a deep neural network model 500 according to an embodiment of the present invention. The training process of the deep neural network model 500 is combined with the above Figure 6 The process of step 230 of determining row probabilities and column probabilities is basically the same, the main difference is that the input data and output data of the deep neural network model 500 are different, and step 250 also includes a process of iterating and updating the weight parameters of the deep neural network model 500 based on the output data.

[0189] like Figure 14 As shown in , step 250 may include sub-step 251, in which the computing device 10 may obtain a training data set for the deep neural network model 500. The training data set may include multiple training data, each training data includes information of multiple training blocks contained in a training table corresponding to the training data, and the information of each training block includes the characters contained in the training block, the coordinate position of the training block, and the row and column information of the training block in the training table.

[0190] In one example, the SciTSR dataset can be used as the training data set for the deep neural network model 500. The dataset contains 12,000 training data points, each of which corresponds to a training table (which further includes 2,885 complex tables). Each training table contains multiple character blocks (referred to herein as training blocks), and the information for each character block includes the characters contained in the character block, the coordinate position of the character block, and the row and column information of the character block in the training table. In addition, the dataset also includes test data from 3,000 test tables (which further include 716 complex tables) for testing the effectiveness of the trained model or further optimizing the trained model.

[0191] In sub-step 252, similar to sub-step 231 above, at the input layer 510 of the deep neural network model 500, the computing device 10 may determine the training input data of the deep neural network model 500 for the first training block and the second training block of the two training blocks in a training table. The training input data includes the text ID and position vector of the first training block, the text ID and position vector of the second training block, the relative position vector between the first training block and the second training block, and the adjacency matrix and weight matrix of the training table.

[0192] In sub-step 253, similar to the above sub-step 232, in the BioBERT network layer 520 of the deep neural network model 500, the computing device 10 can determine the feature vector of the first training block and the feature vector of the second training block based on the text ID of the first training block and the text ID of the second training block, respectively.

[0193] In sub-step 254, similar to the above sub-step 233, in the first fusion vector layer 530 of the deep neural network model 500, the computing device 10 may concatenate the position vector and feature vector of the first training block to generate a fusion vector of the first training block, and concatenate the position vector and feature vector of the second training block to generate a fusion vector of the second training block.

[0194] In sub-step 255, similar to the above sub-step 234, in the GCN network layer 540 of the deep neural network model 500, the computing device 10 can determine the convolution output vector of the first training block and the convolution output vector of the second training block based on the fusion vector of the first training block, the fusion vector of the second training block, and the adjacency matrix and weight matrix of the training table.

[0195] In sub-step 256, similar to the above sub-step 235, in the second fusion vector layer 550 of the deep neural network model 500, the computing device 10 may concatenate the relative position vector between the first training block and the second training block, the fusion vector and convolution output vector of the first training block, and the fusion vector and convolution output vector of the second training block to determine the fusion feature vector of the first training block and the second training block.

[0196] In sub-step 257, similar to the above sub-step 236, in the fully connected network layer 560 of the deep neural network model 500, the computing device 10 can determine the row probability that the first training block and the second training block are in the same row based on the fused feature vector of the first training block and the second training block and the first fully connected network 562, and determine the column probability that the first training block and the second training block are in the same column based on the fused feature vector of the first training block and the second training block and the second fully connected network 564.

[0197] Different from the above-mentioned step 230, in sub-step 258, the computing device 10 may perform Softmax regression on the row probability that the first training word block and the second training word block are in the same row and the column probability that they are in the same column, and use the cross-entropy loss function to determine the row loss value and column loss value of the first training word block and the second training word block.

[0198] Here, as described in sub-step 236 above, the outputs of the first fully connected network 562 and the second fully connected network 564 are each a two-dimensional probability vector. Therefore, in sub-step 258, the computing device 10 may perform softmax regression on each of these two-dimensional probability vectors. The results of the softmax regression may indicate the probability distribution of the first training block and the second training block being in the same row and column.

[0199] Based on Softmax regression, the output values ​​of the multi-classification are converted into a probability distribution in the range [0, 1] with a sum of 1. That is, the sum of the probabilities of two blocks being in the same row and not in the same row is 1. Similarly, the sum of the probabilities of two blocks being in the same column and not in the same column is also 1. The specific formula should be followed:

[0200]

[0201] Among them, x i indicates the input data, which is the two-dimensional probability vector generated in sub-step 257, θ indicates the parameter vector of the Softmax model, θ l Indicates the lth parameter in the parameter vector θ, where l = 1, 2, …, k; j represents x i The category (in this article, x i In the same row and not in the same row or in the same column and not in the same column), where j = 1, 2. Therefore, p(y i =j|x i ;θ) also represents the input data x i The probability distribution of being in the same row and not in the same row, or the probability distribution of being in the same column and not in the same column.

[0202] In one embodiment, the row loss value or column loss value calculated based on the cross entropy loss function Loss can be expressed as:

[0203]

[0204] in Represents the input data x in The probability that the sample label of the first training block and the second training block (here referring to the first training block and the second training block) is 1 (that is, the first training block and the second training block are in the same row or the same column).

[0205] In sub-step 259 , the computing device 10 may determine a loss value of the deep neural network model 500 based on the row loss values ​​and the column loss values ​​of the first training word block and the second training word block and update a parameter matrix of the deep neural network model 500 based on the loss value.

[0206] In some embodiments of the present invention, the loss value of the deep neural network model 500 is determined as the sum of the row loss value and the column loss value.

[0207] In some embodiments of the present invention, a back propagation algorithm is used to adjust the weight functions W of each layer of the deep neural network model 500. k and the bias function b k (k=1, 2, ... K), where K is the number of layers of the deep neural network model 500. Therefore, in sub-step 259, the loss value of the deep neural network model 500 and the weight function W of the Kth layer can be used to update. K and the bias function b K To determine the gradient value of the last layer (Kth layer) of the deep neural network model 500 and

[0208] The computing device 20 may then update the weight function of each of the multiple layers of the deep neural network model 500 based on the gradient value of the last layer of the deep neural network model 500.

[0209] For example, any one of the batch, mini-batch or stochastic gradient descent methods can be used to calculate the gradient value of the Kth layer of the deep neural network model 500. and Determine the gradient values ​​of the K-1th layer, the K-2th layer, ... the 1st layer in turn, and use the gradient value of each layer to calculate the weight function W of the layer k (and the bias function b k ) to update.

[0210] The above sub-step 259 is repeated based on a preset iteration step size until the maximum number of iterations is reached or the iteration stopping threshold is reached. At this point, the weight function W (and bias function b) of the deep neural network model 500 are trained to convergence values ​​and can be used to calculate the probability that two word blocks are in the same row or column.

[0211] Figure 15 FIG1 is a block diagram showing a structure of a computing device 1500 suitable for implementing an embodiment of the present invention. The computing device 1500 may be, for example, the computing device 10 or the server 20 described above.

[0212] like Figure 15As shown in FIG, the computing device 1500 may include one or more central processing units (CPUs) 1510 (only one of which is schematically shown in the figure), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 1520 or loaded from a storage unit 1580 into a random access memory (RAM) 1530. Various programs and data required for the operation of the computing device 1500 may also be stored in the RAM 1530. The CPU 1510, the ROM 1520, and the RAM 1530 are connected to each other via a bus 1540. An input / output (I / O) interface 1550 is also connected to the bus 1540.

[0213] Various components in computing device 1500 are connected to I / O interface 1550, including an input unit 1560, such as a keyboard and mouse; an output unit 1570, such as various types of displays and speakers; a storage unit 1580, such as a magnetic disk and optical disk; and a communication unit 1590, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1590 allows computing device 1500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0214] The method 200 described above may be executed, for example, by a CPU 1510 of a computing device 1500 (e.g., computing device 10 or server 20). For example, in some embodiments, the method 200 may be implemented as a computer software program tangibly included in a machine-readable medium, such as a storage unit 1580. In some embodiments, part or all of the computer program may be loaded and / or installed on the computing device 1500 via the ROM 1520 and / or the communication unit 1590. When the computer program is loaded into the RAM 1530 and executed by the CPU 1510, one or more operations of the method 200 described above may be performed. In addition, the communication unit 1590 may support wired or wireless communication capabilities.

[0215] Those skilled in the art will understand that Figure 15 The computing device 1500 shown is for illustration only. In some embodiments, the computing device 10 or the server 20 may include more or fewer components than the computing device 1500.

[0216] The above describes the method 200 for processing a form according to the present invention and the computing device 1500 that can be used as the computing device 10 or the server 20 in conjunction with the accompanying drawings. However, those skilled in the art will appreciate that the execution of the steps and sub-steps of the method 200 is not limited to the order shown in the drawings and described above, but can be executed in any other reasonable order. In addition, the computing device 1500 does not necessarily include Figure 15All components shown in the figure may only include some components necessary to perform the functions described in the present invention, and the connection method of these components is not limited to the form shown in the figure.

[0217] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0218] In one or more exemplary designs, the functions described herein may be implemented using hardware, software, firmware, or any combination thereof. For example, if implemented using software, the functions may be stored on or transmitted as one or more instructions or codes on a computer-readable medium.

[0219] The various units of the apparatus disclosed herein may be implemented using discrete hardware components or may be integrated into a single hardware component, such as a processor. For example, the various exemplary logic blocks, modules, and circuits described in conjunction with the present invention may be implemented or executed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof for performing the functions described herein.

[0220] Those skilled in the art should also understand that the various illustrative logical blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments of the present invention may be implemented as electronic hardware, computer software, or a combination of both.

[0221] The above description of the present invention is intended to enable any person skilled in the art to make or use the present invention. Various modifications of the present invention will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the examples and designs described herein, but is intended to be embodied in accordance with the widest scope of the principles and novel features disclosed herein.

Claims

1. A method for processing a form, comprising: Extract one or more table sub-images from the image using the object detection model, where each table sub-image includes a table; performing optical character recognition on each table sub-image to detect a plurality of character blocks in the table sub-image, wherein each character block includes one or more characters; Predicting, using a deep neural network model, a row probability that any two character blocks among the plurality of character blocks are in the same row and a column probability that any two character blocks are in the same column; as well as Based on the row probability that any two word blocks among the plurality of word blocks are in the same row and the column probability that any two word blocks are in the same column, the plurality of word blocks are structurally reorganized to reconstruct the table into a structured table. The deep neural network model includes an input layer, a BioBERT network layer, a first fusion vector layer, a GCN network layer, a second fusion vector layer, a fully connected network layer, and an output layer. The deep neural network model is used to predict the row probability that any two character blocks in the plurality of character blocks are in the same row and the column probability that any two character blocks are in the same column, including: At the input layer, determining input data of the deep neural network model for a first character block and a second character block of two character blocks to be predicted, wherein the input data includes a first text ID and a first position vector of the first character block, a second text ID and a second position vector of the second character block, a relative position vector between the first character block and the second character block, and an adjacency matrix and a weight matrix of the table subgraph; At the BioBERT network layer, determining a first feature vector of the first character block and a second feature vector of the second character block based on a first text ID of the first character block and a second text ID of the second character block, respectively; At the first fusion vector layer, concatenating the first position vector and the first feature vector of the first block to generate a first fusion vector for the first block, and concatenating the second position vector and the second feature vector of the second block to generate a second fusion vector for the second block; At the GCN network layer, based on the first fusion vector of the first word block, the second fusion vector of the second word block, and the adjacency matrix and weight matrix of the table subgraph, respectively determine a first convolution output vector of the first word block and a second convolution output vector of the second word block; At the second fusion vector layer, the relative position vector between the first block and the second block, the first fusion vector and the first convolution output vector of the first block, and the second fusion vector and the second convolution output vector of the second block are concatenated to determine a fusion feature vector of the first block and the second block; At the fully connected network layer, based on the fused feature vector of the first character block and the second character block and the first fully connected network, a row probability that the first character block and the second character block are in the same row is predicted, and based on the fused feature vector and the second fully connected network, a column probability that the first character block and the second character block are in the same column is predicted; and At the output layer, the row probabilities and the column probabilities of the first word block and the second word block are output.

2. The method of claim 1 , wherein determining input data of the deep neural network model for a first block and a second block of two blocks to be predicted comprises: Converting the text of the first character block and the second character block into the first text ID and the second text ID respectively; Obtaining a first position vector of the first character block and a second position vector of the second character block based on position information of the first character block and the second character block respectively; determining a relative position vector between the first word block and the second word block based on a first position vector of the first word block and a second position vector of the second word block; An adjacency matrix of the table subgraph is determined based on distances between the plurality of character blocks of the table subgraph; and a weight matrix of the table subgraph is determined based on the adjacency matrix.

3. The method of claim 2 , wherein obtaining a first position vector of the first word block and a second position vector of the second word block based on the position information of the first word block and the second word block respectively comprises: determining normalized coordinate information of the first character block based on the position information of the first character block, determining a normalized center position of the first character block and a normalized width and a normalized height of the first character block based on the normalized coordinate information of the first character block, and determining a first position vector of the first character block based on the normalized coordinate information, normalized center position, normalized width, and normalized height of the first character block, and Based on the position information of the second character block, the normalized coordinate information of the second character block is determined; based on the normalized coordinate information of the second character block, the normalized center position of the second character block and the normalized width and normalized height of the second character block are determined; and based on the normalized coordinate information, normalized center position, normalized width and normalized height of the second character block, the second position vector of the second character block is determined.

4. The method of claim 1 , wherein structurally reorganizing the plurality of word blocks based on a row probability that any two word blocks are in the same row and a column probability that any two word blocks are in the same column to reconstruct the table into a structured table comprises: For each character block in the plurality of character blocks, determining the number of rows and columns of the structured table based on a row probability that the character block is in the same row and a column probability that the character block is in the same column as other character blocks in the plurality of character blocks, and a positional relationship between the character block and the other character blocks; determining a boundary of each candidate cell in the structured table based on position information of each character block in the plurality of character blocks and the number of rows and columns of the structured table; determining whether two adjacent candidate cells in the structured table should be merged based on row probabilities and column probabilities between word blocks contained in the two adjacent candidate cells; In response to determining that the two adjacent candidate cells should be merged, merging the two adjacent candidate cells into one cell; In response to determining that the two adjacent candidate cells should not be merged, determining the two adjacent candidate cells as two separate cells; as well as The character blocks contained in each cell are merged based on the position information of the multiple character blocks to reconstruct the structured table.

5. The method of claim 4, wherein determining the number of rows and columns of the structured table comprises: For each target word block in the plurality of word blocks, determining a candidate right word block set, a candidate left word block set, a candidate upper word block set, and a candidate lower word block set for the target word block; determining a right block, a left block, an upper block, and a lower block of the target block based on the position information of each block in the candidate right block set, the candidate left block set, the candidate upper block set, and the candidate lower block set of the target block and the position information of the target block; determining a rightmost block set of the table subgraph based on the right block of each block in the plurality of blocks; and determining the number of left blocks of each rightmost block in the rightmost block set; determining the number of columns of the structured table based on the number of left blocks of each rightmost block in the rightmost block set; determining a bottom-most character block set of the table subgraph based on a bottom character block of each character block in the plurality of character blocks; For each bottom block in the bottom block set, determining the number of blocks above it; as well as The number of rows of the structured table is determined based on the number of upper character blocks of each lowermost character block in the lowermost character block set.

6. The method of claim 4, wherein determining the boundaries of each candidate cell in the structured table comprises: Constructing a row feature vector of each character block based on the upper boundary coordinate, the lower boundary coordinate and the center position ordinate of each character block; clustering the plurality of character blocks based on the row feature vectors of the plurality of character blocks and the number of rows of the structured table to determine a plurality of row categories of the plurality of character blocks, wherein the number of the row categories is equal to the number of rows of the structured table; For each row category of the plurality of row categories, determining an average central vertical coordinate of the character blocks included in the row category; sorting the plurality of row categories according to the size of the average central vertical coordinate of the character blocks included in each row category; Determine the average ordinate of each row category after sorting; as well as The ordinate of the row boundary between the two adjacent row categories is determined based on the average ordinate of the two adjacent row categories after sorting.

7. The method of claim 4, wherein determining the boundaries of each candidate cell in the structured table comprises: Constructing a column feature vector of each character block based on the left boundary coordinate, the right boundary coordinate and the horizontal coordinate of the center position of each character block; Clustering the plurality of character blocks based on the column feature vectors of the plurality of character blocks and the number of columns of the structured table to determine a plurality of column categories of the plurality of character blocks, wherein the number of the column categories is equal to the number of columns of the structured table; determining, for each of the plurality of column categories, an average central abscissa of the character blocks included in the column category; sorting the plurality of column categories according to the size of the average central horizontal coordinate of the character blocks included in each column category; Determine the average horizontal coordinate of each column category after sorting; as well as The abscissa of the column boundary between the two adjacent column categories is determined based on the average abscissa of the two adjacent column categories after sorting.

8. The method of claim 4 , wherein the two adjacent candidate cells are located in the same column and include a first candidate cell located in the i-th row and a second candidate cell located in the i+1-th row, and determining whether the two adjacent candidate cells should be merged comprises: Determining a third character block set in the i-th row excluding the first candidate cell; Determine a fourth block set in the (i+1)th row excluding the second candidate cell; determining a row merging value between the first candidate cell and the second candidate cell based on a row probability between any character block in the i-th row and any character block in the (i+1)-th row, the number of character blocks included in the first candidate cell, the number of character blocks included in the second candidate cell, the number of character blocks included in the third character block set, and the number of character blocks included in the fourth character block set; Determining whether the row merge value is greater than a predetermined threshold; In response to determining that the row merge value is greater than the predetermined threshold, determining that the first candidate cell and the second candidate cell should be merged; as well as In response to determining that the row merge value is less than or equal to the predetermined threshold, it is determined that the first candidate cell and the second candidate cell should not be merged.

9. The method of claim 4 , wherein the two adjacent candidate cells are located in the same row and include a first candidate cell located in the jth column and a second candidate cell located in the j+1th column, and determining whether the two adjacent candidate cells should be merged comprises: Determining a fifth word block set in the j-th column excluding the first candidate cell; Determine a sixth word block set in the j+1th column excluding the second candidate cell; determining a column merging value between the first candidate cell and the second candidate cell based on a column probability between any character block in the j-th column and any character block in the j+1-th column, the number of character blocks included in the first candidate cell, the number of character blocks included in the second candidate cell, the number of character blocks included in the fifth character block set, and the number of character blocks included in the sixth character block set; Determining whether the column merge value is greater than a predetermined threshold; In response to determining that the column merge value is greater than the predetermined threshold, determining that the first candidate cell and the second candidate cell should be merged; as well as In response to determining that the column merge value is less than or equal to the predetermined threshold, it is determined that the first candidate cell and the second candidate cell should not be merged.

10. The method of claim 1, further comprising: Obtaining a training data set for the deep neural network model, the training data set comprising a plurality of training data, each training data comprising information of a plurality of training blocks contained in a training table corresponding to the training data, the information of each training block comprising characters contained in the training block, coordinate positions of the training block, and row and column information of the training block in the training table; Determining, at an input layer of the deep neural network model, training input data for the deep neural network model for a first training block and a second training block of the two training blocks in the training table, wherein the training input data includes a text ID and a position vector of the first training block, a text ID and a position vector of the second training block, a relative position vector between the first training block and the second training block, and an adjacency matrix and a weight matrix of the training table; In the BioBERT network layer of the deep neural network model, determining a feature vector of the first training block and a feature vector of the second training block based on the text ID of the first training block and the text ID of the second training block respectively; In a first fusion vector layer of the deep neural network model, the position vector and the feature vector of the first training block are concatenated to generate a fusion vector of the first training block, and the position vector and the feature vector of the second training block are fused to generate a fusion vector of the second training block; In the GCN network layer of the deep neural network model, based on the fusion vector of the first training block, the fusion vector of the second training block, and the adjacency matrix and weight matrix of the table subgraph, respectively determine the convolution output vector of the first training block and the convolution output vector of the second training block; In a second fusion vector layer of the deep neural network model, the relative position vector between the first training block and the second training block, the fusion vector and the convolution output vector of the first training block, and the fusion vector and the convolution output vector of the second training block are concatenated to determine a fusion feature vector of the first training block and the second training block; In a fully connected network layer of the deep neural network model, a row probability that the first training block and the second training block are in the same row is determined based on the fused feature vector of the first training block and the second training block and the first fully connected network, and a column probability that the first training block and the second training block are in the same column is determined based on the fused feature vector of the first training block and the second training block and the second fully connected network; Performing Softmax regression on the row probabilities that the first training block and the second training block are in the same row and the column probabilities that they are in the same column, and determining row loss values ​​and column loss values ​​of the first training block and the second training block using a cross entropy loss function; as well as A loss value of the deep neural network model is determined based on the row loss values ​​and the column loss values ​​of the first training word block and the second training word block, and a parameter matrix of the deep neural network model is updated based on the loss value.

11. A computing device comprising: at least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions, when executed by the at least one processor, causing the computing device to perform the steps of the method according to any one of claims 1 to 10.

12. A computer-readable storage medium having computer program code stored thereon, wherein the computer program code, when executed, performs the method according to any one of claims 1 to 10.

Images