Table recognition method and device, electronic equipment and readable storage medium

Abstract

Embodiments of the present application provide a table recognition method and device, electronic equipment and readable storage medium, which are applied to the field of computer vision, and the method comprises the following steps: segmenting a table region from a target image to obtain a table image; using a pre-trained table recognition model to recognize the table structure and the position information of the cells in the table image; wherein the table recognition model comprises an encoding layer for extracting image features of the table image, and a decoding layer for generating the table structure; the input of the decoding layer is the input content of the encoding layer and the output content of the encoding layer; the text region of the table image and the text content in the text region are recognized; and an electronic spreadsheet file is generated based on the position information of the cells, the table structure, the text region and the text content in the text region. Through the present solution, the accuracy of table recognition can be improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a table recognition method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Artificial intelligence (AI) refers to the ability of engineered (i.e., designed and manufactured) systems to perceive their environment, and to acquire, process, apply, and represent knowledge. Table recognition is an important branch of AI in the field of image analysis and recognition, with the goal of acquiring and accessing data and other useful information.

[0003] Specifically, table recognition technology refers to reconstructing tables from images containing tables into spreadsheets, such as Excel spreadsheets, using certain image processing strategies.

[0004] However, tables are complex and diverse in style. Different tables have different row and column structures, different text types, different background fills, etc. In addition, the lighting environment in which the images containing tables were captured and the texture of the tables in the images vary greatly. All of these problems have made table recognition a difficult problem.

[0005] Therefore, improving the accuracy of table recognition is an urgent problem to be solved. Summary of the Invention

[0006] The purpose of this invention is to provide a table recognition method, apparatus, electronic device, and readable storage medium to improve the accuracy of table recognition. The specific technical solution is as follows:

[0007] In a first aspect, embodiments of the present invention provide a table recognition method, the method comprising:

[0008] From the target image, segment the table region to obtain the table image;

[0009] Using a pre-trained table recognition model, the table structure and cell position information in the table image are identified; wherein, the table recognition model includes an encoding layer for extracting image features of the table image, and a decoding layer for generating the table structure; the input of the decoding layer is: the input content of the encoding layer and the output content of the encoding layer;

[0010] Identify the text region of the table image and the text content within the text region;

[0011] A spreadsheet file is generated based on the cell location information, the table structure, the text area, and the text content in the text area.

[0012] Optionally, the decoding layer includes a first network layer, and a second and a third network layer located after and in parallel with the first network layer; the first network layer includes a self-attention mechanism transformer layer; the second network layer is used to identify the position of cells in the table image and includes two cross-shaped window self-attention mechanism Cswin transformer layers; the third network layer is used to predict the table structure in the table image and includes a self-attention mechanism transformer layer.

[0013] Optionally, the table structure is a structure represented using a hypertext language.

[0014] Optionally, segmenting the table region from the target image to obtain a table image includes:

[0015] Image features of the target image are extracted using a pre-trained image feature extraction network;

[0016] Based on the image features of the target image, candidate positions of tables in the target image are identified, and identification results for the candidate positions of the tables are obtained;

[0017] The target position of the table is obtained by removing overlapping results from the recognition results using a non-maximum suppression algorithm.

[0018] From the target image, extract the image region at the target location of the table to obtain the table image.

[0019] Optionally, identifying the text region of the table image and the text content within the text region includes:

[0020] The text regions in the table image are identified using a pre-trained text region recognition network and pre-extracted text region feature maps; wherein the text region recognition network is a differentiable binarization network DBNet; and the text region feature maps are feature maps processed by a convolutional block attention module.

[0021] The text region is subjected to text recognition to obtain the text content in the text region.

[0022] Optionally, the step of performing text recognition on the text region to obtain the text content in the text region includes:

[0023] The text content of the text region is identified using a pre-trained text recognition network; wherein the text recognition network is a network trained using a specified comprehensive loss; wherein the specified comprehensive loss is a comprehensive loss obtained by fusing temporal class classification loss and aggregated cross-entropy loss based on a neural network.

[0024] Optionally, generating a spreadsheet file based on the cell's location information, the table structure, the text area, and the text content within the text area includes:

[0025] Based on the cell location information, the table structure, and the text area, determine the position in the table structure corresponding to the text content in each text area;

[0026] The text content in each of the text regions is filled into the corresponding positions of the structural sequence to generate a table represented using hypertext language.

[0027] Optionally, the method further includes:

[0028] When a visual question-and-answer instruction for the target image is received, the target question in the visual question-and-answer instruction is obtained;

[0029] Determine the answer content corresponding to the target question.

[0030] Optionally, determining the answer content corresponding to the target question includes:

[0031] The image features of the target image, the position information of the cell, the text region, the text content in the text region, and the target question are input into a pre-trained visual question answering model to generate the answer content corresponding to the target question; wherein, the visual question answering model is a BERT pre-trained model trained using a sample table image, each text region in the sample table image, the text content of the sample table image, the image features of the sample table image, the input question, and the ground truth of the corresponding answer content.

[0032] Secondly, embodiments of the present invention provide a table recognition device, the device comprising:

[0033] The segmentation module is used to segment table regions from the target image to obtain a table image;

[0034] The first recognition module is used to recognize the table structure and cell position information in the table image using a pre-trained table recognition model; wherein, the table recognition model includes an encoding layer for extracting image features of the table image and a decoding layer for generating the table structure; the input of the decoding layer is: the input content of the encoding layer and the output content of the encoding layer;

[0035] The second recognition module is used to recognize the text area of ​​the table image and the text content in the text area;

[0036] The generation module is used to generate a spreadsheet file based on the cell location information, the table structure, the text area, and the text content in the text area.

[0037] Optionally, the decoding layer includes a first network layer, and a second and a third network layer located after and in parallel with the first network layer; the first network layer includes a self-attention mechanism transformer layer; the second network layer is used to identify the position of cells in the table image and includes two cross-shaped window self-attention mechanism Cswin transformer layers; the third network layer is used to predict the table structure in the table image and includes a self-attention mechanism transformer layer.

[0038] Optionally, the table structure is a structure represented using a hypertext language.

[0039] Optionally, the segmentation module includes:

[0040] The first extraction submodule is used to extract image features of the target image using a pre-trained image feature extraction network;

[0041] The first recognition submodule is used to identify candidate positions of tables in the target image based on the image features of the target image, and obtain recognition results for the candidate positions of the tables;

[0042] The removal submodule is used to remove overlapping results in the recognition results using a non-maximum suppression algorithm to obtain the target position of the table.

[0043] The second extraction submodule is used to extract the image region at the target location of the table from the target image to obtain the table image.

[0044] Optionally, the identification module includes:

[0045] The second recognition submodule is used to recognize the text regions in the table image using a pre-trained text region recognition network and pre-extracted text region feature maps; wherein, the text region recognition network is a differentiable binarization network DBNet; and the text region feature maps are feature maps processed by a convolutional block attention module.

[0046] The third recognition submodule is used to perform text recognition on the text region to obtain the text content in the text region.

[0047] Optionally, the third identification submodule is specifically used for:

[0048] The text content of the text region is identified using a pre-trained text recognition network; wherein the text recognition network is a network trained using a specified comprehensive loss; wherein the specified comprehensive loss is a comprehensive loss obtained by fusing temporal class classification loss and aggregated cross-entropy loss based on a neural network.

[0049] Optionally, the generation module includes:

[0050] The determination submodule is used to determine the position in the table structure corresponding to the text content in each text area based on the position information of the cell, the table structure, and the text area;

[0051] The generation submodule is used to fill the text content in each of the text regions into the corresponding structural sequence positions to generate a table represented using hypertext language.

[0052] Optionally, the device further includes:

[0053] The second acquisition module is used to acquire the target question in the visual question-and-answer instruction when a visual question-and-answer instruction for the target image is received;

[0054] The determination module is used to determine the answer content corresponding to the target question.

[0055] Optionally, the determining module is specifically used for:

[0056] The image features of the target image, the position information of the cell, the text region, the text content in the text region, and the target question are input into a pre-trained visual question answering model to generate the answer content corresponding to the target question; wherein, the visual question answering model is a BERT pre-trained model trained using a sample table image, each text region in the sample table image, the text content of the sample table image, the image features of the sample table image, the input question, and the ground truth of the corresponding answer content.

[0057] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0058] Memory, used to store computer programs;

[0059] When a processor executes a program stored in memory, it implements the steps of the table recognition method described above.

[0060] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described table recognition method.

[0061] This invention also provides a computer program product containing instructions that, when run on a computer, causes the computer to execute the above-described table recognition method.

[0062] Beneficial effects of the embodiments of the present invention:

[0063] The table recognition method provided in this invention first segments the table region from the target image to obtain a table image; then, using a pre-trained table recognition model, it identifies the table structure and cell position information in the table image; and identifies the text region and text content within the text region. Finally, based on the cell position information, table structure, text region, and text content, it generates an electronic spreadsheet file. In this scheme, the input to the decoding layer of the table structure recognition network is the sum of the input and output of the encoding layer, enabling the decoding layer to obtain sufficient information, thereby improving the accuracy of table structure recognition. Simultaneously, by separately identifying the cell position information, table structure, text region, and text content within the text region before generating the electronic spreadsheet file, the accuracy of table recognition is further improved.

[0064] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0066] Figure 1 A flowchart of the table recognition method provided in the embodiments of the present invention;

[0067] Figure 2 This is a flowchart illustrating the process of recognizing a table image in an embodiment of the present invention;

[0068] Figure 3 This is a schematic diagram of the structure recognized by the table recognition model provided in this embodiment of the invention;

[0069] Figure 4 This is a flowchart illustrating text region recognition and text content recognition in an embodiment of the present invention;

[0070] Figure 5 This is another flowchart of the table recognition method provided in an embodiment of the present invention;

[0071] Figure 6 This is a schematic diagram of the structure of the table recognition device provided in an embodiment of the present invention;

[0072] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0073] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0074] Currently, due to the complexity and diversity of table formats—including modern electronic spreadsheets and historical scanned and handwritten forms—and the significant differences in lighting conditions and textures during image acquisition, using artificial intelligence models for table recognition presents numerous challenges. For example, the models often involve a large number of image convolutional layers, resulting in high computational cost and low efficiency. Table recognition methods based on the self-attention mechanism (Terran) exhibit poor convergence. Furthermore, the models are insensitive to features in table images where the foreground and background colors are similar and the foreground texture is complex. These issues have made table recognition a persistent challenge in the field of image analysis and recognition within artificial intelligence.

[0075] To address the above problems, embodiments of the present invention provide a table recognition method, apparatus, electronic device, and readable storage medium. The method can be implemented in an electronic device. In specific applications, the electronic device can be various terminal devices, such as computers, servers, etc., and this disclosure is not limited thereto.

[0076] Specifically, the entity executing this table recognition method can be a table recognition device. For example, when the table recognition method is applied to a terminal device, the table recognition device can be an application running on the terminal device for performing table recognition. For example, when the table recognition method is applied to a server, the table recognition device can be a computer program running on the server, which can be used for table recognition.

[0077] The table recognition method provided in this embodiment of the invention may include the following steps:

[0078] From the target image, segment the table region to obtain the table image;

[0079] Using a pre-trained table recognition model, the table structure and cell position information in the table image are identified; wherein, the table recognition model includes an encoding layer for extracting image features of the table image, and a decoding layer for generating the table structure; the input of the decoding layer is: the input content of the encoding layer and the output content of the encoding layer;

[0080] Identify the text region of the table image and the text content within the text region;

[0081] A spreadsheet file is generated based on the cell location information, the table structure, the text area, and the text content in the text area.

[0082] The table recognition method provided in this invention first segments the table region from the target image to obtain a table image; then, using a pre-trained table recognition model, it identifies the table structure and cell position information in the table image; and identifies the text region and text content within the text region. Finally, based on the cell position information, table structure, text region, and text content, it generates an electronic spreadsheet file. In this scheme, the input to the decoding layer of the table structure recognition network is the sum of the input and output of the encoding layer, enabling the decoding layer to obtain sufficient information, thereby improving the accuracy of table structure recognition. Simultaneously, by separately identifying the cell position information, table structure, text region, and text content within the text region before generating the electronic spreadsheet file, the accuracy of table recognition is further improved.

[0083] The table recognition method provided by the embodiments of the present invention will be described below with reference to the accompanying drawings. Figure 1 As shown, the table recognition method provided in this embodiment of the invention may include the following steps:

[0084] S101, Segment the table region from the target image to obtain the table image;

[0085] The target image is an image whose content contains a table, such as an image containing an Excel spreadsheet. Furthermore, the target image can be acquired in various ways, such as receiving an image sent by an image acquisition device; obtaining the target image containing a table from locally stored images; or obtaining the target image containing a table from an external storage device, and so on.

[0086] Since the target image may contain non-table content, which can affect the extraction of table features and thus reduce the accuracy of table recognition, a better approach is to first identify the area containing the table within the target table, then segment the table area to obtain a table image. This table image can include the cell lines and the text content within the cells.

[0087] Optionally, in one implementation, segmenting the table region from the target image to obtain the table image may include steps A1-A4:

[0088] Step A1: Extract image features of the target image using a pre-trained image feature extraction network;

[0089] The extracted features can be line features, color features, etc. It is understood that when training an image feature extraction network, an image containing a table can be used as a sample set, and the location of the table (i.e., its position in the image) can be marked. Then, supervised training can be performed on this image feature extraction network to obtain the trained network. The method for marking the table's location can be: using coordinate information representing the table's position in the image, for example, using the coordinates of the center point and a combination of the table's height and width. Of course, this is not a limitation, and this embodiment of the invention does not impose specific limitations.

[0090] It should be noted that Mask R-CNN (Mask region-Convolutional Neural Networks) in related technologies utilizes ResNet (Deep residual network) and FPN (feature pyramid networks) to extract image features. However, its efficiency in extracting features from tables is not high. Therefore, in this embodiment, EfficietNet (a type of convolutional neural network) can be selected to extract image features from the target image. It has higher efficiency in extracting features from tables and can reduce the consumption of computing resources. EfficietNet is a network obtained by compound scaling the width, depth, and input resolution of the convolutional neural network. By balancing the three dimensions of width, depth, and input resolution of the convolutional neural network, the accuracy and efficiency of the convolutional neural network can be optimized.

[0091] Step A2: Based on the image features of the target image, identify the candidate positions of the table in the target image to obtain the identification results for the candidate positions of the table;

[0092] The location of a table can be identified by generating bounding boxes. Mask R-CNN, a related technology, uses a network structure with fully connected layers and Softmax (a classification network) layers. Based on image features, it uses bounding boxes to select and classify objects in the image. However, this method is prone to missed detections when applied to table recognition. Therefore, this embodiment of the invention uses a cascaded network to generate bounding boxes, which can reduce the number of missed detections.

[0093] In this context, a cascade network refers to a network that cascades multiple object detection models. An object detection model first uses a convolutional neural network to extract image features, generate candidate boxes, and then selects the target bounding box from these candidate boxes. During model training, candidate boxes are categorized as positive or negative samples based on their Intersection over Union (IOU) threshold with the ground truth. A cascade network, by cascading multiple object detection models, increases the IOU threshold layer by layer. For example, in one implementation, a cascade network cascades three object detection models, with IOU thresholds of 0.5, 0.6, and 0.7 for each layer. This ensures that each layer focuses on detecting candidate boxes within a certain IOU range, resulting in better recognition performance compared to a single-layer structure.

[0094] Since the embodiments of this invention are for table recognition, the generated recognition box can be limited to table recognition only. Of course, other types of recognition boxes can also be generated, and this invention does not impose specific limitations. In one implementation, the recognition box can be characterized by center point coordinates, length, and width. The recognition box that selects the table can be used as the recognition result for the candidate position of the table.

[0095] Furthermore, image features can be simultaneously input into the mask layer. The recognition result generated by the mask layer based on image features will classify the target image into tables and other categories in the form of a mask image. The mask layer can be a fully trained fully convolutional neural network used to identify the category of each pixel in the target image based on its image features. The categories are divided into "table" and "other," generating a mask image for the table. The mask layer can be trained by using an image containing tables as a sample set, and using the table mask image to mark the locations of pixels representing the table. Supervised training of the mask layer is then performed until convergence, resulting in a trained mask layer. The mask image refers to a specific image that occludes the image to be processed, distinguishing it into processing and non-processing areas. The mask image can contain only two colors. For example, in this embodiment, the pixel value of the pixel representing the table location in the mask image can be 255, and the pixel value of other locations can be 0.

[0096] The image feature extraction network and the network for identifying candidate table locations can be trained jointly during training. That is, during training, the output of the image feature extraction network is used as the input of the network for identifying candidate table locations. For example, an image containing a table can be used as a sample set, and the locations of the tables can be labeled. Supervised training can be performed on these two networks until convergence, resulting in the trained image feature extraction network and the network for identifying candidate table locations.

[0097] As can be seen, in this embodiment of the invention, the approach to identifying table positions is consistent with the approach to classifying objects in images using Mask R-CNN, that is, first extracting image features, and then using image features to generate recognition boxes and mask images. However, in this embodiment of the invention, the feature extraction network is improved to EfficietNet to improve the efficiency of extracting table features; the network for generating recognition boxes is improved to Cascade Network. Compared with related technologies, when using the Mask R-CNN improved by this solution to identify tables, the number of missed detections can be reduced, making the improved Mask R-CNN more suitable for table recognition.

[0098] Step A3: Using the nonmaximum suppression algorithm, remove overlapping results from the recognition results to obtain the target position of the table;

[0099] Since tables typically appear independently in an image and do not overlap, the results generated by using bounding boxes to identify tables often contain multiple overlapping bounding boxes. Therefore, non-maximum suppression algorithms can be used to remove redundant bounding boxes. Examples include NMS (Non-Maximum Suppression), softNMS, and matrix_NMS. NMS sorts the bounding boxes according to their confidence level (the probability that each box is a table bounding box), selects the bounding box with the highest confidence level as the reference bounding box, and then removes bounding boxes whose overlap with the reference bounding box reaches a predetermined threshold. SoftNMS uses linear or Gaussian weighting to process the overlap between each bounding box and the reference bounding box, obtaining a score for each bounding box, and then removes bounding boxes with scores below a predetermined threshold. Matrix_NMS uses a matrix to represent the overlap between bounding boxes, and uses this matrix for parallel computation to regenerate the confidence level of each bounding box, finally removing bounding boxes with confidence levels below a predetermined threshold. In this scheme, matrix_NMS is preferred, which avoids the problem of excessively deleting overlapping recognition boxes and has a higher computation speed. After removing the overlapping recognition boxes, the resulting recognition boxes can be used to represent the target positions of the table.

[0100] During implementation, the above steps A1-A3 can be performed as follows: Figure 2 As shown, the target image is input into EfficientNet to obtain image features. The image features are then input into the Cascade network and the mask layer in parallel. Finally, the Matrix_NMS algorithm is used to remove overlapping recognition boxes and output a mask image representing the table position, as well as recognition boxes. The recognition boxes are classified into two categories: table and other.

[0101] Step A4: Extract the image region at the target location of the table from the target image to obtain the table image.

[0102] In one implementation, the area selected by the table's bounding box in the target image can be segmented, that is, all pixels within the range selected by the table's bounding box are extracted to obtain the table image. Alternatively, a table mask image can be used to segment all pixels representing the table from the target image, that is, based on the table mask image, all pixels representing the table are determined and extracted to obtain the table image.

[0103] S102, using a pre-trained table recognition model, identify the table structure and cell position information in the table image; wherein, the table recognition model includes an encoding layer for extracting image features of the table image, and a decoding layer for generating the table structure; the input of the decoding layer is: the input content of the encoding layer and the output content of the encoding layer;

[0104] Encoding an image into an encoding layer and then inputting the encoded output into a decoding layer is a method used by artificial intelligence models to process images. In this embodiment, when using a table recognition model to recognize table structures, both the input and output of the encoding layer are input into the decoding layer. During input, the input and output of the encoding layer can be linearly added or concatenated. The input and output of the encoding layer have a complementary effect, allowing the decoding layer to obtain sufficient information, thus achieving good recognition results even for wireless table structures.

[0105] Cell location identification, i.e., identifying the position and size of the table, can be represented by a vector [x1, y1, w1, h1], where x1 and y1 represent the coordinates of the cell's center point, w1 represents the width, and h1 represents the height. The table structure can be represented using a hypertext language, for example... Declare table labels; declare rows; declare cells. Remove the tags indicating the start of the table or row, such as , , , etc. The remaining tags can correspond to the position of each cell.

[0106] In one implementation, the decoding layer includes a first network layer, and a second and a third network layer located after and in parallel with the first network layer; the first network layer includes a self-attention mechanism transformer layer; the second network layer is used to identify the position of cells in the table image and includes two cross-shaped window self-attention mechanism Cswin transformer layers; the third network layer is used to predict the table structure in the table image and includes a self-attention mechanism transformer layer.

[0107] The transformer is a deep learning model based on a self-attention mechanism. In related technologies, two transformer layers are used to identify the position of cells in a table image, which is computationally intensive and inefficient. However, in this embodiment, as shown... Figure 3As shown, using only one transformer layer for cell position recognition can effectively reduce computation while still achieving good recognition results. For this transformer layer, standard cross-entropy can be used to adjust its parameters until convergence. Furthermore, while related technologies use two transformer layers for table structure prediction, this embodiment utilizes two CSWIN transformer layers, which effectively extracts features in both horizontal and vertical directions, making it more suitable for table recognition tasks. The parameters of these two CSWIN transformer layers can be adjusted using L1 (Least Absolute Error) loss.

[0108] S103, Identify the text region of the table image and the text content in the text region;

[0109] In the implementation process, the text regions within the table image can be determined first. These text regions can be represented by a vector [x2, y2, w2, h2], where x2 and y2 represent the coordinates of the center point of the text region, w2 represents the width, and h2 represents the height. Then, the text content within these regions can be identified. Specifically, a pre-trained text region recognition network can be used for text region recognition, such as the differentiable binarization network DBNet; and a pre-trained text recognition network can be used to recognize the text content within the text regions, such as CRNN (Convolutional Recurrent Neural Network).

[0110] S104. Based on the cell location information, the table structure, the text area, and the text content in the text area, generate a spreadsheet file.

[0111] The spreadsheet file can be a spreadsheet file generated based on a hypertext language, such as an HTML file or an Excel file.

[0112] In one implementation, generating a spreadsheet file based on the cell location information, the table structure, the text area, and the text content within the text area may include steps B1-B2:

[0113] Step B1: Based on the cell location information, the table structure, and the text area, determine the position in the table structure corresponding to the text content in each text area;

[0114] The cell position information represents the position of each cell in the table image; the table structure represents the row and column distribution of each cell in the table; and the text area represents the position of the text content in the table image. In practical applications, the cell position information can be represented by the vector [x1, y1, w1, h1], where x1 and y1 represent the coordinates of the cell's center point, w1 represents the width, and h1 represents the height; the table structure can be represented using hypertext, meaning each cell corresponds to a specific hypertext marker; the text area can also be represented by the vector [x2, y2, w2, h2], where x2 and y2 represent the coordinates of the text area's center point, w2 represents the width, and h2 represents the height. Therefore, based on the text area and cell position information, the cell position information can be matched with the text area to further determine the cell in which the text content in each text area should be filled, which is the position in the hypertext to be filled.

[0115] Step B2: Fill the text content in each of the text regions into the corresponding positions of the structural sequence to generate a table represented using hypertext language.

[0116] Once the position of the text content in each text area to be filled into the corresponding hypertext field is determined, the text content of the text area is filled into the corresponding hypertext field to obtain the hypertext containing the text content. For example, if the position information of cell 1 in the first row and first column matches text area 1, and the position information of cell 2 in the first row and second column matches text area 2, and the text content in text area 1 is "name" and the text content in text area 2 is "age", then the following hypertext can be generated:

[0117]

[0118] The generation method for subsequent hypertext languages ​​can be deduced in the same way. If the hypertext language is HTML (HyperText Markup Language), after generating the hypertext language corresponding to all cells, an HTML format spreadsheet file can be directly generated.

[0119] The matching of cell position information and text area can be based on the following three rules: 1. Center point rule: If the center point of a text area is within the cell range represented by the cell position information, then the cell position information and text area are considered to match; 2. IOU (Intersection over Union) rule: If the first point is not satisfied, the cell with the highest overlap with the text area can be calculated as the matching item for the text area; 3. Distance principle: If neither of the above two points is satisfied, the distance between each cell and the text area is calculated, and the cell with the smallest distance is selected as the matching item for the text area.

[0120] The table recognition method provided in this invention first segments the table region from the target image to obtain a table image; then, using a pre-trained table recognition model, it identifies the table structure and cell position information in the table image; and identifies the text region and text content within the text region. Finally, based on the cell position information, table structure, text region, and text content, it generates an electronic spreadsheet file. In this scheme, the input to the decoding layer of the table structure recognition network is the sum of the input and output of the encoding layer, enabling the decoding layer to obtain sufficient information, thereby improving the accuracy of table structure recognition. Simultaneously, by separately identifying the cell position information, table structure, text region, and text content within the text region before generating the electronic spreadsheet file, the accuracy of table recognition is further improved.

[0121] Optionally, in another embodiment, identifying the text region of the table image and the text content within the text region may include steps C1-C2:

[0122] Step C1: Using a pre-trained text region recognition network and pre-extracted text region feature maps, identify the text regions in the table image; wherein, the text region recognition network is a differentiable binarization network DBNet; and the text region feature maps are feature maps processed by a convolutional block attention module.

[0123] DBNet typically utilizes a Feature Pyramid Network (FPN) structure to extract and process feature maps. Specifically, it first extracts the original feature map, then transforms it into four sub-feature maps of sizes 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original feature map. These four sub-feature maps are then upsampled to 1 / 4 the size of the original feature map, and finally concatenated. The concatenated feature map is then used to identify text regions in the image. In this embodiment, as shown... Figure 4As shown, the technical solution of this embodiment of the invention is based on the DBNet structure. After concatenation, a CBAM (Convolutional Block Attention Module) structure is added. That is, the feature map of the table image is first extracted and processed using the FPN structure in DBNet, and then input into CBAM to obtain the text region feature map. In this embodiment, the attention mechanism of CBAM is utilized, which can effectively extract text features when the background is complex and affects the foreground, or when the foreground features are not obvious.

[0124] Step C2: Perform text recognition on the text region to obtain the text content in the text region.

[0125] In one implementation, this step may include:

[0126] The text content of the text region is identified using a pre-trained text recognition network;

[0127] The text recognition network can be trained using a loss function. That is, the loss function is used to determine whether the text recognition network has converged. If it has not converged, the parameters of the text recognition network are continuously adjusted until the text recognition network converges.

[0128] For example, in one implementation, the text recognition network can be a network trained using a specified comprehensive loss; wherein the specified comprehensive loss is a comprehensive loss obtained by fusing the temporal class classification loss and the aggregated cross-entropy loss based on the neural network.

[0129] In one implementation, the specified comprehensive loss can be:

[0130] L A-CTC =L CTC +λ*L ACE ;

[0131] Among them, L A-CTC To specify the overall loss, λ is a preset parameter, L CTC For the temporal class classification loss based on neural networks, L ACE This represents the aggregated cross-entropy loss.

[0132] The text recognition network can be a CRNN (Convolutional Recurrent Neural Network). In related technologies, when using CRNN for text recognition, the temporal classification loss (CTC Loss) based on neural networks is used to remove erroneous spacing characters and repeated characters from the output text. However, CTC Loss can only handle flat, straight text and cannot handle curved text, artistic fonts, or other similar text. ACE Loss (Convergent Cross-Entropy Loss), on the other hand, can effectively handle the recognition of curved text, artistic fonts, and other similar text. Therefore, in this embodiment of the invention, the temporal classification loss and the convergent cross-entropy loss are fused, i.e., CTC Loss + ACE Loss are jointly trained, thereby enabling the text recognition network to handle multiple types of text.

[0133] In this embodiment, a pre-trained text region recognition network and pre-extracted text region feature maps are used to recognize text regions in a table image. The text region recognition network is a differentiable binarization network DBNet. The text region feature maps are feature maps processed by a convolutional block attention module. Then, text recognition is performed on the text regions to obtain the text content in the text regions. By combining the attention mechanism of the convolutional block attention module, it is possible to effectively recognize text in cases where the background is complex and affects the foreground, or where the foreground features are not obvious.

[0134] Optionally, in another embodiment, the method further includes steps D1-D2:

[0135] Step D1: When a visual question-and-answer instruction for the target image is received, the target question in the visual question-and-answer instruction is obtained;

[0136] In this embodiment of the invention, a visual question-and-answer function can be added to the target image. In one implementation, the user can input a target question in a designated dialog box, thereby generating the visual question-and-answer instruction. Furthermore, the visual question-and-answer instruction can also be for spreadsheet files; that is, it is reasonable to add visual question-and-answer functionality to the output spreadsheet file using the same method.

[0137] Step D2: Determine the answer content corresponding to the target question.

[0138] Tables may contain various statistical data. Retrieving a specific value from a table typically requires manual searching or keyword searches. When the value isn't directly recorded in the table, it needs to be calculated from data in multiple cells. For example, in a table recording the winners of past sports meets, finding the question "Which athlete won the most titles?" requires manually searching for relevant cells, counting their number, and comparing their values ​​to arrive at the answer. Therefore, automatically retrieving the answer to a question related to a target image or spreadsheet file would significantly improve table lookup efficiency.

[0139] In related technologies, BERT (Bidirectional Encoder Representation from Transformers, a pre-trained language representation model) is used to implement the question-answering function of tables. Pre-training refers to training the model with a large sample set for a learning task, continuously adjusting the model's parameters. Then, the obtained parameters are used in the next learning task. In this stage, the parameters are mainly fine-tuned, and a large number of samples are no longer needed to train the model, which can greatly improve the training efficiency of the model.

[0140] In one implementation, determining the answer content corresponding to the target question may include:

[0141] The image features of the target image, the position information of the cell, the text content in the text region, and the target question are input into a pre-trained visual question answering model to generate the answer content corresponding to the target question; wherein, the visual question answering model is a BERT pre-trained model trained using a sample table image, each text region in the sample table image, the text content of the sample table image, the image features of the sample table image, the input question, and the ground truth of the corresponding answer content.

[0142] In related technologies, for the implementation of table-based question answering, the BERT model is pre-trained by first using a large number of table samples to train the BERT model. During the pre-training process, the learning objective of the BERT model is to recover the text that has been masked in the table. During the fine-tuning process, the sample tables and pre-prepared sample questions are used as inputs to the BERT model, and the answers to the sample questions are used as ground truth values ​​to fine-tune the parameters of the BERT model until the BERT model converges, thus obtaining the trained BERT model.

[0143] In related technologies, the BERT model, when learning table-based question answering, only uses text content as embedding (embedding vector, i.e., the input of the BERT model). In this embodiment of the invention, information about each text region and the image features of that text region are added to the embedding. The text region can be represented by a vector (x,y,w,h), where x and y represent the coordinates of the center point of the text region, w represents the width, and h represents the height.

[0144] Thanks to the deep fusion of text content, image features, and text regions, the BERT model's understanding of tables can be enhanced, thereby improving the accuracy of determining the answer content corresponding to the target question.

[0145] Specifically, in one implementation, the BERT model of this invention can be pre-trained using the LayoutLMV2.0 (a multimodal document understanding pre-training model) method. That is, during the pre-training stage, multiple pairs of table images and labeled text regions and text content are prepared, and three self-supervised pre-training tasks are adopted:

[0146] 1. Cover the text in the text content and the corresponding areas in the table image, and require the model to restore the covered text in the text content.

[0147] 2. Cover a portion of the text row by row on the table image, and use a model to predict whether each word is covered.

[0148] 3. In the sample set, some of the originally matched images and texts are shuffled to construct negative samples of image-text mismatch, and the model is used to predict whether the images and text match.

[0149] The fine-tuning process takes the text content of multiple samples, each text region (x,y,w,h), image features, and pre-prepared questions as input to the BERT model, takes the answers to the questions as the ground truth, and fine-tunes the parameters of the BERT model until the BERT model converges, thus obtaining the trained BERT model.

[0150] In practical applications, the image features of the target image, the position information of the cells, the text region, and the text content in the text region can be input into a pre-trained visual question answering model to add visual question answering functionality to the target image. Then, when a visual question answering instruction for the target image is received, the answer corresponding to the target question can be generated.

[0151] In this embodiment, when a visual question-and-answer instruction for the spreadsheet file is received, the target question in the instruction is obtained, and the answer to the target question is determined. Therefore, in this embodiment, when a question-and-answer instruction for the spreadsheet file is received, the answer to the question can be automatically obtained, which can greatly improve the efficiency of spreadsheet lookup.

[0152] To facilitate understanding of the table method provided by this invention, the specific application of this method will be described below by example.

[0153] like Figure 5 As shown, after acquiring the target image, the table is first located, that is, the table region is segmented from the target image to obtain the table image; then the table structure is parsed, that is, the table structure and cell position information in the table image are identified; simultaneously, text recognition is performed, that is, the text region of the table image and the text content in the text region are identified; finally, the table structure and text region are merged, that is, the text content in each text region is filled into the position of the corresponding structure sequence, and the spreadsheet file is output; at the same time, the image features of the target image, the cell position information, the text content in the text region, and the target question can also be input into a pre-trained visual question answering model, thereby adding visual question answering function to the target image.

[0154] In this embodiment, the accuracy of table recognition is improved by recognizing the cell location information, table structure, text area, and text content in the text area, and then generating an electronic spreadsheet file. In addition, a visual question-and-answer function is added to the target image, which can greatly improve the efficiency of table query.

[0155] This invention also provides a table recognition device, such as... Figure 6 As shown, the device includes:

[0156] The segmentation module 610 is used to segment the table region from the target image to obtain a table image;

[0157] The first recognition module 620 is used to recognize the table structure and cell position information in the table image using a pre-trained table recognition model; wherein, the table recognition model includes an encoding layer for extracting image features of the table image and a decoding layer for generating the table structure; the input of the decoding layer is: the input content of the encoding layer and the output content of the encoding layer;

[0158] The second recognition module 630 is used to recognize the text area of ​​the table image and the text content in the text area;

[0159] The generation module 640 is used to generate a spreadsheet file based on the cell location information, the table structure, the text area, and the text content in the text area.

[0160] Optionally, the decoding layer includes a first network layer, and a second and a third network layer located after and in parallel with the first network layer; the first network layer includes a self-attention mechanism transformer layer; the second network layer is used to identify the position of cells in the table image and includes two cross-shaped window self-attention mechanism Cswin transformer layers; the third network layer is used to predict the table structure in the table image and includes a self-attention mechanism transformer layer.

[0161] Optionally, the table structure is a structure represented using a hypertext language.

[0162] Optionally, the segmentation module includes:

[0163] The first extraction submodule is used to extract image features of the target image using a pre-trained image feature extraction network;

[0164] The first recognition submodule is used to identify candidate positions of tables in the target image based on the image features of the target image, and obtain recognition results for the candidate positions of the tables;

[0165] The removal submodule is used to remove overlapping results in the recognition results using a non-maximum suppression algorithm to obtain the target position of the table.

[0166] The second extraction submodule is used to extract the image region at the target location of the table from the target image to obtain the table image.

[0167] Optionally, the identification module includes:

[0168] The second recognition submodule is used to recognize the text regions in the table image using a pre-trained text region recognition network and pre-extracted text region feature maps; wherein, the text region recognition network is a differentiable binarization network DBNet; and the text region feature maps are feature maps processed by a convolutional block attention module.

[0169] The third recognition submodule is used to perform text recognition on the text region to obtain the text content in the text region.

[0170] Optionally, the third identification submodule is specifically used for:

[0171] The text content of the text region is identified using a pre-trained text recognition network; wherein the text recognition network is a network trained using a specified comprehensive loss; wherein the specified comprehensive loss is a comprehensive loss obtained by fusing temporal class classification loss and aggregated cross-entropy loss based on a neural network.

[0172] Optionally, the generation module includes:

[0173] The determination submodule is used to determine the position in the table structure corresponding to the text content in each text area based on the position information of the cell, the table structure, and the text area;

[0174] The generation submodule is used to fill the text content in each of the text regions into the corresponding structural sequence positions to generate a table represented using hypertext language.

[0175] Optionally, the device further includes:

[0176] The second acquisition module is used to acquire the target question in the visual question-and-answer instruction when a visual question-and-answer instruction for the target image is received;

[0177] The determination module is used to determine the answer content corresponding to the target question.

[0178] Optionally, the determining module is specifically used for:

[0179] The image features of the target image, the position information of the cell, the text region, the text content in the text region, and the target question are input into a pre-trained visual question answering model to generate the answer content corresponding to the target question; wherein, the visual question answering model is a BERT pre-trained model trained using a sample table image, each text region in the sample table image, the text content of the sample table image, the image features of the sample table image, the input question, and the ground truth of the corresponding answer content.

[0180] This invention also provides an electronic device, such as... Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0181] Memory 703 is used to store computer programs;

[0182] The processor 701, when executing the program stored in the memory 703, implements the steps of the above-described table recognition method.

[0183] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0184] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0185] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0186] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0187] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of the table recognition method described above.

[0188] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of the table recognition method described in the above embodiments.

[0189] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0190] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0191] The various embodiments in this specification are described in a related manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0192] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A table recognition method, characterized in that, The method includes: From the target image, segment the table region to obtain the table image; Using a pre-trained table recognition model, the table structure and cell position information in the table image are identified; wherein, the table recognition model includes an encoding layer for extracting image features of the table image, and a decoding layer for generating the table structure; the input of the decoding layer is: the input content of the encoding layer and the output content of the encoding layer; the cell position information includes the center point coordinates, width, and height of the cell; Identify the text region of the table image and the text content within the text region; A spreadsheet file is generated based on the cell's location information, the table structure, the text area, and the text content within the text area; each cell in the spreadsheet file is filled with text content from a text area that matches the cell's location information. The decoding layer includes a first network layer, and a second and a third network layer located after and in parallel with the first network layer; the first network layer includes a self-attention mechanism transformer layer; the second network layer is used to identify the position of cells in the table image and includes two cross-shaped window self-attention mechanism cswintransformer layers; the third network layer is used to predict the table structure in the table image and includes a self-attention mechanism transformer layer.

2. The method according to claim 1, characterized in that, The table structure is represented using hypertext language.

3. The method according to claim 1 or 2, characterized in that, The step of segmenting the table region from the target image to obtain the table image includes: Image features of the target image are extracted using a pre-trained image feature extraction network; Based on the image features of the target image, candidate positions of tables in the target image are identified, and identification results for the candidate positions of the tables are obtained; The target position of the table is obtained by removing overlapping results from the recognition results using a non-maximum suppression algorithm. From the target image, extract the image region at the target location of the table to obtain the table image.

4. The method according to any one of claims 1-3, characterized in that, The process of identifying the text region of the table image and the text content within the text region includes: The text regions in the table image are identified using a pre-trained text region recognition network and pre-extracted text region feature maps; wherein the text region recognition network is a differentiable binarization network DBNet; and the text region feature maps are feature maps processed by a convolutional block attention module. The text region is subjected to text recognition to obtain the text content in the text region.

5. The method according to claim 4, characterized in that, The step of performing text recognition on the text region to obtain the text content in the text region includes: The text content of the text region is identified using a pre-trained text recognition network; wherein the text recognition network is a network trained using a specified comprehensive loss; wherein the specified comprehensive loss is a comprehensive loss obtained by fusing temporal class classification loss and aggregated cross-entropy loss based on a neural network.

6. The method according to any one of claims 1-5, characterized in that, The process of generating a spreadsheet file based on the cell location information, the table structure, the text area, and the text content within the text area includes: Based on the cell location information, the table structure, and the text area, determine the position in the table structure corresponding to the text content in each text area; The text content in each of the text regions is filled into the corresponding positions of the structural sequence to generate a table represented using hypertext language.

7. The method according to claim 1, characterized in that, The method further includes: When a visual question-and-answer instruction for the target image is received, the target question in the visual question-and-answer instruction is obtained; Determine the answer content corresponding to the target question.

8. The method according to claim 7, characterized in that, Determining the answer content corresponding to the target question includes: The image features of the target image, the position information of the cell, the text region, the text content in the text region, and the target question are input into a pre-trained visual question answering model to generate the answer content corresponding to the target question; wherein, the visual question answering model is a BERT pre-trained model trained using a sample table image, each text region in the sample table image, the text content of the sample table image, the image features of the sample table image, the input question, and the ground truth of the corresponding answer content.

9. A form recognition device, characterized in that, The device includes: The segmentation module is used to segment table regions from the target image to obtain a table image; The first recognition module is used to recognize the table structure and cell position information in the table image using a pre-trained table recognition model; wherein, the table recognition model includes an encoding layer for extracting image features of the table image and a decoding layer for generating the table structure; the input of the decoding layer is the input content of the encoding layer and the output content of the encoding layer; the cell position information includes the center point coordinates, width, and height of the cell; The second recognition module is used to recognize the text area of ​​the table image and the text content in the text area; The generation module is used to generate a spreadsheet file based on the cell's location information, the table structure, the text area, and the text content in the text area; each cell in the spreadsheet file is filled with text content from a text area that matches the cell's location information; The decoding layer includes a first network layer, and a second and a third network layer located after and in parallel with the first network layer; the first network layer includes a self-attention mechanism transformer layer; the second network layer is used to identify the position of cells in the table image and includes two cross-shaped window self-attention mechanism cswintransformer layers; the third network layer is used to predict the table structure in the table image and includes a self-attention mechanism transformer layer.

10. The apparatus according to claim 9, characterized in that, The table structure is represented using hypertext language.

11. The apparatus according to claim 9 or 10, characterized in that, The segmentation module includes: The first extraction submodule is used to extract image features of the target image using a pre-trained image feature extraction network; The first recognition submodule is used to identify candidate positions of tables in the target image based on the image features of the target image, and obtain recognition results for the candidate positions of the tables; The removal submodule is used to remove overlapping results in the recognition results using a non-maximum suppression algorithm to obtain the target position of the table. The second extraction submodule is used to extract the image region at the target location of the table from the target image to obtain the table image.

12. The apparatus according to any one of claims 9-11, characterized in that, The identification module includes: The second recognition submodule is used to recognize the text regions in the table image using a pre-trained text region recognition network and pre-extracted text region feature maps; wherein, the text region recognition network is a differentiable binarization network DBNet; and the text region feature maps are feature maps processed by a convolutional block attention module. The third recognition submodule is used to perform text recognition on the text region to obtain the text content in the text region.

13. The apparatus according to claim 12, characterized in that, The third identification submodule is specifically used for: The text content of the text region is identified using a pre-trained text recognition network; wherein the text recognition network is a network trained using a specified comprehensive loss; wherein the specified comprehensive loss is a comprehensive loss obtained by fusing temporal class classification loss and aggregated cross-entropy loss based on a neural network.

14. The apparatus according to any one of claims 9-13, characterized in that, The generation module includes: The determination submodule is used to determine the position in the table structure corresponding to the text content in each text area based on the position information of the cell, the table structure, and the text area; The generation submodule is used to fill the text content in each of the text regions into the corresponding structural sequence positions to generate a table represented using hypertext language.

15. The apparatus according to claim 9, characterized in that, The device further includes: The second acquisition module is used to acquire the target question in the visual question-and-answer instruction when a visual question-and-answer instruction for the target image is received; The determination module is used to determine the answer content corresponding to the target question.

16. The apparatus according to claim 15, characterized in that, The determining module is specifically used for: The image features of the target image, the position information of the cell, the text region, the text content in the text region, and the target question are input into a pre-trained visual question answering model to generate the answer content corresponding to the target question; wherein, the visual question answering model is a BERT pre-trained model trained using a sample table image, each text region in the sample table image, the text content of the sample table image, the image features of the sample table image, the input question, and the ground truth of the corresponding answer content.

17. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-8.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-8.

Images