Table restoration method, table restoration model training method, apparatus and device

Abstract

The invention provides a table restoration method, a table restoration model training method, a table restoration model training device and equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring an input image, wherein the input image comprises a table; based on the input image, partition line data and table structure information of the table are obtained, the partition line data are used for indicating partition lines in the table, and the table structure information is used for indicating a distribution structure of cells in the table; according to the table structure information, optimizing the separation line data to obtain separation line optimization data; and reducing each cell in the table based on the separation line optimization data to obtain the table. According to the method, optimization of the separation line is guided through the table structure information, reference information is enriched, and the table restoration accuracy can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a table restoration method, a training method for a table restoration model, an apparatus, and a device. Background Technology

[0002] Table reconstruction, academically known as Table Structure Recognition (TSR), aims to identify and understand the structure of tables in various document types, such as PDF (Portable Document Format), images, and HTML (Hyper Text Markup Language).

[0003] Related technologies treat tables as images and use image processing techniques (such as edge detection, morphological operations, connected component analysis, etc.) to identify the structure of tables. For example, image processing techniques are used to obtain feature maps that characterize tables, and based on the feature maps of the tables, various dividing lines are generated to construct the tables, and then the tables in the images are reconstructed based on the dividing lines.

[0004] However, the relevant technology is easily affected by the complexity of the table. For example, there are problems such as missed detection, false detection, and inaccurate detection for non-visually visible dividing lines or distorted dividing lines, which leads to low accuracy in table reconstruction. Summary of the Invention

[0005] This application provides a table restoration method, a table restoration model training method, apparatus, and device. The technical solution includes the following:

[0006] According to one aspect of the embodiments of this application, a table restoration method is provided, the method comprising: Obtain an input image, which includes a table; Based on the input image, the separator line data and table structure information of the table are obtained. The separator line data is used to indicate the separator lines in the table, and the table structure information is used to indicate the distribution structure of each cell in the table. Based on the table structure information, the separator line data is optimized to obtain optimized separator line data; Based on the separator line, the data is optimized to restore each cell in the table, thus obtaining the table.

[0007] According to one aspect of the embodiments of this application, a method for training a table reconstruction model is provided, the method comprising: Obtain a sample image and label data for the sample image, wherein the sample image includes a table and the label data is used to label the table; Based on the sample image, the table reconstruction model obtains the separator line data and table structure information of the table. The separator line data is used to indicate the separator lines in the table, and the table structure information is used to indicate the distribution structure of each cell in the table. The separator line data is optimized based on the table structure information using the table restoration model to obtain optimized separator line data. The table reconstruction model optimizes the data based on the separator lines and decodes the mapping relationship data to indicate multiple separator lines related to the cell. Based on the mapping relationship data and the label data, the table reconstruction model is trained to obtain the trained table reconstruction model.

[0008] According to one aspect of the embodiments of this application, a table restoration apparatus is provided, the apparatus comprising: An input image acquisition module is used to acquire an input image, wherein the input image includes a table; The table data acquisition module is used to acquire the separator line data and table structure information of the table based on the input image. The separator line data is used to indicate the separator lines in the table, and the table structure information is used to indicate the distribution structure of each cell in the table. The data acquisition optimization module is used to optimize the separator line data based on the table structure information to obtain optimized separator line data. The cell restoration module is used to restore each cell in the table based on the data optimized by the separator line, so as to obtain the table.

[0009] According to one aspect of the embodiments of this application, a training apparatus for a table reconstruction model is provided, the apparatus comprising: A sample image acquisition module is used to acquire sample images and label data of the sample images, wherein the sample images include tables and the label data is used to label the tables; The table data acquisition module is used to acquire the separator line data and table structure information of the table based on the sample image through the table reconstruction model. The separator line data is used to indicate the separator lines in the table, and the table structure information is used to indicate the distribution structure of each cell in the table. The optimized data acquisition module is used to optimize the separator line data based on the table structure information through the table restoration model to obtain optimized separator line data. The relational data acquisition module is used to decode the mapping relational data by optimizing the data based on the separator lines through the table restoration model, and the mapping relational data is used to indicate multiple separator lines related to the cell. The table restoration model training module is used to train the table restoration model based on the mapping relationship data and the label data, so as to obtain the trained table restoration model.

[0010] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described table restoration method, or to implement the above-described table restoration model training method.

[0011] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, the computer program being loaded and executed by a processor to implement the above-described table restoration method, or to implement the above-described table restoration model training method.

[0012] According to one aspect of the embodiments of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the aforementioned table restoration method, or to implement the aforementioned table restoration model training method.

[0013] The technical solutions provided in this application embodiment may include the following beneficial effects.

[0014] Because there is a correspondence between cells and separators, the separator data used to indicate the separators in the table can be optimized by using the table structure information that indicates the distribution structure of each cell in the table. This optimization process allows for reference to the rich structural information of the separators indicated by the distribution structure of the cells, resulting in more accurate separator data (i.e., optimized separator data). In other words, the separators indicated by the optimized separator data are more accurate, thereby improving the accuracy of determining the separators in the table and, consequently, improving the accuracy of the table reconstruction. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of a table restoration system provided in one embodiment of this application; Figure 2 This is a schematic diagram of a table restoration model provided in one embodiment of this application; Figure 3 This is a schematic diagram of an exporter in a candidate component export network provided in one embodiment of this application; Figure 4 This is a schematic diagram of a self-up-down extractor in an exporter provided in one embodiment of this application; Figure 5 This is a schematic diagram of a cross-context synthesizer in an exporter provided in one embodiment of this application; Figure 6 This is a schematic diagram of an extractor in a candidate component export network provided in one embodiment of this application; Figure 7 This is a schematic diagram of a first optimized network for separating lines provided in one embodiment of this application; Figure 8 This is a schematic diagram of a second optimized network for separating lines provided in one embodiment of this application; Figure 9 This is a schematic diagram of a table assembly network provided in one embodiment of this application; Figure 10 This is a flowchart of a table restoration method provided in one embodiment of this application; Figure 11 This is a schematic diagram of a table restoration method provided in one embodiment of this application; Figure 12 This is a schematic diagram of a complex table provided in one embodiment of this application; Figure 13 This is a schematic diagram of a separator line provided in one embodiment of this application; Figure 14 This is a schematic diagram of a table restoration method provided in another embodiment of this application; Figure 15 This is a flowchart of a table restoration method provided in another embodiment of this application; Figure 16 This is a flowchart of a training method for a table restoration model provided in one embodiment of this application; Figure 17 This is a block diagram of a table restoration apparatus provided in one embodiment of this application; Figure 18 This is a block diagram of a training apparatus for a table reconstruction model provided in one embodiment of this application; Figure 19 This is a block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0018] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0019] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0020] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision. Pre-trained models in the field of vision, such as Swin-Transformer, ViT (Vision Transformer), V-MOE (Vision Mixture-of-Experts), and MAE (Masked Autoencoders), can be fine-tuned and quickly and widely applied to specific downstream tasks. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D (Three-Dimensional) technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies.

[0021] Key technologies in speech technology include Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and voiceprint recognition. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech emerging as one of the most promising methods. Large-scale modeling has revolutionized speech technology; pre-trained models such as WavLM and UniSpeech, which utilize the Transformer architecture, possess strong generalization and versatility, enabling them to excel in various speech processing tasks.

[0022] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP deals with natural language, the language people use in daily life, and is closely related to linguistics, while also involving computer science and mathematics. Pre-trained models, a crucial technique for model training in artificial intelligence, evolved from large language models in NLP. After fine-tuning, large language models can be widely applied to downstream tasks. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0023] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning. Pre-trained models are the latest development in deep learning, integrating all of these techniques.

[0024] Autonomous driving technology refers to vehicles driving themselves without driver intervention. It typically includes technologies such as high-precision mapping, environmental perception, computer vision, behavioral decision-making, path planning, and motion control. Autonomous driving encompasses various development paths, including single-vehicle intelligence, vehicle-to-infrastructure (V2I) communication, and networked cloud control. Autonomous driving technology has broad application prospects, currently focusing on logistics, public transportation, taxis, and intelligent transportation systems, and is expected to see further development in the future.

[0025] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, digital twins, virtual humans, robots, AI-generated content (AIGC), conversational interaction, smart healthcare, smart customer service, and game AI. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0026] Table structure recognition technology is a technology involving computer vision and natural language processing. Its purpose is to identify and understand the structure of tables in various types of documents, such as PDF (Portable Document Format), images, and HTML (Hyper Text Markup Language).

[0027] Table structure recognition is a process of extracting and parsing tabular data from documents, web pages, images, etc. It identifies the layout and hierarchical structure of tables to convert visual information into structural descriptions that can be reconstructed. This task is of great significance in fields such as information extraction, data mining, and natural language processing.

[0028] The technical solutions provided in this application involve computer vision and machine learning technologies in artificial intelligence to achieve table reconstruction (i.e., table structure recognition). For example, computer vision technology can be used to obtain table feature maps of an input image, as well as cell data, row-column relationship data, and row-column attribute data of the table. Then, machine learning technology can be used to train a table reconstruction model based on the table feature maps, cell data, row-column relationship data, and row-column attribute data, thereby obtaining a trained table reconstruction model. The trained table reconstruction model can be used to reconstruct tables in input images.

[0029] Table structure recognition technology helps machines understand how data in a table is organized, such as which data belongs to which row, which data belongs to which column, and the relationships between different data. Table structure recognition technology is very useful in many fields, such as data mining, information retrieval, automated data input tasks, and any scenario requiring table reconstruction. The technical solution provided in this application is applicable to any scenario where table structure recognition technology is applicable, and it can improve the accuracy of table reconstruction (i.e., the accuracy of table structure recognition).

[0030] For example, when extracting data from a large number of financial reports in PDF format, the technical solution provided in this application can be used to automatically and accurately reconstruct the various tables corresponding to the financial reports, thereby extracting data from the tables and saving significant time and resources. As another example, when reconstructing tables from an image, the technical solution provided in this application can be used to automatically and accurately reconstruct the tables from the image.

[0031] The table restoration model provided in the embodiments of this application will be described below.

[0032] Please refer to Figure 1 The diagram illustrates a table recognition system according to an embodiment of this application. The table recognition system may include a model training device 10 and a model usage device 20.

[0033] The model training device 10 can be an electronic device such as a mobile phone, desktop computer, tablet computer, laptop computer, PC (Personal Computer), vehicle terminal, server, intelligent robot, smart TV, multimedia playback device, or other electronic devices with strong computing power. This application embodiment does not limit this. The server can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center.

[0034] The model training device 10 is used to train the table reconstruction model 30. The table reconstruction model 30 refers to the AI ​​model used to reconstruct tables. An AI model is a mathematical model based on artificial intelligence technology that can automatically process and analyze input data and output corresponding results. AI models include models with a large number of parameters trained through deep learning algorithms and artificial neural networks, such as the aforementioned large models, large language models, and pre-trained models.

[0035] The table reconstruction model 30 takes an image of a table (such as an image containing a table) as input and a table (such as data used to describe the table) as output. Optionally, the model training device 10 can use machine learning to train the table reconstruction model 30 using images to obtain a trained table reconstruction model 30 with better performance.

[0036] The trained table reconstruction model 30 can be deployed on the model-using device 20 to provide services such as table reconstruction, table structure recognition, data input, and data retrieval. The model-using device 20 can be an electronic device such as a mobile phone, desktop computer, tablet computer, laptop computer, personal computer, vehicle terminal, server, intelligent robot, smart TV, multimedia playback device, or other electronic devices with strong computing power. This application embodiment does not limit this.

[0037] Optionally, the model-using device 20 may have a client running a target application installed, which may include at least one of the following: a table reconstruction application, a data input application, a data retrieval application, a table building application, a social entertainment application, a game application, or a shopping application. Optionally, the client of the aforementioned target application supports the use of the trained table reconstruction model 30.

[0038] In some embodiments, during the table restoration process, the table restoration model 30 may optimize the separator lines twice sequentially to obtain accurate separator lines. Please refer to... Figure 2 The diagram illustrates a table reconstruction model provided in one embodiment of this application. The table reconstruction model 30 includes a candidate component derivation network 301 (extractor), a first separator optimization network 302 (perceptron), a second separator optimization network 303 (associator), and a table assembly network 304 (assembler).

[0039] This application embodiment does not limit the positions of the first optimization network 302 and the second optimization network 303 of the separator line in the table reconstruction model 30. In one example, the candidate component derivation network 301 takes an image as input, the first optimization network 302 takes the output of the candidate component derivation network 301 as input, the second optimization network 303 takes the output of the first optimization network 302 and the output of the candidate component derivation network 301 as input, the table assembly network 304 takes the output of the second optimization network 303 as input, and the output of the table assembly network 304 is table data, which refers to the data used to describe the table.

[0040] In one example, the candidate component export network 301 takes an image as input, the second separator optimization network 303 takes the output of the candidate component export network 301 as input, the first separator optimization network 302 takes the output of the second separator optimization network 303 and the output of the candidate component export network 301 as input, the table assembly network 304 takes the output of the first separator optimization network 302 as input, and the output of the table assembly network 304 is table data, which refers to the data used to describe the table (hereinafter referred to as...). Figure 2 The structure shown is illustrated with an example.

[0041] In this embodiment, the candidate component export network 301 can be a neural network for feature extraction from an image. It can be built based on any neural network suitable for image processing, such as Transformer, Swin-Transformer, ViT, V-MOE, MAE, CNN (Convolutional Neural Networks), RNN (Recurrent Neural Networks), ResNet (Residual Networks), etc. For example, the candidate component export network 301 can be used to extract basic information about tables in an image, such as the cell distribution structure (i.e., cells, the relationships between cells, which row, which column, how many rows, how many columns, how many rows they span, how many columns they span, etc.), separator lines, the relationships between separator lines, and the feature representation of the table (such as the embeddings converted from the corresponding pixels of the table).

[0042] In one example, the candidate component export network 301 may include an exporter 301a and an extractor 301b. The exporter 301a is used to obtain table structure information of a table in an image, which is used to indicate the distribution structure of each cell in the table. The extractor 301b is used to extract the separator line data of the table in the image, which is used to indicate the separator lines in the table.

[0043] Optionally, the exporter 301a takes an image of the table as input and outputs the table's structure information. It can be built based on technologies such as Transformer, Swin-Transformer, ViT, V-MOE, MAE, CNN, RNN, and ResNet. The extractor 301b also takes an image of the table as input and outputs the table's separator line data. It can be a neural network for object detection, such as detecting the separator lines of a table in an image. It can be built based on any neural network that can be used for object detection, such as Transformer, Swin-Transformer, ViT, V-MOE, Deformable DETR (Deformable Detection Transformer, a Transformer-based object detection model), YOLO (YouOnly Look Once, an object detection algorithm), and Faster RCNN (Faster Region-CNN, a deep learning-based object detection method).

[0044] For example, taking the Transformer-based exporter 301a as an example, refer to... Figure 3The exporter 301a may include multiple collaborative modules (such as collaborative module 1 to collaborative module L, where L is an integer greater than 1), a feature extractor 1, and an initialization unit (Init). Each collaborative module includes a pair of CCS (Cross Context Synthesizer) and ECE (Ego Context Extractor). Feature extractor 1 is used to obtain multimodal features based on the input image, such as visual features, geometric features, and content features. ECE is used to generate graph context for each modality feature. CCS is responsible for fusing and modulating the interaction information between different modality features. Through multiple stacking of multiple collaborative modules, the intra-modal context and inter-modal collaboration of modality features can be carried out in a hierarchical and alternating manner, thereby fusing the fused features corresponding to the multimodal features (such as word embeddings of the collaborative graph). This allows the low-level and high-level context information of the multimodal features to cooperate with each other throughout the network to achieve better performance. The initialization unit is used to initialize the multimodal features.

[0045] Both CCS and ECE can be constructed based on the encoder in the Transformer, essentially utilizing the multi-head attention layer within the encoder. For example, see reference... Figure 4 and Figure 5 Both ECE 401 and CCS501 are built on the CMHA structure, which is the encoder in Transformer. It includes: multi-head attention mechanism layer, normalization layer (Add & Norm Layer) and feed forward layer.

[0046] Optionally, the exporter 301a also includes an output layer (such as one built on a fully connected layer, FC) to process the fused features (such as classification) to obtain tabular structure information.

[0047] Taking the extraction tool 301b built based on Transformer as an example, refer to... Figure 6 Extractor 301b may include feature extractor 2 and encoder. Feature extractor 2 may be built based on CNN to extract table feature maps of a table from the input image. The table feature maps are used to indicate the overall visual information of the table. The encoder may be built based on the encoder in Deformable DETR to encode the table feature maps to obtain separator line data.

[0048] The aforementioned first optimization network 302 for separator lines can be a neural network used to optimize the separator lines of a table, such as optimizing the output of extractor 301b or the output of second optimization network 303 for separator lines. This embodiment does not limit this, and it can be constructed based on technologies such as Transformer, Swin-Transformer, ViT, V-MOE, and DeformableDETR. For example, the first optimization network 302 for separator lines is used to assist in processing the output of extractor 301b based on the output of extractor 301a to obtain separator line adjustment data. Alternatively, the first optimization network 302 for separator lines can be used to assist in processing the output of second optimization network 303 for separator lines based on the output of extractor 301a to obtain optimized separator line data. Separator line adjustment data can refer to data after the separator line data has been optimized once using table structure information (such as partial information). Separator line optimization data can refer to data after the separator line data has been optimized a second time using table structure information (such as partial information). That is, the separator line data is used to indicate the initial recognition result of the separator lines, the separator line adjustment data can be used to indicate the first optimization result of the separator lines, and the separator line optimization data can be used to indicate the second optimization result of the separator lines.

[0049] For example, taking the first optimization network 302 for the separator line based on Transformer as an example, refer to... Figure 7 The first optimization network 302 for the separator line may include a decoder and two fully connected layers (FC). The decoder may be built based on the decoder in Deformable DETR. The input of the decoder includes separator line data and one or more pieces of information from the table structure information.

[0050] The aforementioned second optimization network 303 for separator lines can also be a neural network used to optimize the separator lines of a table, such as optimizing the output of extractor 301b or the output of first optimization network 302 for separator lines. This application embodiment does not limit this, and it can be constructed based on technologies such as Transformer, Swin-Transformer, ViT, V-MOE, and DeformableDETR. For example, the second optimization network 303 for separator lines can be used to assist in processing the output of first optimization network 302 for separator lines based on the output of extractor 301a to obtain optimized separator line data. Alternatively, the second optimization network 303 for separator lines can be used to assist in processing the output of extractor 301b based on the output of extractor 301a to obtain adjusted separator line data.

[0051] For example, taking the second optimization network 303 for the separator line based on Transformer as an example, refer to Figure 8The second optimization network 303 for the separator line includes an encoding layer and two fully connected layers. The encoding layer can follow the basic paradigm of the Transformer encoder, such as including a multi-head gross attention mechanism layer, a normalization layer, and a pre-feedback layer. Optionally, the encoding layer of the second optimization network 303 for the separator line includes two encoding lines: one for processing separator line adjustment data and cell information (one type of information in the table structure information), and the other for processing separator line adjustment data and row-column relationship information (another type of information in the table structure information).

[0052] Optionally, the number of encoding lines included in the encoding layer is not limited in this embodiment, and can be adaptively set and adjusted according to actual usage requirements. For example, for each type of information input into the second optimization network 303 of the separator line in the table structure information, one encoding line can be set accordingly. If two types of information are input into the second optimization network 303 of the separator line in the table structure information, two encoding lines can be set to process the two types of information respectively; if three types of information are input into the second optimization network 303 of the separator line in the table structure information, three encoding lines can be set to process the three types of information respectively.

[0053] In some feasible examples, the structure of the first optimization network 302 and the structure of the second optimization network 303 are the same, such as both adopting the following... Figure 7 The structures shown are constructed or all adopt the following methods: Figure 8 The structure is shown. In some feasible examples, the structure of the first optimization network 302 and the structure of the second optimization network 303 are different, such as the first optimization network 302 adopting the following... Figure 7 The structure shown is constructed such that the second optimization network 303 of the separator line adopts the following... Figure 8 The structure shown is constructed as follows; for example, the first optimization network 302 of the separator line adopts the following... Figure 8 The structure shown is constructed such that the second optimization network 303 of the separator line adopts the following... Figure 7 The structural construction shown is not limited in this application embodiment.

[0054] Optionally, the first optimization network 302 and the second optimization network 303 are collectively referred to as the partition line optimization networks of the table restoration model 30.

[0055] The aforementioned table assembly network 304 can be a neural network used to construct tables, such as restoring tables in an image based on the optimized data from the aforementioned dividing lines. It can be constructed based on technologies such as Transformer, Swin-Transformer, ViT, V-MOE, and Deformable DETR. This application embodiment does not limit this.

[0056] For example, taking the table assembly network 304 built based on Transformer as an example, refer to... Figure 9 The table assembly network 304 includes a Non-Maximum Suppression (NMS) layer, an Order Regularization layer, and a decoding layer. The decoding layer follows the basic paradigm of a Transformer decoder, including a multi-head attention mechanism layer, a normalization layer, and a pre-feedback layer. The NMS layer takes the separator optimization data as input, and the decoding layer outputs the mapping relationship data, which indicates the multiple separators associated with the cells.

[0057] In some embodiments, during the table reconstruction process, the table reconstruction model 30 may optimize the separator lines once to obtain accurate separator lines. In one example, the table reconstruction model 30 may only include a candidate component export network 301, a separator line first optimization network 302, and a table assembly network 304. The candidate component export network 301 takes the image as input, the separator line first optimization network 302 takes the output of the candidate component export network 301 (separator line data + table structure information) as input, and the table assembly network 304 takes the output of the separator line first optimization network 302 as input, and the output of the table assembly network 304 is table data (such as the mapping relationship data mentioned above).

[0058] In one example, the table reconstruction model 30 may also consist only of a candidate component export network 301 (extractor), a second separator optimization network 303, and a table assembly network 304 (assembler). The candidate component export network 301 takes the image as input, the second separator optimization network 303 takes the output of the candidate component export network 301 (separator data + table structure information) as input, and the table assembly network 304 takes the output of the second separator optimization network 303 as input, and the output of the table assembly network 304 is the table data (such as the mapping relationship data mentioned above).

[0059] It should be noted that, Figures 2 to 9 The table restoration model 30 shown is merely exemplary and illustrative. The structure of the table restoration model 30 is not limited in this application embodiment and can be set and adjusted according to actual usage requirements.

[0060] The following are embodiments of the method of this application, which illustrate the table restoration process (i.e., the method of using the above-mentioned table restoration model 30). For details not disclosed in the embodiments of the method of this application, please refer to the above embodiments.

[0061] Please refer to Figure 10The diagram illustrates a flowchart of a table restoration method provided in one embodiment of this application. The execution entity for each step of the method can be... Figure 1 The model shown uses device 20, and the method may include the following steps (1001~1004).

[0062] Step 1001: Obtain the input image, which includes a table.

[0063] The input image in this application embodiment can refer to an image of a table, such as a table image captured by photography, a table image cropped from an image, a table image cropped from a video frame, or a table image scanned from a document. For example, for a table in a document (such as PDF, HTML, image, etc.), the corresponding input image of the table can be obtained by scanning, photographing, or other methods.

[0064] The input image described above can be used as input to the table reconstruction model to reconstruct the table in the input image. Optionally, the input image can be a cropped image, such as retaining only the table area in the image, which can reduce information interference from non-table areas.

[0065] For example, refer to Figure 11 Users can upload input images to the model-using device 20. After acquiring the input image, the model-using device 20 can call the trained table reconstruction model to perform table subject detection on the input image and obtain detection results (such as the table feature map, dividing line data, table structure information, etc.). Then, based on the detection results, the trained table reconstruction model performs table structure recognition to obtain recognition results (such as the dividing line adjustment data, dividing line optimization data, mapping relationship data, etc.). Finally, the table in the input image is reconstructed based on the recognition results and displayed to the user.

[0066] In one example, the table described above can be a complex table, which includes tables with inherent structural complexity (complex in their own structure) and tables with external structural distortion (affected by external factors). For example, a table with inherent structural complexity can include at least one of the following: a table without clear dividing lines, a table without clear boundaries, a table with irregularly arranged dividing lines, a table containing nested tables, or a table where cells span across rows / columns. A table with external structural distortion can include at least one of the following: a distorted table (e.g., distorted due to photography, scanning, etc.), or a table where dividing lines are not straight lines. Dividing lines can refer to line segments that construct the table, and can be divided into row dividing lines, column dividing lines, and boundaries.

[0067] For example, refer to Figure 12For Table 1100 in the input image, it has complex elements such as missing separator lines, distorted separator lines (such as due to shooting), and cells spanning multiple rows. Therefore, Table 1100 is a complex table, which has both endogenous structural complexity and external structural distortion.

[0068] Step 1002: Based on the input image, obtain the table's separator data and table structure information. The separator data is used to indicate the separators in the table, and the table structure information is used to indicate the distribution structure of each cell in the table.

[0069] Separator line data can refer to the encoded data for separator lines. This includes the encoded data for each visually visible separator line in the input image. The encoded data can be data encoded from a feature map (such as feature vectors representing the separator lines). Encoding can be the process of converting information from one form or format to another. This separator line data represents a preliminary identification result and may contain issues such as missed detections, false detections, or inaccurate detection. For example, for non-visually visible separator lines in the input image, this separator line data does not include their corresponding encoded data.

[0070] In one example, the candidate component export network in the table reconstruction model can obtain the table's separator line data and table structure information based on the input image. Then, the extractor in the candidate component export network can extract the table's separator line data. For example, refer to... Figure 6 This process may include the following: 1. Extract the table feature map from the input image. This table feature map is used to indicate the overall visual information of the table.

[0071] Optionally, a table feature map is extracted from the input image by feature extractor 2 in extractor 301b. This table feature map can refer to a feature map used to describe the table, which can be a feature representation of the table, such as a feature vector (e.g., embedding) converted from the pixel values ​​of the corresponding pixels in the table, to characterize the overall visual information of the table.

[0072] For example, the input image is first subjected to target detection by feature extractor 2 to determine the pixel region corresponding to the table. Then, the pixel values ​​of each pixel in the pixel region are processed by feature extractor 2 to obtain the table feature map, denoted as F.

[0073] 2. Encode the feature map of the table to obtain the separator line data.

[0074] Optionally, the separator line data is obtained by encoding the table feature map in the extractor 301b, denoted as... The separator data may include the position information and sequence number information of the separator. The position information of the separator is used to indicate the position of the separator, and the sequence number information of the separator is used to indicate the sequence number of the separator, such as each separator corresponding to a sequence number.

[0075] Optionally, the cell distribution structure can be represented by a set of information associated with each cell, with each set of information corresponding to a cell combined to form table structure information. This set of information may include the cell's position, the relationship between cells, how many rows / columns a cell can be divided into, and the row / column in which the cell is located, etc.

[0076] In one example, the table structure information includes at least one of the following: cell information, row-column relationship information, and row-column attribute information. The cell information is used to indicate each cell in the table, the row-column relationship information is used to indicate the rows and columns that the table has, and the row-column attribute information is used to indicate the row and column to which each cell in the table belongs.

[0077] Cell information refers to the data used to describe a cell, such as coded data for a cell (e.g., a token). This cell information may include information indicating the cell's location, size, serial number, etc.

[0078] Row and column relationship information refers to data used to describe the row and column distribution. This distribution indicates how many rows and columns a table has, and how they are distributed. Optionally, the row and column relationship information may include coded data (such as embedding) for each row and coded data for each column. The coded data for each row can be obtained by combining the coded data of all cells corresponding to that row with a learnable coded data set. Similarly, the coded data for each column can be obtained by combining the coded data of all cells corresponding to that column with a learnable coded data set. This avoids missing rows and columns in the reconstructed table and ensures the correct row and column distribution.

[0079] Row and column attribute information refers to data used to describe row and column affiliation. Row and column affiliation indicates which cells belong to the same column / row, or which row / column a given cell belongs to. Optionally, row and column attribute information includes the region code data corresponding to each cell. Cells in the same row / column have the same region code data, while cells in different rows / columns have different region code data. This region code data can be a specific value. For example, all pixels corresponding to text boxes in cells within the same row / column may correspond to the same value, thus allowing the row and column attribute data to indicate which cells belong to the same row and which belong to the same column. Text boxes are used to indicate the area containing the content of a cell; they do not necessarily exist physically.

[0080] Optionally, for cells that span rows / columns, all corresponding pixels are assigned to the area encoding data corresponding to the rows / columns spanned, indicating which rows / columns the cell spans.

[0081] In one example, the exporter in the candidate component export network can obtain the table structure information of a table based on the input image. For example, refer to... Figure 3 This process may include the following: 1. Extract the multimodal features of the table from the input image. The multimodal features include at least two of the following features: visual features, geometric features, and text features. The visual features are used to indicate the visual information of the text box, the geometric features are used to indicate the geometric structure of the text box, the text features are used to indicate the content in the text box, and the text box is used to indicate the area where the content in the cell is located.

[0082] Optionally, the exporter 301a in the candidate component export network extracts the multimodal features of the table from the input image. The multimodal features of the table may include the multimodal features corresponding to each cell. In this embodiment, a text box is used to refer to a cell, which may refer to the area in the cell used for writing content.

[0083] For example, visual features can be the embedding obtained by processing the pixels corresponding to the text box in the cell, geometric features can be the embedding obtained by processing the geometric feature representation (such as position, width and height) of the text box in the cell, and text features can be the embedding obtained by encoding the content in the text box. This application embodiment does not limit these features.

[0084] 2. Fuse multimodal features to obtain fused features. These fused features include interaction information between various modal features and interaction information within different modal features. The fused features are used to indicate the various text boxes in the table.

[0085] Optionally, the multimodal features can be fused through multiple collaborative modules in the exporter 301a to obtain the fused features. This embodiment of the application does not limit the upper limit of the number of modal features included in the multimodal features. For example, if the multimodal features include two modal features, the exporter 301a fuses these two modal features to obtain the fused features; if the multimodal features include three modal features, the exporter 301a fuses these three modal features to obtain the fused features; if the multimodal features include four modal features, the exporter 301a fuses these four modal features to obtain the fused features.

[0086] For example, refer to Figure 3Multimodal features include visual features, geometric features, and text features. These three types of features are processed by L collaborative modules in the exporter 301a and then fused to obtain fused features (i.e., word embeddings of the collaborative graph).

[0087] The first collaborative module takes the output of feature extractor 1 as input, and the second to Lth collaborative modules all take the output of the previous collaborative module as input. Each collaborative module works in the same way. The following will use the working method of a certain collaborative module as an example to illustrate the process.

[0088] For example, refer to Figure 4 and Figure 5 Collaboration module L obtains the output of collaboration module L-1 ( , , ,as well as , , Then, ECE 401 is used to generate the graph context corresponding to each modality feature, such as... The query information (Q) and graph-edge combination features (K and V) are encoded as key-value information (K and V). (Graph context of visual features output by collaboration module L), with The query information (Q) and graph edge combination features (key-value information, K and V) are encoded to obtain... (Graph context of geometric features output by collaboration module L), to The query information (Q) and graph edge combination features (key-value information, K and V) are encoded to obtain... (Graph context of text features output by collaboration module L), then , and Send to CCS 501.

[0089] Then, CCS 501 is used to selectively fuse graph contexts from different modal features, such as... In order to inquire about information, and As key-value information, it is encoded to obtain (The fusion features of the text features output by the collaboration module L), with In order to inquire about information, and As key-value information, it is encoded to obtain (The fused features of the geometric features output by the collaboration module L), with To inquire about information, and As key-value information, it is encoded to obtain (The fusion features of the visual features output by the collaboration module L).

[0090] Finally, using exporter 301a, for... , and By performing fusion, the fusion features corresponding to the table can be obtained, such as... , and By adding the corresponding elements, the fused feature can be obtained.

[0091] 3. Classify the fused features to identify the table structure information of the obtained table.

[0092] Optionally, the output layer in network 301 is derived through candidate components to classify the fused features and identify the table structure information. For example, the output layer may include at least one fully connected layer (or classifier), each fully connected layer being used to acquire one type of information. The aforementioned table structure information includes the information identified by the at least one fully connected layer. This application embodiment does not limit the upper limit of the amount of information included in the table structure information. The more information included in the table structure information, the richer the information that the table reconstruction model can refer to, and the more accurately the dividing lines are determined.

[0093] For example, when the table structure information includes cell information, row-column relationship information, and row-column attribute information, the output layer includes three fully connected layers. One fully connected layer classifies the fused features to obtain cell information, another fully connected layer classifies the fused features to obtain row-column relationship information, and the third fully connected layer classifies the fused features to obtain row-column attribute information. The input to all three fully connected layers is the fused feature. By employing the interaction between multimodal features and the interaction within each modality, a more accurate fused feature is obtained. Based on the fused feature, accurate table structure information can be obtained, thereby improving the accuracy of table reconstruction.

[0094] Step 1003: Optimize the separator line data based on the table structure information to obtain optimized separator line data.

[0095] The optimized separator data is used to indicate the separators after optimization of the table structure information. Optionally, the degree of matching between the separators indicated by the optimized separator data and the table is greater than or equal to the degree of matching between the separators indicated by the optimized separator data and the table; that is, the accuracy of the separators indicated by the optimized separator data is higher than or equal to the accuracy of the separators indicated by the optimized separator data. This degree of matching can be used to indicate the degree of restoration of the separators, and can be determined based on at least one of the following factors: the number of separators, the position of the separators, and the shape of the separators. For example, taking the optimized separator data as an example, the closer the number of separators indicated by the optimized separator data is to the number of separators in the table, the higher the degree of matching; the closer the position of the separators indicated by the optimized separator data is to the position of the separators in the table, the higher the degree of matching; the closer the shape of the separators indicated by the optimized separator data is to the shape of the separators in the table, the higher the degree of matching. This application embodiment does not limit this.

[0096] Optionally, by optimizing the separator data based on the table structure information using the separator optimization network in the table restoration model, optimized separator data can be obtained. Guided by the rich structural information about the separators indicated by the table structure information, the accuracy of determining the table separators can be improved.

[0097] In one example, if the separator optimization network only includes the first separator optimization network, then the process of obtaining the separator optimization data may include the following: 1. Encode the table structure information to obtain the first encoded data.

[0098] The first encoded data refers to the encoded data of the table structure information, which can be obtained through the first optimized network encoding of the separator line.

[0099] Optionally, if the table structure information includes only one type of information, the first type of information is encoded using a first optimization network through a separator line to obtain first encoded data. For example, if the table structure information includes cell information, row-column relationship information, and row-column attribute information, the first encoded data is obtained by encoding the cell information, row-column relationship information, and row-column attribute information using a first optimization network through a separator line.

[0100] When the table structure information includes multiple types of information, these multiple types of information can be merged (e.g., the elements at corresponding positions are added together) and then encoded to obtain the first encoded data. Alternatively, the multiple types of information can be encoded separately and then merged to obtain the encoded sub-data to obtain the first encoded data. This application embodiment does not limit this approach.

[0101] For example, when the table structure information includes two types of information, the two types of information are encoded using a first optimization network with separator lines to obtain first encoded data. Alternatively, the two types of information can be encoded separately and then merged to obtain encoded sub-data, which is also the first encoded data. For instance, when the table structure information includes cell information and row / column relationship information, the cell information and row / column relationship information are merged using a first optimization network with separator lines before encoding to obtain first encoded data; when the table structure information includes cell information and row / column attribute information, the cell information and row / column attribute information are merged using a first optimization network with separator lines before encoding to obtain first encoded data; when the table structure information includes both row / column relationship information and row / column attribute information, the row / column relationship information and row / column attribute information are merged using a first optimization network with separator lines before encoding to obtain first encoded data.

[0102] When the table structure information includes three types of information, the first optimized network using separators encodes these three types of information to obtain the first encoded data. Alternatively, the three types of information can be encoded separately and then merged to obtain the encoded sub-data, which is also the first encoded data. For example, when the table structure information includes cell information, row-column relationship information, and row-column attribute information, the first optimized network using separators merges these three types of information together before encoding to obtain the first encoded data.

[0103] For example, refer to Figure 7 Taking table structure information including row and column attribute information as an example, this row and column attribute information can include row attribute information (i.e., which cells belong to the same row) and column attribute information (i.e., which cells belong to the same column). The first optimization network 302, acting as a separator, converts the region-coded data corresponding to the row attribute information and the region-coded data corresponding to the column attribute information (i.e., the instance index) into embeddings to obtain the first encoded data (i.e.,...). and ).

[0104] 2. Merge the separator data and the first encoded data to obtain the first merged data.

[0105] Optionally, the first fused data can be obtained by adding the separator data and the first encoded data element-wise at corresponding positions using the first optimized network coding. Compared to the separator data, the first fused data carries more information.

[0106] For example, refer to Figure 7 , divide the data and the first encoded data (i.e. and By adding the elements at the corresponding positions, the first fused data can be obtained. .

[0107] 3. Decode the first fused data to obtain the first decoded data.

[0108] The first fused data is decoded by the decoder in the first optimization network of the separator line to obtain the first decoded data. For example, the first fused data is used as key-value information (i.e., K and V) and a set of learnable embeddings is used as query information (i.e., Q) to decode the first decoded data. The first decoded data can be used to indicate the separator line after optimization by the table structure information.

[0109] For example, refer to Figure 7 The decoder in the first optimized network 302, with the first fused data, is separated by a separator. For key-value information, (i.e., a set of learnable embeddings) is the query information, which is decoded to obtain the first decoded data.

[0110] 4. Based on the classification data corresponding to the first decoded data and the regression data corresponding to the first decoded data, the separator optimization data is obtained.

[0111] The classification data indicates the type of the dividing line, and the regression data indicates its location. Optionally, the first decoded data is classified through a fully connected layer in the first optimization network for the dividing line to determine the type of the dividing line (i.e., classification data), such as visually visible, non-visually visible, or boundary. The first decoded data is then regressed through another fully connected layer in the first optimization network for the dividing line to determine its location (i.e., regression data). For example, refer to... Figure 7 The two fully connected layers in the first optimization network 302, which separate the data, perform classification and position regression on the first decoded data, respectively, to obtain classification data Cls and regression data Reg.

[0112] Optionally, by concatenating the classification data corresponding to the first decoded data and the regression data corresponding to the first decoded data, the separator line optimized data can be obtained. For example, by using the separator line first optimization network 302 to concatenate the regression data Reg corresponding to the first decoded data after the classification data Cls corresponding to the first decoded data, the separator line optimized data can be obtained. .

[0113] In one example, considering complex table situations (such as table distortion or warping), straight dividing lines are no longer applicable. Therefore, embodiments of this application use a set of key points to represent the dividing lines. This set of key points includes at least three key points, which makes the candidate data and the data of the dividing lines more accurate.

[0114] For example, in the case where the dividing line is represented by multiple key points extracted from the dividing line, the dividing line data may include key point set data corresponding to each indicated dividing line, and the dividing line optimization data includes key point set data corresponding to each indicated dividing line, the key point set data including the location data of the key points, the location data being used to describe the location, such as coordinates.

[0115] For example, refer to Figure 13 If the separator line 1301 is not a straight line, then multiple key points can be extracted from the separator line 1301 (such as...). Figure 13 (The black dots in the image), and obtain the key point set data corresponding to these multiple key points to describe the separator line 1301.

[0116] Optionally, embodiments of this application may also use a cubic Bézier curve to describe the separator line. The Bézier curve equation has four parameters, which can be learned to serve as separator line data or separator line optimization data. For example... Figure 13 As shown, the separator line 1301 can be described by four parameters: S1, S2, S3, and S4 (S1 and S4 are the endpoints), in order to simplify the separator line data or optimize the separator line data.

[0117] In one example, if the separator optimization network only includes the second separator optimization network, then the process of obtaining the separator optimization data may include the following: 1. Using the separator line data as the query information and the table structure information as the key-value information, decode to obtain the first decoded data.

[0118] Optionally, the first decoded data is obtained by using the encoding layer (which acts as a decoder in this case) in the second optimized network of the separator line as the query information and the table structure information as the key-value information.

[0119] For example, the table structure information includes M types of information, where M is a positive integer. When M equals 1, the first decoded data is obtained by decoding the first type of information through the encoding layer in the second optimization network of the separator line, using the separator line data as the query information and the first type of information as the key-value information. When M is greater than 1, for the s-th type of information among the M types of information, the decoded sub-data corresponding to the s-th type of information is obtained by decoding the second optimization network of the separator line, using the separator line data as the query information and the s-th type of information as the key-value information. The first decoded data is obtained by fusing the decoded sub-data corresponding to the M types of information respectively, where s is a positive integer less than or equal to M.

[0120] For example, when the table structure information includes cell information, row-column relationship information, and row-column attribute information, the first decoded data is obtained by decoding the cell information, row-column relationship information, and row-column attribute information through the encoding layer in the second optimization network of the separator line.

[0121] When the table structure information includes two types of information, the two types of information are decoded separately using two different encoding lines in the encoding layer to obtain the corresponding decoded sub-data for each type of information. These decoded sub-data are then fused to obtain the first decoded data. For example, when the table structure information includes cell information and row / column relationship information, the two types of information are decoded separately using two different encoding lines in the encoding layer to obtain the corresponding decoded sub-data for each type of information. These decoded sub-data are then fused to obtain the first decoded data. Similarly, when the table structure information includes cell information and row / column attribute information, the two types of information are decoded separately using two different encoding lines in the encoding layer to obtain the corresponding decoded sub-data for each type of information. These decoded sub-data are then fused to obtain the first decoded data. Finally, when the table structure information includes both row / column relationship information and row / column attribute information, the two types of information are decoded separately using two different encoding lines in the encoding layer to obtain the corresponding decoded sub-data for each type of information. These decoded sub-data are then fused to obtain the first decoded data.

[0122] When the table structure information includes three types of information, the three types of information are decoded separately using three different encoding circuits in the encoding layer to obtain the decoded sub-data corresponding to each of the three types of information. These decoded sub-data are then fused together to obtain the first decoded data. For example, when the table structure information includes cell information, row-column relationship information, and row-column attribute information, these three types of information are decoded separately using three different encoding circuits in the encoding layer to obtain the decoded sub-data corresponding to each of the three types of information. These decoded sub-data are then fused together to obtain the first decoded data.

[0123] For example, refer to Figure 8 The second optimization network 303 for separator lines, as shown, takes table structure information including cell information and row-column relationship information as an example. Through the first encoder in the second optimization network 303 for separator lines (such as the encoding line on the left), the separator line data is processed. For query information, cell information For key-value information, decoded data yields decoded sub-data Y1, and the data is processed by the second encoder (such as the right-hand encoding line) in the second optimization network 303, using the separator line data. For querying information, row and column relationship information The key-value information is decoded to obtain decoded sub-data Y2. Then, through the normalization layer in the second optimization network 303, decoded sub-data Y1 and decoded sub-data Y2 are fused and normalized to obtain the first decoded data.

[0124] 2. Based on the classification data corresponding to the first decoded data and the regression data corresponding to the first decoded data, the separator optimization data is obtained.

[0125] Optionally, the first decoded data is classified through a fully connected layer in the second optimization network for the separator line to determine the type of separator line (i.e., classified data), and the first decoded data is regressed through another fully connected layer in the second optimization network for the separator line to determine the position of the separator line (i.e., regressed data). For example, refer to... Figure 8 The second optimization network 303 shown in the diagram has two fully connected layers that perform classification and position regression on the first decoded data, respectively, to obtain classification data Cls and regression data Reg.

[0126] Optionally, by concatenating the classification data corresponding to the first decoded data and the regression data corresponding to the first decoded data, the separator line optimized data can be obtained. For example, by using the separator line second optimization network 303 to concatenate the regression data Reg corresponding to the first decoded data with the classification data Cls corresponding to the first decoded data, the separator line optimized data can be obtained. .

[0127] In this embodiment, the table structure information can indicate the distribution structure of each cell in the table. For example, cell information can indicate the cells in the table, and there is a correspondence between cells and separator lines. For example, one cell can correspond to four separator lines, which strengthens the association between cells and separator lines to indicate the distribution of separator lines. This enables the table reconstruction model to identify non-visually visible separator lines, separator lines involving cells spanning rows / columns, and redundant separator lines (i.e., multiple detections) and inaccurate separator lines indicated by the auxiliary filtering separator line data. The row and column relationship information can indicate the table's... The table structure includes rows and columns, which correspond to separator lines. For example, one row / column may correspond to two separator lines. This information can also indicate the distribution of separator lines, enabling the table reconstruction model to identify non-visually visible separator lines, as well as redundant separator lines (i.e., multiple detections) and inaccurate separator lines indicated by the auxiliary filtering data. Row and column attribute information indicates the row and column of a cell, allowing the table reconstruction model to identify non-visually visible separator lines, distorted separator lines, and redundant separator lines (i.e., multiple detections) and inaccurate separator lines indicated by the auxiliary filtering data. Using one or more of the table structure information can enrich the information that the table reconstruction model can refer to, thereby improving the accuracy of separator line determination and ultimately improving the accuracy of table reconstruction.

[0128] For example, refer to Figure 14 Taking table structure information, including cell information, row-column relationship information, and row-column attribute information, as an example, for table 1401 in the input image, using the technical solution provided in this application embodiment, based on the separator line data and combined with cell information, row-column relationship information, and row-column attribute information, table 1402 can be accurately restored. The structure of table 1402 is the same as that of table 1401. However, in related technologies, when restoring a table based solely on cell boundaries (i.e., separator line data), there are problems such as missed detection of separator lines (e.g., failure to restore non-visually visible separator lines in distant table 1401), false detection of separator lines (e.g., restoring extra separator lines for cells spanning multiple rows in table 1401), inaccurate detection of separator lines (e.g., restoring straight lines for distorted separator lines in table 1401), and cell misalignment (e.g., generating extra cells for cells spanning multiple rows in table 1401 (i.e., incorrect separator lines)).

[0129] Step 1004: Optimize the data based on the separator lines to restore each cell in the table, and obtain the table.

[0130] Optionally, the table is restored by optimizing the data based on the separator lines through the table assembly network in the table restoration model to restore each cell in the table. There is a correspondence between cells and separator lines. For example, one cell can correspond to four separator lines. Based on the correspondence between separator lines and cells, the installation position of each separator line can be determined, thereby avoiding the problem of misaligned separator line installation and enabling the restoration of cells that span rows / columns (e.g., cells that span rows / columns are only related to their corresponding separator lines), thus effectively improving the accuracy of table restoration.

[0131] For example, combined Figure 9 Step 1004 may also include the following: 1. Decode the separator line optimization data to obtain mapping relationship data, which is used to indicate multiple separator lines associated with a cell.

[0132] Optionally, the mapping relationship data includes multiple sets of sub-relationship data, each set of sub-relationship data indicating the corresponding relationship between a cell and four separator lines. In this embodiment, the separator lines can be represented by serial numbers, which can correspond to the area coding data of the row / column. For example, the serial numbers can be consecutive integers starting from 0. The serial number of the first row separator line can be 0, and the serial number of the last column separator line is the last value. For example, if the table has 6 rows and 5 columns, the serial number of the first row separator line can be 0, the serial number of the last row separator line (i.e., the 7th separator line) can be 6, the serial number of the first column separator line is 7, and the serial number of the last column separator line is 12.

[0133] After determining the four dividing lines corresponding to a cell, you can obtain the cell's specific location, shape, size, etc. in the table, and thus obtain the entire table.

[0134] Optionally, the mapping relationship data can be obtained by decoding the separator line optimization data through the table assembly network 304. For example, the separator line optimization data can be processed first through the non-maximum suppression layer and the ordered regularization layer in the table assembly network 304 to obtain the processed separator line optimization data. For example, the non-maximum suppression layer can remove redundant separator lines, and the ordered regularization layer can divide the separator lines into two categories: horizontal (i.e., row separator lines) and vertical (i.e., column separator lines). Then, the row separator lines and column separator lines are sorted. For example, the row separator lines are sorted from top to bottom of the table, and the column separator lines are sorted from left to right of the table, with the column separator lines placed after the row separator lines.

[0135] Then, the decoder in network 304 is assembled through a table. A set of text embeddings set in the prior is used as the query information (i.e., Q), and the processed separator optimization data is used as the key information. The mapping relationship data is decoded to obtain the mapping relationship data. The mapping relationship data includes the serial numbers of the four separators surrounding each cell. For example, for the first cell in the table, it corresponds to the four separators with serial numbers 0, 1, 7 and 8. For the second cell in the table (the second one in the first row), it corresponds to the four separators with serial numbers 0, 1, 8 and 10 (spanning 1 column).

[0136] 2. Based on the mapping relationship data, assemble the separator lines according to the position of the cells in the table, restore the individual cells in the table, and obtain the table.

[0137] Optionally, once each cell and the four dividing lines surrounding each cell are determined, the structure of the table is also determined, thus obtaining the table after structural restoration.

[0138] In one example, after restoring the table, this embodiment of the application also supports restoring the content in the table, which may include the following: 1. Based on the fusion features corresponding to the input image, obtain the content of each cell and the position of the content in each cell.

[0139] Optionally, a separate fully connected layer can be set up to process the fusion features to obtain the content in each cell and the position of the content in each cell.

[0140] 2. For each cell, restore the content of the cell to the cell according to the position of the content in the cell to obtain the final table.

[0141] Once the table is finalized, the position of each cell can be determined. By comparing the position of the content with the position of the cells, the restored position of the content can be determined. Adding the content to the restored position will complete the content restoration, which will facilitate the subsequent retrieval of the content.

[0142] Alternatively, the content of the cell (such as the encoded data of the content) can be appended after the serial number of the separator line surrounding each cell. When the cell is restored, the content corresponding to that cell can also be restored. This can avoid the problem of content mismatch with cell, thereby improving the efficiency and accuracy of the final table restoration.

[0143] In summary, the technical solution provided in this application, due to the correspondence between cells and separator lines, optimizes the separator line data by using table structure information that indicates the distribution structure of each cell in the table. This optimization process provides rich structural information about the separator lines, indicated by the cell distribution structure, for reference, resulting in more accurate separator line data (i.e., optimized separator line data). In other words, the separator lines indicated by the optimized separator line data are more accurate, thereby improving the accuracy of determining the separator lines in the table and thus improving the accuracy of table reconstruction.

[0144] Please refer to Figure 15 It illustrates a flowchart of a table restoration method provided in another embodiment of this application, wherein the execution entity of each step of the method can be... Figure 1 The model shown uses device 20, and the method may include the following steps (1501~1505).

[0145] Step 1501: Obtain the input image, which includes a table.

[0146] Step 1501 is the same as step 1001 described above. For any content not described in the embodiments of this application, please refer to the above embodiments, and it will not be repeated here.

[0147] Step 1502: Based on the input image, obtain the table's separator data and table structure information. The separator data is used to indicate the separators in the table, and the table structure information is used to indicate the distribution structure of each cell in the table.

[0148] Optionally, the table structure information includes at least two of the following: cell information, row-column relationship information, and row-column attribute information. The cell information is used to indicate each cell in the table, the row-column relationship information is used to indicate the rows and columns that the table has, and the row-column attribute information is used to indicate the row and column to which each cell in the table belongs.

[0149] For example, the table structure information may include two types of information, such as cell information and row-column relationship information, row-column relationship information and row-column attribute information, or cell information and row-column relationship information. The table structure information may also include three types of information, such as cell information, row-column relationship information, and row-column attribute information. The table structure information may also include four types of information, such as cell information, row-column relationship information, row-column attribute information, and row-column deviation information; this embodiment of the application does not limit this. Among these, the row-column deviation information can be used to indicate the deviation between rows and the deviation between columns.

[0150] In one example, the table structure information may include a first part of information and a second part of information. For instance, if the table structure information includes M types of information, where M is a positive integer, the first part of information includes m types of information, where m is a positive integer, and the second part of information includes n types of information, where n is a positive integer, then the sum of n and m is less than or equal to M. This application does not limit the table structure information, the first part of information, or the second part of information.

[0151] For example, when the table structure information includes two types of information, the first part of the information includes one type of information and the second part of the information includes the other type of information. For instance, when the table structure information includes cell information and row-column relationship information, the first part of the information may include cell information and the second part of the information may include row-column relationship information; or, the first part of the information may include row-column relationship information and the second part of the information may include cell information. This application embodiment does not limit this.

[0152] When the table structure information includes three types of information, the first part of the information includes one type of information, and the second part of the information includes the other two types of information; or, the first part of the information includes two types of information, and the second part of the information includes another type of information. For example, when the table structure information includes cell information, row-column relationship information, and row-column attribute information, the first part of the information may include row-column attribute information, and the second part of the information may include cell information and row-column relationship information; or, the first part of the information may include cell information and row-column attribute information, and the second part of the information may include row-column relationship information. This application embodiment does not limit this.

[0153] In one example, a candidate component export network in the table reconstruction model extracts a table feature map from the input image. This table feature map is used to indicate the overall visual information of the table. The table feature map is then encoded to obtain the separator line data.

[0154] In one example, a candidate component derivation network in a table reconstruction model extracts multimodal features of the table from the input image. These multimodal features include at least two of the following features: visual features, geometric features, and text features. Visual features indicate the visual information of the text boxes, geometric features indicate the geometric structure of the text boxes, and text features indicate the content within the text boxes. The text boxes indicate the area where the content in the cells is located. The multimodal features are then fused to obtain fused features, which include interaction information between the various modal features and interaction information within different modal features. These fused features are used to indicate the various text boxes in the table. Finally, the fused features are classified to identify the table structure information.

[0155] The method for obtaining separator line data and table structure information is the same as step 1002 above. For content not described in the embodiments of this application, please refer to the above embodiments, and it will not be repeated here.

[0156] Step 1503: Optimize the separator line data based on the first part of the table structure information to obtain the separator line adjustment data.

[0157] The degree of matching between the separator line indicated by the separator line adjustment data and the table is greater than or equal to the degree of matching between the separator line indicated by the separator line data and the table. In other words, the accuracy of the separator line indicated by the separator line adjustment data is higher than or equal to the accuracy of the separator line indicated by the separator line data.

[0158] Optionally, by using a separator optimization network in the table reconstruction model, the separator data can be optimized based on the first part of the table structure information to obtain the separator adjustment data. Guided by the rich structural information about the separators indicated by the first part of the information, the accuracy of determining the table separators can be improved. Optionally, the separator optimization network includes a first separator optimization network and a second separator optimization network. The separator data can be optimized based on the first part of the table structure information using either the first or second separator optimization network to obtain the separator adjustment data.

[0159] In one example, taking the first optimization network using separators as an example, the process of obtaining separator adjustment data may include the following: 1. Encode the first part of the information to obtain the second encoded data.

[0160] The second encoded data refers to the encoded data of the first part of the information, which can be obtained through the first optimized network coding of the separator line.

[0161] Optionally, if the first part of the information includes one type of information, the first optimized network encodes this one type of information through a separator line to obtain the second encoded data. For example, if the first part of the information includes cell information / row-column relationship information / row-column attribute information, the second encoded data is obtained by encoding the cell information / row-column relationship information / row-column attribute information through a first optimized network encodes the cell information / row-column relationship information / row-column attribute information through a separator line.

[0162] When the first part of the information includes multiple pieces of information, these multiple pieces of information can be fused together (such as adding elements at corresponding positions) and then encoded to obtain the second encoded data. Alternatively, the multiple pieces of information can be encoded separately and then fused together to obtain the encoded sub-data to obtain the second encoded data. This application embodiment does not limit this approach.

[0163] For example, when the first part of the information includes two types of information, the two types of information are encoded using a first optimization network with separators to obtain second encoded data. Alternatively, the two types of information can be merged and then encoded to obtain second encoded data. Or, the two types of information can be encoded separately and then merged to obtain encoded sub-data, which is the second encoded data. For instance, when the first part of the information includes cell information and row / column relationship information, the cell information and row / column relationship information are merged and encoded using a first optimization network with separators to obtain second encoded data; when the first part of the information includes cell information and row / column attribute information, the cell information and row / column attribute information are merged and encoded using a first optimization network with separators to obtain second encoded data; when the first part of the information includes row / column relationship information and row / column attribute information, the row / column relationship information and row / column attribute information are merged and encoded using a first optimization network with separators to obtain second encoded data.

[0164] When the first part of the information includes three types of information, the three types of information are encoded using a first optimization network with separators to obtain the second encoded data. Alternatively, the three types of information can be encoded separately and then merged to obtain the encoded sub-data, which is the second encoded data. For example, when the first part of the information includes cell information, row-column relationship information, and row-column attribute information, the cell information, row-column relationship information, and row-column attribute information are merged together using the first optimization network with separators before encoding to obtain the second encoded data.

[0165] For example, refer to Figure 7 Taking the first part of the information, which includes row and column attribute information, as an example, this row and column attribute information can include row attribute information (i.e., which cells belong to the same row) and column attribute information (i.e., which cells belong to the same column). The first optimization network 302, through a separator line, converts the region-coded data corresponding to the row attribute information and the region-coded data corresponding to the column attribute information (i.e., the instance index) into embeddings to obtain the second encoded data (i.e.,...). and ).

[0166] 2. Merge the separator data and the second encoded data to obtain the second merged data.

[0167] Optionally, the second fused data can be obtained by adding the separator data and the second coded data element-wise at corresponding positions using the first optimized network coding. Compared to the separator data, the second fused data carries more information.

[0168] For example, refer to Figure 7 , divide the data Second encoded data (i.e.) and By adding the elements at the corresponding positions, the second fused data can be obtained. .

[0169] 3. Decode the second fused data to obtain the second decoded data.

[0170] The second fused data is decoded by the decoder in the first optimization network of the separator line to obtain the second decoded data. For example, the second fused data is used as key-value information (i.e., K and V) and a set of learnable embeddings is used as query information (i.e., Q) to obtain the second decoded data. The second decoded data can be used to indicate the separator line after optimization by the first part of the information.

[0171] For example, refer to Figure 7 The decoder in the first optimized network 302, separated by a separator, uses the second fused data. For key-value information, (i.e., a set of learnable embeddings) is the query information, which is then decoded to obtain the second decoded data.

[0172] 4. Based on the classification data corresponding to the second decoded data and the regression data corresponding to the second decoded data, the separator line adjustment data is obtained.

[0173] The classification data indicates the type of the dividing line, and the regression data indicates its location. Optionally, the second decoded data is classified through a fully connected layer in the first optimization network for the dividing line to determine the type of the dividing line (i.e., classification data), such as visually visible, non-visually visible, or boundary. The second decoded data is then regressed through another fully connected layer in the first optimization network for the dividing line to determine its location (i.e., regression data).

[0174] For example, refer to Figure 7 The two fully connected layers in the first optimization network 302, which separate the data, perform classification and position regression on the second decoded data, respectively, to obtain classification data Cls and regression data Reg.

[0175] Optionally, by concatenating the classification data corresponding to the second decoded data and the regression data corresponding to the second decoded data, the separator line adjustment data can be obtained. For example, by using the separator line first optimization network 302 to concatenate the regression data Reg corresponding to the second decoded data after the classification data Cls corresponding to the second decoded data, the separator line adjustment data can be obtained. .

[0176] In one example, considering complex table conditions (e.g., table distortion or warping), straight dividing lines are no longer suitable. Therefore, this embodiment uses a set of key points to represent the dividing lines. This set of key points includes at least three key points, making the candidate dividing line data and the actual dividing line data more accurate. For example, when the dividing lines are represented by multiple key points extracted from the dividing lines, the dividing line data may include the key point set data corresponding to each indicated dividing line. The dividing line adjustment data includes the key point set data corresponding to each indicated dividing line, and the dividing line optimization data includes the key point set data corresponding to each indicated dividing line. This key point set data includes the location data of the key points, which describes the location, such as coordinates.

[0177] In one example, taking the second optimization network using separators as an example, the process of obtaining separator adjustment data may include the following: 1. Using the separator line data as the query information, the first part of the information is the key-value information, and the second decoded data is obtained by decoding.

[0178] Optionally, the second decoded data is obtained by using the coding layer (which acts as a decoder in this case) in the second optimized network of the separator line as the query information and the first part of the information as the key-value information.

[0179] For example, the first part of the information includes m types of information. When m equals 1, the separator line data is used as the query information and the first part of the information is used as the key-value information to decode and obtain the second decoded data. When m is greater than 1, for the j-th type of information among the m types of information, the separator line data is used as the query information and the j-th type of information is used as the key-value information to decode and obtain the decoded sub-data corresponding to the j-th type of information. The decoded sub-data corresponding to the m types of information are fused to obtain the second decoded data, where j is a positive integer less than or equal to m.

[0180] For example, if the first part of the information includes cell information / row-column relationship information / row-column attribute information, the second decoded data is obtained by decoding the cell information / row-column relationship information / row-column attribute information through the encoding layer in the second optimization network with separators.

[0181] When the first part of the information includes two types of information, the two types of information are decoded separately using two different encoding circuits in the encoding layer to obtain the decoded sub-data corresponding to each of the two types of information. These decoded sub-data are then fused together to obtain the second decoded data. For example, when the first part of the information includes cell information and row / column relationship information, the two types of information are decoded separately using two different encoding circuits in the encoding layer to obtain the decoded sub-data corresponding to each of the two types of information. Similarly, when the first part of the information includes cell information and row / column attribute information, the two types of information are decoded separately using two different encoding circuits in the encoding layer to obtain the decoded sub-data corresponding to each of the two types of information. These decoded sub-data are then fused together to obtain the second decoded data. Likewise, when the first part of the information includes row / column relationship information and row / column attribute information, the two types of information are decoded separately using two different encoding circuits in the encoding layer to obtain the decoded sub-data corresponding to each of the two types of information. These decoded sub-data are then fused together to obtain the second decoded data.

[0182] When the first part of the information includes three types of information, the three types of information are decoded separately using three different encoding circuits in the encoding layer to obtain the decoded sub-data corresponding to each of the three types of information. These decoded sub-data are then fused together to obtain the second decoded data. For example, when the first part of the information includes cell information, row-column relationship information, and row-column attribute information, these three types of information are decoded separately using three different encoding circuits in the encoding layer to obtain the decoded sub-data corresponding to each of the three types of information. These decoded sub-data are then fused together to obtain the second decoded data.

[0183] For example, refer to Figure 8 Taking the first part of the information, which includes cell information and row-column relationship information, as an example, the first encoder in the second optimization network 303 (such as the left-hand encoding line) uses the separator line data. For query information, cell information For key-value information, decoded data yields decoded sub-data Y1, and the data is processed by the second encoder (such as the right-hand encoding line) in the second optimization network 303, using the separator line data. For querying information, row and column relationship information The key-value information is decoded to obtain decoded sub-data Y2. Then, the decoded sub-data Y1 and decoded sub-data Y2 are fused and normalized through the normalization layer in the second optimization network 303 to obtain the second decoded data.

[0184] 2. Based on the classification data corresponding to the second decoded data and the regression data corresponding to the second decoded data, the separator line adjustment data is obtained.

[0185] Optionally, the second decoded data is classified through a fully connected layer in the second optimization network of the separator line to determine the type of separator line (i.e., classified data), and the second decoded data is regressed through another fully connected layer in the second optimization network of the separator line to determine the position of the separator line (i.e., regressed data). For example, refer to Figure 8 The two fully connected layers in the second optimization network 303, which separate the dividing line, perform classification and position regression on the second decoded data, respectively, to obtain classification data Cls and regression data Reg.

[0186] Optionally, by concatenating the classification data corresponding to the second decoded data and the regression data corresponding to the second decoded data, the separator line adjustment data can be obtained. For example, by using the separator line second optimization network 303 to concatenate the regression data Reg corresponding to the second decoded data after the classification data Cls corresponding to the second decoded data, the separator line adjustment data can be obtained. .

[0187] Using one or more pieces of information from the table structure can enrich the information that the table reconstruction model can refer to, thereby improving the accuracy of determining the data for adjusting the separator lines, and thus improving the accuracy of table reconstruction.

[0188] Step 1504: Based on the second part of the information in the table structure information, optimize the separator line adjustment data to obtain the optimized separator line data.

[0189] The degree of matching between the separator line indicated by the optimized separator line data and the table is greater than or equal to the degree of matching between the separator line indicated by the adjusted separator line data and the table. In other words, the accuracy of the separator line indicated by the optimized separator line data is higher than or equal to the accuracy of the separator line indicated by the adjusted separator line data. Optionally, the optimized separator line data can be obtained by optimizing the adjusted separator line data based on the second part of the table structure information through the separator line optimization network in the table reconstruction model. Guided by the rich structural information on the separator lines indicated by the second part of the information, the accuracy of determining the table's separator lines can be further improved. Optionally, the separator line optimization network includes a first separator line optimization network and a second separator line optimization network. The optimized separator line data can be obtained by optimizing the adjusted separator line data based on the second part of the table structure information through either the first or second separator line optimization network.

[0190] In one example, taking the first optimization network using separators as an example, the process of obtaining separator optimization data may include the following: 1. Encode the second part of the information to obtain the third encoded data.

[0191] The third encoded data refers to the encoded data of the second part of the information, which can be obtained through the first optimized network coding of the separator line.

[0192] Optionally, if the second part of the information includes one type of information, the first optimization network through the separator line encodes that one type of information to obtain the third encoded data. For example, if the second part of the information includes cell information / row-column relationship information / row-column attribute information, the first optimization network through the separator line encodes the cell information / row-column relationship information / row-column attribute information to obtain the third encoded data.

[0193] When the second part of the information includes multiple pieces of information, these multiple pieces of information can be merged (e.g., the corresponding positional elements are added together) and then encoded to obtain the third encoded data. Alternatively, the multiple pieces of information can be encoded separately and then merged to obtain the encoded sub-data to obtain the third encoded data. This application embodiment does not limit this approach.

[0194] For example, when the second part of the information includes two types of information, the two types of information are encoded using a first optimization network with separators to obtain third encoded data. Alternatively, the two types of information can be encoded separately and then merged to obtain encoded sub-data, which is the third encoded data. For instance, when the second part of the information includes cell information and row / column relationship information, the cell information and row / column relationship information are merged using a first optimization network with separators before encoding to obtain third encoded data; when the second part of the information includes cell information and row / column attribute information, the cell information and row / column attribute information are merged using a first optimization network with separators before encoding to obtain third encoded data; when the second part of the information includes both row / column relationship information and row / column attribute information, the row / column relationship information and row / column attribute information are merged using a first optimization network with separators before encoding to obtain third encoded data.

[0195] When the second part of the information includes three types of information, the first optimization network using separators encodes these three types of information to obtain the third encoded data. Alternatively, the three types of information can be encoded separately and then merged to obtain the encoded sub-data, which is also the third encoded data. For example, when the second part of the information includes cell information, row-column relationship information, and row-column attribute information, the first optimization network using separators merges these three types of information before encoding to obtain the third encoded data.

[0196] 2. Merge the separator line adjustment data and the third encoded data to obtain the third merged data.

[0197] Optionally, the third fused data can be obtained by adding the separator line adjustment data and the third coded data element-wise at corresponding positions through the first optimized network coding. Compared to the separator line adjustment data, the third fused data carries more information.

[0198] 3. Decode the third fused data to obtain the third decoded data.

[0199] The third fused data is decoded by the decoder in the first optimization network of the separator line to obtain the third decoded data. For example, the third fused data is used as key-value information (i.e., K and V) and a set of learnable embeddings is used as query information (i.e., Q) to decode the third decoded data. The third decoded data can be used to indicate the separator line after optimization by the first part of information and the second part of information (i.e., table structure information).

[0200] 4. Based on the classification data corresponding to the third decoded data and the regression data corresponding to the third decoded data, the separator optimization data is obtained.

[0201] The classification data indicates the type of the dividing line, and the regression data indicates the location of the dividing line. Optionally, the third decoded data is classified through a fully connected layer in the first optimization network of the dividing line to determine the type of the dividing line (i.e., classification data). The third decoded data is then regressed through another fully connected layer in the first optimization network of the dividing line to determine the location of the dividing line (i.e., regression data).

[0202] Optionally, by concatenating the classification data corresponding to the third decoded data and the regression data corresponding to the third decoded data, the separator line optimized data can be obtained. For example, by using the separator line first optimization network to concatenate the regression data corresponding to the third decoded data after the classification data corresponding to the third decoded data, the separator line optimized data can be obtained. .

[0203] In one example, taking the second optimization network using separators as an example, the process of obtaining separator optimization data may include the following: 1. The data is adjusted by the separator line to form the query information, the second part of the information is the key-value information, and the third decoded data is obtained by decoding.

[0204] Optionally, the coding layer in the second optimization network (which acts as a decoder at this time) is used to adjust the data with the separator line as query information, and the second part of the information is key-value information, and the third decoded data is obtained by decoding.

[0205] For example, the second part of the information includes n types of information. When n equals 1, the data is adjusted with a separator line to form query information, and the second part of the information is key-value information. The third decoded data is obtained by decoding. When n is greater than 1, for the i-th type of information among the n types of information, the data is adjusted with a separator line to form query information, and the i-th type of information is key-value information. The decoded sub-data corresponding to the i-th type of information is obtained by decoding. The decoded sub-data corresponding to the n types of information are fused to obtain the third decoded data, where i is a positive integer less than or equal to n.

[0206] For example, if the second part of the information includes cell information / row-column relationship information / row-column attribute information, the cell information / row-column relationship information / row-column attribute information is decoded by the encoding layer in the second optimization network through the separator line to obtain the third decoded data.

[0207] When the second part of the information includes two types of information, the two types of information are decoded separately using two different encoding circuits in the encoding layer to obtain the decoded sub-data corresponding to each type of information. These decoded sub-data are then fused together to obtain the third decoded data. For example, when the second part of the information includes cell information and row / column relationship information, the cell information and row / column relationship information are decoded separately using two different encoding circuits in the encoding layer to obtain the decoded sub-data corresponding to each type of information. These decoded sub-data are then fused together to obtain the third decoded data. Similarly, when the second part of the information includes cell information and row / column attribute information, the cell information and row / column attribute information are decoded separately using two different encoding circuits in the encoding layer to obtain the decoded sub-data corresponding to each type of information. These decoded sub-data are then fused together to obtain the third decoded data.

[0208] When the second part of the information includes three types of information, the three types of information are decoded separately using three different encoding circuits in the encoding layer to obtain the decoded sub-data corresponding to each of the three types of information. These decoded sub-data are then fused together to obtain the third decoded data. For example, when the second part of the information includes cell information, row-column relationship information, and row-column attribute information, these three types of information are decoded separately using three different encoding circuits in the encoding layer to obtain the decoded sub-data corresponding to each of the three types of information. These decoded sub-data are then fused together to obtain the third decoded data.

[0209] 2. Based on the classification data corresponding to the third decoded data and the regression data corresponding to the third decoded data, the separator optimization data is obtained.

[0210] Optionally, the third decoded data is classified through a fully connected layer in the second optimization network of the separator line to determine the type of separator line (i.e., classified data), and the third decoded data is regressed through another fully connected layer in the second optimization network of the separator line to determine the position of the separator line (i.e., regressed data).

[0211] Optionally, by concatenating the classification data corresponding to the third decoded data and the regression data corresponding to the third decoded data, the separator line optimized data can be obtained. For example, by using a separator line second optimization network to concatenate the regression data corresponding to the third decoded data after the classification data corresponding to the third decoded data, the separator line optimized data can be obtained. .

[0212] Based on the data adjustment of the separator lines, another one or more pieces of information from the table structure are used to optimize the separator lines, further enriching the reference information and helping to improve the accuracy of separator line determination, thereby further improving the accuracy of table reconstruction.

[0213] Step 1505: Optimize the data based on the separator lines to restore each cell in the table, thus obtaining the table.

[0214] Optionally, the table is restored by optimizing the data based on the separator lines through the table assembly network in the table restoration model to restore each cell in the table. There is a correspondence between cells and separator lines. For example, one cell can correspond to four separator lines. Based on the correspondence between separator lines and cells, the installation position of each separator line can be determined, thereby avoiding the problem of misaligned separator line installation and enabling the restoration of cells that span rows / columns (e.g., cells that span rows / columns are only related to their corresponding separator lines), thus effectively improving the accuracy of table restoration.

[0215] In one example, the table restoration process can be as follows: decode the separator line optimization data to obtain mapping relationship data, which is used to indicate multiple separator lines associated with cells; based on the mapping relationship data, assemble the separator lines according to the position of the cells in the table, restore the individual cells in the table, and obtain the table.

[0216] In one example, after restoring the table, this application embodiment also supports restoring the content in the table. This process may include the following: obtaining the content in each cell and the position of the content in each cell according to the fusion features corresponding to the input image; for each cell, restoring the content in the cell to the cell according to the position of the content in the cell to obtain the final table.

[0217] Step 1505 is the same as step 1004 described above. For any content not described in the embodiments of this application, please refer to the above embodiments, and it will not be repeated here.

[0218] In summary, the technical solution provided in this application, due to the correspondence between cells and separator lines, optimizes the separator line data by using table structure information that indicates the distribution structure of each cell in the table. This optimization process provides rich structural information about the separator lines, indicated by the cell distribution structure, for reference, resulting in more accurate separator line data (i.e., optimized separator line data). In other words, the separator lines indicated by the optimized separator line data are more accurate, thereby improving the accuracy of determining the separator lines in the table and thus improving the accuracy of table reconstruction.

[0219] In addition, by optimizing the first and second parts of the table structure information, the separator data can be used locally with emphasis, rather than being used without emphasis as a whole. This allows the table structure information to be fully utilized, thereby improving the optimization effect of the separator data.

[0220] In some embodiments, reference Figure 2-9 The table reconstruction model shown, taking the first part of the information including row and column attribute information and the second part of the information including cell information and row and column relationship information as an example, illustrates the technical solution provided by the example in this application, which may include the following contents.

[0221] 1. Obtain the input image by restoring the table model 30. The input image includes a table.

[0222] 2. By reconstructing candidate components in model 30 through a table, network 301 is derived and the table's separator line data is obtained based on the input image. And table structure information.

[0223] Optionally, the table feature map F of the table is extracted from the input image by the feature extractor 2 in the extractor 301b, and the table feature map is encoded by the encoder in the extractor 301b to obtain the separator line data, denoted as F. The exporter 301a in the candidate component export network 301 extracts multimodal features of the table from the input image. These multimodal features include visual, geometric, and textual features. Multiple collaborative modules in exporter 301a fuse these multimodal features to obtain fused features. The output layer of the candidate component export network 301 splits the fused features to identify the table structure information, which includes cell information. Row and column relationship information and row and column attribute information .

[0224] 3. Restore the separator lines in model 30 using tables. The first optimization network 302 is based on row and column attribute information. Optimize separator data Obtain the separator line adjustment data. .

[0225] This row and column attribute information This can include row attribute information (i.e., which cells belong to the same row) and column attribute information. The first optimization network 302, using a separator line, converts the region-coded data corresponding to the row attribute information and the region-coded data corresponding to the column attribute information (i.e., the instance index) into embeddings to obtain the second encoded data (i.e.,...). and ), to separate the data Second encoded data (i.e.) and ), and add the elements at the corresponding positions to obtain the second fused data. The decoder in the first optimization network 302, through the separator line, is used to fuse the second data. For key-value information, (i.e., a set of learnable embeddings) represents the query information. Decoding yields the second decoded data. The second decoded data is then classified and regressed using two fully connected layers in the first optimization network 302, resulting in classification data Cls and regression data Reg. Finally, the first optimization network 302 concatenates the regression data Reg corresponding to the second decoded data after the classification data Cls, resulting in the adjusted data. .

[0226] 4. The second optimization network 303 restores the separator lines in model 30 through the table, based on cell information. and row and column relationship information Optimize the separator line adjustment data to obtain optimized separator line data. .

[0227] Data is adjusted by the separator line through the first encoder (such as the left-hand encoding line) in the second optimization network 303. For query information, cell information For the key-value information, decoded data yields decoded sub-data Y1, and the data is adjusted by the separator line through the second encoder (such as the right-hand encoding line) in the second optimization network 303. For querying information, row and column relationship information For key-value information, decoded sub-data Y2 is obtained. Then, through the normalization layer in the second optimization network 303, decoded sub-data Y1 and decoded sub-data Y2 are fused and normalized to obtain third decoded data. The third decoded data is then classified and regressed using two fully connected layers in the second optimization network 303, yielding classification data Cls and regression data Reg. Finally, the regression data Reg corresponding to the third decoded data is concatenated with the classification data Cls corresponding to the third decoded data using the second optimization network 303 to obtain the optimized data. .

[0228] 5. The table is restored by assembling the network 304 in the table restoration model 30 based on the separator line to restore each cell in the table and obtain the table.

[0229] First, the non-maximum suppression layer and ordered regularization layer in network 304 are assembled using a table to optimize the data for the separator lines. The data is processed to obtain the optimized separator line data. Then, the decoder in the table assembly network 304 decodes the data, using a set of prior text embeddings as the query information (i.e., Q), and the processed separator line optimizes the data. The key-value information is decoded to obtain mapping relationship data. Based on the mapping relationship data, the separator lines are assembled according to the position of the cells in the table, and the individual cells in the table are restored to obtain the table.

[0230] 6. Using the table restoration model 30, based on the fusion features corresponding to the input image, the content of each cell and the position of the content in each cell are obtained. For each cell, the content in the cell is restored to the cell according to the position of the content in the cell, and the final table is obtained.

[0231] Even after optimizing the separator lines based on row and column attribute information, issues such as redundancy (i.e., multiple detections), inaccuracy, and overly regularity may still exist. A second optimization, combining cell information and row / column relationship information, can correct these problems. This includes filtering redundant separator lines, correcting overly regular lines, and optimizing distorted lines, thereby further improving the accuracy of separator line determination and ultimately enhancing the accuracy of table reconstruction. Furthermore, separator lines spanning multiple rows / columns can also be reconstructed based on row and column attribute information and cell information.

[0232] In summary, the technical solution provided in this application, due to the correspondence between cells and separator lines, optimizes the separator line data by using table structure information that indicates the distribution structure of each cell in the table. This optimization process provides rich structural information about the separator lines, indicated by the cell distribution structure, for reference, resulting in more accurate separator line data (i.e., optimized separator line data). In other words, the separator lines indicated by the optimized separator line data are more accurate, thereby improving the accuracy of determining the separator lines in the table and thus improving the accuracy of table reconstruction.

[0233] The training method for the table restoration model will be described below. For content not described in the embodiments of this application, please refer to the above embodiments, and it will not be repeated here.

[0234] Please refer to Figure 16 It illustrates a flowchart of a training method for a table reconstruction model provided in one embodiment of this application. The execution entity of each step of the method can be... Figure 1 The model training device 10 in the implementation environment of the scheme shown can include the following steps (1601~1605).

[0235] Step 1601: Obtain a sample image and label data for the sample image, wherein the sample image includes a table and the label data is used to label the table.

[0236] This sample image serves as training data for a table reconstruction model; it can be an image of a table. The sample image is labeled with corresponding data, including a first sub-label and a second sub-label. The first sub-label is for the data on adjusting or optimizing separator lines, and may include data indicating the location and classification of the true separator lines (e.g., visually visible separator lines, non-visually visible separator lines, and boundary lines). The second sub-label is for the data on the mapping relationships, and may include the actual sequence numbers of the four separator lines surrounding the cell.

[0237] Step 1602: Based on the sample image, the table reconstruction model obtains the table's separator data and table structure information. The separator data is used to indicate the separators in the table, and the table structure information is used to indicate the distribution structure of each cell in the table.

[0238] Step 1603: Optimize the separator line data based on the table structure information using the table restoration model to obtain optimized separator line data.

[0239] Step 1604: Optimize the data by decoding the separator lines through the table restoration model to obtain mapping relationship data, which is used to indicate multiple separator lines associated with the cell.

[0240] The process of restoring the table in the sample image is the same as the process of restoring the table in the input image. Steps 1602-1603 in this embodiment can refer to steps 1002-1004 or steps 1502-1505 in the above embodiment, and will not be repeated here.

[0241] Step 1605: Based on the mapping relationship data and label data, train the table reconstruction model to obtain the trained table reconstruction model.

[0242] Optionally, a first loss is determined based on the difference between the mapping relationship data and the label data. This first loss is used to indicate the prediction accuracy of the mapping relationship between cells and separators. The parameters of the table reconstruction model are iteratively adjusted based on the first loss to obtain the trained table reconstruction model.

[0243] The label data here can refer to the second sub-label data in the label data. The third loss is used to measure the difference between the predicted sequence group (composed of the sequence numbers of the four separator lines corresponding to each cell) and the corresponding true sequence group in the label data. It can be calculated using loss functions such as Cross Entropy (CE), Binary Cross Entropy (BCE), Mean Absolute Loss (MAL), and Mean Square Error (MSE), etc. The embodiments of this application do not limit this.

[0244] The training process of the table reconstruction model is an iterative process. This embodiment of the application uses a certain iteration as an example for illustration. Optionally, one iteration corresponds to multiple batches of sample data, and each batch of sample data includes multiple sample images. After processing multiple batches of sample data, one epoch of iteration optimization is completed. The SGD (Stochastic Gradient Descent) gradient update method can be used for training. For example, if a learning rate of 0.005 is used for the parameters to be learned, the learning rate is reduced to 0.1 times the original value every 10 epochs. The total loss of the table reconstruction model is fed back to the neural network of the table reconstruction model to calculate the gradient of each parameter. Based on the gradient of each parameter, the parameters are updated.

[0245] For example, the termination condition for iterative optimization includes at least one of the following: minimizing the total loss of the table restoration model, the number of iterations being greater than or equal to a threshold, the total loss of the table restoration model being less than or equal to a threshold, etc., and the embodiments of this application do not limit this.

[0246] For example, the first loss is used as the total loss of the table reconstruction model. With the goal of minimizing the total loss of the table reconstruction model, the parameters of the table reconstruction model are iteratively adjusted to obtain the trained table reconstruction model.

[0247] In one example, prior to step 1605, the loss of the table reconstruction model can be constructed based on the separator line optimized data and label data to assist the first loss in training the table reconstruction model. This process may include the following: 1. Based on the difference between the separator optimization data and the label data, determine the second loss, which is used to indicate the prediction accuracy of the separator.

[0248] The label data here can refer to the first sub-label data in the label data, that is, the difference between the optimized data based on the separator line and the first sub-label data is used to determine the second loss. The second loss includes classification loss and regression loss. The regression loss is used to measure the positional difference between the separator line indicated by the optimized data and the true separator line. The regression loss can be calculated using functions such as Mean Absolute Loss (MAL) and Mean Square Error (MSE). This application does not limit this.

[0249] The classification loss is used to measure the type difference between the separator indicated by the separator optimization data and the true separator. The regression loss can be calculated using loss functions such as cross-entropy (CE), binary cross-entropy (BCE), mean absolute error loss, mean squared error loss, etc. The embodiments of this application do not limit this.

[0250] Optionally, the sum of the regression loss and the classification loss can be determined as the second loss.

[0251] Optionally, before constructing the second loss, the LSF (Least Squares Fitting) algorithm can be used to optimize the separator line optimization data, resulting in optimized separator line optimization data. This optimized data is then used to determine the second loss, thus obtaining a more accurate second loss.

[0252] 2. The parameters of the table reconstruction model are iteratively adjusted based on the first loss and the second loss to obtain the trained table reconstruction model.

[0253] Optionally, the first loss and the second loss are summed to obtain the total loss of the table reconstruction model. The parameters of the table reconstruction model are iteratively adjusted with the goal of minimizing the total loss of the table reconstruction model to obtain the trained table reconstruction model.

[0254] In one example, the process of obtaining separator optimization data includes the following: optimizing separator data based on the first part of the table structure information using a table restoration model to obtain separator adjustment data; optimizing separator adjustment data based on the second part of the table structure information using a table restoration model to obtain separator optimization data; wherein, the first part of the information includes m types of information in the table structure information, and the second part of the information includes n types of information in the table structure information, where m is a positive integer and n is a positive integer.

[0255] That is, after the separator line data is optimized twice by the table restoration model, the optimized separator line data is obtained. Then, step 1605 may also include the following: 1. Based on the difference between the mapping relationship data and the label data, a first loss is determined, which is used to indicate the prediction accuracy of the mapping relationship between cells and separator lines.

[0256] 2. Based on the difference between the separator optimization data and the label data, determine the second loss, which is used to indicate the prediction accuracy of the separator.

[0257] 3. Based on the difference between the adjusting data and the label data for the separator line, a third loss is determined, which is used to indicate the prediction accuracy of the separator line.

[0258] The label data here can refer to the first sub-label data in the label data, that is, the difference between the adjusted data and the first sub-label data, which determines the third loss. This third loss can also include classification loss and regression loss. The regression loss measures the positional difference between the adjusted data and the true dividing line. This regression loss can be calculated using functions such as mean absolute error loss function and mean squared error loss function; this embodiment does not limit this. The classification loss measures the type difference between the adjusted data and the true dividing line. This regression loss can be calculated using functions such as cross-entropy loss function, binary cross-entropy loss function, mean absolute error loss function, and mean squared error loss function; this embodiment does not limit this.

[0259] Optionally, the sum of the regression loss and the classification loss can be determined as the third loss.

[0260] Optionally, before constructing the third loss, the LSF (Least Squares Fitting) algorithm can be used to optimize the separator line adjustment data, resulting in optimized separator line adjustment data. This optimized data is then used to determine the third loss, leading to a more accurate third loss.

[0261] 4. Iteratively adjust the parameters of the table reconstruction model based on the first loss, the second loss, and the third loss to obtain the trained table reconstruction model.

[0262] Optionally, the first loss, second loss, and third loss are summed to obtain the total loss of the table reconstruction model. The parameters of the table reconstruction model are iteratively adjusted with the goal of minimizing this total loss, resulting in the trained table reconstruction model. Training the table reconstruction model by combining mapping relationship data, separator optimization data, and separator adjustment data enriches the reference data during training compared to single mapping relationship data, thus accelerating the learning process and improving the training efficiency of the table reconstruction model.

[0263] In a feasible example, step 1605 may also include the following: determining a first loss based on the difference between the mapping relationship data and the label data, the first loss being used to indicate the prediction accuracy of the mapping relationship between cells and separators; determining a third loss based on the difference between the separator adjustment data and the label data, the third loss being used to indicate the prediction accuracy of the separators; iteratively adjusting the parameters of the table reconstruction model based on the first loss and the third loss to obtain the trained table reconstruction model.

[0264] Optionally, the first loss and the third loss are summed to obtain the total loss of the table reconstruction model. The parameters of the table reconstruction model are iteratively adjusted with the goal of minimizing the total loss of the table reconstruction model to obtain the trained table reconstruction model.

[0265] In summary, the technical solution provided in this application, due to the correspondence between cells and separator lines, optimizes the separator line data by using table structure information indicating the distribution structure of each cell in the table. This allows for the reference of rich structural information about the separator lines indicated by the cell distribution structure during the separator line data optimization process, resulting in more accurate separator line data (i.e., optimized separator line data). This improves the learning accuracy of the table reconstruction model, thereby increasing the accuracy of determining the table separator lines and ultimately improving the accuracy of table reconstruction.

[0266] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0267] refer to Figure 17 This diagram illustrates a block diagram of a table restoration apparatus according to an embodiment of this application. The apparatus has the functionality to implement the method example described above; the apparatus can be the computer device described above, or it can be installed within a computer device. Figure 17 As shown, the device 1700 includes: an input image acquisition module 1701, a table data acquisition module 1702, an optimized data acquisition module 1703, and a cell restoration module 1704.

[0268] The input image acquisition module 1701 is used to acquire an input image, which includes a table.

[0269] The table data acquisition module 1702 is used to acquire the separator line data and table structure information of the table based on the input image. The separator line data is used to indicate the separator lines in the table, and the table structure information is used to indicate the distribution structure of each cell in the table.

[0270] The data acquisition optimization module 1703 is used to optimize the separator line data based on the table structure information to obtain optimized separator line data.

[0271] The cell restoration module 1704 is used to restore each cell in the table based on the data optimization of the separator line, so as to obtain the table.

[0272] In some embodiments, the optimized data acquisition module 1703 is used for: Encode the table structure information to obtain the first encoded data; The separator line data and the first encoded data are fused to obtain the first fused data; Decode the first fused data to obtain the first decoded data; Based on the classification data corresponding to the first decoded data and the regression data corresponding to the first decoded data, the optimized data for the dividing line is obtained; wherein, the classification data is used to indicate the type of the dividing line, and the regression data is used to indicate the position of the dividing line.

[0273] In some embodiments, the table structure information includes at least one of the following: cell information, row-column relationship information, and row-column attribute information, wherein the cell information is used to indicate each cell in the table, the row-column relationship information is used to indicate the rows and columns that the table has, and the row-column attribute information is used to indicate the row and column to which each cell in the table belongs.

[0274] In some embodiments, the optimized data acquisition module 1703 is further configured to: optimize the separator line data according to the first part of the information in the table structure information to obtain separator line adjustment data; optimize the separator line adjustment data according to the second part of the information in the table structure information to obtain separator line optimized data; wherein the first part of the information includes m types of information in the table structure information, and the second part of the information includes n types of information in the table structure information, where m is a positive integer and n is a positive integer.

[0275] In some embodiments, the optimized data acquisition module 1703 is further configured to: Encode the first part of the information to obtain second encoded data; fuse the separator line data and the second encoded data to obtain second fused data; decode the second fused data to obtain second decoded data; based on the classification data corresponding to the second decoded data and the regression data corresponding to the second decoded data, obtain the separator line adjustment data; wherein, the classification data is used to indicate the type of the separator line, and the regression data is used to indicate the position of the separator line; Alternatively, using the separator line data as query information and the first part of the information as key-value information, the second decoded data is obtained by decoding; based on the classification data corresponding to the second decoded data and the regression data corresponding to the second decoded data, the separator line adjustment data is obtained.

[0276] In some embodiments, the optimized data acquisition module 1703 is further configured to: When m equals 1, the separator line data is used as the query information, the first part of the information is the key value information, and the second decoded data is obtained by decoding. Alternatively, when m is greater than 1, for the j-th type of information among the m types of information, the separator line data is used as the query information and the j-th type of information is used as the key value information to decode the corresponding decoded sub-data; the decoded sub-data corresponding to the m types of information are fused to obtain the second decoded data, where j is a positive integer less than or equal to m.

[0277] In some embodiments, the optimized data acquisition module 1703 is further configured to: The second part of the information is encoded to obtain third encoded data; the separator adjustment data and the third encoded data are fused to obtain third fused data; the third fused data is decoded to obtain third decoded data; the separator optimization data is obtained based on the classification data corresponding to the third decoded data and the regression data corresponding to the third decoded data; wherein, the classification data is used to indicate the type of the separator, and the regression data is used to indicate the position of the separator. Alternatively, the separator adjustment data can be used as query information, and the second part of the information can be used as key-value information to decode and obtain third decoded data; based on the classification data corresponding to the third decoded data and the regression data corresponding to the third decoded data, the separator optimization data can be obtained.

[0278] In some embodiments, the optimized data acquisition module 1703 is further configured to: When n equals 1, the data is adjusted to query information using the separator line, the second part of the information is key-value information, and the third decoded data is obtained by decoding. Alternatively, when n is greater than 1, for the i-th type of information among the n types of information, the data adjusted by the separator line is used as query information, and the i-th type of information is used as key-value information. The decoded sub-data corresponding to the i-th type of information is obtained by decoding; the decoded sub-data corresponding to the n types of information are fused together to obtain the third decoded data, where i is a positive integer less than or equal to n.

[0279] In some embodiments, the table structure information includes at least two of the following: cell information, row-column relationship information, and row-column attribute information. The cell information is used to indicate each cell in the table, the row-column relationship information is used to indicate the rows and columns that the table has, and the row-column attribute information is used to indicate the row and column to which each cell in the table belongs.

[0280] In some embodiments, the cell restoration module 1704 is used for: Decode the separator line optimization data to obtain mapping relationship data, which is used to indicate multiple separator lines related to the cell; Based on the mapping relationship data, the separator lines are assembled according to the positions of the cells in the table, and the individual cells in the table are restored to obtain the table.

[0281] In some embodiments, the table data acquisition module 1702 is configured to: The table feature map of the table is extracted from the input image, and the table feature map is used to indicate the overall visual information of the table; The table feature map is encoded to obtain the separator line data.

[0282] In some embodiments, the table data acquisition module 1702 is further configured to: The multimodal features of the table are extracted from the input image. The multimodal features include at least two of the following features: visual features, geometric features, and text features. The visual features are used to indicate the visual information of the text box, the geometric features are used to indicate the geometric structure of the text box, and the text features are used to indicate the content in the text box. The text box is used to indicate the area where the content in the cell is located. The multimodal features are fused to obtain fused features, which include interaction information between various modal features and interaction information within different modal features. The fused features are used to indicate various text boxes in the table. The table structure information of the table is obtained by classifying the fused features.

[0283] In some embodiments, the device 1700 further includes a content data acquisition module and a content data restoration module (not shown in the figure).

[0284] The content data acquisition module is used to acquire the content in each cell and the position of the content in each cell based on the fusion features.

[0285] The content data restoration module is used to restore the content in each cell to the cell according to the position of the content in the cell, so as to obtain the final table.

[0286] In some embodiments, the table is reconstructed by a table reconstruction model, which includes a candidate component export network, a separator optimization network, and a table assembly network. The candidate component export network is used to obtain separator data and table structure information of the table based on the input image. The separator optimization network is used to optimize the separator data according to the table structure information to obtain the optimized separator data. The table assembly network is used to reconstruct each cell in the table based on the optimized separator data to obtain the table.

[0287] In summary, the technical solution provided in this application, due to the correspondence between cells and separator lines, optimizes the separator line data by using table structure information that indicates the distribution structure of each cell in the table. This optimization process provides rich structural information about the separator lines, indicated by the cell distribution structure, for reference, resulting in more accurate separator line data (i.e., optimized separator line data). In other words, the separator lines indicated by the optimized separator line data are more accurate, thereby improving the accuracy of determining the separator lines in the table and thus improving the accuracy of table reconstruction.

[0288] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0289] refer to Figure 18 This diagram illustrates a block diagram of a training apparatus for a table reconstruction model according to an embodiment of this application. The apparatus has the functionality to implement the method example described above; the apparatus can be the computer device described above, or it can be integrated into a computer device. Figure 18 As shown, the device 1800 includes: a sample image acquisition module 1801, a table data acquisition module 1802, an optimization data acquisition module 1803, a relational data acquisition module 1804, and a restoration model training module 1805.

[0290] The sample image acquisition module 1801 is used to acquire a sample image and label data of the sample image, wherein the sample image includes a table and the label data is used to label the table.

[0291] The table data acquisition module 1802 is used to acquire the separator line data and table structure information of the table based on the sample image through the table restoration model. The separator line data is used to indicate the separator lines in the table, and the table structure information is used to indicate the distribution structure of each cell in the table.

[0292] The optimized data acquisition module 1803 is used to optimize the separator line data based on the table structure information through the table restoration model to obtain optimized separator line data.

[0293] The relational data acquisition module 1804 is used to decode the mapping relational data by optimizing the data based on the separator lines through the table restoration model. The mapping relational data is used to indicate multiple separator lines related to the cell.

[0294] The table restoration model training module 1805 is used to train the table restoration model based on the mapping relationship data and the label data, so as to obtain the trained table restoration model.

[0295] In some embodiments, the model training module 1805 is configured to: Based on the difference between the mapping relationship data and the label data, a first loss is determined, which is used to indicate the prediction accuracy of the mapping relationship between cells and separator lines; The parameters of the table reconstruction model are iteratively adjusted based on the first loss to obtain the trained table reconstruction model.

[0296] In some embodiments, the model training module 1805 is further configured to: Based on the difference between the optimized data of the separator line and the label data, a second loss is determined, which is used to indicate the prediction accuracy of the separator line. The parameters of the table reconstruction model are iteratively adjusted based on the first loss and the second loss to obtain the trained table reconstruction model.

[0297] In some embodiments, the optimized data acquisition module 1803 is configured to: optimize the separator line data based on a first part of the information in the table structure information using the table restoration model to obtain separator line adjustment data; and optimize the separator line adjustment data based on a second part of the information in the table structure information using the table restoration model to obtain the separator line optimized data; wherein the first part of the information includes m types of information in the table structure information, and the second part of the information includes n types of information in the table structure information, where m is a positive integer and n is a positive integer.

[0298] In some embodiments, the model training module 1805 is further configured to: Based on the difference between the mapping relationship data and the label data, a first loss is determined, which is used to indicate the prediction accuracy of the mapping relationship between cells and separator lines; Based on the difference between the optimized data of the separator line and the label data, a second loss is determined, which is used to indicate the prediction accuracy of the separator line. Based on the difference between the separator line adjustment data and the label data, a third loss is determined, which is used to indicate the prediction accuracy of the separator line. The parameters of the table reconstruction model are iteratively adjusted based on the first loss, the second loss, and the third loss to obtain the trained table reconstruction model.

[0299] In some embodiments, the model training module 1805 is further configured to: Based on the difference between the mapping relationship data and the label data, a first loss is determined, which is used to indicate the prediction accuracy of the mapping relationship between cells and separator lines; Based on the difference between the separator line adjustment data and the label data, a third loss is determined, which is used to indicate the prediction accuracy of the separator line. The parameters of the table reconstruction model are iteratively adjusted based on the first loss and the third loss to obtain the trained table reconstruction model.

[0300] In summary, the technical solution provided in this application, due to the correspondence between cells and separator lines, optimizes the separator line data by using table structure information indicating the distribution structure of each cell in the table. This allows for the reference of rich structural information about the separator lines indicated by the cell distribution structure during the separator line data optimization process, resulting in more accurate separator line data (i.e., optimized separator line data). This improves the learning accuracy of the table reconstruction model, thereby increasing the accuracy of determining the table separator lines and ultimately improving the accuracy of table reconstruction.

[0301] Please refer to Figure 19 This illustration shows a schematic diagram of a computer device according to an embodiment of this application. The computer device can be any electronic device with data computing, processing, and storage functions, and can be implemented as... Figure 1 The model training device 10 or model usage device 20 in the implementation environment of the scheme shown. Specifically, it may include the following:

[0302] The computer device 1900 includes a central processing unit (such as a CPU, GPU, or FPGA) 1901, a system memory 1904 including RAM (Random-Access Memory) 1902 and ROM (Read-Only Memory) 1903, and a system bus 1905 connecting the system memory 1904 and the central processing unit 1901. The computer device 1900 also includes a basic input / output system 1906 to facilitate information transfer between various devices within the server, and a mass storage device 1907 for storing the operating system 1913, application programs 1914, and other program modules 1915.

[0303] The basic input / output system 1906 includes a display 1908 for displaying information and an input device 1909 for user input, such as a mouse or keyboard. Both the display 1908 and the input device 1909 are connected to the central processing unit 1901 via an input / output controller 1910 connected to the system bus 1905. The basic input / output system 1906 may also include the input / output controller 1910 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1910 also provides output to a display screen, printer, or other types of output devices.

[0304] The mass storage device 1907 is connected to the central processing unit 1901 via a mass storage controller (not shown) connected to the system bus 1905. The mass storage device 1907 and its associated computer-readable media provide non-volatile storage for the computer device 1900. That is, the mass storage device 1907 may include computer-readable media (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0305] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the above-mentioned types. The system memory 1904 and mass storage device 1907 described above can be collectively referred to as memory.

[0306] According to embodiments of this application, the computer device 1900 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1900 can be connected to the network 1912 via the network interface unit 1911 connected to the system bus 1905, or the network interface unit 1911 can be used to connect to other types of networks or remote computer systems (not shown).

[0307] The memory also includes a computer program stored in the memory and configured to be executed by one or more processors to implement the above-described table restoration method or the training method for the above-described table restoration model.

[0308] In some embodiments, a computer-readable storage medium is also provided, wherein a computer program is stored therein, which, when executed by a processor, implements the above-described table restoration method or the training method for the above-described table restoration model.

[0309] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0310] In some embodiments, a computer program product is also provided, the computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the table restoration method or the table restoration model training method described above.

[0311] It should be noted that, in this application embodiment, before and during the collection of user-related data, a prompt interface, pop-up window, or voice prompt message can be displayed. This prompt interface, pop-up window, or voice prompt message is used to inform the user that their relevant data is being collected. This ensures that the application only begins executing the steps related to acquiring user-related data after receiving confirmation from the user regarding the prompt interface or pop-up window; otherwise (i.e., without receiving confirmation from the user), the steps to acquire user-related data end, meaning no user-related data is acquired. In other words, all user data collected in this application is processed strictly in accordance with the requirements of relevant national laws and regulations. The acquisition of informed consent or separate consent from the personal information subject is conducted with the user's consent and authorization. Subsequent data use and processing are carried out within the scope of laws and regulations and the authorization of the personal information subject. Furthermore, the collection, use, and processing of relevant user data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the input images, sample images, and tables involved in this application are all obtained with full authorization.

[0312] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0313] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for restoring a table, characterized in that, The method includes: Obtain an input image, which includes a table; Based on the input image, the separator line data and table structure information of the table are obtained. The separator line data is used to indicate the separator lines in the table, and the table structure information is used to indicate the distribution structure of each cell in the table. Based on the table structure information, the separator line data is optimized to obtain optimized separator line data; Based on the separator line, the data is optimized to restore each cell in the table, thus obtaining the table.

2. The method according to claim 1, characterized in that, The step of optimizing the separator line data based on the table structure information to obtain optimized separator line data includes: Encode the table structure information to obtain the first encoded data; The separator line data and the first encoded data are fused to obtain the first fused data; Decode the first fused data to obtain the first decoded data; Based on the classification data corresponding to the first decoded data and the regression data corresponding to the first decoded data, the optimized separator data is obtained; The classification data is used to indicate the type of the dividing line, and the regression data is used to indicate the position of the dividing line.

3. The method according to claim 2, characterized in that, The table structure information includes at least one of the following: cell information, row-column relationship information, and row-column attribute information. The cell information is used to indicate each cell in the table, the row-column relationship information is used to indicate the rows and columns that the table has, and the row-column attribute information is used to indicate the row and column to which each cell in the table belongs.

4. The method according to claim 1, characterized in that, The step of optimizing the separator line data based on the table structure information to obtain optimized separator line data includes: Based on the first part of the information in the table structure information, the separator line data is optimized to obtain separator line adjustment data; Based on the second part of the information in the table structure information, the separator line adjustment data is optimized to obtain the separator line optimized data; The first part of the information includes m types of information in the table structure information, and the second part of the information includes n types of information in the table structure information, where m is a positive integer and n is a positive integer.

5. The method according to claim 4, characterized in that, The step of optimizing the separator line data based on the first part of the table structure information to obtain separator line adjustment data includes: Encode the first part of the information to obtain second encoded data; fuse the separator line data and the second encoded data to obtain second fused data; decode the second fused data to obtain second decoded data; based on the classification data corresponding to the second decoded data and the regression data corresponding to the second decoded data, obtain the separator line adjustment data; wherein, the classification data is used to indicate the type of the separator line, and the regression data is used to indicate the position of the separator line; or, Using the separator line data as query information and the first part of the information as key-value information, the second decoded data is obtained by decoding; based on the classification data corresponding to the second decoded data and the regression data corresponding to the second decoded data, the separator line adjustment data is obtained.

6. The method according to claim 5, characterized in that, The second decoded data is obtained by decoding the data using the separator line as query information and the first part of the information as key-value information, including: When m equals 1, the separator line data is used as the query information, the first part of the information is the key value information, and the second decoded data is obtained by decoding. or, When m is greater than 1, for the j-th type of information among the m types of information, the separator line data is used as the query information and the j-th type of information is used as the key value information to decode the corresponding decoded sub-data of the j-th type of information; the decoded sub-data corresponding to the m types of information are fused to obtain the second decoded data, where j is a positive integer less than or equal to m.

7. The method according to claim 4, characterized in that, The step of optimizing the separator line adjustment data based on the second part of the table structure information to obtain the optimized separator line data includes: The second part of the information is encoded to obtain third encoded data; the separator adjustment data and the third encoded data are fused to obtain third fused data; the third fused data is decoded to obtain third decoded data; the separator optimization data is obtained based on the classification data corresponding to the third decoded data and the regression data corresponding to the third decoded data; wherein, the classification data is used to indicate the type of the separator, and the regression data is used to indicate the position of the separator. or, Using the adjusted separator data as query information and the second part of the information as key-value information, the third decoded data is obtained by decoding; based on the classification data corresponding to the third decoded data and the regression data corresponding to the third decoded data, the optimized separator data is obtained.

8. The method according to claim 7, characterized in that, The data adjusted by the separator line is used as query information, and the second part of the information is key-value information. The third decoded data is obtained by decoding, including: When n equals 1, the data is adjusted to query information using the separator line, the second part of the information is key-value information, and the third decoded data is obtained by decoding. or, When n is greater than 1, for the i-th type of information among the n types of information, the data adjusted by the separator line is used as query information, and the i-th type of information is used as key-value information. The decoded sub-data corresponding to the i-th type of information is obtained by decoding. The decoded sub-data corresponding to the n types of information are fused to obtain the third decoded data, where i is a positive integer less than or equal to n.

9. The method according to claim 4, characterized in that, The table structure information includes at least two of the following: cell information, row-column relationship information, and row-column attribute information. The cell information is used to indicate each cell in the table, the row-column relationship information is used to indicate the rows and columns that the table has, and the row-column attribute information is used to indicate the row and column to which each cell in the table belongs.

10. The method according to any one of claims 1 to 9, characterized in that, The process of optimizing the data based on the separator line to restore each cell in the table, resulting in the table, includes: Decode the separator line optimization data to obtain mapping relationship data, which is used to indicate multiple separator lines related to the cell; Based on the mapping relationship data, the separator lines are assembled according to the positions of the cells in the table, and the individual cells in the table are restored to obtain the table.

11. The method according to any one of claims 1 to 9, characterized in that, The step of obtaining the table structure information of the table based on the input image includes: The multimodal features of the table are extracted from the input image. The multimodal features include at least two of the following features: visual features, geometric features, and text features. The visual features are used to indicate the visual information of the text box, the geometric features are used to indicate the geometric structure of the text box, and the text features are used to indicate the content in the text box. The text box is used to indicate the area where the content in the cell is located. The multimodal features are fused to obtain fused features, which include interaction information between various modal features and interaction information within different modal features. The fused features are used to indicate various text boxes in the table. The table structure information of the table is obtained by classifying the fused features.

12. A training method for a table reconstruction model, characterized in that, The method includes: Obtain a sample image and label data for the sample image, wherein the sample image includes a table and the label data is used to label the table; Based on the sample image, the table reconstruction model obtains the separator line data and table structure information of the table. The separator line data is used to indicate the separator lines in the table, and the table structure information is used to indicate the distribution structure of each cell in the table. The separator line data is optimized based on the table structure information using the table restoration model to obtain optimized separator line data. The table restoration model decodes the separator line optimization data to obtain mapping relationship data, which is used to indicate multiple separator lines related to the cell. Based on the mapping relationship data and the label data, the table reconstruction model is trained to obtain the trained table reconstruction model.

13. The method according to claim 12, characterized in that, The step of training the table reconstruction model based on the mapping relationship data and the label data to obtain the trained table reconstruction model includes: Based on the difference between the mapping relationship data and the label data, a first loss is determined, which is used to indicate the prediction accuracy of the mapping relationship between cells and separator lines; The parameters of the table reconstruction model are iteratively adjusted based on the first loss to obtain the trained table reconstruction model.

14. The method according to any one of claims 12 to 13, characterized in that, The step of optimizing the separator line data based on the table structure information using the table restoration model to obtain optimized separator line data includes: Based on the first part of the information in the table structure information, the table restoration model optimizes the separator line data to obtain separator line adjustment data. Based on the second part of the table structure information, the table restoration model optimizes the separator line adjustment data to obtain the optimized separator line data. The first part of the information includes m types of information in the table structure information, and the second part of the information includes n types of information in the table structure information, where m is a positive integer and n is a positive integer.

15. The method according to claim 14, characterized in that, The step of training the table reconstruction model based on the mapping relationship data and the label data to obtain the trained table reconstruction model includes: Based on the difference between the mapping relationship data and the label data, a first loss is determined, which is used to indicate the prediction accuracy of the mapping relationship between cells and separator lines; Based on the difference between the optimized data of the separator line and the label data, a second loss is determined, which is used to indicate the prediction accuracy of the separator line. Based on the difference between the separator line adjustment data and the label data, a third loss is determined, which is used to indicate the prediction accuracy of the separator line. The parameters of the table reconstruction model are iteratively adjusted based on the first loss, the second loss, and the third loss to obtain the trained table reconstruction model.

16. A table restoration device, characterized in that, The device includes: An input image acquisition module is used to acquire an input image, wherein the input image includes a table; The table data acquisition module is used to acquire the separator line data and table structure information of the table based on the input image. The separator line data is used to indicate the separator lines in the table, and the table structure information is used to indicate the distribution structure of each cell in the table. The data acquisition optimization module is used to optimize the separator line data based on the table structure information to obtain optimized separator line data. The cell restoration module is used to restore each cell in the table based on the data optimized by the separator line, so as to obtain the table.

17. A training device for a table reconstruction model, characterized in that, The device includes: A sample image acquisition module is used to acquire sample images and label data of the sample images, wherein the sample images include tables and the label data is used to label the tables; The table data acquisition module is used to acquire the separator line data and table structure information of the table based on the sample image through the table reconstruction model. The separator line data is used to indicate the separator lines in the table, and the table structure information is used to indicate the distribution structure of each cell in the table. The optimized data acquisition module is used to optimize the separator line data based on the table structure information through the table restoration model to obtain optimized separator line data. The relational data acquisition module is used to decode the mapping relational data by optimizing the data based on the separator lines through the table restoration model, and the mapping relational data is used to indicate multiple separator lines related to the cell. The table restoration model training module is used to train the table restoration model based on the mapping relationship data and the label data, so as to obtain the trained table restoration model.

18. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the table restoration method as described in any one of claims 1 to 11, or the training method for the table restoration model as described in any one of claims 12 to 15.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the table restoration method as described in any one of claims 1 to 11, or the training method for the table restoration model as described in any one of claims 12 to 15.

20. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, wherein a processor reads from and executes the computer program to implement the table restoration method as described in any one of claims 1 to 11, or the training method for the table restoration model as described in any one of claims 12 to 15.

Images