Method, system and device for constructing complex merged cell table image and medium

Abstract

The application discloses a complex merged cell table image construction method, system, device and medium, which are corresponding solutions, and in the solutions: two complementary data sources are adopted to ensure that the constructed data set reaches a high level in structural complexity, scene diversity and annotation quality; a multi-level quality checking mechanism is introduced, which can effectively filter out annotation illusions, structure breaks or cross-model unstable samples, thereby significantly improving the annotation reliability of the final data; finally, by using real geometric field deformation modeling and texture mapping technology, a table image highly consistent with the actual shooting effect can be generated, so that the table recognition model can truly learn the merged cell attributes in training, thereby improving its analysis ability for complex structures, instead of only optimizing the TEDS index of HTML.

Description

Technical Field

[0001] This invention relates to the field of table image data synthesis technology, and in particular to a method, system, device and medium for constructing complex merged cell table images. Background Technology

[0002] In the field of table recognition, large-scale, high-quality table image data is a key resource for training models.

[0003] Existing public datasets contain a limited number of tables with complex merged cell structures; the vast majority of samples have fewer than ten merged cells, leading to insufficient generalization of trained models in complex table scenarios. Furthermore, existing evaluation metrics, such as TEDS (Tree-Edit-Distance-based Similarity), are insensitive to errors in merged cell attributes (number of rows spanned, number of columns spanned), making it difficult for models to truly learn the structural attributes of merged cells during training. Therefore, a high-fidelity data construction method is needed that can systematically and on a large scale generate images of complex merged cell tables and corresponding HTML (Hypertext Markup Language) annotations while ensuring structural accuracy.

[0004] In view of this, the present invention is hereby proposed. Summary of the Invention

[0005] The purpose of this invention is to provide a method, system, device, and medium for constructing complex merged cell table images. Through mechanisms such as automatic annotation, structure checking, and cross-model consistency checking, a high-quality, structurally accurate, and reliably annotated large-scale table image dataset is constructed. This enables the table recognition model to truly learn the merged cell attributes during training, improving its ability to parse complex structures, rather than simply optimizing the TEDS metric for HTML.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A method for constructing complex merged cell table images, including:

[0008] Obtain the original table images, including: the synthesized table image and the real table image; and determine the HTML annotation corresponding to each original table image, where HTML is Hypertext Markup Language;

[0009] The acquired original table images undergo quality verification, including: filtering out original table images with broken structures, abnormal spans, or missing levels through HTML structure consistency checks; performing cross-model consistency verification on the remaining original table images after filtering, generating HTML structures for each original table image using different models, calculating the TEDS of the HTML structures generated by different models, and performing consistency verification on all calculated TEDS, retaining the original table images that pass the consistency verification; for the original table images that pass the consistency verification, counting the number of merged cells and using this as a filter to obtain the filtered original table images; where TEDS is a similarity based on tree edit distance.

[0010] The original table image after filtering is transformed and its texture modulated based on real geometric deformation parameters, and then fused with a natural background image to construct a merged cell table image.

[0011] A complex merged cell table image construction system, used to implement the aforementioned method, includes:

[0012] The original table image acquisition unit is used to acquire original table images, including: synthesized table images and real table images; and to determine the HTML annotation corresponding to each original table image, where HTML is Hypertext Markup Language;

[0013] The quality verification unit is used to perform quality verification on the acquired original table images, including: filtering out original table images with broken structures, abnormal spans, or missing levels through HTML structure consistency checks; performing cross-model consistency verification on the remaining original table images after filtering, generating HTML structures for each original table image using different models, calculating the TEDS of the HTML structures generated by different models, performing consistency verification on all calculated TEDS, and retaining the original table images that pass the consistency verification; for the original table images that pass the consistency verification, counting the number of merged cells and using this to filter, obtaining the filtered original table images; where TEDS is a similarity based on tree edit distance;

[0014] The image enhancement unit, which synthesizes the original table image into a real-world scene, is used to transform and modulate the texture of the filtered original table image based on real geometric deformation parameters, and then merge it with the natural background image to construct a merged cell table image.

[0015] A processing device includes: one or more processors; and a memory for storing one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0017] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0018] As can be seen from the technical solution provided by this invention, two complementary data sources (synthetic and real original table image data) are employed to ensure that the constructed dataset achieves a high level of quality in terms of structural complexity, scene diversity, and annotation quality. The introduction of multi-level quality verification mechanisms, such as HTML structure consistency checks and cross-model consistency verification, effectively filters out annotation illusions, structurally fragmented, or unstable samples across models, thereby significantly improving the annotation reliability of the final data. By utilizing deformation modeling and texture mapping techniques based on real geometric fields, table images highly consistent with actual photographic effects can be generated, allowing the model to learn about deformations and lighting changes that may occur in real shooting scenes during the training phase. Experiments demonstrate that the model trained on complex merged cell table image data constructed based on this invention achieves leading performance in multiple table recognition-related tasks, especially showing significant improvement in merged cell structure parsing, effectively verifying the practical value and technical advantages of the data construction scheme of this invention. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a method for constructing a complex merged cell table image, as provided in an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram illustrating a method for constructing a complex merged cell table image according to an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of a complex merged cell table image construction system provided in an embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0025] First, the following explanations are provided for the terms that may be used in this article:

[0026] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0027] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0028] The following provides a detailed description of a method, system, device, and medium for constructing complex merged cell table images provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Reagents or instruments used in the embodiments of this invention, unless otherwise specified by the manufacturer, are all commercially available conventional products.

[0029] Example 1

[0030] This invention provides a method for constructing complex merged cell table images. It is a high-fidelity table image data construction scheme that integrates Multimodal Large Language Model (MLLM) synthesis, web page crawling and acquisition, automatic HTML annotation, structural consistency checking, cross-model consistency verification, merged cell number filtering, and table image synthesis in real-world shooting scenarios. This method can generate large-scale, structurally complex, and realistically diverse table image datasets, providing effective and abundant training data for subsequent table recognition training, and achieving a significant performance improvement. Figure 1 As shown, the method mainly includes the following steps:

[0031] Step 1: Obtain the original table image.

[0032] In this embodiment of the invention, the obtained original table images mainly include the following two categories: synthesized table images and real table images; and the HTML annotation corresponding to each original table image is determined.

[0033] In this embodiment of the invention, the synthesized table image is obtained in the following manner: based on a preset structural complexity, an HTML-formatted table structure is automatically generated using a first multimodal large model according to the specified number of merged cells and the number of rows and columns spanned; the generated HTML-formatted table structure is converted into a table image with a scanned texture through rendering, which is used as the synthesized table image, wherein the HTML-formatted table structure is the corresponding HTML annotation.

[0034] In this embodiment of the invention, the real table image is obtained in the following manner: Documents or webpage images containing table structures are crawled from the network using an automated crawling strategy; a table detection model is used to locate the table regions in the crawled documents or webpage images containing table structures, identifying the outer regions of each table, and cropping them to obtain the table image as the real table image; furthermore, a second multimodal large model is used to automatically predict the structure of the obtained table image, generating corresponding HTML code as HTML annotations.

[0035] In this embodiment of the invention, the multimodal large model can select an existing model structure according to the actual situation, and the two multimodal large models can use the same or different models.

[0036] Step 2: Perform quality verification on the obtained original table image.

[0037] In this embodiment of the invention, the quality verification process includes: filtering out original table images with broken structures, abnormal spans, or missing levels through HTML structure consistency checks; performing cross-model consistency verification on the remaining original table images after filtering, generating HTML structures for each original table image through different models, calculating the TEDS of the HTML structures generated by different models, performing consistency verification on all calculated TEDS, and retaining the original table images that pass the consistency verification; for the original table images that pass the consistency verification, counting the number of merged cells and filtering accordingly to obtain the filtered original table images.

[0038] In this embodiment of the invention, the preferred implementation of filtering original table images with structural breaks, abnormal spans, or missing layers through HTML structure consistency checks is as follows: The HTML annotations of the original table image are parsed and mapped into a two-dimensional logical matrix. Coverage detection is performed on each cell, i.e., the table width and height are calculated based on the HTML annotations, and a corresponding two-dimensional logical matrix with the specified width and height is established. All grid positions in the two-dimensional logical matrix are initialized to 0, indicating that all cell positions of the HTML annotations are not covered. Then, each cell in the HTML annotations is mapped one-to-one to find its grid position on the two-dimensional logical matrix, and the corresponding grid position is set to 1, indicating that the grid position is covered by a cell in the HTML annotation. When the proportion of empty areas in the grid that are not occupied by any cell (i.e., the proportion of grid areas with a value of 0) exceeds a preset first threshold, the corresponding original table image is determined to have structural breaks, abnormal spans, or missing layers, and is filtered out.

[0039] In this embodiment of the invention, the preferred implementation of generating HTML structures for each original table image using different models, calculating the TEDS of the HTML structures generated by different models, and performing consistency verification by comprehensively calculating all TEDS is as follows: For each original table image, corresponding HTML structures are generated using multiple heterogeneous multimodal models, the TEDS between HTML structures generated by different multimodal models is calculated, and the average TEDS is calculated; if the average TEDS is lower than a preset second threshold, and the variance of the predicted HTML structures is higher than a preset third threshold, then the consistency verification fails and the structure is filtered out; otherwise, the consistency verification passes.

[0040] In this embodiment of the invention, the preferred implementation of counting the number of merged cells and thereby filtering to obtain the filtered original table image is as follows: For each original table image that passes the consistency verification, the number of merged cells is counted; the original table images with the number of merged cells exceeding a preset fourth threshold are retained as the first part of the original table images; and, according to a set sampling probability, sampling is performed on the original table images with the number of merged cells not exceeding the preset fourth threshold to obtain the second part of the original table images; the first part of the original table images and the second part of the original table images constitute the filtered original table image.

[0041] Those skilled in the art will understand that merged cells mainly refer to a cell that spans more than one row or column. Specifically: based on the HTML annotations of the original table image, determine the values ​​of colspan (number of columns spanned) and rowspan (number of rows spanned) of the cell. If colspan or rowspan is greater than 1, the cell is considered a merged cell.

[0042] Step 3: Image enhancement from the original table image to the composite of the real-life scene.

[0043] In this embodiment of the invention, the original table image after screening is transformed and texture modulated based on real geometric deformation parameters, and then fused with a natural background image to construct a merged cell table image. The preferred implementation method is as follows: A set of deformation parameters P is randomly selected from the real document geometry field extracted from a pre-selected dataset as real geometric deformation parameters. The deformation parameters P serve as the basis for simulating paper bending, twisting, or irregular perspective changes under real shooting conditions. The original table image after screening is transformed and texture modulated according to the deformation parameters P using UV (horizontal and vertical axes of two-dimensional texture space) transformation to generate a distorted image. The distorted image is then fused with a randomly sampled natural background image, and the overall visual style is adjusted to construct a merged cell table image.

[0044] The above-described solution provided by the embodiments of the present invention can construct merged cell table images, providing a large amount of training data for the training of table recognition models. During training, the table recognition model truly learns the merged cell attributes, improving its ability to parse complex structures. More specifically, the above-described solution provided by the embodiments of the present invention can be widely applied in fields such as table recognition, intelligent document analysis, automated processing of enterprise reports, parsing of financial and audit documents, OCR (Optical Character Recognition) system training, and intelligent question-answering systems. By significantly improving the table recognition model's ability to understand and recover complex structured tables (especially tables with highly merged cells), it can effectively improve the accuracy, stability, and application coverage of intelligent document systems, possessing significant industrial value and broad prospects for promotion. In implementation, by supervising the training of the dataset proposed in this paper on relevant multimodal large models, a high-performance table recognition model can be obtained. This model can then be deployed on a server, allowing users to complete table recognition and parsing through API (Application Programming Interface) calls, and returning the predicted HTML sequence.

[0045] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0046] I. Overall Overview of the Plan

[0047] This invention provides a method for constructing complex merged cell table images. It is a high-fidelity data construction scheme for complex table structures based on multi-source generation and quality control, addressing issues such as the lack of complex merged cell samples in existing datasets, structural illusions in automatic annotation, and inconsistencies in cross-model prediction results. Specifically, the scheme designed in this invention employs a multi-source decoupled construction process, breaking down the generation and annotation of complex table image data into multiple independent but collaborative steps: original table image collection (including synthesized table images and real table images crawled from web pages, table structure annotations), multi-level quality verification, and synthesis of real-shot table images (converting the collected original scanned images into corresponding real-shot table images). This multi-source decoupled construction process effectively avoids problems such as structural fabrication and misjudgment of merged cells that easily occur when directly relying on multimodal large models for end-to-end annotation, thereby significantly improving the accuracy and consistency of annotation.

[0048] II. Detailed introduction of the plan.

[0049] like Figure 2 The diagram illustrates the overall process of the method described above in this invention. Each step will be described in detail below.

[0050] 1. Obtaining the original table image.

[0051] To obtain a large number of table images in real-world scenarios, this invention employs two complementary data sources to ensure that the constructed dataset achieves a high level of structural complexity, scene diversity, and annotation quality.

[0052] The first category consists of synthetic data (synthetic table images) based on the first multimodal large model. According to preset structural complexity requirements, the multimodal large model automatically generates HTML-formatted table structures based on the specified number of merged cells and their row span and column span (rowspan, colspan). Subsequently, the generated HTML table is converted into a scanned table image through rendering, thus obtaining a synthetic table image that is formally standardized, structurally controllable, and fully accurately labeled.

[0053] The second category is real, complex table data (real table images) from the internet. This invention uses an automated crawling strategy to collect documents or webpage images that may contain complex table structures from publicly available links on the internet, thereby improving the coverage of complex merged cell samples. Since directly crawled documents or webpage images may contain a large number of non-table areas, such as paragraph text, illustrations, and page borders, to ensure that the final target areas are all tables, this invention performs a series of preprocessing steps on the documents and webpage images after data crawling. Specifically, firstly, a table detection model is used to locate table regions in the crawled full-page document images (webpage images), automatically identifying the outer regions of each table within the page. Then, the detection results are cropped, removing non-table content and interfering elements from the images, retaining only image fragments containing table structures, to ensure that subsequent annotation processes focus on valid table regions. Simultaneously, the size, proportions, and edge noise of the table images are standardized to improve overall data quality and stability. Afterwards, this invention uses a second multimodal large model (such as the Qwen-VL class model or the Gemini class model) to automatically predict the structure of these table fragments, generating corresponding HTML code as preliminary coarse annotation results. Although these samples lack native HTML annotations, their content is authentic, their layout is natural, and their structure is complex, effectively compensating for the shortcomings of purely synthetic data in terms of visual naturalness, realistic document layout characteristics, and cross-industry structural distribution. Combined with subsequent structural consistency checks and cross-model consistency verification, this invention can ultimately extract high-quality, realistic, and complex table annotation data from these network samples.

[0054] In the above examples of multimodal large models, Qwen-VL refers to the Qianwen Visual Language Model, and Gemini is a multimodal large model launched by Google.

[0055] 2. Quality verification.

[0056] To ensure that the automatically generated HTML annotations have sufficient structural reliability and semantic accuracy, this invention constructs a multi-level quality verification mechanism, including three stages: HTML structure consistency check, cross-model consistency verification, and merging cell number filtering. Through layer-by-layer filtering and selection, samples with abnormal structures, model illusions, or unstable annotations can be effectively eliminated, thereby ensuring the annotation quality of the final dataset.

[0057] (1) HTML structure consistency check. The HTML annotations of the original table image are parsed and mapped into a two-dimensional logical matrix, and the coverage detection is performed on each cell in the HTML annotation. When the proportion of empty areas in the two-dimensional logical matrix grid that are not occupied by any cell exceeds a preset first threshold (e.g., 5%), it is determined that the HTML of the corresponding original table image has problems such as structural fragmentation, abnormal span, or missing hierarchy, and the corresponding original table image is then removed.

[0058] (2) Cross-model consistency verification. For each original table image retained after HTML structure consistency check, HTML structures are generated using various heterogeneous multimodal models (e.g., MonkeyOCR, Mineru, dots.ocr, etc.), and the TEDS between the prediction results of different models (i.e., the generated HTML structures) is calculated. When the average TEDS score of multiple prediction results is lower than a preset second threshold (e.g., 0.7), and the variance between predictions is set to a preset third threshold (e.g., 0.08), it indicates that the structure of the original table image has high uncertainty or the model generation is unreliable, and it needs to be filtered as a low-quality label.

[0059] Among them, MonkeyOCR, Mineru, and dost.ocr are multimodal document intelligent parsing models launched by Huazhong University of Science and Technology, Shanghai Artificial Intelligence Laboratory, and Xiaohongshu, respectively, which can parse the input table image into the corresponding HTML structure.

[0060] (3) Filtering by the number of merged cells. After the checks and verifications in (1) to (2) above, the number of merged cells in each of the original table images is counted and filtered. All original table images with more than the preset fourth threshold (e.g., 10) are retained to ensure that the dataset contains a sufficient proportion of structurally complex samples; for original table images with fewer merged cells or no merged cells, they are selected according to the set sampling probability (e.g., random sampling in the probability range of 0.3 to 0.5) to maintain the balance of the overall complexity distribution while ensuring the data size.

[0061] Through the above three steps, high-quality HTML table annotation data with reliable structure, rich complexity, and high annotation consistency are finally obtained, providing a solid data foundation for subsequent model training.

[0062] 3. Image enhancement from original table images to composite images of real-life scenes.

[0063] To enable the dataset to cover the appearance features of tables in real shooting scenarios and improve the model's adaptability to various shooting conditions, this invention further introduces a real geometric deformation and UV texture mapping mechanism based on the dataset to construct the synthesis of real-world images. For example, the dataset here can be the UVDoc dataset, which is a dedicated dataset in the fields of computer vision and document analysis, mainly serving document image dedistortion or document flattening tasks.

[0064] Specifically, firstly, a set of deformation parameters P is randomly selected from the real document geometry field extracted from the UVDoc dataset, serving as the basis for simulating paper bending, twisting, or irregular perspective changes under real shooting conditions. Then, the original table images selected for quality verification undergo UV transformation and texture modulation according to the deformation parameters P, causing spatial distortion corresponding to real physical deformation, thus generating an intermediate distorted image with a realistic document bending structure. Next, the distorted image is fused with a randomly sampled natural background image, and the overall visual style is adjusted through color matching and brightness adaptation, ultimately obtaining a simulated photographic image that more closely resembles a real photographic scene in terms of lighting, background, and shooting angle. This approach significantly improves the appearance realism of the synthesized table image data, allowing the subsequent table recognition model to be fully exposed to table formats under various shooting conditions during the training phase, thereby effectively enhancing the model's robustness and generalization ability in real shooting scenarios.

[0065] III. Description of the overall effect of the plan.

[0066] Through the aforementioned steps of multi-source synthesis, automatic annotation, quality verification, and enhancement through real-world table image synthesis, this invention ultimately constructs a complex merged cell table image dataset containing a high proportion of complex merged cell samples, high-quality HTML structure annotations, and covering scanning scenarios, photographic scenarios, multilingual tables, and documents from multiple industries. This is a massive table dataset containing 350,000 table images, abbreviated as (Merged-Cell Table data 350K, MCT-350K). This dataset can be directly used for supervised training, reinforcement learning training, or multi-task joint learning of various table understanding models, significantly improving the model's parsing capabilities in complex real-world scenarios.

[0067] Overall, the present invention has the following significant technical advantages:

[0068] (1) Significantly improves the number and structural diversity of complex merged cell samples.

[0069] This invention effectively addresses the scarcity of complex merged cell samples in existing public datasets by using a multimodal large model-based table image synthesis and complex scene capture strategy, enabling the table recognition model to fully engage with more realistic and comprehensive structural patterns during the training phase.

[0070] (2) Significantly improve the accuracy and consistency of automatic HTML annotation.

[0071] This invention introduces a multi-level quality verification mechanism, including HTML structure consistency checks and cross-model consistency verification, which can effectively filter out annotation illusions, structurally broken or unstable samples across models, thereby significantly improving the annotation reliability of the final data.

[0072] (3) The image enhancement method of synthesizing the original table data into the real shooting scene significantly improves the robustness of the model to the real shooting scene.

[0073] By utilizing deformation modeling of real geometric fields and UV texture mapping technology, this invention can generate table images that are highly consistent with the actual photographic effect, enabling the table recognition model to learn the deformation and lighting changes that may occur in real shooting scenes during the training phase.

[0074] (4) Achieve leading performance in table recognition and merged cell recognition tasks.

[0075] Experiments have shown that the table recognition model trained on the data constructed based on this invention has achieved leading performance in multiple table recognition-related tasks, especially in the parsing of merged cell structures, which has effectively verified the practical value and technical advantages of the data construction method of this invention.

[0076] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0077] Example 2

[0078] This invention also provides a system for constructing complex merged cell table images, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 3As shown, the system mainly includes:

[0079] The original table image acquisition unit is used to acquire original table images, including: synthesized table images and real table images; and to determine the HTML annotation corresponding to each original table image, where HTML is Hypertext Markup Language;

[0080] The quality verification unit is used to perform quality verification on the acquired original table images, including: filtering out original table images with broken structures, abnormal spans, or missing levels through HTML structure consistency checks; performing cross-model consistency verification on the remaining original table images after filtering, generating HTML structures for each original table image using different models, calculating the TEDS of the HTML structures generated by different models, performing consistency verification on all calculated TEDS, and retaining the original table images that pass the consistency verification; for the original table images that pass the consistency verification, counting the number of merged cells and using this to filter, obtaining the filtered original table images; where TEDS is a similarity based on tree edit distance;

[0081] The image enhancement unit, which synthesizes the original table image into a real-world scene, is used to transform and modulate the texture of the filtered original table image based on real geometric deformation parameters, and then merge it with the natural background image to construct a merged cell table image.

[0082] Since the main technical details of this system have been described in detail in previous embodiments, they will not be repeated here.

[0083] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0084] Example 3

[0085] The present invention also provides a processing device, such as Figure 4 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0086] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0087] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0088] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0089] The output device can be a display terminal;

[0090] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0091] Example 4

[0092] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0093] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0094] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for constructing a complex merged cell table image, characterized in that, include: Obtain the original table images, including: the synthesized table image and the real table image; and determine the HTML annotation corresponding to each original table image, where HTML is Hypertext Markup Language; The acquired original table images undergo quality verification, including: filtering out original table images with broken structures, abnormal spans, or missing levels through HTML structure consistency checks; performing cross-model consistency verification on the remaining original table images after filtering, generating HTML structures for each original table image using different models, calculating the TEDS of the HTML structures generated by different models, and performing consistency verification on all calculated TEDS, retaining the original table images that pass the consistency verification; for the original table images that pass the consistency verification, counting the number of merged cells and using this as a filter to obtain the filtered original table images; where TEDS is a similarity based on tree edit distance. The original table image after filtering is transformed and its texture modulated based on real geometric deformation parameters, and then fused with a natural background image to construct a merged cell table image.

2. The method for constructing a complex merged cell table image according to claim 1, characterized in that, The synthesized table image is obtained in the following way: Based on the preset structural complexity, a multimodal large model is used to automatically generate an HTML-formatted table structure according to the specified number of merged cells and the number of rows and columns spanned. The generated HTML-formatted table structure is then converted into a scanned table image through rendering, which is used as the composite table image. The HTML-formatted table structure is the corresponding HTML annotation.

3. The method for constructing a complex merged cell table image according to claim 1, characterized in that, The actual table image was obtained in the following way: Using automated crawling strategies, documents or web page images containing table structures can be crawled from the web. The table detection model is used to locate the table regions in crawled documents or web page images containing table structures, identify the outer regions of each table, and crop them to obtain the table images as real table images. Furthermore, a multimodal large model is used to automatically predict the structure of the obtained table images and generate corresponding HTML code as HTML annotations.

4. The method for constructing a complex merged cell table image according to claim 1, characterized in that, The filtering of original table images containing structural fragmentation, abnormal span, or missing hierarchy through HTML structure consistency checks includes: The HTML annotations of the original table image are parsed and mapped into a two-dimensional logical matrix. Coverage detection is performed on each cell: the table width and height are calculated based on the HTML annotations, and a two-dimensional logical matrix with corresponding width and height is created accordingly. All grid positions in the two-dimensional logical matrix are initialized to 0, indicating that all cell positions of the HTML annotations are not covered. Then, each cell in the HTML annotations is mapped one by one to find its grid position in the two-dimensional logical matrix, and the corresponding grid position is set to 1, indicating that the grid position is covered by the cell in the HTML annotation. When the proportion of grid areas with a value of 0 exceeds a preset first threshold, the corresponding original table image is determined to have structural fragmentation, abnormal span, or missing layers, and is filtered out.

5. The method for constructing a complex merged cell table image according to claim 1, characterized in that, The process of generating HTML structures for each original table image using different models, calculating the TEDS of the HTML structures generated by different models, and performing consistency verification on all calculated TEDS includes: For each original table image, the corresponding HTML structure is generated using multiple heterogeneous multimodal models. The TEDS between the HTML structures generated by different multimodal models is calculated, and the average TEDS is calculated. If the average TEDS is lower than the preset second threshold and the variance of the difference between the predicted HTML structures is higher than the preset third threshold, it means that the consistency verification has failed and will be filtered out; otherwise, it means that the consistency verification has passed.

6. The method for constructing a complex merged cell table image according to claim 1, characterized in that, The process of counting and merging cells to obtain the filtered original table image includes: For each original table image that passes the consistency verification, the number of merged cells is counted. The original table image with more than a preset fourth threshold number of merged cells is retained as the first part of the original table image; and the second part of the original table image is obtained by sampling the original table image with less than a preset fourth threshold number of merged cells according to the set sampling probability. The filtered original table image is composed of the first part of the original table image and the second part of the original table image.

7. The method for constructing a complex merged cell table image according to claim 1, characterized in that, The process of transforming and texturing the filtered original table image based on real geometric deformation parameters, and then fusing it with a natural background image to construct a merged cell table image includes: A set of deformation parameters P is randomly selected from the real document geometry field extracted from the pre-selected dataset as the real geometric deformation parameters. The deformation parameters P serve as the basis for simulating paper bending, twisting or irregular perspective changes under real shooting conditions. The original table image after filtering is subjected to UV transformation and texture modulation according to the deformation parameter P to generate a distorted image. The distorted image is then merged with a randomly sampled natural background image, and the overall visual style is adjusted to construct a merged cell table image. Here, UV refers to the horizontal and vertical axes of the two-dimensional texture space.

8. A system for constructing complex merged cell table images, characterized in that, To implement the method according to any one of claims 1 to 7, comprising: The original table image acquisition unit is used to acquire original table images, including: synthesized table images and real table images; and to determine the HTML annotation corresponding to each original table image, where HTML is Hypertext Markup Language; The quality verification unit is used to perform quality verification on the acquired original table images, including: filtering out original table images with broken structures, abnormal spans, or missing levels through HTML structure consistency checks; performing cross-model consistency verification on the remaining original table images after filtering, generating HTML structures for each original table image using different models, calculating the TEDS of the HTML structures generated by different models, performing consistency verification on all calculated TEDS, and retaining the original table images that pass the consistency verification; for the original table images that pass the consistency verification, counting the number of merged cells and using this to filter, obtaining the filtered original table images; where TEDS is a similarity based on tree edit distance; The image enhancement unit, which synthesizes the original table image into a real-world scene, is used to transform and modulate the texture of the filtered original table image based on real geometric deformation parameters, and then merge it with the natural background image to construct a merged cell table image.

9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images