A document image synthesis and data set automatic generation method and system based on Doctags language

Abstract

The application discloses a document image synthesis and data set automatic generation method and system based on a Doctags language, and first acquires an original document and parses Doctags information of the document; then, based on the Doctags information, a synthetic document image is generated through reverse rendering, the reverse rendering is inputted with the Doctags information, visual elements aligned with the Doctags information are drawn on a canvas according to semantic categories and physical boundary box coordinates, so that the synthetic document image is obtained; finally, the synthetic document image is stored in association with corresponding Doctags information, and an image-label pair is formed to construct a data set. The application adopts reverse thinking, so that the generated image is absolutely logically consistent and spatially corresponding with the labeling information at a pixel level, can eliminate the mismatching error between the image and the labeling, and realizes automatic construction of large-scale high-precision document understanding model training data.

Description

Technical Field

[0001] This invention relates to the field of computer vision and natural language processing, and in particular to a method for constructing a gold standard dataset for training deep learning models. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially multimodal large models, and their widespread application in document understanding, layout analysis, and optical character recognition (OCR), the demand for high-quality training data is becoming increasingly urgent. In the current deep learning training process, data quality directly determines the upper limit of model performance, and accurate alignment between images and labeled information (including text content, logical structure, and coordinate positions) is a key element in building high-performance models.

[0003] However, existing document dataset construction techniques primarily rely on forward annotation logic, which involves collecting existing PDF documents or scanned images and then generating corresponding text and coordinate labels through manual annotation or by using existing OCR tools and PDF parsers. This traditional method has insurmountable technical drawbacks in practical applications. First, manual annotation is extremely costly and inefficient, making it difficult to meet the scalability requirements of large models with massive amounts of data. Second, pseudo-labels generated by automated tools inevitably contain noise, such as character recognition errors, missing characters, extra characters, or offsets between the detection box and the actual text pixels. When models are trained using this data containing inherent errors, they learn incorrect feature mapping relationships, leading to a performance ceiling effect and making it difficult to achieve pixel-level accurate recognition and localization.

[0004] Furthermore, when processing documents with complex layouts (such as academic papers and legal documents containing double-column layouts, nested tables, mathematical formulas, or floating images), existing technologies often struggle to accurately match the logical structure with the physical location. For example, garbled characters or coordinate system conversion errors frequently occur during PDF parsing, causing the extracted document structure tags (Doctags) to fail to accurately and completely reflect the visual information in the images. Current technologies lack a low-cost, automated dataset generation mechanism that guarantees absolute consistency between annotation information and image content (i.e., the "gold standard"). This mismatch between images and annotations has become a major bottleneck restricting further breakthroughs in intelligent document processing technology. Summary of the Invention

[0005] Purpose of the invention: To address the shortcomings of the existing technology, this invention aims to provide a method and system for document image synthesis and automatic dataset generation based on the Doctags language. By re-rendering Doctags as images, the mismatch error between images and annotations is eliminated, thereby achieving the automated construction of large-scale, high-precision document understanding model training data.

[0006] Technical solution: To achieve the above-mentioned objectives, the first aspect of this invention provides a method for document image synthesis and automatic dataset generation based on the Doctags language, comprising the following steps:

[0007] Obtain the original document and parse the Doctags information of the document from the original document, wherein the Doctags information includes at least the semantic category of each element in the document and the physical bounding box coordinates of each element on the page;

[0008] Based on Doctags information, a synthetic document image is generated through reverse rendering. The reverse rendering process involves taking Doctags information as input, drawing visual elements aligned with the Doctags information on the canvas according to the semantic category and physical bounding box coordinates, thereby obtaining a synthetic document image. During the rendering process, each tag node in Doctags is traversed. For text-type tags, the text content and physical bounding box coordinates are parsed, and an adaptive font size calculation is used to determine the font size that maximizes the filling of the specified bounding box with the text content. The determined font size is then used to render the text within the specified bounding box. For non-text tags such as images or formulas, the corresponding pixel regions are cropped from the pre-acquired original document image according to their physical bounding box coordinates, and the cropped pixel regions are pasted to the corresponding coordinate positions on the canvas.

[0009] The synthesized document images are associated and stored with the corresponding Doctags information to form image-tag pairs, thereby constructing a dataset.

[0010] Furthermore, the adaptive font size calculation adopts a binary approximation algorithm, which includes: setting a preset font size search range, continuously taking the intermediate font size for text layout calculation during the iteration process, narrowing the search range based on the comparison results of the calculated text rendering width and height with the target bounding box, until the maximum font size that satisfies the specified bounding box constraint is obtained.

[0011] Furthermore, for streaming text that spans multiple columns or regions, the reverse rendering also includes: merging the parameters of multiple bounding boxes corresponding to the text to estimate the global font size, processing each bounding box sequentially according to the reading order, calculating the maximum number of lines that the current bounding box can accommodate, extracting the text corresponding to the number of lines for rendering, and passing the remaining text to the next bounding box for continued rendering, so as to maintain the reading continuity of the text.

[0012] Furthermore, for special symbols such as checkboxes in text type, the reverse rendering also includes: calculating the geometric dimensions of the checkbox based on the determined font size and line height, and drawing the checkbox and its check mark at the specified coordinate position using vector drawing, so as to achieve characters that do not depend on the font library.

[0013] Furthermore, for OTSL tables, when the Doctags information contains a description instruction for the table structure, the reverse rendering also includes: parsing the table structure description instruction, restoring the cell topology and row / column merging attributes of the table; dynamically allocating the width of each column using a water-filling algorithm based on the cell content and the total width of the table; calculating the physical coordinates of each cell based on the allocated column width, and drawing the table outline and filling the cell text content.

[0014] Furthermore, the method of dynamically allocating the width of each column using a water-filling algorithm includes:

[0015] Based on the basic physical width of the cell when it does not wrap, and combined with the average width across columns and the minimum column width limit, determine the ideal width requirement for each column;

[0016] Calculate the sum of the ideal widths of all columns. Based on the comparison between this sum and the total available width of the table, there are two scenarios: If the total space is ample, the ideal requirements of each column are fully met, and the remaining space is evenly distributed among the columns; if the total available space is limited, a greedy dynamic iterative allocation is initiated. During the iteration process, the current average remaining quota of the unallocated columns is calculated in each round. Priority is given to fully meeting the ideal requirements of low-bandwidth columns that are less than the average remaining quota, and these columns are removed from the unallocated set. The remaining space is carried over to the next round; when the requirements of all remaining columns exceed the average remaining quota, the remaining space is distributed according to the proportion of the ideal requirements of each column, while a minimum column width constraint is applied.

[0017] The last digit is compensated for rounding errors caused by floating-point calculations.

[0018] Furthermore, when rendering text, the method also includes randomly selecting a font from a preset font library for text drawing to enhance the diversity of the generated images.

[0019] Furthermore, the synthetic document images are associated and stored with the corresponding Doctags information, including: dividing the image-tag pairs into training and testing sets according to a preset ratio, and encapsulating them into a dataset file in Parquet columnar storage format; the Parquet columnar storage format is to store the dataset in segments according to a preset number of samples.

[0020] The second aspect of this invention provides a document image synthesis and automatic dataset generation system based on the Doctags language, used to implement the document image synthesis and automatic dataset generation method based on the Doctags language described in the first aspect. The system includes:

[0021] The preprocessing module is used to obtain the original document and parse the Doctags information of the document from the original document, wherein the Doctags information includes at least the semantic category of each element in the document and the physical bounding box coordinates of each element on the page;

[0022] The reverse rendering module is used to generate a synthetic document image based on Doctags information through reverse rendering. The reverse rendering involves taking Doctags information as input, drawing visual elements aligned with the Doctags information on the canvas according to the semantic category and physical bounding box coordinates, thereby obtaining a synthetic document image. During the rendering process, each tag node in Doctags is traversed. For text-type tags, the text content and physical bounding box coordinates are parsed, and the font size that maximizes the filling of the specified bounding box is determined through adaptive font size calculation. The determined font size is then used to render the text within the specified bounding box. For non-text tags such as images or formulas, the corresponding pixel regions are cropped from the pre-acquired original document image according to their physical bounding box coordinates, and the cropped pixel regions are pasted to the corresponding coordinate positions on the canvas.

[0023] The dataset generation module is used to associate and store the synthesized document images with the corresponding Doctags information to form image-tag pairs in order to construct the dataset.

[0024] A computer system provided by a third aspect of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the document image synthesis and automatic dataset generation method based on the Doctags language described in the first aspect.

[0025] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0026] 1. This invention employs a reverse rendering method, using precise Doctags information as input to draw visual elements on a canvas according to semantic categories and physical bounding box coordinates, generating a synthetic document image. This reverse thinking, where annotation precedes image creation, ensures that the generated image maintains absolute logical consistency and spatial correspondence with the annotation information at the pixel level, eliminating annotation noise in traditional methods and providing high-quality "gold standard" training data for document understanding models (especially multimodal large models).

[0027] 2. This invention acquires original documents and parses Doctags information, then generates synthetic document images through reverse rendering, and finally associates and stores them to form image-tag pairs, constructing a complete automated data production pipeline. This process requires no manual annotation, uses existing PDF documents as seed data, and can quickly generate millions or even hundreds of millions of training samples with low computational cost. This automated closed-loop production mode not only eliminates the dependence on manual annotation but also provides efficient technical support for the construction of customized datasets in specific fields (such as law, medicine, and finance).

[0028] 3. For text-based tags, this invention uses adaptive font size calculation combined with random font selection to simulate various document layout styles. This artificially introduced visual diversity ensures that the trained model not only memorizes specific document templates but also learns robust representations of text features and page structure, effectively avoiding overfitting or decreased recognition rates when faced with documents with inconsistent fonts and complex layouts in the real world. For non-text tags such as images or formulas, this invention crops the corresponding pixel regions from the pre-acquired original document images according to the physical bounding box coordinates and pastes them to the corresponding coordinate positions on the canvas. This hybrid rendering mechanism solves the problem of complex mathematical formulas and images being difficult to realistically render through code, ensuring that the dataset retains rich and realistic non-textual visual information while possessing perfect text annotations.

[0029] 4. For cross-column or multi-regional flowing text, this invention merges the parameters of multiple bounding boxes to estimate the global font size, processes each bounding box sequentially according to the reading order, and passes the remaining text to the next bounding box for further rendering, thus ensuring the visual continuity of cross-column text. Furthermore, for special symbols such as checkboxes, this invention uses vector drawing for simulation rather than simple character printing to simulate the actual filling effect, solving the problem that most Chinese fonts do not support special symbols. Simultaneously, for OTSL tables, this invention parses the table structure description instructions, restores the cell topology and row / column merging attributes, uses a water-filling algorithm to dynamically allocate the width of each column, and calculates the physical coordinates of each cell based on the allocated column width, achieving accurate drawing of the table outline and content. These technical means enable this invention to fully handle various complex layout elements, and the generated dataset supports the model's ability to simultaneously learn text recognition, layout analysis, and cross-modal element understanding. Attached Figure Description

[0030] Figure 1 The overall flowchart of the document image synthesis and automatic dataset generation method based on the Doctags language provided in the embodiments of the present invention is shown.

[0031] Figure 2 This is a schematic diagram of the rendering engine processing flow in an embodiment of the present invention.

[0032] Figure 3 This is a screenshot of an original PDF used in an embodiment of the present invention.

[0033] Figure 4 for Figure 3 Screenshot of the corresponding Doctags information section.

[0034] Figure 5 For based on Figure 3 and Figure 4A new image generated by reverse rendering of the content.

[0035] Figure 6 This is another screenshot of the original PDF used in an embodiment of the present invention.

[0036] Figure 7 for Figure 6 Screenshot of the corresponding Doctags information.

[0037] Figure 8 For based on Figure 6 and Figure 7 A new image generated by reverse rendering of the content. Detailed Implementation

[0038] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] like Figure 1 As shown in the figure, an embodiment of the present invention provides a method for document image synthesis and automatic dataset generation based on the Doctags language, which includes the following steps:

[0040] S1. Preprocessing: Obtain the original document and parse the Doctags information of the document from the original document, wherein the Doctags information includes at least the semantic category of each element in the document and the physical bounding box coordinates of each element on the page.

[0041] In this embodiment, this step aims to obtain a high-precision physical baseline map and document Doctags tags, mainly by calling an open-source toolchain to achieve standardized preprocessing.

[0042] Specifically, firstly, multi-source heterogeneous PDF documents are acquired, and then recursively split into independent single-page PDF files using the pypdf library. When converting the vector PDF to a discrete pixel image, to avoid pixel collapse of small characters and subsequent bounding box coordinate drift caused by conventional 72dpi rendering, this embodiment calls the PyMuPDF (fitz) library, setting the zoom transformation matrix parameters to zoom_x=2.0, zoom_y=2.0. This oversampling operation increases the image resolution to over 144dpi, generating a high-fidelity PNG baseline image, ensuring absolute accuracy of the physical coordinate system.

[0043] The Docling document parsing engine is then invoked. To minimize transcription noise at the source, the system explicitly injects the parameter `do_ocr=False` when configuring `PdfPipelineOptions`, forcing the engine to read only the underlying native vector text stream of the PDF; simultaneously, `do_table_structure=True` is enabled to obtain the table topology. After parsing, the two-dimensional layout is serialized into a one-dimensional Doctags markup language stream (e.g., containing...).<section_header> Tags and formats are<loc_x1><loc_y1><loc_x2><loc_y2> Commands for absolute coordinates, such as<loc_67><loc_43><loc_443><loc_291> This serves as the input drawing for subsequent reverse rendering.

[0044] S2. Reverse rendering: Based on Doctags information, a synthetic document image is generated through reverse rendering. The reverse rendering is to take Doctags information as input, draw visual elements aligned with the Doctags information on the canvas according to the semantic category and physical bounding box coordinates, thereby obtaining a synthetic document image.

[0045] Specifically, during the rendering process, each tag node in Doctags is traversed. For text-type tags, the text content and physical bounding box coordinates are parsed, and the font size that can maximize the filling of the specified bounding box is determined through adaptive font size calculation. The determined font size is then used to render the text within the specified bounding box. For non-text-type tags such as images or formulas, the corresponding pixel region is cropped from the pre-acquired original document image according to its physical bounding box coordinates, and the cropped pixel region is pasted to the corresponding coordinate position in the canvas.

[0046] In this embodiment, this step is the core of constructing the "gold standard" dataset. Using a reverse rendering engine, precise Doctags information is input to create a synthetic document image on a blank canvas that is absolutely aligned to the tag information at the pixel level. For text-type tags, adaptive font size calculation ensures the text precisely fills the specified bounding box, eliminating text truncation or white space issues. For non-text-type tags such as images and formulas, a copy-paste mechanism is used to directly crop the original pixel area from the high-fidelity PNG baseline image generated in step S1 and paste it to the corresponding position, perfectly preserving the realism of complex visual elements. Through this reverse thinking of labeling first and then creating the image, this invention fundamentally eliminates the image-tag mismatch problem caused by OCR recognition errors or detection box offsets in traditional forward labeling methods.

[0047] In some possible implementations, adaptive font size calculation employs a binary approximation algorithm, which includes: presetting a font size search range, continuously taking intermediate font sizes for text layout calculation during the iteration process, narrowing the search range based on the comparison results of the calculated text rendering width and height with the target bounding box, until the maximum font size that satisfies the specified bounding box constraints is obtained.

[0048] In some possible implementations, for streaming text that spans multiple columns or regions, the reverse rendering further includes: merging the parameters of multiple bounding boxes corresponding to the text to estimate the global font size, processing each bounding box sequentially according to the reading order, calculating the maximum number of lines that the current bounding box can accommodate, extracting the text corresponding to the number of lines for rendering, and passing the remaining text to the next bounding box for continued rendering, so as to maintain the reading continuity of the text.

[0049] In some possible implementations, for special symbols such as checkboxes of text type, the reverse rendering also includes: calculating the geometric dimensions of the checkbox based on the determined font size and line height, and drawing the checkbox and its check mark at the specified coordinate position using vector drawing, so as to achieve characters that do not depend on the font library.

[0050] In some possible implementations, for OTSL tables, when the Doctags information contains a description instruction for the table structure, the reverse rendering further includes: parsing the table structure description instruction, restoring the cell topology and row / column merging attributes of the table; dynamically allocating the width of each column using a water-filling algorithm based on the cell content and the total width of the table; calculating the physical coordinates of each cell based on the allocated column width, and drawing the table outline and filling the cell text content.

[0051] In some possible implementations, when rendering text, the method also includes randomly selecting a font from a preset font library for text drawing to enhance the diversity of the generated images.

[0052] The steps of this embodiment will be further explained in detail below with an example of a real reverse rendering scenario, such as... Figure 2 As shown, the reverse rendering process mainly includes the following steps.

[0053] S21. Text and symbol drawing.

[0054] To address the issue that existing typesetting libraries cannot perfectly fill dynamic text within a specified absolute physical coordinate frame, this embodiment instantiates a self-developed DocTagRenderer rendering engine. The specific implementation is as follows:

[0055] S211, Randomly assign fonts.

[0056] To enhance the visual diversity of the generated data and prevent downstream multimodal language models from overfitting to single font features during training, the rendering engine constructs a low-level font library matrix during the initialization phase, including: Microsoft YaHei (msyh.ttc), SimSun (simsun.ttc), SimHei (simhei.ttf), KaiTi (simkai.ttf), FangSong (simfang.ttf), and DengXian (Deng.ttf). When allocating rendering resources for each single-page canvas, the rendering engine triggers a pseudo-random generator to assign a globally unique primary font feature to the current canvas. This engineering intervention forces the model to learn the geometric and topological invariants of the glyphs, rather than memorizing the arrangement rules of specific pixels.

[0057] S212. Adaptive calculation of font size based on constrained bisection approximation algorithm.

[0058] In reverse rendering <text>or <paragraph>When rendering plain text tags, the rendering engine faces a complex non-linear space filling constraint problem: it must ensure that discrete strings of arbitrary length perfectly fill a predefined absolute coordinate bounding box. And there is no overflow or abnormal white space at the pixel level, among which The x and y coordinates represent the top left corner. The x and y coordinates represent the bottom right corner.

[0059] Let the physical constraint width of the target bounding box be... The height is In this embodiment, the virtual typesetting mapping function is defined as follows: ,in For the input string, Scalar for font size scaling. and This represents the actual rendered width and height of the text after the underlying line break calculation.

[0060] The rendering engine abstracts the typesetting process into a constrained optimization problem:

[0061]

[0062]

[0063] in The upper and lower bounds of the scalar search space. To achieve computational efficiency, the rendering engine deploys a time complexity of O(log n). The binary approximation operator. The algorithm sets the search space of the font size scalar as follows: In a fixed 12-cycle binary search, the rendering engine calculates the intermediate scalar. The function calls a virtual measurement function to obtain the current line height and total height. This virtual measurement function is used for layout preview; it doesn't actually draw the text on the canvas, but rather simulates how the specified text will be broken into lines under a specified font size and maximum width limit using line break calculations. The returned current line height refers to the height of a single line of text plus the preset line spacing, and the total height is the total vertical space occupied by the entire text block after all line breaks are stacked.

[0064] If in Total text height at this scale If the maximum width of a single line does not exceed the limit, it means that the current font size is valid but has not yet reached the limit of fit, and the algorithm updates the lower bound. Conversely, if pixel overflow occurs in any dimension, the upper bound is updated. After 12 iterations, the algorithm forces convergence to the optimal font size, eliminating text truncation caused by coordinate errors at the underlying level. Furthermore, a flag `prefer_larger` is set to indicate whether text is allowed to slightly exceed the bounding box. For ordinary text, setting `prefer_larger=False` allows for minor tolerance; for strict alignment requirements (such as table cells), `prefer_larger=True` enables strict degradation constraints.

[0065] S213, Layout calculation of long text spanning multiple columns.

[0066] For continuous paragraphs in a document that span multiple columns or are interrupted by charts, existing parsing engines often physically divide them into multiple independent sets of bounding boxes. ,in This represents the number of bounding boxes. For example, the corresponding Doctags language is " <text><loc_47><loc_427><loc_241><loc_454><loc_259><loc_56><loc_453><loc_73> This is a long text.< / text> "It contains two bounding boxes. If each bounding box is rendered in isolation, the font size of the same paragraph will fluctuate."

[0067] To address this complex topology, the rendering engine uses an algorithm for calculating the streaming layout of long, multi-column text. The specific steps are as follows:

[0068] First, merge the parameters of multiple bounding boxes to estimate a global master font size. Then, sum the heights of all individual bounding boxes to obtain a total height. Then, average the widths of all bounding boxes to obtain... Using this hypothetical size as [ The text is placed within a virtual box, and the font size required to write all the characters into the virtual box is calculated. This font size is then used as the global font size for all boxes.

[0069] Upon entering the traversal rendering pipeline, the current font metrics are extracted, including the ascent and descent. Ascent refers to the distance from the baseline upwards to the top of the tallest character (e.g., uppercase T or lowercase h). Descent refers to the distance from the baseline downwards to the bottom of the shortest character (e.g., lowercase g or p). The standard physical line height is calculated by adding Ascent and Descent. And calculate the line spacing according to a preset ratio (such as 30% of the line height). Define step size .

[0070] For the bounding box currently being processed The maximum number of rows it can hold is calculated as follows:

[0071]

[0072] in For the enclosure height, This indicates rounding down to the nearest integer.

[0073] Subsequently, a low-level hard truncation is performed on the long string to extract the preceding text. Write directly to row Within the pixel array. After writing is complete, the rendering engine synchronously updates the text memory pointer of the state machine, using the truncated remaining string as an overflow stream and importing it into the next bounding box. The calculations continue. This mechanism restores the sequential reading semantics of long texts within a two-dimensional pixel matrix, making the rendering results visually consistent with the cross-column layout of real academic papers.

[0074] S214, Special symbols for structured vector geometric reconstruction.

[0075] In form-type documents, checkbox symbols ( , , The alignment quality directly affects the logic recognition accuracy of the model. Because different operating systems and font libraries have different definitions of character spacing (kerning) and baseline, directly printing characters can cause vertical offset or garbled text. The rendering engine implements hard vector geometric constraints to address this.

[0076] Geometric anchoring: based on the currently calculated row height Calculate the geometric constraint side length of the checkbox. ,in This is the current font size.

[0077] Mathematical definition of coordinate point: The coordinate point is defined by the top-left vertex of the checkbox. Define the checked state within the tensor space, with the origin as the origin. Three key features of )

[0078]

[0079]

[0080]

[0081] Vector line drawing: Using a graphics rendering engine to draw lines directly at physical coordinate points. Linear interpolation is performed between them to draw vector line segments.

[0082] This font-independent, geometry-driven drawing logic ensures that checkbox symbols maintain absolute physical centering and pixel-level clarity at any resolution, solving the problem of poor support for special symbols in Chinese fonts.

[0083] S22, OTSL table structure restoration.

[0084] Since Doctags uses OTSL (One-line Table Structure Language) syntax to compress a high-dimensional table space into a one-dimensional instruction stream, its reverse reconstruction process can be divided into two core parts: table structure parsing and optimal column width allocation.

[0085] S221, Table Structure Analysis.

[0086] This embodiment intercepts logical anchor points in a one-dimensional text stream, reversing the highly compressed character sequence into a grid matrix with geometric properties. The existing parsing engine's built-in set of marker operators is not limited to basic cells but also includes depth markers for describing hierarchy and merging relationships. Specific examples of these markers are as follows:

[0087] <fcel>(Functional Cell): Defined as the main cell node containing substantial text content, it is the foundation of the grid index.

[0088] <ecel>(Empty Cell): Defined as a blank master cell, used to maintain the integrity of the table's logical structure and avoid index offset caused by empty cells.

[0089] <lcel>(Left-merge Cell): Column merge operator. When the state machine scans this marker, it triggers a left-backtracking logic, merging the nearest parent cell in the current row. <fcel>or <ecel>The column span cs increases.

[0090] <ucel>(Up-merge Cell): Row merge operator. This operator triggers an upward backtracking logic, locating the corresponding parent cell above the logical grid and incrementing its row span rs.

[0091] <xcel>(Cross-merge Cell): Cross-merge placeholder. Used to identify a spatial region that is simultaneously constrained by row merging and column merging, ensuring that coordinate grids do not overlap in calculations during multi-directional merging scenarios.

[0092] <ched>and <rhed>(Column / Row Header): Semantic header tag. Used to distinguish the header area from the data area. The rendering engine will adjust the border thickness or font bolding attribute accordingly to enhance the semantic features of the layout.

[0093] <srow>(Start Row) and <nl>(Newline): Line start and end marker. Used to define the vertical growth boundary of a two-dimensional matrix, enabling dimensional transformation from one-dimensional to two-dimensional arrays.

[0094] In this embodiment, a two-dimensional matrix `grid_tokens` is constructed. First, all tokens are sequentially scanned and their corresponding coordinates are filled in. Then, a span deduction algorithm is executed: traversing every non-primary cell node in the matrix. If the node type is `lcel`, the algorithm performs a horizontal leftward search until the starting primary node is found and execution is performed. If it is uCEL, then perform a vertical upward search and execute... This recursive backtracking mechanism can accurately deduce the absolute values ​​required for rendering each cell. and Weights are used to solve the border snapping problem when complex nested tables are reverse-drawn.

[0095] S222. Optimal column width is allocated based on the water-filling algorithm.

[0096] After obtaining the topology, the core problem facing the rendering engine is how to allocate the width of each column to ensure that the table can fill the target width perfectly. It also tries to avoid distorting short text columns due to uneven column width distribution.

[0097] In this embodiment, a water-filling algorithm is used for dynamic column width weight calculation. First, based on the basic physical width of the cell without wrapping, combined with cross-column amortization and the minimum column width lower limit, the ideal width requirement for each column is determined. Then, the sum of the ideal widths of all columns is calculated, and based on the comparison with the total available width of the table, two scenarios are considered: if the total space is ample, the ideal requirements of each column are fully satisfied, and the remaining space is evenly distributed among the columns; if the total available space is limited, a greedy dynamic iterative allocation is initiated. During the iteration process, the current average remaining quota of the unallocated columns is calculated in each round. Low-bandwidth columns whose ideal requirements are less than the average remaining quota are prioritized for full satisfaction and removed from the unallocated set, with the remaining space flowing to the next round. When the requirements of all remaining columns exceed the average remaining quota, the remaining space is distributed according to the proportion of the ideal requirements of each column, while a minimum column width constraint is applied. Finally, rounding errors caused by floating-point calculations are compensated for in the last column. The specific algorithm process is as follows:

[0098] S2221. Ideal Width Calculation and Lower Limit Constraint: Calculate the basic physical width of all cells without line breaks. (Including text width and left and right inner margins). If the cell spans... If a column is spanned, its width contribution is evenly distributed across the columns it spans. Traversing the first Extract the maximum value of the contribution of all cell widths in the column and compare it with the set minimum column width lower limit. (e.g., 10 pixels) are compared, and the larger of the maximum value and the minimum column width lower limit is selected as the first value. The final ideal width requirement for the column .

[0099] S2222, Greedy Water Injection Allocation Decision: Let the total available width of the table be... Calculate the sum of the ideal widths of all columns. .

[0100] Scenario A (Ample Space Allocation): If This indicates that there is sufficient space and no line breaks are needed. Allocate the ideal width to each column. and the extra space Distribute evenly across all columns. Column width allocation for:

[0101]

[0102] in, This represents the total number of columns.

[0103] Scenario B (Space-Constrained Iterative Allocation): If This means that some columns must be sacrificed for line breaks. Let the set of unallocated columns be... Current remaining available width Initially In each round of calculation, an absolute fair allocation threshold (quota) is defined. :

[0104]

[0105] in, This represents the number of unassigned columns.

[0106] Perform a greedy filter: If a certain column exists in the set... This indicates that the column is a low-bandwidth column (such as serial number or date), and the rendering engine directly meets its ideal width requirement by allocating column width accordingly. and from Subtract this value from the column and move the column out. Set up and proceed to the next iteration. Forcefully divide proportionally: when... Only the high-bandwidth column remains (i.e., all remaining columns satisfy the condition). When the algorithm allocates the remaining space according to the ideal requirements of each column, it simultaneously applies the minimum column width. constraint:

[0107]

[0108] S2223, Rounding error correction:

[0109] Since pixel-level rendering coordinates must be integers, the above-described rounding down operation will result in the final allocated total width being less than or equal to the integer coordinates. There are slight pixel differences. Calculate the sum of the widths of all currently allocated columns. The difference is directly compensated to the last column to ensure that the table width is absolutely tight.

[0110] This algorithm ensures the readability and aesthetics of the table at the pixel level by protecting narrow columns from excessive compression. Finally, the absolute pixel coordinates of each cell vertex are obtained using the cumulative sum sequence of width and height, which then drives the drawing rendering engine to reconstruct the mesh entity. An example of the cumulative sum sequence is: if the widths (in millimeters) of three cells are 10, 20, and 30 respectively, then the corresponding cumulative width sum sequence is {0, 10, 30, 60}.

[0111] S23, Heterogeneous modality replication.

[0112] In this embodiment, for heterogeneous elements in the document that cannot be rendered using fonts (such as illustrations and complex mathematical formulas) and structured symbols that are prone to causing layout alignment errors, the rendering engine triggers a copy mechanism. That is, for Doctags... <picture> 、 <figure> 、 <formula>Copy tags. This method avoids the problem of redrawing complex formulas or natural images through code, while preserving the visual authenticity of the original document.

[0113] Figures 3 to 5 This demonstrates a test case that combines heterogeneous modality and text reconstruction. Figures 6 to 8 A test example combining table structure and text reconstruction is presented. The test results show that the document image reverse-rendered in this embodiment maintains a high degree of consistency with the Doctags annotation information, eliminating the mismatch error between the original image and the annotation information.

[0114] S3. Dataset encapsulation: Associate and store the synthesized document images with the corresponding Doctags information to form image-tag pairs in order to construct the dataset.

[0115] In this embodiment, after the rendering engine completes the reverse redraw of the entire page, it enters the final data encapsulation and serialization stage, converting the synthesized image and corresponding Doctags into the Parquet dataset format with high-throughput reading capabilities. Specifically, this includes the following steps:

[0116] S31. Dataset Proportional Partitioning and Sample Sampling Logic: Introducing a Random Seed To ensure the reproducibility of the experiment, The preset ratio is used to perform the train / test split. Let the total sample set be... training set With test set The capacity satisfies the constraints:

[0117]

[0118] in, This indicates the number of elements in the set.

[0119] The partitioned samples are stored in separate directory structures, and a metadata.json file is generated synchronously as the index baseline. The structure of a single sample is as follows:

[0120]

[0121] S32. Multimodal Alignment and JSONL Payload Construction Logic: The filename hash verification and metadata mapping program is executed. Since the rendered image filenames (e.g., render_pic_page1.png) and their corresponding tag files (e.g., page1.doctags) have a definite derivation relationship, the encapsulation script establishes a one-to-one mapping index by parsing prefixes and suffixes. For each successfully matched data pair, it is encapsulated into a JSONL format conforming to the multimodal large-scale model training specification. Its single-line structure is defined as:

[0122]

[0123]

[0124] in, For fixed task instructions (such as "Please identify the images in this document and provide the doctags annotation results"), This corresponds to the Doctags string.

[0125] S33. Parquet Dataset Assembly: For large-scale (millions or more) datasets, the traditional ImageFolder format suffers from significant small file I / O bottlenecks. This embodiment employs the Parquet columnar storage format for deep data compression and serialization. The Pyarrow rendering engine is used to convert the image binary data and its corresponding structured tags into an Arrow table structure. To adapt to streaming delivery in a distributed cluster, this embodiment uses automatic sharding logic. The preset single-shard capacity is set to... If there are 10 samples, then the total number of segments is 1. The calculation formula is:

[0126]

[0127] in, This indicates rounding up to the nearest integer.

[0128] The fragmented files use a standardized naming convention, such as train-00000-of-00010.parquet. This format not only embeds image data into columns, reducing file addressing overhead, but also allows training frameworks (such as HuggingFace datasets) to utilize multi-threaded parallel loading, significantly improving data loading throughput during model training. Finally, subset directories are constructed based on the data source style, and a README.md file containing YAML configuration is written in the root directory. This configuration file defines the physical path mapping relationship of each subset, enabling the final "gold standard" dataset to be uploaded to HuggingFace and read by subset directory name.

[0129] This invention provides a document image synthesis and dataset automatic generation system based on the Doctags language, comprising: a preprocessing module for acquiring an original document and parsing its Doctags information, wherein the Doctags information includes at least the semantic category of each element in the document and the physical bounding box coordinates of each element on the page; and a reverse rendering module for generating a synthesized document image based on the Doctags information through reverse rendering, wherein the reverse rendering takes the Doctags information as input and draws visual elements aligned with the Doctags information on a canvas according to the semantic category and physical bounding box coordinates, thereby obtaining the synthesized document image. During the rendering process, each tag node in Doctags is traversed. For text-type tags, the text content and physical bounding box coordinates are parsed. An adaptive font size calculation is used to determine the font size that will maximize the filling of the specified bounding box with the text content. The determined font size is then used to render the text within the specified bounding box. For non-text tags such as images or formulas, the corresponding pixel regions are cropped from the pre-acquired original document images according to their physical bounding box coordinates. The cropped pixel regions are then pasted to the corresponding coordinate positions on the canvas. The dataset generation module is used to associate and store the synthesized document images with the corresponding Doctags information to form image-tag pairs to construct the dataset.

[0130] For the specific implementation details of each module, please refer to the aforementioned method implementation examples, which will not be repeated here.

[0131] The present invention also provides a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the document image synthesis and automatic dataset generation method based on the Doctags language described in any of the foregoing method embodiments.

[0132] The program code used to implement the method of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the steps of the method of the present invention to be performed. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a standalone software package, or entirely on a remote machine or server. All aspects not detailed in this invention are well-known to those skilled in the art.< / formula> < / figure> < / picture> < / nl> < / srow> < / rhed> < / ched> < / xcel> < / ucel> < / ecel> < / fcel> < / lcel> < / ecel> < / fcel> < / paragraph> < / text>

Claims

1. A method for document image synthesis and data set automatic generation based on Doctags language, characterized in that, Includes the following steps: Obtain the original document and parse the Doctags information of the document from the original document, wherein the Doctags information includes at least the semantic category of each element in the document and the physical bounding box coordinates of each element on the page; Based on Doctags information, a synthetic document image is generated through reverse rendering. The reverse rendering is to take Doctags information as input, draw visual elements on the canvas that are aligned with the Doctags information according to the semantic category and physical bounding box coordinates, thereby obtaining a synthetic document image. During the rendering process, each tag node in Doctags is traversed. For text-type tags, the text content and physical bounding box coordinates are parsed. The font size that can make the text content fill the specified bounding box to the maximum extent is determined by adaptive font size calculation, and the text is rendered within the specified bounding box using the determined font size. For non-text tags such as images or formulas, the corresponding pixel area is cropped from the pre-acquired original document image according to its physical bounding box coordinates, and the cropped pixel area is pasted to the corresponding coordinate position in the canvas. The adaptive font size calculation adopts a binary approximation algorithm, which includes: setting a preset font size search range, continuously taking the middle font size for text layout calculation during the iteration process, narrowing the search range based on the comparison results of the calculated text rendering width and height with the target bounding box, until the maximum font size that satisfies the specified bounding box constraint is obtained; The synthesized document images are associated and stored with the corresponding Doctags information to form image-tag pairs, thereby constructing a dataset.

2. The method for document image synthesis and automatic dataset generation based on the Doctags language according to claim 1, characterized in that, For streaming text that spans multiple columns or regions, the reverse rendering also includes: merging the parameters of multiple bounding boxes corresponding to the text to estimate the global font size, processing each bounding box in the order of reading, calculating the maximum number of lines that the current bounding box can accommodate, extracting the text corresponding to the number of lines for rendering, and passing the remaining text to the next bounding box for continued rendering.

3. The method for document image synthesis and automatic dataset generation based on the Doctags language according to claim 1, characterized in that, For special symbols like checkboxes in text type, the reverse rendering also includes: calculating the geometric dimensions of the checkbox based on the determined font size and line height, and drawing the checkbox and its check mark at a specified coordinate position using vector drawing.

4. The method for document image synthesis and automatic dataset generation based on Doctags language according to claim 1, characterized in that, For OTSL tables, when the Doctags information contains a description instruction for the table structure, the reverse rendering further includes: parsing the table structure description instruction, restoring the cell topology and row / column merging attributes of the table; dynamically allocating the width of each column using a water-filling algorithm based on the cell content and the total width of the table; calculating the physical coordinates of each cell based on the allocated column width, and drawing the table outline and filling the cell text content.

5. The method for document image synthesis and automatic dataset generation based on the Doctags language according to claim 4, characterized in that, The method of dynamically allocating the width of each column using a water-filling algorithm includes: Based on the basic physical width of the cell when it does not wrap, and combined with the average width across columns and the minimum column width limit, determine the ideal width requirement for each column; Calculate the sum of the ideal widths of all columns. Based on the comparison between this sum and the total available width of the table, there are two scenarios: If the total space is ample, the ideal requirements of each column are fully met, and the remaining space is evenly distributed among the columns; if the total available space is limited, a greedy dynamic iterative allocation is initiated. During the iteration process, the current average remaining quota of the unallocated columns is calculated in each round. Priority is given to fully meeting the ideal requirements of low-bandwidth columns that are less than the average remaining quota, and these columns are removed from the unallocated set. The remaining space is carried over to the next round; when the requirements of all remaining columns exceed the average remaining quota, the remaining space is divided according to the proportion of the ideal requirements of each column, while a minimum column width constraint is applied. The last digit is compensated for rounding errors caused by floating-point calculations.

6. The method for document image synthesis and automatic dataset generation based on the Doctags language according to claim 1, characterized in that, When rendering text, it also includes: randomly selecting a font from a preset font library for text drawing.

7. The method for document image synthesis and automatic dataset generation based on the Doctags language according to claim 1, characterized in that, The process of associating and storing the synthesized document images with their corresponding Doctags information includes: dividing the image-tag pairs into training and testing sets according to a preset ratio, and encapsulating them into a dataset file in Parquet columnar storage format; wherein the Parquet columnar storage format is used to store the dataset in segments according to a preset number of samples.

8. A document image synthesis and automatic dataset generation system based on the Doctags language, used to implement the document image synthesis and automatic dataset generation method based on the Doctags language according to any one of claims 1-7, characterized in that, include: The preprocessing module is used to obtain the original document and parse the Doctags information of the document from the original document, wherein the Doctags information includes at least the semantic category of each element in the document and the physical bounding box coordinates of each element on the page; The reverse rendering module is used to generate a synthetic document image based on Doctags information through reverse rendering. The reverse rendering is to take Doctags information as input, draw visual elements on the canvas that are aligned with the Doctags information according to the semantic category and physical bounding box coordinates, so as to obtain a synthetic document image. During the rendering process, each tag node in Doctags is traversed. For text-type tags, the text content and physical bounding box coordinates are parsed. The font size that can make the text content fill the specified bounding box to the maximum extent is determined by adaptive font size calculation, and the text is rendered within the specified bounding box using the determined font size. For non-text tags such as images or formulas, the corresponding pixel area is cropped from the pre-acquired original document image according to its physical bounding box coordinates, and the cropped pixel area is pasted to the corresponding coordinate position in the canvas. The dataset generation module is used to associate and store the synthesized document images with the corresponding Doctags information to form image-tag pairs in order to construct the dataset.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the steps of the document image synthesis and automatic dataset generation method based on the Doctags language according to any one of claims 1-7.

Images