A single certificate data synthesis method, device and equipment for multi-modal agent training and a storage medium

Abstract

Embodiments of the present application provide a single certificate data synthesis method and device for multi-modal agent training, equipment and storage medium. Including: obtaining a real single certificate image, erasing variable text and retaining fixed layout, establishing space coordinates and layout mapping of filled fields, obtaining a reference template; template tilt detection and reverse rotation correction, repair the erasing area; generate prompt words based on business logic, use large language model to generate batch of structured text meeting multi-field logic consistency; adaptively render text on the template according to the mapping configuration and superimpose simulated seal, then rotate it back according to the original tilt angle; apply cross-modal environmental noise enhancement to the rotated image, and synchronously output the synthesized image and structured label. The present application solves the problems of sample scarcity, inconsistent business logic and low spatial alignment accuracy in multi-modal agent training, and realizes the automatic generation of high-quality data with pixel-level precise alignment.

Description

Technical Field

[0001] This application relates to the field of multimodal data processing technology, specifically to a method, apparatus, device, and storage medium for synthesizing single-document data for training multimodal intelligent agents. Background Technology

[0002] In vertical sectors such as logistics, chemical engineering, and government affairs, the demand for multimodal intelligent agents to perform end-to-end structured information extraction (such as key-value pair extraction and automatic data entry) from document images is becoming increasingly urgent. To train such agents, it is necessary to rely on massive supervised fine-tuning (SFT) datasets with high alignment of "image visual features - spatial coordinate topology - structured text semantics".

[0003] However, real business document samples are extremely scarce due to the involvement of corporate privacy or confidential data, making large-scale compliant acquisition difficult in a commercial environment. Furthermore, manually annotating spatial coordinates and text content is time-consuming, labor-intensive, costly, and prone to errors. Existing general data augmentation techniques can only perform simple pixel-level operations on images, such as flipping, cropping, or adding Gaussian noise, and cannot generate text content that conforms to business logic, resulting in a lack of logical consistency constraints between multiple fields in the synthesized data. For example, in documents such as hazardous chemical business licenses or business permits, existing methods cannot simultaneously satisfy the check digit rules of coded fields, the temporal logical consistency of date fields, and the spatial semantic alignment requirements between place names.

[0004] Furthermore, existing technologies typically employ rigid, fixed-length splicing when attempting to fill blank templates with text, lacking dynamic control over text wrapping, maximum width limits, and line height. This results in overlapping, overflow, or formatting distortion in long text document fields after rendering. Additionally, traditional methods cannot simulate the characteristics of seals on real documents, as well as complex environmental factors such as geometric angles, lighting changes, and physical degradation during the scanning process.

[0005] The aforementioned defects cause the synthetic data to be distorted at both the semantic and visual levels, making it difficult for the agent to establish a stable "semantic-visual-spatial" three-in-one feature alignment relationship, which severely restricts the performance of information structure extraction in real and complex scenarios. Summary of the Invention

[0006] This application provides a method, apparatus, device, and storage medium for synthesizing single-document data for training multimodal intelligent agents.

[0007] According to a first aspect of this application, a method for synthesizing single-document data for training multimodal intelligent agents is provided, comprising:

[0008] Obtain at least one real document image, erase the variable text to retain a fixed layout, and establish a mapping configuration between the spatial coordinates of the fields to be filled and the layout parameters to obtain a baseline template;

[0009] The reference template is tilted and rotated in the opposite direction to obtain the corrected template. Then, the erased areas in the corrected template are restored.

[0010] Prompt words are generated based on business logic rules, and structured text data that meets multi-field logical consistency constraints is generated in batches using a large language model.

[0011] Based on the mapping configuration, text adaptive rendering is performed on the template after reverse rotation correction, and a simulated stamp is superimposed. Then, the rendered image is rotated back in the positive direction according to the original tilt angle.

[0012] Cross-modal environmental noise enhancement is applied to the cyclic image, and the synthesized image and structured labels are output simultaneously.

[0013] Furthermore, adaptive rendering includes: automatically wrapping text and adjusting the layout according to the preset line height when the width of the text to be filled exceeds the preset maximum width limit.

[0014] Furthermore, multi-field logical consistency constraints include at least one of the following: check bit rules for encoded fields, time logical consistency of date fields, and spatial semantic alignment between place names.

[0015] Furthermore, tilt detection employs a line detection algorithm; background restoration utilizes an image restoration algorithm.

[0016] Furthermore, the simulated stamp is a geometric primitive stamp generated through parameterization. When superimposed, it uses multiply or alpha channel blending for pixel-level fusion and introduces random rotation, transparency, and position offset.

[0017] Furthermore, cross-modal environmental noise enhancement includes at least one of the following: geometric distortion, color and light source perturbation, and physical image degradation.

[0018] Furthermore, the structured tags are in JSON format, and their field values ​​are aligned pixel-level with the text areas in the synthesized image. When there are multiple versions of real documents, an independent mapping configuration is established for each version, and multiple style samples are generated using the same tag pool.

[0019] According to a second aspect of this application, a single-document data synthesis apparatus for multimodal agent training is provided, the apparatus comprising:

[0020] The template building module is used to obtain at least one real document image, erase the variable text in it to retain a fixed layout, and establish a mapping configuration between the spatial coordinates of the fields to be filled and the layout parameters to obtain a baseline template.

[0021] The geometric correction and repair module is used to perform tilt detection and reverse rotation correction on the reference template, and to repair the background of the erased area;

[0022] The semantic generation module is used to generate prompt words based on business logic rules and to generate structured text data in batches that meet the multi-field logical consistency constraints using a large language model.

[0023] The rendering and stamping module is used to perform adaptive text rendering and overlay a simulated stamp on the template after reverse rotation correction based on the mapping configuration, and then rotate the rendered image back in the positive direction according to the original tilt angle.

[0024] The post-processing and output module is used to apply cross-modal environmental noise enhancement to the cyclic image and simultaneously output the synthesized image and structured labels.

[0025] According to a third aspect of this application, an electronic device is provided, the electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the single-document data synthesis method for multimodal agent training as described above.

[0026] According to a fourth aspect of this application, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the single-document data synthesis method for multimodal agent training as described above.

[0027] The technical solution provided in this embodiment achieves fully automatic closed-loop data synthesis through the above steps, effectively solving the problems of sample scarcity, logical inconsistency, and low alignment accuracy.

[0028] It should be understood that the description in the Summary Section is not intended to limit the key or essential features of the embodiments of this application, nor is it intended to restrict the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0029] The above and other features, advantages, and aspects of the embodiments of this application will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this application. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0030] Figure 1This is a flowchart illustrating the single-document data synthesis method for training multimodal intelligent agents provided in this embodiment;

[0031] Figure 2 This is a schematic diagram of the single-document data synthesis device for multimodal agent training provided in this embodiment. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0034] In training multimodal agents for specific industry documents (such as hazardous chemical business licenses, business licenses, customs declarations, etc.), real business samples are extremely scarce and involve privacy risks. Existing general data augmentation techniques cannot simulate the consistency of multi-field business logic across modalities, and manually annotating spatial coordinates and text labels is costly, inefficient, and prone to errors, making it difficult for agents to establish stable semantic-visual-spatial three-dimensional feature alignment. To address the above problems, this application provides a fully automated closed-loop data synthesis method.

[0035] Example 1

[0036] Figure 1 This is a flowchart illustrating the method for synthesizing single-document data for multimodal agent training provided in this embodiment. For example... Figure 1 As shown, as an optional implementation method, the document data synthesis method includes the following steps.

[0037] Step S1: Construct the baseline template and space mapping configuration

[0038] Specifically, obtain at least one authentic document image (e.g., a scanned copy of a hazardous chemicals business license). Using a graphical editing tool, erase the pixels of the variable text fields in the image (i.e., changing values ​​such as company name, certificate number, and issuance date), while fully preserving the fixed layout elements in the image (including titles, table lines, fixed key value fields, background textures, and anti-counterfeiting patterns), and output a clean baseline document template image without any specific text content.

[0039] On the baseline document template image, the absolute spatial coordinates (x, y) of each variable text field to be filled are determined, and these coordinates are bound to the field's preset layout topology parameters to construct a unified document field spatial mapping configuration table. The layout topology parameters include at least: font size (font_size), maximum width limit (max_width), and line height (line_height). For example, for the long text field "Permitted Scope," the configuration table can record its top-left corner coordinates as (120, 450), max_width = 300 pixels, and line_height = 24 pixels.

[0040] Step S2: Geometric Correction and Background Restoration

[0041] Considering that the initial input document images often have slight tilts due to shooting or scanning, this embodiment uses the probabilistic Hough linear transform algorithm to detect straight lines in the main table lines or text edges of the benchmark document template, adaptively calculating the initial tilt angle θ of the template image (e.g., 2.5° counterclockwise). Using the image rotation matrix, the benchmark document template is rotated in the opposite direction (rotation - θ) based on the initial tilt angle θ to obtain a horizontally normalized template image, thereby eliminating the interference of geometric tilt angles on subsequent rendering coordinate calculations.

[0042] Furthermore, for any remaining pre-filled text or noise artifacts after erasing variable text, a corresponding region of interest mask is constructed. An image inpainting algorithm based on the Fast Marching Method is then used for background texture reconstruction and color smoothing to ensure the area to be written is clean, flawless, and has continuous texture. For example, the rectangular area containing the original company name is repaired by using surrounding background pixels to fill the gaps and eliminate the afterimages of the original text.

[0043] Step S3: Structured Text Generation Driven by a Large Language Model

[0044] Based on the compliance requirements of the target documents in real-world industry scenarios, the corresponding multi-dimensional verification logic of entity fields is sorted out and transformed into prompts that can be understood by a large language model. The multi-field logical consistency constraints contained in the prompts include at least the following: the unified social credit code must meet the 18-digit encoding and specific verification bit rules; the difference logic between the issuance date and the validity period (such as 3 years or 5 years); and the administrative division contained in the company name (such as "XX Province XX City") must maintain spatial semantic consistency with the company's registered address.

[0045] Optionally, a small number of standard structured document text dictionary examples (Few-Shot) can be incorporated into the prompt words. These prompt words are then input into a large language model with strong generative capabilities (such as GPT-4 or Wenxin Yiyan) via a remote API call. Under the aforementioned strong rule constraints, the large language model generates massive amounts of highly diverse structured text data in batches, storing it as a JSON-formatted structured tag pool to be populated. For example, if the prompt words explicitly constrain "if the issuing authority is 'Beijing Chaoyang District Market Supervision Administration,' then the company's address must contain the phrase 'Beijing Chaoyang District,'" each piece of data generated by the large language model based on this constraint automatically satisfies the semantic consistency of this space.

[0046] Step S4: Adaptive rendering, stamp overlay, and forward rotation

[0047] A text data entry containing multiple field values ​​is randomly selected from the structured tag pool to be filled. The document field space mapping configuration table is read, and the corresponding text is written at the corresponding coordinates of the horizontally standardized template image after reverse rotation correction using the layout rendering engine. For long text fields (such as permitted ranges), the rendering engine performs boundary determination based on preset max_width and line_height parameters: when the projected width of the target text exceeds max_width, the rendering engine automatically wraps at word boundaries or character gaps and adjusts the line height according to line_height to avoid semantic breaks caused by simple truncation. In this way, the overall layout topology of the document remains stable.

[0048] Building upon this, this embodiment also provides a simulated seal. A circular or elliptical virtual document seal image is dynamically constructed based on a geometric primitive algorithm. The seal includes a central geometric pentagram, a ring-shaped unit name, and a semi-transparent red channel with a specific ratio. When overlaying the seal, independent random variable factors are introduced for each sample image: random rotation angle (e.g., ±15°), random transparency (e.g., 0.7~0.95), and random position offset (Δx, Δy). Multiply blending mode or an Alpha channel blending model based on the characteristics of real ink is used for multi-channel pixel-level fusion, ensuring that the underlying text skeleton of the seal-covered area remains visible, reproducing the physical overlay effect of "red ink overlaying black ink."

[0049] After the layout rendering and stamp overlay are completed, the image is rotated in the positive direction (rotation + θ) using the initial tilt angle θ detected in step S2, so that it is restored to the real physical state of scanning with a slight tilt angle, ensuring that the fixed key value of the document and the synthesized value maintain an absolutely consistent tilt angle.

[0050] Step S5: Enhance environmental disturbance and synchronize tag output

[0051] To make the synthesized document images visually approximate complex real-world physical environments and improve the anti-interference capability of multimodal agents, cross-modal environmental noise fusion is applied to the images after forward rotation. The cross-modal environmental noise enhancement includes at least one of the following:

[0052] Geometric distortion simulation: Introducing minute quadratic rotations (-2°~+2°), random scaling (0.95~1.05), and slight perspective affine transformations to simulate shooting distortion.

[0053] Color and light source disturbance: By adjusting the multi-channel color balance, it simulates various realistic scanning environments such as yellowish paper, bluish office light sources, and grayscale photocopies.

[0054] Physical image degradation: Gaussian blur with a random kernel size (standard deviation 0.5~1.5) simulates inaccurate focusing, dynamic adjustment of brightness and contrast simulates abnormal exposure, and physical spot reflection occlusion.

[0055] While the final heterogeneous environment document sample image is stored on the ground, the system losslessly maps the structured text field dictionary corresponding to the sample, which was initially generated by the large language model, and synchronously binds and outputs it as a standard fine-tuning (SFT) training label file (JSON format), thus completing the closed loop of the entire "image-space-semantic" data synthesis.

[0056] Through the above steps, a high-fidelity industry-level fine-tuning dataset can be constructed with only a small number of real samples. The synthesized images and structured labels achieve pixel-level spatial alignment, which greatly improves the information structure extraction performance of multimodal agents in real and complex scenarios.

[0057] Example 2

[0058] Based on Example 1, this embodiment further defines the specific implementation process of adaptive rendering in order to solve the problem of page overflow or structural distortion caused by forcibly rendering long text fields within fixed spatial coordinates.

[0059] Specifically, the adaptive rendering process includes: when the width of the text to be filled exceeds the preset maximum width limit, automatically wrapping the text and adjusting the layout according to the preset line height.

[0060] Specifically, when the rendering engine calculates the cumulative width of the text to be filled character by character or word by word, it compares it with max_width in real time. Once it detects that the cumulative width of the current line of text is about to exceed the limit, it immediately inserts a newline character before the current character, moves the remaining text to the beginning of the next line, and forces the vertical distance between the baseline of the next line and the baseline of the current line to be strictly equal to line_height.

[0061] For example, when processing the "Permit Scope" field, if max_width=400 pixels and line_height=30 pixels, the total width of the text to be filled, "Operation of Dangerous Chemicals (excluding highly toxic chemicals)," is 520 pixels in the selected font. The rendering engine determines that 520>400, truncates and wraps the text at the 18th character, and arranges the remaining text on the second line, with the baseline spacing between the two lines strictly equal to 30 pixels. In this way, the text is prevented from horizontally breaking through the field boundary, preventing OCR or LayoutLM-type models from misjudging the paragraph level due to uneven line spacing.

[0062] This embodiment ensures that business text of any length is strictly constrained within a preset spatial topology by using dynamic boundary calculation and layout adjustment, effectively avoiding spatial feature learning bias in multimodal agents caused by rendering distortion of long text.

[0063] Example 3

[0064] This embodiment, based on embodiment 1, further defines the specific types of multi-field logical consistency constraints.

[0065] Specifically, this includes the check digit rules for encoded fields, the time logic consistency of date fields, and the spatial semantic alignment between place names.

[0066] Check digit rules for encoded fields: The generated industry-specific codes must be verified by a mathematical verification algorithm defined by national standards or industry specifications. For example, the Unified Social Credit Code must be 18 digits long, and the 18th check digit must be calculated according to GB 32100-2015 to ensure that the modulo 31 weighted remainder relationship between the first 17 body codes and the check digit is valid, thus preventing the problem of legal format but illegal semantics caused by random character concatenation.

[0067] The date field must be logically consistent: multiple time-related fields must satisfy the preset time span logic and sequence constraints. For example, the difference between the certificate issuance date and the expiration date must be equal to the legal validity period (3 or 5 years), and the certificate issuance date must not be later than the current system time to ensure that the certificate is within its validity period.

[0068] Spatial semantic alignment between place names: Names involving administrative divisions in the text must maintain strict consistency in geographical hierarchy and affiliation. For example, if a company name contains "XX Province XX City XX District", then the company's registered address must belong to the same administrative division, and logically contradictory address combinations such as those spanning provinces or cities are prohibited.

[0069] This embodiment internalizes complex business rules into hard constraints in prompt words, guides the large language model to perform multi-field joint reasoning and cross-validation, significantly reduces the illusion rate of generated content, makes the synthesized text have real business rationality, and greatly improves the generalization ability of multimodal intelligent agents in complex logical judgment tasks.

[0070] Example 3A: Specific Implementation of Structured Text Generation Based on a Large Language Model

[0071] This embodiment details how to use a Large Language Model (LLM) to generate structured text data in batches that meets multi-field logical consistency constraints.

[0072] I. Prompt word design and rule encoding

[0073] For the target document type (e.g., "Hazardous Chemicals Business License"), first identify all fields to be filled and their business logic constraints. The constraint types should at least include:

[0074] - Encoding verification rules The Unified Social Credit Code must conform to the GB 32100-2015 standard, and be 18 digits long. The 18th digit is a check code calculated by weighting the first 17 digits.

[0075] - Date logic consistent The certificate issuance date must not be later than the current system time, and the expiry date of the validity period = the certificate issuance date + the statutory validity period (e.g., 3 years or 5 years).

[0076] - Spatial semantic alignment The administrative division in the company name (such as "Chaoyang District, Beijing") must be completely consistent with the administrative division in the company's registered address.

[0077] These rules are converted into natural language prompts, and several standard examples are provided using a few-shot approach. Below is a specific prompt template (applicable to models such as OpenAI GPT-4 and Baidu Wenxin Yiyan):

[0078] You are a professional hazardous chemicals business license data generator. Please generate a complete structured license data entry according to the following rules, with the output format being JSON.

[0079] rule:

[0080] 1. Unified Social Credit Code: 18 digits, conforming to the check digit algorithm. The first 9 digits are fixed as "91110108" (Haidian District code), and the last 9 digits are randomly generated but need to pass the verification.

[0081] 2. Company Name: The format is "Beijing XX Technology Co., Ltd.", where "XX" consists of 2-4 random Chinese characters. The "Beijing" in the name must match the "Beijing Municipality" in the registered address.

[0082] 3. Residence address: Must include "Haidian District, Beijing", and the street name is random.

[0083] 4. Issuance date: Randomly selected from 2020 to 2025, in the format YYYY-MM-DD.

[0084] 5. Expiry date: Issuance date + 5 years, format YYYY-MM-DD.

[0085] 6. Scope of permit: 3 to 5 categories of hazardous chemicals will be randomly generated, with each category separated by a Chinese semicolon.

[0086] Example output: { Unified Social Credit Code: "ABCDEFGHIG12345678" Company Name: Beijing XXX Technology Co., Ltd. Address: No. X, Zhongguancun Street, Haidian District, Beijing "Issuance Date": "2023-05-20" "Expiration Date": "2028-05-19" "Permitted Scope": "Liquid chlorine; ammonia solution [containing >10% ammonia]; hydrogen peroxide solution [containing >8%]" }

[0087] Now please generate a new data entry, outputting only JSON and without any additional explanation.

[0088] text

[0089] Copy and download the generated content.

[0090] II. Batch Generation and Verification Feedback

[0091] To improve generation efficiency and diversity, the following steps are adopted:

[0092] 1. Batch requests The API of the large language model is called asynchronously in a loop, generating one or more (e.g., 10) data entries per request. A temperature parameter (temperature = 0.8) is set in the request parameters to increase randomness, while top_p = 0.95 is set to maintain semantic validity.

[0093] 2. Post-rule validation The system parses the JSON string returned by the model and recalculates the checksum, date difference, string inclusion relationships, etc., using local code (Python script). If a piece of data fails validation, it is discarded and a new request is made (maximum of 3 retries).

[0094] 3. Tag pool storage The validated JSON data is stored in a structured label pool, either locally or in the cloud. Each label pool file can contain 100,000 to 1 million records for subsequent rendering.

[0095] III. Specific Call Examples

[0096] Example 1: Using OpenAI GPT-4

[0097] The following code demonstrates how to use the OpenAI Python SDK:

[0098] Python

[0099] import openai

[0100] openai.api_key = "your-api-key"

[0101] prompt = """(The complete prompt above)""

[0102] response = openai.ChatCompletion.create(

[0103] model="gpt-4",

[0104] messages=[{"role": "user", "content": prompt}],

[0105] temperature=0.7,

[0106] max_tokens=500 )

[0108] generated_json = response.choices[0].message.content

[0109] print(generated_json)

[0110] Example 2: Using Baidu Wenxin Yiyan ERNIE-Bot Construct the request body via Baidu AI Cloud API call:

[0111] Python

[0112] Copy and download the generated content

[0113] import requests

[0114] url = "https: / / aip.baidubce.com / rpc / 2.0 / ai_custom / v1 / wenxinworkshop / chat / completions"

[0115] payload = {

[0116] "messages": [{"role": "user", "content": prompt}],

[0117] "temperature": 0.7,

[0118] "extra_parameters": {"rule_constraint": "strict"}

[0119] }

[0120] headers = {"Content-Type": "application / json", "Authorization": "Bearer your_access_token"}

[0121] response = requests.post(url, json=payload, headers=headers)

[0122] generated_json = response.json()["result"]

[0123] Example 3: Local Deployment of an Open Source Model (LLaMA-3-8B) In offline environments without internet access, text can be generated using a quantized LLaMA-3-8B model via the HuggingFaceTransformers library:

[0124] Python

[0125] Copy and download the generated content

[0126] from transformers import pipeline

[0127] generator = pipeline("text-generation", model="meta-llama / Llama-3-8B", device=0)

[0128] prompt = "(prompt word)"

[0129] output = generator(prompt, max_new_tokens=500, temperature=0.7)

[0130] generated_text = output[0]["generated_text"]

[0131] # Parse the JSON part

[0132] Although open-source models are less capable of following complex rules, a large amount of qualified data can still be generated by strengthening the few-shot examples and combining them with backend validation.

[0133] IV. Examples of handling multi-field joint constraints

[0134] This embodiment particularly emphasizes cross-field joint generation capabilities. For example, a generation request requires the large language model to simultaneously guarantee:

[0135] When the issuing authority is the "Beijing Haidian District Market Supervision and Administration Bureau", the "business address" must include "Haidian District";

[0136] The difference between the "expiration date" and the "issuance date" is exactly 365 × 5 days (taking leap years into account).

[0137] Large language models can simultaneously consider these constraints through attention mechanisms, generating logically consistent and complete records. Below is an example of actual generated data (after anonymization):

[0138] json

[0139] Copy and download the generated content

[0140] {

[0141] Unified Social Credit Code: "ABCDEFGH1234567890"

[0142] Company Name: Beijing XX Chemical Co., Ltd.

[0143] Address: No. X, Beiqing Road, Haidian District, Beijing

[0144] Legal Representative: Zhang San

[0145] "Issuance Date": "2022-03-15"

[0146] "Expiration Date": "2025-03-14"

[0147] "Scope of Permit": Toluene; Sulfuric Acid; Hydrochloric Acid; Sodium Hydroxide

[0148] }

[0149] The data was verified by the local verification program. The unified social credit code check digit was correct, the date difference was accurate to 1826 days (5 years, including one leap year), and the "Beijing" in the company name was aligned with the administrative division of "Haidian District, Beijing" in the address, which fully complies with the business rules.

[0150] Through the above method, this embodiment achieves low-cost, highly diverse, and logically consistent batch generation of structured text, solving the problem of missing business logic in training data.

[0151] Example 4

[0152] This embodiment, based on embodiment 1, further specifies the specific algorithms used for tilt detection and background restoration.

[0153] Step 1: Tilt detection uses a straight line detection algorithm, specifically the probabilistic Hough linear transform.

[0154] The baseline template, after variable text erasure, is grayscaled and Gaussian filtered for noise reduction to highlight strong geometric features such as table lines and border lines. A probabilistic Hough transform is used to accumulate collinear pixels in the parameter space, detecting a set of line segments whose length exceeds a preset threshold. The tilt angle distribution of each detected line segment relative to the horizontal axis is calculated, and the initial tilt angle θ is determined by statistical mode or weighted average. For example, when the document image has slight curvature or partial occlusion, the probabilistic Hough transform can ignore the interference of broken short lines, accurately locking the long table lines that run through the page as the main reference line, and calculating the tilt angle accurate to two decimal places.

[0155] Step 2: Background restoration employs an image restoration algorithm, specifically an image restoration based on the Fast Marching Model (FMM).

[0156] After obtaining the erased area mask of the field to be filled, the algorithm iteratively propagates the color and gradient information of known pixels into the region, starting from the mask boundary. Following the direction of iso-illuminance lines, it moves from the known area to the unknown area, prioritizing the repair of edge structures (such as table lines and background textures) before filling in smooth color areas. For example, when the blank area left after erasing the company name spans the anti-counterfeiting wavy pattern, the algorithm automatically identifies the direction and frequency of the wavy pattern, extracts features from the complete patterns on both sides, and seamlessly reconstructs a continuous wavy texture in the blank area, avoiding breaks or blurred patches.

[0157] This embodiment overcomes the texture breakage defects caused by traditional simple interpolation repair by using a processing mechanism of "first straightening detection, then horizontal repair, and finally corner rotation". The generated reference template achieves pixel-level perfection in geometric structure and visual texture.

[0158] Example 5

[0159] This embodiment, based on embodiment 1, further defines the parametric generation and pixel-level fusion method of the simulated seal.

[0160] Step 1: The simulated stamp is a geometric primitive stamp generated through parameterization.

[0161] Instead of relying on pre-scanned images of physical seals, this system uses vector graphics principles to dynamically construct the seal's components via code: a circular or elliptical outer contour path, a central five-pointed star pattern, and the unit's name text arranged along a circular path. The size, line width, font style, and relative position of each component are controlled by preset parameter variables. For example, adjusting the radius parameter changes the seal size, and modifying the ratio of the major and minor axes switches a circular official seal to an elliptical financial seal.

[0162] Step 2: When overlaying, use either a multiply blending mode or an alpha channel blending mode to perform pixel-level fusion.

[0163] The formula for Multiply blending is Result Color = Base Color × Blend Color / 255, simulating the attenuation effect of light passing through translucent red ink onto dark text or light-colored paper. When the red stamp is overlaid on black printed text, the blended area becomes significantly darker, exhibiting the visual characteristic of "red ink overlaying black ink"; when overlaid on a white background, it retains its vibrant red color. Alpha channel blending controls the weighted average of stamp pixels and background pixels by setting the opacity channel value of the stamp layer, simulating the translucency effect caused by different degrees of ink dryness.

[0164] Step 3: Introduce random rotation, transparency, and position offset.

[0165] In each stamp overlay operation, three random variables are independently sampled: random rotation angle (e.g., ±15°), random transparency coefficient (e.g., 0.7~0.95), and random position offset (Δx, Δy). For example, when generating 1000 composite documents in batches, the first stamp is rotated 3° clockwise, has a transparency of 0.82, and is offset 5 pixels to the lower right; the second stamp is rotated 8° counterclockwise, has a transparency of 0.91, and is offset 3 pixels to the upper left.

[0166] This embodiment achieves the synthesized image possesses visual features in the signature area that are difficult to distinguish from real business documents through the synergistic effect of parametric generation, physical-optical fusion, and multidimensional random perturbation. This avoids repetitive mechanical features in the training data and enhances the model's generalization ability.

[0167] Example 6

[0168] This embodiment, based on embodiment 1, further defines the specific types of cross-modal environmental noise enhancement.

[0169] Specifically, geometric distortion, color and light source perturbation, and physical image degradation.

[0170] Geometric distortion: Introducing minute quadratic rotations (-2°~+2°), random scaling factors (0.95~1.05), and slight perspective affine transformations to simulate lens distortion or planar distortion caused by uneven paper placement during handheld shooting.

[0171] Color and Light Source Disturbance: Adjust multi-channel color balance to simulate the color temperature of yellowish old paper, the bluish hue of office lighting, and the grayscale effect caused by low-quality photocopies.

[0172] Physical image degradation: Applying Gaussian blur with a random kernel size (standard deviation 0.5~1.5) to simulate inaccuracy, dynamically adjusting brightness and contrast to simulate exposure anomalies, introducing JPEG compression artifacts to simulate network transmission loss, and adding random spot reflections to simulate plastic film reflections or stains.

[0173] This embodiment organically combines three heterogeneous perturbations—geometric, color, and physical—to form a systematic joint degradation model, constructing extremely challenging training samples that force the multimodal agent to learn more essential and robust semantic-visual-spatial alignment features.

[0174] Example 7

[0175] Based on Example 1, this embodiment further defines the output format of structured tags and the reuse mechanism in multi-format scenarios.

[0176] Step 1: The structured tags for synchronous output are set to JSON format, and their field values ​​are aligned pixel-level with the text areas in the composite image.

[0177] The JSON tag's data structure contains at least one set of key-value pairs, each corresponding to a business field in the document. The field value not only contains the text content string generated by the large language model but also strictly binds to the pixel-level bounding box coordinates (e.g., xmin, ymin, xmax, ymax) of that text in the final composite image. For example, if the company name field is rendered within a rectangular area at image coordinates (120, 450) to (580, 490), then the bbox attribute value of that field in the JSON tag will be [120, 450, 580, 490], achieving zero-error matching.

[0178] Step 2: When there are multiple versions of real documents, establish an independent mapping configuration for each version and use the same tag pool to generate multiple style samples.

[0179] For each specific document format, a dedicated document field spatial mapping configuration table is constructed, defining the names of each field to be filled under that format, the spatial topology of fixed layout elements, and the corresponding layout parameters. Regardless of the current format being processed, the same structured text tag pool generated by the large language model is invoked. This tag pool only contains semantic content and does not carry spatial information specific to the format. For example, a structured data entry containing "Unified Social Credit Code: 91110108MA00XXXX" and "Issuance Date: May 2023" is generated in the tag pool. When generating a business license format sample, the mapping configuration for the business license is invoked, rendering the code in the center of the top of the layout; when generating a customs declaration format sample, the mapping configuration for the customs declaration is switched, rendering the same code in the upper left corner of the layout.

[0180] This embodiment, through an architecture design of independent configuration + shared pool, not only ensures the independence of different document formats in terms of spatial layout, but also realizes the intensive use of semantic generation resources, which greatly reduces the synthesis cost of multi-category document datasets.

[0181] Example 8

[0182] like Figure 2 As shown in the illustration, this application also provides a single-document data synthesis device for training multimodal intelligent agents, the device comprising:

[0183] Template building module 201: Used to obtain at least one real document image, erase the variable text in it to retain a fixed layout, and establish a mapping configuration between the spatial coordinates of the fields to be filled and the layout parameters to obtain a baseline template.

[0184] Geometric correction and repair module 202: Used to perform tilt detection and reverse rotation correction on the reference template, and to repair the background of the erased area.

[0185] Semantic generation module 203: Used to generate prompt words based on business logic rules, and to generate structured text data in batches that meet the multi-field logical consistency constraints using a large language model.

[0186] Rendering and Stamping Module 204: Based on the mapping configuration, it performs adaptive text rendering on the template after reverse rotation correction and overlays a simulated stamp, then rotates the rendered image back in the forward direction according to the original tilt angle.

[0187] Post-processing and output module 205: used to apply cross-modal environmental noise enhancement to the cyclic image and simultaneously output the synthesized image and structured labels.

[0188] The specific working process of each module corresponds one-to-one with the steps described in Examples 1 to 7, and will not be repeated here. Through the above modular design, this device realizes the pipelined deployment of the document data synthesis process. Each module can be upgraded independently and processed in parallel, improving the scalability and engineering practicality of the system.

[0189] This embodiment provides an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the method as described in any one of Embodiments 1 to 7.

[0190] The processor can be a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), or a digital signal processor (DSP). Memory can include random access memory (RAM), read-only memory (ROM), hard disk, solid-state drive (SSD), optical disk, or removable storage media.

[0191] In one deployment scenario, the electronic equipment is a server cluster, with multi-core CPUs and high-performance GPUs as processors. The program runs as a containerized service, supporting high-concurrency document data synthesis requests. In another deployment scenario, the electronic equipment is an industrial control machine with an integrated NPU. The program is optimized through quantization and compression, enabling it to independently generate small batches of document samples offline. This is suitable for field operation environments with high data privacy requirements and limited network access.

[0192] This embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method as described in any one of embodiments 1 to 7.

[0193] The computer-readable storage medium may be the storage medium included in the device described in the above embodiments, or it may be a standalone storage medium not assembled into the device. It includes, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, such as portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, etc.

[0194] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0195] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for synthesizing single-document data for training multimodal intelligent agents, characterized in that, The method includes: Obtain at least one real document image, erase the variable text to retain a fixed layout, and establish a mapping configuration between the spatial coordinates of the fields to be filled and the layout parameters to obtain a baseline template; The reference template is tilted and rotated in the opposite direction to obtain a corrected template. Then, the erased areas in the corrected template are restored. Prompt words are generated based on business logic rules, and structured text data that meets multi-field logical consistency constraints is generated in batches using a large language model. Based on the mapping configuration, text adaptive rendering is performed on the template after reverse rotation correction, and a simulated stamp is superimposed. Then, the rendered image is rotated back in the positive direction according to the original tilt angle. Cross-modal environmental noise enhancement is applied to the cyclic image, and the synthesized image and structured labels are output simultaneously.

2. The method according to claim 1, characterized in that, The adaptive rendering includes: when the width of the text to be filled exceeds the preset maximum width limit, automatically wrapping the text and adjusting the layout according to the preset line height.

3. The method according to claim 1, characterized in that, The multi-field logical consistency constraints include at least one of the following: check bit rules for encoded fields, time logical self-consistency for date fields, and spatial semantic alignment between place names.

4. The method according to claim 1, characterized in that, The tilt detection uses a straight line detection algorithm; the background restoration uses an image restoration algorithm.

5. The method according to claim 1, characterized in that, The simulated stamp is a geometric primitive stamp generated through parameterization. When superimposed, it uses a multiply or alpha channel blending method for pixel-level fusion and introduces random rotation, transparency, and position offset.

6. The method according to claim 1, characterized in that, The cross-modal environmental noise enhancement includes at least one of the following: geometric distortion, color and light source perturbation, and physical image degradation.

7. The method according to claim 1, characterized in that, The structured tags are in JSON format, and their field values ​​are aligned with the text areas in the synthesized image at the pixel level. When there are multiple versions of real documents, an independent mapping configuration is established for each version, and multiple style samples are generated using the same tag pool.

8. A single-document data synthesis device for training multimodal intelligent agents, characterized in that, The device includes: The template building module is used to obtain at least one real document image, erase the variable text in it to retain a fixed layout, and establish a mapping configuration between the spatial coordinates of the fields to be filled and the layout parameters to obtain a baseline template. The geometric correction and repair module is used to perform tilt detection and reverse rotation correction on the reference template, and to repair the background of the erased area; The semantic generation module is used to generate prompt words based on business logic rules and to generate structured text data in batches that meet the multi-field logical consistency constraints using a large language model. The rendering and stamping module is used to perform adaptive text rendering and overlay a simulated stamp on the template after reverse rotation correction according to the mapping configuration, and then rotate the rendered image back in the positive direction according to the original tilt angle. The post-processing and output module is used to apply cross-modal environmental noise enhancement to the cyclic image and simultaneously output the synthesized image and structured labels.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the single-document data synthesis method for training a multimodal agent as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the single-document data synthesis method for multimodal agent training as described in any one of claims 1-7.

Images