A zero-shot template inference and document structuring identification method and device

Abstract

The present application relates to the technical field of computer vision and natural language processing, and specifically provides a zero sample template inference and document structuring identification method and device, which has the following steps: S1, document image acquisition and preprocessing; S2, layout perception blocking and position coding; S3, prior or example information construction and injection; S4, cross-modal fusion and representation construction; S5, automatic layout analysis and field slot filling; S6, prompt-free extraction instruction generation; S7, field area parallel character recognition; S8, semantic verification and result standardization; S9, output structured field-value data. Compared with the prior art, the present application can avoid the dependence of traditional methods on template making, prompt word writing and large-scale sample training, significantly improve the flexibility, accuracy and online speed of document structuring identification, have good intelligentization and rapid adaptation ability, and be suitable for diversified document identification scenes.

Description

Technical Field

[0001] This invention relates to the field of computer vision and natural language processing, specifically providing a method and apparatus for zero-shot template inference and document structure recognition. Background Technology

[0002] With the accelerating digitalization of various online and offline businesses, it has become commonplace for users to collect images of various documents such as invoices, certificates, and forms via mobile phones and scanning devices. However, most existing document recognition and structured extraction technologies rely on pre-designed templates or multimodal models based on a large number of prompts for information extraction. This still requires manual template creation or prompt writing, making it difficult to adapt to diverse new document formats in a timely manner. Furthermore, the accuracy of structured extraction in zero-sample or few-sample scenarios remains unsatisfactory, failing to meet the needs of rapid deployment and low cost. To address these issues, academia and industry have proposed various multimodal fusion, visual language models, and few-sample fine-tuning solutions. However, they generally lack an end-to-end solution that can automatically infer document formats and output structured field information without prompts in the case of very few samples, making it difficult to simultaneously balance flexibility, accuracy, and deployment efficiency.

[0003] The existing publicly disclosed or authorized relevant invention patents mainly include:

[0004] Application publication number: CN109919014B. This paper proposes a method to improve the adaptability and efficiency of OCR recognition by inputting the image to be recognized into a general OCR template for recognition, obtaining text information and its location information, and then synthesizing them into structured recognition data. Limitations: Although a general template and structured output are proposed, it still relies on a pre-trained recognition model, requiring a large number of samples for model training, and lacks zero-shot inference capability for new document layouts.

[0005] Application publication number: CN113191348B. The method proposes first creating a template based on a fixed-format sample, then accurately locating the position of each character in the template using a character positioning algorithm, followed by using an OCR algorithm to recognize the image to be recognized and match it with the template, finally outputting a structured result. Limitations: The method relies on a pre-made template and character coordinate matching, cannot automatically adapt to situations with missing samples or new document layouts, and the process is relatively complex and lacks flexibility.

[0006] Application publication number: CN110008944B. The proposal suggests collecting sample documents with different layouts, establishing a recognition template database, and calling the corresponding template for OCR recognition based on the document type to improve adaptability to various layout formats. Limitations: Although it can adapt to multiple layouts, it still requires pre-collection and creation of a template library, and new document layouts require re-annotation of samples. It lacks an automatic inference mechanism for zero or few samples.

[0007] A comprehensive analysis of existing technologies reveals that, although the aforementioned patents have made some progress in OCR recognition and structured extraction, they generally suffer from the following problems:

[0008] First, it relies heavily on pre-made templates or a large number of samples, making it difficult to adapt to new document formats in a timely manner;

[0009] Secondly, it often requires manual creation of prompts or rules, lacking automation and intelligence.

[0010] Third, it lacks the ability to automatically complete template inference and structured extraction under zero-sample or few-sample conditions, making it difficult to meet the needs of rapid deployment and flexible deployment. Therefore, there is an urgent need to propose a method and tool that can automatically complete template inference and achieve structured recognition without prompts in zero-sample or few-sample scenarios through a multimodal large model, so as to improve the intelligence level and application scope of document processing. Summary of the Invention

[0011] This invention addresses the shortcomings of the prior art by providing a highly practical zero-sample template inference and document structure recognition method.

[0012] A further technical objective of this invention is to provide a reasonably designed, safe, and applicable zero-sample template inference and document structure recognition device.

[0013] The technical solution adopted by this invention to solve its technical problem is:

[0014] A zero-shot template inference and document structure recognition method comprises the following steps:

[0015] S1. Document image acquisition and preprocessing;

[0016] S2, Page Layout Aware Segmentation and Position Encoding;

[0017] S3, Construction and injection of prior or example information;

[0018] S4, Cross-modal fusion and representation construction;

[0019] S5, Automatic Layout Parsing and Field Slot Filling;

[0020] S6. Generation of prompt-free word extraction instructions;

[0021] S7, Parallel Text Recognition in Field Regions;

[0022] S8. Semantic validation and result standardization;

[0023] S9. Output structured field-value data.

[0024] Furthermore, in step S1, the document image to be processed is acquired through a mobile camera, document scanner or scanning device, and adaptive scaling is performed on the input original image to make the long side of the image uniform to a preset size, and mean and variance normalization is performed to adapt to the subsequent model input requirements.

[0025] In step S2, the preprocessed image is divided into several fixed-size image patches to form a visual token;

[0026] Absolute coordinates and row / column relative coordinates are overlaid on each image block to form a layout-aware feature containing layout space information, which is used by the subsequent model to understand the field spatial layout.

[0027] Furthermore, step S3 includes:

[0028] S3-1. If the business side predefines the field name, then encode it as a text prior token and inject it into the sequence;

[0029] S3-2. If a small number of sample documents are provided, extract their layout features, generate sample embedding tokens through a visual encoder, and inject them into the sequence to guide the model to learn the target layout.

[0030] S3-3. If there are no priors or examples, this step can be skipped, and the system will rely on the general template prior within the model to execute subsequent processes.

[0031] Furthermore, step S4 includes:

[0032] S4-1. Input the visual token generated in step S2, the text prior token injected in step S3, and the example token into the multi-layer cross-attention mechanism.

[0033] S4-2. Deep fusion of visual, linguistic, and example information is achieved through multi-head cross-attention, feedforward networks, and residual normalization structures.

[0034] S4-3, the output fused multimodal shared representation serves as the common input feature for the downstream field slot filling decoder SlotFilling and the instruction generation module.

[0035] Furthermore, in step S5, based on the multimodal shared representation of step S4, the JSON description of the document format is directly output through the Slot Filling decoder, including the field name, its position coordinates in the image, and the hierarchy or association information between the fields, further including:

[0036] S5-1, Slot Query Initialization: Set a preset number of learnable slot query vectors on the decoding end, with each query corresponding to a potential field slot;

[0037] S5-2, Multi-head Decoding and Matching: Input the multimodal shared representation output from step S4 into the Transformer decoder, use the slot query as the Query, perform multi-head cross-attention on the fused features, and output the field name probability distribution, position coordinates and confidence of each slot;

[0038] S5-3, Field Naming and Alignment: When a text prior token exists, field names are aligned using classification or similarity metrics; when no prior token exists, sequence generation or pointer copying strategies are used to output field names.

[0039] S5-4, Coordinate Regression and Normalization: Regress the bounding box of each slot field and normalize it to [0,1] based on the image size to improve the versatility across resolution scenarios;

[0040] S5-5, Set Matching and Filtering: During the training phase, Hungarian matching is used to achieve a one-to-one correspondence between predicted slots and real fields; during the inference phase, non-maximum suppression is performed on the prediction results to remove overlapping redundancy, and low-confidence fields are marked as "to be reviewed" based on the confidence threshold τ.

[0041] S5-6, JSON Format Output: Organize the filtered field names, coordinates, hierarchical relationships, and confidence scores into a unified JSON format for subsequent extraction instruction generation and OCR execution module calls.

[0042] Furthermore, in step S6, without the need for manual prompts or regular expressions, the language decoding submodule LLM-Head of the multimodal model automatically generates extraction instruction chains for each field based on the formatted JSON generated in step S5, specifically including:

[0043] S6-1, JSON Parsing and Type Inference: Parse the field names, bounding box coordinates, hierarchical relationships, and confidence values ​​in the JSON format one by one; and call the built-in lightweight classifier or language model to output the corresponding field type labels based on the field context information. The labels include at least name, date, amount, ID number, and address type, thereby establishing a <field name - type> correspondence table under zero-sample document conditions.

[0044] S6-2, Operation Primitive Combination and Extraction Instruction Generation: Based on field type and location information, operations are automatically retrieved or combined from a preset operation primitive library to form a field-level extraction instruction sequence; wherein the operation primitive library includes at least: region cropping, OCR recognition (ocr), regularization (regex), data normalization (normalize), rule or checkpoint validation (validate), confidence (confidence_check), fallback processing, and human review (human_review).

[0045] S6-3, Conditional and Exceptional Branch Injection: When the field confidence is lower than the threshold τ or the field type inference is unclear, an exception handling branch is automatically inserted into the instruction chain. The threshold τ ranges from 0.5 to 0.9. The exception handling logic includes at least: re-cropping the region, switching to the backup OCR engine, dynamically adjusting the threshold τ, and triggering the manual review channel to ensure the robustness of recognition in low confidence scenarios.

[0046] S6-4, Instruction Chain Verification and Compression: Perform syntax rule verification and dependency topology sorting on the generated instruction chain to ensure the execution order is legal; at the same time, detect and merge redundant operations, remove duplicate OCR calls, and add parallel markers to OCR steps that can be executed independently to support the simultaneous recognition of multiple fields;

[0047] S6-5, Instruction Chain Output: Output the verified and compressed instruction chain in JSON or Domain-Specific Language (DSL) format. The output includes field identifiers, operation sequences, conditional branching rules, parallel group flags, instruction chain version number, and hash check value.

[0048] S6-6 Dynamic Maintenance and Adaptive Updates: When new fields are added due to business requirements or document format changes, differential additions or replacements are performed on the existing instruction chain based on the updated JSON format. At the same time, statistical analysis is performed on the accumulated failure samples during the runtime to dynamically adjust the threshold τ or the combination of update operation primitives, so as to realize online adaptive updates of the instruction chain and ensure that new document types can still be quickly adapted without increasing manual configuration.

[0049] Furthermore, step S7 includes:

[0050] S7-1. Based on the position information of each field in the JSON layout, crop the image to obtain the local image block where the field is located;

[0051] S7-2. For each field region, a lightweight text recognition network is used for parallel recognition to extract the original text information;

[0052] S7-3. Perform OCR processing on multiple field regions simultaneously.

[0053] Furthermore, step S8 includes:

[0054] S8-1. Perform semantic analysis and type determination on the recognition results of each field, and automatically identify the type to which the field belongs through the language model;

[0055] S8-2. Standardize the units or formats of field values ​​to meet the data consistency requirements of downstream business systems;

[0056] S8-3. If the confidence level of the recognition result is low, or there is an abnormal format, you can go back to step S7 to re-execute OCR, region cropping or field parsing. It supports a multi-round error correction mechanism and improves the overall recognition accuracy through Voting.

[0057] S8-4. When detecting abnormal patterns, a manual review process can be automatically triggered;

[0058] S8-5. Integrate all validated fields and identification values ​​to generate structured JSON or CSV for subsequent information verification, data entry, or automated business processes.

[0059] Furthermore, step S9 includes:

[0060] S9-1. Summarize all field names and identification values ​​that have passed the validation to form a structured field-value linked list result;

[0061] S9-2. The results can be output in JSON or CSV format for subsequent information verification, data entry, or automated business processes.

[0062] A zero-sample template inference and document structure recognition device includes: at least one memory and at least one processor;

[0063] The at least one memory is used to store a machine-readable program;

[0064] The at least one processor is used to call the machine-readable program to execute a zero-sample template inference and document structure recognition method.

[0065] Compared with existing technologies, the zero-sample template inference and document structure recognition method and apparatus of the present invention have the following outstanding advantages:

[0066] (1) The present invention realizes end-to-end processing of document images from input to layout inference and structured recognition. There is no need to pre-make templates or manually write prompts. It can automatically adapt to the new document layout under zero or few sample conditions, significantly shorten the new document online cycle, and improve the system flexibility and intelligence level.

[0067] (2) By using a multimodal vision-language model to deeply understand the semantic and layout information of images, and combining contextual information to complete the automatic field location and extraction, manual intervention is reduced and the accuracy and robustness of document structure recognition are improved.

[0068] (3) The structured instruction generation strategy that does not require prompt words is adopted, which avoids the cumbersome process of repeatedly designing prompt words or rules for different document types in the traditional method, reduces system maintenance costs, and enhances the system's ability to adapt to diverse scenarios.

[0069] (4) It supports the rapid processing of various invoices, certificates, forms and complex documents, and can output structured data results containing field names and corresponding values, providing a high-quality data foundation for subsequent information verification, data entry and automated business processes, and meeting the diversified application needs of government affairs, finance, enterprise document processing and other fields. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a flowchart illustrating a zero-shot template inference and document structure recognition method.

[0072] Figure 2 This is a schematic diagram of a zero-sample template inference and document structure recognition method;

[0073] Figure 3 This is an architecture diagram of a multimodal model in a zero-shot template inference and document structure recognition method;

[0074] Figure 4 This is a flowchart of the instruction chain output for JSON format generation and extraction in a zero-sample template inference and document structure recognition method. Detailed Implementation

[0075] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0076] The following is a preferred embodiment:

[0077] like Figure 1-4 As shown, the zero-sample template inference and document structure recognition method in this embodiment has the following steps:

[0078] S1. Document image acquisition and preprocessing;

[0079] include:

[0080] S1-1. Acquire images of documents to be processed through a mobile camera, document scanner, or scanning device, including but not limited to various types of cards and certificates such as tickets, certificates, and forms.

[0081] S1-2. Perform adaptive scaling on the input original image to unify the long side of the image to the preset size, and perform mean and variance normalization to adapt to the input requirements of subsequent models.

[0082] S2, Page Layout Aware Segmentation and Position Encoding;

[0083] include:

[0084] S2-1. Divide the preprocessed image into several fixed-size image patches to form visual tokens;

[0085] S2-2. Overlay absolute coordinates and row / column relative coordinates onto each image block to form a layout-aware feature containing layout space information, which is used by the subsequent model to understand the field space layout.

[0086] S3. Prior or example information construction and injection (optional);

[0087] include:

[0088] S3-1. If the business side predefines the field name, then encode it as a text prior token and inject it into the sequence;

[0089] S3-2. If a small number of sample documents are provided, extract their layout features, generate a sample embedding token (Demo Token) through a visual encoder, and inject it into the sequence to guide the model to learn the target layout.

[0090] S3-3. If there are no priors or examples, this step can be skipped, and the system will rely on the general template prior within the model to execute subsequent processes.

[0091] S4, Cross-modal fusion and representation construction;

[0092] include:

[0093] S4-1. Input the visual token generated in step S2, the text prior token and the example token injected in step S3 into the multi-layer cross-attention mechanism.

[0094] S4-2. Deep fusion of visual, linguistic, and example information is achieved through multi-head cross-attention, feedforward networks, and residual normalization structures.

[0095] S4-3, Output the fused multimodal shared representation as a common input feature for the downstream field slot filling decoder and instruction generation module.

[0096] S5, Automatic Layout Parsing and Field Slot Filling;

[0097] Based on the multimodal shared representation in step S4, the JSON description of the document format is directly output through the field slot filling decoder. This includes the field name, the position coordinates in the image, and the hierarchical or relational information between the fields. Data example: {"Field": "Name", "Coordinates": [x1,y1,x2,y2], "Confidence": 0.93}.

[0098] include:

[0099] S5-1, Slot Query Initialization: Set a preset number of learnable slot query vectors on the decoding end, with each query corresponding to a potential field slot;

[0100] S5-2, Multi-head Decoding and Matching: Input the multimodal shared representation output from step S4 into the Transformer decoder, use the slot query as the Query, perform multi-head cross-attention on the fused features, and output the field name probability distribution, position coordinates and confidence of each slot;

[0101] S5-3, Field Naming and Alignment: When a text prior token exists, field names are aligned using classification or similarity metrics; when no prior token exists, sequence generation or pointer copying strategies are used to output field names.

[0102] S5-4, Coordinate Regression and Normalization: For each slot, regress the bounding box of the field (four-point coordinates or center point + width and height), and normalize it [0,1] based on the image size to improve the versatility in cross-resolution scenarios;

[0103] S5-5, Set Matching and Filtering: During the training phase, Hungarian matching is used to achieve a one-to-one correspondence between predicted slots and real fields; during the inference phase, non-maximum suppression (NMS / Soft-NMS) is performed on the prediction results to remove overlapping redundancy, and low-confidence fields are marked as "to be reviewed" based on the confidence threshold τ.

[0104] S5-5, JSON Format Output: Organize the filtered field names, coordinates, hierarchical relationships, and confidence scores into a unified JSON format for subsequent extraction instruction generation and OCR execution module calls.

[0105] S5-6, JSON Format Output: Organize the filtered field names, coordinates, hierarchical relationships, and confidence scores into a unified JSON format for subsequent extraction instruction generation and OCR execution module calls.

[0106] S6. Generation of prompt-free word extraction instructions;

[0107] Without requiring manual prompts or regular expressions, the language decoding submodule (LLM-Head) built into the multimodal model automatically generates an extraction instruction chain for each field based on the formatted JSON output in step S5. For example, input formatted JSON:

[0108] "fields": [

[0109] {"name": "Name", "bbox": [0.12, 0.20, 0.30, 0.25], "conf": 0.93,

[0110] "level": 1},

[0111] {"name": "ID number", "bbox": [0.15, 0.35, 0.55, 0.40], "conf": 0.78,"level": 1} ]

[0113] }

[0114] The system automatically executes the following sub-steps:

[0115] S6-1, Layout JSON Parsing and Type Inference: The layout JSON is parsed one by one for field names, bounding box coordinates (using 0-1 normalized coordinates relative to the original image size), hierarchical relationships, and confidence values; and the built-in lightweight classifier or language model is called to output the corresponding field type labels according to the field context information. The labels include at least types such as name, date, amount, ID number, and address, thereby establishing a <field name-type> correspondence table under zero-sample document conditions, avoiding manual predefined template configuration;

[0116] Parse the field name and confidence value, call a lightweight classifier to infer the type based on context information, such as "Name" as text type and "ID Number" as document number type, avoiding manual template configuration.

[0117] S6-2, Operation Primitive Combination and Extraction Instruction Generation: Based on field type and location information, operations are automatically retrieved or combined from a preset operation primitive library to form a field-level extraction instruction sequence; wherein the operation primitive library includes at least: region cropping, OCR recognition, regularization, data normalization, rule or checkpoint validation, confidence check, fallback, and human review triggering; preferably, the instruction sequence is arranged in the order of "cropping → OCR → regularization / normalization → validation" to improve cross-scene adaptability and execution stability;

[0118] Based on the field type, combine operation sequences from the primitive library. For example, generate [crop → ocr → normalize → validate] for "name" and [crop → ocr → regex_normalize → validate] for "ID number".

[0119] S6-3, Conditional and Exceptional Branch Injection: When the field confidence level is lower than the threshold τ or the field type inference is unclear, an exception handling branch is automatically inserted into the instruction chain. The preferred range of the threshold τ is 0.5–0.9. The exception handling logic includes at least: re-cropping the region (adding 2%–10% margin around the original bounding box), switching to the backup OCR engine, dynamically adjusting the threshold τ, and triggering the manual review channel to ensure recognition robustness in low-confidence scenarios.

[0120] When the confidence level of a field falls below the threshold τ=0.8, an exception handling branch is automatically injected. If the confidence level of "ID number" is 0.78, the system will expand the bounding box by 5%, switch to the backup OCR engine, and trigger manual review if the conditions are still not met. The preferred range for τ is 0.5–0.9, and the boundary expansion ratio is 2%–10%.

[0121] S6-4, Instruction Chain Verification and Compression: Perform syntax rule verification and dependency topology sorting on the generated instruction chain to ensure the execution order is legal; at the same time, detect and merge redundant operations, remove duplicate OCR calls, add parallel markers to independently executable OCR steps to support simultaneous recognition of multiple fields, reduce redundant calculations, and reduce overall inference latency by at least 20%;

[0122] Redundant OCR steps were removed, and parallel flags were added to the OCR operations for "Name" and "ID Number". In testing, the overall inference latency was reduced by approximately 20%.

[0123] S6-5, Instruction Chain Output: The verified and compressed instruction chain is output in JSON or Domain-Specific Language (DSL) format. The output includes field identifiers, operation sequences, conditional branching rules, parallel group markers, instruction chain version number and hash check value, thereby ensuring consistency, traceability and tamper-proofness when subsequent execution modules are called.

[0124] The optimized instruction chain will be output in JSON / DSL format, for example:

[0125] {

[0126] "field": "ID number",

[0127] "chain": [

[0128] {"op": "crop", "args": {"bbox": [0.15, 0.35, 0.55, 0.40], "margin": "5%"}},

[0129] {"op": "ocr", "args": {"engine": "ocr_v2"}},

[0130] {"op": "regex_normalize", "args": {"pattern": "ID18"}},

[0131] {"op": "validate", "args": {"rule": "luhn_cn_id"}},

[0132] {"if": "conf<0.8 || !validate",

[0133] "then": [

[0134] {"op": "fallback", "args": {"strategy": "reOCR_alt_engine"}},

[0135] {"op": "human_review_trigger", "args": {"reason": "low_conf_or_format"}}

[0136] ]}

[0137] ],

[0138] "version": "v1.0",

[0139] "hash": "b7d3e19f..."

[0140] }

[0141] S6-6 Dynamic Maintenance and Adaptive Updates: When new fields are added due to business requirements or document layout changes, the existing instruction chain is differentially supplemented or replaced based on the updated JSON layout. At the same time, statistical analysis is performed on the accumulated failure samples during the runtime to dynamically adjust the threshold τ or the combination of update operation primitives, so as to realize online adaptive updates of the instruction chain. This ensures that new document types can still be quickly adapted without increasing manual configuration, thereby significantly reducing maintenance costs.

[0142] When a new field "Expiration Date" is added, the system detects this field in the new JSON format, automatically generates an instruction chain [crop → ocr → validate(date_format)], and adds it to the existing chain. Simultaneously, based on failure sample statistics, the threshold τ is dynamically adjusted from 0.8 to 0.75 to improve the overall recall rate.

[0143] S7, Parallel Text Recognition in Field Regions;

[0144] include:

[0145] S7-1. Based on the position information of each field in the layout JSON, perform region cropping on the image to obtain the local image block where the field is located;

[0146] like Figure 4 As shown, the process of generating and extracting the instruction chain from the formatted JSON is as follows:

[0147] (1) Multimodal representation after fusion:

[0148] The shared representation obtained by cross-modal interaction of input features such as visual tokens, text prior tokens, and example tokens is used as the basic input for subsequent parsing and instruction generation.

[0149] (2) Automatic layout parsing and field slot filling unit:

[0150] Slot query initialization: Set a preset number of learnable slot query vectors for the decoding end;

[0151] Multi-head decoding and matching: Slot matching with multimodal features is achieved through a multi-head cross-attention mechanism;

[0152] Field naming and alignment: Output candidate field names and align them by combining prior information or sequence generation strategies;

[0153] Coordinate regression and normalization: Predict bounding boxes for each field slot and normalize their representation;

[0154] Set matching and filtering: The final set of fields is obtained by matching and filtering based on confidence.

[0155] After the above steps, the output is a JSON format, which includes field names, coordinate positions, hierarchical relationships, and confidence levels.

[0156] (3) Prompt-free word extraction instruction generation unit:

[0157] Formatted JSON parsing and type inference: Parsing field attributes and inferring field types;

[0158] Operation primitive combination and extraction instruction generation: Retrieve or combine operation sequences from the primitive library based on field type;

[0159] Conditional and exception branch injection: Automatically inject exception handling and fallback logic when the field confidence is insufficient or the type is ambiguous;

[0160] Instruction chain verification and compression: Perform syntax checking, dependency sorting, and redundancy merging on the generated instruction chain.

[0161] After processing by this unit, an executable extraction instruction chain is obtained.

[0162] (4) Dynamic maintenance and adaptive update unit:

[0163] When business requirements change or new fields are added, the existing instruction chain is differentially supplemented or replaced based on the updated JSON format, and the threshold or primitive combination is adjusted in combination with the statistics of runtime failure samples, thereby realizing online adaptive updates of the instruction chain.

[0164] (5) Instruction chain output:

[0165] Finally, the verified and updated extraction instruction chain will be output in JSON or Domain-Specific Language (DSL) format for subsequent execution and result verification modules to call.

[0166] S7-2. For each field region, a lightweight text recognition network (such as an OCR module based on the ViT-Tiny backbone) is used for parallel recognition to extract the original text information;

[0167] The S7-3 supports parallel OCR processing of multiple field regions, significantly reducing overall inference time compared to traditional serial recognition, and meeting the needs of high concurrency and real-time business.

[0168] S8. Semantic validation and result standardization;

[0169] include:

[0170] S8-1. Perform semantic analysis and type determination on the recognition results of each field, and automatically identify whether the field belongs to text, number, date, amount or other types through language model;

[0171] S8-2. Standardize the units or formats of field values, including but not limited to currency symbol unification, date format conversion, and numerical normalization, to meet the data consistency requirements of downstream business systems.

[0172] S8-3. If the confidence level of the recognition result is low, or there is an abnormal format, you can go back to step S7 to re-execute OCR, region cropping or field parsing. It supports a multi-round error correction mechanism and further improves the overall recognition accuracy through Voting and other methods.

[0173] S8-4: When detecting abnormal patterns such as question marks, garbled characters, and illegal characters, the manual review process can be automatically triggered to ensure the quality of the final output data.

[0174] S8-5. Integrate all validated fields and their identification values ​​to generate structured JSON, CSV or other standard format results for subsequent information verification, data entry or automated business processes.

[0175] S9. Output structured field-value data;

[0176] include:

[0177] S9-1. Summarize all field names and identification values ​​that have passed the validation to form a structured field-value linked list result;

[0178] S9-2. The results can be output in formats such as JSON and CSV for subsequent information verification, data entry, or automated business processes.

[0179] Based on the above method, a zero-sample template inference and document structure recognition device in this embodiment includes: at least one memory and at least one processor;

[0180] The at least one memory is used to store a machine-readable program;

[0181] The at least one processor is used to call the machine-readable program to execute a zero-sample template inference and document structure recognition method.

[0182] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0183] Memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A zero-shot template inference and document structure recognition method, characterized in that, It has the following steps: S1. Document image acquisition and preprocessing; The document image to be processed is acquired through a mobile camera, document scanner or scanning device. Adaptive scaling is performed on the original input image to make the long side of the image uniform to a preset size, and mean and variance normalization is performed to adapt to the input requirements of the subsequent model. S2, Page Layout Aware Segmentation and Position Encoding; The preprocessed image is divided into several fixed-size image patches to form visual tokens; Absolute coordinates and row-column relative coordinates are overlaid on each image block to form a layout-aware feature containing layout space information, which is used by the subsequent model to understand the field spatial layout. S3, Construction and injection of prior or example information; include: S3-1. If the business side predefines the field name, then encode it as a text prior token and inject it into the sequence; S3-2. If a small number of sample documents are provided, extract their layout features, generate sample embedding tokens through a visual encoder, and inject them into the sequence to guide the model to learn the target layout. S3-3. When there are no priors or examples, the system relies on the general layout prior within the model to execute subsequent processes. S4, Cross-modal fusion and representation construction; include: S4-1. Input the visual token generated in step S2, the text prior token injected in step S3, and the example token into the multi-layer cross-attention mechanism. S4-2. Deep fusion of visual, linguistic, and example information is achieved through multi-head cross-attention, feedforward networks, and residual normalization structures. S4-3, The multimodal shared representation after output fusion serves as the common input feature for the downstream field slot filling decoder and instruction generation module; S5, Automatic Layout Parsing and Field Slot Filling; Based on the multimodal shared representation in step S4, the JSON description of the document format is directly output through the Slot Filling decoder, including field names, their position coordinates in the image, and hierarchical or relational information between fields, further including: S5-1, Slot Query Initialization: Set a preset number of learnable slot query vectors on the decoding end, with each query corresponding to a potential field slot; S5-2, Multi-head Decoding and Matching: Input the multimodal shared representation output from step S4 into the Transformer decoder, use the slot query as the Query, perform multi-head cross-attention on the fused features, and output the field name probability distribution, position coordinates and confidence of each slot; S5-3, Field Naming and Alignment: When a text prior token exists, field names are aligned using classification or similarity metrics; when no prior token exists, sequence generation or pointer copying strategies are used to output field names. S5-4, Coordinate Regression and Normalization: Regress the bounding box of each slot field and normalize it to [0,1] based on the image size to improve the versatility across resolution scenarios; S5-5, Set Matching and Filtering: During the training phase, Hungarian matching is used to achieve a one-to-one correspondence between predicted slots and real fields; during the inference phase, non-maximum suppression is performed on the prediction results to remove overlapping redundancy, and low-confidence fields are marked as "to be reviewed" based on the confidence threshold τ. S5-6, JSON Layout Output: Organize the filtered field names, coordinates, hierarchical relationships, and confidence scores into a unified JSON layout for subsequent extraction instruction generation and OCR execution module calls; S6. Generation of prompt-free word extraction instructions; Without the need for manual prompts or regular expressions, the language decoding submodule LLM-Head of the multimodal model automatically generates extraction instruction chains for each field based on the formatted JSON generated in step S5. Specifically, these include: S6-1, JSON Parsing and Type Inference: Parse the field names, bounding box coordinates, hierarchical relationships, and confidence values ​​in the JSON layout one by one; and call the built-in lightweight classifier or language model to output the corresponding field type labels based on the field context information. The labels include at least name, date, amount, ID number, and address type, thereby establishing a field name-type correspondence table under zero-sample document conditions. S6-2, Operation Primitive Combination and Extraction Instruction Generation: Based on field type and location information, operations are automatically retrieved or combined from a preset operation primitive library to form a field-level extraction instruction sequence; wherein the operation primitive library includes at least: region cropping, OCR recognition (ocr), regularization (regex), data normalization (normalize), rule or checkpoint validation (validate), confidence (confidence_check), fallback processing, and human review (human_review). S6-3, Conditional and Exceptional Branch Injection: When the field confidence is lower than the threshold τ or the field type inference is unclear, an exception handling branch is automatically inserted into the instruction chain. The threshold τ ranges from 0.5 to 0.

9. The exception handling logic includes at least: re-cropping the region, switching to the backup OCR engine, dynamically adjusting the threshold τ, and triggering the manual review channel to ensure the robustness of recognition in low confidence scenarios. S6-4, Instruction Chain Verification and Compression: Perform syntax rule verification and dependency topology sorting on the generated instruction chain to ensure the execution order is legal; at the same time, detect and merge redundant operations, remove duplicate OCR calls, and add parallel markers to OCR steps that can be executed independently to support the simultaneous recognition of multiple fields; S6-5, Instruction Chain Output: Output the verified and compressed instruction chain in JSON or Domain-Specific Language (DSL) format. The output includes field identifiers, operation sequences, conditional branching rules, parallel group flags, instruction chain version number, and hash check value. S6-6 Dynamic Maintenance and Adaptive Updates: When new fields are added for business requirements or the document format changes, the existing instruction chain is differentially supplemented or replaced based on the updated JSON format. At the same time, statistical analysis is performed on the accumulated failure samples during the runtime to dynamically adjust the threshold τ or the combination of update operation primitives, so as to realize the online adaptive update of the instruction chain and ensure that new document types can still be quickly adapted without increasing manual configuration. S7, Parallel Text Recognition in Field Regions; S8. Semantic validation and result standardization; S9. Output structured field-value data.

2. The zero-sample template inference and document structure recognition method according to claim 1, characterized in that, Step S7 includes: S7-1. Based on the position information of each field in the JSON layout, crop the image to obtain the local image block where the field is located; S7-2. For each field region, a lightweight text recognition network is used for parallel recognition to extract the original text information; S7-3. Perform OCR processing on multiple field regions simultaneously.

3. The zero-sample template inference and document structure recognition method according to claim 2, characterized in that, Step S8 includes: S8-1. Perform semantic analysis and type determination on the recognition results of each field, and automatically identify the type to which the field belongs through the language model; S8-2. Standardize the units or formats of field values ​​to meet the data consistency requirements of downstream business systems; S8-3. If the confidence level of the recognition result is low, or there is an abnormal format, you can go back to step S7 to re-execute OCR, region cropping or field parsing. It supports a multi-round error correction mechanism and improves the overall recognition accuracy through Voting. S8-4. When detecting abnormal patterns, a manual review process can be automatically triggered; S8-5. Integrate all validated fields and identification values ​​to generate structured JSON or CSV for subsequent information verification, data entry, or automated business processes.

4. The zero-sample template inference and document structure recognition method according to claim 3, characterized in that, Step S9 includes: S9-1. Summarize all field names and identification values ​​that have passed the validation to form a structured field-value linked list result; S9-2. The results can be output in JSON or CSV format for subsequent information verification, data entry, or automated business processes.

5. A zero-sample template inference and document structure recognition device, characterized in that, include: At least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to invoke the machine-readable program to perform the method according to any one of claims 1 to 4.

Images