An intelligent document understanding and automatic form filling system based on a multi-modal large model

Abstract

The application discloses an intelligent document understanding and automatic form filling system based on a multimodal large model, comprising a multimodal document coding module, a document-form semantic alignment module, an information fusion module, a form constraint checking module and an intelligent form filling module. The application fuses multimodal information such as document images, text content, table structures, form fields and knowledge bases, constructs a multimodal document encoder, a document-form semantic alignment module, a cross-document information fusion module, a form constraint understanding module and an intelligent form filling module, realizes extraction of structured information from a source document, understanding of field meanings and filling rules of a target form, automatic matching and filling of information, supports cross-document information completion, format conversion and consistency verification.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and intelligent document processing, specifically to an intelligent document understanding and automatic form filling system based on a multimodal large model, applicable to various application scenarios requiring document processing and form filling, such as government services, financial services, healthcare, education management, and enterprise office work. Background Technology

[0002] With the deepening of digital transformation, various organizations need to process a large number of documents and fill out various forms in their daily operations. Traditional document information extraction and form filling mainly rely on manual operation, which has problems such as low efficiency, high error rate, and high cost. Although there are some automated solutions based on template matching or rule engines, these methods still have significant shortcomings when dealing with complex documents, cross-document information association, and understanding of form constraints.

[0003] Existing technologies mainly face the following problems:

[0004] 1. Document information extraction relies on fixed templates: Traditional methods typically require designing specific templates for each document type. When the document format changes or a new document type appears, the template needs to be redesigned, lacking universality and flexibility. For unstructured documents (such as handwritten documents, scanned documents, and multi-page documents), the accuracy of template matching drops significantly.

[0005] 2. Low table recognition accuracy: Existing table recognition methods mainly focus on the physical structure (row and column division) and lack sufficient understanding of the logical structure of tables, making it difficult to handle complex tables (such as nested tables, irregular tables, cross-page tables, and multi-level headers). For scenarios such as merged cells, the correspondence between headers and data rows, and cross-document table associations, the recognition accuracy is often below 70%.

[0006] 3. Difficulty in cross-document information association: In actual business, the same information may be scattered in multiple documents (such as ID cards, household registration books, and academic certificates, all of which contain information such as name and date of birth). Existing methods lack cross-document information fusion mechanisms and cannot automatically identify information consistency, detect conflicts, or complete information.

[0007] 4. Insufficient semantic understanding of form fields: Existing autofill systems typically use keyword matching, which fails to understand the semantic meaning of form fields. When the field descriptions in the source document do not match the form fields (e.g., "ID number" vs. "ID Number"), the system cannot match them correctly, leading to form filling failures or errors.

[0008] 5. Lack of understanding and validation of form constraints: Forms typically contain multiple constraints (required fields, format requirements, numerical ranges, field dependencies). Existing systems lack the ability to understand and validate these constraints, which can easily lead to formatting errors, logical errors, or omissions of required fields.

[0009] 6. Lack of interpretability and traceability: When errors occur in autofill, the existing system cannot provide information source labeling and error tracing path, making it difficult for users to quickly locate and correct the problem.

[0010] Therefore, there is an urgent need for an intelligent document understanding and automatic form filling system that can fully utilize multimodal information, deeply understand document semantics, accurately identify table structures, achieve cross-document information fusion, understand form constraints, and possess good interpretability. This invention solves the above-mentioned technical challenges by constructing a multimodal document understanding framework, designing a document-form semantic alignment mechanism, establishing a cross-document information fusion module, and realizing form constraint understanding and intelligent verification, thus providing an efficient and reliable technical solution for intelligent document processing and automatic form filling. Summary of the Invention

[0011] To address the aforementioned issues, this invention provides an intelligent document understanding and automatic form filling system based on a multimodal large model. By integrating multimodal information such as document images, text content, table structure, form fields, and knowledge bases, it constructs a multimodal document encoder, a document-form semantic alignment module, a cross-document information fusion module, a form constraint understanding module, and an intelligent form filling module. This enables the extraction of structured information from source documents, understanding the meaning of fields and filling rules in the target form, automatically matching information and filling the form, and supports cross-document information completion, format conversion, and consistency verification.

[0012] The specific plan is as follows:

[0013] A smart document understanding and automatic form filling system based on a multimodal large model includes:

[0014] The multimodal document encoding module extracts and encodes features for different types of document information, achieving a deep understanding of documents within a unified multimodal framework. This module forms the basis for subsequent semantic alignment, information fusion, and intelligent form filling. By extracting structured information from documents through multimodal encoding, it provides high-quality feature representations for the entire system.

[0015] The document-form semantic alignment module, based on the document features and field information extracted by the aforementioned multimodal document encoding module, establishes a semantic mapping relationship between source document fields and target form fields to achieve intelligent information matching from source document to target form. It semantically aligns the field information obtained by multimodal encoding with the target form fields, providing accurate field correspondence for subsequent information extraction and form filling.

[0016] The information fusion module, based on the semantic mapping relationship established by the aforementioned document-form semantic alignment module, enables cross-document information completion and consistency verification when the same information appears in multiple source documents. It utilizes the results of semantic alignment to collect candidate values ​​for the same field from multiple source documents, and obtains reliable field values ​​through consistency detection and conflict resolution, providing high-quality data for subsequent constraint verification and form filling.

[0017] The form constraint validation module, based on the field values ​​obtained by the aforementioned information fusion module, identifies the constraints (required, format, range, dependencies) of the form fields and performs integrity checks, format validation, and logical validation before the form is filled out. This module ensures the integrity and correctness of the information filled out, detects potential errors before the form is filled out, and provides the intelligent form filling module with validated, high-quality data.

[0018] The intelligent form-filling module, based on the outputs of the aforementioned document-form semantic alignment, information fusion, and form constraint validation modules, performs automatic form filling and annotates the information source for each filled field, supporting interpretability and error traceability. This module integrates the processing results of the aforementioned modules, performs the actual form-filling operation, including value extraction, format conversion, and information source annotation, and generates the final form-filling result and report.

[0019] Furthermore, the multimodal document encoding module includes:

[0020] S11, Document Visual Layout Coding

[0021] For document images, a Transformer-based visual encoder is used to extract the document's layout and structural information; let the input document image be... ,in and These are the image height and width, respectively.

[0022] First, the image is divided into a non-overlapping sequence of patches, each patch being of size [size missing]. Get the number of patches Linear projection is performed on each patch to obtain the patch embedding:

[0023]

[0024] in For visual feature dimensions; to preserve location information, add location encoding. This yields a position-enhanced patch embedding:

[0025]

[0026] The document layout features are extracted through multi-layer Transformer encoder processing. The Transformer encoder employs a self-attention mechanism and a feedforward neural network, specifically implemented as follows: for the input... First, the correlation between patches is calculated through a multi-head self-attention mechanism. Then, feature transformation is performed through a feedforward network. After layer normalization and residual connections, deep feature representations are obtained by stacking layers one by one.

[0027]

[0028] The Transformer encoder contains multiple Transformer blocks, each of which contains a self-attention layer, a feedforward network layer, layer normalization, and residual connections. It contains layout features for each patch and can identify the position and structural relationships of document elements such as headings, paragraphs, tables, and charts;

[0029] S12, Semantic Encoding of Text Content

[0030] For the text content in the document, semantic features are extracted using a pre-trained large language model; let the text sequence extracted by OCR be... ,in The number of text tokens. Indicates the first A text token;

[0031] Pre-trained large language models (such as BERT, RoBERTa, etc.) are used as text encoders to obtain contextual semantic representations. The text encoder adopts a bidirectional Transformer architecture and captures the contextual relationships between tokens through a self-attention mechanism. Specifically, the token sequence is input into the encoder and processed through multiple Transformer blocks. The representation of each token incorporates its contextual information.

[0032]

[0033] in For text feature dimensions, For the first The context embedding vector of each token contains the semantic information of that token throughout the sequence; to capture document-level semantic information, an attention pooling mechanism is used:

[0034]

[0035] in The weight matrix is ​​a learnable matrix. For attention weight vectors, For the first Attention score of each token This is a document-level semantic feature vector;

[0036] S13, Understanding and Coding Table Structures

[0037] For tables in a document, we not only identify their physical structure (row and column division) but also understand their logical structure (table header-data row relationship, merged cells, cross-page tables); assuming the detected tables include... OK Column, in which For the number of rows, For column number;

[0038] First, extract the visual and text features of each cell; for the first cell... Line number The visual characteristics of the column cells are: Text features are Cell representations are obtained through feature fusion:

[0039]

[0040] in To fuse the weight matrix, For cell feature dimensions, This indicates vector concatenation;

[0041] To model the semantic relationships between cells, a graph neural network is used to construct the table graph structure. Each cell is treated as a graph node, and edge connections are established based on row-column relationships and semantic similarity. Specifically, edge connections are established for cells in the same row or column, and edge connections are also established for semantically similar cells (such as the header and its corresponding data row). The features of all cells are organized into a matrix. , of which The row corresponds to the first Line number Column cell characteristics ;

[0042] The node representation is updated using a graph convolutional network. The specific implementation of graph convolution is as follows: for each node, the features of its neighboring nodes are aggregated, feature transformation is performed using a learnable weight matrix, and a non-linear activation function is applied; The layer's features are updated as follows:

[0043]

[0044] in For the first The node feature matrix of the layer This is the initial feature matrix (composed of all cell features). For the first The learnable weight matrix of the layer, It is a non-linear activation function (such as ReLU). The normalized adjacency matrix; after The final update is obtained after layer graph convolution:

[0045]

[0046] in The updated cell feature matrix can capture the semantic relationships between the header and data rows, or cells in the same row or column.

[0047] Furthermore, the document-form semantic alignment module includes:

[0048] S21, Learning Field Semantic Representation

[0049] For fields in the source document (such as "ID number", "name", "date of birth"), extract their contextual semantic representation; let the detected fields in the document be... ,in Number of fields;

[0050] For each field The text encoding process extracts the label text and corresponding value text, and obtains the semantic representation of the field through a field encoder. The field encoder is implemented by concatenating the label and value texts and inputting them into a pre-trained text encoder (such as BERT) to obtain the contextual representation of the sequence. Then, it extracts the representation of the [CLS] marker or pools the sequence representation to obtain the semantic vector of the field. Finally, a fully connected layer projects the features onto a unified semantic space.

[0051]

[0052] in This is a pre-trained text encoder that outputs a sequence representation matrix. This indicates the concatenation of field labels and values. The representation vector (or pooled representation) labeled [CLS]. The projection weight matrix is... For bias vectors, This is a semantic representation dimension; similarly, for fields in the target form... Extract its semantic representation using the same encoding method:

[0053]

[0054] in Placeholders or sample values ​​for form fields;

[0055] S22, Cross-language and cross-format semantic alignment

[0056] To address the issue of matching fields in different languages ​​and expressions (such as "ID number" and "ID number"), a contrastive learning mechanism is employed to map the same semantic fields in different expressions to a unified semantic space.

[0057] By optimizing the field encoder through contrastive learning loss, fields with similar semantics are grouped closer together in the embedding space, while fields with different semantics are grouped further apart.

[0058]

[0059] in The cosine similarity function is used. The temperature parameter is used; through this loss function, the system can learn the semantic correspondence of fields across languages ​​and formats.

[0060] S23, Field Matching and Mapping

[0061] Based on the learned semantic representation, calculate the matching score between the source document fields and the target form fields:

[0062]

[0063] in Represents document fields With form fields The matching probability; establishing field mapping relationships through threshold judgment and optimal matching algorithms (such as the Hungarian algorithm). .

[0064] Furthermore, the information fusion module includes:

[0065] S31, Multi-document information aggregation

[0066] Let from The information extracted from the source documents is ,in Indicates the first A collection of information from a document;

[0067] For the fields in the target form Collect candidate values ​​from multiple documents ,in The number of candidate values; each candidate value Link its source document and confidence level ;

[0068] S32. Information Consistency Detection and Conflict Resolution

[0069] Calculate the consistency score between candidate values; for two candidate values and Define a consistency metric:

[0070]

[0071] in The text similarity function is implemented as follows: First, the text values ​​are converted into vector representations (e.g., using a pre-trained word embedding model). Then, the cosine similarity or edit distance similarity between the two vectors is calculated. For short texts, edit distance (Levenshtein distance) normalization is used as the similarity; for long texts, cosine similarity based on word embeddings is used. When the consistency score is below a threshold... When this occurs, it is determined to be a conflict;

[0072] For conflicting candidate values, a confidence-based selection strategy is adopted; define fields. The final value is:

[0073]

[0074] in For document The credibility weight (e.g., the ID card has a higher weight than other documents). This represents the confidence level of the candidate value. This strategy prioritizes values ​​from highly reliable documents with high confidence levels.

[0075] Furthermore, the form constraint validation module includes:

[0076] S41. Constraint Identification

[0077] For form fields By analyzing information such as field labels, prompt text, and example values, the constraints can be identified.

[0078]

[0079] in Indicates whether a field is required. Indicate the format requirements (such as mobile phone number, email address, date format). Indicates a range of values. Indicates field dependencies (e.g., "the end date must be later than the start date");

[0080] S42. Information Integrity Check

[0081] Before filling out the form, check that all required fields have had their corresponding values ​​extracted from the source document:

[0082]

[0083] in This is a set of required fields. For fields The system will fill in the missing values; when the completeness is below the threshold, the system will prompt for missing fields and suggest finding them in other documents or entering them manually.

[0084] S43. Format validation and logical verification

[0085] For the extracted values, perform format validation and logical verification; the format validation function is defined as:

[0086]

[0087] For field dependencies, define logical validation functions; for date field dependencies, it is necessary to check the logical relationship between dates: for the dependency "end date > start date":

[0088]

[0089] For the constraint that "the date must be earlier than the current date", the logic check function is defined as:

[0090]

[0091] When a formatting error or logical error is detected, the system generates an error message and suggested corrections.

[0092] Furthermore, the intelligent form-filling module includes:

[0093] S51, Value Extraction and Format Conversion

[0094] For successfully matched field pairs Extract the corresponding value from the source document The function performs format conversion based on the format requirements of the target form fields. Specifically, it converts the format using regular expression matching and string replacement based on the target format type (e.g., date, phone number, ID card). For date formats, it identifies the source format (e.g., YYYYMMDD, YYYY-MM-DD) and converts it to the target format (e.g., YYYY / MM / DD). For phone numbers, it adds separators according to the target format requirements. For other formats, it uses appropriate conversion rules.

[0095]

[0096] The format conversion function is based on For each data type (date, phone number, ID card, etc.), apply the corresponding conversion rules; for example, convert "2024-01-15" to "2024 / 01 / 15" (date format conversion), and format "13800138000" to "138-0013-8000" (phone number format conversion), etc.

[0097] S52, Information Traceability Labeling

[0098] Record the source of information for each field that is filled in:

[0099]

[0100] in For source documents, For the position in the document (e.g., "page 3, line 5"), This is the confidence level; this information is used to generate the form completion report, supporting manual review and error tracing.

[0101] S53. Generation of form completion results

[0102] The final form completion result is generated. ,in It indicates the field status (filled, missing, format error, logical error, etc.); the system outputs a form filling report, including filling details, missing field prompts, error warnings, and suggested corrections.

[0103] Furthermore, based on the form-filling results generated by the aforementioned intelligent form-filling module, when users make manual corrections, the module learns from these corrections through an incremental learning mechanism to identify the filling preferences of specific institutions and forms, thus establishing a personalized form-filling model. This module continuously learns from user correction patterns to optimize the system's form-filling accuracy, achieving personalized adaptation and improving user experience and system performance.

[0104] Furthermore, the incremental learning mechanism includes:

[0105] When a user corrects a value entered by the system, the correction is recorded. By analyzing correction patterns, personalized rules are learned. The specific implementation of personalized rule learning involves: collecting historical correction records for the same organization or form type, statistically analyzing correction patterns (e.g., certain fields habitually use full names, certain fields habitually use abbreviations, etc.), and constructing a rule base; for new fields, based on contextual information (organization type, form type, etc.) and historical correction records, calculating the applicability probability of each rule, and selecting the rule with the highest probability.

[0106]

[0107] in For a set of rules, To correct historical records, For the current context (organization type, form type, etc.); probability The system employs statistical learning methods, such as frequency-based statistics or machine learning-based classifiers. Through learning, the system can adapt to specific scenario-based filling preferences, improving form completion accuracy and user satisfaction.

[0108] Through the above technical solution, this invention constructs a complete multimodal document understanding and automatic form filling system, which can make full use of the multimodal information of documents to achieve deep semantic understanding and intelligent information matching, support cross-document information fusion and form constraint understanding, significantly improve form filling efficiency and accuracy, and provide strong technical support for the digital transformation of government services and enterprise offices.

[0109] The beneficial effects of this invention are as follows:

[0110] 1. By deeply integrating multimodal information, achieving semantic alignment, and supporting cross-document information fusion, this invention improves form filling accuracy by 30-40 percentage points compared to traditional template matching methods. Its advantages are particularly pronounced in scenarios involving complex documents, cross-language matching, and cross-document information completion.

[0111] 2. This invention achieves field matching through semantic understanding, eliminating the need to design specific templates for each document type, thus possessing excellent versatility and flexibility. Even when new document types emerge or document formats change, the system can still accurately understand and extract information, demonstrating strong adaptability.

[0112] 3. This invention innovatively designs a cross-document information fusion mechanism, which can automatically collect, verify, and complete information from multiple source documents, detect information conflicts, and intelligently resolve them. This capability is of great value in scenarios requiring multi-document collaboration and is not available in existing systems.

[0113] 4. By automatically identifying form constraints, performing integrity checks, format validation, and logical verification, this invention can detect potential errors before the form is filled out, generate error prompts and suggestions, significantly reduce the form filling error rate, and improve user experience.

[0114] 5. By using graph neural networks to model the logical structure of tables, this invention can accurately understand complex tables (nested tables, irregular tables, and cross-page tables), identify the correspondence between table headers and data rows, and handle complex scenarios such as merged cells, improving the accuracy of table recognition by 25-30 percentage points.

[0115] 6. Detailed source information is provided for each entered field, generating a completion report that supports manual review and error tracing. This mechanism enhances the system's credibility and user acceptance, facilitating problem identification and system optimization.

[0116] 7. Through an incremental learning mechanism, the system can learn from user corrections, adapt to filling preferences in specific scenarios, and continuously improve form-filling accuracy. This capability gives the system excellent personalization adaptability, meeting the special needs of different institutions and scenarios.

[0117] 8. This invention can be applied to multiple fields such as government services, financial services, healthcare, education management, and enterprise office work, and has broad market application prospects and commercial value. Especially in scenarios requiring the processing of large numbers of documents and forms, this invention can significantly improve work efficiency and reduce labor costs.

[0118] 9. The technical framework of this invention has good scalability, can easily integrate more modal information (such as handwriting recognition, seal recognition, etc.), can be extended to more languages ​​and cultural backgrounds, and can be adapted to multimodal large models of different sizes, providing a solid foundation for subsequent technology iteration and functional enhancement. Attached Figure Description

[0119] Figure 1 This is a flowchart of the present invention.

[0120] Figure 2 This is a diagram of the architecture of the present invention. Detailed Implementation

[0121] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0122] As shown in the figure, this embodiment provides an intelligent document understanding and automatic form filling system based on a multimodal large model. Taking the automatic filling of a "Personal Basic Information Registration Form" in a government service scenario as an application example, the specific solution is as follows:

[0123] I. System Architecture and Data Preparation

[0124] An intelligent document understanding and automatic form filling system is constructed, comprising a document acquisition module, a multimodal document encoding module, a document-form semantic alignment module, an information fusion module, a form constraint validation module, and an intelligent form filling module. Each module executes sequentially according to the technical solution described in the invention, forming a complete processing flow.

[0125] Application scenario configuration:

[0126] Source document types: ID card, household registration book, academic certificate, employment certificate, etc.;

[0127] Target form: Personal basic information registration form (including fields such as name, gender, date of birth, ID number, ethnicity, education, and work unit);

[0128] Document formats: Supports multiple formats including PDF, images (JPG, PNG), and scanned documents;

[0129] The system receives multiple source documents uploaded by users. First, it automatically identifies the document types and extracts structured information through a multimodal coding module. Then, it establishes field mapping relationships through a semantic alignment module, integrates information from multiple documents through an information fusion module, performs validation through a constraint understanding module, and finally fills the target form through an intelligent form-filling module. The entire process is coherent, with the output of each module serving as the input for the next, forming a complete end-to-end processing chain.

[0130] II. Implementation of Multimodal Document Coding

[0131] 1) Document visual layout coding

[0132] Taking an ID card image as an example, the input document image is... ,in Image height, This represents the image width.

[0133] The image is divided into a non-overlapping sequence of patches, each patch being of size [size missing]. Get the number of patches Linear projection of each patch yields the patch embedding:

[0134]

[0135] in For visual feature dimensions, For the first The embedding vector of each patch. Position encoding is added to preserve positional information. This yields a position-enhanced patch embedding:

[0136]

[0137] Document layout features are extracted through a 12-layer Transformer encoder. The Transformer encoder employs a self-attention mechanism and a feedforward neural network. Each layer contains a self-attention layer, a feedforward network layer, layer normalization, and residual connections, stacked layer by layer to obtain a deep feature representation.

[0138]

[0139] in Including the layout features of each patch, the system can identify fixed areas in the ID card (such as the name area, ID number area, photo area, etc.), providing location guidance for subsequent field extraction.

[0140] 2) Semantic encoding of text content

[0141] Text content is extracted from the ID card using OCR to obtain a text sequence. , .

[0142] Contextual semantic representations are obtained through a pre-trained text encoder (such as BERT-base-chinese). The BERT encoder employs a bidirectional Transformer architecture, capturing the contextual relationships between tokens through a self-attention mechanism. The representation of each token incorporates its preceding and following contextual information.

[0143]

[0144] in For text feature dimensions, For the first The context embedding vector of each token contains the semantic information of that token throughout the sequence. An attention pooling mechanism is used to obtain document-level semantic features.

[0145]

[0146] in The weight matrix is ​​a learnable matrix. For attention weight vectors, For the first The attention score of each token. This mechanism can highlight important information (such as name, ID number, and other key fields) and obtain document-level semantic feature vectors. .

[0147] 3) Understanding and implementing the table structure

[0148] Taking the family member information form in the household registration booklet as an example, the detected forms include: Rows (including headers) List (name, relationship to head of household, gender, date of birth, ID number, education level).

[0149] First, extract the visual and text features of each cell. For the first cell... Line number The visual characteristics of the column cells are: ( ), text features are ( The cell representation is obtained through feature fusion:

[0150]

[0151] in The weight matrix is ​​fused (768 dimensions of visual features + 768 dimensions of text features = 1536 dimensions). For cell feature dimensions, This indicates vector concatenation.

[0152] To model the semantic relationships between cells, a graph neural network is used to construct the table graph structure. Each cell is treated as a graph node, and edges are established based on row-column relationships and semantic similarity. Specifically, edges are established between cells in the same row or column, and also between semantically similar cells (such as the header and its corresponding data row). The features of all 30 cells are organized into an initial feature matrix. Each row corresponds to a cell's characteristics.

[0153] The node representation is updated through a 3-layer graph convolutional network. The specific implementation of graph convolution is as follows: for each node, the features of its neighboring nodes are aggregated, feature transformation is performed using a learnable weight matrix, and the ReLU activation function is applied. The layer's features are updated as follows:

[0154]

[0155] in For the first The node feature matrix of the layer For the first The learnable weight matrix of the layer, This is a normalized table adjacency matrix. After 3 layers of graph convolution ( The final update was obtained after that:

[0156]

[0157] in This is the updated cell feature matrix. The updated features accurately reflect the correspondence between the "Name" column and the "Zhang San" row, as well as the semantic relationships between fields in the same row, providing structured information for subsequent field extraction and matching.

[0158] III. Implementation of Document-Form Semantic Alignment

[0159] 1) Learning field semantic representation

[0160] Based on the document information extracted by the aforementioned multimodal coding module, the system detects and extracts field information. The fields detected from the ID card are:

[0161] ,in This represents the number of fields.

[0162] The target form field is

[0163] ,in This represents the number of form fields.

[0164] For each field, its label text and corresponding value text are extracted, and a semantic representation is obtained through a field encoder. The field encoder is implemented as follows: the field label text and value text are concatenated and input into the BERT encoder to obtain the contextual representation of the sequence. Then, the representation of the [CLS] tag is extracted to obtain the semantic vector of the field. Finally, the features are projected onto a unified semantic space through a fully connected layer. Taking the document field "ID number" (…) as an example… ) and the form field "ID number" For example, although the wording is slightly different, the semantics are the same. The semantic representation is extracted using a field encoder:

[0165]

[0166]

[0167] The BERT encoder output is a sequence representation matrix. This represents the extracted representation vector of the [CLS] tag. Representation vector labeled [CLS] (for text feature dimensions) For the projection weight matrix ( ), For bias vectors, For semantic representation dimension, and These are the semantic representation vectors for document fields and form fields, respectively.

[0168] 2) Cross-language and cross-format semantic alignment

[0169] To handle the matching problem of fields in different languages ​​and expressions (such as "ID number" and "ID number"), the system adopts a contrastive learning mechanism to map the same semantic fields in different expressions to a unified semantic space.

[0170] During the training phase, the data is compared and trained using fields that include both Chinese and English translations. For positive sample pairs (semantically identical fields, such as "ID card number" and "ID card number"), the cosine similarity is maximized; for negative sample pairs (semantically different fields), the similarity is minimized.

[0171] The contrastive learning loss function is:

[0172]

[0173] in The cosine similarity function is used. The temperature parameter is used. The field encoder is optimized using this loss function, ensuring that semantically similar fields are close together in the embedding space, while semantically different fields are far apart. After training, the semantic representations of "ID number" and "ID card number" are very close in the embedding space, with a cosine similarity of 0.92. The system is able to learn cross-language and cross-format semantic correspondences between fields.

[0174] 3) Field matching and mapping

[0175] Based on the learned semantic representation, a matching score is calculated between the source document fields and the target form fields. For document fields... With form fields The matching score is calculated as follows:

[0176]

[0177] in Represents document fields With form fields The matching probability. Based on the "ID number" field ( ) and the "ID number" field ( For example:

[0178]

[0179] A field mapping relationship is established by using a threshold (set to 0.7) and an optimal matching algorithm (such as the Hungarian algorithm). .For example: (Mapping ID card number to ID card number) (Name mapped to name) (Gender mapping to gender), etc. This mapping relationship provides accurate field correspondences for subsequent information extraction and form filling.

[0180] IV. Implementation of Cross-Document Information Integration

[0181] Based on the field mapping relationship established by the aforementioned semantic alignment module, the system collects candidate values ​​of the same field from multiple source documents to perform information fusion and conflict resolution.

[0182] 1) Multi-document information aggregation

[0183] Assuming the user uploaded Source document: ID card (Document 1, Household registration booklet (Document 2) ), academic certificate (document 3, From this The information extracted from the source documents is ,in Indicates the first Information collection of a document.

[0184] For the fields in the target form (Name), collect candidate values from multiple documents , where is the number of candidate values. Each candidate value is associated with its source document and confidence : , source , confidence ; , source , confidence ; , source , confidence

[0185] 2) Information consistency detection and conflict resolution

[0186] Calculate the consistency score between candidate values. For two candidate values and , define the consistency metric:

[0187]

[0188] where is the text similarity function. The specific implementation is as follows: for short texts (such as names), the edit distance (Levenshtein distance) is used to calculate the similarity, and after normalization, a similarity value between 0 and 1 is obtained; for long texts, the cosine similarity based on word embeddings is used. For the current situation, since the three values are exactly the same, the consistency score is 1.0 and it is determined to be consistent.

[0189] Assume another situation: the name in the ID card is "Zhang San" ( ), and the name in the household register is "Zhang Si" ( , there may be an input error). When calculating the consistency, the edit distance is used: the edit distance between two characters is 1 (replace "San" with "Si"), and the similarity obtained after normalization is:

[0190]

[0191] Set the threshold . Since 0.67 < 0.8, it is determined to be in conflict.

[0192] For conflicting candidate values, a credibility-based selection strategy is adopted. Define the final value of the field as:

[0193]

[0194] in For document The credibility weight (e.g., the ID card has a higher weight than other documents). The confidence level of the candidate value. Set the document confidence weight: (ID card is the highest) (Household registration booklet) (Academic certificate).

[0195] Calculate the weighted score for each candidate value. For "Zhang San," which appears in 3 documents, the scores from all sources need to be summed: Zhang San scores: Zhang Si's score:

[0196] The highest-scoring "Zhang San" (score 2.402) was selected as the final value. The system flags conflict warnings to prompt users to verify. This strategy prioritizes values ​​from highly reliable documents with high confidence levels. When the same value appears in multiple documents, scores from all sources are accumulated to ensure the reliability of information fusion.

[0197] V. Understanding Form Constraints and Implementing Intelligent Validation

[0198] Based on the field values ​​obtained from the aforementioned information fusion module, the system needs to understand the form constraints and perform intelligent validation to ensure the completeness and accuracy of the filled-in information.

[0199] 1) Constraint identification

[0200] For form fields By analyzing information such as field labels, prompt text, and example values, the constraints can be identified.

[0201]

[0202] in Indicates whether a field is required. Indicate the format requirements (such as mobile phone number, email address, date format). Indicates a range of values. This indicates field dependencies (e.g., "the end date must be later than the start date").

[0203] Taking the target form as an example, the constraints for each field are as follows: (Name): Required The format is Chinese name (2-4 characters). ); (ID card number): Required The format is 18-digit number or 17-digit number + X ( ); (Date of birth): Required The format is YYYY-MM-DD ( And must be earlier than the current date ( ); (Education level): Optional The value range is {primary school, junior high school, senior high school, junior college, undergraduate, master's, doctoral}. ); (Work unit): Optional No special format requirements.

[0204] 2) Information integrity check

[0205] Before filling out the form, check that all required fields have had their corresponding values ​​extracted from the source document. Based on the aforementioned information fusion results, assume the extracted information is as follows: (Name): Extracted. (From cross-document fusion); (Gender): Extracted (From ID card); (Date of birth): Extracted. (From ID card); (ID card number): Retrieved. (From ID card); (Ethnicity): Extracted. (From ID card); (Education): Not retrieved (This information is not found in the ID card and household registration book; it needs to be retrieved from the education certificate). (Work unit): Not withdrawn

[0206] Required fields set Integrity is calculated as follows:

[0207]

[0208] in Representation field The populated value is not empty. All required fields are complete; the system will continue with format validation.

[0209] 3) Format validation and logical verification

[0210] For the extracted values, perform format validation and logical checks. The format validation function is defined as follows:

[0211]

[0212] ID card number field For example, the value extracted from the ID card (18-digit number), format verification:

[0213]

[0214] For the date of birth field The extracted value is The format validation passed. Further logical validation is performed to ensure the birth date is earlier than the current date. For the constraint that "the date must be earlier than the current date," a logical validation function is used:

[0215]

[0216] For the date of birth field, the current date is 2024-01-01. We need to check if the date of birth is earlier than the current date. Use a logical validation function:

[0217]

[0218] in The date value to be verified. The current date is used. Since 1990-01-01 < 2024-01-01, the logical check passes. Let's consider another scenario: the extracted birth date is... (Future date), logical verification:

[0219]

[0220] Because 2025-01-01 > 2024-01-01, the logical validation failed, and the system generated an error message: "Date of birth cannot be later than the current date. Please check the source document or correct it manually." When a formatting error or logical error is detected, the system generates an error message and suggested corrections to ensure the accuracy of the information entered into the form.

[0221] VI. Intelligent form filling and information traceability

[0222] Based on the results of the aforementioned semantic alignment, information fusion, and constraint verification, the intelligent form filling module performs automatic form filling operations and marks the information source for each filled field, supporting interpretability and error traceability.

[0223] 1) Value extraction and format conversion

[0224] For successfully matched field pairs Extract the corresponding value from the source document Based on the format requirements of the target form fields, necessary format conversions are performed. The specific implementation of the format conversion function is as follows: Based on the target format type, conversion is performed using regular expression matching and string replacement. For date formats, the source format (such as YYYYMMDD, YYYY-MM-DD, etc.) is identified and then converted to the target format; for phone numbers, a separator is added according to the target format requirements; for other formats, corresponding conversion rules are applied.

[0225]

[0226] For example, for fields (Date of birth), the original value extracted from the ID card. (8-digit number, format YYYYMMDD), target format is Perform format conversion: Use regular expressions to identify the date format, extract the year, month, and day parts, and then recombine them according to the target format.

[0227]

[0228] For fields (ID card number), extracted value The target format requires an 18-digit number; the original format should be preserved after conversion. The system adjusts the format according to the actual form requirements to ensure that the entered values ​​conform to the form format specifications.

[0229] 2) Information source traceability labeling

[0230] Record the source of information for each entered field, supporting interpretability and error traceability. Information source tracing annotation is defined as follows:

[0231]

[0232] in For source documents, For the position in the document (e.g., "page 3, line 5"), For confidence level. Based on the "Name" field. For example, based on the aforementioned cross-document information fusion results:

[0233]

[0234] The system generates a form completion report, annotating the source document, location, and confidence level of each field, supporting user review and error tracing. When errors occur in the form, users can quickly locate the source of the problem based on the traceability information.

[0235] 3) Generation of form completion results

[0236] The final form completion result is generated. ,in Indicates the field status (filled, missing, format error, logical error, etc.).

[0237] Based on the aforementioned processing flow, the form filling result includes the following information: Fields have been successfully filled: (Name), (gender), (Date of birth) (ID card number) (Ethnicity), status is "filled"; missing fields: (Educational background is optional, but it is recommended to extract it from the degree certificate.) (Work unit, optional), status is "missing"; Format validation passed: all filled fields are formatted correctly; Logical validation passed: logical relationships such as date of birth are correct; Information traceability: the source of each field has been marked.

[0238] The system outputs a form completion report, which includes completion details, missing field prompts, error warnings, and suggested corrections. It prompts users to check for missing optional fields and provides an entry point for manual completion.

[0239] VII. Incremental Learning and Personalized Adaptation Implementation

[0240] Based on the aforementioned form completion results, when users manually correct the values ​​entered by the system, the system records the corrected values ​​and learns personalized rules to continuously optimize the form completion accuracy.

[0241] When a user corrects a value entered by the system, the correction is recorded. ,in The value entered by the system. This is the value corrected by the user. For example, the user changed the "Work Unit" field... Extracted from the system , revised to .

[0242] By analyzing and correcting patterns, personalized rules are learned. The personalized rule learning function is defined as follows:

[0243]

[0244] in For a set of rules, To correct historical records, Given the current context (organization type, form type, etc.), the system analysis and correction pattern revealed that this organization habitually uses the full company name (including "Co., Ltd.") when filling in the "Work Unit" field. Personalized rules were learned:

[0245]

[0246] In subsequent forms, the system automatically applies this rule, improving accuracy and user satisfaction. Through learning, the system can adapt to specific scenario filling preferences, achieving personalized customization.

[0247] Through the detailed description of this embodiment, those skilled in the art can clearly understand the technical solution of the present invention, and can implement the present invention in actual application scenarios according to the described method and system configuration, so as to achieve efficient and accurate intelligent document understanding and automatic form filling functions.

[0248] In summary, this invention designs specialized feature extraction and encoding modules for different types of information, such as document images, text content, and table structures. It employs a Transformer-based visual encoder to extract document layout features, utilizes a pre-trained large language model to extract text semantic features, and models the semantic relationships between table cells using graph neural networks, thereby achieving deep understanding and unified representation of multimodal information.

[0249] This invention innovatively designs a cross-language, cross-format field semantic alignment method, which maps identical semantic fields with different expressions to a unified semantic space through contrastive learning. This method can handle cross-language matching problems such as "ID card number" and "IDNumber," as well as expression differences such as "ID card number" and "ID card number," achieving true semantic understanding rather than simple keyword matching.

[0250] This invention designs a multi-document information aggregation mechanism that collects candidate values ​​for the same field from multiple source documents, identifies information conflicts through consistency detection, and resolves conflicts using a credibility-based selection strategy. This method can fully utilize multi-document information for completion while ensuring information consistency and reliability.

[0251] This invention automatically identifies field constraints (required, format, range, dependencies) by analyzing information such as labels and prompt text in form fields. It incorporates integrity checks, format validation, and logical verification mechanisms to detect potential errors before form completion, generating error messages and suggested corrections, significantly improving form completion accuracy.

[0252] This invention not only identifies the physical structure of a table (row and column division) but also understands its logical structure (header-data row relationship, merged cells, and cross-page tables). By employing graph neural networks to model the semantic relationships between cells, it can accurately identify field correspondences in complex tables and handle complex scenarios such as nested tables and irregular tables.

[0253] This invention provides detailed source information (document type, location, confidence level) for each entered field, generating a form completion report that supports manual review and error tracing. This mechanism improves the system's interpretability and credibility, making it easier for users to understand and correct the form completion results.

[0254] This invention learns from users' manual corrections, identifies the filling preferences of specific institutions and forms, and establishes a personalized form-filling model. This method enables the system to adapt to the specific needs of different scenarios, continuously improving form-filling accuracy and user satisfaction.

[0255] The above embodiments are merely typical illustrative methods of the present invention, and the scope of protection of the present invention is not limited thereto. All equivalent substitutions and improvements made under the concept of the present invention should fall within the scope of protection. It should be emphasized that any modifications or minor adjustments made by those skilled in the art without departing from the basic principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A smart document understanding and automatic form filling system based on a multimodal large model, characterized in that, include: The multimodal document encoding module performs feature extraction and encoding for different types of document information, achieving a deep understanding of documents within a unified multimodal framework; The document-form semantic alignment module, based on the document features and field information extracted by the aforementioned multimodal document encoding module, establishes a semantic mapping relationship between the source document fields and the target form fields in order to achieve intelligent information matching from the source document to the target form, and semantically aligns the field information obtained by multimodal encoding with the target form fields. The information fusion module, based on the semantic mapping relationship established by the aforementioned document-form semantic alignment module, enables cross-document information completion and consistency verification when the same information appears in multiple source documents; It utilizes the results of semantic alignment to collect candidate values ​​for the same field from multiple source documents, and obtains reliable field values ​​through consistency detection and conflict resolution; The form constraint validation module identifies the constraints of form fields based on the field values ​​obtained by the aforementioned information fusion module, and performs integrity checks, format validation, and logical validation before the form is filled out. The intelligent form filling module, based on the output of the aforementioned document-form semantic alignment, information fusion, and form constraint validation modules, performs automatic form filling operations and marks the information source for each filled field, supporting interpretability and error traceability.

2. The intelligent document understanding and automatic form filling system based on a multimodal large model according to claim 1, characterized in that, The multimodal document encoding module includes: S11, Document Visual Layout Coding For document images, a Transformer-based visual encoder is used to extract the document's layout and structural information; let the input document image be... ,in and These are the image height and width, respectively. First, the image is divided into a non-overlapping sequence of patches, each patch being of size [size missing]. Get the number of patches Linear projection is performed on each patch to obtain the patch embedding: in For visual feature dimensions; to preserve location information, add location encoding. This yields a position-enhanced patch embedding: The document layout features are extracted through multi-layer Transformer encoder processing. The Transformer encoder employs a self-attention mechanism and a feedforward neural network, specifically implemented as follows: for the input... First, the correlation between patches is calculated through a multi-head self-attention mechanism. Then, feature transformation is performed through a feedforward network. After layer normalization and residual connections, deep feature representations are obtained by stacking layers one by one. The Transformer encoder contains multiple Transformer blocks, each of which contains a self-attention layer, a feedforward network layer, layer normalization, and residual connections. It contains layout features for each patch and can identify the position and structural relationships of document elements such as headings, paragraphs, tables, and charts; S12, Semantic Encoding of Text Content For the text content in the document, semantic features are extracted using a pre-trained large language model; let the text sequence extracted by OCR be... ,in For the number of text tokens, Indicates the first A text token; A pre-trained large language model is used as a text encoder to obtain contextual semantic representations. The text encoder adopts a bidirectional Transformer architecture and captures the contextual relationships between tokens through a self-attention mechanism. Specifically, the token sequence is input into the encoder and processed through multiple Transformer blocks. The representation of each token incorporates its contextual information. in For text feature dimensions, For the first The context embedding vector of each token contains the semantic information of that token throughout the sequence; to capture document-level semantic information, an attention pooling mechanism is used: in The weight matrix is ​​a learnable matrix. For attention weight vectors, For the first Attention score of each token This is a document-level semantic feature vector; S13, Understanding and Coding Table Structures For tables in a document, we not only identify their physical structure but also understand their logical structure; suppose the detected tables are... OK Column, among which For the number of rows, For column numbers; First, extract the visual and text features of each cell; for the first cell... Line number The visual characteristics of the column cells are: Text features are Cell representations are obtained through feature fusion: in To fuse the weight matrix, For cell feature dimensions, This indicates vector concatenation; To model the semantic relationships between cells, a graph neural network is used to construct a table graph structure. Each cell is treated as a graph node, and edges are established based on row-column relationships and semantic similarity. Specifically, edges are established for cells in the same row or column, and also for cells with similar semantics. The features of all cells are organized into a matrix. , of which The row corresponds to the first Line number Column cell characteristics ; The node representation is updated using a graph convolutional network. The specific implementation of graph convolution is as follows: for each node, the features of its neighboring nodes are aggregated, feature transformation is performed using a learnable weight matrix, and a non-linear activation function is applied; The layer's features are updated as follows: in For the first The node feature matrix of the layer, The initial feature matrix, For the first The learnable weight matrix of the layer, It is a non-linear activation function. The normalized adjacency matrix; after The final update is obtained after layer graph convolution: in The updated cell feature matrix can capture the semantic relationships between the header and data rows, or cells in the same row or column.

3. The intelligent document understanding and automatic form filling system based on a multimodal large model according to claim 2, characterized in that, The document-form semantic alignment module includes: S21, Learning Field Semantic Representation For each field in the source document, extract its contextual semantic representation; let the detected fields in the document be... ,in Number of fields; For each field The text encoding process extracts the label text and corresponding value text, and obtains the semantic representation of the field through a field encoder. The field encoder is implemented by concatenating the label text and value text and inputting the result into a pre-trained text encoder to obtain the contextual representation of the sequence. Then, it extracts the representation of the [CLS] marker or pools the sequence representation to obtain the semantic vector of the field. Finally, a fully connected layer projects the features onto a unified semantic space. in This is a pre-trained text encoder that outputs a sequence representation matrix. This indicates the concatenation of field labels and values. The representation vector labeled [CLS] The projection weight matrix is... For bias vectors, This is a semantic representation dimension; similarly, for fields in the target form... Extract its semantic representation using the same encoding method: in Placeholders or sample values ​​for form fields; S22, Cross-language and cross-format semantic alignment To address the issue of field matching across different languages ​​and expressions, a contrastive learning mechanism is employed to map identical semantic fields from different expressions to a unified semantic space. By optimizing the field encoder through contrastive learning loss, fields with similar semantics are grouped closer together in the embedding space, while fields with different semantics are grouped further apart. in The cosine similarity function is used. For temperature parameters; S23, Field Matching and Mapping Based on the learned semantic representation, calculate the matching score between the source document fields and the target form fields: in Represents document fields With form fields The matching probability; establishing field mapping relationships through threshold judgment and optimal matching algorithm. .

4. The intelligent document understanding and automatic form filling system based on a multimodal large model according to claim 3, characterized in that, The information fusion module includes: S31, Multi-document information aggregation Let from The information extracted from the source documents is ,in Indicates the first A collection of information from a document; For the fields in the target form Collect candidate values ​​from multiple documents ,in The number of candidate values; each candidate value Link its source document and confidence level ; S32. Information Consistency Detection and Conflict Resolution Calculate the consistency score between candidate values; for two candidate values and Define a consistency metric: in The text similarity function is implemented as follows: First, the text values ​​are converted into vector representations, and then the cosine similarity or edit distance similarity between the two vectors is calculated. For short texts, edit distance normalization is used as the similarity; for long texts, cosine similarity based on word embeddings is used. When the consistency score is below a threshold... When this occurs, it is determined to be a conflict; For conflicting candidate values, a confidence-based selection strategy is adopted; define fields. The final value is: in For document Credibility weight, represents the confidence level of the candidate value.

5. The intelligent document understanding and automatic form filling system based on a multimodal large model according to claim 4, characterized in that, The form constraint validation module includes: S41. Constraint Identification For form fields By analyzing information such as field labels, prompt text, and example values, the constraints can be identified. in Indicates whether a field is required. Indicate the format requirements, Indicates a range of values. Indicates field dependencies; S42. Information Integrity Check Before filling out the form, check that all required fields have had their corresponding values ​​extracted from the source document: in This is a set of required fields. For fields The system will fill in the missing values; when the completeness is below the threshold, the system will prompt for missing fields and suggest finding them in other documents or entering them manually. S43. Format validation and logical verification For the extracted values, perform format validation and logical verification; the format validation function is defined as: For field dependencies, define logical validation functions; for date field dependencies, it is necessary to check the logical relationship between dates: for the dependency "end date > start date": For the constraint that "the date must be earlier than the current date", the logic check function is defined as: When a formatting error or logical error is detected, the system generates an error message and suggested corrections.

6. The intelligent document understanding and automatic form filling system based on a multimodal large model according to claim 5, characterized in that, The intelligent form-filling module includes: S51, Value Extraction and Format Conversion For successfully matched field pairs Extract the corresponding value from the source document The format conversion is performed according to the format requirements of the target form fields. The specific implementation of the format conversion function is as follows: based on the target format type, it uses regular expression matching and string replacement for conversion; for date formats, it identifies the source format and then converts it to the target format; for phone numbers, it adds a separator according to the target format requirements; for other formats, it uses the corresponding conversion rules. The format conversion function is based on Apply the corresponding conversion rules to the type; S52, Information Traceability Labeling Record the source of information for each field that is filled in: in For source documents, For the position in the document, Confidence level; S53. Generation of form completion results The final form completion result is generated. ,in Indicates the field status; the system outputs a form completion report, including completion details, missing field prompts, error warnings, and suggested corrections.

7. The intelligent document understanding and automatic form filling system based on a multimodal large model according to claim 6, characterized in that, Based on the form filling results generated by the aforementioned intelligent form filling module, when the user makes manual corrections, the system learns from the user's manual corrections through an incremental learning mechanism, identifies the filling preferences of specific institutions and specific forms, and establishes a personalized form filling model.

8. The intelligent document understanding and automatic form filling system based on a multimodal large model according to claim 7, characterized in that, The incremental learning mechanism includes: When a user corrects a value entered by the system, the correction is recorded. By analyzing correction patterns, personalized rules are learned. The specific implementation of personalized rule learning involves: collecting historical correction records from the same organization or form type, statistically analyzing correction patterns, and constructing a rule base; for new fields, based on context information and historical correction records, calculating the applicability probability of each rule, and selecting the rule with the highest probability. in For a set of rules, To correct historical records, For the current context; probability Calculated using statistical learning methods.

Images