Heterogeneous document structured data extraction system and method based on multi-modal fusion

Abstract

The invention discloses a heterogeneous document structured data extraction system and method based on multi-modal fusion, and the method comprises the following steps: S1, receiving a heterogeneous document, and carrying out the preprocessing; s2, extracting visual and semantic modal information based on the visual backbone network and an OCR module, and fusing spatial coordinates, text content and layout features; s3, receiving a dynamic target mode Schema, executing field-level semantic matching, and calculating semantic similarity; s4, performing consistency verification, verifying and filling the field values, and correcting the associated fields; s5, converting the format into a data file in a specified format, and reserving fields to be mapped with an original document semantic block; s6, automatically switching a cloud mode and a local mode according to a deployment environment; and S7, outputting the data file of the target field. According to the method, high-precision and low-delay structured data extraction of the heterogeneous format document can be realized, and the automatic processing efficiency and the data credibility are improved.

Description

Technical Field

[0001] This invention relates to the field of structured data extraction technology, and in particular to a system and method for extracting structured data from heterogeneous documents based on multimodal fusion. Background Technology

[0002] With the widespread adoption of IT-based office practices and data governance, massive amounts of structured and unstructured documents are continuously accumulating across various industries, including government, finance, healthcare, telecommunications, and law. These documents exist in diverse heterogeneous formats such as PDF, scanned images, Word, and Excel, containing not only standard text but also complex tables, images, watermarks, annotations, handwriting, signatures, and other structural elements—the main characteristics of "heterogeneous documents." Efficiently and accurately extracting structured field information from these heterogeneous documents has become a fundamental prerequisite for downstream tasks such as information extraction, knowledge graph construction, and intelligent question answering.

[0003] Traditional document structuring methods often rely on rule-driven, template-based field extraction strategies, or use a single OCR tool for text recognition followed by positional segmentation and keyword matching. These methods suffer from serious problems such as poor adaptability, weak structure preservation, unstable table recognition, and inability to generalize to complex documents. Furthermore, in practical applications, users often want to customize the extracted fields and flexibly specify output formats (such as JSON, CSV, and HTML), which places higher demands on the system's dynamic schema support, field alignment, and format conversion capabilities.

[0004] In recent years, deep learning has made significant progress in the field of document understanding, especially with the development of visual language models, multimodal coding networks, and pre-trained language models, making it possible to fuse image and text modalities in modeling. However, most existing methods focus on standard documents such as questionnaires, invoices, and financial reports, lacking universal support for unstructured documents, complex layouts, handwritten markings, and non-standard layouts. Furthermore, most existing models are statically structured, lacking the ability to dynamically adapt to user schemas, and at the deployment level, they struggle to balance the lightweight requirements of local environments with the high-precision inference capabilities of the cloud.

[0005] More importantly, existing structured document extraction methods still have significant shortcomings in semantic consistency verification between fields, field completion for missing data, and multi-format output. For example, some fields may be missing or out of order due to OCR errors or formatting issues, making it difficult to accurately restore field hierarchy, table structure, and semantic blocks. The lack of consistency verification mechanisms for field output also affects the usability of data quality in subsequent stages.

[0006] Therefore, how to provide a system and method for extracting heterogeneous document structured data based on multimodal fusion is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] One objective of this invention is to propose a heterogeneous document structured data extraction system and method based on multimodal fusion. This invention fully utilizes the visual backbone network, OCR recognition network, and Transformer multimodal attention mechanism, and describes in detail the entire process of automatically extracting user-defined structured fields from complex heterogeneous documents and converting them into multi-format output files. It has the advantages of strong document adaptability and high field extraction accuracy.

[0008] The method for extracting heterogeneous document structured data based on multimodal fusion according to embodiments of the present invention includes the following steps: S1. Receive heterogeneous documents to be processed and perform preprocessing; S2. Input the preprocessed heterogeneous document into the multimodal analysis engine. Extract visual modality and semantic modality information in parallel based on the visual backbone network and the embedded OCR module. In the Transformer architecture, integrate spatial coordinates, text content and layout features to construct the document's internal representation vector. S3. Receive the user-defined dynamic target pattern Schema, perform field-level semantic matching on the internal representation vector, combine the Transformer attention mechanism to calculate the semantic similarity of the field candidate regions, complete the field location recognition and field value extraction, and generate an intermediate dataset. S4. Perform field consistency verification on the intermediate dataset. Use rule-based verification functions and field type discriminators to verify the validity of field values ​​and fill in missing values. Correct the matching of context-related fields and output a structured field set. S5. Call the format conversion module to convert the structured field set into a data file of the specified format, and preserve the hierarchical relationship between fields, the table structure and the semantic block mapping in the original document during the output process; S6. Automatically switch computing modes according to the deployment environment. In cloud deployment, call large-scale multimodal models, and in local deployment, call lightweight compressed models. The model output maintains structural consistency and constant field correspondence. S7. Output the data file of the target field and complete the data connection with the subsequent system.

[0009] Optionally, the preprocessing includes eliminating image noise using a Gaussian filtering algorithm or a median filtering algorithm, performing rotational correction on the image using a tilt correction algorithm to ensure that the text is consistent in the horizontal direction, and improving image clarity through operations such as brightness equalization, contrast enhancement, and edge sharpening.

[0010] Optionally, S2 specifically includes: S21. Input the preprocessed heterogeneous document into the visual backbone network composed of Vision Encoder, extract the image feature tensor, and combine it with the two-dimensional coordinate information of each image block to generate an image feature representation with position encoding. S22. Input the preprocessed heterogeneous document into the OCR module composed of Nanonets-OCR2, extract the text recognition sequence, character bounding boxes and character confidence distribution, construct the text embedding tensor, and align the text embedding tensor with the image feature representation in the spatial position dimension. S23. Concatenate the aligned image feature representation with the text embedding tensor into a multimodal input tensor, and input it into a multi-head Transformer structure. Based on the cross-attention mechanism, fuse the image features, text embedding tensor and spatial location embedding, and output the fused multimodal representation tensor. S24. Input the fused multimodal representation tensor into the feature enhancement module with a position-aware self-attention mechanism to extract the contextual semantic relationships across regions and paragraphs in the document and construct the internal representation vector of the document.

[0011] Optionally, S3 specifically includes: S31. Receive the target schema defined by the user, use the BERT encoder to perform context representation calculation on the field name sequence in the schema, and generate field embedding tensors. Each field embedding tensor consists of the name, field type and semantic label of the corresponding field. S32. Input the document's internal representation vector and field embedding tensor into the Transformer-based cross-attention module, and use a multi-head attention mechanism to calculate the attention distribution when the field embedding tensor is used as the query vector and the internal representation vector is used as the key-value pair, respectively. S33. Based on the computational attention distribution, feature mapping is performed, and the semantic similarity score between the field embedding tensor and each document position vector is calculated using the dot product similarity function. S34. Based on semantic similarity scores, the maximum score aggregation function is used to identify the most relevant index interval of each field in the document representation, and the text sequence within the interval is extracted as candidate field value fragments. S35. Perform type consistency verification on the candidate field value fragments, remove candidate fragments that are inconsistent with the field types in the Schema, output the field position set and the corresponding field value, and generate an intermediate dataset that satisfies the Schema constraints.

[0012] Optionally, S4 specifically includes: S41. Input each field value in the intermediate dataset into a regular expression-based rule validation function to verify whether the field value conforms to the structural specification defined by the field type, and output the valid field label. S42. Combine each field value in the intermediate dataset with the field type label to form a field pair, input it into the field type discriminator composed of BiLSTM and Softmax classifiers, calculate the type prediction label corresponding to the field value, and compare the prediction label with the field type definition in the schema to mark the type consistency result. S43. Construct a graph structure based on the field values ​​and context field values ​​in the intermediate dataset. Each field value in the graph is a node, and the semantic relationship between fields is used as an edge. Input the graph structure into the graph convolutional network (GCN) model, calculate the node representation based on the semantic similarity of adjacent field values, perform consistency feature inference between fields, and identify contextual logical conflicts between fields. S44. Make a joint judgment based on the field's valid label, type consistency results, and contextual logic conflict results: If a field value has an abnormal format, it will be discarded. If a field is missing in the structural position, a K-nearest neighbor interpolation operation is performed based on semantically similar fields; If there is a contextual logic conflict between field pairs, the field value with lower confidence will be replaced based on the field confidence level. Output a structured set of fields with consistent structure and valid content.

[0013] Optionally, S42 specifically includes: S421. Encode each field value using BERT to generate the corresponding contextual semantic embedding sequence, which serves as the initial input feature for the field representation. S422. Input the initial input features into a bidirectional long short-term memory network (BiLSTM), calculate the forward and reverse hidden states respectively, and concatenate them to obtain the field representation vector. S423. Input the field representation vector into the fully connected layer and connect it to the Softmax classifier. Perform probability distribution prediction in the predetermined field type label space and output the type prediction label of each field value and the corresponding probability distribution vector. S424. Calculate the maximum probability score of the predicted label based on the probability distribution vector, and match it with the type label of the corresponding field in the Schema. If the labels are consistent and the predicted probability is higher than the set confidence threshold, mark the field as consistent in type; otherwise, mark it as inconsistent in type.

[0014] Optionally, S5 specifically includes: S51. Construct a structure mapping table based on the structured field set. The structure mapping table records field labels, field values, field hierarchical relationships and semantic block numbers corresponding to the fields, and sorts them in the order of semantic block numbers, serving as the structural basis for format conversion. S52. Determine the conversion strategy based on the output format type: When the output format is JSON, hierarchical parsing and key-value nesting are performed, and a hierarchical expansion algorithm is used to construct nested JSON objects based on the field hierarchy relationship in the structure mapping table. When the output format is Markdown, the header encoder function is called to construct the header row from the first-level fields, and the table body is generated using the row alignment algorithm. When the output format is CSV, perform header extraction and field alignment operations, and output a two-dimensional field matrix using a column vector concatenation strategy. When the output format is HTML, the DOM node generation function is called based on the semantic block number and field position index in the structure mapping table to construct an HTML node structure with semantic block identifiers; S53. Perform a structure verification operation on the format conversion result, and compare the field set in the output structure with the field label and position order in the structure mapping table to verify the consistency of the number, order and value content of the fields. When the output format is JSON, verify whether the nested objects are nested according to the field level and whether the field key names are consistent with the mapping table; When the output format is Markdown, verify whether the heading level is consistent with the field level, whether the table structure is complete, whether the field values ​​are correctly filled into the corresponding text blocks and table cells, and whether the paragraph order is consistent with the mapping table. When the output format is CSV, verify whether the number of field columns is fixed, whether the order of field columns is consistent with the order of the mapping table, and whether all records are aligned completely. When the output format is HTML, verify that all field values ​​are inserted into the corresponding semantic block container, that field tags constitute valid DOM nodes, and confirm that all fields are bound to unique position indexes. If all format and structure validations pass, output the data file in the target format.

[0015] Optionally, S6 specifically includes: S61. Perform deployment environment identification operation, read the runtime environment variable configuration items, determine whether there is a high-performance graphics processing unit (GPU) and high-bandwidth memory access permission, and input the detection result into the condition judgment function. If the condition is met, generate a deployment environment label as a cloud environment; otherwise, generate a deployment environment label as a local environment. S62. When the deployment environment is marked as a cloud deployment environment, a large-scale multimodal model trained based on the LLM architecture is invoked. The large-scale multimodal model contains a Transformer structure with more than 7 billion parameters, receives the fused multimodal representation tensor, and outputs the structured probability distribution vector corresponding to the field labels. When the deployment environment is marked as a local deployment environment, a lightweight model that is jointly compressed by sparse pruning, parameter sharing and 8-bit integer quantization is invoked. The lightweight model is based on the TinyBERT backbone structure, extends the visual cross attention unit, receives the fused multimodal representation tensor, and outputs the field position index sequence and the field value sequence. S63. Perform a consistency alignment operation on the field prediction results output by the two types of models, call the field reordering algorithm based on the field order defined by the schema, combine the starting position index of the field in the original document, adjust the output field order to the schema order, and project the output field labels to the schema field label set through the label mapping function to generate field matching results.

[0016] A heterogeneous document structured data extraction system based on multimodal fusion according to an embodiment of the present invention includes: The document receiving module is used to receive documents in heterogeneous formats and perform image normalization, resolution unification, noise removal, and text region enhancement operations. The feature extraction module is used to extract visual and semantic features of the document in parallel. The visual features are composed of image feature tensors and spatial location embeddings generated by Vision Encoder, and the semantic features are composed of text sequences and character bounding boxes extracted by the Nanonets-OCR2 structure, and the fusion representation encoding is completed through the Transformer structure. The target parsing module receives user-defined field schemas, generates field-aware representations based on BERT encoding and Transformer cross-attention mechanism, performs semantic similarity calculation and field position recognition, and outputs candidate field value fragments. The field validation module is used to perform legality verification, field type discrimination and context logic consistency checks on candidate field values, construct graph structure representation and call graph convolutional network to perform inter-field consistency inference, and output a set of fields with valid structure. The format conversion module is used to convert a set of legally structured fields into structured file formats such as JSON, Markdown, CSV, and HTML, build a structure mapping table, and maintain the consistency between field hierarchy, semantic blocks, and table structure. The mode switching module is used to detect the deployment environment configuration, and based on the environment determination result, call the large-scale cloud model with LLM structure or the local lightweight model based on TinyBERT compression to perform field prediction inference, while maintaining the consistency of model output. The results docking module is used to perform field output reordering and tag mapping based on the schema field order and the document starting index order, and output a structured data file that can be used for subsequent system calls.

[0017] The beneficial effects of this invention are: First, this invention achieves synchronous feature extraction of image and text modalities by introducing a parallel architecture of visual backbone network and OCR embedding module. It also introduces spatial location encoding and cross-paragraph context modeling mechanism in the Transformer structure, which effectively solves the problem of the inability to uniformly model complex layout documents, table structures and multiple semantic blocks in the prior art, and significantly improves the accuracy and robustness of field recognition.

[0018] Secondly, this invention proposes a dynamic field matching method based on user-defined schemas, constructing a BERT encoder and a cross-attention fusion mechanism, which can accurately align document content with the field targets defined in the schema, has high adaptability, and is suitable for document extraction tasks under multiple industries and different business templates; at the same time, it is combined with a BiLSTM field type discriminator and a GCN graph structure consistency check model to enhance the logical correctness and data quality of field results.

[0019] Finally, this invention constructs a format-preserving structure conversion mechanism that can output structured data containing field hierarchy, table relationships, and semantic block identifiers, and supports multiple formats such as JSON, CSV, Markdown, and HTML, solving the problems of lost structure information and fixed output format in existing solutions. At the same time, it introduces a deployment environment recognition mechanism, calling the large model in the cloud and the compressed model locally, effectively balancing accuracy and deployment resource limitations, and possessing good engineering practicality and deployment flexibility. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0021] Figure 1 This is an overall flowchart of the heterogeneous document structured data extraction method based on multimodal fusion proposed in this invention; Figure 2 This is a flowchart of the multimodal feature extraction process for the heterogeneous document structured data extraction method based on multimodal fusion proposed in this invention. Figure 3This is a module structure diagram of the heterogeneous document structured data extraction system based on multimodal fusion proposed in this invention. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0023] refer to Figure 1-2 A method for extracting structured data from heterogeneous documents based on multimodal fusion includes the following steps: S1. Receive heterogeneous documents to be processed and perform preprocessing; S2. Input the preprocessed heterogeneous document into the multimodal analysis engine. Extract visual modality and semantic modality information in parallel based on the visual backbone network and the embedded OCR module. In the Transformer architecture, integrate spatial coordinates, text content and layout features to construct the document's internal representation vector. S3. Receive the user-defined dynamic target pattern Schema, perform field-level semantic matching on the internal representation vector, combine the Transformer attention mechanism to calculate the semantic similarity of the field candidate regions, complete the field location recognition and field value extraction, and generate an intermediate dataset. S4. Perform field consistency verification on the intermediate dataset. Use rule-based verification functions and field type discriminators to verify the validity of field values ​​and fill in missing values. Correct the matching of context-related fields and output a structured field set. S5. Call the format conversion module to convert the structured field set into a data file of the specified format, and preserve the hierarchical relationship between fields, the table structure and the semantic block mapping in the original document during the output process; S6. Automatically switch computing modes according to the deployment environment. In cloud deployment, call large-scale multimodal models, and in local deployment, call lightweight compressed models. The model output maintains structural consistency and constant field correspondence. S7. Output the data file of the target field and complete the data connection with the subsequent system.

[0024] In this embodiment, the preprocessing includes eliminating image noise using a Gaussian filtering algorithm or a median filtering algorithm, performing rotational correction on the image using a tilt correction algorithm to ensure that the text is consistent in the horizontal direction, and improving image clarity through operations such as brightness equalization, contrast enhancement, and edge sharpening.

[0025] In this embodiment, S2 specifically includes: S21. Input the preprocessed heterogeneous document into the visual backbone network composed of Vision Encoder, extract the image feature tensor, and combine it with the two-dimensional coordinate information of each image block to generate an image feature representation with position encoding. S22. Input the preprocessed heterogeneous document into the OCR module composed of Nanonets-OCR2, extract the text recognition sequence, character bounding boxes and character confidence distribution, construct the text embedding tensor, and align the text embedding tensor with the image feature representation in the spatial position dimension. The OCR module specifically includes three parts: Feature extraction layer: A multi-layer two-dimensional convolutional network is used to perform feature extraction operations on the input document image, and the output is a two-dimensional feature map representing the local perception of characters; Sequence modeling layer: The convolutional feature map is unfolded in the horizontal direction and input into the bidirectional gated recurrent unit network to learn the contextual dependencies between characters and output a sequence of contextual feature vectors representing the character sequence; Predictive decoding layer: The character sequence is trained using the connection-time classification loss function, and the output is the character recognition sequence, the coordinates of the character bounding box, and the confidence score of each character position, which constitutes the text recognition result; S23. Concatenate the aligned image feature representation with the text embedding tensor into a multimodal input tensor, and input it into a multi-head Transformer structure. Based on the cross-attention mechanism, fuse the image features, text embedding tensor and spatial location embedding, and output the fused multimodal representation tensor. S24. Input the fused multimodal representation tensor into the feature enhancement module with a position-aware self-attention mechanism to extract the contextual semantic relationships across regions and paragraphs in the document and construct the internal representation vector of the document. The feature enhancement module is used to perform contextual semantic enhancement on the fused multimodal representation tensor, and includes the following parts: Position Encoding Injection Layer: The spatial position coordinates of each multimodal vector are encoded into position embedding vectors. The position vectors generated by the sine-cosine function are concatenated with the fused multimodal representation tensor according to the dimensions to enhance its spatial awareness of paragraph, row, column and field layout. Multi-head self-attention layer: Introduces multiple independent attention heads to perform parallel self-attention computation on the fused tensor. Each attention head uses the representation vector at any position in the document as the query, and performs a weighted calculation with all position vectors as the key and value, outputting a context representation containing global dependencies. Position-aware weight adjustment layer: Introduces a relative position bias term into the attention weight calculation and uses the position difference vector as a weight adjustment factor to improve the ability to model long-distance semantic dependencies across paragraphs and table regions. Residual connection and normalization layer: The attention output is residually connected to the original input, and layer normalization is performed to stabilize training and maintain the consistency of feature representation.

[0026] In this embodiment, S3 specifically includes: S31. Receive the target schema defined by the user, use the BERT encoder to perform context representation calculation on the field name sequence in the schema, and generate field embedding tensors. Each field embedding tensor consists of the name, field type and semantic label of the corresponding field. The schema is a dynamic field pattern template defined by the user or application configuration end, used to guide the matching and extraction of structured fields in a document, and has the following structural characteristics: Field Name Field: Used to define the natural language name of the target field, such as "Company Name", "Taxpayer Identification Number", "Contract Signing Date", etc., which serve as the semantic subject of the BERT encoder input; Field type field: Used to specify the data type or format requirements of the field, such as "string", "date", "amount", "address", etc., for subsequent type validation and format validation; Semantic tag field: Used to represent the functional tags or business semantic categories of fields, such as "company information", "time", "amount", "address", etc., to assist in field similarity calculation and semantic alignment; Structure level field: Used to identify the nesting level of a field in the target structure output format, such as first-level field, table field, nested block field, etc., to guide formatted output and hierarchical restoration; The schema as a whole constitutes a structured set of nested objects. After being input into the BERT encoder, the context representation model of all field names is performed, and the field name embedding vector, together with its type and semantic label, forms a field embedding tensor. S32. Input the document's internal representation vector and field embedding tensor into the Transformer-based cross-attention module, and use a multi-head attention mechanism to calculate the attention distribution when the field embedding tensor is used as the query vector and the internal representation vector is used as the key-value pair, respectively. S33. Based on the computational attention distribution, feature mapping is performed, and the semantic similarity score between the field embedding tensor and each document position vector is calculated using the dot product similarity function. S34. Based on semantic similarity scores, the maximum score aggregation function is used to identify the most relevant index interval of each field in the document representation, and the text sequence within the interval is extracted as candidate field value fragments. S35. Perform type consistency verification on the candidate field value fragments, remove candidate fragments that are inconsistent with the field types in the Schema, output the field position set and the corresponding field value, and generate an intermediate dataset that satisfies the Schema constraints.

[0027] In this embodiment, S4 specifically includes: S41. Input each field value in the intermediate dataset into a regular expression-based rule validation function to verify whether the field value conforms to the structural specification defined by the field type, and output the valid field label. The rule validation function is a set of regular expression functions based on field type mapping. It constructs a predefined set of regular expression rules for different field types and determines whether the field value meets the structural format requirements through string matching. S42. Combine each field value in the intermediate dataset with the field type label to form a field pair, input it into the field type discriminator composed of BiLSTM and Softmax classifiers, calculate the type prediction label corresponding to the field value, and compare the prediction label with the field type definition in the schema to mark the type consistency result. The field type discriminator structure includes: Input representation layer: Each field value in the intermediate dataset is segmented at the character or word level, and the field values ​​are encoded into embedding sequences using pre-trained word vectors to form a vectorized input representation of the field values; Context modeling layer: The field embedding sequence is input into BiLSTM, and semantic modeling of the field sequence is performed from both forward and backward directions, outputting a context-aware representation vector for each field value; Global feature pooling layer: Apply average pooling and max pooling operations to the output sequence of BiLSTM and concatenate them to generate a fixed-dimensional global field feature representation; Type prediction layer: Input global field features into the fully connected layer and Softmax classifier, output the type prediction probability distribution of field values ​​based on the set of field type labels defined in the schema, and select the type label corresponding to the highest probability as the prediction result; Consistency labeling module: Compares the predicted label of field type with the definition label of the corresponding field in the schema. If they match, it is marked as type consistent; otherwise, it is marked as type conflict, for use by the subsequent consistency correction mechanism. S43. Construct a graph structure based on the field values ​​and context field values ​​in the intermediate dataset. Each field value in the graph is a node, and the semantic relationship between fields is used as an edge. Input the graph structure into the graph convolutional network (GCN) model, calculate the node representation based on the semantic similarity of adjacent field values, perform consistency feature inference between fields, and identify contextual logical conflicts between fields. S44. Make a joint judgment based on the field's valid label, type consistency results, and contextual logic conflict results: If a field value has an abnormal format, it will be discarded. If a field is missing in the structural position, a K-nearest neighbor interpolation operation is performed based on semantically similar fields; If there is a contextual logic conflict between field pairs, the field value with lower confidence will be replaced based on the field confidence level. Output a structured set of fields with consistent structure and valid content.

[0028] In this embodiment, S42 specifically includes: S421. Encode each field value using BERT to generate the corresponding contextual semantic embedding sequence, which serves as the initial input feature for the field representation. S422. Input the initial input features into a bidirectional long short-term memory network (BiLSTM), calculate the forward and reverse hidden states respectively, and concatenate them to obtain the field representation vector. S423. Input the field representation vector into the fully connected layer and connect it to the Softmax classifier. Perform probability distribution prediction in the predetermined field type label space and output the type prediction label of each field value and the corresponding probability distribution vector. S424. Calculate the maximum probability score of the predicted label based on the probability distribution vector, and match it with the type label of the corresponding field in the Schema. If the labels are consistent and the predicted probability is higher than the set confidence threshold, mark the field as consistent in type; otherwise, mark it as inconsistent in type.

[0029] In this embodiment, S5 specifically includes: S51. Construct a structure mapping table based on the structured field set. The structure mapping table records field labels, field values, field hierarchical relationships and semantic block numbers corresponding to the fields, and sorts them in the order of semantic block numbers, serving as the structural basis for format conversion. S52. Determine the conversion strategy based on the output format type: When the output format is JSON, hierarchical parsing and key-value nesting are performed, and a hierarchical expansion algorithm is used to construct nested JSON objects based on the field hierarchy relationship in the structure mapping table. When the output format is Markdown, the header encoder function is called to construct the header row from the first-level fields, and the table body is generated using the row alignment algorithm. When the output format is CSV, perform header extraction and field alignment operations, and output a two-dimensional field matrix using a column vector concatenation strategy. When the output format is HTML, the DOM node generation function is called based on the semantic block number and field position index in the structure mapping table to construct an HTML node structure with semantic block identifiers; S53. Perform a structure verification operation on the format conversion result, and compare the field set in the output structure with the field label and position order in the structure mapping table to verify the consistency of the number, order and value content of the fields. When the output format is JSON, verify whether the nested objects are nested according to the field level and whether the field key names are consistent with the mapping table; When the output format is Markdown, verify whether the heading level is consistent with the field level, whether the table structure is complete, whether the field values ​​are correctly filled into the corresponding text blocks and table cells, and whether the paragraph order is consistent with the mapping table. When the output format is CSV, verify whether the number of field columns is fixed, whether the order of field columns is consistent with the order of the mapping table, and whether all records are aligned completely. When the output format is HTML, verify that all field values ​​are inserted into the corresponding semantic block container, that field tags constitute valid DOM nodes, and confirm that all fields are bound to unique position indexes. If all format and structure validations pass, output the data file in the target format.

[0030] In this embodiment, S6 specifically includes: S61. Perform deployment environment identification operation, read the runtime environment variable configuration items, determine whether there is a high-performance graphics processing unit (GPU) and high-bandwidth memory access permission, and input the detection result into the condition judgment function. If the condition is met, generate a deployment environment label as a cloud environment; otherwise, generate a deployment environment label as a local environment. S62. When the deployment environment is marked as a cloud deployment environment, a large-scale multimodal model trained based on the LLM architecture is invoked. The large-scale multimodal model contains a Transformer structure with more than 7 billion parameters, receives the fused multimodal representation tensor, and outputs the structured probability distribution vector corresponding to the field labels. When the deployment environment is marked as a local deployment environment, a lightweight model that is jointly compressed by sparse pruning, parameter sharing and 8-bit integer quantization is invoked. The lightweight model is based on the TinyBERT backbone structure, extends the visual cross attention unit, receives the fused multimodal representation tensor, and outputs the field position index sequence and the field value sequence. S63. Perform a consistency alignment operation on the field prediction results output by the two types of models, call the field reordering algorithm based on the field order defined by the schema, combine the starting position index of the field in the original document, adjust the output field order to the schema order, and project the output field labels to the schema field label set through the label mapping function to generate field matching results.

[0031] refer to Figure 3 A heterogeneous document structured data extraction system based on multimodal fusion includes: The document receiving module is used to receive documents in heterogeneous formats and perform image normalization, resolution unification, noise removal, and text region enhancement operations. The feature extraction module is used to extract visual and semantic features of the document in parallel. The visual features are composed of image feature tensors and spatial location embeddings generated by Vision Encoder, and the semantic features are composed of text sequences and character bounding boxes extracted by the Nanonets-OCR2 structure, and the fusion representation encoding is completed through the Transformer structure. The target parsing module receives user-defined field schemas, generates field-aware representations based on BERT encoding and Transformer cross-attention mechanism, performs semantic similarity calculation and field position recognition, and outputs candidate field value fragments. The field validation module is used to perform legality verification, field type discrimination and context logic consistency checks on candidate field values, construct graph structure representation and call graph convolutional network to perform inter-field consistency inference, and output a set of fields with valid structure. The format conversion module is used to convert a set of legally structured fields into structured file formats such as JSON, Markdown, CSV, and HTML, build a structure mapping table, and maintain the consistency between field hierarchy, semantic blocks, and table structure. The mode switching module is used to detect the deployment environment configuration, and based on the environment determination result, call the large-scale cloud model with LLM structure or the local lightweight model based on TinyBERT compression to perform field prediction inference, while maintaining the consistency of model output. The results docking module is used to perform field output reordering and tag mapping based on the schema field order and the document starting index order, and output a structured data file that can be used for subsequent system calls.

[0032] Example 1: To verify the feasibility of this invention, it was applied to a structured field extraction task in a tax-related intelligent file management platform. This platform contains a large number of heterogeneous documents, including scanned VAT invoices, contracts, expense reports, and forms. These documents come from diverse sources, have significantly different formats, and suffer from issues such as text rotation, inconsistent paragraph breaks, irregular table structures, and handwritten content. This makes it difficult for traditional OCR tools to reliably extract target fields, and manual extraction is inefficient, with field accuracy below 70%, severely impacting the efficiency and accuracy of business data aggregation.

[0033] In this scenario, the proposed multimodal fusion-based heterogeneous document structured data extraction method is deployed. The system automatically receives the aforementioned multi-type documents and eliminates scanning distortion and font blurring through image enhancement and tilt correction preprocessing. Subsequently, the documents are simultaneously input into the Vision Encoder backbone network and the Nanonets-OCR2 text recognition network, respectively extracting image modal feature tensors and text embedding tensors, and aligning them based on spatial coordinates. After multimodal fusion, context modeling is performed using a Transformer model, and then combined with BERT-encoded user schemas for field semantic matching, locating field regions and extracting corresponding field values.

[0034] After field extraction, the system uses a triple mechanism—rule functions, field type discrimination networks, and GCN graph structure consistency inference—to perform consistency verification and missing data repair, ultimately outputting a field set with consistent structure and compliant fields. In the output format stage, the platform specifies JSON and CSV structured formats for data output. The system successfully preserves the hierarchical structure and semantic block correspondence between fields in the original document and is compatible with data interface interfaces.

[0035] After a two-week validation period, a total of 13,752 document samples were processed, covering 12 document templates and 13 types of target fields. The platform compared the performance of the method of this invention with traditional OCR+template extraction strategies and single-modal deep learning models in field extraction tasks. The results show that the present invention has significant advantages in field recall, accuracy, consistency, and average inference time. The following is a summary of the specific data results:

[0036] Table 1. Performance Comparison of Structured Field Extraction from Multiple Document Types

[0037] As shown in Table 1, this invention achieves more stable and accurate structured field extraction in heterogeneous document environments. In particular, it significantly outperforms existing methods in terms of field missing repair and field semantic consistency, while maintaining a reasonable inference time to meet the online deployment requirements of large-scale business scenarios. It has strong engineering application value and promotion potential.

[0038] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for extracting structured data from heterogeneous documents based on multimodal fusion, characterized in that, Includes the following steps: S1. Receive heterogeneous documents to be processed and perform preprocessing; S2. Input the preprocessed heterogeneous document into the multimodal analysis engine. Extract visual modality and semantic modality information in parallel based on the visual backbone network and the embedded OCR module. In the Transformer architecture, integrate spatial coordinates, text content and layout features to construct the document's internal representation vector. S3. Receive the user-defined dynamic target pattern Schema, perform field-level semantic matching on the internal representation vector, combine the Transformer attention mechanism to calculate the semantic similarity of the field candidate regions, complete the field location recognition and field value extraction, and generate an intermediate dataset. S4. Perform field consistency verification on the intermediate dataset. Use rule-based verification functions and field type discriminators to verify the validity of field values ​​and fill in missing values. Correct the matching of context-related fields and output a structured field set. S5. Call the format conversion module to convert the structured field set into a data file of the specified format, and preserve the hierarchical relationship between fields, the table structure and the semantic block mapping in the original document during the output process; S6. Automatically switch computing modes according to the deployment environment. In cloud deployment, call large-scale multimodal models, and in local deployment, call lightweight compressed models. The model output maintains structural consistency and constant field correspondence. S7. Output the data file of the target field and complete the data connection with the subsequent system.

2. The method for extracting heterogeneous document structured data based on multimodal fusion according to claim 1, characterized in that, The preprocessing includes eliminating image noise using Gaussian filtering or median filtering algorithms, rotating the image using a tilt correction algorithm to ensure consistent text horizontal direction, and improving image clarity through operations such as brightness equalization, contrast enhancement, and edge sharpening.

3. The method for extracting heterogeneous document structured data based on multimodal fusion according to claim 1, characterized in that, S2 specifically includes: S21. Input the preprocessed heterogeneous document into the visual backbone network composed of Vision Encoder, extract the image feature tensor, and combine it with the two-dimensional coordinate information of each image block to generate an image feature representation with position encoding. S22. Input the preprocessed heterogeneous document into the OCR module composed of Nanonets-OCR2, extract the text recognition sequence, character bounding boxes and character confidence distribution, construct the text embedding tensor, and align the text embedding tensor with the image feature representation in the spatial position dimension. S23. Concatenate the aligned image feature representation with the text embedding tensor into a multimodal input tensor, and input it into a multi-head Transformer structure. Based on the cross-attention mechanism, fuse the image features, text embedding tensor and spatial location embedding, and output the fused multimodal representation tensor. S24. Input the fused multimodal representation tensor into the feature enhancement module with a position-aware self-attention mechanism to extract the contextual semantic relationships across regions and paragraphs in the document and construct the internal representation vector of the document.

4. The method for extracting heterogeneous document structured data based on multimodal fusion according to claim 1, characterized in that, S3 specifically includes: S31. Receive the target schema defined by the user, use the BERT encoder to perform context representation calculation on the field name sequence in the schema, and generate field embedding tensors. Each field embedding tensor consists of the name, field type and semantic label of the corresponding field. S32. Input the document's internal representation vector and field embedding tensor into the Transformer-based cross-attention module, and use a multi-head attention mechanism to calculate the attention distribution when the field embedding tensor is used as the query vector and the internal representation vector is used as the key-value pair, respectively. S33. Based on the computational attention distribution, feature mapping is performed, and the semantic similarity score between the field embedding tensor and each document position vector is calculated using the dot product similarity function. S34. Based on semantic similarity scores, the maximum score aggregation function is used to identify the most relevant index interval of each field in the document representation, and the text sequence within the interval is extracted as candidate field value fragments. S35. Perform type consistency verification on the candidate field value fragments, remove candidate fragments that are inconsistent with the field types in the Schema, output the field position set and the corresponding field value, and generate an intermediate dataset that satisfies the Schema constraints.

5. The method for extracting heterogeneous document structured data based on multimodal fusion according to claim 1, characterized in that, S4 specifically includes: S41. Input each field value in the intermediate dataset into a regular expression-based rule validation function to verify whether the field value conforms to the structural specification defined by the field type, and output the valid field label. S42. Combine each field value in the intermediate dataset with the field type label to form a field pair, input it into the field type discriminator composed of BiLSTM and Softmax classifiers, calculate the type prediction label corresponding to the field value, and compare the prediction label with the field type definition in the schema to mark the type consistency result. S43. Construct a graph structure based on the field values ​​and context field values ​​in the intermediate dataset. Each field value in the graph is a node, and the semantic relationship between fields is used as an edge. Input the graph structure into the graph convolutional network (GCN) model, calculate the node representation based on the semantic similarity of adjacent field values, perform consistency feature inference between fields, and identify contextual logical conflicts between fields. S44. Make a joint judgment based on the field's valid label, type consistency results, and contextual logic conflict results: If a field value has an abnormal format, it will be discarded. If a field is missing in the structural position, a K-nearest neighbor interpolation operation is performed based on semantically similar fields; If there is a contextual logic conflict between field pairs, the field value with lower confidence will be replaced based on the field confidence level. Output a structured set of fields with consistent structure and valid content.

6. The method for extracting heterogeneous document structured data based on multimodal fusion according to claim 5, characterized in that, S42 specifically includes: S421. Encode each field value using BERT to generate the corresponding contextual semantic embedding sequence, which serves as the initial input feature for the field representation. S422. Input the initial input features into a bidirectional long short-term memory network (BiLSTM), calculate the forward and reverse hidden states respectively, and concatenate them to obtain the field representation vector. S423. Input the field representation vector into the fully connected layer and connect it to the Softmax classifier. Perform probability distribution prediction in the predetermined field type label space and output the type prediction label of each field value and the corresponding probability distribution vector. S424. Calculate the maximum probability score of the predicted label based on the probability distribution vector, and match it with the type label of the corresponding field in the Schema. If the labels are consistent and the predicted probability is higher than the set confidence threshold, mark the field as consistent in type; otherwise, mark it as inconsistent in type.

7. The method for extracting heterogeneous document structured data based on multimodal fusion according to claim 1, characterized in that, S5 specifically includes: S51. Construct a structure mapping table based on the structured field set. The structure mapping table records field labels, field values, field hierarchical relationships and semantic block numbers corresponding to the fields, and sorts them in the order of semantic block numbers, serving as the structural basis for format conversion. S52. Determine the conversion strategy based on the output format type: When the output format is JSON, hierarchical parsing and key-value nesting are performed, and a hierarchical expansion algorithm is used to construct nested JSON objects based on the field hierarchy relationship in the structure mapping table. When the output format is Markdown, the header encoder function is called to construct the header row from the first-level fields, and the table body is generated using the row alignment algorithm. When the output format is CSV, perform header extraction and field alignment operations, and output a two-dimensional field matrix using a column vector concatenation strategy. When the output format is HTML, the DOM node generation function is called based on the semantic block number and field position index in the structure mapping table to construct an HTML node structure with semantic block identifiers; S53. Perform a structure verification operation on the format conversion result, and compare the field set in the output structure with the field label and position order in the structure mapping table to verify the consistency of the number, order and value content of the fields. When the output format is JSON, verify whether the nested objects are nested according to the field level and whether the field key names are consistent with the mapping table; When the output format is Markdown, verify whether the heading level is consistent with the field level, whether the table structure is complete, whether the field values ​​are correctly filled into the corresponding text blocks and table cells, and whether the paragraph order is consistent with the mapping table. When the output format is CSV, verify whether the number of field columns is fixed, whether the order of field columns is consistent with the order of the mapping table, and whether all records are aligned completely. When the output format is HTML, verify that all field values ​​are inserted into the corresponding semantic block container, that field tags constitute valid DOM nodes, and confirm that all fields are bound to unique position indexes. If all format and structure validations pass, output the data file in the target format.

8. The method for extracting heterogeneous document structured data based on multimodal fusion according to claim 1, characterized in that, S6 specifically includes: S61. Perform deployment environment identification operation, read the runtime environment variable configuration items, determine whether there is a high-performance graphics processing unit (GPU) and high-bandwidth memory access permission, and input the detection result into the condition judgment function. If the condition is met, generate a deployment environment label as a cloud environment; otherwise, generate a deployment environment label as a local environment. S62. When the deployment environment is marked as a cloud deployment environment, a large-scale multimodal model trained based on the LLM architecture is invoked. The large-scale multimodal model contains a Transformer structure with more than 7 billion parameters, receives the fused multimodal representation tensor, and outputs the structured probability distribution vector corresponding to the field labels. When the deployment environment is marked as a local deployment environment, a lightweight model that is jointly compressed by sparse pruning, parameter sharing and 8-bit integer quantization is invoked. The lightweight model is based on the TinyBERT backbone structure, extends the visual cross attention unit, receives the fused multimodal representation tensor, and outputs the field position index sequence and the field value sequence. S63. Perform a consistency alignment operation on the field prediction results output by the two types of models, call the field reordering algorithm based on the field order defined by the schema, combine the starting position index of the field in the original document, adjust the output field order to the schema order, and project the output field labels to the schema field label set through the label mapping function to generate field matching results.

9. A heterogeneous document structured data extraction system based on multimodal fusion, comprising executing the heterogeneous document structured data extraction method based on multimodal fusion as described in any one of claims 1 to 8, characterized in that, include: The document receiving module is used to receive documents in heterogeneous formats and perform image normalization, resolution unification, noise removal, and text region enhancement operations. The feature extraction module is used to extract visual and semantic features of the document in parallel. The visual features are composed of image feature tensors and spatial location embeddings generated by VisionEncoder, and the semantic features are composed of text sequences and character bounding boxes extracted by the Nanonets-OCR2 structure, and the fusion representation encoding is completed through the Transformer structure. The target parsing module receives user-defined field schemas, generates field-aware representations based on BERT encoding and Transformer cross-attention mechanism, performs semantic similarity calculation and field position recognition, and outputs candidate field value fragments. The field validation module is used to perform legality verification, field type discrimination and context logic consistency checks on candidate field values, construct graph structure representation and call graph convolutional network to perform inter-field consistency inference, and output a set of fields with valid structure. The format conversion module is used to convert a set of legally structured fields into structured file formats such as JSON, Markdown, CSV, and HTML, build a structure mapping table, and maintain the consistency between field hierarchy, semantic blocks, and table structure. The mode switching module is used to detect the deployment environment configuration, and based on the environment determination result, call the large-scale cloud model with LLM structure or the local lightweight model based on TinyBERT compression to perform field prediction inference, while maintaining the consistency of model output. The results docking module is used to perform field output reordering and tag mapping based on the schema field order and the document starting index order, and output a structured data file that can be used for subsequent system calls.

Images