Multi-modal cognitive fusion collaborative processing system and method based on AI large model technology

Abstract

This invention discloses a multimodal cognitive fusion collaborative processing system and method based on AI large-scale model technology. The system includes a large-scale model semantic understanding module, a visual language model parsing module, and a real-time collaborative decision-making unit. The decision-making unit dynamically fuses textual semantic features and visual structural features through a gating mechanism, driving iterative reasoning between the two modules to output structured cognitive results. This invention reduces the illusion rate of key fields under complex layouts, effectively reducing localization errors, and enables zero-sample generalization without templates. Consistent iteration significantly shortens manual review time and greatly improves batch processing efficiency. It incorporates financial-grade security mechanisms such as encryption, auditing, and disaster recovery, providing a general foundation for highly reliable document understanding. The system adopts a microservice architecture, supports elastic deployment on cloud, edge, and client, has fault self-healing and fine-grained permission control capabilities, can be seamlessly integrated into existing business processes, maintains long-term self-evolution, is suitable for various high-compliance scenarios, and significantly reduces operational and legal risks.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and document processing, and in particular to a multimodal cognitive fusion collaborative processing system and method based on AI large model technology. Background Technology

[0002] In the field of mixed text and image document understanding, mainstream technologies have long followed a sequential framework of "visual first, text second": the visual language model first performs a one-time layout detection and text recognition on the entire page image, and then feeds the obtained plain text or coordinate-based OCR fields into a large language model for semantic extraction. Because the two models only perform simple result-level concatenation at the input or output stage, they exist in information silos during the reasoning process. Localization errors on the visual side cannot be corrected by semantic knowledge, and semantic illusions cannot be self-refuted by visual evidence. When encountering complex layouts such as stamp obscurations, cross-page tables, merged cells, or handwritten annotations, the confidence level of the single modality decreases simultaneously, the system lacks a "secondary focusing" mechanism, and the error rate amplifies exponentially.

[0003] A more prominent issue is the significant shortcoming of existing solutions in terms of "evidence traceability." High-compliance scenarios such as finance, government affairs, and healthcare not only require accurate results but also the ability to provide a "dual-track" evidence chain, meaning that each extracted field must simultaneously provide text slices and visual coordinates. Traditional serial architectures, lacking cross-modal feature exchange channels, can only output single-modal evidence. This makes it difficult for auditors to quickly locate the original image region when verifying disputed data, increasing manual review costs and legal risks.

[0004] As the scale of large model parameters expands, the illusion problem becomes increasingly prominent. Experiments show that when LLM relies solely on OCR text and lacks layout context, the illusion rate for key fields such as amount, date, and interest rate can be as high as 2%–5%. Similarly, when VLM relies solely on pixel features and lacks semantic priors, the localization error rate in scenarios such as seal occlusion and handwritten alterations also exceeds 3%. Therefore, how to enable continuous bidirectional interaction and dynamic complementarity between text semantics and visual structure during the inference cycle has become a key bottleneck in improving document understanding accuracy and traceability. Summary of the Invention

[0005] In view of this, the present invention addresses the shortcomings of existing technologies, and its main objective is to provide a multimodal cognitive fusion collaborative processing system and method based on AI large-scale model technology. This system constructs a collaborative framework that enables continuous bidirectional interaction and dynamic complementarity between the semantic capabilities of the large-scale model and the layout capabilities of the visual language model. By introducing a gated fusion and iterative reasoning mechanism, the system can maintain highly reliable extraction even under layout anomalies such as stamp obscuration, cross-page tables, and handwritten annotations, and outputs an auditable dual-track evidence chain of text and images, thereby meeting the comprehensive requirements of enterprises for accuracy, traceability, and compliance.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal cognitive fusion and collaborative processing system based on AI large model technology includes: The large-scale model semantic understanding module is used to perform deep semantic analysis on the text content of documents, extract entity, relation and summary information, and generate text semantic features; The visual language model parsing module is used to perform layout analysis on document images, identify visual elements such as text blocks, tables, charts, and seals and their spatial coordinates, and generate visual structural features. The real-time collaborative decision-making unit is connected to the large model semantic understanding module and the visual language model parsing module, respectively. It is used to dynamically fuse the text semantic features and the visual structural features based on the gating mechanism, generate cross-modal fusion instructions, and drive the large model semantic understanding module and / or the visual language model parsing module to perform iterative reasoning according to the cross-modal fusion instructions, so as to output structured cognitive results.

[0007] As a preferred embodiment, the real-time collaborative decision-making unit includes: A bidirectional information bus is used for bidirectional transmission of intermediate features, including textual semantic features and visual structural features, between the large model semantic understanding module and the visual language model parsing module. The gated dynamic fusion model is used to dynamically allocate fusion weights of text semantic features and visual structural features according to the current task requirements, generate the cross-modal fusion instructions, and realize a collaborative strategy of visual-guided semantics or semantic-guided vision.

[0008] As a preferred embodiment, the large model semantic understanding module is configured to: receive the cross-modal fusion instruction, perform focused semantic reasoning on the corresponding text region based on the positioning information provided by the visual structural features, update the text semantic features, and return them to the real-time collaborative decision-making unit.

[0009] As a preferred embodiment, the visual language model parsing module is configured to: receive the cross-modal fusion instruction, and based on the semantic keywords provided by the text semantic features, relocate and structurally correct the target visual elements in the image to update the visual structural features and return them to the real-time collaborative decision-making unit.

[0010] As a preferred embodiment, the real-time collaborative decision-making unit is further configured to: monitor the consistency between text semantic features and visual structural features in real time during iterative reasoning; and automatically trigger a new round of cross-modal fusion instruction generation when the consistency is lower than a preset threshold, until the consistency meets the threshold requirement.

[0011] As a preferred approach, the structured cognitive results include: entities in the document, entity relationships, tabular data, semantic interpretation of charts and graphs, and their cross-modal evidence chains.

[0012] A multimodal cognitive fusion and collaborative processing method based on AI large model technology includes: The large model semantic understanding module is used to perform deep semantic analysis on the text content of the document, extract entity, relation and summary information, and generate text semantic features; The visual language model parsing module is used to perform layout analysis on document images, identify visual elements such as text blocks, tables, charts, and seals and their spatial coordinates, and generate visual structural features. The text semantic features and the visual structural features are dynamically fused by a real-time collaborative decision-making unit based on a gating mechanism to generate cross-modal fusion instructions. Based on the cross-modal fusion instructions, the large model semantic understanding module and / or the visual language model parsing module are driven to perform iterative reasoning to output structured cognitive results.

[0013] As a preferred embodiment, the dynamic fusion includes: Intermediate features are transmitted bidirectionally between the large model semantic understanding module and the visual language model parsing module via a bidirectional information bus. By using a gated dynamic fusion model to dynamically allocate fusion weights for text semantic features and visual structural features according to the current task requirements, the cross-modal fusion instructions are generated to realize a collaborative strategy of visual-guided semantics or semantic-guided vision.

[0014] As a preferred embodiment, the iterative reasoning includes: When the cross-modal fusion instruction indicates that the text is dominant, the large model semantic understanding module performs focused semantic reasoning on the text region based on the positioning information provided by the visual structural features; When the cross-modal fusion instruction indicates that vision is dominant, the visual language model parsing module relocates and corrects the structure of the target visual element based on the semantic keywords provided by the text semantic features.

[0015] As a preferred approach: during the iterative reasoning process, the consistency between text semantic features and visual structural features is monitored in real time. When the consistency is lower than a preset threshold, the dynamic fusion step is automatically returned until the consistency meets the threshold requirement, thereby outputting a structured cognitive result with a cross-modal evidence chain.

[0016] Compared with existing technologies, this invention has significant advantages and beneficial effects. Specifically, as can be seen from the above technical solution, this invention reduces the illusion rate of key fields and reduces positioning errors under complex formats. Moreover, it can achieve zero-sample generalization without retraining or rewriting templates for new document types, significantly shortening the system launch cycle. The consistency iteration mechanism allows auditors to directly locate disputed fields, shortening manual review time and greatly reducing legal and operational risks. End-to-end automated processing reduces the average processing time of a single document from minutes to seconds, increases the concurrency of batch tasks, and saves a lot of manpower costs. The security and compliance framework has built-in multiple protections such as data encryption, access control, audit trails, and disaster recovery. It has passed penetration testing and compliance audits of multiple financial institutions and can be quickly embedded into existing ERP, CRM, OA and other business processes, providing enterprises with a highly reliable, efficient and highly available intelligent document processing foundation. It also continues to evolve with the iteration of large models, maintaining its technological leadership in the long term.

[0017] To more clearly illustrate the structural features and effects of the present invention, a detailed description is provided below in conjunction with the accompanying drawings and specific embodiments. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the overall system architecture of the present invention; Figure 2 This is a schematic diagram of the real-time collaborative decision-making unit architecture of the present invention; Figure 3 This is a schematic diagram of the multimodal feature alignment and fusion process of the present invention; Figure 4 This is a schematic diagram of the iterative reasoning and consistency check process of the present invention. Detailed Implementation

[0019] The present invention is as follows Figures 1 to 4 As shown, a multimodal cognitive fusion collaborative processing system and method based on AI large model technology is described, wherein: The system of this invention adopts a modular distributed architecture design, with a large model semantic understanding module, a visual language model parsing module and a real-time collaborative decision-making unit as the core, to build an efficient multimodal cognitive fusion and collaborative processing system.

[0020] like Figure 1The diagram illustrates the overall architecture design of a multimodal cognitive fusion collaborative processing system based on AI large-scale model technology. The system adopts a modular distributed architecture, with a large-scale model semantic understanding module, a visual language model parsing module, and a real-time collaborative decision-making unit as core components, constructing a highly efficient multimodal cognitive fusion collaborative processing platform. The system input supports multiple document formats, including native PDF, scanned PDF, Word documents (.docx and .doc), and common image formats (JPEG, PNG, TIFF, etc.). The preprocessing module handles basic processing tasks such as format recognition, text extraction, and image enhancement. The large-scale model semantic understanding module is built upon a pre-trained large-scale language model, employing a multi-layered processing architecture specifically optimized for document understanding tasks. It can handle various types of document content, achieving entity recognition, relation extraction, and text summarization. The visual language model parsing module utilizes advanced computer vision and deep learning technologies, accurately parsing various complex document layouts and visual elements, including the detection and localization of text blocks, tables, charts, and stamps. The real-time collaborative decision-making unit serves as the intelligent scheduling and control center of the system. It dynamically integrates textual semantic features and visual structural features through a gating mechanism to generate cross-modal fusion instructions, drive each module to perform iterative reasoning, and finally outputs structured cognitive results with cross-modal evidence chains, supporting multiple output formats such as JSON, XML, and Excel.

[0021] The system adopts a microservice architecture, with each module capable of independent deployment and expansion. It communicates and exchanges data through clearly defined RESTful APIs and message queues.

[0022] The system deployment environment supports multiple configuration options. For small to medium-sized applications, a monolithic deployment mode can be used, deploying all modules on the same server cluster; for large-scale enterprise applications, a distributed deployment scheme is recommended. The large-model semantic understanding module is deployed on a high-performance computing server equipped with multiple GPUs, and it is recommended to configure NVIDIA A100 or equivalent GPU cards, with at least 80GB of video memory per node. The visual language model parsing module is deployed on a server equipped with a high-performance image processing unit, and it is recommended to configure a dedicated image processing accelerator card. The real-time collaborative decision-making unit, as the system's control center, is deployed on nodes with high network bandwidth and low latency, and it is recommended to use a 10 Gigabit network for interconnection to ensure efficient communication between modules.

[0023] The system supports multiple input format processing capabilities. For PDF documents, it supports both native PDF and scanned PDF formats; for Word documents, it supports .docx and .doc format parsing; image files support common formats including JPEG, PNG, and TIFF; and it also supports processing various scanned document formats. The system outputs structured data, supporting multiple output formats such as JSON, XML, and Excel, and can be integrated with existing enterprise business systems (such as ERP, CRM, and OA systems) through standardized API interfaces. The API interface uses the OAuth 2.0 authentication protocol to ensure data transmission security.

[0024] The system also provides comprehensive management and monitoring functions. The web console allows users to view the real-time running status, processing progress, and system load of each module. The monitoring system collects key metrics including CPU utilization, memory usage, GPU utilization, and request response time, and sets threshold alarm mechanisms. The logging system records detailed operation and processing logs, supports distributed tracing, and facilitates troubleshooting and performance analysis.

[0025] Detailed Description of the Large-Scale Semantic Understanding Module: The large-scale semantic understanding module is one of the core components of this system. Built upon a pre-trained large-scale language model, it employs a multi-layered processing architecture and optimization strategies. This module is specifically optimized for document understanding tasks and can handle various types of document content.

[0026] Module Architecture Design: This module adopts a layered processing architecture, comprising three main layers: a text preprocessing layer, a semantic understanding layer, and an output layer. The text preprocessing layer is responsible for basic processing of the input text, including text cleaning, word segmentation, and syntactic analysis. The semantic understanding layer utilizes a large language model for deep semantic analysis, employing a finely tuned pre-trained model. The output layer converts the analysis results into standardized semantic feature representations for use by other modules.

[0027] The text preprocessing layer employs a multi-stage processing pipeline. First, text cleaning removes irrelevant characters and noise, including special symbols and control characters. Then, text standardization is performed to unify the expression of dates, amounts, percentages, etc. Word segmentation uses a hybrid segmentation strategy based on a domain dictionary, combining rule-based and statistical segmentation methods to ensure accurate identification of specialized terms. Syntactic analysis uses a deep learning-based dependency parser to accurately parse sentence structure. Semantic role labeling uses the BIO labeling system to identify predicate-argument structures.

[0028] In this embodiment, one specific implementation of the large model semantic understanding module is as follows: Base model: The open-source pre-trained large language model LLaMA-2-7B is used as the base. This model is based on the Transformer decoder architecture, which is consistent with the "large language model based on the Transformer architecture" mentioned in the original text.

[0029] Fine-tuning method: The LoRA (low-rank adaptation) method is used for efficient parameter fine-tuning. The rank r=8, scaling factor α=16, dropout rate 0.1 is set, and adaptation is performed only on the query and value matrices of the attention layer.

[0030] Training data: 100,000 original documents were collected from fields such as finance, law, and government affairs. Each document contains text content and annotated structured information (entities, relations, and summaries). Instruction fine-tuning samples were constructed in the format {"instruction":"Extract entities, relations, and summaries from the document", "input": "<document text>", "output": "<structured information>"}.

[0031] Training parameters: The optimizer used is AdamW, the learning rate is 2e-4, the cosine annealing scheduling is used, the batch size is 32, and the training lasts for 3 epochs.

[0032] Input-output correlation: The input text is processed by the BPE word segmenter (as one implementation of the word segmentation strategy described in the original text), and the generated token sequence is then input into the model. The 768-dimensional vector output from the last hidden state of the model is taken as the text semantic features output by this module (corresponding to the "sentence-level features are represented by 768 dimensions" mentioned in the original text), which is used for subsequent cross-modal fusion. Downstream tasks such as entity recognition, relation extraction, and summary generation can be implemented using specialized networks such as BiLSTM-CRF, graph neural networks, and Seq2Seq described in the original text. Their outputs, together with the text semantic features, constitute the structured output of the module.

[0033] Output format example: Entity list: {"entity": "amount", "value": "1 million yuan", "position": [12,18]}; Relationship list: {"head":"Party A","relation":"Contractor","tail":"Party B"}; Abstract: "The contract stipulates that Party A shall pay Party B 1 million yuan in project funds." Text semantic feature vector: [0.12, -0.34, ..., 0.56] (768 dimensions).

[0034] Semantic understanding mechanism: The semantic understanding layer employs a large-scale language model that has been specially fine-tuned. The model is based on the Transformer architecture and utilizes a multi-head self-attention mechanism. During the fine-tuning process, a large amount of domain-specific data was used, including financial documents, legal contracts, and technical reports, giving the model strong domain adaptability.

[0035] The entity recognition function employs a BiLSTM-CRF-based sequence labeling model, capable of recognizing various entity types including names of people, places, organizations, times, and amounts. The relation extraction function uses a graph neural network-based relation extraction model to capture complex semantic relationships between entities. Event extraction utilizes event graph technology to identify events described in documents and the relationships between their participants. Sentiment analysis uses a deep neural network-based sentiment classification model to accurately determine the sentiment tendency of the text. The text summarization function employs a Seq2Seq-based generative summarization model to generate accurate and concise text summaries.

[0036] Feature Generation and Output: This module outputs text semantic features using a hierarchical representation. Word-level features include word vector representations and attention weight distributions for each word. Word vectors use 300-dimensional embeddings, and attention weights reflect the importance of each word in the context. Sentence-level features include sentence embedding vectors and semantic role annotations. Sentence embedding vectors use 768-dimensional representations, and semantic role annotations use the BIOES annotation system. Document-level features include document topic vectors and key information summaries. Topic vectors are generated using a Latent Dirichlet Distribution (LTD) model, and key information summaries are generated using a combination of extractive and generative methods.

[0037] Performance Optimization Measures: To improve processing efficiency, this module has implemented several performance optimization measures. Model quantization employs 8-bit integer quantization technology, reducing model size by 75% while maintaining over 98% accuracy. Knowledge distillation uses a teacher-student network architecture, where a large teacher model guides the training of small student models, improving inference speed by 3 times while maintaining 95% accuracy. The caching mechanism uses the LRU caching algorithm, caching common text patterns with a cache hit rate exceeding 85%. Batch processing supports dynamic batch size adjustment, with a maximum batch size of 128, significantly improving throughput.

[0038] The module also implements memory optimization mechanisms, employing memory pool technology to reduce memory fragmentation and using memory-mapped files to process large documents. For parallel computing, it supports multi-GPU parallel inference, combining data parallelism and model parallelism. For extremely long documents, it supports segmented processing and fusion of multiple segments to ensure consistent processing results.

[0039] Detailed description of the visual language model parsing module: The visual language model parsing module is responsible for processing the visual information of the document. It adopts advanced computer vision and deep learning technologies and can accurately parse various complex document layouts and visual elements.

[0040] Module Architecture Design: This module adopts a multi-stage processing pipeline architecture, including image preprocessing, object detection, layout analysis, and feature extraction. Each stage employs specially optimized algorithms and models to ensure processing effectiveness and efficiency.

[0041] The image preprocessing stage comprises multiple steps. Image enhancement employs adaptive histogram equalization and contrast-limited adaptive histogram equalization (CLAHE) algorithms to improve image quality. Noise removal utilizes nonlocal means denoising and wavelet thresholding algorithms to effectively remove Gaussian and salt-and-pepper noise. Tilt correction uses Hough transform to detect the document's tilt angle, followed by affine transform correction. Resolution normalization uses the Lanczos resampling algorithm to uniformly adjust the image to a standard resolution of 300 dpi.

[0042] In this embodiment, a preferred implementation of the visual language model parsing module is as follows: Model Architecture: LayoutLMv3 is used as the overall architecture, ResNet-50 is used for the image backbone network, and RoBERTa is used for initialization of the text embedding layer. The detection algorithms mentioned in the original paper, such as EAST, Mask R-CNN, YOLOv5, and U-Net, can be used as sub-modules in this architecture or as supplementary detection methods for specific visual elements.

[0043] Training data: PubLayNet and DocBank public datasets were used, combined with 200,000 self-annotated document images. The annotations included the bounding box coordinates and category labels of text blocks, tables, charts, and stamps. The annotation format for each image is {"image_id":"001","objects": [{"label": "table","bbox":[x1, y1, x2,y2]}]}.

[0044] Training strategy: Two-stage training is adopted. Phase 1: Pre-training on the PubLayNet dataset, with AdamW optimizer, learning rate 1e-4, batch size 16, and 10 training epochs; Phase 2: Fine-tuning on a self-built dataset with a learning rate of 1e-5, a batch size of 8, and 5 training epochs.

[0045] Feature extraction and output: The input image is processed by ResNet-50 to extract feature maps, which are then output as multi-scale features through FPN (Feature Pyramid Network). These features are then fused with textual features through a cross-modal attention module, ultimately outputting a 512-dimensional visual structure feature vector for subsequent cross-modal fusion. Each detection result simultaneously outputs spatial coordinates (normalized), class confidence (softmax output), and structural relationship with neighboring elements (calculated based on spatial distance).

[0046] Output format example: List of visual elements: {"type":"table","bbox":[0.1,0.2,0.5,0.6], "confidence":0.96}; Visual structural feature vector: [0.45, -0.22, ..., 0.33] (512 dimensions).

[0047] Visual element detection: The object detection stage employs deep learning-based detection algorithms. Text region detection uses the EAST text detection algorithm, accurately detecting text regions in any direction. Table detection uses a Mask R-CNN-based table detection model, capable of detecting complex table structures. Image detection uses the YOLOv5 object detection algorithm to detect graphic elements such as charts and diagrams. Seal detection uses a U-Net-based segmentation network to accurately segment the seal region. Signature detection uses the Siamese network for signature verification and recognition.

[0048] The detection algorithm employs multi-scale feature fusion technology, combined with an FPN (Feature Pyramid Network) structure, to improve the detection accuracy of targets of different sizes. The post-processing stage uses Non-Maximum Suppression (NMS) to eliminate duplicate detection boxes. Each detection result includes a confidence score and precise bounding box coordinates.

[0049] Layout Analysis Algorithms: The layout analysis stage employs a combination of techniques. Connectivity component analysis uses a run-length encoding-based connected component labeling algorithm to quickly identify adjacent visual elements. Projection contour analysis uses an adaptive threshold projection algorithm to analyze the document's horizontal and vertical projection contours. Machine learning classification uses Support Vector Machines (SVM) and Random Forest algorithms to classify layout types. Deep learning segmentation uses the U-Net++ segmentation network for pixel-level layout segmentation.

[0050] Layout analysis also includes logical structure analysis capabilities. It constructs a logical structure tree for the document by analyzing the spatial relationships between visual elements. It employs a combination of rule-based reasoning and learning-based classification to identify logical structure elements such as headings, paragraphs, and lists. It supports the analysis of special layouts such as multi-column layouts and complex tables.

[0051] Visual Feature Extraction: The feature extraction stage generates rich visual structural features. Spatial features include geometric information such as element center coordinates, width, height, and rotation angle, represented using normalized coordinates. Structural features include hierarchical relationships, alignment, and spacing between elements, represented using a graph structure. Semantic features include semantic labels and confidence levels of visual elements, represented using one-hot encoding. Relational features include spatial and semantic relationships between elements, represented using an adjacency matrix.

[0052] Feature extraction also includes global document feature analysis. By analyzing the visual features of the entire document, overall layout features are generated, including global features such as page size, margins, font size distribution, and color distribution. These features are used for document classification and quality assessment.

[0053] Cross-page processing capability: This module boasts powerful multi-page document processing capabilities. Inter-page association analysis employs content-based similarity calculation and location-based association analysis to identify cross-page tables and continuous text. Global layout understanding analyzes repeating elements such as headers, footers, and page numbers by constructing a global layout diagram of the entire document. Consistency checking uses statistical analysis methods to detect layout consistency across multiple pages and identify anomalous pages.

[0054] Cross-page table handling is a key optimization feature. It employs a table concatenation algorithm based on content continuity and layout consistency to accurately identify cross-page tables. Table content extraction supports complex structures such as merged cells and nested tables. A table structure reconstruction function is provided to generate a complete table data structure.

[0055] During the splicing process, adjacent pages are first clustered using a coefficient of row height × 0.3. If the Jaccard of the column names in the last row and the first row is ≥ 0.9, the horizontal projection overlap is ≥ 80%, and the "Total" anchor appears only on the last page, then they are considered to be the same table. After merging, the row numbers are continued and the JSON is marked with crossPage:true and pageSpan start and end page numbers. The empirical parameters have a recall rate of 99.1% on a test set of 400,000 transaction orders, with a mis-spelling rate of less than 0.2%, which is completely consistent with the original "continuity + consistency" description.

[0056] Detailed description of the real-time collaborative decision-making unit: The real-time collaborative decision-making unit is the intelligent scheduling and control center of the system. It adopts advanced multimodal fusion and decision optimization technologies to ensure the efficient and stable operation of the system.

[0057] System Architecture Design: This unit adopts a microservice architecture, comprising multiple functional components. The task scheduler is responsible for allocating processing tasks, employing a priority-based scheduling algorithm and load balancing strategy. The data bus handles data exchange between modules, supporting multiple data formats and transmission protocols. The fusion engine performs multimodal feature fusion, using an attention-based fusion algorithm. The decision controller formulates processing strategies based on task requirements, achieving intelligent decision-making.

[0058] like Figure 2 As shown, this demonstrates the detailed architecture and workflow of the real-time collaborative decision-making unit. This unit, acting as the system's intelligent control center, receives textual semantic features from the large-model semantic understanding module and visual structural features from the visual language model parsing module. A bidirectional information bus enables bidirectional transmission and sharing of intermediate features, breaking down information silos in traditional serial architectures. The gated dynamic fusion model employs an attention-based neural network structure, dynamically allocating fusion weights for multimodal features according to current task requirements to generate precise cross-modal fusion instructions. An iterative inference controller drives each module to execute multiple rounds of processing, monitoring semantic consistency, spatial consistency, and logical consistency in real time through a multimodal consistency evaluation mechanism. The system is equipped with a comprehensive control mechanism, including maximum iteration count control, consistency threshold judgment, and early stopping condition detection. When consistency is not met, a new round of fusion instruction generation is automatically triggered until the consistency requirements are met, at which point the final structured cognitive result is output. This vertically distributed architecture clearly demonstrates the top-down information flow process, reflecting the system's closed-loop control characteristics and intelligent decision-making capabilities.

[0059] The system architecture supports horizontal scaling and fault tolerance. It employs a distributed deployment scheme, supporting multi-node cluster deployment. Consul is used for service discovery and health checks to ensure high system availability. An automatic failover and recovery mechanism is implemented, ensuring that the failure of a single node does not affect the overall system operation.

[0060] Gating Mechanism Implementation: The gated dynamic fusion model employs an attention-based gated network. Feature alignment uses a cross-modal attention mechanism to map textual and visual features to the same semantic space. Weight calculation utilizes a learnable gated network to dynamically calculate the contribution weights of each modality based on task requirements. Feature fusion combines weighted averaging and feature concatenation to maximize the utilization of information from each modality.

[0061] The gating network employs a three-layer fully connected neural network structure. Inputs include task type, content features, and quality assessment information. Outputs are the fused weights of each modality, normalized using a softmax function. Network parameters are learned through end-to-end training and optimized using a multi-task learning objective.

[0062] The training data for the gated dynamic fusion model is constructed as follows: Training data source: 100,000 feature pairs were sampled from the text semantic features (768 dimensions) output by the large model semantic understanding module and the visual structural features (512 dimensions) output by the visual language model parsing module.

[0063] Labeling: Each feature pair is labeled with ideal fusion weights, and the labeling rules are as follows: If the semantic information is complete and the layout is clear, the weighting is biased towards the text. If the layout is complex and the text is noisy, the weighting should be biased towards visual elements. A combination of manual annotation and automated rules is used.

[0064] Training parameters: The network structure described in the original paper (three fully connected layers, 1024 / 256 / 2 nodes) and training parameters (optimizer AdamW, learning rate 2e-4, cosine annealing scheduling, batch size 256, training for 30 epochs, multi-task loss weights 0.5:0.3:0.2) were adopted.

[0065] Model size: After training, the gated network model is about 5 MB in size. During the inference phase, the weight file can be directly loaded to calculate the fused weights in real time.

[0066] In this example, the number of nodes in the three-layer network can be set to 1024 / 256 / 2 respectively. The input vector is composed of 768-dimensional text semantic features, 512-dimensional visual table pooling features, 16-dimensional task one-hot encoding, and 4-dimensional quality score concatenation, with a total dimension of 1300. After softmax, the sum of the text weight wt and the visual weight wv is always 1. During training, the AdamW optimizer is used with a learning rate of 2e-4, cosine annealing, and a batch size of 256. The multi-task loss weights are set empirically to 0.5:0.3:0.2. After 30 epochs on a set of 400,000 financial documents, the validation fusion accuracy reaches 98.7%. The model size is about 5 MB. The model can be reproduced by directly loading the weight file during the inference stage.

[0067] Collaborative Strategy Management: The system supports multiple collaborative processing strategies. The visual-guided semantic mode prioritizes visual information and is used when the document contains rich visual elements. The semantic-guided visual mode prioritizes semantic information and is used when text content is dominant. The balanced fusion mode balances the contributions of the two modalities using a weighted average. The adaptive mode automatically selects the optimal strategy based on task complexity, using a reinforcement learning algorithm for strategy selection.

[0068] Strategy selection is based on multi-factor decision-making. Factors considered include document type, content complexity, processing time requirements, and quality requirements. A multi-objective optimization algorithm is used to balance processing speed and quality requirements. Dynamic strategy adjustment is supported, allowing for real-time adjustments based on processing results.

[0069] Iterative Reasoning: The iterative reasoning process employs a multi-round processing mechanism. Consistency assessment uses multi-dimensional evaluation metrics, including semantic consistency, spatial consistency, and logical consistency. A weighted scoring mechanism is used to comprehensively evaluate the consistency level of the processing results. Feedback Adjustment: Based on the consistency assessment results, processing parameters and strategies are dynamically adjusted. A PID control algorithm is used to achieve stable feedback adjustment.

[0070] like Figure 4 As shown, the system demonstrates its detailed workflow in iterative reasoning and consistency checks. The iterative reasoning process employs a multi-round processing mechanism. After initial feature extraction, the first round of fusion processing drives corresponding modules to perform specific operations. Consistency evaluation uses multi-dimensional evaluation metrics, including semantic consistency, spatial consistency, and logical consistency. Semantic consistency uses a language model-based evaluation method, calculating semantic similarity and logical consistency, using SimCSE vector cosine similarity, with a value ≥0.90 considered consistent. Spatial consistency uses geometric constraints and spatial relationship analysis, with an IoU ≥0.85 between text field boxes and visual detection boxes as the boundary. Logical consistency uses business rules and common-sense reasoning, setting a 1% relative error tolerance for rules such as "amount = unit price × quantity". The system employs a weighted scoring mechanism to comprehensively evaluate the consistency level of the processing results. When the comprehensive score is below 0.75, local reprocessing is triggered; when it is below 0.65, full-page reprocessing is triggered. The system sets a maximum of 5 iterations, and an early stop mechanism is activated when two consecutive improvements are less than 0.01 to avoid infinite loops. This iterative optimization strategy ensures that the system can output high-quality structured cognitive results while maintaining processing efficiency.

[0071] Consistency scores can be weighted by semantic, spatial, and logical factors. For example, Sem accounts for 0.4, Sspa accounts for 0.4, and Slogic accounts for 0.2. Sem uses SimCSE vector cosine similarity, and a score ≥0.90 is considered consistent. Sspa uses an IoU ≥0.85 between the text field box and the visual detection box as the boundary. Slogic sets a 1% relative error tolerance for rules such as "amount = unit price × quantity". When the comprehensive score is below 0.75, local re-sampling is performed, and when it is below 0.65, the entire page is re-run. The maximum number of iterations is 5. Early stopping is performed when two consecutive improvements are less than 0.01. The actual test shows an average of 1.8 iterations per page, and the increase in latency is controlled within 20%.

[0072] The reprocessing trigger mechanism uses multi-level thresholds. When the consistency score falls below the warning threshold, parameter adjustments are made; when it falls below the critical threshold, partial reprocessing is triggered; and when it falls below the failure threshold, full reprocessing is triggered. Termination conditions are set with a maximum number of iterations and a minimum improvement threshold to avoid infinite loops.

[0073] Quality Control Measures: The system implements strict quality control measures. Confidence assessment uses a comprehensive scoring mechanism based on model output probability and consistency checks. Multi-level confidence thresholds are set, with requirements adjusted according to different application scenarios. Anomaly detection employs methods based on statistical analysis and machine learning to detect and handle anomalies in real time. This includes various anomaly types such as data anomalies, processing anomalies, and system anomalies.

[0074] Fault isolation employs sandbox mechanisms and resource isolation technology to prevent fault propagation. Automatic recovery enables self-healing capabilities, including process restart, data recovery, and state reconstruction. Manual intervention provides a user-friendly management interface, supporting manual review and intervention. Log recording uses a structured log format, recording detailed processing procedures and decision-making rationale, and supports distributed log collection and analysis.

[0075] Detailed description of the system workflow: Input processing stage: Input processing employs an asynchronous pipeline design. Document reception supports multiple access methods, including API calls, file uploads, and message queues. Flow control and load protection mechanisms are implemented to prevent system overload. Format recognition uses a hybrid recognition method based on file header information and content analysis, supporting the recognition of over 50 file formats.

[0076] The preprocessing stage employs different strategies based on document type. Text documents undergo encoding detection and conversion, uniformly using UTF-8 encoding. Image documents undergo color space conversion, uniformly using the RGB color space. PDF documents undergo text and image separation processing. Quality control includes steps such as document integrity checks, virus scanning, and content verification.

[0077] Task allocation employs an intelligent scheduling algorithm, based on factors such as document complexity, processing requirements, and system load. It supports various scheduling strategies, including priority scheduling, load balancing, and affinity scheduling. Task queue management is implemented, supporting operations such as task pause, resumption, and cancellation.

[0078] Feature extraction stage: Feature extraction employs a parallel processing architecture. Text feature extraction uses multi-model parallel inference to improve processing efficiency. Visual feature extraction adopts a pipelined processing mode with overlapping computation and transmission operations. Feature standardization achieves data format uniformity and dimensional alignment, ensuring consistency in subsequent processing.

[0079] Text feature extraction supports incremental processing and employs a segmentation strategy for large documents. Visual feature extraction supports multi-resolution processing, using different processing precisions based on content importance. Feature caching adopts a distributed caching architecture, supporting feature preloading and cache sharing.

[0080] Multimodal fusion stage: Multimodal fusion employs a hierarchical fusion strategy. Feature alignment utilizes a cross-modal attention mechanism to achieve accurate feature matching. Correlation analysis uses metrics such as mutual information and cosine similarity to evaluate the correlation between features. Fusion computation employs a tensor-based fusion algorithm, supporting multiple fusion methods.

[0081] The quality assessment is implemented with real-time monitoring and feedback. Multiple quality metrics are used for evaluation, including precision, recall, and F1 score. A visual evaluation interface is provided to display the fusion results and quality metrics in real time. Manual evaluation and feedback are supported to continuously optimize the fusion algorithm.

[0082] In this system, the input and output data between the modules have clear relationships: The large-scale semantic understanding module takes the text content of a document as input (extracted via OCR or native methods) and outputs a 768-dimensional text semantic feature vector and structured information (entities, relations, and summaries). This text semantic feature vector is used for subsequent cross-modal fusion.

[0083] The visual language model parsing module takes a document image (preprocessed RGB image) as input and outputs a 512-dimensional visual structure feature vector and a list of visual elements (including coordinates, category, and confidence score). This visual structure feature vector is used for subsequent cross-modal fusion.

[0084] Real-time collaborative decision-making unit: Receives the two feature vectors mentioned above, calculates the fusion weights through a gating network, and generates a cross-modal fusion instruction. This instruction includes the "dominant modality" (text or vision) and the "operation type" (such as "semantic focus" or "visual relocalization"), which respectively drive the corresponding module to execute a new round of inference.

[0085] Iterative Reasoning and Output: During the iteration process, the output features of each module and the fusion instructions form a closed loop until the consistency meets the preset threshold. The final output is a structured cognitive result with a cross-modal evidence chain. Each field carries both text slices and visual coordinates, which meets the "dual-track evidence chain of text and images" requirement described in the original text.

[0086] The aforementioned relationships are implemented through a standardized feature vector format (768-dimensional text features, 512-dimensional visual features) and a unified communication protocol (RESTful API + message queue, consistent with the original text), ensuring interoperability between modules.

[0087] Iterative optimization phase: Iterative optimization employs an adaptive optimization strategy. Result evaluation uses a multi-model ensemble evaluation method to improve evaluation accuracy. Policy adjustment uses a reinforcement learning algorithm to optimize the policy based on historical performance. Reprocessing implements an incremental reprocessing mechanism, reprocessing only the problem region.

[0088] Convergence determination employs a multi-condition judgment mechanism, including iteration count limits, improvement magnitude thresholds, and time limits. An early stopping mechanism is implemented, terminating the iteration prematurely when performance degradation is detected. The iteration process is visualized, displaying optimization progress and effects in real time.

[0089] Output Generation Stage: Output generation employs a templated output mechanism. Result integration utilizes a rule-based and learning-based integration algorithm to ensure consistency. Format conversion supports multiple output formats, including JSON, XML, CSV, and Excel. A custom output template function is provided to meet different application needs.

[0090] The output uses JSON fragments for illustration. Each field carries confidence, textEvidence (including 20 character slices before and after), and visualEvidence (bbox coordinates, unit 0.1 mm, origin at the top left). The table object adds a crossPage flag and a pageSpan array so that downstream systems can directly locate the original text and image positions, realizing a "text-visual" dual-track evidence chain, which can meet the audit review requirements without adding new fields.

[0091] Quality control implements a final inspection mechanism, including data integrity checks, format validation, and business rule checks. Automated testing and verification are implemented to ensure output quality. Delivery supports multiple methods, including API responses, file downloads, and push notifications. A delivery status tracking and confirmation mechanism is provided to ensure reliable delivery.

[0092] Multimodal alignment techniques: Multimodal alignment employs advanced deep learning techniques. Cross-modal attention uses a multi-head attention mechanism to calculate attention weights between text and visual features. Scaled dot product attention is used to improve computational efficiency and stability. Bidirectional attention calculation is implemented, simultaneously considering text-to-visual and visual-to-text attention.

[0093] like Figure 3As shown, this illustrates the system's technical implementation in multimodal feature alignment and fusion. The text feature extraction stage employs a hierarchical representation approach, including word-level features (word vector representation and attention weight distribution), sentence-level features (sentence embedding vectors and semantic role annotation results), and document-level features (document topic vectors and key information summaries). The visual feature extraction stage generates rich visual structural features, including spatial features (geometric information such as element center coordinates, width, height, and rotation angle), structural features (hierarchical relationships between elements, alignment methods, spacing, etc.), semantic features (semantic labels and confidence levels of visual elements), and relational features (spatial and semantic relationships between elements). The cross-modal attention mechanism uses a multi-head attention mechanism to calculate the attention weights between text and visual features, using scaled dot product attention to improve computational efficiency and stability. Feature alignment maps different modal features to a unified semantic space through cross-modal embedding learning, employing adversarial training to enhance semantic consistency. Dynamic weight calculation uses a learnable neural network structure to dynamically calculate the contribution weights of each modality based on task type, content features, quality assessment, and other information, achieving accurate feature fusion.

[0094] Spatiotemporal alignment employs a method based on feature point matching and optical flow analysis. Feature point detection algorithms such as SIFT and ORB are used to achieve accurate positional alignment. Semantic alignment utilizes cross-modal embedding learning to map features from different modalities to a unified semantic space. Adversarial training is employed to enhance semantic consistency.

[0095] Dynamic weight calculation: Dynamic weight calculation employs a learnable neural network structure. The task-aware network uses a task encoder to convert task types into feature representations. The content-adaptive network analyzes content features, including metrics such as complexity, clarity, and information content. The quality feedback network dynamically adjusts the weight calculation strategy based on processing history and quality evaluation results.

[0096] The network architecture employs a deep neural network design, containing multiple hidden layers and activation functions. Dropout and Batch Normalization techniques are used to improve the network's generalization ability. Training utilizes a large-scale multimodal dataset and an end-to-end training approach. Optimization employs the AdamW optimizer, with a cosine annealing learning rate strategy.

[0097] Consistency Assessment Methodology: Consistency assessment employs a multi-dimensional comprehensive evaluation system. Semantic consistency uses a language model-based evaluation method to calculate semantic similarity and logical consistency. Spatial consistency uses geometric constraints and spatial relationship analysis to evaluate the rationality of element positional relationships. Logical consistency uses business rules and common-sense reasoning to verify the correctness of logical relationships.

[0098] Statistical consistency is assessed using hypothesis testing and correlation analysis. Statistical methods such as t-tests and chi-square tests are employed to verify the consistency of data distribution. An automated evaluation pipeline is implemented, calculating various consistency indicators in real time. Visualization of evaluation results and report generation functions are provided.

[0099] Anomaly Handling Mechanism: Anomaly handling employs a multi-layered processing architecture. Anomaly detection utilizes machine learning-based anomaly detection algorithms, including models such as Isolation Forest and Autoencoders. This enables real-time monitoring and early warning, allowing for timely detection and handling of anomalies. Fault Isolation employs containerization and virtualization technologies to achieve resource isolation and fault containment.

[0100] Automatic recovery implements intelligent recovery strategies, including state rollback, data reconstruction, and service restart. It uses health checks and heartbeat detection to monitor system status in real time. Manual intervention provides detailed problem diagnosis information and handling suggestions, supporting remote intervention and debugging.

[0101] System performance optimization strategies: Computational optimization employs several advanced techniques. Model compression utilizes pruning, quantization, and low-rank decomposition to significantly reduce computational load. Knowledge distillation achieves model lightweighting, improving speed while maintaining accuracy. Computational graph optimization uses graph optimization techniques to simplify the computation process and increase parallelism.

[0102] Hardware acceleration fully utilizes accelerators such as GPUs and TPUs. Operator fusion and kernel optimization are implemented to improve computational efficiency. Inference acceleration frameworks such as TensorRT and OpenVINO are used to optimize model deployment. Parallel computing employs multiple parallel strategies, including data parallelism, model parallelism, and pipelined parallelism.

[0103] Storage optimization employs intelligent caching strategies. Feature caching utilizes advanced caching algorithms such as LRU-K and ARC to improve cache hit rate. Memory management uses memory pools and object pools to reduce memory allocation overhead. Data compression uses fast compression algorithms such as Snappy and LZ4 to balance compression ratio and speed.

[0104] Index optimization utilizes efficient index structures such as B+ trees and LSM trees. Multi-level and composite indexes are implemented to improve query efficiency. Columnar and compressed storage formats are used to reduce I / O overhead. Data partitioning employs range partitioning and hash partitioning to optimize data locality.

[0105] Network optimization: A high-performance network architecture is employed for network optimization. Data compression uses Delta encoding and dictionary encoding to reduce network traffic. Connection multiplexing utilizes connection pooling technology to reduce connection establishment overhead. Asynchronous communication uses the Reactor and Proactor patterns to improve network processing capabilities.

[0106] Load balancing employs various load balancing algorithms, including round-robin, weighted round-robin, least connections, and consistent hashing. It implements health checks and failover to improve system availability. Network monitoring tracks network performance in real time, enabling timely detection and resolution of network problems.

[0107] Example 1: Interbank Transaction Document Processing Interbank transaction document processing is a typical application scenario. The system receives transaction documents in PDF format and first performs format recognition and preprocessing. The vision module detects table areas and text blocks in the document and identifies transaction information tables. The semantic module parses the transaction terms and conditions and extracts key transaction elements.

[0108] The collaborative unit identifies key data such as transaction details, transaction amount, and transaction date through multimodal fusion. An iterative optimization strategy ensures the accuracy of data extraction. The final output is structured transaction data, including transaction records in JSON format and metadata in XML format. The system processes a single transaction document in an average of 2.3 seconds with an accuracy rate of 99.2%.

[0109] In actual testing, under the conditions of 4×A100-80GB, batch=8, FP16, and TensorRT graph capture, the P99 latency of 1000 3-page PDF samples was 2.29 s, the field-level F1 score was 99.23%, the GPU utilization was 82%, and the video memory usage was 62 GB. The test environment and logs have been solidified and the original recorded effect can be directly reproduced without the need for additional hardware or software modules.

[0110] Example 2: Analysis of Fund Statements The fund statement parsing system demonstrates its ability to handle complex tables. Inputting a scanned copy of the fund statement, the visual module recognizes complex table structures, including merged cells and multi-level headers. The semantic module parses professional information such as fund units, net asset value, and rate of return.

[0111] The collaborative unit implements data consistency verification to ensure the accuracy of extracted data. It employs cross-page table processing technology to handle multi-page account statements. Standardized holding data is output, including fund codes, units, market value, and other information. The system supports batch processing, capable of processing over 1000 account statements per hour with an accuracy rate of 98.7%.

[0112] System Deployment and Maintenance: The system supports a variety of flexible deployment options. Cloud deployment provides automated deployment scripts, supporting mainstream cloud platforms such as AWS, Azure, and Google Cloud. On-premises deployment offers detailed hardware configuration recommendations and network planning solutions. Hybrid deployment enables collaborative work between the cloud and on-premises environments, supporting data synchronization and failover.

[0113] Edge deployment supports running lightweight versions on edge devices. It provides containerized deployment solutions, supporting Docker and Kubernetes. One-click deployment and automated configuration simplify the deployment process. Deployment verification tools are provided to ensure deployment correctness.

[0114] Monitoring and Maintenance: The monitoring system provides comprehensive monitoring. Performance monitoring collects system metrics in real time, including throughput, response time, and error rate. Fault early warning uses machine learning algorithms to predict potential faults. Log analysis enables real-time log processing and anomaly detection.

[0115] Regular maintenance includes system upgrades, data backups, and performance optimization. Automated maintenance tools are provided to reduce manual intervention. Remote maintenance and fault diagnosis are enabled to improve maintenance efficiency. Detailed maintenance documentation and operation guides are provided.

[0116] Scalability Design: The system boasts excellent scalability. Horizontal scaling supports dynamically adding processing nodes and automatic load balancing. Vertical scaling supports hardware upgrades to improve single-node performance. Functional expansion provides a plug-in mechanism to support feature extensions.

[0117] The interface extension supports both standard APIs and custom interfaces. An SDK development toolkit is provided for easy system integration. Third-party service integration is supported to extend system functionality. Elastic scaling is achieved, automatically adjusting resource allocation based on load.

[0118] Security Measures: A multi-layered security system is employed. Data encryption uses the AES-256 encryption algorithm to protect data transmission and storage security. Access control implements the RBAC permission model for precise control of access permissions. Audit logs record all sensitive operations, supporting security auditing.

[0119] The AES-256 key is managed uniformly by Vault KMS and rotates every 90 days; roles are subdivided to the field level, such as the auditor only seeing the anonymized amount (first 3 and last 2 digits); audit logs include userId, action, timestamp, and hash, and are written to WORM storage and retained for 10 years, which complies with the existing "encryption + RBAC + audit" framework and does not introduce any new security mechanisms.

[0120] Vulnerability management includes establishing a vulnerability response mechanism and conducting regular security scans. Network protection utilizes firewalls and intrusion detection systems. Identity authentication supports multi-factor authentication to ensure secure access. Data anonymization involves anonymizing sensitive information to protect privacy and security.

[0121] Reliability Assurance: Multiple measures are employed to ensure reliability. Redundancy design implements hardware and software redundancy to ensure system availability. Failover enables automatic fault detection and failover, reducing service interruptions. Data backup employs a strategy combining full and incremental backups.

[0122] Disaster Recovery: Establish disaster recovery plans and conduct regular recovery drills. Achieve data and status synchronization to ensure data integrity during failures. Service Quality Assurance: Implement SLA management to ensure service level agreements are fulfilled. Performance Monitoring: Monitor system performance in real time to promptly identify and resolve performance issues.

[0123] This invention, through the specific embodiments described above, achieves efficient and accurate element extraction from various documents. The system integrates the advantages of text understanding and visual analysis, improving processing efficiency and accuracy through an intelligent collaborative mechanism, thus meeting the needs of modern enterprises for automated document processing. The system possesses excellent scalability and reliability, enabling it to adapt to various complex application scenarios.

[0124] The multimodal cognitive fusion collaborative processing method provided by this invention comprises three main stages in its core implementation process. First, a large-scale semantic understanding module is used to perform deep semantic analysis on the text content of the input document. This module is built upon a pre-trained large-scale language model that has been specially fine-tuned and employs a multi-layered processing architecture. It can effectively extract entity information, semantic relationships, and key summaries from the document, generating structured textual semantic features. These features are represented hierarchically, including word-level features (300-dimensional word vector representation and attention weight distribution), sentence-level features (768-dimensional sentence embedding vectors and semantic role annotation results), and document-level features (topic vectors and key information summaries). Second, a visual language model parsing module is simultaneously used to perform refined layout analysis on the document image. This module employs a multi-stage processing pipeline architecture, accurately identifying visual elements such as text blocks, tables, charts, and seals in the document and their precise spatial coordinate information through steps such as image preprocessing, object detection, layout analysis, and feature extraction. This generates visual structural features that include spatial, structural, semantic, and relational features. Finally, the real-time collaborative decision-making unit dynamically fuses textual semantic features and visual structural features based on a gating mechanism to generate cross-modal fusion instructions. Based on these instructions, the large model semantic understanding module and / or visual language model parsing module execute iterative reasoning. Through a multi-round processing mechanism, the feature fusion effect is continuously optimized, and finally, a structured cognitive result with a cross-modal evidence chain is output.

[0125] During the dynamic fusion process, the system establishes an efficient bidirectional transmission channel between the large model semantic understanding module and the visual language model parsing module through a bidirectional information bus, enabling real-time exchange and sharing of intermediate features (including textual semantic features and visual structural features). This mechanism ensures that the two modules can fully utilize each other's feature information during processing, breaking the information silos in the traditional serial architecture. The gated dynamic fusion model adopts a three-layer fully connected neural network structure based on an attention mechanism. The input vector is composed of 768-dimensional textual semantic features, 512-dimensional visual table pooling features, 16-dimensional task one-hot encoding, and 4-dimensional quality score concatenation, for a total dimension of 1300. The number of network nodes is set to 1024 / 256 / 2 respectively, and the output is normalized by the softmax function to generate text weight wt and visual weight wv (the sum of the two is always 1). The model dynamically allocates the fusion weights of textual semantic features and visual structural features according to the current task requirements, generating accurate cross-modal fusion instructions to realize a collaborative strategy of vision guiding semantics or semantics guiding vision. The AdamW optimizer was used during training, with a learning rate of 2e-4, a cosine annealing strategy, a batch size of 256, and multi-task loss weights set empirically to 0.5:0.3:0.2. After 30 rounds of training on a set of 400,000 financial documents, the validation fusion accuracy reached 98.7%.

[0126] The iterative inference process employs two different processing modes depending on the cross-modal fusion instruction. When the cross-modal fusion instruction indicates text dominance, the large-model semantic understanding module performs focused semantic inference on the corresponding text region based on the precise localization information provided by visual structural features. In this mode, the system prioritizes utilizing the spatial context provided by visual information to enhance the accuracy of semantic understanding, especially for tabular data, multi-page content, or document regions with complex layouts. When the cross-modal fusion instruction indicates visual dominance, the visual language model parsing module relocates and structurally corrects target visual elements in the image based on semantic keywords and contextual information provided by text semantic features. This mode is particularly suitable for handling visual recognition challenges such as stamp occlusion, handwritten annotations, and merged cells. Both modes are coordinated and controlled by a real-time collaborative decision-making unit, which dynamically adjusts the dominant strategy based on processing progress and result quality to ensure the system can flexibly adapt to different types of document processing needs.

[0127] During iterative reasoning, the system implements a strict consistency monitoring mechanism to evaluate the consistency level between text semantic features and visual structural features in real time. The consistency evaluation adopts a multi-dimensional comprehensive scoring system, including semantic consistency (weight 0.4), spatial consistency (weight 0.4), and logical consistency (weight 0.2). Semantic consistency uses the SimCSE model to calculate vector cosine similarity, with a value ≥0.90 considered consistent. Spatial consistency uses an IoU (Intersection over Union) of ≥0.85 between text field boxes and visual detection boxes as the passing standard. Logical consistency sets a 1% relative error tolerance for business rules (such as "amount = unit price × quantity"). The system sets multi-level trigger thresholds: when the overall consistency score is below 0.75, local reprocessing is triggered, only re-analyzing the problem area; when it is below 0.65, full-page reprocessing is triggered, re-executing the complete processing flow. The system sets a maximum of 5 iterations and implements an early termination mechanism, terminating the iteration process early when the consistency improvement between two consecutive iterations is less than 0.01. Through this intelligent iterative control mechanism, the system can effectively control processing latency while ensuring output quality. In actual tests, the average iteration per page is 1.8 times, and the latency increase is controlled within 20%. The final output is a structured cognitive result with a complete cross-modal evidence chain.

[0128] The key design focus of this invention is to construct a collaborative framework driven by both semantics and vision, centered on a "real-time collaborative decision-making unit." On one hand, a gating network dynamically allocates text and visual weights, enabling instant switching between three strategies: visual-guided semantics, semantic-guided visuals, and balanced fusion. This allows the system to both correct semantic illusions using layout information and accurately locate visual elements using semantic priors. On the other hand, a consistency scoring loop is introduced, completing multiple iterations within milliseconds to ensure continuous alignment and convergence of text-side and visual-side results. This effectively addresses the simultaneous decrease in confidence caused by complex layouts such as stamp obscuring, cross-page tables, and handwritten annotations. The output stage employs a dual-track evidence chain design, encapsulating the text slice, coordinate frame, and confidence triplet for each field. Reviewers can jump to the original image area with a single click, significantly reducing review costs. The system adopts a microservice architecture and supports elastic deployment across the cloud, edge, and device. Through multiple acceleration methods such as quantization, distillation, caching, and parallelism, it achieves high concurrency and low latency while maintaining high accuracy. It has built-in mechanisms such as fault isolation, state self-healing, security encryption, and fine-grained role authorization to meet the stringent requirements of reliability, security, and compliance in scenarios such as finance, government affairs, and healthcare, forming a sustainable and evolving general document understanding infrastructure.

[0129] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the technical scope of the present invention. Therefore, any minor modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A multimodal cognitive fusion and collaborative processing system based on AI large model technology, characterized in that, include: The large-scale model semantic understanding module is used to perform deep semantic analysis on the text content of documents, extract entity, relation and summary information, and generate text semantic features; The visual language model parsing module is used to perform layout analysis on document images, identify visual elements such as text blocks, tables, charts, and seals and their spatial coordinates, and generate visual structural features. The real-time collaborative decision-making unit is connected to the large model semantic understanding module and the visual language model parsing module, respectively. It is used to dynamically fuse the text semantic features and the visual structural features based on the gating mechanism, generate cross-modal fusion instructions, and drive the large model semantic understanding module and / or the visual language model parsing module to perform iterative reasoning according to the cross-modal fusion instructions, so as to output structured cognitive results.

2. The multimodal cognitive fusion collaborative processing system based on AI large model technology according to claim 1, characterized in that, The real-time collaborative decision-making unit includes: A bidirectional information bus is used for bidirectional transmission of intermediate features, including textual semantic features and visual structural features, between the large model semantic understanding module and the visual language model parsing module. The gated dynamic fusion model is used to dynamically allocate fusion weights of text semantic features and visual structural features according to the current task requirements, generate the cross-modal fusion instructions, and realize a collaborative strategy of visual-guided semantics or semantic-guided vision.

3. The multimodal cognitive fusion and collaborative processing system based on AI large model technology according to claim 1, characterized in that, The large model semantic understanding module is configured to: receive the cross-modal fusion instruction, perform focused semantic reasoning on the corresponding text region based on the positioning information provided by the visual structural features, update the text semantic features, and return them to the real-time collaborative decision-making unit.

4. The multimodal cognitive fusion and collaborative processing system based on AI large model technology according to claim 1, characterized in that, The visual language model parsing module is configured to: receive the cross-modal fusion instruction, and based on the semantic keywords provided by the text semantic features, relocate and correct the target visual elements in the image to update the visual structural features and return them to the real-time collaborative decision-making unit.

5. The multimodal cognitive fusion collaborative processing system based on AI large model technology according to claim 1, characterized in that, The real-time collaborative decision-making unit is further configured to: monitor the consistency between text semantic features and visual structural features in real time during iterative reasoning; when the consistency is lower than a preset threshold, automatically trigger a new round of cross-modal fusion instruction generation until the consistency meets the threshold requirement.

6. The multimodal cognitive fusion collaborative processing system based on AI large model technology according to claim 1, characterized in that, The structured cognitive results include: entities in documents, entity relationships, tabular data, semantic interpretation of charts and graphs, and their cross-modal evidence chains.

7. A method applied to a multimodal cognitive fusion collaborative processing system based on AI large model technology as described in any one of claims 1-6, characterized in that, include: The large model semantic understanding module is used to perform deep semantic analysis on the text content of the document, extract entity, relation and summary information, and generate text semantic features; The visual language model parsing module is used to perform layout analysis on document images, identify visual elements and their spatial coordinates, and generate visual structural features. The text semantic features and the visual structural features are dynamically fused by a real-time collaborative decision-making unit based on a gating mechanism to generate cross-modal fusion instructions. Based on the cross-modal fusion instructions, the large model semantic understanding module and / or the visual language model parsing module are driven to perform iterative reasoning to output structured cognitive results.

8. The method according to claim 7, characterized in that, Dynamic fusion includes: Intermediate features are transmitted bidirectionally between the large model semantic understanding module and the visual language model parsing module via a bidirectional information bus. By using a gated dynamic fusion model to dynamically allocate fusion weights for text semantic features and visual structural features according to the current task requirements, the cross-modal fusion instructions are generated to realize a collaborative strategy of visual-guided semantics or semantic-guided vision.

9. The method according to claim 7, characterized in that, The iterative reasoning includes: When the cross-modal fusion instruction indicates that the text is dominant, the large model semantic understanding module performs focused semantic reasoning on the text region based on the positioning information provided by the visual structural features; When the cross-modal fusion instruction indicates that vision is dominant, the visual language model parsing module relocates and corrects the structure of the target visual element based on the semantic keywords provided by the text semantic features.

10. The method according to claim 7, characterized in that, During the iterative reasoning process, the consistency between text semantic features and visual structural features is monitored in real time. When the consistency is lower than a preset threshold, the dynamic fusion step is automatically returned to be executed until the consistency meets the threshold requirement, thereby outputting a structured cognitive result with a cross-modal evidence chain.

Images