A multi-modal information extraction and annotation method and system

Abstract

The application discloses a kind of multimodal information extraction and labeling method and system, the method is automatically collected multi-source data;Through data cleaning agent, the original data is de-duplication and quality filtering, through information extraction agent, entity recognition and relationship extraction are executed, through multimodal alignment agent, cross-modal data matching is realized;Build the human-in-the-loop labeling platform based on crowd-sourcing level, professional level and expert level three-level architecture, manually check and correct the pre-processing result, and based on the difference analysis of expert labeling result and AI pre-processing result, identify model bias and generate targeted optimization strategy;Based on feedback data, dynamically adjust to continuously improve data labeling quality and the accuracy of multimodal alignment.The application can be applied to microbiology, medicine, industrial equipment, finance and other fields, improve the labeling efficiency, processing accuracy and model generalization ability of multimodal data in the field.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for multimodal information extraction and annotation. Background Technology

[0002] With the rapid development of science and technology and the widespread application of digital acquisition methods, the scale of multimodal data, including scientific literature, experimental records, technical documents, image reports, and multimedia materials, is exploding. The amount of academic papers and professional technical data generated annually is enormous, and the data scale of various professional knowledge bases continues to expand rapidly. Taking the field of microbiology as an example, although existing strain databases such as ATCC and DSMZ have achieved centralized management of basic strain information to some extent, public databases still have significant shortcomings in terms of data coverage, update timeliness, and fine-grained feature expression. A large amount of microbial characteristic data with significant scientific research value and industrial application potential, especially key information such as growth parameters, metabolic characteristics, and culture conditions of newly discovered or newly characterized strains, is still mainly scattered and stored in academic papers and related technical literature, and has not yet been systematically and structurally organized, seriously restricting the efficient utilization of microbial data resources and the construction of knowledge bases. At the same time, existing general information extraction and retrieval models are mainly designed for general text scenarios and are difficult to adapt to the characteristics of microbial literature, such as dense professional terminology, strong conditional dependence, and highly intertwined multimodal information, resulting in insufficient accuracy and stability in practical applications. Existing technical solutions mainly have the following problems: (1) Existing data collection and information extraction technologies mainly rely on automated tools and database interfaces, but these technologies have significant limitations when dealing with massive amounts of complex multimodal data. Specific problems include: difficulty in cross-source data integration: In microbial research scenarios, the classification information, physiological characteristics, and culture conditions of the same strain are often scattered across different papers, database records, and supplementary materials, and are presented in various forms such as text descriptions, table parameters, and images and charts. Existing technologies lack the ability to uniformly organize and collaboratively analyze multi-source and multimodal microbial data, making it difficult to achieve complete integration of strain characteristic information. Limited ability to analyze unstructured data: Microbiological literature usually adopts a two-column layout, containing complex structures such as cross-page tables, embedded images, and annotations. Existing methods have difficulty accurately identifying strain entities, characteristic attributes, and their contextual relationships when processing such documents, especially when analyzing growth parameter ranges, conditional dependency descriptions, and cross-paragraph information, resulting in low accuracy. Insufficient model generalization ability: Although deep learning models perform well in general text tasks, their training corpora usually lack professional knowledge in the field of microbiology. When applied to microbial literature with complex strain nomenclature, unique terminology, and implicit logical relationships in conditional expressions, the general model struggles to accurately identify key strain entities and their physiological characteristics, resulting in significantly insufficient model generalization ability.

[0003] (2) Existing technologies lack a rigorous quality control mechanism in the data annotation process. Specifically, this manifests in the following ways: Low annotation efficiency and accuracy: Microbiological data exhibits significant multi-source heterogeneity and strong conditional dependence. Annotators need to process multiple information formats, such as text, tables, and images, and face issues such as differences in strain naming, inconsistent parameter units, and cross-document distribution of features. In the absence of intelligent auxiliary tools, manual annotation is prone to omissions or misjudgments in key stages such as entity recognition, attribute extraction, and conditional association. Especially when it is necessary to comprehensively judge strain characteristics across paragraphs or modalities, the annotation cost is high and the accuracy is difficult to guarantee. Lack of a professional annotation system: Microbial feature data annotation highly depends on domain knowledge and practical experience. However, the professional backgrounds of annotators vary greatly in the existing annotation process, and there is a lack of clear hierarchical division of responsibilities and a unified professional verification mechanism, resulting in inconsistent annotation standards and making it difficult to guarantee the reliability and authority of the benchmark dataset. Lack of standardized processes: There is a lack of unified annotation specifications and operating guidelines for microbial feature data. Different annotators have subjective differences in their understanding and expression of the same strain attributes (such as growth temperature range, metabolic capacity description, etc.), which affects data consistency. Weak verification mechanism: Traditional manual verification suffers from inefficiency and difficulty in collaboration when processing multimodal data due to the lack of systematic platform support.

[0004] (3) Disconnect between manual verification and model training: The correction results of strain feature data by microbiology experts are usually only used for the construction of the current dataset and are not systematically fed back into the information extraction and preprocessing model, making it difficult to form a continuous optimization mechanism for microbial feature error patterns. Insufficient incremental learning support: Existing technologies are mostly based on fixed training sets for model training and lack the ability to adaptively learn from new microbial data. When new strains, new feature types, or new expression patterns appear, the model performance is prone to decline. Coarse optimization granularity: Traditional methods usually only adjust model parameters through macro indicators such as overall accuracy, making it difficult to perform fine-grained optimization for specific error patterns of different data modalities or different types of entities, which limits the performance improvement space of the model in complex professional data scenarios. Summary of the Invention

[0005] This invention proposes a method for multimodal information extraction and annotation.

[0006] Another objective of this invention is to propose a multimodal information extraction and annotation system.

[0007] To achieve the above objectives, a first aspect of the present invention proposes a method for multimodal information extraction and annotation, comprising: Acquire target domain data from multiple heterogeneous data sources, including database interfaces, unstructured documents, and business monitoring data, and output standardized multimodal data after preprocessing. Construct an intelligent agent processing pipeline consisting of a data cleaning agent, an information extraction agent, and a multimodal alignment agent to automatically clean, extract, and align standardized multimodal data across modalities, and output an initial knowledge graph. The initial knowledge graph is verified, corrected, and finalized using a hierarchical human-machine collaborative annotation platform with crowdsourcing, professional, and expert levels to output a gold standard dataset. Based on the difference analysis between the gold standard dataset and the initial knowledge graph, high-frequency error patterns are identified through unsupervised clustering. The amount of manual correction is used as a negative reward signal. Reinforcement learning is used to dynamically adjust the confidence threshold and extraction strategy of the agent's processing pipeline to form a closed-loop feedback.

[0008] In one embodiment of the present invention, target domain data from multiple heterogeneous data sources, including database interfaces, unstructured documents, and business monitoring data, is acquired, preprocessed, and then output as standardized multimodal data, including: Structured data from external databases is collected in batches through standardized API interfaces, and the data is converted into a predefined unified format containing entity identifiers, classification information, and basic feature parameters and stored in a relational database. An OCR engine is used to extract text and parse layout of unstructured documents in PDF and image formats, separating text paragraphs, table areas and image blocks. Based on regular expression matching rules, strain names, experimental parameters and numerical features are pre-identified, and deep neural networks are used to extract image feature vectors. The structured data after format conversion, the encoded standardized text corpus, and the image feature vectors are stored in a unified encoding format, and the output is standardized multimodal data for intelligent agents to process.

[0009] In one embodiment of the present invention, standardized multimodal data is automatically cleaned, information extracted, and cross-modal semantically aligned to output an initial knowledge graph, including: The data cleaning agent performs deduplication, quality filtering and standardization on the standardized multimodal data. The SimHash algorithm and perceptual hash comparison are used to identify and remove duplicate text and images respectively. Low-quality items are removed based on field integrity, logical rules and preset quality standards. The units, names and formats are unified to predefined standards and the cleaned standardized data is output. An information extraction agent is constructed using a pre-trained model fine-tuned with vertical domain corpus. Named entity recognition is performed on the cleaned and standardized data and linked to the industry standard database ID. Relationships between entities are extracted through dependency parsing and business rule base. Numerical attributes are standardized and modeled to generate a structured knowledge graph. A multimodal alignment agent is used to establish semantic alignment relationships between text, images, and sequence data. The CLIP model is used to calculate the cross-modal similarity between text and images to achieve matching retrieval. Faster R-CNN is used for visual localization and annotation of key regions. Finally, a graph neural network is used to jointly model multimodal features to generate unified entity identifiers and output the aligned multimodal knowledge graph.

[0010] In one embodiment of the present invention, the step of verifying, correcting, and finalizing the initial knowledge graph through a hierarchical human-machine collaborative annotation platform with crowdsourcing, professional, and expert levels to output a gold standard dataset includes: The multimodal knowledge graph is subjected to basic annotation through a crowdsourced annotation layer. Annotation tasks are split based on a rule engine and standardized templates are provided. High-confidence AI recognition results are pre-filled and real-time logical verification is performed. The consistency index of multiple annotators is calculated, low-quality annotators are automatically filtered out, and an initial annotation set is output. The professional-grade annotation layer receives the initial annotation set and conflict report, performs cluster analysis on similar conflicts and calls the domain rule base, trains the LightGBM model based on historical decisions to recommend the optimal annotation to intelligently correct the crowdsourcing results, outputs the verified annotation set, and upgrades complex disputed samples to the expert level. An expert-level final review layer is used to perform text-image cross-validation and key feature comparison on the high-risk samples. A double-blind review mechanism is used to make a final decision and lock the annotation results, and output the gold standard dataset.

[0011] In one embodiment of the present invention, the step of analyzing the difference between the gold standard dataset and the initial knowledge graph, identifying high-frequency error patterns through unsupervised clustering, using manual correction as a negative reward signal, and dynamically adjusting the confidence threshold and extraction strategy of the agent's processing pipeline using reinforcement learning to form a closed-loop feedback includes: The system integrates feedback data, compares the gold standard dataset with the initial knowledge graph using a difference analysis engine, extracts the structured differences of entities and attribute values, identifies high-frequency error patterns using an unsupervised clustering algorithm, generates an error pattern analysis report, designs a negative reward function based on manual corrections, dynamically adjusts the confidence threshold and extraction strategy of the agent's processing pipeline using deep reinforcement learning, verifies the optimization effect through A / B testing, and outputs an updated parameter configuration file. Establish a closed loop for model iteration. When the cumulative amount of newly added labeled data reaches a preset threshold, incremental training is triggered. On the basis of the pre-trained language model, a single round of fine-tuning is performed and a cosine annealing learning rate scheduling strategy is adopted. The new model is applied to some labeling tasks through canary release. Core indicators are monitored in real time. If the correction rate drops to a preset threshold, full deployment is performed. Otherwise, it is automatically rolled back to the previous stable version. The system constructs an evaluation dashboard, using interactive and visual charts to display key indicators such as annotation efficiency, expert correction rate, and cross-modal alignment accuracy. It supports drill-down analysis by target domain category, data modality, and time dimension to compare the performance differences of different model versions, and presents a heatmap of error type distribution and a scatter plot of individual annotator performance differences.

[0012] In one embodiment of the present invention, constructing the data cleaning Agent includes: A deduplication module is constructed. The SimHash algorithm is used to calculate the similarity of text content and determine that text with a similarity greater than a preset threshold is duplicate data. The perceptual hash algorithm is used to compare image content. Combined with the text similarity algorithm based on cosine similarity and edit distance, as well as the document encoding and UUID unique identifier precise matching mechanism, duplicate or nearly duplicate data records are identified and removed. A quality filtering module is built to remove documents with missing key information or incomplete metadata from text data, remove low-quality images from image data, and establish multi-level quality assessment standards for structured business data through field integrity checks, data consistency verification, and outlier detection techniques to automatically identify and remove data items that do not meet quality requirements. Standardized functional modules are constructed, and a unified data storage format and encoding standard are established to convert data from different sources into a predefined standard format, including UTF-8 unification of text encoding, ISO 8601 standardization of date format, conversion of numerical units to the International System of Units, and unified mapping of unit symbols and names. The cleaned and standardized data is output to ensure the efficiency and accuracy of subsequent information extraction.

[0013] In one embodiment of the present invention, constructing the information extraction Agent includes: A named entity recognition module is built by fine-tuning a pre-trained model with vertical domain corpus. The module performs entity recognition on the cleaned standardized data and links it to the industry standard database ID. It establishes a hierarchical entity classification system that includes core object names and attribute names, processes entity abbreviations, aliases and synonym variants, and outputs complete and accurate entity recognition results. A relation extraction module is constructed, which uses vertical domain business rules and dependency parsing techniques to extract subject-verb-object triple relationships between entities, identify complex logical associations, and generate structured relation data. An attribute extraction module is constructed to identify and analyze quantitative and semi-quantitative features in text. Range values, approximate values, and conditional limits are uniformly modeled into structured numerical representations containing minimum values, maximum values, and standardized units. The output is a structured knowledge graph for multiple domains to support multimodal alignment processing.

[0014] In one embodiment of the present invention, constructing the multimodal alignment agent includes: A text-image alignment module is constructed using a vision-language pre-trained model. The semantic similarity between the text description and the image is calculated using the CLIP model or other cross-modal representation models to achieve matching retrieval. The key regions of the image are visually located using Faster R-CNN or YOLO series object detection models, and the text semantic features and image visual features are fused and aligned. A sequence-function alignment module is constructed using sequence alignment algorithms or temporal neural networks. Sequence data is matched with predefined business rules using dynamic time warping algorithms or Transformer-based temporal models to generate corresponding functional or state annotations and establish a mapping relationship between measurement sequences and biological significance. A unified entity module is constructed using graph neural networks, including GraphSAGE, GraphConvolutional Network, Graph Isomorphism Network, or hyperbolic graph neural network, to jointly model and embed multimodal features of text entities, image features, and sequence features, thereby achieving cross-modal entity disambiguation and unified identifier generation.

[0015] To achieve the above objectives, a second aspect of the present invention provides a multimodal information extraction and annotation system, comprising: The multi-source data acquisition module is used to acquire target domain data from multiple heterogeneous data sources, including database interfaces, unstructured documents, and business monitoring data, and output standardized multimodal data after preprocessing. The multi-agent construction module is used to build an agent processing pipeline consisting of a data cleaning agent, an information extraction agent, and a multimodal alignment agent to automatically clean, extract information, and perform cross-modal semantic alignment on standardized multimodal data, and output an initial knowledge graph. The collaborative annotation module is used to verify, correct, and finalize the initial knowledge graph through a hierarchical human-machine collaborative annotation platform with crowdsourcing, professional, and expert levels, so as to output the gold standard dataset. The dynamic consistency verification module is used to analyze the difference between the gold standard dataset and the initial knowledge graph, identify high-frequency error patterns through unsupervised clustering, use the manual correction amount as a negative reward signal, and dynamically adjust the confidence threshold and extraction strategy of the agent's processing pipeline using reinforcement learning to form a closed-loop feedback.

[0016] The multimodal information extraction and annotation method and system of this invention can realize cross-modal semantic alignment and consistency verification of microbial multimodal data, effectively solve the problems of fragmented text and image information, difficulty in parsing conditional dependency attributes and entity disambiguation, and significantly improve the structured quality and construction efficiency of knowledge graphs.

[0017] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0018] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart of a multimodal information extraction and annotation method provided in an embodiment of the present invention; Figure 2 A flowchart illustrating another method for multimodal information extraction and annotation provided in this embodiment of the invention; Figure 3 A flowchart of another multimodal information extraction and annotation method provided in an embodiment of the present invention; Figure 4 This is a structural diagram of a multimodal information extraction and annotation system provided in an embodiment of the present invention. Detailed Implementation

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] A method and system for multimodal information extraction and annotation according to an embodiment of the present invention will now be described with reference to the accompanying drawings.

[0022] Example 1 This invention proposes a method for multimodal information extraction and annotation, such as... Figure 1 As shown, the method includes the following steps: S101 acquires target domain data from multiple heterogeneous data sources, including database interfaces, unstructured documents, and business monitoring data, and outputs standardized multimodal data after preprocessing.

[0023] Specifically, for multi-source heterogeneous data scenarios, this invention constructs an automated data acquisition system for the unified acquisition and preliminary standardization of data with complex professional characteristics. In a typical application scenario, taking microbiology research and application data as an example, relevant data typically originates from various channels such as strain preservation databases, academic papers, and experimental records, involving key features such as strain characteristics, classification information, physiological characteristics, and culture conditions. This step establishes standardized API interfaces with authoritative industry databases and third-party data platforms to achieve large-scale acquisition of basic strain information and related structured data. Combined with automatic parsing of professional literature, this provides high-quality, consistent data input for subsequent intelligent agent-based processing pipelines, thereby reducing the complexity of downstream information extraction and semantic parsing. Specifically, it includes: Basic information of strains is obtained in batches from strain preservation databases via a standardized API interface. The obtained JSON or XML format data is converted into a unified structured table, with table fields including strain identifier, classification information, and basic characteristic parameters. Layout parsing and optical character recognition are performed on microbiology literature PDFs or image files. Text paragraphs, parameter tables, and growth curve image blocks are split using a document structure analysis model, and strain numbers, physiological characteristic descriptions, and experimental condition data are identified and extracted. The collected multi-source data undergoes deduplication, quality filtering, and unit unification processing. The SimHash algorithm is used to remove duplicate literature, data lacking core strain information, or data with resolution below a threshold are removed, and the temperature unit is unified to degrees Celsius, generating a standardized dataset for downstream processing.

[0024] Specifically, the multi-source data acquisition system in this embodiment is designed for three types of heterogeneous data sources: the first type is external data platform API interfaces, including professional interfaces such as strain preservation databases and taxonomic databases, used to batch acquire strain numbers, genus and species classification levels, and basic attribute information; the second type is professional technical documents, covering microbiology academic papers, culture medium formulation instructions, and experimental reports in PDF, PNG, or TIFF format; the third type is raw business data, including experimental measurement data, growth curve images, and microscopic images, providing a multi-dimensional information foundation for subsequent processing.

[0025] The structured data acquisition process obtains target strain information in batches through a standardized API interface, including strain number, taxonomic level and known basic parameters, and converts the acquired JSON or XML format data into a unified structured table with fields including at least entity_id (strain identifier), taxonomy (classification information) and target_parameter (basic feature parameter), thus achieving standardization and unification of data format.

[0026] Unstructured document parsing employs a triple processing mechanism: First, OCR text extraction is performed using the Tesseract engine, prioritizing image input with a resolution of at least 300 DPI in microbiology literature scenarios to improve recognition accuracy. Next, key entities and numerical features such as strain names, temperature, and pH values ​​are pre-identified based on rule-based and regular expression matching. Finally, a PDF parser performs layout analysis on the document, separating and splitting text paragraphs, table areas, and image blocks to lay the foundation for cross-modal feature association. After processing, basic strain information and attribute data are stored in a MySQL or PostgreSQL database. OCR and pre-annotated text are stored using a unified encoding format, while feature vectors are extracted from growth curves and microscopic images for subsequent multimodal alignment analysis.

[0027] S102, construct an intelligent agent processing pipeline consisting of a data cleaning agent, an information extraction agent, and a multimodal alignment agent to automatically clean, extract information, and perform cross-modal semantic alignment on standardized multimodal data, and output an initial knowledge graph.

[0028] Specifically, this invention utilizes a data cleaning agent to perform quality filtering on the input standardized dataset, removing low-quality entries that lack core strain information, have image resolution below a threshold, or have incomplete parameter units, and outputting a cleaned, high-quality dataset. An information extraction agent performs semantic parsing on the cleaned dataset, identifying strain entities and linking them to standard database IDs, extracting physiological characteristics and culture conditions to construct subject-verb-object triples, and outputting structured domain knowledge fragments. A multimodal alignment agent inputs text entities, microscopic image features, and growth curve sequences into a graph attention network, calculates cross-modal semantic similarity, aligns descriptions of the same strain in different modalities, and outputs a unified multimodal knowledge graph.

[0029] It is understood that this invention proposes a data preprocessing and information extraction pipeline based on an intelligent agent architecture. By setting up multiple AI agents with clearly defined functional roles, it automates and intelligently processes the multi-source heterogeneous data collected in step one. In the microbial application example, the data simultaneously contains multimodal information such as strain names and classification information, physiological characteristic descriptions, culture condition parameters, and growth curve images, and the relevant features exhibit significant conditional dependence and cross-modal distribution characteristics. The pipeline adopts a modular design, with each agent performing specific data processing tasks and collaborating to automatically transform raw data into structured domain knowledge, thereby improving the efficiency, consistency, and reliability of complex professional data processing. This includes: In one embodiment, the data cleaning agent contains the following details: In modern data processing workflows, the data cleaning agent, as the first step in the data processing pipeline, plays a crucial role. It is primarily responsible for quality control and standardization of the collected raw multimodal data, ensuring the effectiveness and accuracy of subsequent analyses. Particularly in the field of microbiology research, this agent utilizes specialized microbiological knowledge to perform consistency checks and standardization on strain names, experimental parameters, growth conditions, and related image data. Through this series of operations, the agent effectively improves data quality, providing a solid foundation for downstream data analysis. For example, when processing raw data such as microbiological literature texts, culture parameter tables, growth curve images, and microscopic images, the data cleaning agent first performs deduplication, using the SimHash algorithm to calculate text similarity and employing perceptual hashing (pHash) to compare image content, thereby eliminating duplicate data items.

[0030] Next, the data cleaning agent performs rigorous quality filtering on the data. For text data, data missing key strain information or with incomplete experimental condition descriptions is flagged and excluded. For image data, images with too low resolution or that fail to clearly reflect growth trends are automatically identified and removed. Tables containing experimental parameters are also considered unqualified if they lack necessary header information or have incomplete unit labeling. Furthermore, the system establishes a multi-level quality assessment standard, utilizing techniques such as field integrity checks, data consistency verification, and outlier detection to automatically identify and remove incomplete, damaged, or low-quality data entries. This rigorous screening mechanism not only ensures high data quality but also provides a reliable data source for subsequent information extraction, improving the efficiency and effectiveness of the entire data processing pipeline.

[0031] Finally, the data cleaning agent also possesses powerful standardization capabilities. It can unify data formats from different sources, such as converting temperature units "℃" and "°C" to a unified Celsius representation, or concentration units "g / L" and "gl" to... - ¹” represents a standard format. Simultaneously, addressing inconsistencies in strain aliases and abbreviations, the agent performs preliminary normalization to ensure all data conforms to a predefined standard format. Through these measures, the data cleaning agent not only improves internal data consistency but also enhances compatibility with other external systems. The final output of cleaned and standardized data reduces invalid data entering the information extraction agent, significantly improving the stability of strain entity identification and attribute extraction, providing strong support for scientific research and industrial applications.

[0032] In one embodiment, the information extraction Agent contains the following details: The Information Extraction Agent is the core intelligent component of the preprocessing pipeline, specifically responsible for semantic analysis and information extraction from unstructured or semi-structured data. Particularly in the field of microbiology research, this component can perform in-depth semantic parsing and structuring processing of key information such as strain characteristics, physiological features, culture conditions, and experimental results. Input data is typically cleaned text or tabular data that has undergone preliminary quality control and standardization. The Information Extraction Agent first uses Named Entity Recognition (NER) technology, employing a BERT model fine-tuned for the microbiology domain, to identify important entities in the document and links these entities to industry-standard database IDs, thus ensuring the accuracy and completeness of entity recognition. Furthermore, this agent can handle various forms of entity variants, including abbreviations, aliases, and synonyms, enabling the system to comprehensively capture all relevant entities mentioned in the text.

[0033] After successfully identifying key entities, the information extraction agent further leverages relation extraction to reveal the complex logical relationships between these entities. This step relies on a rule base containing over 200 vertical domain business rules and deep learning techniques, particularly dependency parsing (using the SpaCy framework). In this way, the agent can not only extract subject-verb-object triples but also gain a deeper understanding of the text content, constructing a detailed network of relationships between entities. This network is crucial for building a knowledge graph in the microbiology field, as it clearly demonstrates the associations between strains and their growth environment, physiological characteristics, and experimental results. Attribute extraction, another core function, focuses on identifying and standardizing quantitative and semi-quantitative features related to strains from natural language descriptions, such as converting the temperature range "30-37℃" into a structured format of {"min":30, "max":37, "unit":"StandardUnit"}. This step is particularly important for subsequent data modeling and analysis because it provides high-quality, standardized feature input.

[0034] Ultimately, the output of the information extraction agent is a structured knowledge graph for the microbiology domain, typically stored in Neo4j format. This graph contains detailed information about bacterial strains, their physiological and cultural characteristics, relationships between entities, and attribute constraints. This knowledge graph provides a unified and disambiguating foundation for cross-document and cross-modal data fusion, significantly reducing the risk of ambiguity during the data fusion process and improving the accuracy and consistency of the overall knowledge construction. Furthermore, by providing the multimodal alignment agent with clear and unified descriptions of bacterial strains and their characteristics, the information extraction agent effectively supports downstream tasks such as strain screening, comparative analysis, and knowledge reasoning, enhancing the efficiency and effectiveness of the entire data analysis process.

[0035] In one embodiment, the multimodal alignment agent contains the following details: The Multimodal Alignment Agent represents a significant innovation in the technology field, focusing on constructing semantic alignment and consistent mapping between different data modalities. In microbiology applications, this component is particularly crucial for addressing semantic fragmentation and inconsistencies in reference between text descriptions, experimental images, assay sequences, and structured tables. By automatically aligning and uniformly modeling these features from multiple sources, the Multimodal Alignment Agent lays the foundation for creating a unified multimodal representation of microbial knowledge. Input data includes text entities extracted from literature, image features such as microscopic images or growth curves, gene sequencing data or time-series data from experiments, and structured experimental result tables. First, the Agent uses the CLIP model to calculate the semantic similarity between text descriptions and related images to achieve text-to-image matching; simultaneously, it uses Faster R-CNN to annotate key regions in the images, such as colony or cell morphology areas, ensuring consistency between text descriptions and image content.

[0036] Furthermore, the multimodal alignment agent leverages one of its core functions, sequence-function alignment, to compare various sequence data generated during microbial experiments (such as growth curves and OD value changes) with predefined biological or experimental rules, generating corresponding functional or state annotations. This alignment goes beyond simple pattern matching, involving complex sequence alignment algorithms, rule matching, and model prediction, aiming to establish a direct link between experimental data and biological interpretation, thereby improving data interpretability and usability. For example, different stages of a growth curve can be precisely associated with specific biological meanings, such as a rapid growth phase or a nutrient-limited phase. In addition, to overcome the problems of inconsistent naming or ambiguous referencing between different data sources, the agent also uses graph neural networks (GAT) to jointly model all available multimodal features, generating a unified strain entity ID. This method effectively achieves entity disambiguation and attribute integration across literature, images, and experimental data through cross-modal feature embedding and similarity calculation.

[0037] Ultimately, the multimodal alignment agent outputs a meticulously aligned multimodal knowledge graph containing unified strain entities and their cross-modal associations of physiological characteristics, experimental conditions, and measurement results. This knowledge graph not only resolves the semantic fragmentation and inconsistency in reference inherent in traditional data processing methods but also significantly improves the efficiency and accuracy of microbiological research. By providing a comprehensive platform for integrating and analyzing various types of data, the multimodal alignment agent not only offers researchers powerful tools to explore the complexity of the microbial world but also lays a solid foundation for subsequent knowledge reasoning and decision support systems.

[0038] In some embodiments of the present invention, a text-image alignment module is constructed using a vision-language pre-trained model. The semantic similarity between the text description and the image is calculated using the CLIP model or other cross-modal representation models to achieve matching retrieval. The key regions of the image are visually located using Faster R-CNN or YOLO series object detection models, and the text semantic features and image visual features are fused and aligned.

[0039] The text-image alignment module employs advanced vision-language pre-trained models (such as CLIP, ALIGN, or BLIP) as its foundation. It calculates cross-modal semantic similarity between natural language descriptions (e.g., "colonies are round and milky white," "cells are rod-shaped") and microbial-related images (e.g., petri dish photographs, microscopic images), achieving high-precision image-text matching retrieval (Top-3 accuracy exceeding 90%). Building upon this, an object detection model (such as Faster R-CNN, YOLOv8, or DETR) is introduced to perform fine-grained visual localization of key semantic regions in the image—for example, identifying and selecting colony areas, cell morphology, experimental readings, etc., and extracting their local visual features. Subsequently, the morphological and experimental conditions extracted from the text are fused cross-modal with the visual embeddings of the corresponding image regions (e.g., through cross-attention mechanisms or feature concatenation + projection) to achieve precise alignment at the "word-image region" level, effectively solving problems such as fragmented text-image descriptions and ambiguous referential meanings.

[0040] In some embodiments of the present invention, a sequence-function alignment module is constructed using sequence alignment algorithms or temporal neural networks. The sequence data is matched with predefined business rules using dynamic time warping algorithms or Transformer-based temporal models to generate corresponding functional or state annotations and establish a mapping relationship between the determination sequence and its biological significance.

[0041] Specifically, the sequence-function alignment module constructs a dual-path alignment mechanism based on sequence alignment and temporal modeling for time-series or numerical sequence data generated during experiments (such as OD600 growth curves, pH change trajectories, and metabolite concentration dynamics). On one hand, it employs classic sequence alignment algorithms such as Dynamic Time Warping (DTW) and Longest Common Subsequence (LCS) to flexibly match the original measured sequences with predefined biological pattern templates (such as "exponential growth phase," "stationary phase," and "decay phase"). On the other hand, it utilizes Transformer-based temporal models (such as Temporal Fusion Transformer and Informer) or recurrent neural networks (such as LSTM) to learn the deep dynamic features of the sequences and combines this with a domain rule base (such as "OD value continuously increases within 4–12 hours with a slope > 0.2 → rapid growth phase") for joint inference. Finally, the system automatically generates functional or state annotations with clear biological significance (such as "logarithmic growth phase under adequate nutrition"), establishing an interpretable mapping relationship between the original measured data and its high-level semantic interpretation, significantly improving the usability and scientific value of experimental data.

[0042] In some embodiments of the present invention, a unified entity module is constructed using a variant of a graph neural network, including GraphSAGE, Graph Convolutional Network, Graph Isomorphism Network, or hyperbolic graph neural network, for jointly modeling and embedding multimodal features of text entities, image features, and sequence features, thereby achieving cross-modal entity disambiguation and unified identifier generation.

[0043] Specifically, the entity unification module addresses the issues of inconsistent naming, ambiguous referencing, or missing identification of the same microbial strain across different modalities such as text, images, and sequences. It designs a cross-modal entity unification framework based on a variant of Graph Neural Networks (GNNs). This module first constructs a heterogeneous multimodal graph using features from various modalities (such as strain names identified in text, colony phenotype embeddings extracted from images, and growth kinetic fingerprints calculated from sequences) as nodes. Then, it employs GNN variants such as GraphSAGE, Graph Convolutional Networks (GCNs), Graph Isomorphic Networks (GINs), or Hyperbolic Graph Neural Networks (Hyperbolic GNNs) optimized for hierarchical structures to perform message passing and aggregation on the nodes in the graph, generating a unified multimodal joint embedding representation. Within this embedding space, the system identifies different modal observations pointing to the same real strain using similarity metrics (such as cosine distance or hyperbolic distance) and assigns a unique global entity ID. This mechanism not only enables entity disambiguation across documents, experiments, and modalities, but also supports attribute fusion (such as verifying the consistency between the optimal temperature described in the text and the growth peak temperature in the sequence inference), laying the foundation for building a highly consistent multimodal knowledge graph.

[0044] Furthermore, the Semantic-Enhanced Large Language Model (SELLM) is an advanced natural language processing architecture designed for specific vertical domains. It aims to significantly improve the accuracy and semantic consistency of converting unstructured text to structured data by deeply integrating domain knowledge graphs. This model takes the full text of the original professional document as input. First, it introduces domain ontology knowledge—such as industry-standard classification systems, authoritative terminology databases, or pre-built microbial knowledge graphs—at the low-level language understanding stage. This structured semantic information is then injected into the attention mechanism of pre-trained models like BERT, guiding the model to more accurately identify key concepts and their logical relationships within the context. Simultaneously, for PDF documents containing complex business tables, the system integrates a high-precision table parsing module that automatically converts poorly formatted tables into a clear Markdown format, laying the foundation for subsequent semantic alignment and information extraction. The final output is JSON-LD format text with rich semantic annotations, preserving the integrity of the original content while embedding machine-readable domain semantic tags. For example, in microbial literature processing, this model can ensure that the "culture temperature" field in the experimental parameter table is semantically consistent with the description "strain X grows at 37°C" in the text, and is uniformly mapped to the standard attribute nodes in the knowledge graph.

[0045] Building upon this foundation, the system further introduces an intelligent entity annotation and correction mechanism to address low-confidence entity problems caused by linguistic ambiguity, terminology variations, or OCR recognition errors in practical applications. The core of this mechanism is a dynamic entity annotation algorithm based on an attention mechanism, which can adjust the annotation weights of each candidate entity in real time according to the context and historical annotation patterns, prioritizing high-confidence, frequently occurring, or domain-compliant naming forms. For low-quality entities with a confidence level below 0.7, the system automatically triggers an error correction process: on the one hand, it trains a high-performance XGBoost error correction model (with a measured AUC of 0.93) using a large amount of historically manually verified annotation data to identify and correct typical problems such as misspelled strain names, non-standard unit symbols, and inconsistent aliases; on the other hand, through a dynamic weight adjustment strategy, the system prioritizes high-frequency error types with high impact in the domain, thereby maximizing the error correction benefits with limited computing resources. This mechanism effectively improves the robustness and reliability of entity recognition and prevents noisy data from propagating to downstream tasks.

[0046] In summary, the semantically enhanced large language model and the intelligent entity annotation and error correction mechanism together constitute a closed-loop, self-optimizing information structuring engine. The former achieves "understanding equals structuring" through knowledge graph guidance, while the latter achieves "recognition equals correction" through data-driven feedback learning. Working together, they not only significantly improve the accuracy of extracting structured knowledge from complex professional documents but also significantly enhance the system's adaptability to multi-source, heterogeneous, and low-quality inputs. The final output, semantically annotated JSON-LD text and corrected high-confidence entity lists, provides a high-quality, semantically consistent data foundation for subsequent knowledge fusion, multimodal alignment, and intelligent reasoning, strongly supporting the construction of intelligent knowledge management systems for vertical domains.

[0047] S103 uses a hierarchical human-machine collaborative annotation platform with crowdsourcing, professional, and expert levels to verify, correct, and finalize the initial knowledge graph, thereby outputting a gold standard dataset.

[0048] Specifically, the extraction results of the multimodal knowledge graph are distributed to the crowdsourced annotation layer. Trained annotators process the well-defined strain names and numerical culture conditions, and verify the consistency between the pre-filled high-confidence AI recognition results and the repeated annotations by multiple people, outputting an initial annotation set. The initial annotation set and annotation conflict reports are input into the professional annotation layer, where professionals with microbiology backgrounds perform cluster analysis on similar conflicts and call the domain rule base to intelligently correct samples of moderate complexity, outputting a verified annotation set. High-risk samples in the verified annotation set are submitted to the expert annotation layer, where senior microbiologists conduct a final review and adjudication of the characteristics of rare strains and novel physiological indicators through text-image cross-validation and key feature comparison, outputting a corrected dataset.

[0049] This invention proposes a human-in-the-loop data annotation platform for vertical domain characteristic data, particularly suitable for applications in microbiology. This platform integrates human expertise with artificial intelligence technology to form an efficient and accurate data annotation system specifically designed for the manual verification and professional annotation of multimodal data related to bacterial strains. This data includes at least the strain name, physiological and biochemical characteristics, culture conditions, experimental results, and their interrelationships. Furthermore, a hierarchical architecture design and the application of intelligent auxiliary tools enhance annotation efficiency and quality.

[0050] First, the Human-in-the-Environment (HIU) annotation platform employs a three-tiered annotation architecture. Annotation tasks are allocated and validated layer by layer based on the complexity, specialization, and potential risk level of the microbial feature data. The crowdsourced annotation layer consists of trained annotators responsible for basic feature annotation tasks, such as the location of strain names and numerical culture conditions. This layer ensures the reliability of the annotation results through multi-user repeated annotation and consistency verification mechanisms. The professional annotation layer comprises industry professionals with relevant experience, responsible for annotating samples of moderate complexity. They handle conflict reports using cluster analysis and domain rule base invocation, and recommend optimal annotations based on historical decision-trained models. The expert-level final review is conducted by senior industry experts, focusing on the final review of highly complex samples. Text-image cross-validation and double-blind review are used to ensure the dataset reaches the highest standards of professional accuracy.

[0051] Furthermore, to further improve annotation efficiency and accuracy, the platform integrates an intelligent auxiliary annotation system. This system utilizes machine learning models to provide intelligent support for annotators. The pre-annotated model is trained based on historical data with completed annotations and can automatically generate preliminary feature annotation results for annotators to verify and correct. As data accumulates, the system uses a progressive learning mechanism to iteratively update the pre-annotated model, forming a closed-loop process of "annotation-feedback-optimization." Simultaneously, an active learning mechanism identifies samples that contribute significantly to improving model performance and prioritizes their annotation, reducing unnecessary workload and increasing the annotation coverage of key data and the model's generalization ability. A real-time feedback and continuous improvement mechanism for annotation quality has also been implemented. By dynamically monitoring the annotation process and results, it automatically identifies deviations or inconsistencies and provides improvement suggestions to annotators, ensuring the high reliability of the data foundation for subsequent knowledge base construction and model training.

[0052] In summary, this data annotation platform for the field of microbiology not only improves the accuracy and efficiency of data annotation, but also solves a number of challenges faced in the traditional data annotation process through its innovative hierarchical architecture design and the application of intelligent auxiliary tools, providing strong support for microbial research.

[0053] S104. Based on the difference analysis between the gold standard dataset and the initial knowledge graph, high-frequency error patterns are identified through unsupervised clustering. The amount of manual correction is used as a negative reward signal. Reinforcement learning is used to dynamically adjust the confidence threshold and extraction strategy of the agent's processing pipeline to form a closed-loop feedback.

[0054] Specifically, the input correction dataset is processed by a differential analysis engine to extract manual correction patterns and cluster high-frequency error types. A negative reward function R = -α·N_corrections is constructed based on the number of corrections. A deep Q-network is used to dynamically adjust the entity recognition confidence threshold and extraction strategy parameters of the information extraction agent, and the optimized model configuration is output. When the cumulative newly labeled data reaches a preset scale, incremental training is triggered. Based on the pre-trained language model, single-round fine-tuning is performed using labeled data from the microbiology domain. Batch size and learning rate scheduling strategies are set, and the upgraded model version is output. The upgraded model is then applied to 10% of the labeling tasks through canary release. The manual correction rate and entity recognition accuracy are monitored in real time. When the correction rate decreases by ≥5%, a full deployment is performed and the optimized parameters are fed back, forming a self-evolving system with closed-loop optimization.

[0055] This invention designs a feedback mechanism based on online learning, which continuously optimizes the data preprocessing and verification effects by updating AI model parameters in real time. Based on the core concept of "human-machine collaboration and closed-loop iteration," this mechanism constructs a complete process in microbial feature annotation tasks, from difference identification and model tuning to deployment and verification. This ensures that the system continuously improves its automated processing capabilities and professional accuracy while accumulating high-quality labeled data. The entire mechanism consists of three key steps: feedback data integration, closed-loop model iteration, and performance evaluation dashboards. Each step deeply integrates domain knowledge and machine learning techniques to form an efficient and adaptive intelligent optimization system.

[0056] Feedback Data Integration: This step aims to systematically compare the results of human expert annotation with the AI ​​preprocessing output to extract actionable optimization signals. First, the difference analysis engine receives two types of input: one is the microbial feature annotation results generated by the AI ​​preprocessing module (such as strain taxonomic information, physiological and biochemical parameters, growth condition ranges, morphological descriptions, and key functional characteristics); the other is the "gold standard" annotation data confirmed by expert review. The engine performs structured difference extraction, accurately identifying additions, deletions, or modifications in entity names, key attribute fields (such as optimal growth temperature, pH range, metabolic rate, etc.) and their values. Subsequently, unsupervised clustering methods (such as K-means) are used to summarize high-frequency errors, forming a typical error type library—for example, "missing units (e.g., '37' without ℃ annotation)," "inconsistent terminology abbreviations (e.g., 'E. coli' vs 'Escherichia coli')," and "reversed upper and lower limits of ranges (e.g., 'pH 4–6' incorrectly labeled as '6–4')." Finally, a microbial feature annotation error pattern analysis report is output, identifying the top 10 most frequent errors and their frequency of occurrence, providing data support for subsequent model optimization. Building upon this, the system further introduces a reinforcement learning optimization mechanism: a reward function is designed with negative rewards for the number of manual corrections (formula: ...).R = α N corrections, among which α To adjust the coefficients, key strategy parameters of the preprocessed model are dynamically adjusted using a Deep Q-Network (DQN). For example, the entity recognition confidence threshold is increased from 0.7 to 0.8, or the feature extraction rules are optimized to prioritize high-confidence fields. The optimized strategy undergoes A / B testing to verify its performance, comparing the new and old versions in terms of annotation efficiency, manual correction rate, and overall accuracy. Finally, an optimized intelligent agent parameter configuration file is output to guide the next stage of model updates.

[0057] Model Iteration Closed Loop: After obtaining optimization signals, the system initiates an incremental training pipeline to achieve continuous model evolution. When the cumulative number of newly added, quality-validated microbial annotation data reaches a preset threshold (e.g., 1000), the system automatically triggers a fine-tuning process: based on the existing pre-trained language model (e.g., BERT), a single round of incremental training is performed only on parameters related to microbial feature recognition, preserving the model's general language understanding capabilities while enhancing its professional performance in vertical domains. During training, a batch size of 32 is used, combined with a cosine annealing learning rate scheduling strategy to improve convergence stability and generalization performance. After training, an upgraded microbial feature annotation model is output, whose F1-score on the validation set should be significantly better than the previous version. Subsequently, a safe and controllable online deployment is achieved through a dynamic deployment system: first, a canary release is conducted, applying the new model to approximately 10% of real-world annotation tasks; simultaneously, core metrics are monitored in real time, including manual correction rate, microbial entity recognition accuracy, and key feature field extraction accuracy.

[0058] Effectiveness Evaluation: To comprehensively and transparently present the optimization results, the system has constructed a multi-dimensional visual effectiveness evaluation dashboard. The dashboard includes two core modules: First, the core indicator dashboard, which displays three key indicators in real time—microbial feature annotation efficiency (such as the number of samples processed per unit time), expert correction rate (reflecting the quality of AI output), and cross-modal alignment accuracy (measuring the degree of consistency between text descriptions and experimental images, sequencing data, or tables); Second, the drill-down analysis function, which allows users to drill down by microbial category (such as Gram-positive bacteria vs. Gram-negative bacteria), feature type (such as temperature, pH, metabolites), or data modality (text, image, sequence), etc., to deeply analyze the annotation performance of specific subsets. For example, users can view the optimal salinity annotation accuracy of "Actinomycetes" strains or compare the consistency of "colony morphology" descriptions under different modalities. All data is presented in interactive visualizations: line charts show the trends of various indicators over time or model version evolution; heatmaps visually display the distribution density of various error patterns across different microbial communities or features; scatter plots depict the individual performance differences among annotators in terms of correction frequency and accuracy, assisting team management and training optimization. This dashboard not only provides the technical team with a basis for model iteration but also provides decision support for domain experts and project managers, truly realizing a closed-loop intelligent annotation ecosystem of "data-driven optimization and visualization-enabled governance."

[0059] In summary, the online learning feedback mechanism proposed in this invention, through five technical pillars—differential analysis, reinforcement learning, incremental training, gray-scale deployment, and visual evaluation—constructs a highly adaptive, professional, reliable, and interpretable intelligent annotation and optimization system, significantly improving the automation level and knowledge construction quality of microbial multimodal data processing.

[0060] It should be understood that the above embodiments are only used to explain the technical solutions of the present invention, and are not intended to limit the present invention. Those skilled in the art can easily apply the above methods to other vertical fields that require processing multimodal data, such as, but not limited to, new material research and development, industrial equipment fault diagnosis, and financial compliance review.

[0061] Example 2 Understandably, existing technologies struggle to achieve efficient and standardized collection and processing of multi-source heterogeneous data for specific vertical fields. Taking microbiology research as an example, this type of data typically originates from various carriers such as academic papers, strain preservation records, and experimental reports, involving multiple modalities including text descriptions, structured tables, and charts. Furthermore, key characteristics such as strain names, classification information, physiological characteristics, and culture conditions exhibit inconsistencies in expression and diverse formats across different data sources. The goal is to overcome the limitations of existing tools in integrating cross-data source and multi-type professional documents, and to solve the problems of missing and incomplete extraction of key information caused by differences in data formats and semantic inconsistencies. How to construct an intelligent data processing pipeline with semantic understanding capabilities, targeting the conditional descriptions and cross-modal expression features commonly found in complex professional data such as microbiology literature, such as the strong correlation between strain growth characteristics and environmental factors such as temperature, pH, and nutrient conditions, as well as the correspondence between text descriptions, table values, and chart information, to achieve high-quality cleaning, information extraction, and cross-modal alignment of multimodal data, while overcoming the problems of traditional manual verification relying on subjective experience, difficulty in standardization, and difficulty in meeting industrial-grade repeatability and consistency requirements; How to establish a hierarchical and controllable professional data annotation system, targeting highly specialized fields such as microbial characteristic data, by constructing a three-level annotation architecture of expert, professional and crowdsourced levels, and combining intelligent auxiliary annotation tools constrained by domain knowledge and multi-dimensional quality assessment mechanisms, to ensure the accuracy, consistency and reliability of results such as strain entity identification, attribute annotation and condition association, and solve the data deviation and inconsistency problems caused by differences in professional background and lack of effective quality control in the existing annotation system; How can we introduce a closed-loop optimization mechanism based on online learning, continuously incorporating actual business feedback, such as microbial literature feature extraction and annotation, into the model training and parameter update process, to achieve dynamic linkage and iterative optimization between data, models, and application scenarios, thereby continuously improving the performance, stability, and generalization ability of AI models in complex professional data processing scenarios?

[0062] Based on this, this embodiment provides another method for multimodal information extraction and annotation, applied to the field of microbiology, such as... Figure 2 As shown, the method includes: S1 acquires multi-source heterogeneous data in the field of microbiology, which includes text descriptions, structured tables, and image information, and performs text, table, and image block separation processing on unstructured documents through a layout parsing model.

[0063] Specifically, acquiring and processing multi-source heterogeneous data in the microbiology field, including text descriptions, structured tables, and image information, is a fundamental step in achieving subsequent information extraction and knowledge graph construction. This step, by integrating OCR, PDF parsing, and layout analysis models, separates unstructured documents into text, table, and image blocks, thereby providing high-quality input for semantic alignment and structured modeling of multimodal data.

[0064] This step first uses an OCR engine (such as Tesseract) to extract text from PDF or image-formatted documents, requiring the input image resolution to be no less than 300 DPI to ensure the accuracy of recognizing technical terms and numerical information. Then, a deep learning-based layout parsing model (such as a PDF parser combined with CNN or Transformer architecture) is used to perform structured analysis of the document, identifying and segmenting text paragraphs, table regions, and image blocks. Text blocks are segmented using paragraph boundary detection and font feature analysis; table regions are identified using table detection algorithms (such as TableMaster) to recognize table boundaries and cell structure, and converted to Markdown or CSV format; image blocks are extracted using image segmentation models (such as U-Net), and image classification models are used to preliminarily determine the image type (such as microscopic images, growth curves, etc.).

[0065] The OCR accuracy should reach over 95% to ensure the reliability of key terms (such as strain names and experimental conditions). The F1-score of the layout parsing model should be no less than 0.85 to ensure accurate separation of text, tables, and images. Furthermore, the extraction of image blocks must meet a perceptual hash (pHash) similarity threshold of less than 0.1 to avoid duplicate images interfering with subsequent processing.

[0066] In application scenarios, this step is widely applicable to the automated processing of microbiological literature, experimental reports, and strain preservation records. For example, when processing a two-column formatted microbiology paper, the system can separate cross-page tables, embedded images, and the main text, providing a clearly structured data input for subsequent entity recognition and cross-modal alignment. This step also supports the recognition and processing of multilingual text, ensuring compatibility between international journals and Chinese literature.

[0067] Furthermore, S1 includes: The S11 uses the Tesseract engine to perform OCR text extraction on images with a resolution of at least 300 DPI, thereby improving the accuracy of technical terms and numerical recognition.

[0068] Specifically, in the automated multi-source data acquisition step of this invention, using the Tesseract engine to perform OCR text extraction on images with a resolution of at least 300 DPI is a key technical means to achieve high-quality unstructured document data parsing. Tesseract is currently a mainstream open-source optical character recognition engine. Its deep learning-based OCR model (such as LSTM network) has significant recognition advantages on high-resolution images, and is especially suitable for image text extraction tasks containing complex layouts, small font sizes, special symbols, and professional terminology in professional documents.

[0069] The OCR text extraction process first requires that the image input meets certain resolution requirements. To ensure clear character edges and regular text layout, image preprocessing is typically performed, including grayscale conversion, binarization, denoising, and contrast enhancement, to improve the robustness of OCR recognition. The Tesseract engine uses language models (such as eng.traineddata) to model character sequences during recognition, combining contextual semantic information to accurately recognize technical terms (such as "mesophilic" and "halophilic") and numerical expressions (such as "37°C" and "pH 6.8–7.2"). In this invention, the OCR output is used as input to the information extraction agent for subsequent entity recognition and attribute extraction.

[0070] The recognition accuracy of Tesseract is closely related to image resolution, text font, and layout. In this invention, setting the image resolution to be no less than 300 DPI is based on a comprehensive consideration of OCR recognition accuracy and character positioning precision. Experiments show that when the image resolution is below 300 DPI, the error rate of professional terminology recognition increases significantly, especially in expressions involving combinations of numbers and units (such as "10g / L" and "5–7°C"), where the recognition accuracy can drop by more than 15%. Therefore, this invention ensures that the OCR output text has high semantic integrity and structural consistency by setting a resolution threshold.

[0071] This step is primarily used to process the text content in academic papers, experimental reports, and supplementary image materials in the field of microbiology. For example, when documents contain multi-page tables or embedded images, OCR extraction can restore the text information in the image to structured text, providing foundational data for subsequent entity recognition and multimodal alignment. Furthermore, this step also supports the extraction of text information embedded in visual data such as microscopic images and colony morphology diagrams, thereby achieving preliminary fusion of image and text information.

[0072] S12 uses a PDF parser to analyze the layout structure of unstructured documents, separating text paragraphs, table areas, and image blocks to provide a foundation for subsequent cross-modal feature association.

[0073] Specifically, in some implementations, analyzing the layout structure of unstructured documents using a PDF parser is a key preprocessing step in the multimodal information extraction and annotation system of this invention. This technology is based on document structure modeling and content segmentation algorithms, aiming to effectively separate text paragraphs, table regions, and image blocks, providing a structured foundation for subsequent cross-modal feature association. This step is primarily aimed at documents in specialized fields such as microbiology literature, whose layouts typically include two-column layouts, multi-page tables, embedded images, and complex annotations. Traditional OCR and text extraction methods struggle to accurately distinguish between different content types, leading to contextual breaks and semantic misalignments in information extraction.

[0074] In its specific implementation, the PDF parser employs a deep learning-based layout analysis model (such as PDFPlumber combined with YOLOv5 or LayoutLM) to divide the document page into regions. The model first performs image processing on the PDF page, extracting text blocks, table cells, and image bounding boxes. It then uses a combination of semantic and geometric features to determine the content type of each region. Text paragraphs are clustered based on features such as font size, line spacing, and paragraph continuity; table regions are determined by cell alignment, row and column structure, and content density; and image blocks are classified based on content type detection (e.g., whether they are charts, microscopic images, or experimental flowcharts).

[0075] The system requires the PDF parser to achieve a text recognition accuracy of no less than 95%, a table structure restoration error rate of less than 5%, and an image localization recall rate of greater than 90%. To improve parsing performance, the system prioritizes PDF input with a resolution of at least 300 DPI and employs the Tesseract OCR engine for text extraction, while combining it with PDFMiner for coordinate information extraction to ensure the preservation of the relative positional information between text and images. Furthermore, the system supports language detection and segmentation for multilingual documents (such as mixed Chinese and English) to adapt to international microbiological literature data sources.

[0076] S2. Construct a cross-modal alignment model based on graph neural networks. Use the model to jointly model the separated text entities, image features and sequence data, calculate the semantic similarity of cross-modal features to achieve text and image matching, and establish the mapping relationship between experimental data and functional description through sequence comparison algorithm.

[0077] Specifically, in the multimodal data processing flow of this invention, constructing a cross-modal alignment model based on graph neural networks (GAT) is a key step in realizing the semantic fusion of text, image, and sequence data. This step models the semantic relationships between different modalities through graph structures, thereby solving the problems of cross-modal information fragmentation, ambiguous referencing, and insufficient entity disambiguation capabilities in traditional methods.

[0078] The model first embeds text entities, image feature vectors, and sequence data (such as growth curves and OD value sequences) into a unified semantic space. Text entities are encoded using a fine-tuned BERT model to extract their contextual semantic representation; image features are extracted by a pre-trained CNN model (such as ResNet-50) and further localized using Faster R-CNN to enhance the matching accuracy with the text description; sequence data is modeled temporally using LSTM or Transformer structures to capture its dynamic changes. Subsequently, these heterogeneous features are input into a Graph Attention Network (GAT) to construct a heterogeneous graph structure containing nodes (entities, image regions, sequence fragments) and edges (semantic associations, conditional dependencies). GAT uses a multi-head attention mechanism to weight the semantic similarity between nodes; its core formula is:

[0079] in, Indicates the first Layer nodes eigenvectors, The weight matrix is ​​a learnable matrix. For nodes and Attention coefficient between them This is a non-linear activation function. Through this mechanism, the model can dynamically adjust the fusion weights of features from different modalities, achieving cross-modal semantic alignment.

[0080] The GAT model employs a 3-layer stacked structure, with each layer containing 8 attention heads and a hidden layer dimension of 256. The embedding dimension for both text and images is uniformly 768 to ensure the alignment accuracy of cross-modal features in the semantic space. During model training, a negative sampling strategy is used, with the loss function being the triplet loss, to maximize the similarity of positive sample pairs and the dissimilarity of negative sample pairs.

[0081] In practical applications, this step is mainly used for the integration of multimodal features of microbial strains. For example, it matches the text description "colonies are round and milky white" with the corresponding microscopic image region, or maps the changes in the OD value of the growth curve with textual functional descriptions such as "rapid growth stage" and "nutrient-limited stage." Through this model, the system can achieve semantic consistency verification of cross-modal information, significantly improving the structured quality and data integrity of strain features.

[0082] Furthermore, S2 includes: S21. The CLIP model is used to calculate the semantic similarity between text descriptions and microbial-related images, achieving Top-3 matching from text to image.

[0083] Specifically, in the text-image alignment submodule of the multimodal alignment agent, the CLIP (Contrastive Language-Image Pretraining) model is used to calculate the semantic similarity between text descriptions and microbial-related images, which is a key technical means to achieve cross-modal information fusion. This step identifies the image content most relevant to the text description by matching text descriptions and image features in a unified semantic space, achieving a Top-3 text-to-image match with an accuracy of approximately 90%.

[0084] The CLIP model employs a dual-encoder structure, embedding text and images separately. The text encoder, based on the Transformer architecture, converts the input natural language description (e.g., "colonies are round and milky white") into a fixed-dimensional semantic vector. The image encoder, based on ResNet or Vision Transformer (ViT), extracts visual feature vectors from microscopic or culture images. In this invention, the text description typically consists of strain features extracted by the information extraction agent, while the images are derived from experimental images collected in step one or illustrations from literature. By calculating the cosine similarity between the text embedding vector and the image embedding vector, the system can score each text-image pair and select the top-3 images with the highest similarity as the matching results.

[0085] The input length of the text encoder in the CLIP model is limited to [number]. 1 token, image encoder input size is 0. and adopt The system uses RGB channel format. In this system, text descriptions are standardized to remove noise and ambiguous expressions, ensuring clear input semantics. Images are then processed through OCR and layout parsing, extracting key regions (such as colony areas and cell morphology areas) for encoding to improve matching accuracy. The Top-3 accuracy of the matching results is approximately 90%, indicating that this method has high reliability in the semantic alignment task between microbial images and text descriptions.

[0086] S22 uses Faster R-CNN to annotate key regions in the image, such as colony regions, cell morphology regions, or experimental reading regions, to assist in cross-modal alignment.

[0087] Specifically, in the multimodal data processing flow of this invention, using Faster R-CNN to annotate key regions (such as colony regions, cell morphology regions, or experimental reading regions) in the image is an important technical means to achieve cross-modal alignment. This step identifies visual regions in the image that are semantically consistent with the text description through an object detection model, providing structured visual input for subsequent image-text semantic matching and knowledge graph construction.

[0088] Faster R-CNN is a region-based convolutional neural network. Its core architecture includes a Region Proposal Network (RPN) and a Fast R-CNN classification and regression module. The RPN is responsible for generating candidate regions (Region Proposals), while Fast R-CNN classifies these regions and performs bounding box regression. In this invention, Faster R-CNN is used to identify key regions in images that are related to textual descriptions, such as the image region corresponding to "colonies are round and milky white," or the microscopic image portion referred to by "cells are rod-shaped." The model learns visual representations of features such as colony morphology, cell structure, and experimental readings by introducing a microbial image annotation dataset during the training phase, thereby achieving high-precision localization of the target region during the inference phase.

[0089] The Faster R-CNN detection confidence threshold is set to >0.7 to ensure that only high-confidence regions are labeled. The model uses ResNet-50 as the backbone network, and the input image size is uniformly 800×1400 to meet the high-resolution requirements of microbial images. During training, the standard anchor box configuration of the COCO dataset is used, and the anchor size and proportion are fine-tuned for targets such as colonies and cells to improve detection accuracy. Furthermore, the model's output bounding boxes are evaluated using the Intersection over Union (IoU) metric to ensure that the overlap between labeled regions and ground truth targets reaches >0.5, thus meeting the accuracy requirements for cross-modal alignment.

[0090] This step primarily processes image data from microbiology literature, such as microscopic images, petri dish images, and experimental measurement images. By semantically matching text descriptions with key regions in the images, the system can automatically establish semantic relationships between text and images; for example, aligning "colony diameter 2–3 mm" with the size measurement results of the corresponding region in the image. This technology significantly improves the consistency and interpretability of cross-modal data, providing crucial support for constructing a unified microbial knowledge graph.

[0091] S3 employs a domain knowledge-guided entity unification algorithm to disambiguate the multi-level naming structure of the same microbial entity in cross-modal data, generating a unified entity identifier.

[0092] Specifically, in some implementations, this invention employs a domain knowledge-guided entity unification algorithm to disambiguate the multi-level naming structure of the same microbial entity in cross-modal data, generating a unified entity identifier. This step is a crucial link in multimodal data fusion and knowledge graph construction. Its core technology lies in using graph neural networks (GAT) to jointly model entities in text, image, and sequence data, achieving semantic alignment and unique identifier generation for cross-modal entities.

[0093] The algorithm first extracts features from microbial entities derived from text, images, and tables. Text entities are identified using a BERT-based Named Entity Recognition (NER) model and linked to standard bacterial strain databases (such as ATCC and DSMZ), mapping non-standard names like aliases and abbreviations to unified entity IDs. In image data, Faster R-CNN is used to detect key visual regions such as colony morphology and cell structure, extracting their visual feature vectors. Table and sequence data undergo rule matching and numerical normalization to extract structured attribute values. Subsequently, the GAT model embeds these multimodal features into a unified graph structure and calculates semantic similarity between entities using a heterogeneous graph attention mechanism, with the formula: ,in , Representing entities respectively , The multimodal feature vector.

[0094] The entity unification algorithm requires a text-to-image matching accuracy of no less than 90%, and sets the entity recognition confidence threshold at 0.7. Entities below this threshold will enter the error correction process. Simultaneously, the system supports multi-level disambiguation of fields such as strain name, classification information, and growth conditions to ensure consistent entity representation across different modalities.

[0095] The technical effect of this step is to significantly improve the efficiency and accuracy of cross-modal data integration, providing a high-quality, traceable entity identification system for subsequent knowledge graph construction and intelligent retrieval. By introducing domain knowledge graphs and graph neural networks, the system can effectively solve the identification conflict problem of microbial entities in multimodal data caused by naming differences and semantic ambiguity, thereby achieving data structuring and semantic unification, laying the foundation for building a highly reliable microbial knowledge base.

[0096] Furthermore, S3 includes: S31 uses a graph neural network (GAT) to jointly model text entities, image features, and sequence features to generate a unified strain entity ID.

[0097] Specifically, in the multimodal alignment agent of this invention, the core technical means to achieve cross-modal entity unification and semantic alignment is to jointly model text entities, image features, and sequence features using a graph attention network (GAT). This step is technically implemented based on a heterogeneous graph structure, mapping data nodes from different modalities (such as strain names in text, colony morphology features in images, and growth curve data in sequences) into a unified embedding space, thereby generating a uniquely identified strain entity ID.

[0098] In some implementations, GAT introduces an attention mechanism to perform weighted aggregation of features from different nodes in the graph. Its core formula is:

[0099] in, Represents a node In the Layer embedding vectors, The weight matrix is ​​a learnable matrix. For nodes With adjacent nodes Attention coefficient between them It is a non-linear activation function (such as ReLU). Represents a node The set of adjacent nodes. This mechanism allows the model to dynamically focus on features that are more semantically relevant to the current node when processing heterogeneous modal information, thereby improving the accuracy of cross-modal alignment.

[0100] In the specific implementation, semantic embeddings of text entities are extracted using a fine-tuned BERT model, while image features are extracted using visual embeddings from ResNet-50. Sequence features (such as growth curves) are encoded into temporal embeddings using LSTM or Transformer. All types of embedding vectors are input as node features into the GAT model within a unified graph structure. The model fuses cross-modal information through a multi-hop attention mechanism, ultimately outputting a unified strain entity embedding representation. This embedding representation generates a unique strain entity ID through clustering algorithms (such as DBSCAN) or hash encoding, achieving entity disambiguation and unified modeling across modal data.

[0101] This step plays a crucial role in the present invention, and its technical effects are reflected in: significantly improving the consistency and semantic alignment accuracy of cross-modal data, reducing entity recognition errors caused by naming inconsistencies or modal fragmentation, and providing a high-quality, structured data foundation for subsequent knowledge graph construction and manual verification.

[0102] S32, based on multimodal feature embedding and similarity calculation, solves the problem of inconsistent naming or ambiguous reference of strains in different literature, images and experimental data.

[0103] Specifically, in the multimodal alignment agent of this invention, the step of multimodal feature embedding and similarity calculation is the core link to achieve cross-modal data semantic alignment and entity disambiguation. This step solves the problem of inconsistent naming or ambiguous referencing of microbial strains in different documents, images and experimental data by fusing semantic representations of text, images and sequence data, thereby constructing a unified and traceable strain knowledge graph.

[0104] This step first involves multimodal embedding modeling of features from text, image, and sequence data. In the text modality, a domain-knowledge-enhanced BERT model is used to semantically encode strain names, classification information, and physiological characteristics, generating fixed-dimensional embedding vectors. In the image modality, Faster R-CNN is used to detect and extract features from key regions in microscopic or colony images, generating image embeddings. Sequence data (such as growth curves and OD value changes) are then processed by a time series encoder (such as LSTM or Transformer) to extract their dynamic features and generate... Furthermore, a heterogeneous graph structure is constructed by jointly modeling the three types of embeddings using a graph attention network (GAT), where nodes represent entities of different modalities and edges represent cross-modal semantic associations.

[0105] In terms of similarity calculation, the system uses cosine similarity as the core metric to calculate the matching degree between text and image embeddings: .

[0106] When the similarity is greater than a set threshold (e.g.) When the sequence data and text descriptions are aligned, the system identifies them as cross-modal expression of the same strain. Furthermore, to ensure consistency between key experimental features and text descriptions, the system employs a combination of rule-based pattern matching and deep learning model prediction.

[0107] S4, based on cross-modal alignment results and unified entity identifiers, constructs a microbial knowledge graph containing strain entities, physiological characteristics, experimental conditions, and cross-modal relationships, and verifies the logical correlation between text descriptions, image information, and experimental data through a consistency verification mechanism.

[0108] Specifically, in the steps of this invention, a microbial knowledge graph containing strain entities, physiological characteristics, experimental conditions, and cross-modal relationships is constructed based on cross-modal alignment results and unified entity identifiers. A consistency verification mechanism is then used to verify the logical correlation between text descriptions, image information, and experimental data. This step is the core component of the entire system's knowledge fusion and structured modeling, and its technical implementation is based on heterogeneous graph modeling and semantic consistency verification strategies.

[0109] The system first uses a unified strain entity ID generated by the multimodal alignment agent as the unique identifier for the graph nodes, ensuring that the same strain in cross-modal data (such as text, images, and tables) has a unique mapping in the knowledge graph. Subsequently, the system cleanses and extracts structured information output by the agent (such as strain classification, growth temperature, pH value, metabolic capacity, etc.) as attributes and relation edges of the graph, constructing a graph database based on Neo4j. Node types include strain entities, physiological properties, and experimental conditions, while edge types cover semantic relationships such as "has property" and "under condition." During construction, the system employs a graph neural network (GAT) to jointly embed cross-modal features to enhance the semantic association between entities.

[0110] This step is widely applicable to the integrated analysis of microbiological literature, experimental reports, and image data. For example, in tasks such as strain screening and functional prediction, the system can quickly retrieve the physiological performance of specific strains under different experimental conditions based on a knowledge graph, assisting researchers in comparing strain characteristics and inferring functions. Furthermore, this graph can also serve as a structured knowledge source for subsequent AI model training, improving the model's generalization ability in strain identification and feature prediction.

[0111] This step achieves semantic unification and logical verification of multimodal data, significantly improving the integrity and credibility of the knowledge graph. Through the consistency verification mechanism, the system can automatically identify and mark logical conflicts in cross-modal information (such as inconsistencies between textual descriptions of colony morphology and image recognition results), thereby reducing the burden of manual verification and improving the automation and professionalism of data processing.

[0112] Furthermore, S4 includes: S41 uses Fleiss' Kappa consistency check to perform statistical analysis on annotation results from multiple people, automatically filtering out low-quality annotators.

[0113] Specifically, through Fleiss' Kappa consistency check (threshold) Statistical analysis of annotation results from multiple annotators, and automatic filtering of low-quality annotators (accuracy <80%), is a key step in building a highly reliable data annotation workflow. This step aims to address inconsistencies in microbial characteristic data annotation caused by differences in the professional backgrounds, misunderstandings, and non-standard operations of annotators, thereby improving overall annotation quality and system stability.

[0114] Fleiss' Kappa is a statistical metric used to measure annotation consistency among multiple annotators, suitable for scenarios with multiple categories and multiple annotators. Its calculation formula is:

[0115] in, Indicates the observed proportion of consistency. This indicates the expected consistency ratio. In this invention, this metric is used to evaluate the consistency of annotations by crowdsourced annotators on the same microbial characteristics (such as strain name, growth temperature, pH value, etc.). The system assigns the same annotation task to multiple annotators (usually 3 people) and collects their annotation results. Subsequently, the Kappa value of all annotators on each feature field is calculated. If the overall Kappa value is... If the labeling consistency is within the acceptable range, the task is considered to have reached an acceptable level; if it is below the threshold, further verification or manual review mechanisms will be triggered.

[0116] Furthermore, the system incorporates the individual accuracy rates of the annotators for quality assessment. The annotator's accuracy rate is defined as the proportion of consistency between their annotation results and those of the experts, calculated using the following formula:

[0117] in, For the annotation staff The number of correctly labeled samples, This represents the total number of samples annotated by the annotator. When an annotator's accuracy rate is less than 80%, the system will automatically pause their task assignment and mark their annotation results as data awaiting review, to be verified a second time by professional annotators.

[0118] By quantifying annotation consistency and annotator accuracy, the system enables automated quality control and annotator management in large-scale crowdsourced annotation scenarios. Its innovation lies in combining statistical consistency indicators with annotator performance evaluation to form a dynamic annotation quality monitoring mechanism. This significantly improves the authority and repeatability of annotation results while ensuring annotation efficiency. This mechanism provides a high-quality initial data foundation for subsequent professional and expert-level annotation, and is a crucial guarantee for constructing the gold standard dataset.

[0119] S42, when constructing the gold standard dataset, a double-blind review mechanism was adopted (and the final annotation results were locked to ensure that the error rate was ≤10.5%).

[0120] Specifically, in constructing the gold standard dataset, employing a double-blind review mechanism (independent judgment by two experts) and locking in the final annotation results is a crucial step in ensuring the data's authority and consistency. This mechanism, by introducing independent judgment from senior microbiology experts, effectively avoids annotation errors caused by the subjective bias or experience limitations of a single expert, thereby controlling the error rate of the final dataset to within a certain range. Within this range, it meets the needs of constructing high-precision knowledge graphs.

[0121] This step is executed based on the expert-level final review module of the hierarchical annotation platform. The specific operation process is as follows: First, the system automatically pushes high-risk samples (such as rare strain characteristics, novel physiological indicators, or data with cross-modal conflicts) that have undergone crowdsourcing and professional-level annotation to the expert annotation layer. Two experts, in a completely isolated annotation environment, independently judge the same data sample based on unified annotation specifications and knowledge graph-assisted tools. The system provides multimodal validation support, such as text-image cross-validation (e.g., matching colony morphology descriptions with microscopic images) and key feature comparison (e.g., the similarity of the strain's 16S rRNA sequence with the standard reference library >99%), to enhance the objectivity and accuracy of expert judgment.

[0122] This step strictly adheres to standards of annotation consistency and authority. Expert annotation results must meet a cross-modal matching rate of >99%, and the final annotation result must pass a double-blind consistency check. If the annotation results of two experts are consistent, the system automatically locks that annotation as the final version; if there is a disagreement, the system will call the BERT embedding and image feature fusion model for auxiliary decision-making, and combine it with a decision tree voting mechanism to generate a recommendation result for expert review. This process ensures the authority and traceability of the annotation results.

[0123] This step is widely used in scenarios such as building microbial strain databases, validating the characteristics of new strains, and integrating cross-literature data. For example, when constructing a knowledge graph of strain growth conditions, experts need to conduct a final review of key attributes such as "optimal growth temperature" and "pH range" to ensure their consistency with experimental images, literature descriptions, and sequence data. This mechanism is particularly suitable for handling complex data with cross-modal ambiguity, strong condition dependence, or the presence of terminology variations.

[0124] This step is technically effective, and its core value lies in controlling the error rate of data annotation within a certain range through an expert double-blind review mechanism. This resulted in the construction of a highly authoritative and consistent gold standard dataset. This dataset not only provides high-quality supervision signals for subsequent model training, but also lays a solid foundation for the continuous optimization of cross-modal alignment, knowledge graph construction, and automated annotation systems.

[0125] Also includes: S5 deploys a semantically enhanced large language model, injects domain ontology knowledge into the BERT attention layer, converts PDF tables into Markdown format, and outputs semantically annotated JSON-LD format text.

[0126] Specifically, in some implementations, the semantically enhanced large language model deployed in this invention focuses on improving the model's ability to understand professional domain texts and its information extraction accuracy by injecting domain ontology knowledge (such as industry standard classification systems and business terminology databases) into the BERT attention mechanism. The input to this model is unstructured text data, such as PDF documents like academic papers and experimental reports in the field of microbiology. Its processing flow includes the structured transformation of PDF tables and semantic annotation output.

[0127] Specifically, PDF table parsing employs a combination of OCR and layout analysis. First, a PDF parser (such as PDFMiner or PyMuPDF) extracts the table regions. Then, an OCR engine (such as Tesseract) performs text recognition. To improve the robustness of table recognition, the OCR input image resolution must be at least 300 DPI to ensure accurate recognition of technical terms and numerical values. The recognized table content is then cleaned and structured using a rule engine and a contextual semantic model, ultimately converting it into Markdown format for easier subsequent processing and display.

[0128] During the semantic annotation phase, the model outputs text in JSON-LD format, where each entity and its attributes are accompanied by semantic identifiers (such as `@id`, `@type`), conforming to the W3C JSON-LD 1.1 standard. The BERT model introduces embedding vectors from the domain knowledge graph into the attention layer, enhancing its ability to recognize technical terms, conditional dependencies, and semantic associations between entities through a knowledge-guided mechanism. For example, when processing "strain A grows at 30-37℃", the model can identify "30-37℃" as a temperature range attribute, normalize it to `{"min": 30, "max": 37, "unit": "℃"}`, and associate it with the entity ID of strain A.

[0129] Furthermore, this step plays a crucial role in the entire system, acting as a bridge between preceding and subsequent steps. On one hand, it transforms tabular information in unstructured documents into structured text, providing high-quality input for subsequent information extraction and multimodal alignment. On the other hand, through semantic annotation output, it lays the foundation for constructing microbial knowledge graphs (such as Neo4j), ensuring semantic consistency of entities and attributes in cross-modal data fusion. This technical solution not only improves the quality of data structuring but also significantly enhances the model's ability to capture semantics within specialized domains, providing reliable data support for subsequent automated annotation and closed-loop optimization.

[0130] The multimodal information extraction and annotation method of this invention is applied to the field of microbiology to achieve semantic alignment and consistency verification of cross-modal data, thereby improving the structured quality and annotation efficiency of multi-source heterogeneous microbial data.

[0131] It should be understood that the above embodiments are only used to explain the technical solutions of the present invention, and are not intended to limit the present invention. Those skilled in the art can easily apply the above methods to other vertical fields that require processing multimodal data, such as, but not limited to, new material research and development, industrial equipment fault diagnosis, and financial compliance review.

[0132] Example 3 This invention proposes another method for multimodal information extraction and annotation, which can be used in high-end manufacturing and the industrial internet, and applied to the full lifecycle management of industrial equipment, such as... Figure 3 As shown, the method includes: S10 acquires industrial equipment data from multiple heterogeneous data sources, including industrial database interfaces, equipment technical manual PDF documents, real-time sensor log sequence data, flaw detection images, and business monitoring data, and outputs standardized industrial multimodal data after preprocessing.

[0133] Specifically, for the entire lifecycle management scenario of industrial equipment, the multi-source data automated acquisition system constructed in this invention achieves comprehensive integration of industrial data through three core modules. The input layer interfaces with standardized APIs of the Enterprise Equipment Management System (EAM), Manufacturing Execution System (MES), and third-party industrial internet platforms to acquire structured data such as equipment operating parameters, maintenance records, and fault logs in real time. Simultaneously, it collects PDF documents such as equipment technical manuals, maintenance procedures, and fault diagnosis guidelines, as well as PNG / TIFF format files such as equipment structural drawings, flaw detection images, and infrared thermal imaging samples. It also accesses raw business data streams such as real-time sensor logs and SCADA system time-series monitoring data. This system is particularly designed for the high temporality, strong correlation, and multimodal characteristics of industrial data, establishing unified data access standards and secure transmission mechanisms to ensure that data across the entire chain, from equipment-level sensors to enterprise-level management systems, is collectable, usable, and controllable.

[0134] The processing workflow adopts a three-stage architecture of "structured data import - unstructured data parsing - semantic enhancement". First, basic equipment information is acquired in batches via a standardized API interface. Equipment operating parameters, maintenance records, and fault logs in JSON / XML format are converted into unified structured tables (core fields include device_id, equipment_category, operational_indicator, maintenance_record, etc.) and stored in an InfluxDB industrial time-series database or a PostgreSQL relational database, supporting second-level data writing and efficient querying. Second, the Tesseract engine (resolution ≥300 DPI) is used to perform OCR recognition on the equipment technical manuals. Based on regular expression matching from an industrial rule base, key technical parameters such as equipment model (e.g., "6-Axis-Robot-3000"), fault code (e.g., "ERR-5021"), and operating parameter thresholds (e.g., "temperature limit: 85℃") are automatically extracted. The LayoutLM model is then used to analyze the document layout, intelligently separating multimodal blocks such as text paragraphs, equipment parameter tables, structural diagrams, and maintenance flowcharts. At the technical enhancement level, a semantically enhanced LLM for the industrial equipment field was developed. Manufacturing ontology knowledge such as the ISO standard classification system and equipment fault terminology library was injected into the BERT attention layer. The PDF parameter table was converted into standard Markdown format. For low confidence entities (<0.7), an XGBoost error correction model (AUC 0.93) was trained based on historical equipment annotation data to prioritize the correction of high-frequency errors such as spelling errors in equipment models and non-standard fault code naming.

[0135] The data acquisition system ultimately outputs a structured industrial database table, standardized manufacturing text corpus encoded in UTF-8 (with initial entity annotations), and 2048-dimensional equipment detection image feature vectors extracted using ResNet-50, forming multimodal data assets that comply with industrial data security standards. In application scenarios, this system provides high-quality data input for building an equipment fault knowledge base, supporting cross-modal intelligent retrieval and diagnosis. When technicians input fault descriptions such as "bearing abnormal noise," the system can automatically match the corresponding vibration spectrum diagram, abnormal area in infrared thermal imaging, and standard handling sections in the maintenance manual, reducing the traditional troubleshooting time of several hours to minutes. This significantly reduces equipment operation and maintenance costs and unplanned downtime, achieving a digital transformation from reactive maintenance to predictive maintenance.

[0136] S20: Construct an intelligent agent processing pipeline consisting of a data cleaning agent, an information extraction agent, and a multimodal alignment agent to automatically clean the standardized industrial multimodal data, extract equipment fault information, and perform cross-modal semantic alignment, outputting an initial industrial equipment knowledge graph.

[0137] Specifically, in the field of high-end manufacturing and industrial internet, this invention constructs a complete intelligent data processing and knowledge graph construction system for the whole life cycle management of industrial equipment. It realizes the automated transformation from raw equipment data to a structured fault knowledge base through three core agents.

[0138] As the first processing node in the pipeline, the data cleaning agent bears the heavy responsibility of quality control and standardization of multimodal industrial data. Inputs include equipment technical manuals (PDF documents), real-time sensor logs, flaw detection images, and SCADA system data tables. In the deduplication function, the SimHash algorithm (identifying duplicates with a similarity >95%) is used for equipment maintenance records and technical documents to identify and merge duplicate inspection reports or similarly versioned maintenance guides. For flaw detection images and equipment status photos, perceptual hashing (pHash) is used to eliminate redundant inspection images. This agent uses a dual mechanism—combining text similarity algorithms such as cosine similarity and edit distance with unique identifiers such as equipment codes and UUIDs—to accurately identify duplicate equipment monitoring records and maintenance logs, ensuring the uniqueness of each record in the industrial dataset. In the quality filtering stage, at the text level, technical documents lacking key equipment parameters or incomplete metadata (such as maintenance records without equipment model and serial number markings) are removed; at the image level, blurry flaw detection images with a resolution lower than 72 DPI or PSNR < 20dB are removed; at the table level, sensor data tables lacking header descriptions or with incomplete unit markings are removed. Agent has established a rigorous industrial data quality assessment system. This system verifies the completeness of equipment operating parameters through field integrity checks, ensures consistent coding for the same equipment across different systems through data consistency checks, and identifies invalid data such as sensor jumps through outlier detection. For equipment technical documents, the system checks the completeness of core content, the validity of key process parameters, and the standardization of source information; for industrial images, it evaluates sharpness, resolution, and color fidelity; for structured time-series data, it verifies the rationality of numerical ranges, unit consistency, and logical verification pass rates, among other equipment business indicators. In standardization, the temperature unit "°C" is unified to "Celsius," the pressure unit "MPa / bar" is unified to the International System of Units (SI), the date format is forcibly converted to the ISO 8601 standard, and equipment model aliases are uniformly mapped to standard codes. This agent establishes a unified storage format covering all industrial data types, ensuring the consistency and compatibility of equipment data as it flows across systems and platforms, outputting high-quality standardized industrial data, and significantly reducing invalid calculations caused by duplicate equipment records or low-quality data in subsequent agents.

[0139] The information extraction agent, as the core intelligent component of the preprocessing pipeline, deeply integrates deep learning models and industrial knowledge bases to accurately identify key elements throughout the equipment's lifecycle. Inputs include cleaned equipment technical text, maintenance records, and sensor logs. In named entity recognition, a BERT model fine-tuned based on industrial equipment corpus is used to construct a three-level entity classification system encompassing core equipment, key components, and operating parameters. This accurately identifies equipment models such as "6-Axis-Robot-3000," core components such as "spindle bearing," and monitoring parameters such as "vibration amplitude," and links to industry standard database IDs (e.g., "equipment alias A" → standard ID: 1234). It also handles abbreviations, aliases, and synonym variations of equipment codes to ensure the completeness and accuracy of entity recognition. In relation extraction, over 200 vertical domain rules (such as "Equipment A triggers fault C under temperature exceeding limit condition B" → [Equipment A] - [Operating Condition] → [Condition B] - [Triggers] → [Fault C]) are integrated with a deep learning model based on dependency parsing to accurately extract complex logic such as the compositional relationship between equipment and components, the operating condition relationship between equipment and the environment, and the causal relationship between parameters and states, laying the relational foundation for building an equipment fault knowledge graph. In attribute extraction, "Operating temperature range 30-85℃" is normalized to {"min":30, "max":85, "unit":"Celsius"}, and "Vibration threshold <0.5mm / s" is parsed to {"threshold":0.5, "unit":"mm / s", "condition":"less_than"}. Uncertain expressions such as range values, approximate values, and conditional limits of equipment parameters are automatically processed, and unit standardization and numerical verification are performed. The agent outputs a structured device knowledge graph (Neo4j format), which includes device entities, component composition, operating parameters, fault records, and attribute constraints. This provides a unified entity basis for multimodal alignment after disambiguation, avoiding matching ambiguities caused by naming differences in different documents and systems.

[0140] The multimodal alignment agent, a key technological innovation of this invention, achieves a breakthrough in semantic-level fusion of text, image, and time-series data from industrial equipment. Inputs include equipment text descriptions, detection image features, and sensor sequence data. In text-image alignment, the CLIP model is used to calculate the semantic similarity between text descriptions such as "abnormal bearing wear marks" in the technical manual and flaw detection images, achieving a 90% Top-3 matching accuracy in cross-modal retrieval. Faster R-CNN is used to annotate key defect areas such as cracks and wear in the images (IoU > 0.7), fusing and aligning defect features and operating conditions in the text with visual features in the images, avoiding information separation between equipment fault descriptions and detection images. In sequence-function alignment, a time-series pattern matching algorithm compares vibration, temperature, and pressure sequence data collected by sensors with a predefined equipment operation rule base, generating functional annotations such as "startup phase," "overload state," and "abnormal vibration" (matching confidence < 1e-5), establishing a mapping relationship between raw sensor data and equipment operating status, and improving the interpretability of industrial time-series data. In entity unification, a graph neural network (GAT) is used to fuse device text descriptions, detection image features, and sensor sequence patterns to generate a unified device entity ID. This resolves the issue of inconsistent coding or ambiguous designations of the same device in technical manuals, inspection photos, and maintenance records, achieving cross-modal device disambiguation and attribute integration. The agent outputs an aligned multimodal device knowledge graph, uniformly representing the device entity and its cross-modal associations' operating parameters, fault characteristics, and detection results.

[0141] Semantic Enhanced Large Language Model: Develop a dedicated LLM for the industrial equipment field, inject equipment manufacturing ontology knowledge (such as the ISO 81346 standard classification system and GB equipment fault terminology library) into the BERT attention layer, automatically convert complex parameter tables in PDF technical manuals into structured Markdown format, and output equipment semantic annotation text in JSON-LD format to ensure the consistency between table parameters and manual text content.

[0142] Intelligent entity annotation and error correction mechanism: For low-confidence device entities (confidence <0.7), an XGBoost error correction model (AUC 0.93) is trained based on historical device annotation data. The annotation weights are dynamically adjusted to prioritize the handling of high-frequency errors (such as the misspelling of the device model "DCS-100" as "DSC-100", or non-standard fault code naming, etc.). The corrected list of device entities is output to continuously improve the accuracy of the industrial knowledge graph.

[0143] These three agents work together to automatically transform scattered equipment technical documents, real-time monitoring data, and detection images into a structured, aligned, and reasonable industrial knowledge graph. This provides a high-quality data foundation for equipment fault diagnosis, predictive maintenance, and repair decision support, driving the evolution of the manufacturing industry from a digital to an intelligent, knowledge-driven operation and maintenance model.

[0144] S30 utilizes a hierarchical human-machine collaborative annotation platform with crowdsourcing, professional, and expert levels to perform equipment fault verification, correction, and final review on the initial industrial equipment knowledge graph, thereby outputting an industrial equipment gold standard dataset.

[0145] Specifically, for the whole life cycle management scenario of industrial equipment, the human-in-the-loop annotation platform constructed by this invention deeply integrates with the intelligent auxiliary system through a three-level architecture, transforming equipment technical manuals, sensor logs and flaw detection images into a high-precision fault knowledge base, and realizing hierarchical collaborative annotation from junior technicians to senior equipment experts.

[0146] Crowdsourced annotation is performed by frontline maintenance personnel trained by equipment manufacturers, primarily responsible for annotating basic equipment information. Input includes multimodal raw data such as industrial equipment technical manuals (PDF), real-time sensor logs, and flaw detection images. Task allocation utilizes a rule engine to automatically split tasks by equipment type and fault module. Technical documents for a single piece of equipment are assigned to 2-3 technicians for parallel annotation. The system provides standardized equipment annotation templates, a built-in equipment model dropdown menu (e.g., "6-Axis-Robot-3000", "CNC-Mill-5000"), a standard fault code library (e.g., "ERR-5021 bearing overheating", "ERR-3012 hydraulic pressure abnormality"), and a maintenance case example library. Intelligent auxiliary functions pre-fill high-confidence entity recognition results (e.g., BERT model prediction probability >0.9 for equipment number and parameter thresholds), and real-time logic checks ensure annotation compliance (e.g., physical rationality checks such as temperature parameter range -50℃ to 150℃, pressure value >0MPa, etc.). The output is an initial set of equipment fault annotations, containing descriptions, locations, and preliminary classifications of the same equipment faults by multiple technicians. Quality control ensures the reliability of basic annotations by calculating annotation consistency (Fleiss' Kappa > 0.6), automatically filtering out inefficient annotators with an accuracy rate below 80% and suspending their task permissions, thus ensuring the basic quality of crowdsourced data.

[0147] The professional-level annotation layer consists of equipment engineers or production line technical supervisors with over 5 years of experience, responsible for verifying and correcting medium-complexity equipment fault samples. Input includes crowdsourced annotation results and conflict reports (e.g., differences in how different technicians annotate the same bearing noise as "mechanical wear" versus "insufficient lubrication"). The difference analysis module clusters conflicting similar fault descriptions and calls upon the equipment manufacturer's rule base (e.g., ISO 81346 equipment structure classification standard, GB / T fault mode terminology specification) for standardization verification. The intelligent correction function trains a LightGBM model based on historical maintenance decision data, recommends the optimal fault root cause classification (recommendation accuracy > 85%), supports engineers to accept or reject crowdsourced results with one click, and outputs a verified equipment fault annotation set (correction rate controlled within 15%). Upgrade conditions are automatically triggered in complex dispute scenarios, such as new equipment faults (the first occurrence of "servo motor encoder drift") or rare anomalies ("abnormal plasma discharge in vacuum pumps"). Such samples are automatically pushed to expert-level final review.

[0148] The expert-level final review is conducted by senior technical experts or industry-certified senior engineers from the equipment manufacturer, responsible for the authoritative determination of highly complex equipment failure samples. Input consists of high-risk samples submitted by professionals, including undefined failure categories, major equipment accident cases, and complex cross-system related failures. The multimodal verification process involves cross-validation of technical manual descriptions and flaw detection images, comparing the consistency between the equipment's external damage descriptions and X-ray crack features; key time-series data comparison matches vibration spectrum sequences with a standard failure mode library (consistency >99%) to confirm precise diagnoses such as "bearing inner ring spalling." The authoritative decision-making process employs a double-blind review mechanism, with two senior equipment experts independently determining the root cause of the failure. If the results are consistent, the final labeling is locked; if there is disagreement, arbitration is initiated. The output is a gold-standard equipment failure dataset with an error rate of less than 0.5%, covering equipment failure mechanisms, maintenance decision-making logic, and preventative maintenance recommendations, building the equipment manufacturer's core knowledge assets.

[0149] The intelligent assisted annotation system deeply integrates big data on industrial equipment maintenance. Through a pre-annotated model trained on over 100,000 historical maintenance records, it automatically generates preliminary fault mode annotations (e.g., "vibration amplitude exceeds limits → bearing wear") for newly collected abnormal sensor data. Engineers only need to verify and correct these annotations instead of starting from scratch, improving efficiency by more than three times. An active learning mechanism intelligently selects samples that provide the greatest benefit to the equipment fault diagnosis model, prioritizing data from new equipment models, rare fault modes, and abnormal samples from high-value core equipment (such as robots at critical workstations on the production line) for manual annotation. This increases model iteration efficiency by 40% while reducing manual annotation by 30%. The real-time quality feedback system monitors the equipment fault labeling process, automatically identifies common labeling deviations (such as misdiagnosing "broken gear teeth" as "tooth surface wear"), pushes targeted improvement prompts, and displays labeling accuracy trends for different production lines and equipment types through a visual dashboard. This assists engineers in continuously improving their fault diagnosis and labeling skills, ultimately forming a high-quality industrial knowledge base that can support predictive maintenance and digital twin model training. Through cross-modal retrieval, technicians can accurately locate the cause of the fault and obtain standard maintenance guidelines within 3 minutes, reducing the mean time to repair (MTTR) by more than 50%, significantly reducing equipment maintenance costs and production line downtime losses.

[0150] S40, based on the difference analysis between the gold standard dataset of industrial equipment and the initial industrial equipment knowledge graph, high-frequency error patterns of industrial equipment faults are identified through unsupervised clustering. The amount of manual correction is used as a negative reward signal. Reinforcement learning is used to dynamically adjust the confidence threshold of the agent's processing pipeline and the equipment fault extraction strategy to form a closed-loop feedback, continuously optimize the equipment fault diagnosis model, and support cross-modal retrieval to quickly locate the cause of the fault.

[0151] Specifically, in high-end manufacturing and the Industrial Internet, this is applied to the full lifecycle management of industrial equipment. The system can automatically aggregate equipment technical manuals (PDF documents), real-time sensor logs (sequence data), and flaw detection images (image data) to establish an equipment fault knowledge base. Through cross-modal retrieval, technicians can quickly locate the cause of the fault and obtain maintenance guidelines, significantly reducing equipment operation and maintenance costs. This includes: S401: Feedback Data Integration: The operation steps include two core modules: a difference analysis engine and reinforcement learning optimization. The difference analysis engine inputs the preliminary equipment fault labeling results generated by AI preprocessing and the gold standard data confirmed by the expert-level labeling layer. Industrial fault characteristics include equipment type, fault code, abnormal sensor parameters, fault phenomenon description, and maintenance measures. The structured difference extraction stage identifies newly added, deleted, or modified fault entities in the two types of labeling results by comparison, focusing on extracting differences in equipment number, key fault fields (such as vibration amplitude, temperature deviation, and pressure anomaly), and their corresponding attribute values. Error pattern clustering uses unsupervised learning (such as K-means) to intelligently classify high-frequency labeling errors, forming typical error patterns in industrial equipment data, including but not limited to omissions or inconsistencies in units of measurement (e.g., mixing "rpm" and "revolutions per minute"), inconsistent fault code abbreviations (e.g., "ERR-5021" and "E5021" coexisting), and confusion between upper and lower limits of parameter ranges (e.g., mislabeling "temperature upper limit 85℃" as "85℃±5"). The output is an error pattern analysis report for equipment fault labeling, which statistically analyzes the top 10 most frequent problem types and their distribution frequency in terms of production line and equipment model.

[0152] The reinforcement learning optimization design uses the number of manual corrections as a negative reward signal; the fewer the corrections, the higher the reward the model receives. The reward function is expressed as R = α Ncorrections, where α is the adjustment coefficient for the equipment fault type (α=1.5 for critical equipment, α=1.0 for auxiliary equipment). Strategy optimization employs a deep Q-network (DQN) to dynamically adjust key parameters of the AI ​​Agent, including but not limited to the confidence threshold for equipment fault entity recognition (increased from 0.7 to 0.85), vibration spectrum anomaly detection sensitivity, and strictness of time-series data pattern matching. Performance validation involves A / B testing to compare the efficiency, manual correction rate, and overall diagnostic accuracy of the model before and after optimization in equipment fault labeling tasks. The test dataset covers typical equipment fault cases from four major processes: stamping, welding, painting, and assembly. The output is an optimized Agent parameter configuration file for industrial equipment fault diagnosis tasks.

[0153] S402: Model Iteration Closed Loop: The incremental training pipeline input consists of newly labeled and quality-verified equipment fault data. When the cumulative number of newly added fault cases reaches 500, the model update process is triggered (the trigger threshold is lowered to account for the scarcity of industrial data). Model micro-calls, based on the pre-trained language model (BERT), combine industrial domain-annotated data such as equipment technical manuals, maintenance records, and fault logs for single-round incremental training. A strategy of freezing top-level parameters and updating only the underlying domain adaptation layer is adopted, enhancing equipment fault pattern recognition capabilities while retaining general language understanding capabilities. The training configuration sets the batch size to 16 (adapting to the small sample size characteristics of industrial applications) and employs a cosine annealing learning rate scheduling strategy, with an initial learning rate set to 2e-5 to improve the model's stability and convergence performance in equipment fault extraction tasks. The output is the upgraded equipment fault diagnosis model version, with a target F1-score improvement of ≥3% on the validation set compared to the previous version.

[0154] The dynamic deployment system employs a canary release strategy, prioritizing the testing of new models on approximately 5% of non-critical auxiliary equipment labeling tasks to avoid risks to the diagnosis of core production line equipment. Real-time monitoring tracks key performance indicators (KPIs) such as manual correction rate, accuracy of equipment fault entity identification, and recall rate of critical fault modes. A decision-making mechanism is in place: if the new model causes a decrease in the manual correction rate to reach or exceed a preset threshold (3%), a full deployment to all equipment types is executed; otherwise, it automatically rolls back to the previous stable version and triggers in-depth error mode analysis. This mechanism ensures the continuous and robust optimization of the equipment fault diagnosis model.

[0155] S403: Performance Evaluation Dashboard: Visualized metrics include a core metrics dashboard and drill-down analysis functionality. The core metrics dashboard monitors real-time key performance indicators such as equipment fault labeling efficiency (numbers / hour), senior engineer correction rate, and cross-modal alignment accuracy (e.g., consistency between text fault descriptions and flaw detection images). Drill-down analysis supports analysis by equipment type (e.g., industrial robots, CNC machine tools, conveyor systems), production line category (e.g., welding lines, assembly lines), and fault mode (e.g., mechanical wear, electrical faults, sensor malfunctions). For example, it allows viewing the fault labeling accuracy of a specific brand of industrial robot at a specific workstation. Trend comparison supports evolutionary analysis of diagnostic performance across different model versions along a timeline, identifying model drift phenomena.

[0156] The data is presented using interactive visualization charts, including: line charts showing the improvement trends of equipment fault labeling accuracy and recall as the model version iterates; heatmaps showing the error distribution density of different equipment types and fault categories, highlighting areas with high false alarms; and scatter plots showing the individual performance differences of labelers on different production lines, identifying personnel requiring targeted training. This dashboard provides equipment operation and maintenance managers with a panoramic view of model performance, assisting in optimizing labeling resource allocation and fault diagnosis strategies.

[0157] The multimodal information extraction and annotation method according to embodiments of the present invention is applied to the full lifecycle management of industrial equipment. It can automatically aggregate equipment technical manuals (PDF documents), real-time sensor logs (sequence data), and flaw detection images (image data) to establish an equipment fault knowledge base. Through cross-modal retrieval, technicians can quickly locate the cause of the fault and obtain maintenance guidelines, significantly reducing equipment operation and maintenance costs.

[0158] It should be understood that the above embodiments are only used to explain the technical solutions of the present invention, and are not intended to limit the present invention. Those skilled in the art can easily apply the above methods to other vertical fields that require processing multimodal data, such as, but not limited to, new material research and development, industrial equipment fault diagnosis, and financial compliance review.

[0159] In summary, the intelligent multimodal information extraction and annotation method constructed in this invention, through its modular, configurable, and self-optimizing architecture, not only solves the data fragmentation problem within a single domain but also demonstrates strong adaptability and value potential in cross-industry applications. Whether in life sciences, advanced manufacturing, materials research and development, or fintech, this system can serve as a core infrastructure, driving the efficient transformation of data into knowledge and knowledge into decision-making, thus assisting various vertical fields in achieving intelligent upgrades.

[0160] Furthermore, as the trend of artificial intelligence shifting from general-purpose large models to vertical industry large models accelerates, this invention has broad market prospects and significant strategic value: Addressing the data challenges in vertical industries: Current general-purpose models lack in-depth knowledge specific to particular industries. This invention provides a standardized, industrial-grade data production pipeline that can transform the vast amounts of unstructured "dark data" accumulated within industries into high-quality, labeled "golden data," serving as the core fuel for training industry-specific AI models.

[0161] Redefining the "human-machine collaboration" annotation paradigm: Through a three-tiered architecture of "crowdsourcing-professional-expert" and an online reinforcement learning feedback mechanism, this invention effectively solves the pain points of high cost, poor quality, and wasted expert resources in traditional data annotation. The described human-in-the-loop model will be the mainstream direction for future high-difficulty data processing.

[0162] Driving the automation upgrade of knowledge engineering: This system realizes a closed-loop process from raw data collection to knowledge graph construction, greatly reducing the threshold for enterprises to build industry knowledge bases. It not only improves the efficiency of information retrieval, but also provides underlying technical support for enterprises to achieve digital transformation and accumulate core data assets.

[0163] Example 4 To implement the methods of the above embodiments, the present invention also provides a multimodal information extraction and annotation system 10, such as... Figure 2 As shown, it includes: The multi-source data acquisition module 100 is used to acquire target domain data from multiple heterogeneous data sources, including database interfaces, unstructured documents, and business monitoring data, and output standardized multimodal data after preprocessing. The multi-agent construction module 200 is used to build an agent processing pipeline consisting of a data cleaning agent, an information extraction agent, and a multimodal alignment agent, so as to automatically clean, extract information, and perform cross-modal semantic alignment on standardized multimodal data and output an initial knowledge graph. The collaborative annotation module 300 is used to verify, correct, and finalize the initial knowledge graph through a hierarchical human-machine collaborative annotation platform with crowdsourcing, professional, and expert levels, so as to output the gold standard dataset. The dynamic consistency verification module 400 is used to analyze the difference between the gold standard dataset and the initial knowledge graph, identify high-frequency error patterns through unsupervised clustering, use the manual correction amount as a negative reward signal, and dynamically adjust the confidence threshold and extraction strategy of the agent's processing pipeline using reinforcement learning to form a closed-loop feedback.

[0164] This invention's multimodal information extraction and annotation system automatically collects multi-source domain data; constructs a data processing pipeline based on an intelligent agent architecture; uses a data cleaning agent to perform quality screening on the raw data; uses an information extraction agent to identify domain entities and their conditional dependencies; and uses a multimodal alignment agent to achieve semantic matching and consistency verification of cross-modal feature information. Furthermore, a hierarchical human-in-the-loop annotation platform is built to manually verify and correct the automatic processing results, and the annotation feedback is used for dynamic updates of model parameters, forming a closed-loop optimization mechanism. Through the above technical solutions, this invention can achieve full-process collaborative processing of domain data with complex feature structures, from multi-source collection, intelligent parsing, manual verification to continuous optimization, significantly improving the structured quality and annotation efficiency of multimodal feature data, and providing reliable support for data construction and application in related fields.

[0165] The multimodal information extraction and annotation system of this invention applies the steps of the aforementioned multimodal information extraction and annotation method, so there will be no further elaboration.

[0166] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0167] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0168] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for multimodal information extraction and annotation, characterized in that, include: Acquire target domain data from multiple heterogeneous data sources, including database interfaces, unstructured documents, and business monitoring data, and output standardized multimodal data after preprocessing. Construct an intelligent agent processing pipeline consisting of a data cleaning agent, an information extraction agent, and a multimodal alignment agent to automatically clean, extract, and align standardized multimodal data across modalities, and output an initial knowledge graph. The initial knowledge graph is verified, corrected, and finalized using a hierarchical human-machine collaborative annotation platform with crowdsourcing, professional, and expert levels to output a gold standard dataset. Based on the difference analysis between the gold standard dataset and the initial knowledge graph, high-frequency error patterns are identified through unsupervised clustering. The amount of manual correction is used as a negative reward signal. Reinforcement learning is used to dynamically adjust the confidence threshold and extraction strategy of the agent's processing pipeline to form a closed-loop feedback.

2. The method as described in claim 1, characterized in that, Acquire target domain data from multiple heterogeneous data sources, including database interfaces, unstructured documents, and business monitoring data, and output standardized multimodal data after preprocessing, including: Structured data from external databases is collected in batches through standardized API interfaces, and the data is converted into a predefined unified format containing entity identifiers, classification information, and basic feature parameters and stored in a relational database. An OCR engine is used to extract text and parse layout of unstructured documents in PDF and image formats, separating text paragraphs, table areas and image blocks. Pre-recognition is performed based on regular expression matching rules, and image feature vectors are extracted using deep neural networks. The structured data after format conversion, the encoded standardized text corpus, and the image feature vectors are stored in a unified encoding format, and the output is standardized multimodal data for intelligent agents to process.

3. The method as described in claim 1, characterized in that, Automated cleaning, information extraction, and cross-modal semantic alignment are performed on standardized multimodal data to output an initial knowledge graph, including: The data cleaning agent performs deduplication, quality filtering and standardization on the standardized multimodal data. The SimHash algorithm and perceptual hash comparison are used to identify and remove duplicate text and images respectively. Low-quality items are removed based on field integrity, logical rules and preset quality standards. The units, names and formats are unified to predefined standards and the cleaned standardized data is output. An information extraction agent is constructed using a pre-trained model fine-tuned with vertical domain corpus. Named entity recognition is performed on the cleaned and standardized data and linked to the industry standard database ID. Relationships between entities are extracted through dependency parsing and business rule base. Numerical attributes are standardized and modeled to generate a structured knowledge graph. A multimodal alignment agent is used to establish semantic alignment relationships between text, images, and sequence data. The CLIP model is used to calculate the cross-modal similarity between text and images to achieve matching retrieval. Faster R-CNN is used for visual localization and annotation of key regions. Finally, a graph neural network is used to jointly model multimodal features to generate unified entity identifiers and output the aligned multimodal knowledge graph.

4. The method as described in claim 1, characterized in that, The initial knowledge graph is verified, corrected, and finalized using a hierarchical human-machine collaborative annotation platform with crowdsourcing, professional, and expert levels to output a gold-standard dataset, including: The multimodal knowledge graph is subjected to basic annotation through a crowdsourced annotation layer. Annotation tasks are split based on a rule engine and standardized templates are provided. High-confidence AI recognition results are pre-filled and real-time logical verification is performed. The consistency index of multiple annotators is calculated, low-quality annotators are automatically filtered out, and an initial annotation set is output. The professional-grade annotation layer receives the initial annotation set and conflict report, performs cluster analysis on similar conflicts and calls the domain rule base, trains the LightGBM model based on historical decisions to recommend the optimal annotation to intelligently correct the crowdsourcing results, outputs the verified annotation set, and upgrades complex disputed samples to the expert level. An expert-level final review layer is used to perform text-image cross-validation and key feature comparison on high-risk samples. A double-blind review mechanism is used to make a final decision and lock the annotation results, and output the gold standard dataset.

5. The method as described in claim 1, characterized in that, The differential analysis based on the gold standard dataset and the initial knowledge graph, through unsupervised clustering to identify high-frequency error patterns, using manual correction as a negative reward signal, and employing reinforcement learning to dynamically adjust the confidence threshold and extraction strategy of the agent's processing pipeline to form a closed-loop feedback, includes: The system integrates feedback data, compares the gold standard dataset with the initial knowledge graph using a difference analysis engine, extracts the structured differences of entities and attribute values, identifies high-frequency error patterns using an unsupervised clustering algorithm, generates an error pattern analysis report, designs a negative reward function based on manual corrections, dynamically adjusts the confidence threshold and extraction strategy of the agent's processing pipeline using deep reinforcement learning, verifies the optimization effect through A / B testing, and outputs an updated parameter configuration file. Establish a closed loop for model iteration. When the cumulative amount of newly added labeled data reaches a preset threshold, incremental training is triggered. On the basis of the pre-trained language model, a single round of fine-tuning is performed and a cosine annealing learning rate scheduling strategy is adopted. The new model is applied to some labeling tasks through canary release. Core indicators are monitored in real time. If the correction rate drops to a preset threshold, full deployment is performed. Otherwise, it is automatically rolled back to the previous stable version. The system constructs an evaluation dashboard, using interactive and visual charts to display key indicators such as annotation efficiency, expert correction rate, and cross-modal alignment accuracy. It supports drill-down analysis by target domain category, data modality, and time dimension to compare the performance differences of different model versions, and presents a heatmap of error type distribution and a scatter plot of individual annotator performance differences.

6. The method as described in claim 3, characterized in that, Constructing the data cleaning agent includes: A deduplication module is constructed. The SimHash algorithm is used to calculate the similarity of text content and determine that text with a similarity greater than a preset threshold is duplicate data. The perceptual hash algorithm is used to compare image content. Combined with the text similarity algorithm based on cosine similarity and edit distance, as well as the document encoding and UUID unique identifier precise matching mechanism, duplicate or nearly duplicate data records are identified and removed. A quality filtering module is built to remove documents with missing key information or incomplete metadata from text data, remove low-quality images from image data, and establish multi-level quality assessment standards for structured business data through field integrity checks, data consistency verification, and outlier detection techniques to automatically identify and remove data items that do not meet quality requirements. Standardized functional modules are constructed, and a unified data storage format and encoding standard are established to convert data from different sources into a predefined standard format, including UTF-8 unification of text encoding, ISO 8601 standardization of date format, conversion of numerical units to the International System of Units, and unified mapping of unit symbols and names. The cleaned and standardized data is output to ensure the efficiency and accuracy of subsequent information extraction.

7. The method as described in claim 3, characterized in that, Constructing the information extraction Agent includes: A named entity recognition module is built by fine-tuning a pre-trained model with vertical domain corpus. The module performs entity recognition on the cleaned standardized data and links it to the industry standard database ID. It establishes a hierarchical entity classification system that includes core object names and attribute names, processes entity abbreviations, aliases and synonym variants, and outputs complete and accurate entity recognition results. A relation extraction module is constructed, which uses vertical domain business rules and dependency parsing techniques to extract subject-verb-object triple relationships between entities, identify complex logical associations, and generate structured relation data. An attribute extraction module is constructed to identify and analyze quantitative and semi-quantitative features in text. Range values, approximate values, and conditional limits are uniformly modeled into structured numerical representations containing minimum values, maximum values, and standardized units. The output is a structured knowledge graph for multiple domains to support multimodal alignment processing.

8. The method as described in claim 3, characterized in that, Constructing the multimodal alignment agent includes: A text-image alignment module is constructed using a vision-language pre-trained model. The semantic similarity between the text description and the image is calculated using the CLIP model or other cross-modal representation models to achieve matching retrieval. The key regions of the image are visually located using Faster R-CNN or YOLO series object detection models, and the text semantic features and image visual features are fused and aligned. A sequence-function alignment module is constructed using sequence alignment algorithms or temporal neural networks. Sequence data is matched with predefined business rules using dynamic time warping algorithms or Transformer-based temporal models to generate corresponding functional or state annotations and establish a mapping relationship between measurement sequences and biological significance. An entity unification module is constructed using graph neural network variants, including GraphSAGE, GraphConvolutional Network, Graph Isomorphism Network, or hyperbolic graph neural network, to jointly model and embed multimodal features of text entities, image features, and sequence features, thereby achieving cross-modal entity disambiguation and unified identifier generation.

9. A multimodal information extraction and annotation system, characterized in that, include: The multi-source data acquisition module is used to acquire target domain data from multiple heterogeneous data sources, including database interfaces, unstructured documents, and business monitoring data, and output standardized multimodal data after preprocessing. The multi-agent construction module is used to build an agent processing pipeline consisting of a data cleaning agent, an information extraction agent, and a multimodal alignment agent to automatically clean, extract information, and perform cross-modal semantic alignment on standardized multimodal data, and output an initial knowledge graph. The collaborative annotation module is used to verify, correct, and finalize the initial knowledge graph through a hierarchical human-machine collaborative annotation platform with crowdsourcing, professional, and expert levels, so as to output the gold standard dataset. The dynamic consistency verification module is used to analyze the difference between the gold standard dataset and the initial knowledge graph, identify high-frequency error patterns through unsupervised clustering, use the manual correction amount as a negative reward signal, and dynamically adjust the confidence threshold and extraction strategy of the agent's processing pipeline using reinforcement learning to form a closed-loop feedback.

Images