Multi-modal experimental data extraction method and system embedded with intelligent agent search mechanism

Abstract

This invention discloses a multimodal experimental data extraction method and system with an embedded intelligent agent search mechanism, relating to the field of data extraction technology. The method includes: performing multimodal parsing on input target documents, identifying multimodal regions and constructing a source data index system; identifying research objects in the target documents and mapping synonymous expressions to entity identifiers; invoking multiple data extraction agents of different modalities to perform parallel data extraction on the target documents; performing cross-modal alignment on triplet data, detecting and arbitrating data conflicts, determining data truth values ​​and performing reverse source tracing verification, and generating structured knowledge base entries. This invention solves the technical problems of low experimental data extraction efficiency and difficulty in collaborative analysis and conflict detection of multi-source heterogeneous data in existing technologies, leading to low accuracy of the constructed knowledge base. It achieves the technical effects of improving the accuracy and reliability of data extraction, realizing intelligent fusion verification of cross-modal information, and ensuring data traceability.

Description

Technical Field

[0001] This invention relates to the field of data extraction technology, specifically to a method and system for extracting multimodal experimental data with an embedded intelligent agent search mechanism. Background Technology

[0002] Efficiently and accurately extracting experimental data from massive amounts of scientific literature is crucial for building domain knowledge bases, as the accumulation and reuse of experimental data directly impacts research efficiency. Currently, data extraction from scientific literature primarily relies on traditional manual methods. Researchers typically need to go through a series of tedious steps, including literature retrieval and screening, full-text reading, manual data entry, data standardization, and cross-validation, to transform information scattered across text, tables, and charts into structured data. However, this data extraction model, heavily reliant on manual operations, has significant drawbacks. On the one hand, it is extremely inefficient, with excessively long processing cycles in the face of rapidly growing volumes of literature, severely restricting the speed of knowledge base construction and data update frequency. On the other hand, manual operations inevitably introduce errors, and although cross-validation can partially correct these errors, the overall cost is high.

[0003] With the development of natural language processing and image recognition technologies, some automated or semi-automated data extraction tools have emerged. However, these tools are typically limited to the isolated extraction of information from a single modality, such as focusing only on plain text content in literature or parsing only structured tabular data. They generally lack the ability to collaboratively analyze and fuse multi-source heterogeneous data. In actual scientific literature, information about the same experimental object is often scattered across multiple modalities, such as text descriptions, tabular data, and figure / table titles. Existing methods struggle to effectively identify and associate cross-modal information, leading to data fragmentation and an inability to form a complete knowledge view of the research object. Furthermore, when data from different modalities are inconsistent or even conflicting—for example, when descriptive values ​​in the text differ from specific test values ​​in a table—current methods lack intelligent detection and arbitration mechanisms to autonomously determine the credibility of the data and make appropriate choices, thus affecting the accuracy and reliability of the final knowledge base.

[0004] Therefore, current technologies suffer from low efficiency in extracting experimental data and difficulty in conducting collaborative analysis and conflict detection of multi-source heterogeneous data, resulting in low accuracy of the constructed knowledge base. Summary of the Invention

[0005] This application provides a multimodal experimental data extraction method and system with an embedded intelligent agent search mechanism. It solves the technical problems of low experimental data extraction efficiency, difficulty in collaborative analysis and conflict detection of multi-source heterogeneous data, and low accuracy of the constructed knowledge base in the prior art. It achieves the technical effects of improving the accuracy and reliability of data extraction, realizing intelligent fusion and verification of cross-modal information, and ensuring data traceability.

[0006] This application provides a multimodal experimental data extraction method with an embedded intelligent agent search mechanism. The method includes: performing multimodal parsing on the input target document, identifying multimodal regions and assigning a globally unique source identifier to each independent semantic unit, and constructing a traceable source data index system; identifying research objects in the target document based on a domain ontology knowledge base, and mapping synonymous expressions to standardized entity identifiers through ontology anchoring; performing parallel data extraction of each modality on the target document by calling multiple data extraction agents of different modalities, combined with the source data index system, and outputting structured triple data; performing cross-modal alignment on the triple data based on entity identifiers, detecting and arbitrating data conflicts, determining the truth value of the fused data and performing reverse source tracing verification, and generating structured knowledge base entries.

[0007] In a possible implementation, before performing multimodal parsing on the input target documents, the following steps are taken: constructing a dual-database knowledge system for multimodal experimental data, wherein the dual-database knowledge system includes a domain ontology knowledge base and a data extraction rule base; and constructing an extraction tool based on task definition according to the dual-database knowledge system, wherein the extraction tool includes an ontology recognition agent and a data extraction agent.

[0008] In possible implementations, a traceable source data indexing system is constructed, including: performing layout analysis on the target documents to divide them into multimodal regions; for each multimodal region, performing differentiated segmentation on each region, breaking it down into the smallest semantic units, wherein the text modality is segmented based on independent sentences, the table modality is segmented based on data behaviors, and the chart modality is extracted based on core annotation elements; for each smallest semantic unit, a globally unique identifier containing the region type and hierarchical relationship is assigned to construct a fine-grained source data indexing system.

[0009] In possible implementations, ontology anchoring maps synonymous expressions to standardized entity identifiers. This includes: activating an ontology recognition agent based on predefined entity naming rules in a domain ontology knowledge base to perform a full-text scan of the target document and identify multiple expressions of the same research object; and for the multiple expressions, semantic mapping uniformly maps synonymous expressions to standardized entity identifiers, where the entity identifier is a unique primary key for document data association, satisfying cross-modal data anchoring.

[0010] In a possible implementation, the data extraction agent includes a text expert agent, a table expert agent, and a visual relationship expert agent, and the data extraction agent is scalable; the text expert agent uses material composition, performance parameters, values, units, and test conditions as identification targets; the table expert agent is used to parse table structures and extract structured data; and the visual relationship expert agent is used to extract key parameters from chart titles and annotation information.

[0011] In a possible implementation, data extraction of various modalities is performed in parallel on the target document to output structured triple data, including: the text expert agent, table expert agent and visual relationship expert agent in the data extraction agent perform data extraction in parallel to obtain extraction results; the extraction results are marked based on a unified entity identifier, and the data is structurally transformed according to the standardized output format in the data extraction rule base to determine the triple data.

[0012] In a possible implementation, a text expert agent extracts the target and converts it into text triples representing material-performance-value; a table expert agent parses the table structure and converts it into table triples representing sample-test value-condition; a visual relationship expert agent identifies key parameters in the chart title and annotation through image analysis and converts them into chart triples representing identifier-performance-value; and the text triples, table triples, and chart triples are added to the triplet data.

[0013] In possible implementations, after reverse tracing verification, the following steps are taken: if the verification fails, a loop error correction is automatically triggered, wherein at least one of the following is used as the triggering condition: missing data, inconsistent data, or broken tracing chain; as the loop error correction step is triggered, iterative arbitration processing is performed until the verification passes, wherein the iterative arbitration processing method is either the corresponding data extraction proxy call or cross-modal fusion arbitration.

[0014] In a possible implementation, generating structured knowledge base entries includes: attaching quality control information to each verified data truth value, wherein the quality control information includes a data consistency score, a traceability integrity marker based on the source identifier, and a verification status identifier; generating the structured knowledge base entries by jointly encapsulating the quality control information, standardized entity identifiers, triplet data, source identifier traceability chains, and conflict arbitration records.

[0015] This application also provides a multimodal experimental data extraction system with an embedded intelligent agent search mechanism. The system includes: a source data index system construction module, used to perform multimodal parsing on the input target documents, identify multimodal regions, assign a globally unique source identifier to each independent semantic unit, and construct a traceable source data index system; a synonym mapping module, used to identify research objects in the target documents based on a domain ontology knowledge base, and map synonyms to standardized entity identifiers through ontology anchoring; a data extraction execution module, used to execute data extraction of each modality in parallel on the target documents by calling multiple data extraction agents of different modalities, combined with the source data index system, and output structured triple data; and a knowledge base entry generation module, used to perform cross-modal alignment of the triple data based on entity identifiers, detect and arbitrate data conflicts, determine the truth value of the fused data and perform reverse source verification, and generate structured knowledge base entries.

[0016] This application proposes a multimodal experimental data extraction method and system with an embedded intelligent agent search mechanism. The method performs multimodal parsing on input target documents, identifies multimodal regions, and constructs a source data index system. It identifies research objects within the target documents, mapping synonyms and variants to entity identifiers. Multiple data extraction agents of different modalities are invoked to perform parallel data extraction on the target documents. Cross-modal alignment of triplet data is performed, data conflicts are detected and arbitrated, data truth values ​​are determined, and reverse tracing verification is conducted to generate structured knowledge base entries. This addresses the technical problems of low experimental data extraction efficiency, difficulty in collaborative analysis and conflict detection of multi-source heterogeneous data, and low accuracy of the constructed knowledge base in existing technologies. It achieves improved accuracy and reliability of data extraction, intelligent fusion and verification of cross-modal information, and ensures data traceability. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments of this disclosure will be briefly described below. Flowcharts are used in this application to illustrate the operations performed by the system according to the embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from these processes.

[0018] Figure 1 This is a schematic diagram of the multimodal experimental data extraction method with an embedded intelligent agent search mechanism provided in an embodiment of this application.

[0019] Figure 2 This is a schematic diagram of the structure of a multimodal experimental data extraction system with an embedded intelligent agent search mechanism provided in an embodiment of this application.

[0020] Figure labeling: Source data index system construction module 10, synonym and synonym representation mapping module 20, data extraction execution module 30, knowledge base entry generation module 40. Detailed Implementation

[0021] To further illustrate the technical means and effects adopted by the present invention in order to achieve the intended purpose, the following detailed description is provided in conjunction with the accompanying drawings and preferred embodiments, based on the specific implementation methods, structures, features and effects of the present invention.

[0022] This application provides a method for extracting multimodal experimental data by embedding an intelligent agent search mechanism, such as... Figure 1 As shown, the method includes: Step S100: Perform multimodal parsing on the input target documents, identify multimodal regions, assign a globally unique source identifier to each independent semantic unit, and construct a traceable source data index system.

[0023] Preferably, the input PDF or electronic scientific literature file is automatically analyzed to identify different types of components in the literature, including text, tables, and images (such as charts and graphs), to distinguish regions belonging to text paragraphs, tables, and charts / images. Then, the boundaries of each modality are defined to determine multimodal regions. For example, the lower half of page 2 in the target document is defined as a table region, the upper left corner of page 3 as an image region, and the remaining continuous text blocks as text paragraph regions. Then, within each identified modal region, the smallest semantic units with independent meaning are further decomposed, and each independent semantic unit is assigned a globally unique source identifier. Specifically, for text regions, paragraphs are decomposed into independent sentences. For example, a paragraph is divided into sentence 1 and sentence 2, and sentence 1 is further divided into... Assign ID "#SID-101" to sentence 2 and ID "#SID-102" to sentence 2. For table areas, break the table down into independent rows. For example, assign ID "#SID-T1" to a table and sub-ID "#SID-T1-R1" to the first row of data, "#SID-T1-R2" to the second row, and so on. For chart areas, extract core annotation elements such as titles or key labels. For example, assign ID "#SID-F1" to a chart and assign a related ID to its title. Finally, integrate all assigned identifiers and their corresponding original content fragments and their location information in the document to form a structured list or mapping table. This allows for precise location of the source data in the original document and its original content through any source identifier.

[0024] Furthermore, step S100 also includes constructing a dual-database knowledge system for multimodal experimental data, wherein the dual-database knowledge system includes a domain ontology knowledge base and a data extraction rule base; and constructing an extraction tool based on task definition according to the dual-database knowledge system, wherein the extraction tool includes an ontology recognition agent and a data extraction intelligent agent.

[0025] Preferably, before performing multimodal analysis on the input target literature, a dual-base knowledge system for multimodal experimental data is pre-established, including a domain ontology knowledge base and a data extraction rule base. This serves as the knowledge basis and operational standard for all automated processing. Specifically, a standardized concept dictionary is constructed based on the standard definitions, naming conventions, and relationships of all core concepts within a specific discipline, thus defining the domain ontology knowledge base. In particular, material naming is standardized, specifying a unified representation format for materials. For example, high-entropy alloys must use the format "element symbol + atomic percentage," and the use of terms like "Sample-A," "Alloy 1," and "FeCoNiC" in the literature is standardized. Different names such as "rMn-HEA" are all mapped to this standard format; performance parameters are standardized, and the standard definitions of performance indicators are clearly defined. For example, the symbolic representation of "overpotential," standard measurement conditions, and unit requirements are specified to ensure the comparability of the same parameter in different documents; test conditions are standardized, and the standardized description of experimental environment parameters is recorded, such as electrolyte type, pH value, and scan rate; unit conversion is regularized, and the conversion relationship between different units is predefined to ensure that all data is ultimately stored in a unified unit; data traceability and identification are specified, and the metadata types that each piece of data must be associated with are specified, such as literature DOI, paragraph ID, table number, and other metadata information.

[0026] Preferably, an operational instruction set is constructed based on the conversion rules and format requirements from original documents to structured output, and a data extraction rule base is determined. Specifically, this includes: extraction target definition, which clarifies what information needs to be extracted from the documents, for example, specifying that text extraction must identify five items: material composition, performance parameters, numerical values, units, and test conditions; output format specifications, which specify the standard data structure of the extraction results, for example, specifying that all extraction results must be output in JSON format; conflict handling rules, which predefine the arbitration priority when the same data exists from multiple sources, for example, specifying that the credibility priority of tabular data is higher than that of the main text description, and the priority of original test data is higher than that of summary statements; and verification standards, which specify the conditions for data to pass verification, such as the traceability chain must be complete and the numerical values ​​must be within a reasonable range.

[0027] Preferably, based on the dual-database knowledge system, a task-defined extraction tool is constructed to perform specific data extraction tasks. That is, the function and output format of each extraction tool are predefined by the specifications in the knowledge base. The extraction tool includes an ontology recognition agent and a data extraction agent. The ontology recognition agent is responsible for entity recognition and standardized mapping. Its workflow includes: acquiring the parsed and segmented document content; traversing all text content; identifying possible research object names; matching the identified various expressions with the standard names in the domain ontology knowledge base; assigning a standardized, globally unique entity identifier to each identified research object as the primary key for all data related to that research object; generating an entity-identifier mapping table; and recording each research object appearing in the document and its corresponding standardized identifier. That is, receiving input → full-text scanning → semantic mapping → entity identifier allocation → output.

[0028] Preferably, optimized tools are configured for four core tasks: PDF parsing, text extraction, table recognition, and image analysis. Dedicated training datasets are constructed based on experimental data in the HEA-OER field. The PDF parsing tool dataset contains structural annotation data from 5,000 materials science literature articles, the text extraction tool dataset contains 15,000 experimental parameter extraction samples, the table recognition tool dataset contains parsing annotations of 8,000 complex tables, and the image analysis tool dataset contains feature annotations of 6,000 electrochemical test graphs. All training datasets adopt the form of entity-attribute-value triples. The entity part contains raw data fragments, such as text paragraphs, table images, and graph images. The attribute part clearly defines the specific extraction task, and the value part provides a standardized structured output format.

[0029] Preferably, the PDF parsing tool deploys a deep learning-based document layout analysis model capable of identifying different document elements such as body paragraphs, table areas, chart areas, and heading levels. This tool segments each element into sentences; for example, it divides body paragraphs into independent sentences based on punctuation marks such as periods and semicolons, assigning each sentence a unique SID number, such as #SID-101. For table areas, the tool extracts the entire table and assigns a table ID (e.g., #SID-T1), while also assigning a row-level ID (e.g., #SID-T1-R2) to each row of data within the table, ensuring the accuracy of data traceability. For chart areas, the tool extracts chart titles, legends, and annotation text, assigning a chart ID (e.g., #SID-F2), providing structured input for visual analysis. In the text extraction tool, a natural language processing model based on named entity recognition (NER) and relation extraction is deployed. This tool, fine-tuned with domain-specific data, can accurately identify material composition, performance parameters, numerical information, and test conditions.

[0030] Furthermore, step S100 also includes the following: the data extraction agent comprises a text expert agent, a table expert agent, and a visual relationship expert agent; the data extraction agent is scalable; the text expert agent uses material composition, performance parameters, values, units, and test conditions as identification targets; the table expert agent is used to parse the table structure and extract structured data; and the visual relationship expert agent is used to extract key parameters from the chart title and annotation information.

[0031] Preferably, the data extraction agent is a collection of multiple types of program units, each used to extract specific modal data. These units include text expert agents, table expert agents, and visual relationship expert agents. The text expert agent is responsible for extracting information from text paragraphs, operating according to the text extraction targets defined in the data extraction rule base, namely material composition, performance parameters, values, units, and test conditions. The output format also follows the rules base specifications. The table expert agent is responsible for parsing table structures, identifying the correspondence between table headers and data rows, and extracting structured data. The visual relationship expert agent is responsible for analyzing charts and images, identifying chart titles, legends, and annotations, and extracting key parameters. Furthermore, the data extraction agent is scalable, allowing for the addition of extraction agents for other modal data such as spectral data and 3D models as needed, thereby ensuring the accuracy and reliability of data extraction from different modalities.

[0032] Furthermore, step S100 also includes performing layout analysis on the target document to divide it into multimodal regions; for each multimodal region, performing differentiated segmentation on each modal region to break it down into the smallest semantic units, wherein the text modality is segmented based on independent sentences, the table modality is segmented based on data behavior, and the chart modality is extracted based on core annotation elements; for each smallest semantic unit, assigning a globally unique identifier containing the region type and hierarchical relationship to construct a fine-grained source data indexing system.

[0033] Preferably, layout analysis tools, such as document layout analysis models based on deep learning, are used to analyze each page of the target document. Based on visual and structural features such as text block distribution, table lines, image features, font size, and page coordinates, different types of content blocks are identified, and the document page is clearly divided into multiple non-overlapping regions. Each region is labeled with a modality category; for example, continuous text blocks are labeled as text regions, blocks with a grid structure are labeled as table regions, and blocks containing graphic elements with titles or labels are labeled as chart regions. Based on the modality type of each region, different segmentation strategies are used to perform differentiated segmentation of each modality region, breaking it down into the smallest semantic units. Specifically, for regions labeled as "…",… For continuous paragraphs in the "text area," natural language processing is used to segment them into multiple independent sentences based on sentence boundary punctuation marks such as periods, semicolons, question marks, and exclamation marks. For structured data marked as "table area," the overall structure of the table is parsed, the table header, column definitions, and row definitions are identified, and the table content is then broken down into several independent data rows, each representing a complete sample record or an independent test result. For images marked as "chart area," image recognition and optical character recognition are used to extract core annotation elements that carry key numerical information, such as the chart title, each item in the legend and its corresponding curve / region, labels and tick values ​​on the coordinate axes, and numerical points directly labeled in the chart. Finally, for each smallest semantic unit, a globally unique identifier containing the region type and hierarchical relationship is assigned, and an index file or database table is created. All assigned identifiers are used as "keys", and the corresponding original content fragments, page numbers, region coordinates, etc. are used as "values" for associated storage, generating a fine-grained source data index system. This allows any identifier to be used to immediately and accurately trace its specific location in the original document and the original text / image content, achieving traceability from abstract data to the original source.

[0034] Step S200: Based on the domain ontology knowledge base, identify the research objects in the target documents, and map synonymous expressions to standardized entity identifiers through ontology anchoring.

[0035] Step S200 further includes: based on the predefined entity naming rules in the domain ontology knowledge base, activating the ontology recognition agent to perform a full-text scan of the target document and identify the multiple expressions of the same research object; for the multiple expressions, using semantic mapping, uniformly mapping the synonymous expressions to standardized entity identifiers, wherein the entity identifier is the unique primary key for document data association, satisfying cross-modal data anchoring.

[0036] Preferably, a domain ontology knowledge base is loaded, and an ontology recognition agent scans the target document sentence by sentence and word by word according to standard naming rules. The text content of the target document is matched with the standard naming rules, definitions, and expression patterns of all core concepts in the domain ontology knowledge base. The research object is identified and its corresponding position is marked, thereby determining the multiple expressions of the same research object, such as the full name when it first appears, the abbreviation in the following text, the label in the table, the annotation in the figure, and descriptive references. Then, through ontology anchoring, synonymous expressions are uniformly mapped to standardized entity identifiers. That is, based on semantic mapping matching, a standardized, globally unique entity identifier is generated for each real research object, and all multiple expressions pointing to the object are mapped to the identifier. The entity identifier is the unique primary key for document data association, which satisfies cross-modal data anchoring such as text, tables, and figures. Data aggregation and cross-modal alignment are achieved through entity identifiers to facilitate data conflict detection.

[0037] Step S300: By calling multiple data extraction agents of different modalities and combining them with the source data indexing system, data extraction of each modality is performed in parallel on the target documents, and structured triplet data is output.

[0038] Step S300 further includes the following steps: the text expert agent, table expert agent, and visual relationship expert agent in the data extraction agent perform data extraction in parallel to obtain extraction results; the extraction results are marked based on a unified entity identifier, and a structured transformation is performed according to the standardized output format in the data extraction rule base to determine the triplet data.

[0039] Preferably, data extraction is performed in parallel by calling multiple data extraction agents of different modalities. Among them, text expert agents are used to process text modal data, table expert agents are used to process table modal data, and visual relationship expert agents are used to process chart modal data. When performing data extraction tasks, all three types of agents rely on the source data index system to locate the content to be processed. The text expert agent obtains a list of all text class minimum semantic units and their corresponding original text content. The table expert agent obtains a list of all table class minimum semantic units and their corresponding table row data. The visual relationship expert agent obtains a list of all chart class minimum semantic units and their corresponding chart titles and label elements, thereby obtaining the extraction results. Then, each extraction result is marked with a unified entity identifier, and the data about the same identification object in each modality are associated. The data is then structured according to the standardized output format in the data extraction rule base to ensure that all outputs have unified field names, data types, and structural hierarchy. Finally, triple data is determined, including text triples, table triples, and chart triples. Each triple data structure contains an entity, an attribute, and a value.

[0040] Furthermore, step S300 also includes: a text expert agent extracting the target and converting it into text triples representing material-performance-value; a table expert agent parsing the table structure and converting it into table triples representing sample-test value-condition; a visual relationship expert agent identifying key parameters in the chart title and annotation through image analysis and converting them into chart triples representing identifier-performance-value; and adding the text triples, table triples, and chart triples into the triplet data.

[0041] Preferably, the text expert agent receives sentence index data output from the PDF parsing tool using an NLP model based on named entity recognition and relation extraction. It iterates through all main text sentences (#SID-101 to #SID-N). For each sentence, the agent calls a text extraction tool for entity recognition and relation extraction, focusing on identifying key information such as material composition, performance parameters, values, units, and test conditions, and converting it into text triples representing material-performance-value. The table expert agent receives all table area data (#SID-T1 to #SID-TN) identified by the PDF parsing tool. For each table, the agent calls a table recognition tool for structure parsing and data extraction. It parses the table structure using the table recognition tool, identifies the meaning of the obtained table row data according to the table header definition, and converts it into table triples representing sample-test value-condition. The visual relation expert agent receives all chart area data (#SID-F1 to #SID-FN) identified by the PDF parsing tool. For each chart, the agent calls image analysis tools to extract visual features and recognize text information, determine the key parameters in the chart title and labels, such as legend, axis labels, and values ​​directly labeled in the chart, and convert them into chart triples representing identifier-performance-value; finally, the text triples, table triples, and chart triples are added to the triplet data.

[0042] Step S400: Based on entity identifiers, perform cross-modal alignment on the triplet data, detect and arbitrate data conflicts, determine the truth value of the fused data and perform reverse tracing verification, and generate structured knowledge base entries.

[0043] Preferably, standardized entity identifiers are used to perform cross-modal alignment of triplet data. That is, all triples are grouped and aggregated according to entity identifiers, integrating all data that were originally scattered across different modalities according to the research object they all point to, forming a complete dataset about each entity. Then, all data under each entity identifier are scanned to identify whether there are multiple different values ​​corresponding to the same attribute. If so, a data conflict is constituted. The conflict is then adjudicated according to the predefined confidence priority rules in the data extraction rule base. The confidence priority rules include the priority of tabular data over text description, the priority of original test data over summary statement, the priority of high confidence sources over low confidence sources, and the majority consensus principle. According to the confidence priority rules, one of the multiple conflicting values ​​is selected as the true value of the fused data, and the conflict information and arbitration basis are recorded.

[0044] Furthermore, step S400 also includes adding quality control information to each verified data truth value, wherein the quality control information includes a data consistency score, a traceability integrity mark based on the source identifier, and a verification status identifier; by jointly encapsulating the quality control information, standardized entity identifiers, triple data, source identifier traceability chains, and conflict arbitration records, the structured knowledge base entries are generated.

[0045] Preferably, the truth value of the fused data is verified by reverse tracing. This involves searching for the original content fragment corresponding to the source identifier in the source data index system, comparing the fused truth value with the original content, and verifying for extraction or format conversion errors. Simultaneously, the integrity of the entire tracing chain from the original document to the final truth value is checked. For example, can the fused data be traced back to the used triples, from the triples to the identifier of the smallest semantic unit, and from the identifier to the original content fragment? Then, based on the verification results, a verification status identifier is added to each data point, indicating whether the verification passed or failed. Finally, a quality control information is added to each verified truth value. The quality control information includes data consistency scores, source identifier-based traceability integrity markers, and verification status identifiers. The data consistency score reflects the degree of consistency between the fused data and each source, ranging from 0 to 1, such as 0.95. The source integrity marker indicates whether the original document can be fully traced back, and the verification status identifier indicates whether the data has passed verification. Finally, the verified data, quality control information, standardized entity identifiers, triplet data, source identifier traceability chains, and conflict arbitration records are encapsulated into high-quality structured knowledge base entries. This enables intelligent fusion and verification of cross-modal information and ensures the complete traceability of the data, improving the accuracy and reliability of experimental data extraction. The performance comparison of different extraction methods on multimodal experimental data extraction tasks is shown in Table 1. Table 1 Comparison of Multimodal Data Extraction Results

[0046] Furthermore, step S400 also includes automatically triggering cyclic error correction if the verification fails, wherein at least one of the following is used as the triggering condition: missing data, inconsistent data, or broken traceability chain; as the cyclic error correction step is triggered, iterative arbitration processing is performed until the verification passes, wherein the iterative arbitration processing method is either the corresponding data extraction proxy call or cross-modal fusion arbitration.

[0047] Preferably, if at least one of the following exists—missing data, inconsistent data, or a broken traceability chain—the verification is deemed unsuccessful. Missing data refers to key fields in the fused data being null or nonexistent; inconsistent data refers to the fused values ​​not matching the original content checked during reverse verification; and a broken traceability chain refers to the inability to trace the fused data back to the specific location in the original document. A quality check gateway is set up; if any check fails, the data is marked as "FAILED," and a loop correction is automatically triggered, performing iterative arbitration processing. A maximum number of iterations (e.g., 3) or a timeout threshold is preset to prevent infinite loops. For data that still fails verification after exceeding the limit, it is marked as "FAILED," and an error log is recorded for manual review. Specifically, for data that fails verification, corresponding data extraction proxy calls or cross-platform calls are performed. Modal fusion arbitration involves several steps. For missing or inconsistent data, the corresponding data extraction agent is invoked based on the source identifier of the fused data. Data extraction is then re-executed with adjusted extraction parameters to obtain more accurate triplet data. Fusion arbitration is then performed again until the data passes verification and is marked as "VERIFIED," generating a structured knowledge base entry. For inconsistent data or broken tracing chains, cross-modal fusion arbitration is re-executed. This involves re-aggregating all triplet data related to the same entity identifier, adjusting arbitration rules (e.g., changing the credibility priority of different modalities, using different conflict resolution algorithms), and re-arbitrating conflicting data to determine a new fusion truth value. The newly determined fusion truth value is then subjected to reverse tracing verification again until the data passes verification and is marked as "VERIFIED," generating a structured knowledge base entry.

[0048] In the above text, refer to Figure 1 A method for extracting multimodal experimental data using an embedded intelligent agent search mechanism according to an embodiment of the present invention is described in detail. Next, reference will be made to... Figure 2 A multimodal experimental data extraction system based on an embedded intelligent agent search mechanism according to an embodiment of the present invention is described.

[0049] The multimodal experimental data extraction system with an embedded intelligent agent search mechanism according to embodiments of the present invention addresses the technical problems in existing technologies, such as low efficiency in experimental data extraction, difficulty in collaborative analysis and conflict detection of multi-source heterogeneous data, leading to low accuracy of the constructed knowledge base. It achieves the technical effects of improving the accuracy and reliability of data extraction, realizing intelligent fusion and verification of cross-modal information, and ensuring data traceability. Figure 2 As shown, the multimodal experimental data extraction system with embedded intelligent agent search mechanism includes: source data index system construction module 10, synonym and synonym representation mapping module 20, data extraction execution module 30, and knowledge base entry generation module 40.

[0050] The source data indexing system construction module 10 is used to perform multimodal parsing on the input target documents, identify multimodal regions, and assign globally unique source identifiers to each independent semantic unit to construct a traceable source data indexing system. The synonym mapping module 20 is used to identify research objects in the target documents based on the domain ontology knowledge base, and map synonyms to standardized entity identifiers through ontology anchoring. The data extraction execution module 30 is used to perform data extraction of each modality in parallel on the target documents by calling multiple data extraction agents of different modalities, combined with the source data indexing system, and output structured triple data. The knowledge base entry generation module 40 is used to perform cross-modal alignment of the triple data based on entity identifiers, detect and arbitrate data conflicts, determine the truth value of the fused data and perform reverse source verification to generate structured knowledge base entries.

[0051] The specific configuration of the source data indexing system construction module 10 will be described in detail below. The source data indexing system construction module 10 further includes: constructing a dual-database knowledge system for multimodal experimental data, wherein the dual-database knowledge system includes a domain ontology knowledge base and a data extraction rule base; and constructing an extraction tool based on task definition according to the dual-database knowledge system, wherein the extraction tool includes an ontology recognition agent and a data extraction intelligent agent.

[0052] The following will describe in detail the specific configuration of the source data indexing system construction module 10. The source data indexing system construction module 10 further includes: performing layout analysis on the target documents to divide them into multimodal regions; for each multimodal region, performing differentiated segmentation on each region, breaking it down into the smallest semantic units, wherein the text modality uses independent sentences as the segmentation target, the table modality uses data lines as the segmentation target, and the chart modality uses core annotation elements as the extraction target; for each smallest semantic unit, assigning a globally unique identifier containing the region type and hierarchical relationship, thus constructing a fine-grained source data indexing system.

[0053] The following will describe in detail the specific configuration of the synonym mapping module 20. The synonym mapping module 20 further includes: based on predefined entity naming rules in the domain ontology knowledge base, activating an ontology recognition agent to perform a full-text scan of the target document and identify multiple representations of the same research object; for the multiple representations, using semantic mapping, uniformly mapping synonyms to standardized entity identifiers, wherein the entity identifier is a unique primary key for document data association, satisfying cross-modal data anchoring.

[0054] The following will describe in detail the specific configuration of the source data indexing system construction module 10. The source data indexing system construction module 10 further includes: the data extraction agent comprises a text expert agent, a table expert agent, and a visual relationship expert agent; the data extraction agent is scalable; the text expert agent uses material composition, performance parameters, values, units, and test conditions as identification targets; the table expert agent is used to parse table structures and extract structured data; and the visual relationship expert agent is used to extract key parameters from chart titles and annotation information.

[0055] The specific configuration of the data extraction execution module 30 will be described in detail below. The data extraction execution module 30 further includes: the text expert agent, table expert agent and visual relationship expert agent in the data extraction agent performing data extraction in parallel to obtain extraction results; marking the extraction results based on unified entity identifiers, and performing structured transformation according to the standardized output format in the data extraction rule base to determine the triplet data.

[0056] The specific configuration of the data extraction execution module 30 will be described in detail below. The data extraction execution module 30 further includes: a text expert agent that extracts data by identifying the target and converts it into text triples representing material-performance-value; a table expert agent that parses the table structure and converts it into table triples representing sample-test value-condition; a visual relationship expert agent that identifies key parameters in the chart title and annotation through image analysis and converts them into chart triples representing identifier-performance-value; and adding the text triples, table triples, and chart triples into the triplet data.

[0057] The specific configuration of the knowledge base entry generation module 40 will be described in detail below. The knowledge base entry generation module 40 further includes: if the verification fails, automatically triggering a cyclic error correction, wherein at least one of the following is a trigger condition: missing data, inconsistent data, or broken traceability chain; as the cyclic error correction step is triggered, iterative arbitration processing is performed until the verification passes, wherein the iterative arbitration processing method is either the corresponding data extraction proxy call or cross-modal fusion arbitration.

[0058] The specific configuration of the knowledge base entry generation module 40 will be described in detail below. The knowledge base entry generation module 40 further includes: attaching quality control information to each verified data truth value, wherein the quality control information includes a data consistency score, a source identifier-based traceability integrity marker, and a verification status identifier; and generating the structured knowledge base entry by jointly encapsulating the quality control information, standardized entity identifiers, triplet data, source identifier traceability chains, and conflict arbitration records.

[0059] The multimodal experimental data extraction system with embedded intelligent agent search mechanism provided in the embodiments of the present invention can execute the multimodal experimental data extraction method with embedded intelligent agent search mechanism provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0060] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for extracting multimodal experimental data by embedding an intelligent agent search mechanism, characterized in that, The method includes: Multimodal parsing is performed on the input target documents to identify multimodal regions and assign a globally unique source identifier to each independent semantic unit, thereby constructing a traceable source data index system; Based on the domain ontology knowledge base, the research objects in the target documents are identified, and synonymous expressions are mapped to standardized entity identifiers through ontology anchoring. By calling multiple data extraction agents with different modalities and combining them with the source data indexing system, data extraction of each modality is performed in parallel on the target documents, and structured triplet data is output. Based on entity identifiers, the triplet data is cross-modal aligned, data conflicts are detected and arbitrated, the truth value of the fused data is determined and reverse tracing verification is performed, and structured knowledge base entries are generated.

2. The multimodal experimental data extraction method with embedded intelligent agent search mechanism as described in claim 1, characterized in that, Before performing multimodal analysis on the input target document, the following steps are included: Construct a dual-database knowledge system for multimodal experimental data, wherein the dual-database knowledge system includes a domain ontology knowledge base and a data extraction rule base; Based on the aforementioned dual-database knowledge system, an extraction tool based on task definition is constructed, wherein the extraction tool includes an ontology recognition agent and a data extraction intelligent agent.

3. The multimodal experimental data extraction method with embedded intelligent agent search mechanism as described in claim 1, characterized in that, Construct a traceable source data indexing system, including: Perform layout analysis on the target documents and segment them into multimodal regions; For the multimodal regions, differentiated segmentation is performed on each modal region, breaking it down into the smallest semantic units. Specifically, the text modality is segmented based on independent sentences, the table modality is segmented based on data lines, and the chart modality is extracted based on core labeled elements. For each smallest semantic unit, a globally unique identifier containing the region type and hierarchical relationship is assigned to construct a fine-grained source data indexing system.

4. The multimodal experimental data extraction method with embedded intelligent agent search mechanism as described in claim 1, characterized in that, Through ontology anchoring, synonymous representations are mapped to standardized entity identifiers, including: Based on the predefined entity naming rules in the domain ontology knowledge base, the ontology recognition agent is activated to perform a full-text scan of the target document and identify multiple expressions of the same research object. For the aforementioned diverse representation methods, a semantic mapping method is used to uniformly map synonymous representations to standardized entity identifiers, wherein the entity identifier is a unique primary key for document data association, satisfying cross-modal data anchoring.

5. The multimodal experimental data extraction method with embedded intelligent agent search mechanism as described in claim 2, characterized in that, The data extraction agent includes a text expert agent, a table expert agent, and a visual relationship expert agent, and the data extraction agent is scalable. The text expert agent identifies material composition, performance parameters, values, units, and test conditions as identification targets. The table expert agent is used to parse the table structure and extract structured data; The visual relationship expert agent is used to extract key parameters from chart titles and annotation information.

6. The multimodal experimental data extraction method with embedded intelligent agent search mechanism as described in claim 5, characterized in that, Data extraction for each modality is performed in parallel on the target literature, outputting structured triplet data, including: The text expert agent, table expert agent, and visual relationship expert agent in the data extraction agent perform data extraction in parallel to obtain the extraction results. The extraction results are labeled based on a unified entity identifier, and the data is structurally transformed according to the standardized output format in the data extraction rule base to determine the triplet data.

7. The multimodal experimental data extraction method with embedded intelligent agent search mechanism as described in claim 6, characterized in that, The text expert agent extracts the target by identifying it and converts it into text triples that characterize the material-performance-numerical value. The table expert agent parses the table structure and converts it into a table triple representing sample-test value-condition; Visual relationship expert agents identify key parameters in chart titles and annotations through image analysis and convert them into chart triples representing identifier-performance-value. Add the text triples, table triples, and chart triples to the triple data.

8. The multimodal experimental data extraction method with embedded intelligent agent search mechanism as described in claim 1, characterized in that, After performing reverse tracing verification, the following is included: If the verification fails, a loop error correction will be automatically triggered, with at least one of the following as the triggering condition: missing data, inconsistent data, or broken traceability chain. As the cyclic error correction step is triggered, iterative arbitration processing is executed until the verification is passed. The iterative arbitration processing method is either the corresponding data extraction proxy call or cross-modal fusion arbitration.

9. The multimodal experimental data extraction method with embedded intelligent agent search mechanism as described in claim 8, characterized in that, Generate structured knowledge base entries, including: For each verified data truth value, quality control information is attached, including a data consistency score, a traceability integrity mark based on the source identifier, and a verification status identifier. The structured knowledge base entries are generated by jointly encapsulating the quality control information, standardized entity identifiers, triplet data, source identifier traceability chains, and conflict arbitration records.

10. A multimodal experimental data extraction system with an embedded intelligent agent search mechanism, characterized in that, The system is used to implement the multimodal experimental data extraction method with embedded agent search mechanism as described in any one of claims 1 to 9, the system comprising: The source data indexing system construction module is used to perform multimodal analysis on the input target documents, identify multimodal regions, assign a globally unique source identifier to each independent semantic unit, and build a traceable source data indexing system. The synonym and heteronym mapping module is used to identify research objects in target documents based on the domain ontology knowledge base and map synonym and heteronym representations to standardized entity identifiers through ontology anchoring. The data extraction execution module is used to call multiple data extraction agents of different modalities, and in conjunction with the source data indexing system, to perform data extraction of each modality in parallel on the target documents, and output structured triplet data; The knowledge base entry generation module is used to perform cross-modal alignment of the triplet data based on entity identifiers, detect and arbitrate data conflicts, determine the truth value of the fused data and perform reverse tracing verification, and generate structured knowledge base entries.

Images