Multi-modal knowledge extraction method and system based on multi-agent collaborative optimization

Abstract

The invention provides a multi-modal knowledge extraction method and system based on multi-agent collaborative optimization, and relates to the technical field of knowledge extraction, and the method comprises the steps: carrying out the multi-modal deconstruction of an original document to be extracted; constructing a multi-modal agent, respectively executing feature extraction and preliminary knowledge extraction, and outputting a single-modal multi-component system; based on a cross-modal knowledge graph, mapping information of different modals to a unified semantic node, and establishing cross-modal association and analyzing a logic chain through a graph neural network and a causal reasoning module; dynamically allocating resources according to the importance of map nodes, and screening structured knowledge; and through confidence analysis and node traceability evaluation, an intelligent agent cooperation mechanism is optimized, and increment correction is carried out on a result. According to the method and the device, the technical problem of low knowledge extraction accuracy and efficiency caused by insufficient multi-modal knowledge collaborative mining capability due to knowledge extraction of literatures by adopting a single agent in the prior art can be solved, and the knowledge extraction quality and efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of knowledge extraction technology, specifically to a multimodal knowledge extraction method and system based on multi-agent collaborative optimization. Background Technology

[0002] Documents typically contain multiple modalities, including text, images, tables, and formulas, which complement each other and contain rich and complex deep-seated knowledge. Different modalities within a document differ significantly in their expressive structure, information density, and semantic characteristics. However, current mainstream document knowledge mining methods generally have limitations, especially when dealing with complex documents containing multiple modalities such as text, images, tables, and formulas. Existing technologies often employ single models or lack modality-specific processing strategies, failing to fully leverage the independent value and synergistic potential of each modality. This leads to the omission of key information and biases in semantic understanding. Furthermore, existing methods are insufficient in multimodal knowledge collaborative mining, lacking modeling and systematic joint reasoning mechanisms for complex cross-modal semantic relationships. This limits the depth and breadth of knowledge mining, preventing personalized adjustments based on specific content and needs, resulting in low accuracy and efficiency in knowledge mining.

[0003] In summary, existing technologies suffer from low accuracy and efficiency in knowledge extraction due to the use of single models or agents for knowledge extraction from documents and insufficient ability to collaboratively mine multimodal knowledge. Summary of the Invention

[0004] This application provides a multimodal knowledge extraction method and system based on multi-agent collaborative optimization, which addresses the technical problems of low accuracy and efficiency in knowledge extraction caused by the use of single models or agents to extract knowledge from documents and insufficient ability to collaboratively mine multimodal knowledge.

[0005] In view of the above problems, this application provides a multimodal knowledge extraction method and system based on multi-agent cooperative optimization.

[0006] Firstly, this application provides a multimodal knowledge extraction method based on multi-agent collaborative optimization. This method is implemented through a multimodal knowledge extraction system based on multi-agent collaborative optimization. The method includes: performing multimodal deconstruction on the original document to be extracted to obtain standardized text data, image and title data, and table and title data; constructing multimodal agents to perform feature extraction and preliminary knowledge extraction on the original document to be extracted, respectively, and outputting single-modal tuples; and based on cross-modal knowledge graphs... The spectrum maps the text entities, key image regions, and table fields of the single-modal tuples to unified semantic nodes. It models the relationships between nodes using a graph neural network and embeds a causal reasoning module to parse the cross-modal logic chains of the initial knowledge extraction. Based on the importance of the graph nodes, it dynamically allocates collaborative resources to each agent, performs structured filtering of the cross-modal logic chains based on a preset knowledge pattern template, and outputs standardized tuple knowledge results. Based on the extraction confidence analysis and node tracing evaluation of the standardized tuple knowledge results, it optimizes the agent collaboration mechanism and constructs a collaborative optimization processing space to incrementally correct the knowledge extraction results.

[0007] Optionally, the original text of the document to be extracted is subjected to image recognition and layout analysis to extract paragraph hierarchical structure, image areas, table areas and corresponding titles and explanatory information; the extracted paragraph hierarchical structure, image areas, table areas and corresponding titles and explanatory information are subjected to text, image and table standardization processing respectively to obtain standardized text data, image and title data, table and title data, wherein the standardized data is a multimodal input representation with consistent structure and uniform format.

[0008] Optionally, the multimodal intelligent agent includes at least: a text mining intelligent agent, an image mining intelligent agent, and a table mining intelligent agent.

[0009] Optionally, based on the deconstructed input data type, the corresponding modality's agent is invoked to extract modal features from the standardized text data, images, and tables, respectively, and to identify entities, relationships, and data. A preliminary structured tuple is constructed according to the characteristics of each modality, and a single-modality tuple is output according to the identified modal features. Specifically, for the text modality, entity-relation triples are output based on syntactic analysis; for the image modality, quantitative description triples are generated by combining the title; and for the table modality, table triples are output based on the mapping relationship of indicator fields.

[0010] Optionally, based on a preset semantic unification standard, text entities, key regions of images, and table fields are mapped to nodes of a knowledge graph, and the source modality and type label of the nodes are labeled; based on the modal feature relationships in the single-modal tuple, initial semantic edge relationships between nodes are established, including co-occurrence relationships of co-originating entities, cross-modal descriptive referential relationships, and structural mapping relationships, to form a multimodal fusion cross-modal knowledge graph.

[0011] Optionally, semantic alignment is performed on nodes of different modalities, and semantic embedding updates are performed on cross-modal knowledge graph nodes based on graph attention mechanism. The semantic similarity and structural association weights between nodes are learned through adjacency propagation. A causal reasoning module is embedded to identify causal paths based on the edge relationships of directional semantics in the graph and output the cross-modal logical chain.

[0012] Optionally, the causal reasoning module identifies edges with directional semantics by parsing the directional field of the relational predicate in the triples and performs semantic tagging, including causal words and semantic directions; based on the semantic tags, multiple triples with causal related entities are spliced ​​into a coherent causal path to construct the cross-modal logic chain.

[0013] Optionally, importance coefficients are configured based on the centrality index, modal semantic density, and relevance to the current task objective of each node in the graph; the task scheduling order, processing frequency, and resource allocation ratio among the modal agents are dynamically adjusted according to the importance coefficients; for the triples fused in the cross-modal logic chain, a matching verification of the preset knowledge pattern template is performed, including entity type verification, triple structure integrity check, and predicate semantic classification operation; the triples that meet the matching verification are formatted and output to obtain the standardized tuple knowledge result.

[0014] Optionally, the standardized tuple knowledge results are evaluated by fusion based on modality consistency, path support, and entity structure stability to obtain confidence. For tuple knowledge results with confidence below a preset threshold, node tracing is performed to identify the corresponding modality source and extraction path. Based on the node tracing results, the collaborative strategy and model parameters of the corresponding modality agents are adjusted to obtain a collaborative optimization processing space. The multiple agents further interact with each other for incremental correction of the knowledge extraction results.

[0015] Secondly, this application also provides a multimodal knowledge extraction system based on multi-agent collaborative optimization, used to execute the multimodal knowledge extraction method based on multi-agent collaborative optimization as described in the first aspect. The multimodal knowledge extraction system based on multi-agent collaborative optimization includes: a multimodal deconstruction module for performing multimodal deconstruction on the original document to be extracted, obtaining standardized text data, image and title data, and table and title data; a knowledge extraction module for constructing multimodal agents to perform feature extraction and preliminary knowledge extraction of the corresponding modality on the original document to be extracted, outputting a single-modal tuple; and a cross-modal parsing module for using a cross-modal knowledge graph. The system maps the text entities, key image regions, and table fields of the single-modal tuples to unified semantic nodes, models the relationships between nodes using a graph neural network, and embeds a causal reasoning module to parse the cross-modal logical chains of the preliminary knowledge extraction. A structured filtering module dynamically allocates collaborative resources to each agent based on the importance of graph nodes, performs structured filtering of cross-modal logical chains based on a preset knowledge pattern template, and outputs standardized tuple knowledge results. An incremental correction module optimizes the agent collaboration mechanism based on the extraction confidence analysis and node tracing evaluation of the standardized tuple knowledge results, constructs a collaborative optimization processing space, and incrementally corrects the knowledge extraction results.

[0016] One or more technical solutions provided in this application have at least the following beneficial effects:

[0017] By performing multimodal deconstruction on the original text of the document to be extracted, standardized text data, image and title data, and table and title data are obtained. A multimodal agent is constructed to perform feature extraction and preliminary knowledge extraction of the original text of the document to be extracted in the corresponding modality, and output a single-modal tuple. Based on a cross-modal knowledge graph, the text entities, key regions of images, and table fields of the single-modal tuple are mapped to unified semantic nodes. The relationship between nodes is modeled through a graph neural network, and a causal reasoning module is embedded to parse the cross-modal logical chain of the preliminary knowledge extraction. The collaborative resources of each agent are dynamically allocated according to the importance of the graph nodes. The cross-modal logical chain is structured and filtered based on a preset knowledge pattern template, and standardized tuple knowledge results are output. Based on the extraction confidence analysis and node source tracing evaluation of the standardized tuple knowledge results, the agent collaboration mechanism is optimized, and a collaborative optimization processing space is constructed to incrementally correct the knowledge extraction results. In other words, by performing multimodal deconstruction of the original text of the document to be extracted, a multimodal intelligent agent is constructed to achieve differentiated processing, cross-modal knowledge graphs and graph neural networks are used to achieve deep collaborative reasoning, and the quality and efficiency of knowledge extraction are improved through dynamic resource allocation and adaptive optimization mechanisms. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the multimodal knowledge extraction method based on multi-agent cooperative optimization proposed in this application.

[0020] Figure 2 This is a schematic diagram of the structure of the multimodal knowledge extraction system based on multi-agent collaborative optimization in this application.

[0021] Figure labeling: Multimodal deconstruction module 11, knowledge extraction module 12, cross-modal parsing module 13, structured filtering module 14, incremental correction module 15. Detailed Implementation

[0022] This application provides a multimodal knowledge extraction method and system based on multi-agent collaborative optimization to address the technical problem of low accuracy and efficiency in knowledge extraction caused by the insufficient ability to collaboratively mine multimodal knowledge in existing technologies, which often rely on single models or agents for document knowledge extraction. By performing multimodal deconstruction of the original document to be extracted, constructing multimodal agents for differentiated processing, utilizing cross-modal knowledge graphs and graph neural networks to achieve deep collaborative reasoning, and improving the quality and efficiency of knowledge extraction through dynamic resource allocation and adaptive optimization mechanisms, this application addresses this issue.

[0023] Example 1, as Figure 1 As shown, this application provides a multimodal knowledge extraction method based on multi-agent cooperative optimization. This method is applied to a multimodal knowledge extraction system based on multi-agent cooperative optimization. The method specifically includes the following steps:

[0024] The original text of the extracted document is deconstructed in a multimodal manner to obtain standardized text data, image and title data, and table and title data.

[0025] Furthermore, this application also includes the following steps: performing image and text recognition and layout analysis on the original text of the document to be extracted, extracting paragraph hierarchical structure, image areas, table areas and corresponding titles and explanatory information; performing text, image and table standardization processing on the extracted paragraph hierarchical structure, image areas, table areas and corresponding titles and explanatory information respectively, to obtain standardized text data, image and title data, and table and title data, wherein the standardized data is a multimodal input representation with consistent structure and uniform format.

[0026] Specifically, the original text of the document to be extracted is the document from which knowledge extraction is required, such as scientific and technological literature. These documents are typically in PDF, Word, or other formats and contain multimodal information including text, images, tables, and formulas. Multimodal refers to data formats that simultaneously contain multiple types of information, such as text, images, tables, and formulas, and are characterized by diverse information sources and rich expression methods. Multimodal deconstruction involves breaking down the different information modalities (such as text, images, and tables) in the document so that each modality can be processed and analyzed independently to extract its respective information.

[0027] The process involves image and text recognition and layout analysis of the original document to be extracted. Image and text recognition identifies the images and text within the original document, while layout analysis analyzes the document's visual layout, identifying paragraphs, headings, list structures, and the position and hierarchy of non-text elements such as images and tables, reflecting the document's content hierarchy and organization. Image and text recognition processing is then performed on the original document, using OCR tools (such as PaddleOCR and Tesseract) to convert mixed text and images into a machine-readable format. OCR not only identifies words but also locates the precise coordinates of each text block on the page. A trained layout recognition model (such as LayoutLMv3 trained on PubLayNet) is used to analyze the page layout, identifying which areas are headings, body paragraphs, images, and tables. Specifically, through comprehensive analysis of features such as font size, position, and numbering rules, the hierarchical structure of paragraphs is identified, and a semantic structure tree is generated. For image areas, the bounding boxes of the images are extracted and associated with the headings and explanatory text above and below them. Table area extraction is also based on layout structure features and OCR recognition information. Table boundaries are identified by analyzing horizontal and vertical lines, field name distribution, and whitespace alignment features, and then the header fields and numerical content are extracted. Simply put, the system identifies the images and text information in the original document to be extracted, analyzes the document's layout structure, and extracts the paragraph hierarchy, the location of images and tables, and the corresponding titles and descriptions. For example, if the document contains an image, the system will identify the boundaries of the image area, mark its location, and extract the descriptive text related to the image (such as the title and description).

[0028] Paragraph hierarchy refers to the way text content is organized, such as chapter titles, section titles, body paragraphs, list items, etc. These have parent-child or parallel relationships, forming a hierarchical structure. Each level corresponds to a semantically progressively refined content unit, which helps with contextual understanding and knowledge extraction. In addition, it includes image areas and their corresponding titles and explanatory information; table areas and their corresponding titles and explanatory information. Titles are brief summaries of the image or table content, while explanatory information provides further explanation or supplementary information. For example, in a 15-page PDF paper of a scientific document, image-text recognition and layout analysis were performed, extracting 6 first-level headings, 17 second-level headings, and 31 third-level headings, with a recognition accuracy of 95.2% (compared to manual annotation); 9 images were extracted, with an average size of 572×460px; the title extraction accuracy was 91.8%; and the explanatory text association accuracy was 89.1%. Six tables were extracted, with an average of 8 columns; the field recognition accuracy was 93.4%; and the title and explanatory text recognition accuracy was 87.6%.

[0029] The extracted paragraph hierarchy, image areas, table areas, and corresponding titles and descriptions are standardized for text, images, and tables, respectively. This process transforms data from different sources, formats, and structures into a unified and standardized form, resulting in standardized text data, image and title data, and table and title data. After standardization, the three modalities of text, images, and tables are all converted into data sets with the same data structure (e.g., represented as a record or tensor) and the same format (e.g., specific data types and length limits). Through precise image and text recognition and layout analysis, the rich content of the document is obtained, and its organizational structure is understood. Standardization solves the problem of diverse sources and chaotic formats of multimodal data, transforming the original, unstructured document content into a clearly structured and uniformly formatted data representation. This ensures consistency in the input received by different agents and avoids processing errors or inefficiencies caused by data format mismatches.

[0030] Construct a multimodal intelligent agent to perform feature extraction and preliminary knowledge extraction of the corresponding modality in the original text of the document to be extracted, and output a single-modal tuple.

[0031] Furthermore, this application also includes the following steps: the multimodal intelligent agent includes at least: a text mining intelligent agent, an image mining intelligent agent, and a table mining intelligent agent.

[0032] Furthermore, this application also includes the following steps: based on the deconstructed input data type, calling the corresponding modality's agent to extract modal features from the standardized text data, images, and tables respectively, and identifying entities, relationships, and data; constructing preliminary structured tuples according to the characteristics of each modality, and outputting single-modality tuples according to the identified modal features, wherein the text modality outputs entity-relationship triples based on syntactic analysis, the image modality outputs quantitative description triples based on the title, and the table modality outputs table triples based on the mapping relationship of indicator fields.

[0033] Specifically, multimodal intelligent agents are intelligent models built upon large language models, possessing autonomous decision-making, information processing, and interaction capabilities. They can independently execute specific tasks and collaborate with other intelligent agents. Each agent focuses on data processing of a particular modality (such as text, images, or tables) and can work collaboratively with other agents to complete complex knowledge mining tasks. Multimodal intelligent agents include text mining agents, image mining agents, and table mining agents, among others. Each agent possesses independent task processing capabilities and can share information with other agents, thereby improving overall efficiency.

[0034] Text mining agents focus on extracting structured knowledge from text data. Using Natural Language Processing (NLP) techniques, they analyze the semantics, entities, relationships, and events within the text and transform them into structured information. For example, if a document discusses the efficiency improvement of solar cells, the text mining agent can identify "solar cell" as an entity and "efficiency improvement" as a relationship, generating structured knowledge of "solar cell → efficiency improvement." Image mining agents analyze images and their related titles, extracting scientific knowledge relevant to the document content. Using computer vision techniques, they perform image recognition, region extraction, and object detection to obtain information from images and associate it with the document content. For example, if a document contains a trend chart of solar cell efficiency changes, the image mining agent can identify key information in the image (such as year and efficiency value) and transform it into structured data. Table mining agents focus on extracting structured information from tables and their titles. They can identify data, fields, and row / column structures in tables and transform them into structured data suitable for analysis. For example, if a document contains a table displaying the efficiency of different solar cell materials, the table mining agent can extract fields from the table, such as material name and efficiency, and transform them into structured data.

[0035] By constructing a unified collaboration protocol, information sharing and knowledge complementarity among different agents are achieved, further improving overall performance. Each agent, after completing its task, shares the results with other agents. During knowledge extraction, a feedback mechanism evaluates the accuracy of the extraction results. If there are deviations, the collaboration strategy of the agents is adjusted based on the evaluation feedback, undergoing iterative optimization. For example, if there are errors in the extraction of table data, the table agent will correct them based on feedback to ensure the accuracy of the final result. For instance, a text mining agent extracts the relationship between material A and efficiency improvement, identifying that material A's efficiency was 20% in 2018 and increased to 22% in 2020; an image mining agent identifies the efficiency of material A in charts in different years, extracting data showing that material A's efficiency was 20% in 2018 and 22% in 2020; a table mining agent identifies and extracts data from tables showing material A at 20% and material B at 22%, converting it into structured data. Through the collaboration framework, this information is integrated, resulting in the knowledge extraction result that material A's efficiency improved from 20% to 22%.

[0036] Using general-purpose models such as LLaVA (for joint image and text processing), TAPAS (for table comprehension), and ChatGLM (for text processing) as a foundation, we fine-tune the basic knowledge mining tasks for text, images, and tables respectively, ensuring that each agent possesses basic extraction capabilities in its respective modality. Each time, we select a batch of multimodal task data for joint training of the multimodal agents. Through a multi-agent collaborative learning mechanism, we strengthen the information fusion and complementarity between modalities, continuously improving knowledge extraction capabilities. We input real scientific literature samples, scheduling different modal data in each round of joint training, such as simultaneously inputting text paragraphs, experimental images, and corresponding tables, constructing a task: extracting the correspondence between material preparation methods and performance indicators. Through mutual feedback learning among the multi-agents, we develop collaborative extraction capabilities.

[0037] During the inference phase, the multimodal agent automatically schedules the corresponding agent to extract information in a specific modality based on the content of the sample to be extracted. Through information feedback and fusion among multiple agents, the final tuple knowledge output is formed. Dynamic collaboration among multiple agents ensures the comprehensiveness and accuracy of the extraction results. For example, if the task is to extract performance change trends, the table and image agents are activated first, while the text agent only assists in confirming the meaning of fields. Based on user or human feedback, the knowledge extraction results are evaluated and corrected. The parameters of each agent are further fine-tuned using the feedback information, the collaboration mechanism is optimized, and the adaptability and robustness of the multimodal agent to complex multimodal tasks are continuously improved.

[0038] By customizing specialized agents for each modality and fine-tuning and jointly training them based on a general large model, we ensure that each agent has the basic ability to handle tasks in its domain. We also improve overall performance through collaborative learning between modalities, fully utilize the complementarity of information from different modalities, and significantly improve the coverage and accuracy of knowledge extraction.

[0039] Based on the deconstructed data type, the corresponding modality's intelligent agent is invoked. If the deconstructed data type is text and image, then the text mining intelligent agent and image mining intelligent agent are invoked; if the deconstructed data type is image and table, then the image mining intelligent agent and table mining intelligent agent are invoked. Based on the invoked intelligent agent, modal features are extracted from the standardized text data, images, and tables respectively to identify entities, relationships, and data. Entities are concepts or objects with clear boundaries and semantics in the literature; relationships refer to the semantic connections between two entities; and data are specific numerical values ​​or facts related to entities.

[0040] Based on the characteristics of each modality, preliminary structured tuples are constructed, transforming raw, unstructured modal data into data units with specific formats and meanings that are easier for computers to process, according to certain rules. Single-modal tuples are output based on the identified modal features, including text modality, image modality, and tabular modality. The text mining agent constructs entity-relation-data entity relation triples through syntactic analysis; for example, extracting a triple like "solar cell - efficiency improvement - 22%" from text. Similarly, the image mining agent identifies key regions in images (such as efficiency trend charts, graphs, etc.) using computer vision technology and outputs quantitative description triples based on image title information. For image modality, since images themselves lack directly readable semantics, their titles or descriptive text are needed to understand the image content. The agent analyzes the image content (e.g., identifying cell structures under a microscope) and its title (e.g., changes in cell nucleus morphology after 48 hours of drug X treatment), and then generates a triple containing quantitative information or a more specific description.

[0041] For tabular modalities, tables typically have a defined row and column structure, containing metrics (row names, column names) and specific values ​​(cell data). Metric field mapping refers to identifying the logical relationships between different fields (columns) in the table and constructing triples based on these mappings. This involves parsing the fields, rows, and columns in the table to identify the data in each row or column. Based on the table's title and field information, corresponding triples are constructed. For example, identifying the value 4.8 at the intersection of the "Average" column (Experimental Group A) and the "Telomere Length" row (4.8) generates the table triple (Experimental Group A, Average Telomere Length, 4.8).

[0042] Information scattered across text, images, and tables, after feature extraction, is transformed into unified, structured single-modal knowledge representation units (triplets). Each agent generates triplets that best match the characteristics of its own modality (syntax of text, visual and title combination of images, structure and field mapping of tables), ensuring the targeting and effectiveness of information extraction.

[0043] Based on cross-modal knowledge graphs, the text entities, key regions of images, and table fields of the single-modal tuples are mapped to unified semantic nodes. The relationships between nodes are modeled through graph neural networks, and a causal reasoning module is embedded to parse the cross-modal logic chain of the preliminary knowledge extraction.

[0044] Furthermore, this application also includes the following steps:

[0045] Based on a pre-defined semantic standard, text entities, key regions of images, and table fields are mapped to nodes of a knowledge graph, and the source modality and type label of the nodes are marked. Based on the modal feature relationships in the single-modal tuple, initial semantic edge relationships between nodes are established, including co-occurrence relationships of co-originating entities, cross-modal descriptive referential relationships, and structural mapping relationships, forming a multimodal fusion cross-modal knowledge graph.

[0046] Specifically, the predefined semantic unification standard is a predefined unified semantic framework or standard used to ensure that knowledge extracted from different modalities (text, images, tables) has a consistent representation. The predefined semantic unification standard defines how information from different sources (such as text entities, key regions of images, table fields, etc.) is mapped to unified knowledge graph nodes, and their modality types and labels are annotated. This ensures that entities from text, images, and tables, even with different representations, can be mapped to nodes in the knowledge graph with consistent or similar meanings.

[0047] Based on a pre-defined semantic standard, text entities, key regions of images, and table fields are mapped to nodes in a knowledge graph. Each node is labeled with its source modality (e.g., text, image, table) and type (e.g., entity, relation, data), which helps distinguish information from different sources and types. Entities (e.g., solar cells) in text data are extracted and mapped to nodes in the knowledge graph. Key regions in image data (e.g., data points and trend lines in tables) are also mapped to nodes, while fields in tables (e.g., material name, efficiency) are mapped to nodes.

[0048] Based on the modal feature relationships in unimodal tuples, edge relationships are established between nodes. Initial semantic edge relationships are the relationship lines connecting two nodes in the knowledge graph. "Initial" means that these edge relationships are mainly based on unimodal information or simple cross-modal associations and have not yet undergone complex reasoning verification. Semantic representation edges have explicit meanings, such as regulation or measurement values, and are established based on the relational information of unimodal triples.

[0049] In knowledge graphs, edges represent relationships between nodes. Initial semantic edge relationships refer to the preliminary relationships between nodes constructed based on features and relationships extracted from unimodal tuples. These include co-occurrence of text entities, descriptive reference relationships between images and text, or structural mapping relationships between table fields and other modal information. Co-occurrence relationships of homologous entities are relationships between entities that appear simultaneously in the same unimodal data unit (such as the same sentence, the same image, or the same table row). Cross-modal descriptive reference relationships are relationships established when describing the same or related entities in different modal data. Structural mapping relationships are the cases where the structured relationships between table fields in a table modality are mapped to knowledge graph edges.

[0050] This approach aggregates co-occurrence relationships, cross-modal descriptive and referential relationships, and structural mapping relationships of entities from the same source, integrating them through a graph structure to construct a complete cross-modal knowledge graph. Each node represents a specific piece of information (such as an entity, field, or region), and edges represent relationships between them (such as co-occurrence, referential, or mapping relationships). By converging information from different sources into unified graph nodes, it achieves structural fusion of heterogeneous information.

[0051] Furthermore, this application also includes the following steps: semantic alignment of nodes of different modalities, semantic embedding and updating of cross-modal knowledge graph nodes based on graph attention mechanism, learning semantic similarity and structural association weights between nodes through adjacency propagation; embedding a causal reasoning module, identifying causal paths based on the edge relationships of directional semantics in the graph, and outputting the cross-modal logical chain.

[0052] Furthermore, this application also includes the following steps: the causal reasoning module identifies edges with directional semantics by parsing the directional field of the relational predicate in the triple and performs semantic labeling, including causal words and semantic directions; based on the semantic labeling, multiple triples with causal related entities are spliced ​​into a coherent causal path to construct the cross-modal logic chain.

[0053] Specifically, to refine the cross / deep correlation analysis between multimodal agents and enhance their dynamic collaborative adaptability, graph neural network modeling and causal reasoning are deeply integrated into the original multi-agent collaboration framework. A new cross-modal knowledge graph construction module is added, using graph neural networks (GNNs) to achieve unified semantic mapping, modality alignment and embedding optimization, and dynamic resource allocation. Entities in text, key regions in images (such as experimental charts), and structured fields in tables are mapped to graph nodes, assigned unified semantic encoding, and defined as nodes. Edge types are defined based on cross-modal associations (such as text description - chart display - table verification), and graph attention networks are used to dynamically learn the association weights between nodes. A cross-modal contrastive learning algorithm is designed to address the semantic gap problem of heterogeneous data by maximizing the embedding similarity of nodes with similar semantic meanings (such as synthesis methods in text and process flow diagrams in images). The collaboration priority of each agent (text, image, table) is dynamically adjusted based on the importance of nodes in the graph (such as centrality indicators) and task objectives (such as focusing on method mining or data verification). Based on cross-modal graphs, a causal reasoning module is introduced to solve the problem of accurately extracting method-effect logic chains in scientific literature. For example, in materials science literature, if the graph shows a strong correlation between microstructure images and performance parameter tables, then the collaborative reasoning of the image and table agents is activated first.

[0054] Semantic alignment of nodes from different modalities (such as text, images, and tables) is performed to match and unify them, ensuring semantic consistency and enabling them to work collaboratively within the same knowledge graph. Semantic alignment is fundamental to cross-modal data fusion, ensuring that information from different sources can correctly correspond and interact. A graph attention mechanism is used to update the semantic embeddings of nodes in the cross-modal knowledge graph. This mechanism dynamically adjusts node embeddings by weighted propagation of relationships between nodes, allowing semantically similar nodes to be better integrated. For example, if a text node representing a solar cell and an image node representing a solar cell efficiency graph are semantically related, the graph attention mechanism will strengthen their connection.

[0055] Graph attention is a deep learning method based on graph-structured data, which assigns different weights to different nodes in the graph using an attention mechanism. Through graph attention, the relationships between nodes can be dynamically weighted, highlighting important nodes or edges and improving the efficiency and accuracy of graph structure learning. Node semantic embeddings represent the node's features and semantic information. With the help of graph attention, node embeddings are updated through adjacency propagation, enabling them to better capture and represent relationships and structural information with other nodes. Adjacency propagation is a method in graph learning that refers to the propagation of node information through connections with its neighboring nodes, updating the node's embedding representation. In cross-modal knowledge graphs, adjacency propagation helps nodes share information and learn similarities and structural associations between nodes.

[0056] The GATv2 model is used to propagate the graph structure, where each node aggregates semantics through its neighboring nodes and assigns different weights. For example, the neighboring nodes of NiCo2O4 are specific capacitance and morphological features, and the attention weights for these two edges are calculated to be 0.72 and 0.28, respectively. After the update, the NiCo2O4 node vector will incorporate more information about specific capacitance, thereby enhancing the semantic capabilities related to performance indicators.

[0057] The causal reasoning module is used to identify and reason about causal relationships between nodes in a knowledge graph. Specifically, it identifies edges with directional semantics by parsing the directional field of the relational predicates of triples, connecting entities with causal relationships to form causal paths. It traverses all unimodal and cross-modal triples (e.g., temperature increase, cause, decrease in conductivity), identifying causal terms (e.g., cause, promote, inhibit) from the predicate. If the predicate has directionality (e.g., cause is from cause to effect), a directional marker is added to the graph for subsequent reasoning. Directional edges are labeled as directed causal edges, such as catalyst A, promotion, reaction rate, and are marked as causal edges.

[0058] The causal reasoning module identifies causal relationships in a cross-modal knowledge graph, labels edges with directional semantics, and forms causal paths. It analyzes the relational predicates of triples and identifies directional causal terms (such as promoting, causing, influencing, and regulating). When a triple such as Material A—causing—efficiency improvement is detected, the module identifies the causal relationship between Material A and efficiency improvement and labels it as a directional edge. The identified causal relationships are labeled with causal terms and semantic directions, and ordered in the form of <cause entity, relation, effect entity>. Causal terms refer to words that express causal relationships, such as promoting, causing, influencing, and regulating; semantic direction refers to the directionality of the causal relationship, i.e., who is the cause and who is the effect. Edges with causal labels are ordered in <cause entity, relation, effect entity>, and multiple causal paths are concatenated, that is, multiple triples with causally related entities are concatenated into a coherent causal path: cause entity → intermediate entity → effect entity, constructing a cross-modal logical chain.

[0059] Semantic alignment avoids ambiguity and repetition in the representation of information from different modalities, making nodes in the knowledge graph more accurate and consistent. Semantic embedding updates based on graph attention mechanisms enable each node's representation to dynamically absorb information from its most relevant neighbors, thereby accurately capturing the semantics of the node in cross-modal contexts.

[0060] Based on the importance of graph nodes, collaborative resources for each agent are dynamically allocated. Based on preset knowledge pattern templates, cross-modal logic chains are structurally filtered, and standardized plural knowledge results are output.

[0061] Furthermore, this application also includes the following steps:

[0062] Based on the centrality index, modal semantic density, and relevance to the current task objective of each node in the graph, importance coefficients are configured. According to these importance coefficients, the task scheduling order, processing frequency, and resource allocation ratio among the modal agents are dynamically adjusted. For triples fused in the cross-modal logic chain, a preset knowledge pattern template matching verification is performed, including entity type verification, triple structure integrity check, and predicate semantic classification. Triples that satisfy the matching verification are formatted and output to obtain the standardized tuple knowledge results.

[0063] Specifically, in graph theory, centrality metrics are used to measure the importance or influence of a node in a network, including degree centrality (how many nodes it is directly connected to), betweenness centrality (how many shortest paths pass through the node), and proximity centrality (the average distance to all other nodes). Higher node centrality generally means it has more connections or is more critical in the knowledge graph, potentially representing a more core concept or entity. Modal semantic density is the richness or concentration of information represented by a node or its neighboring nodes from a specific modality (text, image, table). For example, if a node has many neighboring nodes from the text modality, then that region has a high semantic density for the text modality. High density may mean the region is more information-rich, but it may also mean redundancy. Relevance to the current task objective is the degree to which nodes in the knowledge graph or the relationships between nodes match the specific task the user currently hopes to complete (such as answering a scientific question, generating a summary in a specific field, etc.). Nodes and relationships with high relevance are more valuable for completing the current task.

[0064] Based on the centrality index, modal semantic density, and relevance to the current task objective of each node in the knowledge graph, the importance of each node is comprehensively evaluated, resulting in an importance coefficient for each node. This quantifies the relative importance of each node in the entire knowledge extraction task. The importance coefficient I of each node is calculated using a weighted average. n :I n =α·C n +β·D n +γ·R n , where C n It is centrality, D n It is semantic density, R n α represents task relevance; β and γ are adjustment parameters (e.g., 0.4, 0.3, 0.3). For example, assuming the centrality index of text is 0.43, the modal semantic density is 0.32, and the relevance to the current task objective is 0.91, then the text importance coefficient is 0.584; if the centrality index of image is 0.21, the modal semantic density is 0.85, and the relevance to the current task objective is 0.72, then the text importance coefficient is 0.562; if the centrality index of table is 0.35, the modal semantic density is 0.63, and the relevance to the current task objective is 0.88, then the text importance coefficient is 0.608. The task scheduling order among the modal agents is table, text, image, with priority given to table mining agents.

[0065] Based on the importance coefficient of each modality, the task scheduling order, processing frequency, and resource allocation ratio among agents are dynamically adjusted to ensure that critical modalities are processed first. The processing frequency is a linear expansion of the number of times each modality is called within a time window according to its importance coefficient; for example, a high-weight modality is called 4 times every 10 seconds, a medium-weight modality 2 times, and a low-weight modality 1 time. Resources are allocated according to the normalized ratio of the coefficients, such as memory / threads / number of model replicas. During task execution, multimodal agents dynamically adjust the task execution order and resource allocation based on their importance coefficients. For example, if a text agent occupies an important position in the current task, it will receive more computing resources and be prioritized for task execution. If a node is determined to be very important, agents related to that node (if the node mainly relates to text information, then the text agent is prioritized) are given priority for processing, or their processing frequency is increased (allowing them to process data faster), or even more computing resources are allocated to them to ensure the efficiency and accuracy of critical information processing. Conversely, for nodes or modalities with lower importance, investment is reduced, and effort is allocated according to the actual value of the knowledge to improve overall efficiency. For example, tables are assigned to the first task, processed 6 times per minute, and account for 40% (48 units) of resources; text is assigned to the second task, processed 5 times per minute, and account for 35% (42 units) of resources; and images are assigned to the third task, processed 3 times per minute, and account for 25% (30 units) of resources.

[0066] After triple extraction, validation is performed according to a pre-defined knowledge pattern template. This includes entity type verification (checking whether the entity types in the triples conform to predefined types), triple structure integrity checking (checking whether the structure of the triples conforms to entities, format, and data), and predicate semantic classification (semantic classification and annotation based on the semantic type of the relational predicates). Triples that pass the matching validation are converted into a unified and standardized output format, forming standardized tuple knowledge results. The validated triples are output according to standardized formats (such as RDF, JSON-LD, etc.), and the results are standardized to ensure they conform to a unified data structure.

[0067] By dynamically adjusting the task scheduling, processing frequency, and resource allocation of the intelligent agent, computing resources are concentrated on the most important knowledge nodes, avoiding wasting resources on low-value information, thereby improving the overall efficiency and processing speed of knowledge extraction.

[0068] Based on the extraction confidence analysis and node tracing evaluation of the standardized tuple knowledge results, the agent collaboration mechanism is optimized, and a collaborative optimization processing space is constructed to incrementally correct the knowledge extraction results.

[0069] Furthermore, this application also includes the following steps: performing a fusion evaluation on the standardized tuple knowledge results based on modality consistency, path support, and entity structure stability to obtain confidence; performing node tracing on tuple knowledge results with confidence below a preset threshold to identify the corresponding modality source and extraction path; adjusting the collaborative strategy and model parameters of the corresponding modality agents based on the node tracing results to obtain a collaborative optimization processing space, and further engaging in information interaction among multiple agents for incremental correction of knowledge extraction results.

[0070] Specifically, based on modal consistency, path support, and entity structure stability, the standardized tuple knowledge results are fused and evaluated to assess their credibility and ensure their quality. Confidence is an evaluation value of the credibility of a tuple knowledge result in the knowledge graph, typically used to represent the accuracy and reliability of the result. Tuples with confidence scores below a certain threshold require further examination and correction. Modal consistency refers to the semantic consistency and logical matching between different modalities of data (such as text, images, tables, etc.). Modal consistency ensures that data from different sources can be correctly fused in the knowledge graph, avoiding semantic conflicts. Path support is the frequency and support of a knowledge path (such as causal chains, entity relationship chains, etc.) in the knowledge graph. Paths with high path support usually indicate strong semantic support in the data and are considered credible. Entity structure stability is the structural stability of entities in the knowledge graph. Entities with high entity structure stability appear consistently in multiple modalities or multiple triples, indicating that the representation and relationships of the entity in the knowledge graph have strong consistency and reliability.

[0071] Modal consistency, path support, and entity structure stability are typically values ​​between 0 and 1. A weighted sum is calculated based on these three dimensions to determine the confidence level, usually with weights of 0.3, 0.4, and 0.3. For example, if modal consistency is calculated to be 0.9, path support to be 0.85, and entity structure stability to be 0.95, the overall confidence level is 0.89.

[0072] A preset threshold is a pre-defined numerical limit used to determine whether the confidence level of a tuple is high enough to be considered reliable knowledge. Tuples below this threshold are considered questionable or require further verification. If the confidence level of a tuple falls below the preset threshold, it is not discarded directly. Instead, a deeper diagnostic process is initiated, involving node tracing to identify the origin of the tuple in the knowledge graph. This process identifies the modality (text, image, table, etc.) from which the tuple was extracted and its extraction path (i.e., the steps or agents involved in processing and reasoning). Node tracing traces the original source (whether from text, image, or table) of each node (entity or relation) in a tuple's knowledge result and how it was extracted and combined into the tuple, similar to labeling knowledge with source tags and process records. Through tracing, the process pinpoints the link that caused the tuple's low confidence level—whether it's a deviation in text understanding, an error in image recognition, inaccurate table parsing, or a problem during cross-modal fusion.

[0073] The system traces tuple knowledge results with confidence levels below a preset threshold, identifies their modality (e.g., text, image, table, etc.), and determines their extraction path (i.e., which agents process and infer them). Based on the node tracing results, it dynamically adjusts the collaboration methods between agents of the corresponding modality. For example, if the collaboration between the image agent and the text agent is found to be inefficient, the information transmission method between them is adjusted to increase their collaboration frequency; if the processing capacity of a certain agent is found to be insufficient, its model parameters are adjusted to improve its processing efficiency and accuracy. Based on the tracing results, the collaboration strategy of the corresponding modality agents is adjusted, such as changing the task scheduling order between agents, allowing a certain agent to process first, which may help improve overall accuracy; or adjusting the frequency and content of information sharing between agents, so that they can better verify each other; at the same time, the model parameters inside the corresponding agent are fine-tuned, such as adjusting the entity recognition model weights of the text agent to make it more focused on specific contexts, or adjusting the target detection model threshold of the image agent to make it more robust.

[0074] After adjustments, a collaborative optimization processing space is obtained, defining how to improve its collaborative working mechanism and agent capabilities. Within this space, further interaction among multiple agents incrementally corrects the knowledge extraction results, gradually improving low-confidence tuples and making the final knowledge extraction results more accurate. Incremental correction refers to applying improved strategies and parameters when processing new data, or reprocessing or correcting previously extracted results that have been found to have problems, thereby continuously improving the accuracy and robustness of the entire knowledge extraction system. Through multi-dimensional (modal consistency, path support, entity stability) fusion evaluation and confidence threshold filtering, a large number of low-quality and unreliable knowledge results are effectively filtered out, ensuring the accuracy and credibility of the final output knowledge.

[0075] In summary, the multimodal knowledge extraction method based on multi-agent cooperative optimization provided in this application has the following beneficial effects:

[0076] By performing multimodal deconstruction on the original text of the document to be extracted, standardized text data, image and title data, and table and title data are obtained. A multimodal agent is constructed to perform feature extraction and preliminary knowledge extraction of the original text of the document to be extracted in the corresponding modality, and output a single-modal tuple. Based on a cross-modal knowledge graph, the text entities, key regions of images, and table fields of the single-modal tuple are mapped to unified semantic nodes. The relationship between nodes is modeled through a graph neural network, and a causal reasoning module is embedded to parse the cross-modal logical chain of the preliminary knowledge extraction. The collaborative resources of each agent are dynamically allocated according to the importance of the graph nodes. The cross-modal logical chain is structured and filtered based on a preset knowledge pattern template, and standardized tuple knowledge results are output. Based on the extraction confidence analysis and node source tracing evaluation of the standardized tuple knowledge results, the agent collaboration mechanism is optimized, and a collaborative optimization processing space is constructed to incrementally correct the knowledge extraction results. In other words, by performing multimodal deconstruction of the original text of the document to be extracted, a multimodal intelligent agent is constructed to achieve differentiated processing, cross-modal knowledge graphs and graph neural networks are used to achieve deep collaborative reasoning, and the quality and efficiency of knowledge extraction are improved through dynamic resource allocation and adaptive optimization mechanisms.

[0077] Example 2: Based on the same inventive concept as the multi-agent cooperative optimization-based multimodal knowledge extraction method in Example 1, this application also provides a multi-agent cooperative optimization-based multimodal knowledge extraction system. Please refer to the appendix. Figure 2 The multimodal knowledge extraction system based on multi-agent collaborative optimization includes:

[0078] The multimodal deconstruction module 11 is used to perform multimodal deconstruction on the original text of the document to be extracted, and obtain standardized text data, image and title data, and table and title data. The knowledge extraction module 12 is used to construct a multimodal agent, and perform feature extraction and preliminary knowledge extraction of the corresponding modality on the original text of the document to be extracted, and output a single-modal tuple. The cross-modal parsing module 13 is used to map the text entities, key regions of images, and table fields of the single-modal tuple to unified semantic nodes based on the cross-modal knowledge graph, model the relationship between nodes through graph neural network, and embed a causal reasoning module to parse the cross-modal logical chain of the preliminary knowledge extraction. The structured filtering module 14 is used to dynamically allocate the collaborative resources of each agent according to the importance of the graph nodes, perform structured filtering of the cross-modal logical chain based on the preset knowledge pattern template, and output standardized tuple knowledge results. The incremental correction module 15 is used to optimize the agent collaboration mechanism based on the extraction confidence analysis and node tracing evaluation of the standardized tuple knowledge results, and construct a collaborative optimization processing space to incrementally correct the knowledge extraction results.

[0079] Furthermore, the multimodal deconstruction module 11 in the multimodal knowledge extraction system based on multi-agent collaborative optimization is also used for:

[0080] The original text of the document to be extracted is subjected to image and text recognition and layout analysis to extract paragraph hierarchical structure, image areas, table areas and corresponding titles and descriptions. The extracted paragraph hierarchical structure, image areas, table areas and corresponding titles and descriptions are then subjected to text, image and table standardization processing to obtain standardized text data, image and title data, and table and title data. The standardized data is a multimodal input representation with consistent structure and uniform format.

[0081] Furthermore, the knowledge extraction module 12 in the multi-agent collaborative optimization-based multimodal knowledge extraction system is also used for:

[0082] The multimodal intelligent agent includes at least: a text mining intelligent agent, an image mining intelligent agent, and a table mining intelligent agent.

[0083] Furthermore, the knowledge extraction module 12 in the multi-agent collaborative optimization-based multimodal knowledge extraction system is also used for:

[0084] Based on the deconstructed input data type, the corresponding modality agent is invoked to extract modal features from the standardized text data, images, and tables, and to identify entities, relationships, and data. Preliminary structured tuples are constructed according to the characteristics of each modality. Single-modality tuples are output according to the identified modal features. Specifically, for the text modality, entity-relation triples are output based on syntactic analysis; for the image modality, quantitative description triples are generated by combining the title; and for the table modality, table triples are output based on the mapping relationship of indicator fields.

[0085] Furthermore, the cross-modal parsing module 13 in the multimodal knowledge extraction system based on multi-agent collaborative optimization is also used for:

[0086] Based on a pre-defined semantic standard, text entities, key regions of images, and table fields are mapped to nodes of a knowledge graph, and the source modality and type label of the nodes are marked. Based on the modal feature relationships in the single-modal tuple, initial semantic edge relationships between nodes are established, including co-occurrence relationships of co-originating entities, cross-modal descriptive referential relationships, and structural mapping relationships, forming a multimodal fusion cross-modal knowledge graph.

[0087] Furthermore, the cross-modal parsing module 13 in the multimodal knowledge extraction system based on multi-agent collaborative optimization is also used for:

[0088] Semantic alignment is performed on nodes of different modalities, and semantic embedding updates are performed on cross-modal knowledge graph nodes based on graph attention mechanism. Semantic similarity and structural association weights between nodes are learned through adjacency propagation. A causal reasoning module is embedded to identify causal paths based on the edge relationships of directional semantics in the graph and output the cross-modal logical chain.

[0089] Furthermore, the cross-modal parsing module 13 in the multimodal knowledge extraction system based on multi-agent collaborative optimization is also used for:

[0090] The causal reasoning module analyzes the directional field of the relational predicate in the triples, identifies edges with directional semantics and performs semantic labeling, including causal words and semantic direction; based on the semantic labeling, it splices multiple triples with causal related entities into a coherent causal path to construct the cross-modal logic chain.

[0091] Furthermore, the structured filtering module 14 in the multimodal knowledge extraction system based on multi-agent collaborative optimization is also used for:

[0092] Based on the centrality index, modal semantic density, and relevance to the current task objective of each node in the graph, importance coefficients are configured. According to these importance coefficients, the task scheduling order, processing frequency, and resource allocation ratio among the modal agents are dynamically adjusted. For triples fused in the cross-modal logic chain, a preset knowledge pattern template matching verification is performed, including entity type verification, triple structure integrity check, and predicate semantic classification. Triples that satisfy the matching verification are formatted and output to obtain the standardized tuple knowledge results.

[0093] Furthermore, the incremental correction module 15 in the multi-agent cooperative optimization-based multimodal knowledge extraction system is also used for:

[0094] The standardized plural knowledge results are evaluated by fusion based on modality consistency, path support, and entity structure stability to obtain confidence. For plural knowledge results with confidence below a preset threshold, node tracing is performed to identify the corresponding modality source and extraction path. Based on the node tracing results, the collaborative strategy and model parameters of the corresponding modality agents are adjusted to obtain a collaborative optimization processing space. The multiple agents further interact with each other for incremental correction of the knowledge extraction results.

[0095] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Figure 1 The multimodal knowledge extraction method and specific examples based on multi-agent cooperative optimization in Example 1 are also applicable to the multimodal knowledge extraction system based on multi-agent cooperative optimization in this embodiment. Through the foregoing detailed description of the multimodal knowledge extraction method based on multi-agent cooperative optimization, those skilled in the art can clearly understand the multimodal knowledge extraction system based on multi-agent cooperative optimization in this embodiment. Therefore, for the sake of brevity, it will not be described in detail here.

[0096] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0097] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of this application and its equivalents, this application also intends to include such modifications and variations.

Claims

1. A multimodal knowledge extraction method based on multi-agent cooperative optimization, characterized in that, include: The original text of the document to be extracted is deconstructed in a multimodal manner to obtain standardized text data, image and title data, and table and title data; Construct a multimodal intelligent agent to perform feature extraction and preliminary knowledge extraction of the corresponding modality in the original text of the document to be extracted, and output a single-modal tuple; Based on cross-modal knowledge graphs, the text entities, key regions of images, and table fields of the single-modal tuples are mapped to unified semantic nodes. The relationships between nodes are modeled through graph neural networks, and a causal reasoning module is embedded to parse the cross-modal logic chain of the initial knowledge extraction. Based on the importance of graph nodes, collaborative resources for each agent are dynamically allocated. Based on a preset knowledge pattern template, cross-modal logic chains are structurally filtered, and standardized plural knowledge results are output. Based on the extraction confidence analysis and node tracing evaluation of the standardized tuple knowledge results, the agent collaboration mechanism is optimized, and a collaborative optimization processing space is constructed to incrementally correct the knowledge extraction results.

2. The multimodal knowledge extraction method based on multi-agent cooperative optimization according to claim 1, characterized in that, The multimodal intelligent agent includes at least: a text mining intelligent agent, an image mining intelligent agent, and a table mining intelligent agent.

3. The multimodal knowledge extraction method based on multi-agent cooperative optimization according to claim 1, characterized in that, Multimodal deconstruction of the original text of the document to be extracted includes: The original text of the document to be extracted is subjected to image and text recognition and layout analysis to extract paragraph hierarchical structure, image areas, table areas and corresponding titles and explanatory information; The extracted paragraph hierarchy, image area, table area, and corresponding title and description information are subjected to text, image, and table standardization processing, respectively, to obtain standardized text data, image and title data, and table and title data. The standardized data is a multimodal input representation with consistent structure and uniform format.

4. The multimodal knowledge extraction method based on multi-agent cooperative optimization according to claim 2, characterized in that, Each text document to be extracted undergoes feature extraction and preliminary knowledge extraction for its corresponding modality, outputting a single-modality tuple, including: Based on the deconstructed input data type, the corresponding modal agent is invoked to extract modal features from the standardized text data, images, and tables, and to identify entities, relationships, and data. Based on the characteristics of each modality, a preliminary structured tuple is constructed. According to the features of the identified modality, a single-modality tuple is output. Specifically, for the text modality, entity relation triples are output based on syntactic analysis; for the image modality, quantitative description triples are generated by combining the title; and for the table modality, table triples are output based on the mapping relationship of index fields.

5. The multimodal knowledge extraction method based on multi-agent cooperative optimization according to claim 4, characterized in that, Based on a cross-modal knowledge graph, the text entities, key image regions, and table fields of the unimodal tuples are mapped to unified semantic nodes, including: Based on a pre-defined semantic standard, text entities, key regions of images, and table fields are mapped to nodes in a knowledge graph, and the source modality and type label of the nodes are annotated. Based on the modal feature relationships in the single-modal tuple, initial semantic edge relationships between nodes are established, including co-occurrence relationships of co-originating entities, cross-modal descriptive referential relationships, and structural mapping relationships, forming a multimodal fusion cross-modal knowledge graph.

6. The multimodal knowledge extraction method based on multi-agent cooperative optimization according to claim 5, characterized in that, The graph neural network models the relationships between nodes and embeds a causal reasoning module to parse the cross-modal logic chain of preliminary knowledge extraction, including: Semantic alignment is performed on nodes of different modalities, and semantic embedding updates are performed on cross-modal knowledge graph nodes based on graph attention mechanism. Semantic similarity and structural association weights between nodes are learned through adjacency propagation. An embedded causal reasoning module identifies causal paths based on the edge relationships of directional semantics in the graph and outputs the cross-modal logic chain.

7. The multimodal knowledge extraction method based on multi-agent cooperative optimization according to claim 6, characterized in that, An embedded causal reasoning module identifies causal paths based on the edge relationships of directional semantics in the graph and outputs the cross-modal logic chain, including: The causal reasoning module identifies edges with directional semantics by parsing the directional field of the relational predicate in the triple and performs semantic labeling, including causal words and semantic direction. Based on semantic tagging, multiple triples with causal relationships are concatenated into a coherent causal path to construct the cross-modal logic chain.

8. The multimodal knowledge extraction method based on multi-agent cooperative optimization according to claim 1, characterized in that, Based on the importance of graph nodes, collaborative resources for each agent are dynamically allocated. Cross-modal logic chains are structurally filtered based on preset knowledge pattern templates, and standardized tuple knowledge results are output, including: Importance coefficients are configured based on the centrality index, modal semantic density, and relevance to the current task objective of each node in the graph; Based on the importance coefficient, the task scheduling order, processing frequency, and resource allocation ratio among the various modal agents are dynamically adjusted. For triples fused in cross-modal logic chains, perform matching verification of preset knowledge pattern templates, including entity type verification, triple structure integrity check, and predicate semantic classification operation; The triples that satisfy the matching verification are formatted and output to obtain the standardized tuple knowledge results.

9. The multimodal knowledge extraction method based on multi-agent cooperative optimization according to claim 1, characterized in that, Based on the extraction confidence analysis and node tracing evaluation of the standardized tuple knowledge results, the agent collaboration mechanism is optimized, and a collaborative optimization processing space is constructed to incrementally correct the knowledge extraction results, including: The standardized plural knowledge results are evaluated by fusion based on modality consistency, path support, and entity structure stability to obtain confidence level; For tuple knowledge results with confidence levels below a preset threshold, node tracing is performed to identify the corresponding modality source and extraction path. Based on the node tracing results, the collaborative strategies and model parameters of the corresponding modal agents are adjusted to obtain a collaborative optimization processing space. The multiple agents further interact with each other for incremental correction of the knowledge extraction results.

10. A multimodal knowledge extraction system based on multi-agent collaborative optimization, characterized in that, The steps for implementing the multimodal knowledge extraction method based on multi-agent cooperative optimization as described in any one of claims 1 to 9, wherein the multimodal knowledge extraction system based on multi-agent cooperative optimization comprises: The multimodal deconstruction module is used to perform multimodal deconstruction of the original document to be extracted, and obtain standardized text data, image and title data, and table and title data; The knowledge extraction module is used to construct a multimodal intelligent agent, which performs feature extraction and preliminary knowledge extraction of the corresponding modality on the documents to be extracted, and outputs a single-modal tuple. The cross-modal parsing module is used to map the text entities, key regions of images, and table fields of the single-modal tuples into unified semantic nodes based on the cross-modal knowledge graph, model the relationship between nodes through graph neural networks, and embed a causal reasoning module to parse the cross-modal logic chain of the initial knowledge extraction. The structured filtering module is used to dynamically allocate collaborative resources to each agent based on the importance of graph nodes, perform structured filtering on cross-modal logic chains based on preset knowledge pattern templates, and output standardized plural knowledge results. The incremental correction module is used to perform extraction confidence analysis and node tracing evaluation based on the standardized tuple knowledge results, optimize the agent collaboration mechanism, and construct a collaborative optimization processing space to incrementally correct the knowledge extraction results.

Images