Multi-modal knowledge extraction method and system based on multi-agent collaborative optimization

Abstract

The application provides a multi-modal knowledge extraction method and system based on multi-agent collaborative optimization, relating to the technical field of knowledge extraction. The method comprises: multi-modal deconstruction of the original text of the literature to be extracted; constructing multi-modal agents to perform feature extraction and preliminary knowledge extraction respectively, and outputting single-modal multi-tuples; based on the cross-modal knowledge graph, mapping information of different modalities to unified semantic nodes, establishing cross-modal association and analyzing logical chains through a graph neural network and a causal reasoning module; dynamically allocating resources according to the importance of the graph nodes, and screening structured knowledge; through confidence analysis and node traceability evaluation, optimizing the agent collaborative mechanism, and incrementally correcting the results. Through the application, the technical problem of low precision and efficiency of knowledge extraction due to the lack of multi-modal knowledge collaborative mining ability of single agent for literature knowledge extraction in the prior art can be solved, and the quality and efficiency of knowledge extraction are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge extraction, and particularly relates to a multi-modal knowledge extraction method and system based on multi-agent collaborative optimization. BACKGROUND

[0002] Literature usually contains text, images, tables, formulas and other modalities, which complement each other and contain rich and complex deep knowledge. Different modalities in the literature differ significantly in expression structure, information density and semantic characteristics. However, the current mainstream literature knowledge mining methods generally have limitations, especially when facing complex literature containing text, images, tables, formulas and other multi-modal information. The existing technology often uses a single model or lacks modality-specific processing strategies, which makes it difficult to fully exploit the independent value and collaborative potential of each modality information, leading to key information being easily missed and semantic understanding being biased. At the same time, the existing methods lack the ability to collaboratively mine multi-modal knowledge, and lack the modeling and systematic joint reasoning mechanism for cross-modal complex semantic relationships, which limits the depth and breadth of knowledge mining and makes it impossible to adjust according to specific content and needs. This leads to low accuracy and efficiency of knowledge mining.

[0003] In summary, the existing technology has the technical problem of low accuracy and efficiency of knowledge extraction due to the use of a single model or agent for knowledge extraction from literature and the lack of ability to collaboratively mine multi-modal knowledge. SUMMARY

[0004] The present application provides a multi-modal knowledge extraction method and system based on multi-agent collaborative optimization, which is used to solve the technical problem of low accuracy and efficiency of knowledge extraction due to the use of a single model or agent for knowledge extraction from literature and the lack of ability to collaboratively mine multi-modal knowledge in the existing technology.

[0005] In view of the above problems, the present application provides a multi-modal knowledge extraction method and system based on multi-agent collaborative optimization.

[0006] In a first aspect, the application provides a multi-modal knowledge extraction method based on multi-agent collaborative optimization, which is implemented by a multi-modal knowledge extraction system based on multi-agent collaborative optimization. The multi-modal knowledge extraction method based on multi-agent collaborative optimization includes: multi-modal deconstruction of the original text of the literature to be extracted to obtain standardized text data, image and title data, and table and title data; constructing multi-modal agents to perform feature extraction and preliminary knowledge extraction of the corresponding modal for the original text of the literature to be extracted, and outputting single-modal multi-tuples; mapping the text entities, image key regions, and table fields of the single-modal multi-tuples into unified semantic nodes based on a cross-modal knowledge graph, modeling the association relationship between the nodes through a graph neural network, and embedding a causal reasoning module to analyze the cross-modal logical chain of the preliminary knowledge extraction; dynamically allocating the collaboration resources of each agent according to the importance of the graph nodes, structuring and filtering the cross-modal logical chain based on a preset knowledge mode template, and outputting a standardized multi-tuple knowledge result; based on the extraction confidence analysis and node traceability evaluation of the standardized multi-tuple knowledge result, optimizing the agent collaboration mechanism, constructing a collaborative optimization processing space to incrementally correct the knowledge extraction result.

[0007] Optionally, the original text of the literature to be extracted is subjected to image-text recognition and layout analysis to extract paragraph hierarchical structure, picture area, table area, and corresponding title and description information; the extracted paragraph hierarchical structure, picture area, table area, and corresponding title and description information are subjected to text, image, and table standardization processing to obtain standardized text data, image and title data, and table and title data, wherein the standardized data is a multi-modal input representation with consistent structure and unified format.

[0008] Optionally, the multi-modal agent includes at least a text mining agent, a picture mining agent, and a table mining agent.

[0009] Optionally, based on the deconstructed input data type, the agent of the corresponding modal is called to perform modal feature extraction on the standardized text data, image, and table to identify entities, relationships, and data; a preliminary structured multi-tuple is constructed according to the characteristics of each modal, and single-modal multi-tuples are output according to the identified modal features, wherein the text modal is a syntax analysis-based entity relationship triple, the image modal is a quantitative description triple generated in combination with the title, and the table modal is a table triple output based on the mapping relationship of the index field.

[0010] Optionally, based on a preset semantic unification standard, the text entity, the image key region, and the table field are mapped as nodes of a knowledge graph, and a node source mode and a type label are labeled; based on a modal feature relationship in the single-modal multi-tuple, an initial semantic edge relationship between the nodes is established, including a co-occurrence relationship of homonymous entities, a cross-modal description reference relationship, and a structure mapping relationship, to form a multi-modal fusion cross-modal knowledge graph.

[0011] Optionally, the nodes of different modes are semantically aligned, the nodes of the cross-modal knowledge graph are updated based on a graph attention mechanism for semantic embedding, and the semantic similarity and the structure correlation weight between the nodes are learned through adjacency propagation; a causal reasoning module is embedded, the causal path is identified according to the directional semantic edge relationship in the graph, and the cross-modal logical chain is output.

[0012] Optionally, the causal reasoning module identifies the edges with directional semantics by analyzing the directional field of the relationship predicate in the triple, and performs semantic labeling on the edges with directional semantics, including causal words and semantic directions; based on the semantic labeling, a plurality of triples with causal correlation entities are spliced into a coherent causal path to construct the cross-modal logical chain.

[0013] Optionally, based on the centrality index, the modal semantic density, and the relevance to the current task target of each node in the graph, an importance coefficient is configured; according to the importance coefficient, the task scheduling order, the processing frequency, and the resource allocation proportion between the intelligent agents of each mode are dynamically adjusted; the triples fused in the cross-modal logical chain are matched and verified according to a preset knowledge mode template, including entity type verification, triple structure integrity checking, and predicate semantic classification operation; the triples that satisfy the matching verification are formatted and output to obtain the standardized multi-tuple knowledge result.

[0014] Optionally, the standardized multi-tuple knowledge result is fused and evaluated according to modal consistency, path support, and entity structure stability to obtain a confidence degree; the multi-tuple knowledge result with a confidence degree lower than a preset threshold is traced back to the node source, and the corresponding modal source and extraction path are identified; based on the node tracing result, the coordination strategy and the model parameters of the corresponding modal intelligent agent are adjusted to obtain a collaborative optimization processing space, and the multi-intelligent agent further performs information interaction for incremental correction of the knowledge extraction result.

[0015] In a second aspect, the present application also provides a multi-modal knowledge extraction system based on multi-agent collaborative optimization, for executing the multi-modal knowledge extraction method based on multi-agent collaborative optimization as described in the first aspect, wherein the multi-modal knowledge extraction system based on multi-agent collaborative optimization comprises: a multi-modal deconstruction module, configured to perform multi-modal deconstruction on the original text of the literature to be extracted, to obtain standardized text data, image and title data, and table and title data; a knowledge extraction module, configured to construct multi-modal agents, to perform feature extraction and preliminary knowledge extraction of corresponding modalities on the original text of the literature to be extracted respectively, and to output single-modality multi-tuples; a cross-modal analysis module, configured to map text entities, image key regions, and table fields of the single-modality multi-tuples into unified semantic nodes based on a cross-modal knowledge graph, to model the association relationship between nodes through a graph neural network, and to embed a causal reasoning module to analyze the cross-modal logical chain of the preliminary knowledge extraction; a structured screening module, configured to dynamically allocate collaboration resources of each agent according to the importance of the graph nodes, to perform structured screening on the cross-modal logical chain based on a preset knowledge mode template, and to output a standardized multi-tuple knowledge result; and an incremental correction module, configured to optimize the agent collaboration mechanism based on extraction confidence analysis and node traceability evaluation of the standardized multi-tuple knowledge result, to construct a collaborative optimization processing space, and to perform incremental correction on the knowledge extraction result.

[0016] The one or more technical solutions provided in the present application have at least the following beneficial effects:

[0017] By performing multi-modal deconstruction on the original text of the literature to be extracted, standardized text data, image and title data, and table and title data are obtained; multi-modal agents are constructed to perform feature extraction and preliminary knowledge extraction of corresponding modalities on the original text of the literature to be extracted respectively, and single-modality multi-tuples are output; text entities, image key regions, and table fields of the single-modality multi-tuples are mapped into unified semantic nodes based on a cross-modal knowledge graph, the association relationship between nodes is modeled through a graph neural network, and a causal reasoning module is embedded to analyze the cross-modal logical chain of the preliminary knowledge extraction; collaboration resources of each agent are dynamically allocated according to the importance of the graph nodes, structured screening is performed on the cross-modal logical chain based on a preset knowledge mode template, and a standardized multi-tuple knowledge result is output; and the agent collaboration mechanism is optimized based on extraction confidence analysis and node traceability evaluation of the standardized multi-tuple knowledge result, a collaborative optimization processing space is constructed, and incremental correction is performed on the knowledge extraction result. That is, by performing multi-modal deconstruction on the original text of the literature to be extracted, multi-modal agents are constructed to realize differential processing, a cross-modal knowledge graph and a graph neural network are used to realize deep collaborative reasoning, and dynamic resource allocation and adaptive optimization mechanism are used to improve the quality and efficiency of knowledge extraction. BRIEF DESCRIPTION OF DRAWINGS

[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the multimodal knowledge extraction method based on multi-agent cooperative optimization proposed in this application.

[0020] Figure 2 This is a schematic diagram of the structure of the multimodal knowledge extraction system based on multi-agent cooperative optimization in this application.

[0021] Figure labeling: Multimodal deconstruction module 11, knowledge extraction module 12, cross-modal parsing module 13, structured filtering module 14, incremental correction module 15. Detailed Implementation

[0022] This application provides a multimodal knowledge extraction method and system based on multi-agent collaborative optimization to address the technical problem of low accuracy and efficiency in knowledge extraction caused by the insufficient ability to collaboratively mine multimodal knowledge in existing technologies, which often rely on single models or agents for document knowledge extraction. By performing multimodal deconstruction of the original document to be extracted, constructing multimodal agents for differentiated processing, utilizing cross-modal knowledge graphs and graph neural networks to achieve deep collaborative reasoning, and improving the quality and efficiency of knowledge extraction through dynamic resource allocation and adaptive optimization mechanisms, this application addresses this issue.

[0023] Example 1, as Figure 1 As shown, this application provides a multimodal knowledge extraction method based on multi-agent cooperative optimization. This method is applied to a multimodal knowledge extraction system based on multi-agent cooperative optimization. The method specifically includes the following steps:

[0024] The original text of the extracted document is deconstructed in a multimodal manner to obtain standardized text data, image and title data, and table and title data.

[0025] Furthermore, this application also includes the following steps: performing image and text recognition and layout analysis on the original text of the document to be extracted, extracting paragraph hierarchical structure, image areas, table areas and corresponding titles and explanatory information; performing text, image and table standardization processing on the extracted paragraph hierarchical structure, image areas, table areas and corresponding titles and explanatory information respectively, to obtain standardized text data, image and title data, and table and title data, wherein the standardized data is a multimodal input representation with consistent structure and uniform format.

[0026] Specifically, the original text to be extracted is a document, such as a scientific and technological document, which needs to be subjected to knowledge extraction, and is usually in the form of a PDF, Word or other format, containing text, pictures, tables, formulas and other multi-modal information. Multi-modal refers to a data form that contains multiple types of information such as text, images, tables, formulas, etc., with characteristics such as diverse information sources and rich expression methods. Multi-modal deconstruction is to separate different information modalities (such as text, images, tables, etc.) in the document, so as to independently process and analyze each modality and extract their respective information.

[0027] The original text to be extracted is subjected to image-text recognition and layout analysis. Image-text recognition is to recognize the pictures and text in the original text to be extracted, and layout analysis is to analyze the visual layout of the document, recognize the location and hierarchical relationship of the text paragraphs, titles, list structures, and non-text elements such as pictures and tables in the document, reflecting the hierarchical nature and organization method of the document content. The original text to be extracted is subjected to image-text recognition processing, and an OCR tool (such as PaddleOCR, Tesseract) is called to convert the image-text mixed layout content into a machine-readable format. OCR not only recognizes words, but also locates the precise coordinate information of each text block on the page. A trained layout recognition model (such as LayoutLMv3 trained on PubLayNet) is used to analyze the page layout and identify which areas are titles, body paragraphs, images, and tables. In particular, by analyzing features such as font size, position, numbering rules, etc., the hierarchical structure of the paragraphs is identified, and a semantic structure tree is generated. For image areas, the bounding box of the image is extracted, and the title and description text above and below it are associated. The extraction of table areas is also based on layout structure features and OCR recognition information. By analyzing horizontal and vertical lines, field name distribution, and blank alignment features, the table boundaries are identified, and the table header fields and numerical content are extracted. In short, the picture and text information of the original text to be extracted are recognized, the layout structure of the document is analyzed, the hierarchical structure of the paragraphs, the location of the image and table areas, and the corresponding title and description information are extracted. For example, if the document contains an image, the system will identify the boundary of the image area, mark the location of the area, and extract the related description text (such as image title, image description).

[0028] The paragraph hierarchy refers to the organization of the text content, such as chapter titles, section titles, body paragraphs, list items, etc., which have parent-child or parallel relationships, forming a hierarchical structure. Each level corresponds to a content unit that is gradually refined in semantics, which helps to understand the context and extract knowledge. In addition, it also includes picture areas and corresponding titles, and corresponding explanatory information; table areas and corresponding titles, and corresponding explanatory information. The title is a brief summary of the content of the picture or table, and the explanatory information is further explanation or supplementary information of the picture or table. For example, for a 15-page PDF format paper of a certain scientific literature, after image recognition and layout analysis, 6 first-level titles, 17 second-level titles, and 31 third-level titles are extracted, with a recognition accuracy of 95.2% (compared with manual annotation); 9 images, with an average size of 572x460px; the title extraction accuracy is 91.8%; and the explanatory text association accuracy is 89.1%. Six tables are extracted, with an average field column number of 8; the field recognition accuracy is 93.4%; and the title and explanation recognition accuracy is 87.6%.

[0029] The extracted paragraph hierarchy, picture area, table area, and corresponding title and explanatory information are subjected to text, image, and table standardization processing, converting data of different sources, formats, and structures into a unified and standardized form, obtaining standardized text data, image and title data, and table and title data. After standardization processing, the data of the three modalities of text, image, and table are converted into data sets with the same data structure (such as a certain record or tensor) and the same format (such as a specific data type, length limit). Through accurate image recognition and layout analysis, the rich content of the literature is obtained, and the organization structure of the content is understood; the standardization processing solves the problem of different sources and formats of multi-modal data, converts the original, unstructured literature content into a clear structure and unified format data representation, ensuring the consistency of the input received by different intelligent agents, and avoiding processing errors or low efficiency caused by data format mismatch.

[0030] A multi-modal intelligent agent is constructed to perform feature extraction and preliminary knowledge extraction of the corresponding modality on the original text of the literature to be extracted, and output single-modal multi-tuples.

[0031] Further, the present application further includes the following steps: the multi-modal intelligent agent at least includes a text mining intelligent agent, a picture mining intelligent agent, and a table mining intelligent agent.

[0032] Further, the application further includes the following steps: based on the deconstructed input data type, calling the agent of the corresponding modal, respectively extracting modal features from the standardized text data, images, and tables, identifying entities, relationships, and data; constructing a preliminary structured multi-tuple according to the characteristics of each modal, and outputting a single-modal multi-tuple according to the identified modal features, wherein the text modal outputs entity relationship triples based on syntax analysis, the image modal generates quantitative description triples in combination with the title, and the table modal outputs table triples based on the mapping relationship between the index field.

[0033] Specifically, the multi-modal agent is an intelligent model based on a large language model, which has autonomous decision-making, information processing and interaction capabilities, and can independently perform specific tasks and work collaboratively with other agents. Each agent focuses on data processing of a certain modal (such as text, image or table) and can work collaboratively with other agents to complete complex knowledge mining tasks. The multi-modal agent includes a text mining agent, a picture mining agent, a table mining agent, etc. Each agent has independent task processing capability and can share information with other agents to improve overall efficiency.

[0034] The text mining agent focuses on structured knowledge extraction of text data, analyzes the semantics, entities, relationships and events in the text through natural language processing (NLP) technology, and converts them into structured information. For example, if the literature involves the efficiency improvement of solar cells, the text agent will identify solar cells as an entity and efficiency improvement as a relationship, and generate structured knowledge of solar cells → efficiency improvement. The picture mining agent is responsible for analyzing images and their related titles, extracting scientific knowledge related to the content of the literature, and performing image recognition, region extraction, object detection, etc. through computer vision technology to obtain information from the image and associate it with the content of the literature. For example, if the literature contains a trend chart of solar cell efficiency changes, the picture agent will identify the key information (such as year, efficiency value) in the image and convert it into structured data. The table mining agent focuses on structured information extraction of tables and their titles, and can identify data, fields, row and column structures in the table and convert them into structured data for analysis. For example, if the literature contains a table showing the efficiency of different solar cell materials, the table agent will extract the fields in the table, such as material name and efficiency, and convert them into structured data.

[0035] By constructing a unified collaboration protocol, information sharing and knowledge complementarity between different agents are achieved, further improving overall performance. After each agent completes its own task, the results are shared with other agents. During knowledge extraction, the accuracy of the extraction results is evaluated through a feedback mechanism. If the results are biased, the collaboration strategy of the agent is adjusted based on the evaluation feedback for iterative optimization. For example, if there is an error in table data extraction, the table agent will correct it based on the feedback to ensure the accuracy of the final results. For example, the text mining agent extracts the relationship between material A and efficiency improvement, and identifies that the efficiency of material A in 2018 is 20%, and in 2020 it is improved to 22%; the picture mining agent identifies the efficiency of material A in different years in the chart and extracts the data that the efficiency of material A in 2018 is 20% and in 2020 is 22%; the table mining agent identifies and extracts the data material A 20%, material B 22% in the table and converts it into structured data. Through the collaboration framework, these information is integrated to obtain the knowledge extraction result that the efficiency of material A is improved from 20% to 22%.

[0036] Using LLaVA (for picture and text joint processing), TAPAS (table understanding), ChatGLM (text processing), etc. as the base of general large models, fine-tuning is performed for basic knowledge mining tasks in text, picture, and table modes to ensure that each agent has basic extraction capabilities in its own mode. A batch of multi-modal task data is selected each time for joint training of multi-modal agents, and through multi-agent collaborative learning mechanism, information fusion and complementarity between modes are strengthened to continuously improve the knowledge extraction capability. Input real scientific literature samples, and schedule different modal data in each round of joint training, such as inputting text paragraphs + experimental images + corresponding tables at the same time to construct tasks: extracting the corresponding relationship between material preparation method and performance indicators, and forming the ability of collaborative extraction through mutual feedback learning of multiple agents.

[0037] In the reasoning phase, multi-modal agents will automatically schedule corresponding agents for information extraction in specific modes according to the content of the sample to be extracted, and through information feedback and fusion between multiple agents, form the final multi-element knowledge output. Dynamic collaboration between multiple agents ensures the comprehensiveness and accuracy of the extraction results. For example, if the task is to extract performance trends, the table and image agents are activated first, while the text agent only assists in confirming the field meaning. Based on user or artificial feedback, the knowledge extraction results are evaluated and corrected, and the feedback information is used to further fine-tune the parameters of each agent to optimize the collaboration mechanism and continuously improve the adaptability and robustness of multi-modal agents to complex multi-modal tasks.

[0038] By customizing specialized agents for each modality and fine-tuning and joint training based on general large models, it is ensured that each agent has the basic ability to handle tasks in the field, and the overall performance is improved through inter-modal collaborative learning, fully utilizing the complementarity of information in different modalities, significantly improving the coverage and accuracy of knowledge extraction.

[0039] According to the deconstructed data type, the agent of the corresponding modality is called. If the deconstructed data type is text and picture, the text mining agent and the picture mining agent are called. If the deconstructed data type is picture and table, the picture mining agent and the table mining agent are called. According to the called agent, the standardized text data, image and table are respectively subjected to modality feature extraction, entity, relationship and data recognition. Entity is a concept or object with clear boundaries and semantics in literature, relationship refers to the semantic connection between two entities, and data refers to specific numerical values or facts related to entities.

[0040] According to the characteristics of each modality, a preliminary structured tuple is constructed. The original and unstructured modality data is converted into a data unit with a certain format and meaning, which is easier for computer processing, according to certain rules. According to the recognized modality features, single-modality tuples are output, including text modality, image modality and table modality. The text mining agent constructs entity-relation-data entity relation triples through syntactic analysis, for example, extracting the triple of solar cell-efficiency improvement-22% from the text. Similarly, the image mining agent identifies key areas in the image (such as efficiency trend chart, chart, etc.) through computer vision technology, and outputs quantitative description triples combined with image title information. For image modality, since the image itself lacks direct readable semantics, it needs to rely on its title or explanatory text to understand the image content. The agent analyzes the image content (such as identifying the cell structure under the microscope) and its title (such as the nuclear morphology change of cells after 48 hours of drug X treatment), and then generates a triple containing quantitative information or more specific description.

[0041] For table modality, tables usually have clear row and column structures, containing indicators (row names, column names) and specific numerical values (cell data). The indicator field mapping relationship refers to the logical connection between different fields (columns) in the table, and the triple based on the table field mapping relationship is constructed. The fields, rows and columns in the table are parsed, and the data in each row or column in the table is recognized. According to the title and field information of the table, the corresponding triple is constructed. For example, the value 4.8 at the intersection of the column of experimental group A average value and the row of telomere length is recognized, and the table triple (experimental group A, average telomere length, 4.8) is generated.

[0042] The information extracted from text, images, and tables is converted into a unified and structured single-modal knowledge representation unit (triplet). Each agent generates a triplet that best represents the characteristics of the information in its respective modality (text syntax, image visual and title combination, table structure and field mapping), ensuring the relevance and effectiveness of information extraction.

[0043] Based on the cross-modal knowledge graph, the text entities, image key regions, and table fields of the single-modal multi-tuple are mapped to unified semantic nodes. The association between nodes is modeled using a graph neural network, and a causal reasoning module is embedded to analyze the cross-modal logical chain of preliminary knowledge extraction.

[0044] Further, the present application further comprises the following steps:

[0045] Based on the preset semantic unification standard, the text entities, image key regions, and table fields are mapped to the nodes of the knowledge graph, and the source modality and type label of the nodes are labeled. Based on the modal feature relationship in the single-modal multi-tuple, the initial semantic edge relationship between nodes is established, including the co-occurrence relationship of homologous entities, cross-modal description reference relationship, and structure mapping relationship, forming a multi-modal fusion cross-modal knowledge graph.

[0046] Specifically, the preset semantic unification standard is a pre-defined unified semantic framework or standard to ensure that the knowledge extracted from different modalities (text, image, table) has a consistent representation. The preset semantic unification standard defines how to map different sources of information (such as text entities, image key regions, and table fields) to unified knowledge graph nodes and label their modal types and labels, ensuring that entities from text, image, and table, even if their expressions are different, can be mapped to nodes with consistent or similar meanings in the knowledge graph.

[0047] According to the preset semantic unification standard, the text entities, image key regions, and table fields are mapped to the nodes of the knowledge graph, and the source modality (such as text, image, table, etc.) and type label (such as entity, relationship, data, etc.) of each node are labeled, which helps to distinguish different sources and different types of information. The entities in the text data (such as solar cells) are extracted and mapped to the nodes in the knowledge graph. The key regions in the image data (such as data points and trend lines in the chart) are also mapped to the nodes, and the fields in the table (such as material name and efficiency) are mapped to the nodes.

[0048] Based on the modal feature relationships in unimodal tuples, edge relationships are established between nodes. Initial semantic edge relationships are the relationship lines connecting two nodes in the knowledge graph. "Initial" means that these edge relationships are mainly based on unimodal information or simple cross-modal associations and have not yet undergone complex reasoning verification. Semantic representation edges have explicit meanings, such as regulation or measurement values, and are established based on the relational information of unimodal triples.

[0049] In knowledge graphs, edges represent relationships between nodes. Initial semantic edge relationships refer to the preliminary relationships between nodes constructed based on features and relationships extracted from unimodal tuples. These include co-occurrence of text entities, descriptive reference relationships between images and text, or structural mapping relationships between table fields and other modal information. Co-occurrence relationships of homologous entities are relationships between entities that appear simultaneously in the same unimodal data unit (such as the same sentence, the same image, or the same table row). Cross-modal descriptive reference relationships are relationships established when describing the same or related entities in different modal data. Structural mapping relationships are the cases where the structured relationships between table fields in a table modality are mapped to knowledge graph edges.

[0050] This approach aggregates co-occurrence relationships, cross-modal descriptive and referential relationships, and structural mapping relationships of entities from the same source, integrating them through a graph structure to construct a complete cross-modal knowledge graph. Each node represents a specific piece of information (such as an entity, field, or region), and edges represent relationships between them (such as co-occurrence, referential, or mapping relationships). By converging information from different sources into unified graph nodes, it achieves structural fusion of heterogeneous information.

[0051] Furthermore, this application also includes the following steps: semantic alignment of nodes of different modalities, semantic embedding and updating of cross-modal knowledge graph nodes based on graph attention mechanism, learning semantic similarity and structural association weights between nodes through adjacency propagation; embedding a causal reasoning module, identifying causal paths based on the edge relationships of directional semantics in the graph, and outputting the cross-modal logical chain.

[0052] Furthermore, this application also includes the following steps: the causal reasoning module identifies edges with directional semantics by parsing the directional field of the relational predicate in the triple and performs semantic labeling, including causal words and semantic directions; based on the semantic labeling, multiple triples with causal related entities are spliced ​​into a coherent causal path to construct the cross-modal logic chain.

[0053] Specifically, to refine the cross / deep relevance analysis between multi-modal agents, increase the dynamic collaborative adaptability of agents, integrate graph neural network modeling and causal reasoning deeply into the original multi-agent collaboration framework, add a cross-modal knowledge graph construction module, and realize unified semantic mapping, modal alignment and embedding optimization, dynamic resource allocation, etc. through graph neural network (GNN). Map entities in text, key areas in images (such as experimental charts), and structured fields in tables to graph nodes, assign them unified semantic codes, and define nodes. Define edge types based on cross-modal associations (such as text description-chart display-table verification), and dynamically learn the association weights between nodes using graph attention networks. Design a cross-modal contrast learning algorithm to maximize the embedding similarity of nodes with the same semantics (such as synthetic methods in text and process flowcharts in images), solving the semantic gap problem of heterogeneous data. Based on the importance of nodes in the graph (such as centrality indicators) and task objectives (such as focusing on method mining or data verification), dynamically adjust the collaboration priority of each agent (text, image, table). Based on the cross-modal graph, introduce a causal reasoning module to solve the precise extraction of method-effect logical chains in scientific literature. For example, in materials science literature, if the graph shows a strong association between microstructure images and performance parameter tables, the image and table agents will be activated for collaborative reasoning.

[0054] Align the semantics of nodes from different modalities (such as text, images, tables, etc.), match and unify them, so that they have consistency in semantics and can work together in the same knowledge graph. Semantic alignment is the basis for cross-modal data fusion, ensuring that information from different sources can be correctly matched and interacted. Update the semantic embedding of nodes in the cross-modal knowledge graph through the graph attention mechanism. The graph attention mechanism propagates the relationship between nodes by weighting, dynamically adjusts the node embedding, and makes semantically similar nodes better integrated. For example, if the text node solar cell and the image node solar cell efficiency graph are semantically related, the graph attention mechanism will strengthen their connection.

[0055] The graph attention mechanism is a deep learning method based on graph structure data, which uses attention mechanisms to assign different weights to different nodes in the graph. Through the graph attention mechanism, the relationships between nodes can be dynamically weighted, highlighting important nodes or edges and improving the efficiency and accuracy of graph structure learning. The semantic embedding of a node represents the characteristics and semantic information of the node. With the help of the graph attention mechanism, the embedding of the node is updated through adjacency propagation, so that they can better capture and represent the relationship and structure information with other nodes. Adjacency propagation is a method in graph learning that refers to the propagation of information from a node to its adjacent nodes, updating the embedding representation of the node. In cross-modal knowledge graphs, adjacency propagation helps nodes share information and learn the similarity and structural association between nodes.

[0056] The GATv2 model is used to propagate the graph structure, and each node aggregates semantics through its adjacent nodes and assigns different weights. For example: the neighbor nodes of NiCo2O4 are specific capacitance and morphology characteristics, and the attention weights of these two edges are calculated as 0.72 and 0.28 respectively; after updating, the NiCo2O4 node vector will integrate more information about specific capacitance, thereby enhancing the semantic ability related to performance indicators.

[0057] The causal reasoning module is used to identify and reason about the causal relationships between nodes in the knowledge graph, particularly by analyzing the directionality field of the relationship predicate in the triple, identifying edges with directional semantics, and connecting entities with causal relationships to form causal paths. All single-modal and cross-modal triples (such as temperature rise, cause, and conductivity decrease) are traversed to identify causal words (such as cause, promote, and inhibit) from the predicate part. If the predicate has directionality (such as cause is from cause to effect), a directional marker is added to the graph for subsequent reasoning. Marking directional edges as directed causal edges, such as catalyst A, promote, and reaction rate, adds a marker as a causal edge.

[0058] By the causal reasoning module, the causal relationship in the cross-modal knowledge graph is identified, the edges with directional semantics are labeled, and the causal path is formed. The causal reasoning module analyzes the relationship predicate of the triple, and identifies the causal words (such as promote, cause, influence, and regulate) with directionality. When detecting a triple such as material A—causes—efficiency improvement, the causal reasoning module identifies the causal relationship between material A and efficiency improvement, and labels it as an edge with directionality. The identified causal relationship is labeled as a causal word and a semantic direction, and is sorted in the form of 〈cause entity, relationship, result entity〉. The causal word refers to a word representing the causal relationship, such as promote, cause, influence, and regulate; the semantic direction is the directionality in the causal relationship, i.e., who is the cause and who is the result. The edges with causal labels are sorted in the form of 〈cause entity, relationship, result entity〉, and multiple causal paths are spliced, i.e., multiple triples with causal related entities are spliced into coherent causal paths, cause entity→intermediate entity→result entity, to construct cross-modal logical chains.

[0059] By semantic alignment, the ambiguity and repetition of different modal information in representation are avoided, making the nodes in the knowledge graph more accurate and consistent. Semantic embedding updating based on graph attention mechanism enables each node's representation to dynamically absorb the most relevant neighbor information, thereby accurately capturing the node's semantics in the cross-modal context.

[0060] According to the importance of the graph nodes, the collaboration resources of each agent are dynamically allocated, the cross-modal logical chains are structured and filtered based on the preset knowledge mode template, and the standardized multi-tuple knowledge result is output.

[0061] Further, the present application further includes the following steps:

[0062] Based on the centrality index, modal semantic density, and relevance to the current task target of each node in the graph, an importance coefficient is configured; according to the importance coefficient, the task scheduling order, processing frequency, and resource allocation ratio between the agents of each modality are dynamically adjusted; the matching verification of the preset knowledge mode template is performed on the triples fused in the cross-modal logical chain, including entity type verification, triple structure integrity check, and predicate semantic classification operation; the triples that meet the matching verification are formatted and output to obtain the standardized multi-tuple knowledge result.

[0063] Specifically, in graph theory, centrality metrics are used to measure the importance or influence of a node in a network, including degree centrality (how many nodes it is directly connected to), betweenness centrality (how many shortest paths pass through the node), and proximity centrality (the average distance to all other nodes). Higher node centrality generally means it has more connections or is more critical in the knowledge graph, potentially representing a more core concept or entity. Modal semantic density is the richness or concentration of information represented by a node or its neighboring nodes from a specific modality (text, image, table). For example, if a node has many neighboring nodes from the text modality, then that region has a high semantic density for the text modality. High density may mean the region is more information-rich, but it may also mean redundancy. Relevance to the current task objective is the degree to which nodes in the knowledge graph or the relationships between nodes match the specific task the user currently hopes to complete (such as answering a scientific question, generating a summary in a specific field, etc.). Nodes and relationships with high relevance are more valuable for completing the current task.

[0064] Based on the centrality index, modal semantic density, and relevance to the current task objective of each node in the knowledge graph, the importance of each node is comprehensively evaluated, resulting in an importance coefficient for each node. This quantifies the relative importance of each node in the entire knowledge extraction task. The importance coefficient I of each node is calculated using a weighted average. n :I n =α·C n +β·D n +γ·R n , where C n It is centrality, D n It is semantic density, R n α represents task relevance; β and γ are adjustment parameters (e.g., 0.4, 0.3, 0.3). For example, assuming the centrality index of text is 0.43, the modal semantic density is 0.32, and the relevance to the current task objective is 0.91, then the text importance coefficient is 0.584; if the centrality index of image is 0.21, the modal semantic density is 0.85, and the relevance to the current task objective is 0.72, then the text importance coefficient is 0.562; if the centrality index of table is 0.35, the modal semantic density is 0.63, and the relevance to the current task objective is 0.88, then the text importance coefficient is 0.608. The task scheduling order among the modal agents is table, text, image, with priority given to table mining agents.

[0065] According to the importance coefficient of each modality, the task scheduling order, processing frequency and resource allocation ratio between agents are dynamically adjusted to ensure that the key modalities are processed first. The processing frequency is the number of calls of each modality in a time window linearly expanded according to the importance coefficient, for example, a high-weight modality is called 4 times every 10 seconds, a medium-weight 2 times, and a low-weight 1 time. Resource allocation is carried out according to the normalized proportion of the coefficient, such as memory, threads, and model copy number. During the task execution process, multi-modal agents dynamically adjust the task execution order and resource allocation according to the size of their importance coefficients. For example, if the text agent plays an important role in the current task, it will get more computing resources and execute the task first. If a node is determined to be very important, the agents related to the node (if the node is mainly associated with text information, the text agent is given priority to work) are given priority to process, or the processing frequency of these agents is increased (they process data faster), or more computing resources are allocated to them to ensure the processing efficiency and accuracy of key information. Conversely, for nodes or modalities with low importance, resources are reduced and efforts are allocated according to the actual value of knowledge to improve overall efficiency. For example, the allocation task order of the table is first, the processing frequency is 6 times per minute, and the resource proportion is 40% (48 units); the allocation task order of the text is second, the processing frequency is 5 times per minute, and the resource proportion is 35% (42 units); the task order of the image is third, the processing frequency is 3 times per minute, and the resource proportion is 25% (30 units).

[0066] After triple extraction, verification is performed according to the preset knowledge mode template, including entity type verification (checking whether the entity type in the triple conforms to the predetermined type), triple structure integrity check (checking whether the structure of the triple conforms to the entity, format, and data), and predicate semantic classification operation (performing semantic classification and annotation according to the semantic type of the relationship predicate). For triples that pass the matching verification, they are converted into a unified and standardized format output to form a standardized multi-triple knowledge result. The verified triples are output in a standardized format (such as RDF, JSON-LD, etc.) and standardized, ensuring that they conform to a unified data structure.

[0067] By dynamically adjusting the task scheduling, processing frequency and resource allocation of agents, computing resources are concentrated on the most important knowledge nodes, avoiding wasting resources on low-value information, thereby improving the efficiency and processing speed of overall knowledge extraction.

[0068] Based on the extraction confidence analysis and node traceability evaluation of the standardized multi-triple knowledge result, the agent collaboration mechanism is optimized, and a collaborative optimization processing space is constructed to incrementally correct the knowledge extraction result.

[0069] Further, the application further includes the following steps: fusion evaluation of the standardized multi-tuple knowledge result according to modal consistency, path support degree, and entity structure stability to obtain a confidence degree; node tracing of the multi-tuple knowledge result with a confidence degree lower than a preset threshold to identify corresponding modal sources and extraction paths; adjustment of the coordination strategy and model parameters of the corresponding modal agent based on the node tracing result to obtain a coordination optimization processing space, and further information interaction of the multi-agent for incremental correction of the knowledge extraction result.

[0070] Specifically, the standardized multi-tuple knowledge result is fusion evaluated according to modal consistency, path support degree, and entity structure stability to evaluate the reliability of the standardized multi-tuple knowledge result and ensure its quality. The confidence degree is an evaluation value of the reliability of a certain multi-tuple knowledge result in the knowledge graph, and is usually used to represent the accuracy and reliability of the result. The multi-tuple with a confidence degree lower than a certain threshold will need to be further checked and corrected. The modal consistency is the semantic consistency and logical matching between different modal data (such as text, image, table, etc.). The modal consistency ensures that data from different sources can be correctly fused in the knowledge graph, avoiding semantic conflicts. The path support degree is the existence frequency and support degree of a certain knowledge path (such as a causal chain, an entity relationship chain, etc.) in the knowledge graph. The path with a high path support degree usually represents that the path has strong semantic support in the data and is reliable. The entity structure stability is the structural stability of an entity in the knowledge graph. The entity with a high entity structure stability consistently appears in multiple modalities or multiple triples, indicating that the representation and relationship of the entity in the knowledge graph have strong consistency and reliability.

[0071] The modal consistency, the path support degree, and the entity structure stability are usually numbers in the interval of 0-1, and the confidence degree is calculated by weighted summation according to the three dimensions, and the weight parameters are usually 0.3, 0.4, and 0.3. For example, if the modal consistency is calculated as 0.9, the path support degree is calculated as 0.85, and the entity structure stability is calculated as 0.95, then the comprehensive confidence degree is 0.89.

[0072] The preset threshold is a pre-set numerical limit for determining whether the confidence of a multi-tuple is high enough to be considered as reliable knowledge. Multi-tuples below this threshold are considered suspicious or require further verification. If the confidence of a certain multi-tuple is below the pre-set threshold, it will not be discarded directly, but a deep diagnostic process will be initiated to trace the source of a certain multi-tuple in the knowledge graph, identify which modality (text, image, table, etc.) it was extracted from, and its extraction path (i.e. through which steps or agents for processing and reasoning). Node tracing is to track the original source of each node (entity or relationship) in a multi-tuple knowledge result (from text, image, or table) and the specific path or step it was extracted and combined into this multi-tuple, similar to labeling knowledge with source and process records. Through tracing, the link causing the low confidence of the multi-tuple is accurately located, whether it is a deviation in text understanding, an error in image recognition, or inaccurate table parsing, or a problem in cross-modal fusion.

[0073] Tracing the multi-tuple knowledge result with confidence below the preset threshold, identifying which modality it comes from (such as text, image, table, etc.), and determining its extraction path (i.e. through which agents for processing and reasoning). According to the node tracing results, dynamically adjust the collaboration mode between the corresponding modality agents. For example, if it is found that the collaboration between the image agent and the text agent is not smooth, adjust the information transmission mode between them and increase their collaboration frequency; if it is found that the processing capacity of a certain agent is insufficient, adjust its model parameters to improve its processing efficiency and accuracy. Based on the tracing results, adjust the collaboration strategy of the corresponding modality agents, such as changing the task scheduling order between agents to let a certain agent handle first which may help improve overall accuracy; or adjusting the information sharing frequency and content between agents to allow them to better check each other; at the same time, fine-tune the model parameters inside the corresponding agent, such as adjusting the entity recognition model weight of the text agent to make it pay more attention to specific context, or adjusting the target detection model threshold of the image agent to make it more robust.

[0074] After adjustment, a synergistic optimization processing space is obtained, and how to improve its synergistic working mechanism and agent capability is defined. In the synergistic optimization processing space, the knowledge extraction result is incrementally corrected through further interaction of multi-agent information, and low-confidence multi-tuples are gradually improved, so that the final knowledge extraction result is more accurate. Incremental correction refers to applying improved strategies and parameters when processing new data, or reprocessing or correcting the results that have been extracted but are found to be problematic through backtracking, thereby continuously improving the accuracy and robustness of the entire knowledge extraction system. Through multi-dimensional (modal consistency, path support, entity stability) fusion evaluation and confidence threshold filtering, a large number of low-quality and unreliable knowledge results are effectively screened out, ensuring the accuracy and reliability of the final output knowledge.

[0075] In summary, the multi-modal knowledge extraction method based on multi-agent synergistic optimization provided in the present application has the following beneficial effects:

[0076] By multi-modal deconstruction of the original text of the literature to be extracted, standardized text data, image and title data, table and title data are obtained; multi-modal agents are constructed to perform feature extraction and preliminary knowledge extraction of the corresponding modal for the original text of the literature to be extracted, and single-modal multi-tuples are output; based on the cross-modal knowledge graph, the text entities, image key regions, and table fields of the single-modal multi-tuples are mapped to unified semantic nodes, the association relationship between the nodes is modeled through a graph neural network, and a causal reasoning module is embedded to analyze the cross-modal logical chain of the preliminary knowledge extraction; the collaboration resources of each agent are dynamically allocated according to the importance of the graph nodes, the cross-modal logical chain is structured and filtered based on a preset knowledge mode template, and standardized multi-tuple knowledge results are output; based on the extraction confidence analysis and node backtracking evaluation of the standardized multi-tuple knowledge results, the synergistic mechanism of the agents is optimized, and a synergistic optimization processing space is constructed to incrementally correct the knowledge extraction results. That is, by multi-modal deconstruction of the original text of the literature to be extracted, multi-modal agents are constructed for differential processing, cross-modal knowledge graphs and graph neural networks are used for deep synergistic reasoning, and dynamic resource allocation and adaptive optimization mechanisms are used to improve the quality and efficiency of knowledge extraction.

[0077] In the second embodiment, based on the same inventive concept as the multi-modal knowledge extraction method based on multi-agent synergistic optimization in the first embodiment, the present application also provides a multi-modal knowledge extraction system based on multi-agent synergistic optimization. Please refer to the accompanying drawings Figure 2 The multi-modal knowledge extraction system based on multi-agent synergistic optimization comprises:

[0078] The multi-modal deconstruction module 11 is configured to deconstruct the original text of the literature to be extracted in a multi-modal manner, to obtain standardized text data, image and title data, table and title data; the knowledge extraction module 12 is configured to construct multi-modal intelligent agents, to respectively perform feature extraction and preliminary knowledge extraction of the corresponding modal on the original text of the literature to be extracted, and to output single-modal multi-tuples; the cross-modal analysis module 13 is configured to map the text entities, image key regions and table fields of the single-modal multi-tuples into unified semantic nodes based on a cross-modal knowledge graph, to model the association relationship between the nodes through a graph neural network, and to embed a causal reasoning module to analyze the cross-modal logical chain of the preliminary knowledge extraction; the structured screening module 14 is configured to dynamically allocate the collaborative resources of each intelligent agent according to the importance of the graph nodes, to perform structured screening on the cross-modal logical chain based on a preset knowledge mode template, and to output a standardized multi-tuple knowledge result; the incremental correction module 15 is configured to analyze and evaluate the extraction confidence and node traceability of the standardized multi-tuple knowledge result, to optimize the intelligent agent collaboration mechanism, to construct a collaborative optimization processing space, and to perform incremental correction on the knowledge extraction result.

[0079] Further, the multi-modal deconstruction module 11 in the multi-modal knowledge extraction system based on multi-agent collaborative optimization is further configured to:

[0080] The original text of the literature to be extracted is subjected to image-text recognition and layout analysis, to extract paragraph hierarchical structure, picture area, table area and corresponding title and description information; the extracted paragraph hierarchical structure, picture area, table area and corresponding title and description information are respectively subjected to text, image and table standardization processing, to obtain standardized text data, image and title data, table and title data, wherein the standardized data is a multi-modal input representation with consistent structure and unified format.

[0081] Further, the knowledge extraction module 12 in the multi-modal knowledge extraction system based on multi-agent collaborative optimization is further configured to:

[0082] The multi-modal intelligent agent at least includes a text mining intelligent agent, a picture mining intelligent agent and a table mining intelligent agent.

[0083] Further, the knowledge extraction module 12 in the multi-modal knowledge extraction system based on multi-agent collaborative optimization is further configured to:

[0084] Based on the deconstructed input data type, the agent of the corresponding modality is called to extract modality features from the standardized text data, images, and tables respectively, identify entities, relationships, and data; according to the characteristics of each modality, a preliminary structured multi-tuple is constructed, and according to the identified modality features, a single-modality multi-tuple is output, wherein the text modality is a syntax-based entity relationship triple, the image modality is a quantitative description triple combined with the title, and the table modality is a table triple based on the mapping relationship between the index field.

[0085] Further, the cross-modality analysis module 13 in the multi-agent collaborative optimization-based multi-modal knowledge extraction system is further used for:

[0086] Based on the preset semantic unification standard, the text entity, the image key area, and the table field are mapped to the node of the knowledge graph, and the node source modality and type label are labeled; based on the modality feature relationship in the single-modality multi-tuple, the initial semantic edge relationship between the nodes is established, including the co-occurrence relationship of homologous entities, the cross-modality description reference relationship, and the structure mapping relationship, forming a multi-modal fusion cross-modality knowledge graph.

[0087] Further, the cross-modality analysis module 13 in the multi-agent collaborative optimization-based multi-modal knowledge extraction system is further used for:

[0088] The nodes of different modalities are semantically aligned, the nodes of the cross-modality knowledge graph are semantically embedded and updated based on the graph attention mechanism, and the semantic similarity and structure correlation weight between the nodes are learned through adjacency propagation; the embedded causal reasoning module identifies the causal path according to the directional semantic edge relationship in the graph, and outputs the cross-modality logic chain.

[0089] Further, the cross-modality analysis module 13 in the multi-agent collaborative optimization-based multi-modal knowledge extraction system is further used for:

[0090] The causal reasoning module identifies the edges with directional semantics by analyzing the directional field of the relationship predicate in the triple, including causal words and semantic directions; based on the semantic marking, the multiple triples with causal correlation entities are spliced into a coherent causal path to construct the cross-modality logic chain.

[0091] Further, the structured screening module 14 in the multi-agent collaborative optimization-based multi-modal knowledge extraction system is further used for:

[0092] An importance coefficient is configured based on a centrality index of each node in the graph, a modal semantic density, and a relevance to a current task target; a task scheduling order, a processing frequency, and a resource allocation proportion among the modal agents are dynamically adjusted according to the importance coefficient; matching verification of a preset knowledge mode template is performed on the triplets fused in the cross-modal logical chain, including entity type verification, triplet structure integrity checking, and predicate semantic classification operation; the triplets meeting the matching verification are formatted and output to obtain the standardized multi-triplet knowledge result.

[0093] Further, the incremental correction module 15 in the multi-agent collaborative optimization-based multi-modal knowledge extraction system is further used for:

[0094] The standardized multi-triplet knowledge result is fused and evaluated according to modal consistency, path support, and entity structure stability to obtain a confidence degree; the multi-triplet knowledge result with a confidence degree lower than a preset threshold is subjected to node tracing to identify corresponding modal sources and extraction paths; the collaborative strategy and model parameters of the corresponding modal agents are adjusted based on the node tracing result to obtain a collaborative optimization processing space, and the multi-agents further perform information interaction for incremental correction of the knowledge extraction result.

[0095] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The foregoing Figure 1 The multi-agent collaborative optimization-based multi-modal knowledge extraction method and specific examples in Embodiment One are also applicable to the multi-agent collaborative optimization-based multi-modal knowledge extraction system of the present embodiment. Based on the foregoing detailed description of the multi-agent collaborative optimization-based multi-modal knowledge extraction method, those skilled in the art can clearly understand the multi-agent collaborative optimization-based multi-modal knowledge extraction system of the present embodiment. Therefore, for the sake of brevity of the specification, no further detailed description is given here.

[0096] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0097] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application also intends to include these modifications and variations.

Claims

1. A multimodal knowledge extraction method based on multi-agent cooperative optimization, characterized in that, include: The original text of the document to be extracted is deconstructed in a multimodal manner to obtain standardized text data, image and title data, and table and title data; Construct a multimodal intelligent agent to perform feature extraction and preliminary knowledge extraction of the corresponding modality in the original text of the document to be extracted, and output a single-modal tuple; Based on cross-modal knowledge graphs, the text entities, key regions of images, and table fields of the single-modal tuples are mapped to unified semantic nodes. The relationships between nodes are modeled through graph neural networks, and a causal reasoning module is embedded to parse the cross-modal logic chain of the initial knowledge extraction. Based on the importance of graph nodes, collaborative resources for each agent are dynamically allocated. Based on a preset knowledge pattern template, cross-modal logic chains are structurally filtered, and standardized plural knowledge results are output. Based on the extraction confidence analysis and node tracing evaluation of the standardized tuple knowledge results, the intelligent agent collaboration mechanism is optimized, and a collaborative optimization processing space is constructed to incrementally correct the knowledge extraction results. The graph neural network models the relationships between nodes and embeds a causal reasoning module to parse the cross-modal logic chain of preliminary knowledge extraction, including: Semantic alignment is performed on nodes of different modalities, and semantic embedding updates are performed on cross-modal knowledge graph nodes based on graph attention mechanism. Semantic similarity and structural association weights between nodes are learned through adjacency propagation. An embedded causal reasoning module identifies causal paths based on the edge relationships of directional semantics in the graph and outputs the cross-modal logic chain.

2. The multimodal knowledge extraction method based on multi-agent cooperative optimization according to claim 1, characterized in that, The multimodal intelligent agent includes at least: a text mining intelligent agent, an image mining intelligent agent, and a table mining intelligent agent.

3. The multimodal knowledge extraction method based on multi-agent cooperative optimization according to claim 1, characterized in that, Multimodal deconstruction of the original text of the document to be extracted includes: The original text of the document to be extracted is subjected to image and text recognition and layout analysis to extract paragraph hierarchical structure, image areas, table areas and corresponding titles and explanatory information; The extracted paragraph hierarchy, image area, table area, and corresponding title and description information are subjected to text, image, and table standardization processing, respectively, to obtain standardized text data, image and title data, and table and title data. The standardized data is a multimodal input representation with consistent structure and uniform format.

4. The multimodal knowledge extraction method based on multi-agent cooperative optimization according to claim 2, characterized in that, Each text document to be extracted undergoes feature extraction and preliminary knowledge extraction for its corresponding modality, outputting a single-modality tuple, including: Based on the deconstructed input data type, the corresponding modal agent is invoked to extract modal features from the standardized text data, images, and tables, and to identify entities, relationships, and data. Based on the characteristics of each modality, a preliminary structured tuple is constructed. According to the features of the identified modality, a single-modality tuple is output. Specifically, for the text modality, entity relation triples are output based on syntactic analysis; for the image modality, quantitative description triples are generated by combining the title; and for the table modality, table triples are output based on the mapping relationship of index fields.

5. The multimodal knowledge extraction method based on multi-agent cooperative optimization according to claim 4, characterized in that, Based on a cross-modal knowledge graph, the text entities, key image regions, and table fields of the unimodal tuples are mapped to unified semantic nodes, including: Based on a pre-defined semantic standard, text entities, key regions of images, and table fields are mapped to nodes in a knowledge graph, and the source modality and type label of the nodes are annotated. Based on the modal feature relationships in the single-modal tuple, initial semantic edge relationships between nodes are established, including co-occurrence relationships of co-originating entities, cross-modal descriptive referential relationships, and structural mapping relationships, forming a multimodal fusion cross-modal knowledge graph.

6. The multimodal knowledge extraction method based on multi-agent cooperative optimization according to claim 1, characterized in that, An embedded causal reasoning module identifies causal paths based on the edge relationships of directional semantics in the graph and outputs the cross-modal logic chain, including: The causal reasoning module identifies edges with directional semantics by parsing the directional field of the relational predicate in the triple and performs semantic labeling, including causal words and semantic direction. Based on semantic tagging, multiple triples with causal relationships are concatenated into a coherent causal path to construct the cross-modal logic chain.

7. The multimodal knowledge extraction method based on multi-agent cooperative optimization according to claim 1, characterized in that, Based on the importance of graph nodes, collaborative resources for each agent are dynamically allocated. Cross-modal logic chains are structurally filtered based on preset knowledge pattern templates, and standardized tuple knowledge results are output, including: Importance coefficients are configured based on the centrality index, modal semantic density, and relevance to the current task objective of each node in the graph; Based on the importance coefficient, the task scheduling order, processing frequency, and resource allocation ratio among the various modal agents are dynamically adjusted. For triples fused in cross-modal logic chains, perform matching verification of preset knowledge pattern templates, including entity type verification, triple structure integrity check, and predicate semantic classification operation; The triples that satisfy the matching verification are formatted and output to obtain the standardized tuple knowledge results.

8. The multimodal knowledge extraction method based on multi-agent cooperative optimization according to claim 1, characterized in that, Based on the extraction confidence analysis and node tracing evaluation of the standardized tuple knowledge results, the agent collaboration mechanism is optimized, and a collaborative optimization processing space is constructed to incrementally correct the knowledge extraction results, including: The standardized plural knowledge results are evaluated by fusion based on modality consistency, path support, and entity structure stability to obtain confidence level; For tuple knowledge results with confidence levels below a preset threshold, node tracing is performed to identify the corresponding modality source and extraction path. Based on the node tracing results, the collaborative strategies and model parameters of the corresponding modal agents are adjusted to obtain a collaborative optimization processing space. The multiple agents further interact with each other for incremental correction of the knowledge extraction results.

9. A multimodal knowledge extraction system based on multi-agent collaborative optimization, characterized in that, The steps for implementing the multimodal knowledge extraction method based on multi-agent cooperative optimization as described in any one of claims 1 to 8, wherein the multimodal knowledge extraction system based on multi-agent cooperative optimization comprises: The multimodal deconstruction module is used to perform multimodal deconstruction of the original document to be extracted, and obtain standardized text data, image and title data, and table and title data; The knowledge extraction module is used to construct a multimodal intelligent agent, which performs feature extraction and preliminary knowledge extraction of the corresponding modality on the documents to be extracted, and outputs a single-modal tuple. The cross-modal parsing module is used to map the text entities, key regions of images, and table fields of the single-modal tuples into unified semantic nodes based on the cross-modal knowledge graph, model the relationship between nodes through graph neural networks, and embed a causal reasoning module to parse the cross-modal logic chain of the initial knowledge extraction. The structured filtering module is used to dynamically allocate collaborative resources to each agent based on the importance of graph nodes, perform structured filtering on cross-modal logic chains based on preset knowledge pattern templates, and output standardized plural knowledge results. The incremental correction module is used to perform extraction confidence analysis and node tracing evaluation based on the standardized tuple knowledge results, optimize the agent collaboration mechanism, and construct a collaborative optimization processing space to incrementally correct the knowledge extraction results.

Images