Document intelligent analysis and question-answering method and system based on RAG and multi-modal knowledge graph

Abstract

The invention discloses an intelligent document analysis and question-answering method and system based on an RAG and a multi-modal knowledge graph, and aims to solve the problems that the existing GraphRAG mainly depends on text retrieval and generation, and lacks utilization of images or other modal information, so that the comprehensiveness and accuracy of answers are limited in a complex application scene. Analyzing the multi-modal document to obtain text modal information and image modal information; converting the text modal information into a text knowledge graph, and converting the image modal information into an image knowledge graph; fusing the text mapping knowledge domain and the image mapping knowledge domain to obtain a multi-modal mapping knowledge domain; and sequentially inputting the multi-modal knowledge graph into the embedded model and the vector database to obtain an entity vector database. And sequentially inputting a to-be-queried question of a user into the embedded model and the vector database to obtain information corresponding to the to-be-queried question, inputting the information corresponding to the to-be-queried question into the large language model, and outputting an answer to the to-be-queried question. The invention belongs to the field of data processing.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a document intelligent parsing and question-answering method and system based on retrieval-enhanced generation technology (RAG) and multimodal knowledge graphs. Background Art

[0002] In intelligent question-answering systems, with the development of multimodal technology, retrieval-enhanced generation techniques have begun to explore how to process inputs from different modalities and how to improve the quality of generated results through multimodal information fusion. Existing multimodal RAG systems, such as M3DOCRAG and VisRAG, typically embed data from different modalities into a shared space for retrieval and generation. These methods perform well in implicit fact queries (RAG level 2), but due to the lack of modeling of structured relationships in the data, their performance is limited in more complex reasoning tasks, such as explainable reasoning path queries and implicit reasoning path queries (RAG levels 3 and 4).

[0003] Embedding-based retrieval can only measure the similarity between modalities and has difficulty capturing complex semantic relationships and cross-modal logical relationships. In addition, generative models that rely on fuzzy matching have difficulty providing clear reasoning paths and high-quality interpretable outputs. Therefore, multimodal RAG systems that rely solely on embedding spaces cannot meet the needs of complex tasks. Microsoft's GraphRAG introduces structured knowledge graph information, which significantly improves the logical coherence and interpretability of text generation tasks, demonstrating the potential of knowledge graphs in RAG research. However, existing traditional RAG-based systems mainly rely on text retrieval and generation, and lack the use of image or other modal information, resulting in limited comprehensiveness and accuracy of answers. For example, for questions involving visual content, such as what is the function of the object in the picture? Text retrieval alone cannot provide accurate answers. In addition, the answers generated by traditional RAG systems often lack clear reasoning paths and are difficult to meet users' needs for explanatory answers.

[0004] Therefore, although the existing RAG system has shown certain effects in some basic tasks, its performance and functions are significantly limited in complex application scenarios due to the lack of support for multimodal information fusion and structured knowledge. Summary of the Invention

[0005] In order to solve the problem that the existing GraphRAG mainly relies on text retrieval and generation and lacks the utilization of images or other modal information, which leads to limited comprehensiveness and accuracy of answers in complex application scenarios, the present invention proposes a document intelligent parsing and question-answering method and system based on RAG and multimodal knowledge graph.

[0006] The technical solution adopted by the present invention is:

[0007] It includes the following steps:

[0008] S1. Obtain a multimodal document and parse it using MinerU and a large multimodal model to obtain text modal information and image modal information.

[0009] S2. Use entity extraction technology to process text modal information and generate a text knowledge graph. At the same time, use image segmentation and scene graph construction tasks to process image modal information and generate an image knowledge graph.

[0010] S3. Fuse the text knowledge graph and the image knowledge graph to obtain a multimodal knowledge graph;

[0011] S4. Input the multimodal knowledge graph into the embedding model, output the corresponding vector, input the corresponding vector into the vector database, and construct and obtain the entity vector database;

[0012] S5. Input the user's query question into the embedding model, output the corresponding vector, input the corresponding vector into the entity vector database, obtain the information corresponding to the query question, input the information corresponding to the query question into the large language model, and output the answer to the query question.

[0013] Furthermore, the multimodal document in S1 is a PDF document, a docx document, an image document jpg, an image document png or a txt document.

[0014] Furthermore, in S1, a multimodal document is obtained, and MinerU and a multimodal large model are used to parse the multimodal document to obtain text modal information and image modal information. The specific process is as follows:

[0015] Obtain multimodal documents and use the open source tool MinerU to perform structured analysis on them:

[0016] For the text content in the multimodal document, all the text content is divided into multiple text blocks, each text block is numbered and labeled with an ID, and the information of each text block is obtained, which is the text modality information;

[0017] For each image in the multimodal document, extract the image number, storage path, image title, footnote and context text of the current image in the multimodal document;

[0018] The multimodal large model is then used to generate a detailed description for each image. The detailed description includes each element in the image, the attributes of each element, and the emotion or atmosphere conveyed by the image content. The information of each image is obtained, namely the image modal information. The elements include people, objects, scenes and backgrounds, and the attributes of the elements include color, shape, size and posture.

[0019] Furthermore, in S2, the text modal information is processed using entity extraction technology to generate a text knowledge graph. At the same time, the image modal information is processed using image segmentation and scene graph construction tasks in sequence to generate an image knowledge graph. The specific process is as follows:

[0020] S21. Process each text block using entity extraction technology to generate a text knowledge graph for each text block;

[0021] S22. Perform semantic segmentation on each image using a Yolo v8 network to obtain image feature blocks for each image, where the number of image feature blocks for each image is N, where N is an integer and N is greater than or equal to 0.

[0022] S23. Use a large multimodal model to build a scene graph task:

[0023] S231. Generate a detailed description for each image feature block using a multimodal large model, wherein the detailed description includes entity types and features within the image feature block, and obtain each image feature block after the detailed description;

[0024] The entity types include objects, organisms and people, and the entity characteristics include entity name, shape, color, size, age, posture, structural characteristics, size and material;

[0025] S232. Using the multimodal large model, extract all entities and relationships between entities in each image, construct entity relationship information of the corresponding image based on the relationships between the entities, and obtain entity relationship information for each image;

[0026] S233, using the multimodal large model to align each image feature block described in detail in each image with the corresponding entity;

[0027] S24. According to S22-S23, a global entity description is generated for each image to obtain an image knowledge graph for each image.

[0028] Furthermore, in S3, the text knowledge graph and the image knowledge graph are fused to obtain a multimodal knowledge graph. The specific process is as follows:

[0029] S31. For each text knowledge graph and the image knowledge graph corresponding to the text knowledge graph, first, based on the information of the entities in the image knowledge graph, delete the entities that have established relationships with the text blocks. Next, based on the information of the remaining entities in the image knowledge graph and the entities and relationships in the text knowledge graph, use the spectral clustering method to classify all entities in the text knowledge graph to obtain candidate entities. Finally, use the large language model to align the candidate entities with the remaining entities in the image knowledge graph to obtain image knowledge graphs that only contain aligned entities.

[0030] S32. In each image knowledge graph containing only aligned entities, entities requiring enhanced information are screened and supplemented with descriptive information for the entities requiring enhanced information based on the text knowledge graph corresponding to the image knowledge graph containing only aligned entities and the context of the image knowledge graph containing only aligned entities in the multimodal document, thereby obtaining each enhanced image knowledge graph containing only aligned entities.

[0031] S33. Using each enhanced image knowledge graph containing only aligned entities and the corresponding text knowledge graph obtained in S32, the image knowledge graph corresponding to the enhanced image knowledge graph containing only aligned entities obtained in S24 is updated. In the image knowledge graph obtained in S24, if the updated entity already exists, the relationship and information of the corresponding entity are added or modified. If the updated entity does not exist, the entity and relationship are added to obtain each updated image knowledge graph;

[0032] S34. Fuse each updated image knowledge graph with the corresponding text knowledge graph obtained in S21 to obtain each multimodal knowledge graph.

[0033] Furthermore, in S31, based on the information of the remaining entities in the image knowledge graph and the entities and relationships in the text knowledge graph, a spectral clustering method is used to classify all entities in the text knowledge graph to obtain candidate entities. The specific process is as follows:

[0034] 1) Taking entities as nodes, calculate the cosine similarity between the remaining nodes in the image knowledge graph and the nodes in the text knowledge graph:

[0035]

[0036] Among them, v i ,v j are the vector representations of nodes i and j respectively, ||·|| is the L2 norm of the vector;

[0037] 2) Use the information of the remaining entities in the image knowledge graph and the relationship edge weights to construct the adjacency matrix A. Let the relationship between nodes i and j be r ij By weight weight(r ij ) is given. If there is a clear relationship definition between nodes i and j, the weight value is known. If there is no clear relationship definition between nodes i and j, the weight is 1.

[0038] 3) The product of cosine similarity and weight is used as the element of adjacency matrix A, which is:

[0039] A ij =sim(v i ,vj )·weight(r ij )

[0040] 4) The degree matrix D is a diagonal matrix, where each diagonal element D ii is the degree of node i, that is, the connection strength of node i with other nodes, and each diagonal element D is defined according to the adjacency matrix A. ii for:

[0041]

[0042] Therefore, the degree of each node in the degree matrix D is the sum of the similarities of the node with all other nodes;

[0043] 5) Calculate the Laplace matrix L = DA of the image knowledge graph based on the adjacency matrix A and the degree matrix D, perform eigendecomposition on the Laplace matrix, extract the eigenvectors corresponding to the first k smallest eigenvalues, and use the extracted eigenvectors as the matrix Q = [q1,q1,…,q k ] columns, where Indicates the number of entities in the nearby entity list;

[0044] 6) Use the DBSCAN clustering algorithm to cluster the matrix Q and obtain the clustering results C1, C2, ..., C k , map the clustering results to obtain entity clustering results A1, A2, ..., A k , where A i ={v j |r j ∈C i};

[0045] 7) Based on the entity clustering results, entities in the text knowledge graph that are not closely related or irrelevant to the remaining entities in the image knowledge graph are screened out, and only entities that are highly matched with the remaining entities in the image knowledge graph are retained as candidate entities.

[0046] A document intelligent parsing and question-answering system based on RAG and multimodal knowledge graph, which includes a multimodal knowledge graph construction module, a retrieval module, and a generation module;

[0047] The multimodal knowledge graph construction module is used to extract the modal information of multimodal documents and fuse the modal information to obtain a multimodal knowledge graph;

[0048] The multimodal document is a PDF document, a docx document, an image document jpg, an image document png or a txt document;

[0049] The multimodal knowledge graph construction module includes a preprocessing submodule, a single-modal processing submodule, and a cross-modal fusion submodule in sequence;

[0050] The preprocessing submodule is used to preprocess the multimodal document, output each modal information, and input each modal information into the single modal processing submodule;

[0051] The single-modal processing submodule is used to process the received modal information, generate a knowledge graph for each modal information, and input the knowledge graph of all modal information into the cross-modal fusion submodule;

[0052] The cross-modal fusion submodule is used to fuse the knowledge graphs of all received modal information to obtain a multimodal knowledge graph;

[0053] The retrieval module is used to extract entities, entity relationships, and context information corresponding to the query question from the multimodal knowledge graph obtained by the multimodal knowledge graph construction module based on the user's query question, and input the extracted entities, entity relationships, and context information corresponding to the query question into the generation module;

[0054] The generation module is used to integrate the received entities, entity relationships and context information corresponding to the query question and output the answer corresponding to the query question.

[0055] Furthermore, the specific process of the pre-processing submodule is as follows:

[0056] The preprocessing submodule includes the open source tool MinerU and a large multimodal model;

[0057] The multimodal document is input into the preprocessing submodule, and the open source tool MinerU is used to perform structural analysis on the multimodal document:

[0058] For the text content in the multimodal document, all the text content is divided into multiple text blocks, each text block is numbered and labeled with an ID, and the information of each text block is obtained, which is the text modality information;

[0059] For each image in the multimodal document, the image number, storage path, image title, footnote and context text of the current image in the multimodal document are extracted, and then a detailed description is generated for each image using the multimodal large model. The detailed description includes each element in the image and the attributes of each element, the emotion or atmosphere conveyed by the image content, and the information of each image is obtained, which is the image modal information.

[0060] Furthermore, the specific process of the single-mode processing submodule is as follows:

[0061] The single-modal processing submodules include the Text2Graph module and the Img2Graph module;

[0062] The text modal information output by the preprocessing submodule is input into the Text2Graph module, and each text block is processed using entity extraction technology to generate a text knowledge graph for each text block;

[0063] The image modality information output by the preprocessing submodule is input into the Img2Graph module, and the image modality information is processed by image segmentation and scene graph construction in sequence to generate an image knowledge graph for each image. The specific process is as follows:

[0064] The Img2Graph module includes the Yolo v8 network and the multimodal large model in turn;

[0065] 1) Using the Yolo v8 network to perform semantic segmentation on each image, obtaining image feature blocks for each image, where the number of image feature blocks for each image is N, where N ≥ 0, and the number is an integer;

[0066] 2) using the multimodal large model to generate a detailed description for each image feature block, wherein the detailed description includes the entity types and features within the image feature block, thereby obtaining each image feature block after the detailed description;

[0067] The entity types include objects, organisms and people, and the entity characteristics include entity name, shape, color, size, age, posture, structural characteristics, size and material;

[0068] 3) Using a large multimodal model to extract all entities and the relationships between entities in each image, construct entity relationship information for the corresponding image based on the relationships between entities, and obtain entity relationship information for each image;

[0069] 4) Using a large multimodal model, each image feature block described in detail in each image is aligned with the corresponding entity;

[0070] 5) Based on 1)-4), generate a global entity description for each new image and obtain the image knowledge graph of each image.

[0071] Furthermore, the specific process of the cross-modal fusion submodule is as follows:

[0072] 1) For each text knowledge graph and the image knowledge graph corresponding to the text knowledge graph, first, delete the entities that have established relationships with the text block based on the information of the entities in the image knowledge graph;

[0073] Next, based on the information of the remaining entities in the image knowledge graph and the entities and relationships in the text knowledge graph, the spectral clustering method is used to classify all entities in the text knowledge graph to obtain candidate entities;

[0074] Finally, the large language model is used to align the candidate entities with the remaining entities in the image knowledge graph, obtaining an image knowledge graph that only contains aligned entities.

[0075] 2) In each image knowledge graph containing only aligned entities, entities requiring enhanced information are screened and supplemented with descriptive information for the entities requiring enhanced information based on the text knowledge graph corresponding to the image knowledge graph containing only aligned entities and the context of the image knowledge graph containing only aligned entities in the multimodal document, thereby obtaining each enhanced image knowledge graph containing only aligned entities;

[0076] 3) using each enhanced image knowledge graph containing only aligned entities and the corresponding text knowledge graph to update the image knowledge graph corresponding to the enhanced image knowledge graph containing only aligned entities obtained by the unimodal processing submodule, and in the image knowledge graph obtained by the unimodal processing submodule, if the updated entity already exists, then adding or modifying the relationship and information of the corresponding entity; if the updated entity does not exist, then adding a new entity and relationship to obtain each updated image knowledge graph;

[0077] 4) Each updated image knowledge graph is fused with the corresponding text knowledge graph obtained by the unimodal processing submodule to obtain each multimodal knowledge graph.

[0078] The beneficial effects of the present invention are:

[0079] The method of the present invention proposes a multimodal GraphRAG framework (MMGraphRAG), which improves the performance of document understanding and reasoning tasks by integrating text information and image information, that is, by decomposing image data into structured scene graphs, significantly enhancing the intelligent reasoning ability and interpretability of documents. In addition, a spectral clustering method is used for multimodal data fusion to obtain a refined multimodal knowledge graph, which overcomes the limitations of current cross-modal information fusion, improves fusion accuracy while reducing computational overhead, and constructs an entity vector database based on the multimodal knowledge graph. The entity vector database includes explainable reasoning path queries and implicit reasoning path queries. When a user queries a question, a fast and accurate query answer can be generated only through the entity vector database. The experimental results of the present invention demonstrate the advantages of the present invention in multi-domain and multi-page document question-answering tasks.

[0080] In order to realize the function of multimodal GraphRAG, the system of the present invention adopts a modular design method, splitting the three main modules of multimodal knowledge graph construction module, retrieval and generation into independent and interchangeable sub-modules, thereby enhancing the flexibility and scalability of the system. The multimodal knowledge graph construction module is used to extract the modal information of each modality of multimodal documents, fuse the modal information, and obtain a multimodal knowledge graph; the retrieval module is used to extract the entities, entity relationships and context information corresponding to the query question from the multimodal knowledge graph according to the user's query question; the generation module is used to integrate the entities, entity relationships and context information corresponding to the query question, and output the answer corresponding to the query question. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 It is the overall framework of the system of the present invention;

[0082] Figure 2 This is a processing diagram of the Img2Graph module;

[0083] Figure 3 This is a pseudo code diagram of the cross-modal fusion submodule;

[0084] Figure 4 The experimental results of the present invention on the document question answering dataset DocBench;

[0085] Figure 5 The experimental results of the present invention on the document question answering dataset MMLongBench; DETAILED DESCRIPTION

[0086] Specific implementation method 1: Combination Figure 1-Figure 5 This embodiment describes a method for intelligent document parsing and question answering based on RAG and multimodal knowledge graph, which includes the following steps:

[0087] S1. Obtain a multimodal document and perform structural analysis on it using the open source tool MinerU and a multimodal large model to obtain text modal information and image modal information. The specific process is as follows:

[0088] The multimodal document is a PDF document, a docx document, an image document jpg, an image document png or a txt document, etc. When the multimodal document is a PDF document or a docx document, the content of the multimodal document includes images, text and tables.

[0089] Obtain the multimodal documents of the Docbench dataset, which are PDF documents. Use the open source tool MinerU to perform structural analysis on the PDF documents:

[0090] For the text content in the PDF document, all text content is divided into multiple text blocks, each text block is numbered and marked with an ID, and the information of each text block is obtained, which is the text modal information.

[0091] For each image in the PDF document, the image number, storage path, image title, footnote and context text of the current image in the PDF document are extracted to ensure that each image can be associated with the corresponding text block later.

[0092] The multimodal large model is then used to generate a detailed description for each image. The detailed description includes each element in the image and the attributes of each element, the emotion or atmosphere conveyed by the image content, etc. The elements include people, objects, scenes, backgrounds, etc. The attributes of the elements include color, shape, size, posture, etc. This step can describe the emotion or atmosphere (such as warmth, happiness, tension, tranquility) conveyed by images with emotional or atmospheric cues (such as scenes, portraits, or dynamic events). The detailed description helps to improve the semantic expression ability of the image, enrich the image information content, and provide more comprehensive data for subsequent multimodal fusion and reasoning. The information obtained for each image is the image modal information.

[0093] Finally, the information of each text block and each image is integrated according to the position and contextual relationship of the text and image in the PDF document, and then stored in a JSON file in Markdown (MD) format.

[0094] To adapt to the multimodal GraphRAG (MMGraphRAG) framework of the present invention, first, all multimodal documents of the Docbench dataset or the Mmlongbench dataset are obtained as a training set, and the above-mentioned structured parsing steps are performed for each multimodal document to generate a large amount of standardized data format (such as JSON, JPG, MD, etc.) information of different modalities (images, texts) that meet the requirements, so as to ensure the format consistency of the data and provide a high-quality, structured input data foundation for subsequent multimodal knowledge graph construction, retrieval and result generation.

[0095] S2. Use entity extraction technology to process text modal information and generate a text knowledge graph. At the same time, use image segmentation and scene graph construction tasks to process image modal information and generate an image knowledge graph. The specific process is as follows:

[0096] S21. Process each text block using entity extraction technology to generate a text knowledge graph for each text block;

[0097] S22. Perform semantic segmentation on each image using a Yolo v8 network. The Yolo v8 network can parse the image into multiple regions with independent semantics to obtain image feature blocks for each image. The number of image feature blocks for each image is N, where N is an integer and N is greater than or equal to 0.

[0098] S23. Use a large multimodal model to build a scene graph task:

[0099] S231. Generate a detailed description for each image feature block using the multimodal large model to enrich the image feature information. The detailed description includes the entity types and characteristics within the image feature block. The entity types include objects, organisms, and people. The entity characteristics include entity name, shape, color, dimensions, age, posture, structural characteristics, size, material, possible functions, and other significant features. Each image feature block is obtained after the detailed description.

[0100] S232. Utilize the multimodal large model to extract all entities and the relationships between entities in each image, construct entity relationship information of the corresponding image based on the relationships between entities, and obtain entity relationship information of each image.

[0101] S233. Utilize the recognition and reasoning capabilities of the multimodal large model to align each image feature block described in detail in each image with the corresponding entity to ensure semantic consistency.

[0102] S24. According to S22-S23, a global entity description is generated for each image to obtain an image knowledge graph for each image.

[0103] Global entity description summarizes the entire image into a main entity, which includes the entity name and various descriptive information of the entity, establishes the association between the main entity and all entities in the image, and supplements the overall semantic information.

[0104] To achieve fine-grained representation of image modality information within the multimodal GraphRAG framework, this paper proposes a strategy for generating fine-grained scene graphs based on a large multimodal model. This strategy leverages high-precision semantic segmentation and text description generation techniques to effectively extract explicit and implicit relationships within images and achieve deep alignment between image and text modalities, thereby constructing a high-quality, fine-grained scene graph. Traditional methods typically only identify explicit spatial relationships (e.g., "girl—handheld camera—camera"). The proposed method not only identifies explicit spatial relationships but also infers more complex implicit relationships (e.g., "boy—boy and girl have a good relationship, possibly friends or lovers—girl"). Furthermore, the proposed method provides richer descriptive information for entities within images. Compared to the traditional description of "boy," the proposed method can add more nuanced semantics, such as "university student with a languid look." Most importantly, the proposed method does not require extensive annotated data. By refining and aligning image feature blocks, it accurately represents relationships within images and performs entity reasoning based on text descriptions, thereby generating a more transparent and interpretable scene graph suitable for a wider range of application scenarios.

[0105] S3. Fuse the text knowledge graph and the image knowledge graph to obtain a multimodal knowledge graph. The specific process is as follows:

[0106] S31. For each text knowledge graph and the image knowledge graph corresponding to the text knowledge graph, first, based on the information of the entities in the image knowledge graph, delete the entities that have established relationships with the text blocks, especially the image feature block entities. Next, based on the information of the remaining entities in the image knowledge graph and the entities and relationships in the text knowledge graph, use the spectral clustering method to classify all entities in the text knowledge graph to obtain candidate entities. Finally, use the large language model LLM to align the candidate entities with the remaining entities in the image knowledge graph, and find the corresponding text entity for each remaining entity in the image knowledge graph, thereby completing the matching and alignment of the image knowledge graph and the text knowledge graph. Each image knowledge graph containing only aligned entities is obtained.

[0107] The spectral clustering algorithm used in the present invention has the following characteristics and specific implementation methods:

[0108] 1. Using entities as nodes, calculate the cosine similarity between the remaining nodes (entities) in the image knowledge graph and the nodes (entities) in the text knowledge graph:

[0109]

[0110] Among them, v i ,v j are the vector representations of nodes i and j respectively, and ||·|| is the L2 norm of the vector.

[0111] 2. Use the information of the remaining entities in the image knowledge graph and the weight of the relationship edge to construct the adjacency matrix A to reflect the similarity between nodes and the importance of their relationships. Let the relationship between nodes i and j be r ij By weight weight(r ij ). If there is a clear relationship definition between nodes i and j, the weight value is known. If there is no clear relationship definition between nodes i and j, the weight is assumed to be 1.

[0112] 3. The product of cosine similarity and weight is used as the element of adjacency matrix A, which is:

[0113] A ij =sim(v i ,v j )·weight(r ij )

[0114] If no clear relationship exists, weight(r ij )=1, that is, A ij =sim(v i ,v j ).

[0115] 4. The degree matrix D is a diagonal matrix where each diagonal element D ii is the degree of node i, that is, the connection strength between node i and other nodes. According to the adjacency matrix A, each diagonal element D can be defined ii for:

[0116]

[0117] Therefore, the degree of each node in the degree matrix D is the sum of the similarities between the node and all other nodes.

[0118] 5. Calculate the Laplace matrix L = DA of the image knowledge graph based on the adjacency matrix A and the degree matrix D, perform eigendecomposition on the Laplace matrix, extract the eigenvectors corresponding to the first k smallest eigenvalues, and use the extracted eigenvectors as the matrix Q = [q1,q1,…,q k ] columns, where Indicates the number of entities in the nearby entity list.

[0119] 6. Use the DBSCAN clustering algorithm to cluster the matrix Q and obtain the clustering results C1, C2, ..., C k , map the clustering results to obtain the real entity clustering results A1, A2, ..., A k , where A i ={v j |r j ∈C i}.

[0120] 7. Based on the entity clustering results, entities in the text knowledge graph that are not closely related or relevant to the remaining entities in the image knowledge graph are filtered out, and only entities that highly match the remaining entities in the image knowledge graph are retained as candidate entities. Next, categories are assigned to the image entities, and the present invention uses a multimodal large model (LLM) to semantically align the image entities with the candidate entities.

[0121] S32. Although the alignment method based on spectral clustering has achieved good results, in some cases information may be lost or entities cannot be directly aligned. At this time, the present invention further enriches the description of the image entity by combining the relevant text entity information in the multimodal document. For example, the image entity "flooded community" itself cannot provide information about the affected location and cause. However, two related text entities are mentioned in the multimodal document: "Hurricane Ian" and "Florida". Combining these two entities with the information in the multimodal document, the description of the entity can be enhanced to a more detailed description: "The flooded community is an area in Florida that was affected by Hurricane Ian, resulting in severe flooding and damage, and many houses were uninhabitable." This enhancement process is crucial to the integrity of the information. In order to achieve the enhanced image knowledge graph, follow the steps below:

[0122] First, in each image knowledge graph containing only aligned entities, entities that need enhanced information are screened out. According to the text knowledge graph corresponding to the image knowledge graph containing only aligned entities and the context of the image knowledge graph containing only aligned entities in the PDF document, descriptive information is supplemented for the entities that need enhanced information to update the nodes and edges in the image knowledge graph containing only aligned entities, thereby obtaining each enhanced image knowledge graph containing only aligned entities.

[0123] S33. Using each enhanced image knowledge graph containing only aligned entities and the text knowledge graph corresponding to the enhanced image knowledge graph containing only aligned entities, update the entities (nodes) and edges in the image knowledge graph corresponding to the enhanced image knowledge graph containing only aligned entities obtained in S24. If the updated entity already exists in the image knowledge graph obtained in S24, then add or modify the relationship and description information of the corresponding entity in the image knowledge graph obtained in S24; if the updated entity does not exist in the image knowledge graph obtained in S24, then add new entities (nodes) and relationships to the image knowledge graph obtained in S24 to obtain each updated image knowledge graph. This step establishes a connection between the entire image and text knowledge graph.

[0124] S34. Fuse each updated image knowledge graph with the text knowledge graph corresponding to the updated image knowledge graph obtained in S21, merge the nodes and edges in the two knowledge graphs according to the alignment relationship, ensure the uniqueness of entities and the correct connection of relationships, delete duplicate entities and connect related entities, and obtain each multimodal knowledge graph that contains completeness and consistency.

[0125] This achieves a deep fusion of image and text information and constructs a complete multimodal knowledge graph to support subsequent reasoning and question-answering tasks.

[0126] S4. Input all multimodal knowledge graphs (MMKG) into the embedding model, output the corresponding vectors, input the corresponding vectors into the vector database, and construct and obtain the entity vector database (vector data base, vdb) for use in the retrieval stage.

[0127] S5. Input the user's query question into the embedding model, output the corresponding vector, input the corresponding vector into the entity vector database, search in the multimodal knowledge graph, perform intelligent analysis on the entities, entity relationships and context information related to the query question, obtain the information corresponding to the query question, input the information corresponding to the query question into the large language model, and output the answer to the query question.

[0128] Specific implementation method 2: Combination Figure 1-Figure 5 This embodiment describes a document intelligent parsing and question-answering system based on RAG and multimodal knowledge graph, which includes a multimodal knowledge graph construction module (Indexing), a retrieval module (Retrieval) and a generation module (Generation).

[0129] The multimodal knowledge graph construction module is used to extract the modal information of multimodal documents and fuse the modal information to obtain a multimodal knowledge graph.

[0130] The multimodal document is a PDF document, a docx document, an image document jpg, an image document png or a txt document, etc. When the multimodal document is a PDF document or a docx document, the content of the multimodal document includes images, text and tables.

[0131] The multimodal knowledge graph construction module includes a preprocessing submodule, a unimodal processing submodule, and a cross-modal fusion submodule in sequence.

[0132] The preprocessing submodule is used to preprocess multimodal documents, output each modal information, and input each modal information into the single modal processing submodule. The specific process is as follows:

[0133] The preprocessing submodule includes the open source tool MinerU and a large multimodal model.

[0134] The multimodal document is input into the preprocessing submodule, and the open source tool MinerU is used to perform structural analysis on the multimodal document:

[0135] For the text content in the multimodal document, all the text content is divided into multiple text blocks, each text block is numbered and labeled with an ID, and the information of each text block is obtained, which is the text modality information.

[0136] For each image in the multimodal document, the image number, storage path, image title, footnote and context text of the current image in the multimodal document are extracted to ensure that each image can be associated with the corresponding text block in the future. The multimodal large model is then used to generate a detailed description for each image, thereby enriching the image information content. The detailed description includes each element in the image and the attributes of each element, the emotion or atmosphere conveyed by the image content, etc. The elements include people, objects, scenes, backgrounds, etc. The attributes of the elements include color, shape, size, posture, etc. For images with emotional or atmospheric cues (such as scenes, portraits or dynamic events), this step can describe the emotions or atmosphere they convey (such as warmth, happiness, tension, tranquility). The detailed description helps to improve the semantic expression ability of the image and provide more comprehensive data for subsequent multimodal fusion and reasoning. The information obtained for each image is the image modality information.

[0137] Finally, the information of each text block and each image is integrated according to the position and contextual relationship of the text and image in the PDF document, and then stored in a JSON file in Markdown (MD) format.

[0138] To adapt to the multimodal GraphRAG (MMGraphRAG) framework of the present invention, the preprocessing submodule is responsible for performing structured parsing on the input multimodal documents and generating standardized data formats (such as JSON, JPG, MD, etc.) of different modalities (images, text) that meet the system requirements, providing a high-quality, structured input data foundation for the subsequent multimodal knowledge graph construction, retrieval module, and generation module.

[0139] The single-modal processing submodule is used to process the received modal information, generate a knowledge graph for each modal information, and input the knowledge graph of all modal information into the cross-modal fusion submodule. The specific process is as follows:

[0140] The single-modal processing sub-module includes the Text2Graph module and the Img2Graph module.

[0141] The text modal information output by the preprocessing submodule is input into the Text2Graph module, and each text block is processed using entity extraction technology to generate a text knowledge graph for each text block.

[0142] The image modality information output by the preprocessing submodule is input into the Img2Graph module, which processes the image modality information using image segmentation and scene graph construction to generate an image knowledge graph for each image. The specific process is as follows:

[0143] The Img2Graph module consists of a Yolo v8 network and a multimodal large model in turn.

[0144] 1) Perform semantic segmentation on each image using the Yolo v8 network. The Yolo v8 network parses each image into multiple regions with independent semantics, and obtains image feature blocks for each image. The number of image feature blocks for each image is N, where N ≥ 0, and is an integer.

[0145] 2) Generate a detailed description for each image feature block using the multimodal large model to enrich the image feature information. The detailed description includes the entity types and features within the image feature block, and obtains each image feature block after the detailed description.

[0146] The entity types include objects, organisms or people, and the entity characteristics include name, shape, color, size, age, posture, structural characteristics, size, material, possible functions and other significant characteristics.

[0147] 3) Use the multimodal large model to extract all entities and the relationships between entities in each image, construct the entity relationship information of the corresponding image based on the relationships between entities, and obtain the entity relationship information of each image.

[0148] 4) Utilizing the recognition and reasoning capabilities of the multimodal large model, each image feature block described in detail in each image is aligned with the corresponding entity to ensure semantic consistency.

[0149] 5) Based on 1)-4), a global entity description is generated for each new image, resulting in an image knowledge graph for each image. This global entity description summarizes the entire image into a main entity, including the entity name and various entity descriptions. This entity is then associated with all entities in the image, supplementing the overall semantic information.

[0150] To achieve fine-grained representation of image information within the multimodal GraphRAG framework, this paper proposes a fine-grained scene graph generation strategy within the Img2Graph module based on a large multimodal model. This strategy leverages high-precision semantic segmentation and text generation techniques to effectively extract explicit and implicit relationships within images and achieve deep alignment between image and text modalities, thereby constructing a high-quality, fine-grained scene graph. While traditional methods typically only identify explicit spatial relationships (e.g., "girl—handheld camera—camera"), the proposed method can also infer more complex implicit relationships (e.g., "boy—boy and girl have a good relationship, possibly friends or lovers—girl"). Furthermore, the proposed method provides richer descriptive information for entities within images. Compared to the traditional description of "boy," the proposed method can add more nuanced semantics, such as "university student with a languid look." Most importantly, the proposed method does not require extensive annotated data. By refining and aligning image feature blocks, it accurately represents relationships within images and performs entity reasoning based on textual descriptions, thereby generating a more transparent and interpretable scene graph suitable for a wider range of application scenarios.

[0151] The processing of the Img2Graph module ensures that the fine-grained semantic information in the image is fully captured, providing a solid foundation for building high-precision scene graphs, facilitating the subsequent deep fusion and reasoning of images and text, and improving the performance of multimodal GraphRAG in complex scene understanding.

[0152] The cross-modal fusion submodule is used to fuse the knowledge graphs of all received modal information to obtain a multimodal knowledge graph. The specific process is as follows:

[0153] 1. For each text knowledge graph and the image knowledge graph corresponding to the text knowledge graph, first, delete the entities that have established relationships with the text blocks based on the information of the entities in the image knowledge graph, especially the image feature block entities.

[0154] Next, based on the information of the remaining entities after deletion in the image knowledge graph and the entities and relationships in the text knowledge graph, the spectral clustering method is used to classify all entities in the text knowledge graph to obtain candidate entities. The specific process is as follows:

[0155] 1) Taking entities as nodes, calculate the cosine similarity between the remaining nodes in the image knowledge graph and the nodes in the text knowledge graph:

[0156]

[0157] Among them, v i ,v j are the vector representations of nodes i and j respectively, ||·|| is the L2 norm of the vector;

[0158] 2) Use the information of the remaining entities in the image knowledge graph and the relationship edge weights to construct the adjacency matrix A. Let the relationship between nodes i and j be r ij By weight weight(r ij ) is given, if there is no clear relationship definition between nodes i and j, the weight is assumed to be 1;

[0159] 3) Calculate the elements of the adjacency matrix A based on cosine similarity and weights, which are:

[0160] A ij =sim(v i ,v j )·weight(r ij )

[0161] When weight(r ij )=1, that is, A ij =sim(v i ,v j );

[0162] 4) The degree matrix D is a diagonal matrix, where each diagonal element D ii is the degree of node i, that is, the connection strength of node i with other nodes, and each diagonal element D is defined according to the adjacency matrix A. ii for:

[0163]

[0164] Therefore, the degree of each node in the degree matrix D is the sum of the similarities of the node with all other nodes;

[0165] 5) Calculate the Laplace matrix L = DA of the image knowledge graph based on the adjacency matrix A and the degree matrix D, perform eigendecomposition on the Laplace matrix, extract the eigenvectors corresponding to the first k smallest eigenvalues, and use the extracted eigenvectors as the matrix Q = [q1,q1,…,q k ] columns, where Indicates the number of entities in the nearby entity list;

[0166] 6) Use the DBSCAN clustering algorithm to cluster the matrix Q and obtain the clustering results C1, C2, ..., C k , map the clustering results to obtain entity clustering results A1, A2, ..., A k , where A i ={v j |r j ∈C i};

[0167] 7) Based on the entity clustering results, entities in the text knowledge graph that are not closely related or irrelevant to the remaining entities in the image knowledge graph are screened out, and only entities that are highly matched with the remaining entities in the image knowledge graph are retained as candidate entities.

[0168] Finally, the Large Language Model (LLM) is used to align the candidate entities with the remaining entities in the image knowledge graph. For each remaining entity in the image knowledge graph, a corresponding text entity is found, thus completing the matching and alignment between the image knowledge graph and the text knowledge graph. The resulting image knowledge graph contains only aligned entities.

[0169] 2. Although the alignment method based on spectral clustering has achieved good results, in some cases information may be lost or entities cannot be directly aligned. At this time, the present invention further enriches the description of the image entity by combining the relevant text entity information in the multimodal document. For example, the image entity "flooded community" itself cannot provide information about the affected location and cause. However, two related text entities are mentioned in the multimodal document: "Hurricane Ian" and "Florida". Combining these two entities with the information in the multimodal document, the description of the entity can be enhanced to a more detailed description: "The flooded community is an area in Florida that was affected by Hurricane Ian, resulting in severe flooding and damage, and many houses were uninhabitable." This enhancement process is critical to the integrity of the information. In order to achieve the enhanced image knowledge graph, follow the steps below:

[0170] First, in each image knowledge graph containing only aligned entities, entities that need enhanced information are screened. According to the text knowledge graph corresponding to the image knowledge graph containing only aligned entities and the context of the image knowledge graph containing only aligned entities in the multimodal document, descriptive information is supplemented for the entities that need enhanced information to update the nodes and edges in the image knowledge graph containing only aligned entities, thereby obtaining each enhanced image knowledge graph containing only aligned entities.

[0171] 3. Use each enhanced image knowledge graph containing only aligned entities and the text knowledge graph corresponding to the enhanced image knowledge graph containing only aligned entities to update the entities (nodes) and edges in the image knowledge graph corresponding to the enhanced image knowledge graph containing only aligned entities obtained by the unimodal processing submodule. If the updated entity already exists in the image knowledge graph obtained by the unimodal processing submodule, then add or modify the relationship and description information of the corresponding entity in the image knowledge graph obtained by the unimodal processing submodule. If the updated entity does not exist in the image knowledge graph obtained by the unimodal processing submodule, then add new entities (nodes) and relationships to the image knowledge graph obtained by the unimodal processing submodule to obtain each updated image knowledge graph.

[0172] 4. Each updated image knowledge graph is merged with the corresponding text knowledge graph obtained by the unimodal processing submodule. According to the alignment relationship, the nodes and edges in the two knowledge graphs are merged to ensure the uniqueness of entities and the correct connection of relationships. Duplicate entities are deleted and related entities are connected to obtain a multimodal knowledge graph that is complete and consistent.

[0173] This achieves a deep fusion of image and text information and constructs a complete multimodal knowledge graph to support subsequent reasoning and question-answering tasks.

[0174] The cross-modal fusion submodule achieves a deep fusion of the image knowledge graph and the text knowledge graph, forming a unified multimodal knowledge graph. At the same time, through the complementary enhancement of multimodal information, the semantic expression capability of the knowledge graph is improved, effectively enhancing the integrity and semantic expression capabilities of multimodal information, and strengthening the reasoning ability and interpretability of the overall system.

[0175] The multimodal knowledge graph construction module is the starting point of the entire system process. It is responsible for converting the input multimodal information into a structured multimodal knowledge graph to support subsequent retrieval and generation tasks.

[0176] The retrieval module is used to extract entities, entity relationships, and contextual information corresponding to the user's query from the multimodal knowledge graph generated by the multimodal knowledge graph construction module, providing accurate basic data for subsequent generation. The extracted entities, entity relationships, and contextual information corresponding to the query are input into the generation module. The entities, entity relationships, and contextual information include image information and text information.

[0177] The generation module integrates the received entities, entity relationships, and context information corresponding to the query question and outputs the answer to the query question. The specific process is:

[0178] First, a context is constructed based on the received entities, entity relationships, and contextual information corresponding to the query. A single-modal large model is used to generate a preliminary textual response to the textual information. Subsequently, a multimodal large model is called to combine image and textual information to generate a multimodal textual response. The preliminary textual response and the multimodal textual response are merged using a single-modal large language model. The generated text and multimodal information are integrated to output a comprehensive and consistent answer to the query. Given the current immaturity of multimodal large language model technology, the Generation module combines search results with user queries, generating high-quality responses that meet user needs using a hybrid generation method using a single-modal large language model (a model based solely on text) and a multimodal large language model (capable of processing multimodal information such as text and images). This process fully utilizes multimodal information (text and images) to ensure the accuracy and richness of the output content, as well as its high relevance and interpretability. This improves the accuracy and stability of the final results and effectively enhances the user experience.

[0179] This multimodal hybrid strategy not only fully leverages the maturity and advantages of single-modal models in specific tasks, ensuring high-quality semantic understanding and language expression in generated content, but also further enriches the details and contextual relevance of generated results through the fusion of multi-dimensional information from multimodal models. This approach effectively alleviates the limitations of current multimodal models while ensuring improved overall system generation quality.

[0180] Example

[0181] 1. Software Installation and Environment Configuration

[0182] This example uses the DocBench and MMLongBench datasets as examples, selecting PDF documents from each dataset. To achieve optimal PDF document extraction, we install the Mineru tool. The required libraries are listed in `cache / requirements.sh`. It is recommended that you configure this tool in a Conda environment running Python 3.10.

[0183] The necessary dependency library installation commands are as follows:

[0184] pip install openai

[0185] pip install sentence-transformers

[0186] pip install nano-vectordb

[0187] pip install python-docx

[0188] pip install PyMuPDF

[0189] pip install ultralytics

[0190] pip install tiktoken

[0191] pip install-U"magic-pdf[full]"

[0192] Among them, `magic-pdf` is the MinerU body.

[0193] If you need to deploy the embedding model locally, you can install multiple embedding models. We recommend using `sentence-transformers / all-MiniLM-L6-v2` as the text embedding model and `stella-en-1.5B-v5` as the embedding model. The embedding model can be downloaded through the ModelScope platform. The specific command is as follows:

[0194] pip install modelscope

[0195] modelscope download--model sentence-transformers / all-MiniLM-L6-v2

[0196] All embedding model parameter configurations are stored in the ` / mmgraphrag / parameter.py` file, including the local embedding model storage path and model API settings.

[0197] 2. Configuration Parameter Description

[0198] `mineru_dir`: The directory where documents processed by MinerU are stored. It can be used to store the document results obtained by MinerU analysis during testing.

[0199] `QueryParam` parameter configuration:

[0200] `response_type`: defines the response format.

[0201] `top_k`: Maximum number of entities to retrieve.

[0202] `local_max_token_for_text_unit`: The maximum number of tokens for entity and relationship text in the search results.

[0203] `local_max_token_for_local_context`: The maximum number of tokens for the content block (context text) being retrieved.

[0204] `number_of_mmentities`: The maximum number of images in the search results.

[0205] Image detection: By default, the YOLOv8n-segmentation model located at ` / cache / yolov8n-seg.pt` is used.

[0206] 3. User Guide

[0207] After the configuration is complete, you can run the sample test script `mmgraphrag / mmgraphrag_test.py` to verify the process. The calling method example is as follows:

[0208] Python

[0209] Call Example

[0210] pdf_path="path / to / input.pdf"

[0211] working_dir="path / to / output"

[0212] question="User Question"

[0213] Constructing test commands

[0214] python mmgraphrag / mmgraphrag_test.py--pdf_path=YOUR_PDF_PATH--working_dir=OUTPUT_DIR--question=YOUR_QUESTION--input_mode=0 / 1 / 2--query_mode=True / False

[0215] Input mode (`input_mode`) description:

[0216] Mode 2: Applicable to processing PDF documents after installing MinerU.

[0217] Mode 0: Process DOCX format documents.

[0218] Mode 1: Handles well-structured PDF documents, but performs poorly on complex PDFs.

[0219] Query mode (`query_mode`):

[0220] When set to `True`, the system will perform question-answering reasoning based on user questions.

[0221] Note: Enabling `query_mode` requires setting `True`.

[0222] The above process design ensures the flexibility and scalability of the document intelligent parsing and question-answering system. Combined with different input types and system configurations, it can adapt to diverse and complex document understanding and question-answering needs.

[0223] The effectiveness of the present invention is illustrated by experimental results. Figure 4 and Figure 5 shown.

[0224] Multimodal information processing capabilities: Compared with multimodal LLM methods, relying solely on the understanding capabilities of multimodal LLM may not effectively improve the answering of image-related questions. In some cases, answers based on image-related text are even better (such as NaiveRAG). MMGraphRAG uses fusion and retrieval steps to accurately locate relevant images, maximizing the visual understanding advantages of multimodal LLM, thereby improving the quality of answering image-related questions.

[0225] Multimodal Information Fusion and Reasoning Capabilities: In experiments on the DocBench and MMLongBench datasets, MMGraphRAG outperformed GraphRAG on text-only questions, demonstrating the effectiveness of multimodal information fusion. Introducing additional modal information significantly improves the accuracy of answering text questions, particularly in models with strong reasoning capabilities, such as Internvl.

[0226] Multi-domain adaptability: In DocBench experimental results, while current RAG methods have achieved good results in text-heavy tasks (such as Gov and Law), MMGraphRAG significantly improves performance in the Academy and Finance domains. This is because questions in these two domains often involve graphs and tables, and MMGraphRAG can effectively integrate this additional multimodal information, improving question-answering accuracy. Furthermore, combined with the MMLongBench domain results in the appendix \ref{appendix.h}, MMGraphRAG demonstrates its adaptability to a wider range of domains, demonstrating strong flexibility and adaptability.

[0227] The system parameters are as follows. To ensure the fairness of the experiment and eliminate the possibility of incorrect evaluation when using the same model for RAG and answering tasks, a variety of single-modal large models and multi-modal large models were selected as comparison benchmarks. Only two of each model were selected for recording, including the single-modal Llama3.1-70B-Instruct (L), Qwen2.5-72B-Instruct (Q) and the multi-modal Qwen2-VL-72B (Qvl), InternVL2.5-38B-MPO (Intvl).

[0228] In order to make a valid comparison, the following methods were selected:

[0229] LLM: All text and questions in the Markdown document processed by Mineru are fed to LLM for question answering. If the text and questions exceed the context limit of the model, the questions are asked in batches and multiple answers are stitched together.

[0230] MMLLM: All images are concatenated together and scaled according to the maximum limit of the model input. The spliced ​​images are provided to the multimodal LLM together with the question to answer in order to evaluate its ability in multimodal information processing.

[0231] NaiveRAG: The text in the Markdown document processed by MinerU is divided into blocks according to a fixed number of tokens (500), and each text block and question are embedded using the embedding model. Then, the k text blocks that are most relevant to the question are retrieved by calculating the cosine similarity. Finally, these relevant text blocks and the question are provided to the LLM for generation task.

[0232] GraphRAG: Based on Microsoft's open-source GraphRAG, this method was modified for this experiment. To ensure fair comparison with other methods and multimodal GraphRAG, the community clustering component was removed, and queries were performed in local mode. Retrieval was performed using top-k entities (10 selected), with a maximum of 4000 tokens for entity and relationship text and a maximum text block length of 6000 tokens. All other parameters were consistent with multimodal GraphRAG.

[0233] The above examples are merely illustrative of the calculation model and process of the present invention and are not intended to limit the embodiments of the present invention. Persons skilled in the art will readily appreciate that other variations or modifications based on the above description are possible. This list of embodiments is not exhaustive; however, any obvious variations or modifications derived from the technical solution of the present invention remain within the scope of protection of the present invention.

Claims

1. A document intelligent parsing and question-answering method based on RAG and multimodal knowledge graph, characterized by: It includes the following steps: S1. Obtain a multimodal document and parse it using MinerU and a large multimodal model to obtain text modal information and image modal information. S2. Use entity extraction technology to process text modal information and generate a text knowledge graph. At the same time, use image segmentation and scene graph construction tasks to process image modal information and generate an image knowledge graph. S3. Fuse the text knowledge graph and the image knowledge graph to obtain a multimodal knowledge graph; S4. Input the multimodal knowledge graph into the embedding model, output the corresponding vector, input the corresponding vector into the vector database, and construct and obtain the entity vector database; S5. Input the user's query question into the embedding model, output the corresponding vector, input the corresponding vector into the entity vector database, obtain the information corresponding to the query question, input the information corresponding to the query question into the large language model, and output the answer to the query question.

2. The document intelligent parsing and question-answering method based on RAG and multimodal knowledge graph according to claim 1 is characterized by: The multimodal document in S1 is a PDF document, a docx document, an image document jpg, an image document png or a txt document.

3. The document intelligent parsing and question-answering method based on RAG and multimodal knowledge graph according to claim 1 is characterized by: In S1, a multimodal document is obtained, and MinerU and a multimodal large model are used to parse the multimodal document to obtain text modal information and image modal information. The specific process is as follows: Obtain multimodal documents and use the open source tool MinerU to perform structured analysis on them: For the text content in the multimodal document, all the text content is divided into multiple text blocks, each text block is numbered and labeled with an ID, and the information of each text block is obtained, which is the text modality information; For each image in the multimodal document, extract the image number, storage path, image title, footnote and context text of the current image in the multimodal document; The multimodal large model is then used to generate a detailed description for each image. The detailed description includes each element in the image, the attributes of each element, and the emotion or atmosphere conveyed by the image content. The information of each image is obtained, namely the image modal information. The elements include people, objects, scenes and backgrounds, and the attributes of the elements include color, shape, size and posture.

4. The document intelligent parsing and question-answering method based on RAG and multimodal knowledge graph according to claim 3 is characterized by: In S2, the text modal information is processed using entity extraction technology to generate a text knowledge graph. At the same time, the image modal information is processed using image segmentation and scene graph construction tasks to generate an image knowledge graph. The specific process is as follows: S21. Process each text block using entity extraction technology to generate a text knowledge graph for each text block; S22. Perform semantic segmentation on each image using a Yolo v8 network to obtain image feature blocks for each image, where the number of image feature blocks for each image is N, where N is an integer and N is greater than or equal to 0. S23. Use a large multimodal model to build a scene graph task: S231. Generate a detailed description for each image feature block using a multimodal large model, wherein the detailed description includes entity types and features within the image feature block, and obtain each image feature block after the detailed description; The entity types include objects, organisms and people, and the entity characteristics include entity name, shape, color, size, age, posture, structural characteristics, size and material; S232. Using the multimodal large model, extract all entities and relationships between entities in each image, construct entity relationship information of the corresponding image based on the relationships between the entities, and obtain entity relationship information for each image; S233, using the multimodal large model to align each image feature block described in detail in each image with the corresponding entity; S24. According to S22-S23, a global entity description is generated for each image to obtain an image knowledge graph for each image.

5. The document intelligent parsing and question-answering method based on RAG and multimodal knowledge graph according to claim 1 is characterized by: In S3, the text knowledge graph and the image knowledge graph are fused to obtain a multimodal knowledge graph. The specific process is as follows: S31. For each text knowledge graph and the image knowledge graph corresponding to the text knowledge graph, first, based on the information of the entities in the image knowledge graph, delete the entities that have established relationships with the text blocks. Next, based on the information of the remaining entities in the image knowledge graph and the entities and relationships in the text knowledge graph, use the spectral clustering method to classify all entities in the text knowledge graph to obtain candidate entities. Finally, use the large language model to align the candidate entities with the remaining entities in the image knowledge graph to obtain image knowledge graphs that only contain aligned entities. S32. In each image knowledge graph containing only aligned entities, entities requiring enhanced information are screened and supplemented with descriptive information for the entities requiring enhanced information based on the text knowledge graph corresponding to the image knowledge graph containing only aligned entities and the context of the image knowledge graph containing only aligned entities in the multimodal document, thereby obtaining each enhanced image knowledge graph containing only aligned entities. S33. Using each enhanced image knowledge graph containing only aligned entities and the corresponding text knowledge graph obtained in S32, the image knowledge graph corresponding to the enhanced image knowledge graph containing only aligned entities obtained in S24 is updated. In the image knowledge graph obtained in S24, if the updated entity already exists, the relationship and information of the corresponding entity are added or modified. If the updated entity does not exist, the entity and relationship are added to obtain each updated image knowledge graph; S34. Fuse each updated image knowledge graph with the corresponding text knowledge graph obtained in S21 to obtain each multimodal knowledge graph.

6. The document intelligent parsing and question-answering method based on RAG and multimodal knowledge graph according to claim 5 is characterized by: In S31, based on the information of the remaining entities in the image knowledge graph and the entities and relationships in the text knowledge graph, a spectral clustering method is used to classify all entities in the text knowledge graph to obtain candidate entities. The specific process is as follows: 1) Taking entities as nodes, calculate the cosine similarity between the remaining nodes in the image knowledge graph and the nodes in the text knowledge graph: Among them, v i ,v j are the vector representations of nodes i and j respectively, ||·|| is the L2 norm of the vector; 2) Use the information of the remaining entities in the image knowledge graph and the relationship edge weights to construct the adjacency matrix A. Let the relationship between nodes i and j be r ij By weight weight(r ij ) is given. If there is a clear relationship definition between nodes i and j, the weight value is known. If there is no clear relationship definition between nodes i and j, the weight is 1. 3) The product of cosine similarity and weight is used as the element of adjacency matrix A, which is: A ij =sim(v i ,v j )·weight(r ij ) 4) The degree matrix D is a diagonal matrix, where each diagonal element D ii is the degree of node i, that is, the connection strength of node i with other nodes, and each diagonal element D is defined according to the adjacency matrix A. ii for: Therefore, the degree of each node in the degree matrix D is the sum of the similarities of the node with all other nodes; 5) Calculate the Laplace matrix L = DA of the image knowledge graph based on the adjacency matrix A and the degree matrix D, perform eigendecomposition on the Laplace matrix, extract the eigenvectors corresponding to the first k smallest eigenvalues, and use the extracted eigenvectors as the matrix Q = [q1,q1,…,q k ] columns, where Indicates the number of entities in the nearby entity list; 6) Use the DBSCAN clustering algorithm to cluster the matrix Q and obtain the clustering results C1, C2, ..., C k , map the clustering results to obtain entity clustering results A1, A2, ..., A k , where A i ={v j |r j ∈C i }; 7) Based on the entity clustering results, entities in the text knowledge graph that are not closely related or irrelevant to the remaining entities in the image knowledge graph are screened out, and only entities that are highly matched with the remaining entities in the image knowledge graph are retained as candidate entities.

7. Intelligent document parsing and question-answering system based on RAG and multimodal knowledge graph, characterized by: It includes a multimodal knowledge graph construction module, a retrieval module, and a generation module; The multimodal knowledge graph construction module is used to extract the modal information of multimodal documents and fuse the modal information to obtain a multimodal knowledge graph; The multimodal document is a PDF document, a docx document, an image document jpg, an image document png or a txt document; The multimodal knowledge graph construction module includes a preprocessing submodule, a single-modal processing submodule, and a cross-modal fusion submodule in sequence; The preprocessing submodule is used to preprocess the multimodal document, output each modal information, and input each modal information into the single modal processing submodule; The single-modal processing submodule is used to process the received modal information, generate a knowledge graph for each modal information, and input the knowledge graph of all modal information into the cross-modal fusion submodule; The cross-modal fusion submodule is used to fuse the knowledge graphs of all received modal information to obtain a multimodal knowledge graph; The retrieval module is used to extract entities, entity relationships, and context information corresponding to the query question from the multimodal knowledge graph obtained by the multimodal knowledge graph construction module based on the user's query question, and input the extracted entities, entity relationships, and context information corresponding to the query question into the generation module; The generation module is used to integrate the received entities, entity relationships and context information corresponding to the query question and output the answer corresponding to the query question.

8. The document intelligent parsing and question-answering system based on RAG and multimodal knowledge graph according to claim 7, characterized in that: The specific process of the pre-processing submodule is as follows: The preprocessing submodule includes the open source tool MinerU and a large multimodal model; The multimodal document is input into the preprocessing submodule, and the open source tool MinerU is used to perform structural analysis on the multimodal document: For the text content in the multimodal document, all the text content is divided into multiple text blocks, each text block is numbered and labeled with an ID, and the information of each text block is obtained, which is the text modality information; For each image in the multimodal document, the image number, storage path, image title, footnote and context text of the current image in the multimodal document are extracted, and then a detailed description is generated for each image using the multimodal large model. The detailed description includes each element in the image and the attributes of each element, the emotion or atmosphere conveyed by the image content, and the information of each image is obtained, which is the image modal information.

9. The document intelligent parsing and question-answering system based on RAG and multimodal knowledge graph according to claim 7, characterized in that: The specific process of the single-mode processing submodule is as follows: The single-modal processing submodules include the Text2Graph module and the Img2Graph module; The text modal information output by the preprocessing submodule is input into the Text2Graph module, and each text block is processed using entity extraction technology to generate a text knowledge graph for each text block; The image modality information output by the preprocessing submodule is input into the Img2Graph module, and the image modality information is processed by image segmentation and scene graph construction in sequence to generate an image knowledge graph for each image. The specific process is as follows: The Img2Graph module includes the Yolo v8 network and the multimodal large model in turn; 1) Using the Yolo v8 network to perform semantic segmentation on each image, obtaining image feature blocks for each image, where the number of image feature blocks for each image is N, where N ≥ 0, and the number is an integer; 2) using the multimodal large model to generate a detailed description for each image feature block, wherein the detailed description includes the entity types and features within the image feature block, thereby obtaining each image feature block after the detailed description; The entity types include objects, organisms and people, and the entity characteristics include entity name, shape, color, size, age, posture, structural characteristics, size and material; 3) Using a large multimodal model to extract all entities and the relationships between entities in each image, construct entity relationship information for the corresponding image based on the relationships between entities, and obtain entity relationship information for each image; 4) Using a large multimodal model, each image feature block described in detail in each image is aligned with the corresponding entity; 5) Based on 1)-4), generate a global entity description for each new image and obtain the image knowledge graph of each image.

10. The document intelligent parsing and question-answering system based on RAG and multimodal knowledge graph according to claim 7, characterized in that: The specific process of the cross-modal fusion submodule is as follows: 1) For each text knowledge graph and the image knowledge graph corresponding to the text knowledge graph, first, delete the entities that have established relationships with the text block based on the information of the entities in the image knowledge graph; Next, based on the information of the remaining entities in the image knowledge graph and the entities and relationships in the text knowledge graph, the spectral clustering method is used to classify all entities in the text knowledge graph to obtain candidate entities; Finally, the large language model is used to align the candidate entities with the remaining entities in the image knowledge graph, obtaining an image knowledge graph that only contains aligned entities. 2) In each image knowledge graph containing only aligned entities, entities requiring enhanced information are screened and supplemented with descriptive information for the entities requiring enhanced information based on the text knowledge graph corresponding to the image knowledge graph containing only aligned entities and the context of the image knowledge graph containing only aligned entities in the multimodal document, thereby obtaining each enhanced image knowledge graph containing only aligned entities; 3) using each enhanced image knowledge graph containing only aligned entities and the corresponding text knowledge graph to update the image knowledge graph corresponding to the enhanced image knowledge graph containing only aligned entities obtained by the unimodal processing submodule, and in the image knowledge graph obtained by the unimodal processing submodule, if the updated entity already exists, then adding or modifying the relationship and information of the corresponding entity; if the updated entity does not exist, then adding a new entity and relationship to obtain each updated image knowledge graph; 4) Each updated image knowledge graph is fused with the corresponding text knowledge graph obtained by the unimodal processing submodule to obtain each multimodal knowledge graph.

Images