Text-graph multimodal document processing and search enhancement method based on entity community relationship

Abstract

The present invention discloses a text-graph multimodal document processing and search enhancement method based on entity community relations, characterized in that it comprises the following steps: a user uploads a document containing text and images; preprocesses the document; text processing; uses a large visual language model to determine the type of each image; image processing; knowledge graph construction and integration; hierarchical clustering; query keyword extraction; keyword matching; retrieval-enhanced answer generation. The present invention combines the ability to understand multimodal data (such as images and videos) with search enhancement technology based on knowledge graphs to achieve efficient processing of multimodal documents of entity community relations and significant enhancement of search functions.

Description

Technical Field

[0001] The invention relates to a multimodal input processing method, and belongs to the technical field of computer vision and natural language processing. Background Art

[0002] With the explosive growth of digital information, especially the large amount of illustrated content in social media, online publications, and academic literature, entity community relationships have developed into multimodal forms such as images and tables, and they change dynamically over time. Most existing multimodal input processing technologies are limited to the separate processing of text or images, and lack effective means to fuse data from different modalities and conduct in-depth understanding. Although current knowledge graph-based search enhancement technologies (such as GraphRAG) can enhance text generation through knowledge graphs, they currently only support text input, which limits their application in the field of multimodal data processing. Summary of the invention

[0003] The technical problem to be solved by the present invention is that the prior art lacks a method that can effectively convert multimodal inputs (such as images, videos) into entity relationship representations and use these relationships to enhance the generation of knowledge graphs.

[0004] In order to solve the above technical problems, the technical solution of the present invention is to disclose a text-graph multi-modal document processing and search enhancement method based on entity community relationship, which is characterized by comprising the following steps:

[0005] Step 1: The user uploads a document containing text and images;

[0006] Step 2: pre-process the document, extract text content and image content from the document, and record the positional relationship between the image and the text;

[0007] Step 3: Text processing, including entity recognition, relationship extraction, and claim extraction, wherein entity recognition is used to identify key entities in the text, relationship extraction is used to obtain entity relationships in the text, and claim extraction is used to obtain key information or claims expressed in each text unit;

[0008] Step 4: Use the visual language model to determine the type of each image;

[0009] Step 5: Image processing, including:

[0010] Use prompts with specific label symbols to extract entity relationships from different types of images;

[0011] Use LVLM to identify key elements in different types of images, as well as different entities in general images, and generate a short text description for each element, identify entities and relationships in flowcharts, and identify claims expressed in images;

[0012] Step 6: Knowledge graph construction and integration:

[0013] If there is no pre-built knowledge graph in the system, build a new knowledge graph based on the entities, relationships, and claims extracted in steps 3, 4, and 5;

[0014] If there is a pre-built knowledge graph in the system, map the entities, relationships, and claims extracted in steps 3, 4, and 5 to the pre-built knowledge graph, and use the reasoning function of the knowledge graph to discover the relationships between existing entities;

[0015] Step 7: Hierarchical clustering:

[0016] Use Leiden technology to perform hierarchical clustering on the initial knowledge graph, assign the entities in each cluster to different communities, and use a bottom-up approach to generate summaries for each community and its important parts;

[0017] Step 8: Query keyword extraction:

[0018] Based on user queries, local keywords and global keywords are extracted for matching retrieval;

[0019] Step 9: Keyword matching;

[0020] Using the vector database, local keywords are matched to related entities, and global keywords are matched to corresponding communities;

[0021] Step 10: Retrieval-enhanced answer generation, including:

[0022] Global search: Reasoning about holistic questions involving the entire data corpus by leveraging community summaries;

[0023] Local search: Reasoning about a specific entity by expanding to its neighbors and related concepts;

[0024] Fusion search: further integrate the results of global search and local search to obtain answers that are more in line with the query context.

[0025] Preferably, in step 1, the format of the document includes PDF, DOCX or HTML, and the document contains non-text content including tables, flow charts or general images.

[0026] Preferably, in step 2, the preprocessing includes: first converting documents of different formats into a unified Markdown format using a format parsing tool, and then performing text extraction and image extraction.

[0027] Preferably, in step 3, the entire input text is divided into a plurality of text units by text segmentation, and these text units are the smallest analyzable units.

[0028] Preferably, in step 6, a graph database is used to store entities, relationships and claims, and relationships between entities are established, and the knowledge graph is preliminarily cleaned and integrated.

[0029] Preferably, in step 7, the summary includes the main entities in the community, the relationships between the entities and the key claims.

[0030] The present invention combines the ability to understand multimodal data (such as images and videos) and search enhancement technology based on knowledge graphs to achieve efficient processing of multimodal documents of entity community relationships and significant enhancement of search functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0033] The embodiment of the present invention discloses a method for text-graph multimodal document processing and search enhancement based on entity-community relations, comprising the following steps:

[0034] Step 1: User uploads a document

[0035] The user uploads a document containing text and images. The document can be in multiple formats such as PDF, DOCX or HTML. The document may contain non-text content such as tables, flow charts or general images.

[0036] Step 2: Document preprocessing, including:

[0037] Format parsing: Use format parsing tools to convert documents in different formats into a unified Markdown format;

[0038] Text extraction: extract text content from documents, including titles, text, text in tables, etc.;

[0039] Image extraction: Extract image content from documents and record the positional relationship between the image and text.

[0040] Step 3: Text processing, including:

[0041] Text segmentation: All input text is divided into multiple text units (text blocks). These text blocks are the smallest analyzable units, which can be paragraphs, sentences or other logical units.

[0042] Entity recognition: using large models to identify key entities in text (for example, medical documents often contain entities such as step numbers, surgical tool names, and drug names);

[0043] Relation extraction: Use a large model to identify entity relationships in text, for example: "Step 1 and Step 2 are a sequential relationship" and "Surgical instruments and surgical operations are tool usage relationships";

[0044] Claim extraction: Use a large model to identify and extract the key information or claims expressed in each text unit.

[0045] Step 4: Image Classification

[0046] Use a large visual language model (LVLM), such as Qwen-VL, to determine the type of each image, such as whether it is a flowchart, a table, or a general image.

[0047] Step 5: Image processing, including:

[0048] Use prompts with specific label symbols to extract entity relationships from different types of images;

[0049] Use LVLM to identify key elements in different types of images, such as step boxes, arrows, text annotations in flowcharts, table headers, formats, and different entities in general images, and generate a short text description for each element to identify entities and relationships in flowcharts;

[0050] Also, try to identify the claims expressed in the image, such as key steps or results in a flowchart, titles or key data points in a table, or titles or themes in an image.

[0051] Step 6: Knowledge graph construction and integration

[0052] If there is no pre-built knowledge graph in the system, a new knowledge graph is constructed based on the entities, relationships and claims extracted in steps 3, 4 and 5. In the present invention, a graph database is used to store entities, relationships and claims, and the relationships between entities are established. The knowledge graph is preliminarily cleaned and integrated, such as eliminating redundant information and merging synonyms.

[0053] If there is a pre-built knowledge graph in the system, the entities, relationships, and claims extracted in steps 3, 4, and 5 are mapped to the pre-built knowledge graph, and the reasoning function of the knowledge graph is used to discover the relationship between existing entities. For example, it can be discovered through reasoning that "Mr. Li" is a doctor in "a certain hospital".

[0054] Step 7: Hierarchical Clustering

[0055] Hierarchical clustering is performed on the initial knowledge graph using the Leiden technique. Entities in each cluster are assigned to different communities for deeper analysis. At the same time, a bottom-up approach generates summaries for each community and its important parts. These summaries include the main entities within the community, the relationships between entities, and key claims. This step provides an overview of the entire dataset and provides useful contextual information for subsequent queries.

[0056] Step 8: Query keyword extraction

[0057] In response to user queries, the system automatically extracts local keywords (low-level) and global keywords (high-level) for matching retrieval.

[0058] Step 9: Keyword Matching

[0059] Using the vector database, local keywords will match the relevant Entity, and global keywords will match the corresponding Community.

[0060] Step 10: Retrieve enhanced answer generation, including:

[0061] Global search: Reasoning about holistic questions involving the entire data corpus by leveraging community summaries;

[0062] Local search: Reasoning about a specific entity by expanding to its neighbors and related concepts;

[0063] Fusion search: further integrate the results of global search and local search to obtain answers that are more in line with the query context.

Claims

1. A text-graph multimodal document processing and search enhancement method based on entity community relations, characterized in that: The following steps are involved: Step 1: The user uploads a document containing text and images; Step 2: pre-process the document, extract text content and image content from the document, and record the positional relationship between the image and the text; Step 3: text processing, including entity recognition, relationship extraction and claim extraction, wherein entity recognition is used to identify key entities in the text, relationship extraction is used to obtain entity relationships in the text, and claim extraction is used to obtain key information or claims expressed in each text unit; Step 4: Use the visual language model to determine the type of each image; Step 5: Image processing, including: Use prompts with specific label symbols to extract entity relationships from different types of images; Use LVLM to identify key elements in different types of images, as well as different entities in general images, and generate a short text description for each element, identify entities and relationships in flowcharts, and identify claims expressed in images; Step 6: Knowledge graph construction and integration: If there is no pre-built knowledge graph in the system, build a new knowledge graph based on the entities, relationships, and claims extracted in steps 3, 4, and 5; If there is a pre-built knowledge graph in the system, map the entities, relationships, and claims extracted in steps 3, 4, and 5 to the pre-built knowledge graph, and use the reasoning function of the knowledge graph to discover the relationships between existing entities; Step 7: Hierarchical clustering: Use Leiden technology to perform hierarchical clustering on the initial knowledge graph, assign the entities in each cluster to different communities, and use a bottom-up approach to generate summaries for each community and its important parts; Step 8: Query keyword extraction: Based on user queries, local keywords and global keywords are extracted for matching retrieval; Step 9: Keyword matching; Using the vector database, local keywords are matched to related entities, and global keywords are matched to corresponding communities; Step 10: Retrieval-enhanced answer generation, including: Global search: Reasoning about holistic questions involving the entire data corpus by leveraging community summaries; Local search: Reasoning about a specific entity by expanding to its neighbors and related concepts; Fusion search: further integrate the results of global search and local search to obtain answers that are more in line with the query context.

2. A method for text-graph multimodal document processing and search enhancement based on entity-community relations as claimed in claim 1, characterized in that: In step 1, the format of the document includes PDF, DOCX or HTML, and the document contains non-text content such as tables, flow charts or general images.

3. The method for text-graph multimodal document processing and search enhancement based on entity-community relationship as claimed in claim 1, characterized in that: In step 2, the preprocessing includes: first using a format parsing tool to convert documents of different formats into a unified Markdown format, and then performing text extraction and image extraction.

4. The method for text-graph multimodal document processing and search enhancement based on entity-community relationship as claimed in claim 1, characterized in that: In step 3, all input text is divided into multiple text units through text segmentation, and these text units are the smallest analyzable units.

5. The method for text-graph multimodal document processing and search enhancement based on entity-community relationship as claimed in claim 1, characterized in that: In step 6, a graph database is used to store entities, relationships, and claims, and relationships between entities are established, and the knowledge graph is preliminarily cleaned and integrated.

6. The method for text-graph multimodal document processing and search enhancement based on entity-community relationship as claimed in claim 1, characterized in that: In step 7, the summary includes the main entities in the community, the relationships between the entities, and the key claims.

Images