An image encyclopedia question answering method and device based on a multi-modal hypergraph

Abstract

The application discloses a kind of based on multimodal supergraph image encyclopedia question and answer method and device, it is related to electric digital data processing technical field;Method includes step 1 multimodal document structural analysis, step 2 constructs text supergraph, step 3 constructs visual graph, step 4 constructs document level multimodal knowledge graph, step 5 constructs multimodal retrieval index, step 6 based on question and the multimodal recall of query image and candidate document aggregation and step 7 constructs query relevant local multimodal evidence subgraph, device includes computer readable storage medium, by the text fact in encyclopedia knowledge document, image, visual object and corresponding relation are organized into unified knowledge structure, according to the image and question of user provided relevant evidence is retrieved, forms the complete evidence chain that can be used for answer generation, so that next step combines evidence chain and large language model, output is used to answer question answer.

Description

Technical fields unless

[0001] This invention relates to the field of electronic digital data processing technology, and in particular to an image encyclopedia question-and-answer method and apparatus based on a multimodal hypergraph. Background Technology

[0002] While existing multimodal knowledge graph question answering methods can utilize both textual and image information simultaneously, they suffer from the following technical problems when handling image question answering tasks that require external encyclopedic knowledge.

[0003] 1. Difficulty in fully representing facts involving multiple conditions. A single fact in an encyclopedia document often contains multiple objects, attributes, values, units, and limiting conditions. For example, the statement that a female animal weighs 20 to 30 kilograms includes the animal's name, sex, weight attribute, numerical range, and unit of measurement. Existing methods typically break this fact down into multiple relationships, which easily leads to the loss of the original overall connections between the various elements. During a search, the system might find the weight value but miss the limiting condition of "female," resulting in inaccurate evidence or answers.

[0004] 2. The correspondence between textual facts and image content is not complete. A textual fact may not be supported by a single visual object, but rather by the entire image, multiple visual objects, or the relationships between objects. For example, a textual description of two creatures of the same species usually appears in pairs. This fact needs to be supported by two objects in the image and their relative relationships, and cannot be fully represented solely by the connection between a textual entity and a visual object. Although existing methods place text and images into the same knowledge graph, the connections between them are still relatively simple, making it difficult to clearly indicate which content in the image corresponds to which complete fact in the text.

[0005] 3. The retrieved evidence is scattered and difficult to form a complete chain of evidence. Existing retrieval methods typically retrieve document fragments, entity nodes, relation edges, or image objects separately. The retrieval results are often just a few independent pieces of content, lacking further organization of the connections between different pieces of evidence. For example, when a user uploads a bird image and asks what the bird's next-level classification is, the system may retrieve the bird image, species name, classification relationship, and encyclopedia article paragraph, but it cannot guarantee that these contents belong to the same species, nor can it guarantee that the classification relationship correctly corresponds to the bird object in the query image. In this case, the answer generation model still needs to determine the connection between the image object, text entity, classification relationship, and source paragraph, which is easily influenced by irrelevant content and makes it difficult to explain which set of evidence the final answer is based on.

[0006] Therefore, how to obtain well-sourced and interconnected multimodal evidence based on the query images and questions provided by users has become an urgent technical problem to be solved. Summary of the Invention

[0007] This invention provides an image encyclopedia question-answering method and apparatus based on a multimodal hypergraph, which solves the technical problem of obtaining well-sourced and interrelated multimodal evidence based on query images and questions provided by users.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: Step 1: The original multimodal encyclopedia knowledge document is parsed using multimodal document structure parsing to obtain a structured document. The structured document includes a document identifier, document title, section set, image set, and image association mapping relationship between document, section, and image. The section set includes multiple section nodes, and the image set includes multiple images. Step 2: Based on segmented text extraction, obtain a set of text entities, a set of text relation edges, and a set of text hyperedges. The set of text entities includes multiple text entity nodes, each containing a source segment identifier. The set of text relation edges includes multiple text relation edges, each containing a source segment and the original evidence text. The set of text hyperedges includes multiple text hyperedges. Perform name normalization, deduplication, and merging on the text entity nodes, text relation edges, and text hyperedges, retaining the source segment and the original evidence text of the text entity nodes. This yields the text entity source mapping relationship between text entity nodes and source segments, forming a document-level text hypergraph. ; Step 3: For each image in the image set of the same structured document, perform overall image semantic understanding, visual object detection, and visual relation extraction. This yields image nodes, visual object nodes, image-object containment relationships, and visual relation edges between visual object nodes. It also obtains the image source mapping relationship between each image node and its corresponding visual object node, including the source document, source section, and source image, forming a document-level visual graph. Image nodes include image node identifier, image description, overall semantic summary, local image storage path, source document, and source section; visual object nodes include visual object node identifier, object category, source image, bounding box, detection confidence, and object description. Step 4: Develop the document-level visual graph obtained in Step 3. Image nodes, visual object nodes, and visual relation edges are integrated into the document-level text hypergraph obtained in step 2. By performing cross-modal alignment between text entity nodes and visual object nodes, the entity-object cross-modal alignment relationship between text entity nodes and visual object nodes is obtained, forming a document-level heterogeneous multimodal knowledge graph. Document-level heterogeneous multimodal knowledge graph This includes segmentation nodes, text entity nodes, text relation edges, text hyperedges, image association mapping relationships, image nodes, visual object nodes, visual relation edges, entity-object alignment relationships, and image source mapping relationships; Step 5: Based on the document-level heterogeneous multimodal knowledge graph obtained in Step 4 The process involves segmented text vector encoding, whole-image visual feature encoding, object region extraction and visual feature encoding, and index-graph mapping construction. This results in segmented text retrieval indexes, whole-image visual retrieval indexes, visual object retrieval indexes, and index-graph mapping relationships between each index item and graph node and its document, forming a multimodal retrieval index. Step 6: Obtain the user-input query, which includes a query image and a natural language question. Extract question text features from the natural language question, and extract query whole image features and object region visual features from the query image. Use the question text features to perform recall in the segmented text retrieval index to obtain segmented text recall results, use the query whole image features to perform recall in the whole image visual retrieval index to obtain whole image visual recall results, and use the object region visual features to perform recall in the visual object retrieval index to obtain visual object recall results. Normalize, weight, and aggregate the segmented text recall results, whole image visual recall results, and visual object recall results to obtain several documents with high document relevance scores as candidate documents, forming a candidate document set. Candidate documents include those that hit segments, hit images, and hit visual objects. Based on the index-graph mapping relationship and candidate document transformation, obtain the graph node identifiers of the candidate documents to form a set of hit graph nodes for each candidate document. Step 7: Based on the corresponding document-level heterogeneous multimodal knowledge graph obtained in Step 4 Using the hit segment nodes, hit image nodes, and hit visual object nodes in the candidate documents obtained in step 6 as the starting point for expansion, the process extends along the document-level heterogeneous multimodal knowledge graph. The text relation edges, text hyperedges, visual relation edges, and entity-object alignment relations in the graph are locally expanded in relation to the query to form a local multimodal evidence subgraph. This allows us to obtain the context of structured evidence.

[0009] A further technical solution is as follows: In step 1, the multimodal document structured parsing step includes extracting document identifier, document title, section number, section title, section text, and images from the original multimodal encyclopedia knowledge document, as well as the image association mapping relationship between document-section-image. The section number, section title, and section text form sections, and one section corresponds to one section node. All sections form a section set. Images include image identifier, image link, image content, image description, and local cache path. All images form an image set. A structured document is formed based on the document identifier, document title, section set, image set, and image association mapping relationship.

[0010] A further technical solution is as follows: Step 2 specifically includes the following steps: Step 201: Extract text entities from the segmented text to form a text entity set. Text entities include categories and information. A text entity corresponds to a text entity node. The categories of text entities include species, locations, people, and buildings. The information of a text entity includes entity name, type, description, and source section. The source section includes source section identifier. Step 202: Extract the text relation edges between every two text entities from the segmented text to form a text relation set. Textual relation edges include the starting entity identifier, the target entity identifier, the relation type, the relation description, the source section, and the original evidence text. Step 203: Identify and obtain higher-order composite facts from the segmented text, and construct a text hyperedge based on each higher-order composite fact to form a set of text hyperedges. Higher-order composite facts include facts involving multiple entities. Each fact includes each entity, attribute conditions, numerical range, unit of measurement, and time conditions. The text hyperedge includes the role of each entity, fact type, semantic description, source section, and original evidence text. Step 204: For each section of the same structured document, perform name normalization, deduplication, and merging on the text entities, text relation edges, and text hyperedges obtained, retaining the source section and original evidence text of the text entities, to obtain the text entity source mapping relationship between the text entities and the source sections, forming a document-level text hypergraph. .

[0011] A further technical solution is as follows: Step 3 specifically includes the following steps: Step 301: For each image in the image set, all image nodes are obtained through overall semantic understanding. The image nodes include image node identifier, image description, overall semantic summary, local image storage path, source document, and source section. The step of obtaining the overall semantic summary includes generating an overall semantic summary for each image based on the image content and image description. Step 302: For each image in the image set, visual object detection is performed to obtain all visual object nodes and form a visual object node set. Establish an image-object containment relationship between an image node and the visual object nodes contained in that image node. The visual object node includes the visual object node identifier, object category, source image, bounding box, detection confidence, and object description. Step 303: Based on visual objects with clear semantic relationships in the same image, extract the spatial relationships, action relationships, and semantic relationships between the visual objects, and establish visual relationship edges between the corresponding visual object nodes; the visual relationship edge includes the visual object node identifiers at both ends of the relationship, the relationship type, the relationship description, and the source image, forming a set of visual relationships. ; Step 304: Obtain the image source mapping relationship between each image node and its corresponding visual object node, including the source document, source section, and source image. Based on the image node, visual object node, image-object containment relationship, visual relationship edge, and image source mapping relationship corresponding to each image within the same structured document, form a document-level visual graph. .

[0012] A further technical solution is as follows: Step 4 specifically includes the following steps: Step 401: Add the image nodes, visual object nodes, image-object containment relationships, and visual relationship edges obtained in Step 3 to the document-level graph structure; based on the document-section-image image association mapping relationship obtained in Step 1, establish the source association relationship between the section node and the corresponding image node; Step 402: Based on the object category and object description of each visual object node, as well as the corresponding image description, overall semantic summary of the image, and source segmentation, select semantically related candidate text entities from the text entity set obtained in Step 2 to form an entity-object alignment candidate set; Step 403: Combining the semantic similarity, source segment consistency, and image description of candidate text entities and visual object nodes, perform semantic matching on candidate text entities and visual object nodes to obtain candidate combinations and establish cross-modal alignment relationships between text entity nodes and visual object nodes, i.e., entity-object alignment relationships; each cross-modal alignment relationship includes text entity identifier, visual object node identifier, associated image node identifier, matching score, judgment criteria, and source information; Step 404: Form a document-level multimodal knowledge graph based on segment nodes, text entity nodes, text relation edges, text hyperedges, image association mapping relationships, image nodes, visual object nodes, visual relation edges, entity-object alignment relationships, and image source mapping relationships. Section nodes, text entity nodes, text relation edges, and text super edges form text side nodes and relations, while image nodes, visual object nodes, and visual relation edges form visual side nodes and relations.

[0013] A further technical solution is as follows: Step 5 specifically includes the following steps: Step 501: Divide the segment nodes in the multimodal knowledge graph into segment text retrieval units. Each segment text retrieval unit includes the document identifier, segment identifier, segment title, segment text, and corresponding graph node identifier. Step 502: Generate a text representation for each segmented text retrieval unit and perform vector encoding to obtain segmented text features; establish a segmented text retrieval index based on the segmented text features, which is used to recall relevant segments and the documents to which the segments belong based on natural language questions; Step 503: Read the corresponding image according to the local storage path of the image node, extract the visual features of the whole image, and associate the visual features of the whole image with the image node identifier, the document identifier, and the source section identifier to establish a visual retrieval index for the whole image; Step 504: Based on the source image and bounding box of the visual object node, extract the object region from the corresponding image, extract the visual features of the obtained object region, and associate the visual features of the object region with the visual object node identifier, image node identifier, document identifier, and source section identifier to establish a search index for the obtained visual object. Step 505: Record the corresponding graph node identifier and the document identifier for each index item to obtain the index-graph mapping relationship, so that subsequent search results can return the corresponding segment node, image node or visual object node, and can enter the heterogeneous multimodal knowledge graph of the corresponding document for local evidence expansion.

[0014] A further technical solution is as follows: Step 6 specifically includes the following steps: Step 601: Obtain the query image and natural language question input by the user; perform text vector encoding on the natural language question to obtain the question text features; perform overall visual feature encoding on the query image to obtain the query image features; perform object detection on the query image, extract the object region based on the detected bounding box, and extract the visual features of the object region. Step 602: Use the question text features to recall relevant sections in the segmented text retrieval index and obtain segmented text retrieval results; use the query whole image features to recall similar images in the whole image visual retrieval index and obtain whole image visual retrieval results; use the visual features of each object region to recall similar visual objects in the visual object retrieval index and obtain visual object retrieval results; each retrieval result includes retrieval type, similarity score, graph node identifier, and document identifier; Step 603: Normalize the similarity scores of the segmented text recall results, the whole image visual recall results, and the visual object recall results, and perform weighted fusion according to the corresponding retrieval type to obtain the comprehensive relevance score of each recall result; the weights of different retrieval types are preset according to actual application needs. Step 604: Group the recall results according to the document identifier to which each recall result belongs, merge the recall results of segments, images and visual objects belonging to the same document, calculate the document relevance score based on the multi-way recall results contained in each document and the corresponding comprehensive relevance score, select a number of documents with high document relevance scores as candidate documents, and form a candidate document set. The candidate documents include hit segments, hit images and hit visual objects. Step 605: Based on the index-graph mapping relationship obtained in Step 5, convert the hit segments, hit images and hit visual objects in each candidate document into corresponding graph node identifiers to form a set of hit graph nodes corresponding to each candidate document.

[0015] A further technical solution is that step 7 specifically includes the following steps: Step 701: Based on the set of hit segment nodes, hit image nodes, hit visual object nodes, and hit graph nodes in each candidate document obtained in Step 6, the comprehensive relevance score of each recall result in the segment text recall result, the whole image visual recall result, and the visual object recall result is used as the initial relevance score of each corresponding hit node. Step 702: Expanding the neighborhood around the hit node within the same modality includes the following steps: For a hit segment node, expand the text entity nodes, text relationship edges, text hyperedges, and source image associated with the segment node; For a hit image node, expand the visual objects contained in the image node, the visual relationships between objects, and the source segment corresponding to the image node; For a hit visual object node, expand the image node to which the object node belongs, other visual objects that have visual relationships with the object node, and the corresponding visual relationships. Step 703: For text entity nodes that have an alignment relationship with visual object nodes, extend along the entity-object alignment relationship to the corresponding text entity node; extend from the text entity node to the source section of the text entity node, the related text relationship edge, and the text hyperedge containing the text entity node; for the extended text hyperedge, supplement the other member entities associated with the text hyperedge, the role of the member in the composite fact, and the corresponding evidence text. Step 704: Based on the initial relevance score of the hit nodes, propagate the query relevance along text relation edges, text hyperedges, visual relation edges, and entity-object alignment relationships; during the propagation process, set weights according to node type, hyperedge member role, and connection relationship, and combine with the answer type required by the natural language question to increase the priority of nodes related to location, time, value, category, or other target answer types; select relevant nodes, relation edges, and text hyperedges based on the relevance scores after propagation, limit the number of expansion layers and the number of evidence, delete duplicate nodes, low-relevance nodes, and isolated nodes that cannot form effective evidence connections, and obtain the filtered segment nodes, text entity nodes, text relation edges, text hyperedges, image nodes, visual object nodes, visual relation edges, and entity-object alignment relationships; Step 705: Organize the filtered segment nodes, text entity nodes, text relation edges, text hyperedges, image nodes, visual object nodes, visual relation edges, and entity-object alignment relationships into a unified local multimodal evidence subgraph. From the local multimodal evidence subgraph Extract candidate document titles, segmented text, entity descriptions, text hyperedges, overall semantic summaries of images, visual object descriptions, cross-modal correspondences, and source information to form a structured evidence context.

[0016] A further technical solution includes the following steps following step 7: Step 8: Input the structured evidence context output in Step 7, along with the query image and natural language question, into the multimodal large language model to generate the final answer. The final answer includes the question answer, the answer credibility score, the basis for the answer, the content of the evidence used, and the source identifier.

[0017] A further technical solution is as follows: Step 8 specifically includes the following steps: Step 801: Combine the natural language question, query image, and structured evidence context into a multimodal prompt message and input it into the multimodal large language model; the query image is used to identify or distinguish the target object pointed to by the question, and the structured evidence context is used to provide attributes and factual evidence related to the target object; Step 802: The multimodal large language model performs joint reasoning on the query image, natural language question, and structured evidence context to generate the question answer, answer credibility score, answer basis, and the evidence content used, which are then output as the results. Step 803: Perform format parsing and field validation on the output of the multimodal large language model, standardize the answer credibility score to the range of 0 to 1, and associate and save the generated answer with the corresponding candidate document identifier or section identifier; when the existing evidence is insufficient, reduce the answer credibility score and output a conservative answer or insufficient evidence prompt.

[0018] An image encyclopedia question-answering device based on a multimodal hypergraph includes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the corresponding steps in the aforementioned image encyclopedia question-answering method based on a multimodal hypergraph.

[0019] The beneficial effects of adopting the above technical solution are as follows: A method for image encyclopedia question answering based on multimodal hypergraphs includes step 1: performing multimodal document structured parsing on the original multimodal encyclopedia knowledge document to obtain a structured document; and step 2: constructing a document-level text hypergraph based on the structured document. Step 3: Build a document-level visual graph based on the structured document. Step 4: Based on document-level text hypergraph and document-level visual diagrams Constructing a document-level heterogeneous multimodal knowledge graph Step 5: Based on document-level heterogeneous multimodal knowledge graph Step 6: Based on the multimodal retrieval index, the user-input query image, and the natural language question, perform multi-path recall and candidate document aggregation to obtain a candidate document set and a corresponding hit graph node set; Step 7: Based on the candidate document set, the hit graph node set, and the document-level heterogeneous multimodal knowledge graph... Constructing a local multimodal evidence subgraph And the corresponding structured evidence context. By organizing textual facts, images, visual objects, and corresponding relationships in encyclopedic knowledge documents into a unified knowledge structure, relevant evidence is retrieved based on the images and questions provided by the user, forming a complete evidence chain that can be used to generate answers. This evidence chain is then combined with a large language model to output answers to the questions.

[0020] A multimodal hypergraph-based image encyclopedia question-answering device includes a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the corresponding steps in the aforementioned multimodal hypergraph-based image encyclopedia question-answering method. By organizing textual facts, images, visual objects, and corresponding relationships in encyclopedia knowledge documents into a unified knowledge structure, and retrieving relevant evidence based on user-provided images and questions, a complete chain of evidence that can be used to generate answers is formed. Attached Figure Description

[0021] Figure 1 This is a flowchart of Embodiment 1 of the present invention; Figure 2 This is a principle block diagram of Embodiment 3 of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this application or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0024] Example 1:

[0025] like Figure 1 As shown, this invention discloses an image encyclopedia question-answering method based on a multimodal hypergraph, comprising the following steps: Step 1: Multimodal document structured parsing.

[0026] The input raw multimodal encyclopedia knowledge document is subjected to structured parsing, extracting document identifiers, document titles, section numbers, section titles, section text, images, and image sources, as well as the image association mapping relationship between document, section, and image. Section numbers, section titles, and section text form sections, and all sections form a section set. Images include image identifiers, image links, image content, image descriptions, and local cache paths, and all images form an image set. Image sources include image paths, sizes, and processing status on the computer. The image association mapping relationship represents the position of the image within the raw multimodal encyclopedia knowledge document. Based on the document identifier, document title, section set, image set, image source, and image association mapping relationship, a structured document with a unified format is formed.

[0027] As can be seen, the structured document is obtained by simplifying the multimodal encyclopedic knowledge document. The obtained images and their sources are used to support full-image and local target retrieval.

[0028] Input: Original multimodal encyclopedia knowledge document; Processing algorithm: document structured parsing; Output: Structured document.

[0029] In this application, the input set of original multimodal encyclopedia knowledge documents is represented as... In the formula, any structured document It includes text content, image content, and image correspondence.

[0030] For any structured document The structured representation of a document includes: document identifiers. Document title Segmented sets Image collection The image association mapping relationship between document, section, and image is used to determine the document and related section to which each image belongs.

[0031] Any section Includes section number, section title, and section text; any image It includes image identifiers, image links, image descriptions, and local cache paths.

[0032] For ease of description, the following symbols are used in this application, see Table 1.

[0033] Table 1: Technical Parameter Symbol Table

[0034] Step 2: Construct a text hypergraph .

[0035] Each section in a structured document As a basic text processing unit, it performs text knowledge extraction on the segmented text and image sources in the structured document to obtain a set of text entities. Textual Relationship Set and text superedge set To form a document-level text hypergraph .

[0036] The specific division includes the following steps: Step 201: Extract text entities such as species, locations, people, and buildings from the segmented text, and record the text entity name, type, description, and source segment to obtain a set of text entities. .

[0037] Step 202: Identify simple facts from the segmented text that can be fully expressed by two text entities and their semantic relationship, and establish text relationship edges between the corresponding two text entity nodes. Each text relationship edge records the starting entity identifier, target entity identifier, relationship type, relationship description, source segment, and original evidence text, forming a text relationship set. .

[0038] Step 203: Identify higher-order composite facts from the segmented text that cannot be fully expressed by a single binary relation. These higher-order composite facts include facts involving multiple text entities, and facts composed of multiple pieces of information such as text entities, attribute conditions, numerical ranges, units of measurement, or time conditions. For each higher-order composite fact, construct a text hyperedge that simultaneously associates multiple entities involved in the fact, and record the role, fact type, semantic description, source segment, and original evidence text of each member entity, forming a set of text hyperedges. .

[0039] Step 204: Normalize, deduplicate, and merge the names of text entities, text relation edges, and text hyperedges extracted from different sections of the same structured document, while retaining the source section and original evidence text of the text entities, to obtain the text entity source mapping relationship between the text entities and the source section, forming a document-level text hypergraph. .

[0040] When performing knowledge extraction on each segment, the current segment number and original evidence text are recorded for the extracted entities, text relation edges, and text hyperedges. After entity normalization and merging are completed, the segment numbers corresponding to each original entity record before merging are summarized and deduplicated to obtain the mapping relationship between entity numbers and the set of source segment numbers; for text relations and text hyperedges, the mapping relationship between their numbers and the source segment number from which the knowledge unit was extracted is recorded respectively. The source mapping relationship is used for subsequent cross-modal alignment, graph connection, local evidence subgraph expansion, and answer evidence tracing.

[0041] Input: Sectional text and image sources from the structured document output in step 1; Processing algorithms: entity extraction, relation extraction, construction of higher-order fact hyperedges, and normalization and merging of knowledge within documents; Output: Document-level text hypergraph , including text entity sets Textual Relationship Set Text hyperedge set And the mapping relationship of text entity sources.

[0042] Step 3: Construct a visual graph .

[0043] For image collections in structured documents For each image in the document, global image information extraction, visual object detection, and visual relationship extraction are performed one by one. The processing results of all images within the same structured document are then organized into a document-level visual graph. .

[0044] The specific division includes the following steps: Step 301: For each image, generate an overall semantic summary of the image based on the image content and image description information; using the image as an image node, record the image node identifier, image description, overall semantic summary, local image storage path, source document, and source section.

[0045] Step 302: Perform visual object detection on each image to obtain the visual object node identifier, object category, bounding box, detection confidence, and object description of each visual object in the image, forming a visual object node set. Simultaneously, an image-object containment relationship is established between the image node and the visual object node it contains.

[0046] Step 303: For visual objects in the same image with clear semantic relationships, extract the spatial relationships, action relationships, and other semantic relationships between the visual objects, and establish visual relationship edges between the corresponding visual object nodes; each visual relationship edge records the visual object node identifiers at both ends of the relationship, the relationship type, the relationship description, and the source image, forming a visual relationship set. .

[0047] Step 304: Organize the image nodes, visual object nodes, image-object containment relationships, and visual relationship edges corresponding to each image within the same structured document in a unified manner, and retain the image source mapping relationship between each node and relationship and the source document, source section, and source image to form a document-level visual graph. The local image storage path and the visual object bounding box are used in step 5 to extract overall image features and object region features and to build a visual index.

[0048] Input: Images, image descriptions, and image sources from the structured document output in step 1; Processing algorithms: overall image semantic understanding, visual object detection, and visual relation extraction; Output: Document-level visual diagram This includes image nodes and sets of visual object nodes. Image-object containment relationship, visual relationship set Mapping relationship between image source and image source.

[0049] Step 4: Construct a document-level multimodal knowledge graph .

[0050] The document-level text hypergraph output in step 2 And the document-level visual diagram output in step 3 Based on this, image nodes, visual object nodes, and visual relationship edges are connected to the text hypergraph. Furthermore, it establishes cross-modal alignment relationships between text entity nodes and visual object nodes, forming a document-level heterogeneous multimodal knowledge graph. .

[0051] Specifically, it includes the following sub-steps: Step 401: Add the image nodes, visual object nodes, image-object containment relationships, and visual relationship edges obtained in Step 3 to the document-level graph structure; based on the document-section-image correspondence recorded in Step 1, establish source association relationships between section nodes and corresponding image nodes.

[0052] Step 402: For each visual object, select semantically relevant candidate text entities from the text entity set obtained in Step 2 based on its object category, object description, image description, overall semantic summary of the image, and source segmentation, to form an entity-object alignment candidate set.

[0053] Step 403: Perform semantic matching on candidate text entities and visual objects, and make a judgment based on their semantic similarity, consistency of source segmentation, and image description information; retain candidate combinations that meet the matching conditions, and establish cross-modal alignment relationships between text entity nodes and visual object nodes. Each alignment relationship records the text entity identifier, visual object node identifier, associated image node identifier, matching score, judgment basis, and source information.

[0054] Step 404: Organize the segment nodes, text entity nodes, text relation edges, text hyperedges, image nodes, visual object nodes, visual relation edges, and entity-object alignment relationships in a unified manner to form a document-level multimodal knowledge graph. The graph preserves the mapping relationships between each node and relationship and the source document, source section, and source image, for use in subsequent index construction and expansion of local evidence subgraphs.

[0055] Input: The document-level text hypergraph output from step 2 And the document-level visual diagram output in step 3 ; Processing algorithms: cross-modal alignment at the entity level, relation level, and higher-order fact level; Output: Document-level heterogeneous multimodal knowledge graph This includes text-side nodes and relationships, visual-side nodes and relationships, and entity-object cross-modal alignment relationships.

[0056] Step 5: Build a multimodal retrieval index.

[0057] For the document-level heterogeneous multimodal knowledge graph output in step 4 The system constructs segmented text retrieval indexes, whole-image visual retrieval indexes, and visual object retrieval indexes, and establishes index-graph mapping relationships between each index item and graph nodes and their respective documents, forming a multimodal retrieval index.

[0058] Specifically, it includes the following sub-steps: Step 501: Divide the segment nodes in the multimodal knowledge graph into segmented text retrieval units. Each segmented text retrieval unit includes at least the document identifier, segment identifier, segment title, segment text, and corresponding graph node identifier.

[0059] Step 502: Generate text representations for each segmented text retrieval unit and perform vector encoding to obtain segmented text features; establish a segmented text retrieval index based on the segmented text features, which is used to recall relevant segments and their corresponding documents based on natural language questions.

[0060] Step 503: Read the corresponding image according to the local storage path of the image recorded by the image node, extract the visual features of the whole image, and associate the visual features of the whole image with the image node identifier, the document identifier, and the source section identifier to establish a whole image visual retrieval index.

[0061] Step 504: Based on the source image and bounding box recorded by the visual object node, extract the object region from the corresponding image, extract the visual features of the object region, and associate the visual features of the object region with the visual object node identifier, image node identifier, document identifier, and source section identifier to establish a visual object retrieval index.

[0062] Step 505: Record the corresponding graph node identifier and the document identifier for each index item to obtain the index-graph mapping relationship, so that subsequent search results can return the corresponding segment node, image node or visual object node, and can enter the heterogeneous multimodal knowledge graph of the corresponding document for local evidence expansion.

[0063] Among them, text entity nodes, text relation edges, text hyperedges, visual relation edges, and entity-object alignment relations are preserved in the document-level heterogeneous multimodal knowledge graph for subsequent local evidence subgraph construction and graph structure propagation, without requiring the establishment of separate global retrieval indexes.

[0064] Input: The document-level heterogeneous multimodal knowledge graph output from step 4 , and the local image file pointed to by the image node; Processing algorithms: segmented text vector encoding, whole-image visual feature encoding, object region extraction and visual feature encoding, and index-map mapping construction; Output: Multimodal retrieval index, including segmented text retrieval index, whole-image visual retrieval index, visual object retrieval index, and index-graph mapping relationship between each index item and graph node and its corresponding document.

[0065] Step 6: Multi-path recall and candidate document aggregation based on question and query image.

[0066] The query includes user-provided query images and natural language questions. The system extracts features from the question text, the overall query image, and visual objects within the query image, and retrieves them in the corresponding search index established in step 5. Then, it aggregates the search results according to the documents to which they belong, obtaining candidate documents and the corresponding hit graph nodes for each candidate document.

[0067] Specifically, it includes the following sub-steps: Step 601: Perform text vector encoding on the natural language question to obtain the question text features; perform overall visual feature encoding on the query image to obtain the query image features; simultaneously perform object detection on the query image, extract the object region based on the detected bounding box, and extract the visual features of the object region respectively.

[0068] Step 602: Retrieve relevant sections in the segmented text retrieval index using the features of the question text; retrieve similar images in the whole-image visual retrieval index using the features of the query whole image; retrieve similar visual objects in the visual object retrieval index using the visual features of each object region. Each retrieval result should at least record the retrieval type, similarity score, graph node identifier, and document identifier.

[0069] Step 603: Normalize the similarity scores of the segmented text recall results, the whole-image visual recall results, and the visual object recall results, and then perform weighted fusion according to the corresponding retrieval type to obtain the comprehensive relevance score of each recall result. The weights of the different retrieval types can be preset according to actual application needs.

[0070] Step 604: Group the recall results according to the document identifier of each recall result record, merge the recall results of sections, images and visual objects belonging to the same document, calculate the document relevance score based on the multi-way recall results contained in each document and their comprehensive relevance score, and select several documents with high document relevance scores to form a candidate document set.

[0071] Step 605: Based on the index-graph mapping relationship established in Step 5, convert the hit segments, hit images and hit visual objects in each candidate document into corresponding graph node identifiers to form a set of hit graph nodes corresponding to each candidate document, which serves as the starting point for expanding the local evidence subgraph in Step 7.

[0072] Input: The segmented text retrieval index, whole-image visual retrieval index, visual object retrieval index and index-graph mapping relationship output from step 5, as well as the query image and natural language question input by the user; Processing algorithms: Question text encoding, query full-image encoding, query object detection and region encoding, multi-path vector recall, recall score fusion and candidate document aggregation; Output: a set of candidate documents, the recall results for each candidate document, and the corresponding set of hit graph nodes.

[0073] Step 7: Construct a query-related local multimodal evidence subgraph .

[0074] For the candidate documents obtained in step 6, read the corresponding document-level heterogeneous multimodal knowledge graph constructed in step 4. Using the hit segment node, hit image node, and hit visual object node obtained in step 6 as the starting point for expansion, a query-related local expansion is performed along the text relation edges, text hyperedges, visual relation edges, and entity-object alignment relationships in the graph, forming a local multimodal evidence subgraph. .

[0075] Specifically, it includes the following sub-steps: Step 701: For each candidate document, based on the set of hit graph nodes output in Step 6, determine the hit segment nodes, hit image nodes, and hit visual object nodes, and use the corresponding recall scores as the initial relevance scores for each hit node.

[0076] Step 702: Expand the neighborhood around the hit node within the same modality: (1) For a hit segment node, expand the text entity node, text relation edge, text hyperedge and source image associated with the segment node; (2) For a hit image node, expand the visual objects contained in the image node, the visual relationships between the objects, and the source segment corresponding to the image node; (3) For a hit visual object node, expand the image node to which the object node belongs, other visual objects that have a visual relationship with it, and the corresponding visual relationship.

[0077] Step 703: For text entity nodes that have an alignment relationship with visual object nodes, extend along the entity-object alignment relationship to the corresponding text entity node; then extend from the text entity node to its source section, related text relationship edges, and text hyperedges containing the text entity node. For the extended text hyperedges, further supplement the information with other member entities associated with the text hyperedge, the role of the members in the composite facts, and the corresponding evidence text.

[0078] Step 704: Based on the initial relevance score of the hit nodes, propagate query relevance along text relation edges, text hyperedges, visual relation edges, and entity-object alignment relationships. During propagation, different weights are set according to node type, hyperedge member role, and connection relationship, and nodes related to location, time, value, category, or other target answer types are given higher priority based on the answer type required by the natural language question.

[0079] Based on the relevance score after propagation, relevant nodes, relation edges, and superedges are selected, the number of expansion layers and the amount of evidence are limited, and duplicate nodes, low-relevance nodes, and isolated nodes that cannot form effective evidence connections are deleted.

[0080] Step 705: Organize the filtered segment nodes, text entity nodes, text relation edges, text hyperedges, image nodes, visual object nodes, visual relation edges, and entity-object alignment relationships into a unified local multimodal evidence subgraph. Simultaneously, from the aforementioned local multimodal evidence subgraph Extract candidate document titles, segmented text, entity descriptions, text hyperedges, overall semantic summaries of images, visual object descriptions, cross-modal correspondences, and source information to form a structured evidence context for subsequent answer generation modules to call.

[0081] Inputs: The candidate document set output from step 6, the set of hit graph nodes corresponding to each candidate document, and the document-level heterogeneous multimodal knowledge graph constructed in step 4. ; Processing algorithm: Starting from the hit retrieval unit, local expansion is performed in the multimodal knowledge graph corresponding to the candidate document; Output: Relevant local multimodal evidence subgraphs And its corresponding structured evidence context.

[0082] Step 8: Organize the evidence and generate an answer.

[0083] The structured evidence context output from step 7 is then input into the multimodal large language model along with the query image and the natural language question to generate the final answer.

[0084] Specifically, the steps include the following: Step 801: Combine the natural language question, query image, and structured evidence context into a multimodal prompt message, and input it into the multimodal large language model. The query image is used to identify or distinguish the target object pointed to by the question, and the structured evidence context is used to provide attributes and factual basis related to the target object.

[0085] Step 802: The multimodal large language model performs joint reasoning on the query image, natural language question, and structured evidence context to generate the question answer, answer credibility score, answer basis, and the evidence content used.

[0086] Step 803: Perform format parsing and field validation on the output of the multimodal large language model, standardize the answer credibility score to the range of 0 to 1, and associate and save the generated answer with the corresponding candidate document identifier or section identifier; when the existing evidence is insufficient, reduce the answer credibility score and output a conservative answer or insufficient evidence prompt.

[0087] Input: The local multimodal evidence subgraph output from step 7, and the query image and natural language question input by the user; Processing algorithms: structured evidence organization, multimodal cue construction, multimodal large language model reasoning, and output result verification; Output: Question answer, answer credibility score, basis for answer, content of evidence used and its source identifier.

[0088] In Example 1, the encyclopedia document containing text and images is first structured. A text hypergraph capable of storing complete and complex facts is constructed on the text side, and a visual graph containing images, visual objects, and visual relationships is constructed on the image side. Then, a correspondence between textual knowledge and image content is established, forming a document-level multimodal hypergraph. When a user provides a query image and asks a question, the system first recalls potentially relevant candidate documents, then retrieves and organizes evidence from the multimodal hypergraph of the candidate documents, constructing a local evidence subgraph composed of the query image, visual objects, textual entities, complete facts, and source sections. Finally, an answer is generated based on the evidence.

[0089] Compared with the prior art, Example 1 has the following beneficial technical effects.

[0090] 1. By introducing hypergraph representation on the text side, it is possible to more completely express multi-entity composite facts in document sections, avoid complex knowledge being simply broken down into a few isolated binary relations, and improve the knowledge representation capability on the text side.

[0091] 2. By establishing cross-modal alignment relationships at the entity, relation, and higher-order fact levels, we can enhance the fine-grained fusion capability between textual knowledge and visual evidence, and improve the quality of joint text-image modeling.

[0092] 3. By dividing heterogeneous knowledge graphs into multi-type graph substructure units and establishing a unified index, redundant noise generated when directly searching for unstructured documents can be reduced, and the structuring degree of search results can be improved.

[0093] 4. By performing document aggregation and local graph expansion around the graph substructure of the query hit, it is possible to construct a compact and information-complete local multimodal evidence subgraph, which is beneficial for the subsequent answer generation module to utilize key evidence.

[0094] 5. It is suitable for knowledge-intensive visual question answering, multimodal knowledge retrieval, and multimodal knowledge organization scenarios, and has good engineering applicability and scalability.

[0095] The technical contributions are explained below.

[0096] The textual facts, images, visual objects, and their corresponding relationships in the encyclopedia documents are organized into a unified knowledge structure, and relevant evidence is retrieved based on the images and questions provided by the user to form a complete chain of evidence that can be used to generate answers.

[0097] The technical contributions of this application are as follows: 1. Introducing a high-order semantic hypergraph modeling mechanism on the text side, which not only extracts text entities and binary relations, but also organizes composite facts involving multiple entities into text hyperedges, thereby enhancing the high-order semantic expression capability of text knowledge; 2. In the cross-modal fusion stage, not only is the alignment relationship between text entities and visual objects established, but the structural correspondence between high-order facts in the text and the image as a whole, object groups, or visual relation groups is further established; 3. In the retrieval stage, instead of directly recalling unstructured document blocks, a unified index is established for multiple types of substructure units in heterogeneous knowledge graphs, and a query-oriented local multimodal evidence subgraph is constructed through document aggregation and local graph expansion.

[0098] The technical concept is explained below.

[0099] This addresses a type of question that cannot be answered directly from an image alone and requires consulting external encyclopedic resources. For example, a user uploads an image of an animal and asks about its classification, distribution area, or weight. The image only provides clues about the animal's appearance; the actual answer is usually found in a section of the encyclopedia document. Therefore, the system must not only identify the target in the image but also establish a complete chain of evidence, from the query image to the encyclopedia entity, from the encyclopedia entity to relevant facts, and from the relevant facts to the specific answer and source section.

[0100] Existing knowledge graphs typically use a single relation to connect two entities. When a fact simultaneously includes a subject, attribute, value, unit, and limiting condition, it needs to be split into multiple relations, which can easily lead to incomplete factual content. For example, the statement that a female animal weighs between 20 and 30 kilograms includes the animal, sex, weight attribute, value range, and unit of measurement. If these elements are stored separately, a search might only find the weight value, omitting the limiting condition of "female."

[0101] Based on the aforementioned problems, the first concept of this application is to move beyond simply using ordinary binary relations to represent textual knowledge. Instead, it employs textual hyperedges to connect multiple components of a complete fact and record the role of each component within the fact. In this way, the subject, attributes, attribute values, units, limiting conditions, and sources can be stored as a whole, preventing complex facts from being fragmented during the knowledge extraction stage.

[0102] The second concept of this application is that the connection between text and image should not be limited to the level of one text entity corresponding to one visual object. In addition to entity-level correspondences, correspondences are also established between textual and visual relationships, as well as between complete textual facts and the entire image, multiple visual objects, or groups of visual relationships. This allows for a clear indication of which elements in the image support a textual fact.

[0103] The third concept of this application is to use a phased approach to find evidence for the answer. First, based on the query image and natural language question provided by the user, potentially relevant candidate documents are recalled from the entire knowledge base. Then, in the multimodal hypergraph corresponding to the candidate documents, evidence is searched along entity relations, fact hyperedges, and cross-modal alignment relations, starting from the hit image, visual object, text entity, or fact hyperedge.

[0104] During graph-based retrieval, the retrieval priority of different fact members is adjusted according to the type of answer required by the user's question. For example, for classification questions, classification relationships and classification entities are searched first; for numerical questions, attributes, numerical values, units of measurement, and limiting conditions are searched first; and for location questions, location entities and location relationships are searched first. Simultaneously, evidence propagation is controlled based on the credibility of the correspondence between text and images, reducing the impact of erroneous text-image matching on the retrieval results.

[0105] The fourth concept of this application is: instead of directly feeding the retrieved document fragments to the answer generation module, it supplements the selected evidence with relevant entities, attributes, values, fact hyperedges, images, visual objects, and source segments to construct a compact local multimodal evidence subgraph. It restricts evidence with highly repetitive content while retaining complementary textual evidence, visual evidence, and image-text correspondence evidence.

[0106] Finally, the local multimodal evidence subgraphs are organized into a structured evidence context, and an answer is generated based on the query image, user question, and structured evidence. The answer's confidence level, supporting evidence, and source are also output. When the existing evidence is insufficient to support a definitive answer, the system outputs an insufficient evidence warning to avoid relying solely on the model's memory for guesswork.

[0107] The overall concept of this application can be summarized as follows: using text hyperedges to completely preserve encyclopedic facts, using multi-level cross-modal alignment to connect textual facts and image content, and then using a process of candidate document recall, hypergraph evidence retrieval, local evidence organization, and evidence-based answer generation to establish a traceable evidence chain from the target object in the query image to the external encyclopedic answer and its source.

[0108] Example 2:

[0109] This invention discloses an image encyclopedia question-answering method based on a multimodal hypergraph, comprising the following steps: This embodiment uses the InfoSeek dataset and its accompanying multimodal Wikipedia knowledge base for verification. The knowledge base contains approximately 100,000 Wikipedia documents, each of which may include a title, section text, an image, and image descriptions; the query data includes the query image, a natural language question, and a reference answer. InfoSeek categorizes queries into unseen questions and unseen entities, and calculates the overall question-and-answer accuracy according to its prescribed evaluation method.

[0110] The first stage, the construction of a multimodal knowledge graph, is described below.

[0111] First, the multimodal Wikipedia documents in the InfoSeek knowledge base are structured and parsed to extract document titles, section titles, section text, image links, image descriptions and their source correspondences, and unique identifiers are assigned to documents, sections and images.

[0112] On the text side, document segments are used as the basic processing unit to extract entities, entity relationships, and higher-order facts from the segments. For composite facts composed of multiple entities, attributes, or limiting conditions, they are not broken down into independent binary relationships, but are saved as a whole in the form of hyperedges. The extraction results from different segments within the same document are normalized in name, duplicates are merged, and source associations are performed to form a document-level text hypergraph.

[0113] On the image side, a holistic semantic description is generated for the images in the document, and visual objects, object locations, object categories, and relationships between objects are identified to form a document-level visual graph. Subsequently, text entities are matched with visual objects, text relationships with visual relationships, and higher-order text facts with the image as a whole or object groups, merging the text hypergraph and visual graph into a document-level multimodal knowledge graph.

[0114] In the full processing results of this embodiment, a total of 91,321 documents were processed to construct a knowledge map, resulting in 747,101 segmentation units, 2,033,477 entity units, 668,454 text hyperedges, 333,310 image units, 1,498,607 visual object units, and 678,893 cross-modal alignment units, providing a knowledge foundation for subsequent structured retrieval.

[0115] The second stage, retrieval and answer generation, is explained below.

[0116] Based on the multimodal knowledge graph constructed in the first phase, retrieval representations are generated and indexes are established for segments, entities, hyperedges, images, visual objects, and cross-modal alignment relationships.

[0117] For a query image and its natural language question, the semantics of the question and image features are first jointly extracted, and relevant text units, visual units, and cross-modal alignment units are retrieved from the index. Then, the units are aggregated according to the documents to which they belong to determine candidate Wikipedia documents. Finally, local expansion is performed around the hit entities, hyperedges, images, and alignment relationships, and the relevant segmented text, entity descriptions, composite facts, image summaries, and visual objects are organized into a local multimodal evidence subgraph, and the answer is generated based on the evidence subgraph.

[0118] For example, for the query: "What is the IUCN protection status of the animal in the picture?", the system recalls documents and visual objects related to the African wild dog based on the query image, and retrieves textual evidence from the candidate documents that the African wild dog has been listed as an endangered species since 1990. The system organizes the animal object in the query image, the text entity "African wild dog", the protection status fact and its source into local evidence subgraphs, and finally outputs the answer as endangered, while retaining the corresponding evidence source.

[0119] This embodiment performs question-answering validation on 37,991 query samples in the InfoSeek validation set. String and time-related questions are evaluated using question-answering accuracy, while numerical questions are evaluated using accuracy with relaxed matching conditions. The overall accuracy is calculated based on the results of two subsets: unseen questions and unseen entities, as shown in Table 2.

[0120] Table 2: Comparison of Question-Answer Results for Different Methods on the InfoSeek Dataset

[0121] As shown in Table 2, the method described in this application achieves an overall accuracy of 48.8% on InfoSeek, which is 8.3 percentage points higher than the multimodal knowledge graph retrieval method mKG-RAG. These results demonstrate that this application can complete the entire processing from multimodal knowledge document parsing, text hypergraph and visual graph construction, cross-modal fusion, graph substructure retrieval to answer generation, and is engineering-feasible.

[0122] Example 3:

[0123] like Figure 2 As shown, this invention discloses an image encyclopedia question-answering device based on a multimodal hypergraph, including a multimodal document parsing module, a text hypergraph construction module, a visual graph construction module, a document-level multimodal knowledge graph construction module, a multimodal retrieval index construction module, a multi-path recall and candidate document aggregation module, a local multimodal evidence subgraph construction module, and an answer generation module.

[0124] The multimodal document parsing module is used to obtain structured documents from the original multimodal encyclopedia knowledge documents through document structure parsing. The structured document includes document identifier, document title, section set, image set, and image association mapping relationship between document-section-image. The section set includes multiple section nodes, and the image set includes multiple images.

[0125] The steps of document structured parsing include extracting document identifiers, document titles, section numbers, section titles, section text, and images from the original multimodal encyclopedia knowledge document, as well as the image association mapping relationship between document, section, and image. Section numbers, section titles, and section text form sections, with each section corresponding to a section node, and all sections form a section set. Images include image identifiers, image links, image content, image descriptions, and local cache paths, and all images form an image set. A structured document is then formed based on the document identifier, document title, section set, image set, and image association mapping relationship.

[0126] The text hypergraph module is used to extract text entities, text relation edges, and text hyperedges based on segmented text extraction. The text entity set includes multiple text entity nodes, each containing a source segment identifier. The text relation edge set includes multiple text relation edges, each containing a source segment and original evidence text. The text hyperedge set includes multiple text hyperedges. The module performs name normalization, deduplication, and merging on the text entity nodes, text relation edges, and text hyperedges, retaining the source segment and original evidence text of the text entity nodes. This yields the text entity source mapping relationship between text entity nodes and source segments, forming a document-level text hypergraph. .

[0127] The specific division includes the following steps: extracting text entities from the segmented text to form a text entity set. Text entities include categories and information. Each text entity corresponds to a text entity node. The categories of text entities include species, locations, people, and buildings. The information of a text entity includes the entity name, type, description, and source section. The source section includes a source section identifier. Relationships are extracted from the segmented text to obtain the text relation edges between every two text entities, forming a text relation set. Textual relation edges include the starting entity identifier, the target entity identifier, the relation type, the relation description, the source segment, and the original evidence text; higher-order composite facts are identified from the segmented text, and a textual hyperedge is constructed based on each higher-order composite fact, forming a set of textual hyperedges. Higher-order composite facts include facts involving multiple entities. Each fact includes the entity itself, attribute conditions, numerical range, unit of measurement, and time condition. Text hyperedges include the role of each entity, fact type, semantic description, source section, and original evidence text. For each section of the same structured document, the text entities, text relation edges, and text hyperedges are normalized, deduplicated, and merged, retaining the source section and original evidence text of the text entities. This yields the text entity source mapping relationship between text entities and source sections, forming a document-level text hypergraph. .

[0128] The visual graph construction module performs image-wide semantic understanding, visual object detection, and visual relation extraction on each image in the same structured document's image set. This yields image nodes, visual object nodes, image-object containment relationships, and visual relation edges between visual object nodes. It also obtains the image source mapping relationship between each image node and its corresponding visual object node's source document, source section, and source image, thus forming a document-level visual graph. Image nodes include image node identifier, image description, overall semantic summary, local image storage path, source document, and source section; visual object nodes include visual object node identifier, object category, source image, bounding box, detection confidence, and object description.

[0129] The specific partitioning includes the following steps: For each image in the image set, all image nodes are obtained through overall semantic understanding. Each image node includes an image node identifier, image description, overall semantic summary, local image storage path, source document, and source section. The step of obtaining the overall semantic summary includes generating an overall semantic summary for each image based on its content and description. For each image in the image set, all visual object nodes are obtained through visual object detection, forming a visual object node set. The process involves establishing image-object inclusion relationships between image nodes and the visual object nodes they contain. Each visual object node includes its visual object node identifier, object category, source image, bounding box, detection confidence score, and object description. Based on visual objects with clear semantic connections within the same image, spatial, action, and semantic relationships between them are extracted, and visual relationship edges are established between the corresponding visual object nodes. Each visual relationship edge includes the visual object node identifiers at both ends of the relationship, relationship type, relationship description, and source image, forming a set of visual relationships. Obtain the image source mapping relationship between each image node and its corresponding visual object node, including the source document, source section, and source image. Based on the image node, visual object node, image-object containment relationship, visual relationship edge, and image source mapping relationship corresponding to each image within the same structured document, form a document-level visual graph. .

[0130] A module for constructing document-level multimodal knowledge graphs is used to integrate document-level visual graphs. Image nodes, visual object nodes, and visual relationship edges are integrated into a document-level text hypergraph. By performing cross-modal alignment between text entity nodes and visual object nodes, the entity-object cross-modal alignment relationship between text entity nodes and visual object nodes is obtained, forming a document-level heterogeneous multimodal knowledge graph. Document-level heterogeneous multimodal knowledge graph This includes segment nodes, text entity nodes, text relation edges, text hyperedges, image association mapping relationships, image nodes, visual object nodes, visual relation edges, entity-object alignment relationships, and image source mapping relationships.

[0131] The specific partitioning includes the following steps: Image nodes, visual object nodes, image-object inclusion relationships, and visual relationship edges are added to the document-level graph structure; based on the document-section-image image association mapping relationship, source association relationships are established between section nodes and corresponding image nodes; based on the object category and description of each visual object node, as well as the corresponding image description, overall image semantic summary, and source section, semantically relevant candidate text entities are selected from the text entity set to form an entity-object alignment candidate set; combining the semantic similarity between candidate text entities and visual object nodes, source section consistency, and image description, semantic matching is performed on candidate text entities and visual object nodes to obtain candidate combinations, establishing cross-modal alignment relationships between obtained text entity nodes and visual object nodes, i.e., entity-object alignment relationships; each cross-modal alignment relationship includes a text entity identifier, a visual object node identifier, an associated image node identifier, a matching score, a judgment criterion, and source information; based on section nodes, text entity nodes, text relationship edges, text hyperedges, image association mapping relationships, image nodes, visual object nodes, visual relationship edges, entity-object alignment relationships, and image source mapping relationships, a document-level multimodal knowledge graph is formed. Section nodes, text entity nodes, text relation edges, and text super edges form text side nodes and relations, while image nodes, visual object nodes, and visual relation edges form visual side nodes and relations.

[0132] Construct a multimodal retrieval index module for use with document-level heterogeneous multimodal knowledge graphs. The process involves segmented text vector encoding, whole-image visual feature encoding, object region extraction and visual feature encoding, and index-graph mapping construction. This results in segmented text retrieval indexes, whole-image visual retrieval indexes, visual object retrieval indexes, and index-graph mapping relationships between each index item and graph node and its corresponding document, forming a multimodal retrieval index.

[0133] The specific partitioning includes the following steps: dividing the segment nodes in the multimodal knowledge graph into segment text retrieval units, each segment text retrieval unit including its own document identifier, segment identifier, segment title, segment text, and corresponding graph node identifier; generating a text representation for each segment text retrieval unit and performing vector encoding to obtain segment text features; establishing a segment text retrieval index based on the segment text features, used to recall relevant segments and their respective documents according to natural language questions; reading the corresponding image according to the local image storage path of the image node, extracting the overall image visual features, and combining the overall image visual features with the image node identifier, its own document identifier, and its source. By associating segment identifiers, a visual retrieval index for the entire image is established. Based on the source image and bounding box of the visual object node, the object region is extracted from the corresponding image, and the visual features of the object region are extracted. The visual features of the object region are then associated with the visual object node identifier, image node identifier, document identifier, and source segment identifier to establish a visual object retrieval index. For each index item, the corresponding graph node identifier and document identifier are recorded to obtain the index-graph mapping relationship, so that subsequent retrieval results can return the corresponding segment node, image node, or visual object node, and can enter the heterogeneous multimodal knowledge graph of the corresponding document for local evidence expansion.

[0134] A multi-path recall and candidate document aggregation module is used to obtain user-input queries, which include query images and natural language questions. Question text features are extracted from the natural language questions, and query whole-image features and object region visual features are extracted from the query images. The question text features are used to perform recall in the segmented text retrieval index to obtain segmented text recall results; the query whole-image features are used to perform recall in the whole-image visual retrieval index to obtain whole-image visual recall results; and the object region visual features are used to perform recall in the visual object retrieval index to obtain visual object recall results. The segmented text recall results, whole-image visual recall results, and visual object recall results are normalized, weighted, and aggregated to obtain several documents with high document relevance scores as candidate documents, forming a candidate document set. Candidate documents include those that hit segments, hit images, and hit visual objects. Based on the index-graph mapping relationship and candidate document transformation, the graph node identifiers of the candidate documents are obtained, forming a set of hit graph nodes for each candidate document.

[0135] The specific segmentation includes the following steps: obtaining the user-input query image and natural language question; performing text vector encoding on the natural language question to obtain question text features; performing overall visual feature encoding on the query image to obtain query whole image features; performing object detection on the query image, extracting object regions based on the detected bounding boxes, and extracting visual features of the object regions; using the question text features to recall relevant sections in the segmented text retrieval index to obtain segmented text recall results; using the query whole image features to recall similar images in the whole image visual retrieval index to obtain whole image visual recall results; using the visual features of each object region to recall similar visual objects in the visual object retrieval index to obtain visual object recall results; each recall result includes retrieval type, similarity score, graph node identifier, and document identifier; and analyzing the segmented text recall results, whole image visual recall results, and visual... The similarity scores of the object recall results are normalized and weighted and fused according to the corresponding retrieval type to obtain the comprehensive relevance score of each recall result; the weights of different retrieval types are pre-set according to actual application needs; the recall results are grouped according to the document identifier to which each recall result belongs, and the recall results of segments, images and visual objects belonging to the same document are merged. The document relevance score is calculated based on the multiple recall results contained in each document and the corresponding comprehensive relevance score. Several documents with high document relevance scores are selected as candidate documents to form a candidate document set. The candidate documents include hit segments, hit images and hit visual objects; according to the index-map mapping relationship, the hit segments, hit images and hit visual objects in each candidate document are converted into corresponding map node identifiers to form a hit map node set corresponding to each candidate document.

[0136] Constructing a local multimodal evidence subgraph module for use with corresponding document-level heterogeneous multimodal knowledge graphs. Starting with the hit segment nodes, hit image nodes, and hit visual object nodes in the candidate documents, the expansion proceeds along the document-level heterogeneous multimodal knowledge graph. The text relation edges, text hyperedges, visual relation edges, and entity-object alignment relations in the graph are locally expanded in relation to the query to form a local multimodal evidence subgraph. This allows us to obtain the context of structured evidence.

[0137] The specific partitioning includes the following steps: Based on the set of hit segment nodes, hit image nodes, hit visual object nodes, and hit graph nodes in each candidate document, the comprehensive relevance score of each recall result in the segment text recall result, the whole image visual recall result, and the visual object recall result is used as the initial relevance score of each corresponding hit node; the neighborhood expansion around the hit node within the same modality includes the following steps: For hit segment nodes, expand the text entity nodes, text relationship edges, text hyperedges, and source images associated with the segment node; for hit image nodes, expand the visual objects contained in the image node, the visual relationships between objects, and the source segment corresponding to the image node; for hit visual object nodes, expand the image node to which the object node belongs, other visual objects with visual relationships with the object node, and the corresponding visual relationships; for text entity nodes with alignment relationships with visual object nodes, expand along the entity-object alignment relationship to the corresponding text entity node; expand from the text entity node to the source segment, related text relationship edges, and text containing the text entity node. Hyperedges; for the expanded text hyperedges, supplement the other member entities associated with the text hyperedge, the role of the members in the composite facts, and the corresponding evidence text; based on the initial relevance score of the hit node, propagate the query relevance along text relation edges, text hyperedges, visual relation edges, and entity-object alignment relationships; during the propagation process, set weights according to node type, hyperedge member role, and connection relationship, and combine with the answer type required by the natural language question to increase the priority of nodes related to location, time, value, category, or other target answer types; select relevant nodes, relation edges, and text hyperedges according to the relevance score after propagation, limit the number of expansion layers and the number of evidence, delete duplicate nodes, low-relevance nodes, and isolated nodes that cannot form effective evidence connections, and obtain the filtered segment nodes, text entity nodes, text relation edges, text hyperedges, image nodes, visual object nodes, visual relation edges, and entity-object alignment relationships; organize the filtered segment nodes, text entity nodes, text relation edges, text hyperedges, image nodes, visual object nodes, visual relation edges, and entity-object alignment relationships in a unified manner to form a local multimodal evidence subgraph. From the local multimodal evidence subgraph Extract candidate document titles, segmented text, entity descriptions, text hyperedges, overall semantic summaries of images, visual object descriptions, cross-modal correspondences, and source information to form a structured evidence context.

[0138] The answer generation module is used to input structured evidence context, query image, and natural language question into a multimodal large language model to generate the final answer.

[0139] The specific steps include: combining natural language questions, query images, and structured evidence context into multimodal prompts, and inputting this information into a multimodal large language model; using the query image to identify or distinguish the target object pointed to by the question, and using the structured evidence context to provide attributes and factual basis related to the target object; performing joint reasoning on the query image, natural language questions, and structured evidence context by the multimodal large language model to generate the question answer, answer credibility score, answer basis, and the evidence content used, which are then output as the result; performing format parsing and field validation on the output of the multimodal large language model, standardizing the answer credibility score to the range of 0 to 1, and associating and saving the generated answer with the corresponding candidate document identifier or section identifier; when the existing evidence is insufficient, reducing the answer credibility score and outputting a conservative answer or insufficient evidence prompt.

[0140] Example 4:

[0141] This invention discloses an image encyclopedia question-answering device based on a multimodal hypergraph, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory and processor form an electronic terminal. When the processor executes the computer program, it implements the steps of Embodiment 1. The computer program includes the multimodal document parsing module, text hypergraph construction module, visual graph construction module, document-level multimodal knowledge graph construction module, multimodal retrieval index construction module, multi-path recall and candidate document aggregation module, local multimodal evidence subgraph construction module, and answer generation module as described in Embodiment 3.

[0142] Example 5:

[0143] This invention discloses a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps in Embodiment 1. The computer program includes the multimodal document parsing module, text hypergraph construction module, visual graph construction module, document-level multimodal knowledge graph construction module, multimodal retrieval index construction module, multi-path recall and candidate document aggregation module, local multimodal evidence subgraph construction module, and answer generation module as described in Embodiment 3.

Claims

1. A method for answering image encyclopedia questions based on multimodal hypergraphs, characterized in that: The process includes the following steps: Step 1: Perform multimodal document structured parsing on the original multimodal encyclopedia knowledge document to obtain a structured document; Step 2: Construct a document-level text hypergraph based on the structured document. Step 3: Build a document-level visual graph based on the structured document. Step 4: Based on document-level text hypergraph and document-level visual diagrams Constructing a document-level heterogeneous multimodal knowledge graph Step 5: Based on document-level heterogeneous multimodal knowledge graph Step 6: Based on the multimodal retrieval index, the user-input query image, and the natural language question, perform multi-path recall and candidate document aggregation to obtain a candidate document set and a corresponding hit graph node set; Step 7: Based on the candidate document set, the hit graph node set, and the document-level heterogeneous multimodal knowledge graph... Constructing a local multimodal evidence subgraph And the corresponding structured evidence context.

2. The image encyclopedia question-answering method based on multimodal hypergraphs according to claim 1, characterized in that: In step 1, the multimodal document structured parsing process includes extracting document identifiers, document titles, section numbers, section titles, section text, and images from the original multimodal encyclopedia knowledge document, as well as the document-section-image image association mapping relationship. Section numbers, section titles, and section text form sections, with each section corresponding to a section node, and all sections form a section set. Images include image identifiers, image links, image content, image descriptions, and local cache paths, and all images form an image set. A structured document is formed based on the document identifier, document title, section set, image set, and image association mapping relationship. The structured document includes document identifiers, document titles, section sets, image sets, and the document-section-image image association mapping relationship. The section set includes multiple section nodes, and the image set includes multiple images.

3. The image encyclopedia question-answering method based on a multimodal hypergraph according to claim 1, characterized in that: In step 2, a document-level text hypergraph is obtained based on the structured document. The steps include extracting a set of text entities, a set of text relation edges, and a set of text hyperedges based on segmented text extraction. The text entity set includes multiple text entity nodes, each containing a source segment identifier. The text relation edge set includes multiple text relation edges, each containing a source segment and the original evidence text. The text hyperedge set includes multiple text hyperedges. The text entity nodes, text relation edges, and text hyperedges are then normalized, deduplicated, and merged. The source segments and original evidence text of the text entity nodes are retained, resulting in a text entity source mapping relationship between text entity nodes and source segments, forming a document-level text hypergraph. .

4. The image encyclopedia question-answering method based on a multimodal hypergraph according to claim 1, characterized in that: In step 3, a document-level visual graph is obtained based on the structured document. The steps include performing overall image semantic understanding, visual object detection, and visual relation extraction on each image in the image set of the same structured document, thereby obtaining image nodes, visual object nodes, image-object containment relationships, and visual relation edges between visual object nodes. It also obtains the image source mapping relationship between each image node and its corresponding visual object node's source document, source section, and source image, forming a document-level visual graph. Image nodes include image node identifier, image description, overall semantic summary, local image storage path, source document, and source section; visual object nodes include visual object node identifier, object category, source image, bounding box, detection confidence, and object description.

5. The image encyclopedia question-answering method based on a multimodal hypergraph according to claim 1, characterized in that: In step 4, based on the document-level text hypergraph and document-level visual diagrams Constructing a document-level heterogeneous multimodal knowledge graph The steps include processing the document-level visual graph obtained in step 3. Image nodes, visual object nodes, and visual relation edges are integrated into the document-level text hypergraph obtained in step 2. By performing cross-modal alignment between text entity nodes and visual object nodes, the entity-object cross-modal alignment relationship between text entity nodes and visual object nodes is obtained, forming a document-level heterogeneous multimodal knowledge graph. Document-level heterogeneous multimodal knowledge graph This includes segment nodes, text entity nodes, text relation edges, text hyperedges, image association mapping relationships, image nodes, visual object nodes, visual relation edges, entity-object alignment relationships, and image source mapping relationships.

6. The image encyclopedia question-answering method based on a multimodal hypergraph according to claim 1, characterized in that: In step 5, the document-level heterogeneous multimodal knowledge graph is used as a basis. The steps to construct and obtain a multimodal retrieval index include building upon the document-level heterogeneous multimodal knowledge graph obtained in step 4. The process involves segmented text vector encoding, whole-image visual feature encoding, object region extraction and visual feature encoding, and index-graph mapping construction. This results in segmented text retrieval indexes, whole-image visual retrieval indexes, visual object retrieval indexes, and index-graph mapping relationships between each index item and graph node and its corresponding document, forming a multimodal retrieval index.

7. The image encyclopedia question-answering method based on multimodal hypergraphs according to claim 1, characterized in that: In step 6, the step of performing multi-path recall and candidate document aggregation based on the multimodal retrieval index, the user-input query image, and the natural language question to obtain a candidate document set and a corresponding hit graph node set includes: obtaining the user-input query, which includes a query image and a natural language question; extracting question text features from the natural language question; extracting query whole-image features and object region visual features from the query image; using the question text features to perform recall in the segmented text retrieval index to obtain segmented text recall results; using the query whole-image features to perform recall in the whole-image visual retrieval index to obtain whole-image visual recall results; using the object region visual features to perform recall in the visual object retrieval index to obtain visual object recall results; normalizing, weighting, and aggregating the segmented text recall results, whole-image visual recall results, and visual object recall results to obtain several documents with high document relevance scores as candidate documents, forming a candidate document set, which includes hit segments, hit images, and hit visual objects; and obtaining the graph node identifiers of the candidate documents based on the index-graph mapping relationship and candidate document transformation to form a hit graph node set for each candidate document.

8. The image encyclopedia question-answering method based on a multimodal hypergraph according to claim 1, characterized in that: In step 7, the candidate document set, the hit graph node set, and the document-level heterogeneous multimodal knowledge graph are used as the basis. Constructing a local multimodal evidence subgraph The steps involving the corresponding structured evidence context include the document-level heterogeneous multimodal knowledge graph obtained in step 4. Using the hit segment nodes, hit image nodes, and hit visual object nodes in the candidate documents obtained in step 6 as the starting point for expansion, the process extends along the document-level heterogeneous multimodal knowledge graph. The text relation edges, text hyperedges, visual relation edges, and entity-object alignment relations in the graph are locally expanded in relation to the query to form a local multimodal evidence subgraph. This allows us to obtain the context of structured evidence.

9. The image encyclopedia question-answering method based on multimodal hypergraphs according to claim 1, characterized in that: It also includes the following steps following step 7: Step 8: Input the structured evidence context obtained in step 7, along with the query image and natural language question input by the user, into the multimodal large language model to generate the final answer. The final answer includes the question answer, the answer credibility score, the basis of the answer, the content of the evidence used, and the source identifier.

10. An image encyclopedia question-answering device based on a multimodal hypergraph, comprising a computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the corresponding steps in the image encyclopedia question-and-answer method based on a multimodal hypergraph as described in any one of claims 1 to 9.

Images