Knowledge graph-based multi-modal retrieval enhanced generation result consistency checking method

Abstract

The application relates to the technical field of multi-modal information processing, and provides a multi-modal retrieval enhanced generation result consistency verification method based on a knowledge graph. After an initial result is parsed into a candidate semantic triple and mapped to a knowledge graph and a local graph substructure is extracted, explicit facts are verified through relationship existence verification, implicit knowledge relationships are verified through multi-hop path reasoning verification, logical contradictions are excluded through conflict verification, a consistency score is calculated and aggregated into an overall consistency score, and when there is a verification failure or the overall consistency score is lower than a threshold, modification or re-generation of the initial result is automatically triggered, so that the problems of lacking structured verification, being unable to verify implicit relationships, being difficult to identify conflicts and lacking modification basis in the background art are solved, model hallucinations are reduced, and the accuracy and reliability of the generated initial result are improved.

Description

Technical Field

[0001] This application relates to the field of multimodal information processing technology, and in particular to a method for consistency verification of multimodal retrieval enhancement generation results based on knowledge graphs. Background Technology

[0002] With the development of large-scale pre-trained models and retrieval-enhanced generative techniques, generative artificial intelligence systems can leverage external knowledge bases to perform complex tasks such as question answering, content generation, and decision support. In multimodal scenarios, the system typically needs to process multiple types of data, such as images and text, simultaneously, and achieve cross-modal retrieval and content generation through a unified semantic space.

[0003] Currently, mainstream multimodal retrieval enhancement generation schemes follow a basic process of "multimodal input—vector retrieval—generation model output." While these schemes can improve the richness and relevance of generated content to some extent by leveraging external knowledge, their core reliance is on vector similarity for evidence recall. The generation stage primarily depends on the implicit fusion of retrieval results using the model's parameterized knowledge, lacking structured constraints and verification mechanisms oriented towards factual relationships.

[0004] Specifically, existing solutions have the following technical limitations: First, the generated results lack explicit methods for verifying factual consistency, making them prone to factual errors, the so-called "model illusion." Second, for implicit knowledge relationships that require derivation through multiple relationship chains, direct vector retrieval or single-hop relationship matching alone cannot determine whether the generated content can be supported by multi-hop path reasoning in the knowledge base. Third, it is difficult to identify logical conflicts within the generated results or between the generated results and external knowledge. These limitations severely restrict the application of multimodal retrieval-enhanced generation systems in high-reliability scenarios such as medical auxiliary diagnosis, clinical decision support, financial risk control, and legal analysis.

[0005] Furthermore, existing solutions, when detecting that the generated results may be inaccurate, can usually only simply reduce the confidence level or discard the results, without providing specific conflict locations, missing relationship paths, or actionable correction criteria, making it even more difficult to achieve an automated correction and regeneration loop.

[0006] Therefore, how to perform structured fact consistency verification on the results generated by multimodal retrieval enhancement, especially the ability to simultaneously verify explicit and implicit facts, identify logical conflicts, and provide interpretable correction basis, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] This application provides a method for verifying the consistency of generated results based on multimodal retrieval using knowledge graphs, which reduces model illusion and improves the accuracy and credibility of the generated initial results.

[0008] The technical solution adopted in this application is as follows: A method for consistency verification of multimodal retrieval enhancement results based on knowledge graphs, the method comprising: Receive and encode image and text data to generate image and text vectors; Based on the image vector and the text vector, a preset database is retrieved to obtain a candidate evidence set. The candidate evidence set, the image data, and the text data are then input into a preset generation model to generate an initial result. The initial results are semantically parsed to extract entities, attributes, and relationships, and at least one candidate semantic triple is constructed. The candidate semantic triples are mapped to a knowledge graph, and local graph substructures are extracted from the knowledge graph. Based on the local graph substructure, a consistency check is performed on each candidate semantic triple, and the consistency check includes at least relation existence check, multi-hop path reasoning check and conflict check. Calculate the consistency score for each candidate semantic triple and aggregate all the consistency scores to obtain the overall consistency score; If any of the candidate semantic triples fails the consistency check, or if the overall consistency score is lower than a preset threshold, then the initial result is corrected or regenerated.

[0009] Preferably, the step of mapping the candidate semantic triples to a knowledge graph and extracting local graph substructures from the knowledge graph includes: The head and tail entity nodes in the candidate semantic triples are matched with the standard entity nodes in the knowledge graph. Then, the relation types in the candidate semantic triples are aligned with the standard relation types in the knowledge graph; Centered on the head entity node and the tail entity node after entity link matching, a local graph substructure is extracted from the knowledge graph. The local graph substructure includes the head entity node, the tail entity node, first-order neighbor nodes directly connected to the head entity node and the tail entity node and corresponding relationship edges, and multi-hop path information obtained by expanding within a preset number of hops centered on the head entity node and the tail entity node.

[0010] Preferably, performing consistency verification on each of the candidate semantic triples includes: Perform the relation existence check on each of the candidate semantic triples, including: Check whether there is an edge in the local graph substructure that points from the head entity node to the tail entity node, and whether the relation type of the edge is consistent with the relation type after relation alignment; If there are edges of the same type, or if there are no edges of the same type, but there are edges of the type of equivalence relation, hierarchical relation, or pre-defined substitutable relation, then the relation existence check is passed. If there are neither edges of the same type nor edges of the aforementioned equivalence relation, hierarchical relation, or preset substitutable relation, then the relation existence check is deemed to have failed.

[0011] Preferably, performing consistency verification on each of the candidate semantic triples includes: Perform the multi-hop path reasoning verification for each of the candidate semantic triples, including: In the local graph substructure, starting from the head entity node and ending at the tail entity node, a candidate path is searched for where the number of hops between the head entity node and the tail entity node does not exceed a preset number. For each candidate path, the path support is calculated based on the path length, the semantic matching degree between the relation sequence and the relation in the candidate semantic triple, the path confidence, and the weight of the intermediate entity node. If the path support of a candidate path exceeds a preset support threshold, it is determined to pass the multi-hop path inference verification. If the path support of multiple candidate paths exceeds the preset support threshold, it is determined that the multi-hop path inference verification is passed, and the multiple candidate paths are sorted according to the path support to select the optimal path. If the path support of the candidate path does not exceed the preset support threshold, it is determined that the multi-hop path reasoning verification has failed.

[0012] Preferably, performing consistency verification on each of the candidate semantic triples includes: Perform the conflict check on each of the candidate semantic triples, including: In the local graph substructure, retrieve existing facts that have the same head entity node and relationship as the candidate semantic triple; If there exists another candidate semantic triple whose tail entity node is mutually exclusive with the tail entity node of the candidate semantic triple in the existing facts, then a factual conflict is determined to exist. If the relations in the candidate semantic triples belong to a pre-defined mutually exclusive relation pair, then a relation conflict is determined to exist; If the semantics of the candidate semantic triple is contrary to the existing facts or the path reasoning conclusion obtained by the multi-hop path reasoning verification, then a reasoning conflict is determined to exist. If any of the factual conflict, relational conflict, or inference conflict exists, the candidate semantic triple is determined to have failed the conflict check, and the candidate semantic triple is modified or deleted based on the existing facts in the local graph substructure. If there are no factual conflicts, relational conflicts, or reasoning conflicts, then the conflict check is passed.

[0013] Preferably, the step of receiving and encoding image data and text data to generate image vectors and text vectors includes: A multimodal coding model is used to map the image data and the text data to the same shared semantic space, generating the image vector and the text vector.

[0014] Preferably, the step of retrieving a preset database based on the image vector and the text vector to obtain a set of candidate evidence includes: Based on a mechanism combining vector recall and semantic reordering, and based on the image vector and the text vector, the similarity with the evidence vector stored in the preset database is calculated in the shared semantic space to recall initial candidate evidence. The initial candidate evidence is reordered using keyword matching or relevance scoring to obtain a set of candidate evidence.

[0015] Preferably, the step of calculating the consistency score for each candidate semantic triple and aggregating all the consistency scores to obtain the overall consistency score includes: Based on the relation existence verification score and its weight, the multi-hop path reasoning verification score and its weight, and the conflict verification score and its weight, the consistency score of each candidate semantic triple is determined. All the consistency scores are aggregated based on a preset aggregation algorithm to obtain the overall consistency score; The preset aggregation algorithm includes an average value algorithm, a weighted average value algorithm, or a minimum value algorithm.

[0016] Preferably, after correcting or regenerating the initial result, the method further includes: Use the corrected or regenerated result as the new initial result, and repeat the operations of semantic parsing, knowledge graph mapping, consistency verification and consistency scoring; If the overall consistency score after re-execution is still lower than the preset threshold and has not reached the preset number of iterations, then correction or regeneration is triggered again until the overall consistency score meets the preset threshold or reaches the preset number of iterations.

[0017] Preferably, the step of inputting the candidate evidence set, the image data, and the text data into a preset generation model includes: Combine the candidate evidence set with the text data to form a text-side context; The image data is converted into a structured semantic description to construct image semantic triples, which serve as image-side context; The text-side context and the image-side context are concatenated into a unified format of multimodal prompt information, which is then input into the preset generation model.

[0018] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application relates to the field of multimodal information processing technology, and provides a method for consistency verification of multimodal retrieval enhancement generation results based on knowledge graphs. The method includes: receiving and encoding image data and text data to generate image vectors and text vectors; retrieving a preset database based on the image vectors and text vectors to obtain a candidate evidence set, and inputting the candidate evidence set, image data, and text data into a preset generation model to generate an initial result; performing semantic parsing on the initial result to extract entities, attributes, and relations, and constructing at least one candidate semantic triple; mapping the candidate semantic triple to a knowledge graph, and extracting local graph substructures from the knowledge graph; performing consistency verification on each candidate semantic triple based on the local graph substructure, the consistency verification including at least relation existence verification, multi-hop path reasoning verification, and conflict verification; calculating the consistency score of each candidate semantic triple, and aggregating all consistency scores to obtain an overall consistency score; if the consistency verification of a candidate semantic triple fails, or if the overall consistency score is lower than a preset threshold, the initial result is corrected or regenerated. This application parses the initial results into candidate semantic triples, maps them to a knowledge graph, and extracts local graph substructures. It verifies explicit facts through relation existence checks, verifies implicit knowledge relations through multi-hop path reasoning, eliminates logical contradictions through conflict checks, calculates a consistency score, and aggregates it into an overall consistency score. When a check fails or the overall consistency score is below a threshold, it triggers the correction or regeneration of the initial results. This solves the problems of lack of structured verification, inability to verify implicit relations, difficulty in identifying conflicts, and lack of correction basis in the background technology, reduces model illusion, and improves the accuracy and credibility of the generated initial results. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the consistency verification method for multimodal retrieval enhancement based on knowledge graphs provided in this application embodiment; Figure 2 This is a flowchart provided in an embodiment of the present application for mapping image data and text data to the same shared semantic space; Figure 3 This is a flowchart provided in an embodiment of the present application for obtaining a set of candidate evidence by searching a preset database; Figure 4 This is a flowchart provided in an embodiment of the present application, in which candidate evidence sets, image data, and text data are input into a preset generation model; Figure 5 This is a flowchart provided in an embodiment of the present application for mapping candidate semantic triples to a knowledge graph and extracting local graph substructures from the knowledge graph; Figure 6 This is a flowchart of the calculation of the overall consistency score provided in the embodiments of this application; Figure 7 This is a flowchart of the initial result correction and regeneration provided in the embodiments of this application; Figure 8 This is a schematic diagram of a knowledge graph-based multimodal retrieval enhancement generation result consistency verification system provided in this application embodiment; Figure 9 This is a schematic diagram of the computer device provided in the embodiments of this application. Detailed Implementation

[0021] The technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art are within the scope of protection of this application.

[0022] As described in the background section, existing solutions have the following limitations: First, the generated results lack explicit methods for verifying factual consistency, making them prone to factual errors, also known as "model illusions." Second, for implicit knowledge relationships that require derivation through multiple relationship chains, direct vector retrieval or single-hop relationship matching alone cannot determine whether the generated content can be supported by multi-hop path reasoning in the knowledge base. Third, it is difficult to identify logical conflicts within the generated results or between the generated results and external knowledge. These limitations severely restrict the application of multimodal retrieval-enhanced generation systems in high-reliability scenarios such as medical auxiliary diagnosis, clinical decision support, financial risk control, and legal analysis.

[0023] Furthermore, existing solutions, when detecting that the generated results may be inaccurate, can usually only simply reduce the confidence level or discard the results, without providing specific conflict locations, missing relationship paths, or actionable correction criteria, making it even more difficult to achieve an automated correction and regeneration loop.

[0024] Based on this, this application provides a consistency verification method for multimodal retrieval enhancement based on knowledge graphs, which solves the problems of structured verification, inability to verify implicit relationships, difficulty in identifying conflicts, and lack of correction basis in the background technology.

[0025] The embodiments of this application will be analyzed in detail below with reference to the accompanying drawings.

[0026] Example 1 To address the technical problems existing in the background art, this embodiment provides a method for consistency verification of generated results based on knowledge graph-based multimodal retrieval enhancement. This method is executed by a computing device, which includes, but is not limited to, servers, cloud nodes, edge computing devices, or a distributed processing platform composed of multiple computing devices. (Reference) Figure 1 The methods include: S1: Receive and encode image data and text data, and generate image vectors and text vectors; S2: Based on image vectors and text vectors, a preset database is retrieved to obtain a set of candidate evidence. The set of candidate evidence, image data, and text data are then input into a preset generation model to generate initial results. S3: Perform semantic parsing on the initial results, extract entities, attributes and relations, and construct at least one candidate semantic triple; S4: Map candidate semantic triples to the knowledge graph and extract local graph substructures from the knowledge graph; S5: Based on the local graph substructure, perform consistency checks on each candidate semantic triple. The consistency checks include at least relation existence checks, multi-hop path reasoning checks, and conflict checks. S6: Calculate the consistency score for each candidate semantic triple and aggregate all consistency scores to obtain the overall consistency score; S7: If the consistency check of a candidate semantic triple fails, or if the overall consistency score is lower than the preset threshold, the initial result will be corrected or regenerated.

[0027] In step S1, image data includes, but is not limited to, medical images, natural images, invoice images, document screenshots, or other visual information; text data includes, but is not limited to, user questions, explanatory text, search suggestions, historical dialogues, or other natural language input. An encoder maps the multimodal input of image and text data into vector representations, providing a unified numerical basis for subsequent retrieval and verification.

[0028] As an example, during encoding, an image encoder and a text encoder are called separately. The image encoder is used to perform semantic encoding on the image data to generate an image vector, and the text encoder is used to perform semantic encoding on the text data to generate a text vector.

[0029] As a preferred embodiment, refer to Figure 2 It receives and encodes image data and text data, and generates image vectors and text vectors, including: S10: A multimodal coding model is used to map image data and text data to the same shared semantic space, generating image vectors and text vectors.

[0030] Multimodal coding models, such as the CLIP model, align the output spaces of image and text encoders to the same shared semantic space through contrastive learning. During training, a large number of image-text pairing samples are constructed to ensure that matching image-text pairs are close in distance within the vector space, while mismatched pairs are far apart. During inference, the input image and text are processed by their respective encoders, generating vectors of the same dimension. This transforms the raw data from different modalities into vector representations that can be compared in the same space, eliminating modal differences and providing a unified mathematical foundation for subsequent cross-modal retrieval and consistency analysis.

[0031] In step S2, the preset database can be an external knowledge base, a vector knowledge base, a structured triplet database, or a text fragment database. The evidence content stored therein is pre-encoded and has the same vector dimension as in step S1. The retrieval process uses the image vectors and text vectors generated in step S1 to calculate the similarity in the preset database, recalling the top k pieces of evidence most relevant to the input, forming a candidate evidence set.

[0032] As a preferred embodiment, refer to Figure 3 Based on image vectors and text vectors, a pre-defined database is retrieved to obtain a set of candidate evidence, including: S20: Based on a mechanism combining vector recall and semantic reordering, and based on image vectors and text vectors, calculate the similarity with the evidence vectors stored in the preset database in the shared semantic space to recall initial candidate evidence; S21: Reorder the initial candidate evidence by keyword matching or relevance scoring to obtain a set of candidate evidence.

[0033] In step S20, vector recall refers to using cosine similarity, inner product, and other distance metrics to quickly filter out the top few evidence vectors that are closest to the query vector from the vector index of the preset database. Semantic re-ranking refers to using a more accurate semantic matching model on the recall results to re-score the semantic relevance of each candidate evidence to the original query, and then sorting them from high to low scores. In this way, candidate evidence is quickly filtered through vector recall, and then accuracy is improved through semantic re-ranking, balancing retrieval efficiency and evidence quality, providing reliable external knowledge for the generative model, and reducing model illusion.

[0034] In step S21, keyword matching is a secondary screening based on features such as entity and term co-occurrence in the text; the relevance score is the score output by the pre-trained relevance regression model. Through re-ranking, the evidence with the most semantic relevance and the best factual match is placed at the top, further eliminating noise in the recall results, ensuring the high quality of the candidate evidence set, and providing accurate external knowledge support for the subsequent generation model.

[0035] As a preferred embodiment, refer to Figure 4 The candidate evidence set, image data, and text data are input into a pre-defined generative model, including: S22: Combine the candidate evidence set with text data to form a text-side context; S23: Convert image data into structured semantic descriptions to construct image semantic triples as image-side context; S24: Concatenate the text-side context and image-side context into a unified format of multimodal prompt information, and input the preset generation model.

[0036] The purpose of combining the candidate evidence set with the text data in step S22 is to preserve the complete semantics of the retrieved evidence and the user's question. Step S23 can extract candidate entities and relationships between entities from the image data through methods such as object detection, visual language models, or region description, and construct image semantic triples consistent with the text-side format. In step S24, the text-side context and the image-side context are then concatenated according to a preset format to form multimodal prompt information that the preset generation model can understand. Subsequently, the preset generation model outputs the initial result. The preset generation model includes, but is not limited to, large language models, multimodal generation models, or a combination of both.

[0037] As an example, suppose the user inputs text data as "describe the main elements of this landscape image," and the image data is a natural landscape image containing snow-capped mountains, lakes, and forests. The system first calls a multimodal coding model to generate text vectors and image vectors respectively, and then fuses and retrieves them in a shared semantic space. A pre-stored database contains a large number of image-text pairs and corresponding image semantic triples, such as "(snow-capped mountain, located in, background)", "(lake, located in, foreground)", "(forest, surrounding, lake)", etc. Based on the image vectors and text vectors, the system calculates the similarity with the evidence vectors in the database in the shared semantic space, recalling 20 initial candidate evidences semantically related to "landscape," "snow-capped mountain," and "lake." Subsequently, it re-ranks the evidences using keyword matching (such as "snow-capped mountain," "lake," and "forest") and a relevance scoring model, selecting the 5 most relevant high-quality candidate evidences. Finally, the system inputs these 5 candidate pieces of evidence, along with the original text "describe the main elements in this landscape picture" and the image data, into the large language model. The model integrates the visual features of the image with the textual evidence to generate the initial result: "In the distance of the picture are continuous snow-capped mountains, at the foot of which is a deep blue lake surrounded by dense forests. The sky is clear, and the overall picture presents a typical landscape of a plateau lake." The purpose of step S3 is to provide standardized input for subsequent mapping and verification.

[0038] As an example, step S3 employs an information extraction model or a large language model based on prompt words to extract entities (such as "snow mountain" and "lake"), attributes (such as "dark blue"), and relations (such as "surrounding" and "located") from the initial results, and organizes them into candidate semantic triples in a uniform format of (head entity, relation, tail entity), such as (lake, surround, forest). Semantic triples are the smallest unit of expression for structured facts, transforming unstructured natural language into a form that can be directly verified by the knowledge graph.

[0039] It should be noted that the system performs semantic parsing on the initial results, extracting semantic information such as entities, attributes, relations, negation, comparison, attribution, and temporal sequence to construct a set of candidate semantic triples. Each candidate semantic triple is represented by a unified structure T = (h, r, t), where h represents the head entity, r represents the relation, and t represents the tail entity or attribute value. Time labels, source labels, confidence labels, and modality source labels can also be added if necessary. If image semantic triples have already been obtained from the input, the system performs unified normalization processing on the text-side and image-side triples to form a cross-modal consistent set of candidate semantic triples.

[0040] The purpose of step S4 is to provide the graph data foundation of the knowledge graph for subsequent verification.

[0041] As a preferred embodiment, refer to Figure 5Candidate semantic triples are mapped to a knowledge graph, and local graph substructures are extracted from the knowledge graph, including: S40: Perform entity link matching between the head and tail entity nodes in the candidate semantic triples and the standard entity nodes in the knowledge graph; S41: Align the relation types in the candidate semantic triples with the standard relation types in the knowledge graph; S42: Taking the head entity node and tail entity node after entity link matching as the center, extract the local graph substructure from the knowledge graph. The local graph substructure includes the head entity node, tail entity node, first-order neighbor nodes directly connected to the head entity node and tail entity node and corresponding relationship edges, as well as multi-hop path information obtained by expanding within a preset number of hops with the head entity node and tail entity node as the center.

[0042] Step S40 links the head and tail entities in the candidate semantic triples to the uniquely corresponding standard nodes in the knowledge graph, eliminating synonyms or referential ambiguities. Step S41 aligns the candidate relations to the standard relation types in the knowledge graph, ensuring that the edge types used in subsequent verification are defined within the graph. Step S42 uses the linked head and tail entity nodes as centers to extract first-order neighbors and multi-hop paths within a preset hop count range, forming a local graph substructure. Thus, through entity linking and relation alignment, the candidate semantic triples expressed in natural language are transformed into a standardized form operable in the knowledge graph, eliminating semantic ambiguity. Furthermore, by extracting local graph substructures, the search scope for subsequent verification is limited to the local graph substructures surrounding the head and tail entities, preserving the information needed to determine explicit facts and implicit paths while avoiding the computational overhead of searching the entire graph.

[0043] As an example, step S4 first matches the entities and relations in the candidate semantic triples with the standard entity nodes and standard relation types in the knowledge graph (i.e., aligning entity links with relations), eliminating ambiguity caused by different synonyms, referents, or expressions. Then, centered on the matched head and tail entity nodes, a local graph substructure is extracted from the knowledge graph. This local graph substructure includes the head and tail nodes themselves, their first-order neighbor nodes and edges, and multi-hop paths expanded within a preset number of hops (e.g., 2 to 4 hops). The local graph substructure limits the search scope for subsequent consistency checks, ensuring the completeness of the information required for verification while avoiding the inefficiency of searching the entire graph.

[0044] As a preferred implementation, a consistency check is performed on each candidate semantic triple, including: Perform relation existence checks on each candidate semantic triple, including: Check if there are edges in the local graph substructure that point from the head entity node to the tail entity node, and whether the relation type of the edges is consistent with the relation type after relation alignment; If there are edges of the same type, or if there are no edges of the same type, but there are edges of the type of equivalence relation, hierarchical relation, or pre-defined substitutable relation, then the relation existence check is passed. If there are no edges of the same type, nor any edges with equivalent, hierarchical, or pre-defined substitutable relationships, then the relationship existence check is deemed to have failed.

[0045] The core of the relation existence verification lies in directly comparing candidate semantic triples with explicit edges in the knowledge graph to verify whether the explicit facts in the initial results are directly supported by the knowledge graph. After aligning entity links with relations, the system can quickly determine whether there are type-matching relation edges between the head and tail entities: if they exist, it means that the fact has been explicitly recorded in the graph, with the highest credibility; if only equivalent, hierarchical, or pre-set substitutable relations exist, although the relation existence verification passes, it belongs to weak consistency, indicating that although the fact has no direct match, it is semantically acceptable, providing a flexible basis for subsequent scoring; if neither exists, it is judged as failing, triggering path reasoning or correction. This verification solves the problem of lack of structured verification and reduces model illusion.

[0046] It should be noted that for candidate semantic triples in a weakly consistent state, subsequent multi-hop path reasoning and conflict checks will continue to be performed, and a score lower than that for explicit fact matching will be assigned in the subsequent consistency score. If necessary, the relation expression will be replaced, weakened, or modified based on the multi-hop path reasoning results or existing facts; if the subsequent score is insufficient or there is a conflict, correction or regeneration will be triggered.

[0047] As a preferred implementation, a consistency check is performed on each candidate semantic triple, including: Perform multi-hop path reasoning verification for each candidate semantic triple, including: In the local graph substructure, starting from the head entity node and ending at the tail entity node, a candidate path is searched for where the number of hops between the head entity node and the tail entity node does not exceed a preset hop value. For each candidate path, the path support is calculated based on the path length, the semantic matching degree between the relation sequence and the relation in the candidate semantic triple, the path confidence, and the weight of the intermediate entity node. If the path support of a candidate path exceeds the preset support threshold, it is determined to pass the multi-hop path inference verification. If the path support of multiple candidate paths exceeds the preset support threshold, it is determined to be a multi-hop path inference verification, and the multiple candidate paths are sorted by path support to select the optimal path. If the path support of no candidate path exceeds the preset support threshold, it is determined that the multi-hop path reasoning verification has failed.

[0048] The core of multi-hop path reasoning verification lies in verifying implicit facts through indirect relationship chains in the knowledge graph. This verifies implicit facts that are not explicitly stored in the knowledge graph but can be derived through indirect relationship chains, thereby improving the coverage of consistency checks. When there are no direct edges between the head and tail entities, the system searches for paths with limited length (e.g., 2-4 hops) in the local graph substructure and calculates path support by combining path length, semantic matching degree between the relationship sequence and the relationship in the candidate semantic triples, path confidence, and the weight of intermediate entity nodes. If a path with support exceeding the threshold exists, it indicates that although the candidate relationship is not explicitly stored, it can be derived through relationship chains in the knowledge graph, thus being judged as consistent. If multiple valid paths exist, the optimal path is selected as the basis for interpretation based on path support ranking. This mechanism solves the defect of being unable to verify implicit knowledge relationships, expands the verification scope from single-hop explicit facts to multi-hop reasoning facts, and enhances the credibility coverage of the generated results.

[0049] It should be noted that the candidate paths mentioned above consist of multiple intermediate nodes and relation edges, which can be represented as h→r1→n1→r2→n2→…→rk→t. The calculation of path support is a quantitative assessment of the credibility of the candidate paths. Shorter path lengths generally indicate more direct reasoning and higher credibility; the semantic matching degree between the relation sequence and the candidate relation measures whether the semantics of each relation combination on the path are equivalent or imply the target relation (e.g., "father → brother" can deduce "uncle"); path confidence can be derived from the statistical reliability of each edge in the knowledge graph or the weights of manually labeled edges; the weights of intermediate entity nodes reflect the importance or frequency of occurrence of the nodes in the domain (e.g., core concept entities have higher weights). These factors are weighted and integrated to obtain a support score between 0 and 1, which is used to compare with a preset support threshold, thereby selecting reasoning paths that semantically effectively support the candidate semantic triples.

[0050] As a preferred implementation, a consistency check is performed on each candidate semantic triple, including: Perform conflict checking on each candidate semantic triple, including: Within the local graph substructure, retrieve existing facts that share the same head entity node and relation as the candidate semantic triple; If there exists another candidate semantic triple whose tail entity node is mutually exclusive with the tail entity node of the candidate semantic triple, then a factual conflict is determined to exist. If the relation in the candidate semantic triple belongs to a pre-defined mutually exclusive relation pair, then a relation conflict is determined to exist; If the semantics of the candidate semantic triple contradict the conclusion of the path reasoning obtained from existing facts or multi-hop path reasoning verification, then a reasoning conflict is determined to exist. If any of the factual conflict, relational conflict, or inference conflict exists, the candidate semantic triple is determined to have failed the conflict check, and the candidate semantic triple is modified or deleted based on the existing facts in the local graph substructure. If there are no factual conflicts, relational conflicts, or inference conflicts, the conflict check is passed.

[0051] The conflict verification mechanism is used to exclude content in the initial results that contradicts the facts or logical rules of the knowledge graph. Conflict verification is performed from three dimensions: first, factual conflict, i.e., mutually exclusive tail entities exist under the same head entity and relation; second, relational conflict, i.e., the candidate relation itself belongs to a pre-defined mutually exclusive relation pair; and third, reasoning conflict, i.e., the semantics of the candidate semantic triple contradict existing facts or the conclusion of a previously passed path inference. If any type of conflict is detected, the result is deemed unsuccessful, and the candidate semantic triple can be corrected or directly deleted based on the correct facts in the graph. This mechanism solves the problem of difficulty in identifying logical conflicts, ensuring that the verification results not only require the existence or derivability of facts but also do not contradict other facts, thereby improving the logical consistency of the generated results.

[0052] In summary, through consistency verification, this embodiment comprehensively verifies the credibility of the generated results from three levels: explicit facts, implicit relationships, and logical conflicts. Relationship existence verification ensures that direct facts are verifiable, multi-hop path reasoning verification supplements the derivability of indirect knowledge, and conflict verification eliminates contradictions between facts and logic. Therefore, this embodiment effectively reduces model illusion, improves the accuracy and interpretability of the generated results, and provides a structured and reliable guarantee for enhanced multimodal retrieval generation in high-reliability scenarios.

[0053] As a preferred embodiment, refer to Figure 6 Calculate the consistency score for each candidate semantic triple, and aggregate all consistency scores to obtain the overall consistency score, including: S60: Determine the consistency score of each candidate semantic triple based on the relation existence verification score and its weight, the multi-hop path reasoning verification score and its weight, and the conflict verification score and its weight. S61: Aggregate all consistency scores based on a preset aggregation algorithm to obtain an overall consistency score; wherein the preset aggregation algorithm includes an average value algorithm, a weighted average value algorithm, or a minimum value algorithm.

[0054] For each candidate semantic triple, its consistency score can be expressed as: S = α × S_exist + β × S_path - γ × S_conflict. S_exist is the relation existence check score (1 for direct matching, 0.5 for weak consistency, and 0 otherwise), S_path is the multi-hop path reasoning support score (normalized based on path support), and S_conflict is the conflict penalty (1 for conflict, 0 otherwise). A larger S_conflict value indicates a higher degree of conflict between the candidate semantic triple and the knowledge graph or reasoning conclusion. Therefore, a subtraction method is used in the consistency score to reduce the score of conflicting candidate semantic triples. α, β, and γ are preset weight parameters that can be adjusted according to the application scenario (γ can be increased in high-reliability scenarios). Aggregation employs average, weighted average, or minimum algorithms: in the weighted average algorithm, the weights can be determined based on the importance of the triples in the initial results, entity confidence, or modality source; the minimum algorithm takes the lowest score among all triples, suitable for demanding scenarios sensitive to any errors. The overall consistency score serves as a quantitative indicator of the generated result's reliability, used to determine whether correction or regeneration is triggered.

[0055] As a preferred embodiment, refer to Figure 7 After correcting or regenerating the initial result, the method also includes: S8: Use the corrected or regenerated result as the new initial result, and repeat the operations of semantic parsing, knowledge graph mapping, consistency verification and consistency scoring; S9: If the overall consistency score after re-execution is still lower than the preset threshold and the preset number of iterations has not been reached, then correction or regeneration will be triggered again until the overall consistency score meets the preset threshold or the preset number of iterations is reached.

[0056] The closed-loop mechanism in steps S8 to S9 re-inputs the corrected result into the verification process to achieve iterative optimization. If the score after re-verification is still lower than the threshold and the preset number of iterations has not been reached, the correction will continue to be triggered until the conditions are met or the maximum number of iterations is reached.

[0057] As an example, continuing with the landscape scene from the previous text, the initial result, after consistency verification, found that the "surrounded" relationship in "the lake is surrounded by a dense forest" did not directly match in the knowledge graph. However, it was supported by multi-hop path reasoning (lake—adjacent—forest edge—containment—forest), so the overall consistency score was 0.85, higher than the preset threshold of 0.7, requiring no correction. If the score is 0.6, lower than the threshold, correction is triggered. The system corrects the relationship to "lake, adjacent, forest" according to the optimal path, and regenerates the description: "In the distance, there are continuous snow-capped mountains. At the foot of the mountains is a deep blue lake. The lake is adjacent to a dense forest. The sky is clear, presenting a highland lake landscape." After another verification, the score increases to 0.88, meeting the requirements. If, after multiple corrections, the score is still lower than the threshold and the number of iterations has not been reached (e.g., 3 times), the iteration continues until the target is met or the process terminates.

[0058] Example 2 This second embodiment provides a consistency verification system for multimodal retrieval enhancement based on knowledge graphs, referring to... Figure 8 The system includes: The vector generation unit is used to receive and encode image data and text data, and generate image vectors and text vectors. The initial result generation unit is used to retrieve a set of candidate evidences from a preset database based on image vectors and text vectors, and then input the set of candidate evidences, image data and text data into a preset generation model to generate initial results. The candidate semantic triple construction unit is used to perform semantic parsing on the initial results, extract entities, attributes and relations, and construct at least one candidate semantic triple. The local graph substructure extraction unit is used to map candidate semantic triples to the knowledge graph and extract local graph substructures from the knowledge graph. The consistency verification unit is used to perform consistency verification on each candidate semantic triple based on the local graph substructure. The consistency verification includes at least relation existence verification, multi-hop path reasoning verification, and conflict verification. The overall consistency score calculation unit is used to calculate the consistency score of each candidate semantic triple and aggregate all consistency scores to obtain the overall consistency score. The correction unit is used to correct or regenerate the initial result when the consistency check of candidate semantic triples fails, or when the overall consistency score is lower than a preset threshold.

[0059] Example 3 This embodiment three provides a computer device, including a memory and a processor; the memory stores a computer program that can run on the processor, and when the computer program is executed by the processor, it executes the knowledge graph-based multimodal retrieval enhancement generation result consistency verification method provided in embodiment one above.

[0060] Among them, such as Figure 9 As shown, an exemplary computer device of this embodiment is illustrated, which may specifically include a processor 1510, a video display adapter 1511, a disk drive 1512, an input / output interface 1513, a network interface 1514, and a memory 1520. The processor 1510, video display adapter 1511, disk drive 1512, input / output interface 1513, network interface 1514, and memory 1520 can be communicatively connected via a communication bus 1530.

[0061] The processor 1510 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided in this application.

[0062] The memory 1520 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1520 can store the operating system 1521 for controlling the operation of the computer device, and the basic input / output system 1522 for controlling the low-level operations of the computer device. Additionally, it can store a web browser 1523, a data storage management system 1524, and a device identification information processing system 1525, etc. The aforementioned device identification information processing system 1525 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 1520 and is called and executed by the processor 1510.

[0063] Input / output interface 1513 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0064] Network interface 1514 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0065] The communication bus 1530 includes a pathway for transmitting information between various components of the device, such as processor 1510, video display adapter 1511, disk drive 1512, input / output interface 1513, network interface 1514, and memory 1520.

[0066] In addition, the device can also obtain information on specific claim conditions from the virtual resource object claim condition information database for condition judgment, and so on.

[0067] It should be noted that although the above-described device only shows the processor 1510, video display adapter 1511, disk drive 1512, input / output interface 1513, network interface 1514, memory 1520, communication bus 1530, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0068] Example 4 This fourth embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed, it implements the consistency verification method for the multimodal retrieval enhancement generation result based on knowledge graph provided in the first embodiment.

[0069] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of the embodiments of this application.

[0070] The foregoing has provided a detailed description of the consistency verification method for multimodal retrieval enhancement based on knowledge graphs provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for consistency verification of multimodal retrieval enhancement generation results based on knowledge graphs, characterized in that, The method includes: Receive and encode image and text data to generate image and text vectors; Based on the image vector and the text vector, a preset database is retrieved to obtain a candidate evidence set. The candidate evidence set, the image data, and the text data are then input into a preset generation model to generate an initial result. The initial results are semantically parsed to extract entities, attributes, and relationships, and at least one candidate semantic triple is constructed. The candidate semantic triples are mapped to a knowledge graph, and local graph substructures are extracted from the knowledge graph. Based on the local graph substructure, a consistency check is performed on each candidate semantic triple, and the consistency check includes at least relation existence check, multi-hop path reasoning check and conflict check. Calculate the consistency score for each candidate semantic triple and aggregate all the consistency scores to obtain the overall consistency score; If any of the candidate semantic triples fails the consistency check, or if the overall consistency score is lower than a preset threshold, then the initial result is corrected or regenerated.

2. The method for consistency verification of multimodal retrieval enhancement generation results based on knowledge graphs according to claim 1, characterized in that, The step of mapping the candidate semantic triples to a knowledge graph and extracting local graph substructures from the knowledge graph includes: The head and tail entity nodes in the candidate semantic triples are matched with the standard entity nodes in the knowledge graph. Then, the relation types in the candidate semantic triples are aligned with the standard relation types in the knowledge graph; Centered on the head entity node and the tail entity node after entity link matching, a local graph substructure is extracted from the knowledge graph. The local graph substructure includes the head entity node, the tail entity node, first-order neighbor nodes directly connected to the head entity node and the tail entity node and corresponding relationship edges, and multi-hop path information obtained by expanding within a preset number of hops centered on the head entity node and the tail entity node.

3. The method for consistency verification of multimodal retrieval enhancement generation results based on knowledge graphs according to claim 2, characterized in that, The consistency check performed on each of the candidate semantic triples includes: Perform the relation existence check on each of the candidate semantic triples, including: Check whether there is an edge in the local graph substructure that points from the head entity node to the tail entity node, and whether the relation type of the edge is consistent with the relation type after relation alignment; If there are edges of the same type, or if there are no edges of the same type, but there are edges of the type of equivalence relation, hierarchical relation, or pre-defined substitutable relation, then the relation existence check is passed. If there are neither edges of the same type nor edges of the aforementioned equivalence relation, hierarchical relation, or preset substitutable relation, then the relation existence check is deemed to have failed.

4. The method for consistency verification of multimodal retrieval enhancement generation results based on knowledge graphs according to claim 3, characterized in that, The consistency check performed on each of the candidate semantic triples includes: Perform the multi-hop path reasoning verification for each of the candidate semantic triples, including: In the local graph substructure, starting from the head entity node and ending at the tail entity node, a candidate path is searched for where the number of hops between the head entity node and the tail entity node does not exceed a preset number. For each candidate path, the path support is calculated based on the path length, the semantic matching degree between the relation sequence and the relation in the candidate semantic triple, the path confidence, and the weight of the intermediate entity node. If the path support of a candidate path exceeds a preset support threshold, it is determined to pass the multi-hop path inference verification. If the path support of multiple candidate paths exceeds the preset support threshold, it is determined that the multi-hop path inference verification is passed, and the multiple candidate paths are sorted according to the path support to select the optimal path. If the path support of the candidate path does not exceed the preset support threshold, it is determined that the multi-hop path reasoning verification has failed.

5. The method for consistency verification of multimodal retrieval enhancement generation results based on knowledge graphs according to claim 4, characterized in that, The consistency check performed on each of the candidate semantic triples includes: Perform the conflict check on each of the candidate semantic triples, including: In the local graph substructure, retrieve existing facts that have the same head entity node and relationship as the candidate semantic triple; If there exists another candidate semantic triple whose tail entity node is mutually exclusive with the tail entity node of the candidate semantic triple in the existing facts, then a factual conflict is determined to exist. If the relations in the candidate semantic triples belong to a pre-defined mutually exclusive relation pair, then a relation conflict is determined to exist; If the semantics of the candidate semantic triple is contrary to the existing facts or the path reasoning conclusion obtained by the multi-hop path reasoning verification, then a reasoning conflict is determined to exist. If any of the factual conflict, relational conflict, or inference conflict exists, the candidate semantic triple is determined to have failed the conflict check, and the candidate semantic triple is modified or deleted based on the existing facts in the local graph substructure. If there are no factual conflicts, relational conflicts, or reasoning conflicts, then the conflict check is passed.

6. The method for consistency verification of multimodal retrieval enhancement generation results based on knowledge graphs according to claim 1, characterized in that, The process of receiving and encoding image data and text data to generate image vectors and text vectors includes: A multimodal coding model is used to map the image data and the text data to the same shared semantic space, generating the image vector and the text vector.

7. The method for consistency verification of multimodal retrieval enhancement generation results based on knowledge graphs according to claim 6, characterized in that, The step of retrieving a preset database based on the image vector and the text vector to obtain a candidate evidence set includes: Based on a mechanism combining vector recall and semantic reordering, and based on the image vector and the text vector, the similarity with the evidence vector stored in the preset database is calculated in the shared semantic space to recall initial candidate evidence. The initial candidate evidence is reordered using keyword matching or relevance scoring to obtain a set of candidate evidence.

8. The method for consistency verification of multimodal retrieval enhancement generation results based on knowledge graphs according to claim 1, characterized in that, The process involves calculating the consistency score for each candidate semantic triple and aggregating all the consistency scores to obtain an overall consistency score, including: Based on the relation existence verification score and its weight, the multi-hop path reasoning verification score and its weight, and the conflict verification score and its weight, the consistency score of each candidate semantic triple is determined. All the consistency scores are aggregated based on a preset aggregation algorithm to obtain the overall consistency score; The preset aggregation algorithm includes an average value algorithm, a weighted average value algorithm, or a minimum value algorithm.

9. The method for consistency verification of multimodal retrieval enhancement generation results based on knowledge graphs according to claim 1, characterized in that, After the initial result is corrected or regenerated, the method further includes: Use the corrected or regenerated result as the new initial result, and repeat the operations of semantic parsing, knowledge graph mapping, consistency verification and consistency scoring; If the overall consistency score after re-execution is still lower than the preset threshold and has not reached the preset number of iterations, then correction or regeneration is triggered again until the overall consistency score meets the preset threshold or reaches the preset number of iterations.

10. The method for consistency verification of multimodal retrieval enhancement generation results based on knowledge graphs according to claim 1, characterized in that, The step of inputting the candidate evidence set, the image data, and the text data into a preset generation model includes: Combine the candidate evidence set with the text data to form a text-side context; The image data is converted into a structured semantic description to construct image semantic triples, which serve as image-side context; The text-side context and the image-side context are concatenated into a unified format of multimodal prompt information, which is then input into the preset generation model.

Images