Multi-modal knowledge graph question answering system and method based on spatiotemporal perception cognitive agent, electronic device and storage medium

Abstract

The application discloses a kind of multi-modal knowledge graph question answering system and method based on space-time perception cognitive agent.The system includes dynamic knowledge graph construction module (1), thinking map reasoning module (2), multi-modal data processing module (3), confidence check module (4) and feedback optimization module (5), and its innovation is: introduce the dynamic weight decay algorithm based on current time, realize the active forgetting and updating of knowledge;Multi-modal mutual verification mechanism of specific scene is constructed, three language mixed understanding and landmark visual retrieval are supported;Establish real-time thinking distillation human-computer coevolution closed loop, realize system-level real-time reasoning logic optimization.The application solves the problems of knowledge update lag, single reasoning path and insufficient multi-modal support in the prior art through deep system-level coupling, and is suitable for intelligent question answering scenarios with regional culture perception ability in specific space-time background.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a multimodal knowledge graph question-answering system, method, electronic device and storage medium based on spatiotemporal perception cognitive intelligent agents, which is particularly suitable for complex question-answering scenarios that require processing multimodal information and have spatiotemporal perception capabilities. Background Technology

[0002] With the development of artificial intelligence technology, retrieval-enhanced generation (RAG)-based knowledge base systems have become the mainstream technology for intelligent question answering. However, existing technologies still have the following shortcomings:

[0003] 1. Lagging Knowledge Updates: Traditional RAG systems mostly use static knowledge bases, which cannot respond promptly to the addition of new knowledge and the updating of old knowledge. The update mechanism of intelligent RAG systems that integrate dynamic knowledge graphs is mainly based on the passive reception of new data streams, lacking the ability to actively metabolize knowledge at the current point in time.

[0004] 2. Single reasoning path: When dealing with complex problems, existing systems often adopt a forward search approach, such as a decision-evaluation collaborative mechanism. This linear search method is inefficient when faced with fuzzy or complex queries and lacks the leap-like "insight" ability of human thinking.

[0005] 3. Insufficient multimodal support: It mainly focuses on the unified representation of data and lacks the ability to handle multilingual mixing and cultural cognition for specific regional scenarios.

[0006] 4. Lack of real-time evolution mechanism: The feedback optimization of existing systems is mostly an offline training process, which cannot achieve system-level evolution based on real-time user interaction and is difficult to adapt to rapidly changing application environments.

[0007] Therefore, there is a need for an intelligent question-answering service with the ability to perceive regional culture and evolve in real time, in order to solve the above-mentioned problems in existing technologies. Summary of the Invention

[0008] The purpose of this invention is to provide a multimodal knowledge graph question-answering system, method, electronic device and storage medium based on spatiotemporal perception cognitive intelligent agents, so as to solve the problems of lagging knowledge updates, low reasoning efficiency and poor scene adaptability in the prior art, and realize intelligent question-answering services with regional cultural perception capabilities and real-time evolution capabilities.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] The first aspect of this invention discloses a multimodal knowledge graph question-answering system based on a spatiotemporally aware cognitive agent, comprising:

[0011] The dynamic knowledge graph construction module (1) is used to preprocess the original data and construct a dynamically updatable knowledge graph. The dynamic knowledge graph construction module (1) includes a spatiotemporal awareness weight calculation unit, which has a built-in current timestamp to calculate the difference between the knowledge node time and the current time, and sets an attenuation coefficient according to the timeliness category of the knowledge node. When the difference exceeds the preset threshold, the weight of the node is automatically reduced and a supplementary search is triggered.

[0012] The mind graph reasoning module (2) is used to generate a reasoning path graph containing hypothetical intermediate nodes and to perform path matching verification using an anchor point backtracking algorithm. The anchor point backtracking algorithm includes: identifying the core semantic anchor in the query and directly predicting the higher-order abstract concept connecting the semantic anchor as a hypothetical intermediate node.

[0013] The multimodal data processing module (3) is used to process various heterogeneous contents such as text, images, tables, and mathematical formulas, and extract multimodal features to construct a unified multimodal knowledge graph;

[0014] The confidence verification module (4) is used to perform logical consistency self-check and cross-modal triangulation verification before outputting the answer. If the confidence is lower than the threshold, it will trigger supplementary retrieval or directly reject the answer.

[0015] The feedback optimization module (5) is used to optimize the system based on user feedback, including adjusting the path generation strategy of the mind graph reasoning module (2) based on the real-time mind distillation algorithm.

[0016] Furthermore, the anchor point backtracking algorithm specifically includes: first, identifying the core semantic anchor points in the problem; then, skipping multiple linear reasoning paths, directly predicting the higher-order abstract concepts that may connect to the anchor points as intermediate hypothesis nodes; then, forward matching the graph data from the hypothesis node and reverse verification of the path feasibility to form a two-way verification.

[0017] Furthermore, the multimodal data processing module (3) integrates a multilingual visual understanding engine, supports simultaneous recognition and understanding of mixed text in at least three languages, and can associate and bind visual landmark image features with geographical location entities in the knowledge graph.

[0018] Furthermore, the feedback optimization module (5) adopts a real-time mind distillation algorithm, using real-time behavioral data during user use as a reward signal to update the weight parameters of the path generation strategy in the mind graph reasoning module (2).

[0019] Furthermore, the confidence verification module (4) performs cross-modal triangulation before outputting the answer: if the search result contains text description and corresponding image data, it uses a visual language model to generate a description of the image and performs a semantic comparison between the generated description and the retrieved text description; if the comparison difference exceeds a preset threshold, it is determined to be a cognitive conflict, and the answer is rejected and the knowledge node is marked as questionable.

[0020] Furthermore, the spatiotemporal perception weight calculation unit adopts an exponential decay formula:

[0021] Calculate the current weight of the knowledge node, where Based on the weights, The attenuation coefficient is set according to the timeliness category of knowledge nodes. This represents the time difference between the current time and the node's timestamp.

[0022] Furthermore, the mind graph reasoning module (2) also includes a hypothetical node library for storing predefined high-order abstract concepts; when generating hypothetical intermediate nodes, the anchor backtracking algorithm obtains high-order abstract concepts that may connect to core semantic anchors from the hypothetical node library as candidate hypothetical nodes.

[0023] A second aspect of the present invention also provides a multimodal knowledge graph question-answering method based on the system described in the first aspect, comprising the following steps:

[0024] Step S1: Receive user query requests and parse the semantic content and multimodal features of the query;

[0025] Step S2: Obtain the current timestamp, evaluate the timeliness of relevant knowledge nodes based on the current time, and trigger supplementary retrieval for outdated nodes;

[0026] Step S3: Use the anchor backtracking algorithm to generate a reasoning path containing hypothetical intermediate nodes, including: identifying the core semantic anchor in the query, predicting the higher-order abstract concept connecting the anchor as a hypothetical node, and then matching the graph data from the hypothetical node in a forward direction.

[0027] Step S4: Retrieve multimodal data based on the inference path and perform cross-modal triangulation to verify the consistency between the text description and the image content;

[0028] Step S5: Determine whether to output the final answer or reject the answer based on the confidence level of cross-modal mutual verification;

[0029] Step S6: Collect user feedback on the answers and update the reasoning path generation strategy based on the real-time thought distillation algorithm.

[0030] A third aspect of the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the method disclosed in the second aspect of the present invention.

[0031] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method disclosed in the second aspect of the present invention.

[0032] This invention has the following significant advantages over the prior art:

[0033] 1. Significantly improved timeliness: By introducing a spatiotemporal decay factor and an active forgetting mechanism, the system can proactively identify and update outdated knowledge, ensuring the up-to-dateness of the output information in the context of time.

[0034] 2. Increased reasoning efficiency: The anchor backtracking algorithm simulates human "insight" thinking, which significantly improves the efficiency of reasoning path search for complex problems compared to the traditional forward search algorithm.

[0035] 3. Enhanced scene adaptability: The processing engine, customized for multilingual and multicultural characteristics, significantly improves the accuracy of the system in specific regions.

[0036] 4. Significantly improved credibility: The cross-modal triangulation mechanism effectively identifies and filters contradictory information, significantly reducing the rate of illusory responses.

[0037] 5. Breakthrough in Evolutionary Capabilities: The real-time thought distillation mechanism enables the system to continuously optimize its reasoning logic with each user interaction, achieving true online learning and evolution. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the overall architecture of the system of the present invention;

[0039] Figure 2 This is a flowchart of the method of the present invention.

[0040] The diagram shows: 1. Dynamic knowledge graph construction module; 2. Mind graph reasoning module; 3. Multimodal data processing module; 4. Confidence verification module; 5. Feedback optimization module. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0042] Example 1

[0043] See Figure 1 The multimodal knowledge graph question-answering system based on spatiotemporal cognitive intelligent agents of the present invention includes the following functional modules:

[0044] 1. Dynamic Knowledge Graph Construction Module 1, used to preprocess raw data and construct a dynamically updatable knowledge graph; this module contains the following sub-units:

[0045] (1) Real-time detection unit: continuously monitors external data sources (such as database updates, web crawlers, user input, etc.) to identify new knowledge and knowledge changes.

[0046] (2) Automatic fusion unit: The online incremental learning technology is used to integrate new knowledge into the existing map in real time, avoiding the computational overhead caused by full reconstruction.

[0047] (3) Spatiotemporal awareness weight calculation unit: It has a built-in current timestamp and calculates the difference between the node timestamp and the current time for each knowledge node in the knowledge graph. The system pre-labels each node with a timeliness category (permanently valid, periodically updated, or immediately expired) and sets different attenuation coefficients according to the category. .

[0048] when When the preset threshold is exceeded, the weight of the node is automatically reduced according to the exponential decay formula:

[0049]

[0050] in This serves as the base weight. Simultaneously, it triggers a knowledge blind spot detection mechanism, proactively initiating supplementary searches and targeting the latest alternative information. For nodes with high time sensitivity, when... When the threshold T is reached, a supplementary search is triggered, and the weights are adjusted accordingly. Calculate; when When T ≤ T, the weight remains unchanged. Through the above mechanism, the system achieves proactive forgetting and updating of knowledge, solving the problem of lagging knowledge updates in existing technologies.

[0051] 2. Mind Graph Reasoning Module 2 is used to generate a reasoning path graph containing hypothetical intermediate nodes and to perform path matching verification using an anchor point backtracking algorithm. This module contains the following sub-units:

[0052] (1) Thinking path generation unit: The reasoning path is generated using the anchor point backtracking algorithm. The anchor point backtracking algorithm specifically includes: firstly, extracting the core semantic anchor point from the user's question, then skipping the multi-step linear reasoning path and directly predicting the higher-order abstract concept that may connect to the anchor point as an intermediate hypothesis node; then, matching the graph data in the forward direction from the hypothesis node and verifying the feasibility of the path in the reverse direction to form a two-way verification.

[0053] The reasoning path is adopted only when both forward matching and backward verification pass. This process forms a closed-loop verification, significantly improving the accuracy of reasoning for complex problems.

[0054] (2) Path matching verification unit: Verify the feasibility of the generated reasoning path.

[0055] (3) Hypothetical Node Library: Stores predefined high-order abstract concepts (such as "natural disasters", "economic policies", "cultural landmarks", etc.) and their semantic vector representations as reasoning springboards. When generating hypothetical intermediate nodes, the anchor backtracking algorithm retrieves high-order abstract concepts that may connect to core semantic anchors from this library as candidate hypothetical nodes.

[0056] 3. Multimodal Data Processing Module 3 is used to process various heterogeneous content such as text, images, tables, and mathematical formulas, and extract multimodal features to construct a unified multimodal knowledge graph. This module includes the following sub-units:

[0057] (1) Visual language processing unit: The pre-trained visual language model (such as CLIP, BLIP, etc.) is used to encode visual information such as text, images, tables, and mathematical formulas, and extract visual feature vectors.

[0058] (2) Multilingual understanding unit: integrates a multilingual visual understanding engine, supporting simultaneous recognition and understanding of mixed text in at least three languages. This engine uses a cross-language pre-trained model, which can process mixed language inputs without language labeling.

[0059] (3) Image-text alignment unit: realizes bidirectional alignment of image and text knowledge, and can associate and bind visual landmark image features with geographical location entities in the knowledge graph.

[0060] 4. Confidence Verification Module 4 is used to perform logical consistency self-checks and cross-modal triangulation verification before outputting a response. If the confidence level is lower than the threshold, supplementary retrieval is triggered or the response is directly rejected. This module contains the following sub-units:

[0061] (1) Logical consistency check unit: verifies the logical rationality of the reasoning path.

[0062] (2) Cross-modal verification unit: performs consistency comparison of multimodal results such as text and images. Specifically, if the search results contain text descriptions and corresponding image data, a visual language model is used to generate a description of the image, and the generated description is semantically compared with the retrieved text description; if the comparison difference exceeds a preset threshold, it is determined to be a cognitive conflict, and the answer is rejected and the knowledge node is marked as questionable;

[0063] Specifically:

[0064] For cases where the search results contain both text T and image I, the following mutual verification process is executed:

[0065] 1) Use a visual language model (such as BLIP) to generate image description text TI for image I;

[0066] 2) Calculate the semantic similarity S between the original text description T and the production description TI;

[0067] 3) If S < threshold θ (e.g., 0.6), it is judged as cognitive conflict, and the information expressed by the text and the image is considered inconsistent.

[0068] 4) At this point, refuse to generate an answer and mark the knowledge node as questionable, and store it in the verification queue for manual or further automatic verification.

[0069] (3) Risk Control Unit: Calculate the overall confidence level based on the results of comprehensive logical consistency checks and cross-modal mutual verification. Trigger supplementary retrieval if the confidence level is insufficient; if the threshold still cannot be reached after supplementary retrieval, directly refuse to answer and prompt the user.

[0070] 5. Feedback Optimization Module 5 is used to optimize the system based on user feedback, including adjusting the path generation strategy of Mind Graph Reasoning Module 2 based on the real-time mind distillation algorithm. This module contains the following sub-units:

[0071] (1) User behavior collection unit: collects user interaction data such as click behavior, dwell time, and correction operations in real time;

[0072] (2) Strategy Optimization Unit: Based on the real-time mind distillation algorithm, the weight parameters of the path generation strategy in the mind graph reasoning module 2 are updated using real-time behavioral data during user interaction as reward signals. The update formula is as follows:

[0073] ,

[0074] in:

[0075] The learning rate (ranging from 0 to 1, and 0.1 in this embodiment).

[0076] Reward users with feedback values ​​(e.g., +1 for clicking "useful", -0.5 for clicking "useless", and 0 for no action).

[0077] The time discount factor is used to make the impact of recent feedback greater than that of long-term feedback (0.95 in this embodiment).

[0078] (3) Model update unit: The system model parameters are updated periodically and fixed into the system model to form long-term memory.

[0079] Example 2

[0080] See Figure 2 A multimodal knowledge graph question answering method based on spatiotemporally aware cognitive agents includes the following steps:

[0081] Step S1: Receive user query requests and parse the semantic content and multimodal features of the query;

[0082] The system accepts user input queries and supports multiple input methods such as text, voice, and images.

[0083] Step S2: Obtain the current timestamp, evaluate the timeliness of relevant knowledge nodes based on the current time, and trigger supplementary retrieval for outdated nodes.

[0084] Step S3: Generate a reasoning path containing hypothetical intermediate nodes using the anchor backtracking algorithm, including:

[0085] Identify the core semantic anchor points in the query. For example, for the question "In which year was this building built?", anchor the anchor points as "the year of initial construction" or "the period of construction" as candidate hypothesis nodes.

[0086] Retrieve high-level abstract concepts related to the semantics of "foundation date" from the hypothetical node library, such as "historical era" and "construction period" as candidate hypothetical nodes.

[0087] Starting from the hypothetical node, we then perform forward matching of the graph data to obtain possible reasoning paths, such as: Potala Palace → Year of its founding → 7th century AD;

[0088] Simultaneously, reverse verification was performed, starting from the answer "7th century AD," to check if the edge connects to "Potala Palace" and "year of its founding." After two-way verification, the reasoning path was confirmed to be valid.

[0089] Step S4: Retrieve multimodal data based on the inference path and perform cross-modal triangulation to verify the consistency between the text description and the image content;

[0090] Based on a defined reasoning path, relevant data is retrieved from a multimodal knowledge graph. If the text description of the retrieval result matches the corresponding image data, cross-modal triangulation is performed.

[0091] Step S5: Determine whether to output the final answer or reject the answer based on the confidence level of cross-modal mutual verification;

[0092] If cross-modal verification passes and logical consistency checks are consistent, a final answer is generated. If the confidence level is below the threshold, a supplementary search is triggered; if the score still fails after the supplementary search, the answer is rejected and the user is notified.

[0093] Step S6: Collect user feedback on the answers and update the reasoning path generation strategy based on the real-time thought distillation algorithm.

[0094] After receiving a response, the user clicks the "Helpful" or "Useless" button, or corrects their input. The system collects these actions as a reward signal R, according to the formula... The weight parameters of the path generation strategy are updated in real time.

[0095] Through continuous iteration of the above steps, the system achieves a complete closed loop from knowledge construction, reasoning, multimodal verification to feedback evolution.

[0096] Example 3

[0097] The present invention also provides an electronic device, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements any one or more steps in the above method embodiments. The electronic device may be a server, a cloud computing node, a personal computer, a smart terminal (such as a smartphone or tablet), or any other device with computing capabilities.

[0098] Example 4

[0099] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any one or more steps in the above method embodiments. The storage medium can be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory, random access memory, magnetic disk, or optical disk.

[0100] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art can make various improvements and modifications without departing from the spirit and principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal knowledge graph question-answering system based on spatiotemporal cognitive intelligent agents, characterized in that, include: The dynamic knowledge graph construction module (1) is used to preprocess the raw data and construct a dynamically updatable knowledge graph; The dynamic knowledge graph construction module (1) includes a spatiotemporal perception weight calculation unit, which has a built-in current timestamp to calculate the difference between the knowledge node time and the current time, and sets an attenuation coefficient according to the timeliness category of the knowledge node. When the difference exceeds the preset threshold, the weight of the node is automatically reduced and a supplementary search is triggered. The mind graph reasoning module (2) is used to generate a reasoning path graph containing hypothetical intermediate nodes and to perform path matching verification using an anchor point backtracking algorithm. The anchor point backtracking algorithm includes: identifying the core semantic anchor in the query and directly predicting the higher-order abstract concept connecting the semantic anchor as a hypothetical intermediate node. The multimodal data processing module (3) is used to process various heterogeneous contents such as text, images, tables, and mathematical formulas, and extract multimodal features to construct a unified multimodal knowledge graph; The confidence verification module (4) is used to perform logical consistency self-check and cross-modal triangulation verification before outputting the answer. If the confidence is lower than the threshold, it will trigger supplementary retrieval or directly reject the answer. The feedback optimization module (5) is used to optimize the system based on user feedback, including adjusting the path generation strategy of the mind graph reasoning module (2) based on the real-time mind distillation algorithm.

2. The system according to claim 1, characterized in that, The anchor point backtracking algorithm of the mind graph reasoning module (2) specifically includes: first, identifying the core semantic anchor point in the problem, then skipping the multi-step linear reasoning path, directly predicting the higher-order abstract concept that may connect to the anchor point as an intermediate hypothesis node, then matching the graph data from the hypothesis node in the forward direction, and verifying the feasibility of the path in the reverse direction to form a two-way verification.

3. The system according to claim 1, characterized in that, The multimodal data processing module (3) integrates a multilingual visual understanding engine, supports simultaneous recognition and understanding of mixed text in at least three languages, and can associate and bind visual landmark image features with geographical location entities in the knowledge graph.

4. The system according to claim 1, characterized in that, The feedback optimization module (5) uses a real-time mind distillation algorithm to update the weight parameters of the path generation strategy in the mind graph reasoning module (2) by using real-time behavioral data during user use as a reward signal.

5. The system according to claim 1, characterized in that, The confidence verification module (4) performs cross-modal triangulation before outputting the answer: if the search result contains text description and corresponding image data, it uses a visual language model to generate a description of the image and performs a semantic comparison between the generated description and the retrieved text description; if the comparison difference exceeds a preset threshold, it is determined to be a cognitive conflict, and the answer is rejected and the knowledge node is marked as questionable.

6. The system according to claim 1, characterized in that, The spatiotemporal perception weight calculation unit adopts an exponential decay formula: Calculate the current weight of the knowledge node, where Based on weights, The attenuation coefficient is set according to the timeliness category of knowledge nodes. This represents the time difference between the current time and the node's timestamp.

7. The system according to claim 1, characterized in that, The mind graph reasoning module (2) also includes a hypothetical node library for storing predefined high-order abstract concepts; when generating hypothetical intermediate nodes, the anchor backtracking algorithm obtains high-order abstract concepts that may connect to core semantic anchors from the hypothetical node library as candidate hypothetical nodes.

8. A multimodal knowledge graph question-answering method based on the system described in any one of claims 1-7, characterized in that, Includes the following steps: Step S1: Receive user query requests and parse the semantic content and multimodal features of the query; Step S2: Obtain the current timestamp, evaluate the timeliness of relevant knowledge nodes based on the current time, and trigger supplementary retrieval for outdated nodes; Step S3: Use the anchor backtracking algorithm to generate a reasoning path containing hypothetical intermediate nodes, including: identifying the core semantic anchor in the query, predicting the higher-order abstract concept connecting the anchor as a hypothetical node, and then matching the graph data from the hypothetical node in a forward direction. Step S4: Retrieve multimodal data based on the inference path and perform cross-modal triangulation to verify the consistency between the text description and the image content; Step S5: Determine whether to output the final answer or reject the answer based on the confidence level of cross-modal mutual verification; Step S6: Collect user feedback on the answers and update the reasoning path generation strategy based on the real-time thought distillation algorithm.

9. An electronic device comprising a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the method of claim 8.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of claim 8.

Images