Low-altitude economic intelligent question answering method and system based on multi-modal time sequence knowledge graph

Abstract

The invention relates to a low-altitude economic intelligent question-answering method and system based on a multi-modal time sequence knowledge graph, and the method comprises the steps: constructing a multi-modal time sequence knowledge graph reasoning question-answering model MTQA based on time perception constraint, and enabling the MTQA model to comprise a multi-modal time sequence knowledge graph embedding module MMTE, the knowledge graph is constructed by integrating multi-modal information such as texts, images and videos and fusing time sequence dimensions, the semantic relation and time sequence relation of entities are clearly expressed, a more accurate knowledge system is achieved, the semantic relation between knowledge is clearly expressed, and the problems of knowledge dispersion, poor relevance and the like are effectively solved. In the subgraph extraction of the MTQA inference model, a subject entity is used as an anchor fusion subgraph, and after time-aware constraint pruning, a time sequence graph neural network is used for transmitting time sequence correlation, balancing scale and key information retention. The processing flow of the intelligent question and answer system is developed around a multi-modal time sequence knowledge graph module and an intelligent question and answer module. According to the method and the system, the accuracy and the comprehensiveness of semantic matching are obviously improved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and in particular to a low-altitude economic intelligent question-answering method and system based on a multimodal temporal knowledge graph. Background Technology

[0002] With the rapid development of the low-altitude economy, the industry's demand for professional knowledge has surged. However, newcomers to the industry or learners generally face difficulties in acquiring knowledge, as well as fragmented and poorly correlated information. Although some information query tools are currently available for the low-altitude economy, they have significant shortcomings. Specifically, they are limited in modal resources, mainly text-based, and cannot integrate multimodal information such as low-altitude flight route data, policy documents, and equipment operation videos. This makes it difficult to meet diverse knowledge needs. When identifying knowledge entities such as "specific airspace take-off and landing points" and "drone flight suitability rules," they struggle to effectively capture subtle changes in entity boundaries, resulting in low information matching accuracy. Furthermore, they lack the ability to integrate temporal information, making it impossible to adapt to dynamic changes in knowledge such as policy updates and flight route adjustments. At the same time, when facing professional questions such as "low-altitude logistics approval process" and "drone safety operation specifications," semantic similarity calculation is complex, leading to insufficient accuracy and efficiency in question-and-answer responses.

[0003] Knowledge graphs are generally represented using a triple structure, with each triple containing a subject, predicate, and object, used to characterize the relationships between entities. In practical applications, the structure between entity pairs in a knowledge graph changes over time, leading to incompleteness and inadequacy in the knowledge graph system. To obtain a more refined knowledge system, temporal knowledge graphs are introduced, associating each fact with a constraint timestamp to further analyze the trend of topological changes over time. Multimodal temporal knowledge graphs extend static knowledge graphs in the time dimension, increasing triples to quadruples. It includes not only subject entities, relations, and object entities, but also the timestamps or time ranges contained in the relations, effectively improving the efficiency of knowledge question answering.

[0004] Compared to traditional search systems, AI-powered question-answering systems can directly provide answers to users' natural language questions, rather than providing webpage links related to the question's keywords. In the field of multimodal feature extraction and fusion, various algorithms and models have been explored. Examples include multimodal fusion recommendation algorithms, the cross-attention fusion strategy TMFISA, and the entity alignment model EAFE.

[0005] Knowledge reasoning is a technique that utilizes nodes (entities) and edges (relationships) in knowledge graphs to perform logical reasoning, path querying, and other operations. In particular, reasoning based on multimodal dynamic knowledge graphs can construct dynamically evolving knowledge networks and resolve complex knowledge relationships across different points in time. Existing technical solutions in this field include adaptive graph neural networks, the Att-HousE model, and the DL+ two-layer reasoning framework.

[0006] The existing technology still has the following shortcomings.

[0007] 1. At the semantic understanding level, deep semantic parsing of complex questions remains significantly challenging. Complex questions posed by users often contain ambiguous expressions, implicit logical connections, or multiple overlapping intentions. Existing models mostly rely on surface text feature matching, making it difficult to accurately capture the true semantic needs behind the questions. This can easily lead to problems such as entity misidentification and relationship mismatch, creating potential errors for subsequent reasoning stages.

[0008] 2. At the knowledge representation level, the optimal solution for selecting subgraphs and weighing candidate entities in knowledge graphs has not yet been found. To cover the reasoning paths of complex problems, relevant subgraphs need to be extracted from massive amounts of knowledge. However, excessively large subgraph sizes can lead to a surge in computational costs and a decrease in reasoning efficiency. On the other hand, excessively compressing the scope of subgraphs may result in the omission of key entities and relationships, leading to an incomplete set of candidate entities and thus affecting the comprehensiveness and accuracy of the reasoning results.

[0009] 3. At the feature processing level, multimodal information fusion and implicit time constraint recognition remain weak links. On the one hand, existing models lack the ability to integrate multimodal information and struggle to effectively associate visual features, speech features, and structured information from knowledge graphs. On the other hand, many complex questions contain implicit time constraints, which are not presented as explicit time words. The models struggle to actively identify these constraints and transform them into reasoning conditions, resulting in low accuracy of time-related answers. Summary of the Invention

[0010] To address the shortcomings of existing technologies, this invention proposes a low-altitude economic intelligent question-answering method based on a multimodal temporal knowledge graph. The question-answering method constructs a time-aware, time-series knowledge graph reasoning question-answering model (MTQA). The MTQA reasoning model includes a multimodal temporal knowledge graph embedding module (MMTE), which integrates multimodal information such as text, images, and videos and incorporates a temporal dimension to construct a knowledge graph that clearly represents the semantic and temporal relationships of entities. The subgraph extraction of the MTQA reasoning model fuses subgraphs using topic entities as anchors. After time-aware constraint pruning, a temporal graph neural network is used to transmit temporal associations, balancing scale with the preservation of key information. Specifically, this includes:

[0011] Step 1: Collection and preprocessing of text, image, and video datasets. First, define the data schema, crawl policy data related to each item in the policy list and clean it, then construct policy quadruple data. The quadruple data includes entity-attribute-attribute value-timestamp and entity-relationship-entity-timestamp.

[0012] Step 2: Based on the quadruple data, construct a multimodal temporal knowledge graph;

[0013] Step 3: Knowledge retrieval. Based on the constructed multimodal temporal knowledge graph and MTQA inference model, the system queries the policies input by the user and returns the corresponding attribute values ​​and partial relationship displays, as well as related images and videos.

[0014] Step 31: Based on the constructed knowledge graph, accurately retrieve the quadruple knowledge data generated by the MTQA model, transform fragmented reasoning knowledge into structured and understandable policy information, and at the same time associate it with supporting multimodal materials to provide users with policy knowledge services that are both accurate and complete.

[0015] Step 32: Build a retrieval interaction layer to receive policy query keywords or statements input by users, and after text preprocessing, connect to the retrieval interface of the knowledge graph;

[0016] Step 33: Based on the MTQA model, reason about the structural features of the quadruplets, construct targeted retrieval rules, and map the user's query intent to the retrieval dimensions of the quadruplets in the graph;

[0017] Step 34: Call the Neo4j query interface and execute a customized Cypher statement using the Py2neo library to retrieve MTQA inference quadruple data that match the query from the knowledge graph, and extract the policy attribute values, inter-policy relationships, and associated multimedia identifiers.

[0018] Step 35: Parse and reconstruct the retrieved MTQA inference quadruple data. The policy attribute values ​​are organized in key-value pairs, and the policy relationships are displayed in a visual graph fragment. At the same time, retrieve the corresponding policy interpretation images and implementation case video materials according to the multimedia identifiers.

[0019] Step 36: Integrate the parsed quadruple knowledge, relational visualizations, and multimodal materials, and return them to the user through the front-end interface. Among them, the core quadruple information of MTQA reasoning is displayed as the core module to ensure that the user can clearly obtain the key reasoning knowledge of the policy.

[0020] Step 4: Construct an intelligent question-answering model, which includes the MTQA model and the Zidong Changkong low-altitude large model. The MTAQ model includes a question embedding module, a time-aware constraint subgraph extraction module, a multimodal temporal knowledge graph embedding module (MMTE), and an information aggregation and answer prediction module. The two work together to achieve accurate question answering in the low-altitude economic field.

[0021] Step 41: The MTQA model goes through four stages: question embedding, subgraph extraction, knowledge graph embedding, and answer prediction, to infer highly relevant vertical domain knowledge from the low-altitude economic knowledge base;

[0022] Step 411: The question embedding module preprocesses the input question text and then inputs it into the pre-trained language model to output the question embedding vector Q. T ;

[0023] Step 412: The time-aware constraint-based subgraph extraction module runs, which includes three stages: subgraph extraction, subgraph pruning, and temporal graph neural network modeling. First, entity and time information of the input question are extracted and neighbor subgraphs are constructed. Then, the subgraph is pruned by selecting the time constraint acquisition method to obtain the answer subgraph. Finally, the subgraph structure and time information are mined by the temporal graph neural network TGNN to output the answer subgraph information of the question.

[0024] Step 413: The knowledge graph embedding stage is implemented through the MMTE module, including four stages: multimodal embedding extraction and alignment, adaptive fusion and expert mechanism, time-modal interaction modeling, and multimodal quadruple scoring. In the first stage, the textual, visual, structural, and temporal embeddings of entities are extracted and aligned to a unified complex representation space. In the second stage, a multi-expert learning mechanism is introduced to extract expert features to obtain expert-driven entity embeddings. In the third stage, complex embeddings of entities, relations, and timestamps are generated based on the TComplEx algorithm. The multimodal features are mapped to the complex domain through the expert-driven entity embedding to obtain multimodal enhanced embeddings. In the fourth stage, the score is calculated based on the quadruple scoring formula.

[0025] Step 414: Information aggregation and answer prediction module, input question embedding vector and subgraph information, generate enhanced question representation based on stacking operation, then output entity prediction vector and timestamp prediction vector through two layers of MLP, and then obtain probability distribution through Softmax function;

[0026] Step 42: Construct the Zidong Changkong Low-Altitude Large Model Question Answering Module, set up in the backend of the MTQA inference model, and design four question answering processes including knowledge input, structured design of prompt words, parameter configuration and answer generation. Through these four question answering processes, the vertical domain knowledge output by the MTQA inference model is transformed into natural language answers.

[0027] Step 43: Train the MTQA inference model using a publicly available question-answering dataset. Optimize the model using the cross-entropy loss function and iterate until the entity prediction accuracy and timestamp prediction recall of the inference model on the validation set stabilize. Stop training at this point.

[0028] According to a preferred embodiment, the three-stage processing of the subgraph extraction module specifically includes:

[0029] Subgraph extraction stage: For the input problem, extract the entire set of topic entities, retrieve all related facts for each entity in the temporal knowledge graph, extract relevant entity and time information, construct an m-hop neighbor subgraph, merge duplicate entities in all neighbor subgraphs, and construct a fusion subgraph through bidirectional expansion;

[0030] Subgraph pruning stage: Select the time constraint acquisition method through gating mechanism, merge all timestamps of related entities and sort them, take the minimum value as the start time and the maximum value as the end time to obtain the time range; prune the edges and nodes in the fused subgraph that do not meet the time constraint, and finally obtain the answer subgraph;

[0031] The temporal graph neural network modeling stage involves using TGNN to mine subgraph structure and temporal information, calculating the correlation scores between adjacent nodes, updating node representations through an attention mechanism, and finally obtaining the subgraph information Q of the problem through average pooling. G .

[0032] According to a preferred embodiment, the four-stage processing procedure of the MMTE module includes:

[0033] Multimodal embedding extraction and alignment stage: Extract text embedding, visual embedding, structural embedding and temporal embedding of entities respectively, and align the four types of embedding to a unified complex representation space through modality-specific MLP mapping function;

[0034] Adaptive fusion and expert mechanism stage: Text and visual embeddings are fused through a gating function, a multi-expert learning mechanism is introduced, expert features are extracted through a feature extraction function, and a weighted sum is calculated by combining gating weights to obtain expert-driven entity embeddings;

[0035] The time-modal interaction modeling stage: Based on the TComplEx algorithm, complex embeddings of entities, relations, and timestamps are generated. Through expert-driven entity embedding, multimodal features are mapped to the complex domain, and multimodal enhanced embeddings are obtained, including entities, relations, and time.

[0036] Multimodal quadruplet scoring stage: Scores are calculated using the quadruplet scoring formula, and the embedding quality is optimized using a loss function.

[0037] According to a preferred embodiment, step 42, the four question-and-answer processes of the Zidong Changkong Low-Altitude Large Model Question-and-Answer Module, specifically include:

[0038] Knowledge input stage: The MTQA model is invoked to extract domain knowledge highly relevant to the user's question from the low-altitude economic knowledge base, including quadruple reasoning results, entity attributes, and relationships;

[0039] Structured design of prompt words: The prompt content is integrated with the user's original question and the reasoning knowledge of the MTQA model. The prompt content consists of two parts: first, question guidance: clarifying the direction of the query to avoid misunderstanding of intent; second, domain knowledge anchors: marking key reasoning knowledge to constrain the professional scope of the answer.

[0040] Parameter configuration: Set the model temperature coefficient parameter to 0.1 to reduce output randomness and prioritize high-probability semantic sequences;

[0041] Answer generation: Based on the prompts and parameter configurations, the Zidong Changkong Low-Altitude Big Data Model generates natural language answers that conform to the standards of the low-altitude economic field and returns them to the user.

[0042] A low-altitude economic intelligent question-answering system based on a multimodal temporal knowledge graph, wherein the intelligent question-answering system is used to implement the method described in claim 1, and the system includes at least: a multimodal data acquisition and preprocessing module, a knowledge graph construction module, a knowledge retrieval module, a multimodal temporal knowledge graph module, an intelligent question-answering module, and a reasoning module, wherein,

[0043] The multimodal data acquisition and preprocessing module is used to acquire and preprocess text, image, and video data. First, it defines the data pattern, crawls the policy data related to each item in the policy list and cleans the data, and then constructs the policy quadruple data.

[0044] The knowledge graph construction module is used to create nodes and relationships through the local Neo4j service and store knowledge point quadruple data in the Neo4j graph database;

[0045] The knowledge retrieval module is used to query policy questions entered by users. The intelligent question-answering system returns attribute values, relationship displays, and video displays corresponding to the user's input.

[0046] The multimodal temporal knowledge graph module is used to align text, visual, structural and timestamp embeddings to a unified complex representation space. It introduces a multi-expert learning mechanism to extract expert features and maps multimodal features to the complex domain. The resulting multimodal enhanced embeddings are then aggregated and scores are calculated based on the quadruplet scoring formula.

[0047] The intelligent question-answering module includes a reasoning module, which takes the user's question and infers highly relevant vertical domain knowledge from the low-altitude economic knowledge base. This knowledge is then integrated with the user's original question and input into the Zidong Changkong low-altitude big model. Finally, the big model outputs an intelligent answer.

[0048] The intelligent question-answering system's processing flow revolves around two main modules: a multimodal temporal knowledge graph and intelligent question answering. First, the user inputs a question, which is simultaneously transmitted to both modules. In the multimodal temporal knowledge graph module, knowledge is first embedded to transform it into a graph-recognizable form. Then, based on the question, subgraph retrieval extracts related knowledge subgraphs. Next, subgraph processing filters effective information, ultimately yielding an answer subgraph which is then transmitted to the prediction and generation stage of the intelligent question-answering module. In the intelligent question-answering module, entity recognition is performed on the user's question to extract key entities. Then, question embedding is completed based on these entities. Subsequently, prediction and generation are performed based on the answer subgraph to obtain preliminary answer information. Afterward, the question-answering model enters the stage where information is structured through prompt word engineering, answer generation is performed, and finally, an enhanced answer is completed, outputting the final user response.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0050] The reasoning question answering method MTQA used in this invention can improve efficiency by embedding user questions into a pre-trained language model;

[0051] 1. The knowledge graph embedding module MMTE proposed in this invention is based on TComplEx to extend multimodal and temporal modal interaction. It uses time sequence to assist tasks in giving timestamp temporal awareness, ensuring the consistency of multimodal semantics in the temporal dimension. By extending the multimodal-temporal modal interaction mechanism through the complex-valued embedding framework of TComplEx, the knowledge graph embedding can simultaneously capture the multimodal semantic features and temporal dependencies of entity relationships. After the temporal awareness is given, the consistency matching accuracy of multimodal semantics in the time dimension is improved, effectively reducing the semantic offset error across time steps, and improving the degree of compliance of the embedding vector with temporal constraints.

[0052] 2. The subgraph extraction proposed in this invention uses the main entity as the anchor point to fuse subgraphs. After time-aware constraint pruning, a temporal graph neural network is used to transmit temporal associations, balancing scale and key information retention. Under the premise of subgraph scale compression of 40%-60%, the key temporal information retention rate reaches more than 85%, effectively solving the contradiction between "scale redundancy" and "information loss" in traditional subgraph extraction.

[0053] 3. The information fusion of this invention integrates MMTE's multimodal temporal embedding and DistilBERT's question context embedding through a multi-layer Transformer knowledge fusion layer. Adaptive fusion achieves deep semantic collaboration, generating enhanced question representations. While achieving deep semantic collaboration, it improves the semantic similarity matching accuracy of the enhanced question representations by 20%-27%. Compared with simple concatenation or weighted summation strategies, the fusion method of this invention can more accurately capture the semantic alignment relationship between multimodal temporal knowledge and natural language questions. Attached Figure Description

[0054] Figure 1 This is a flowchart of the intelligent question-answering method of the present invention;

[0055] Figure 2 This is a detailed structural diagram of the MTQA model of the present invention;

[0056] Figure 3 This is the specific structure of the MMTE module of the present invention;

[0057] Figure 4 This is a flowchart of the intelligent question-and-answer process of the present invention;

[0058] Figure 5 This is a structural diagram of the intelligent question-and-answer system of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0060] The following is a detailed explanation with reference to the accompanying drawings.

[0061] The MTQA of this invention refers to: Multi-Modal Temporal Knowledge Graph Reasoning for Question Answering based on time-aware constraints.

[0062] In this invention, MMTE refers to: Multi-Modal TComplEx, a multimodal temporal knowledge graph embedding module.

[0063] In this invention, TGNN refers to Temporal Graph Neural Network.

[0064] The MLP in this invention refers to a multilayer perceptron.

[0065] The TComplEx of this invention is an embedding algorithm commonly used in time-series knowledge graphs. It captures the complex relationships between entities, relations and timestamps through complex space modeling.

[0066] To address the problems of existing technologies, this invention proposes a low-altitude economic question-answering method and system based on a multimodal temporal knowledge graph. By integrating multimodal information such as text, airspace data, and video, and incorporating a temporal dimension, a knowledge graph is constructed. This graph clearly represents the semantic relationships and temporal associations between low-altitude economic knowledge, overcoming the shortcomings of existing tools and providing newcomers to the industry with a precise and efficient way to acquire knowledge. The multimodal temporal knowledge graph, by integrating information from multiple modalities such as text and video and introducing temporal attributes, possesses a more accurate knowledge system and clearly represents the semantic relationships between knowledge points, effectively improving problems such as knowledge dispersion and weak correlation. The AI ​​intelligent question-answering system uses the multimodal temporal knowledge graph as its core technology, automatically generating answers by fusing multimodal data and knowledge reasoning techniques. The application of multimodal information can mitigate the impact of outliers and missing data, improving the reliability and robustness of predictions.

[0067] This invention builds an AI-powered intelligent question-answering system for low-altitude economic policies based on a multimodal temporal knowledge graph and the Zidong Changkong low-altitude large-scale model. First, entity and relational data patterns are defined, policy-related data is crawled and cleaned, and quadruples are constructed. This data is then imported into Neo4j to form a knowledge graph containing 1938 policies and 7829 relations. Next, the question-answering accuracy is optimized using the MTQA inference model (combining dataset training, time-aware subgraph extraction, and MMTE multimodal embedding). Finally, the question-answering system is developed to provide real-time information assistance to industry newcomers.

[0068] Step 1: Collection and preprocessing of text, image, and video datasets. First, define the data pattern, crawl policy data related to each item in the policy list and clean it, and then construct policy quadruple data.

[0069] Step 11: Define the data schema and divide the entities in the knowledge graph into seven types: "Policy Documents", "Drone Technology", "Government Departments", "Enterprises", "Cities", "Application Scenarios", and "Technical Functions". Define the relationships as "Issuing Agency", "Associated Technology", "Production", "Pilot Cities", "Located in", "Applied to", and "Possessing Functions".

[0070] Step 12: Save the policy documents into the policy list according to the official website policy list.

[0071] Step 13: Using web crawling technology, visit the Baidu Encyclopedia page and the official website in sequence, crawl data for each entry in the policy list, obtain policy-related concepts, and save the crawled data into a CSV file.

[0072] Step 14: Perform data cleaning on the crawled data to ensure the accuracy and usability of the information.

[0073] Step 15: Construct detailed policy quadruple data, including entity-attribute-attribute value-timestamp and entity-relationship-entity-timestamp.

[0074] The knowledge graph construction of this invention includes not only the subject entity, relation and object entity of the quadruple, but also the timestamp or time range contained in the relation. By constructing a knowledge graph of quadruples, the efficiency of knowledge question answering can be effectively improved.

[0075] Step 2: Construct a multimodal temporal knowledge graph.

[0076] Step 21: Use cmd to type "neo4j console" to start the local Neo4j service.

[0077] Step 22: Install the Py2neo library and use the Py2neo.Graph class to create a new connection.

[0078] Step 23: Create nodes and relationships, and store the quadruple data obtained in Step 5 in the Neo4j graph database. Since time information has been added to the quadruple data, the resulting knowledge graph not only contains multimodal information but also temporal information, which helps improve the accuracy of entity matching in subsequent large models.

[0079] Step 24: Import the knowledge graph. There are 1938 policies in total, and there are 7829 relationships between the policies.

[0080] Step 3: Knowledge retrieval. Based on the constructed multimodal temporal knowledge graph and MTQA inference model, the system queries the policies entered by the user and returns the corresponding attribute values ​​and partial relationship displays, as well as related images and videos.

[0081] Step 31: Based on the constructed knowledge graph, accurately retrieve the quadruple knowledge data generated by the MTQA model to quickly respond to users' policy query needs, transform fragmented reasoning knowledge into structured and understandable policy information, and at the same time associate it with supporting multimodal materials to provide users with policy knowledge services that are both accurate and complete.

[0082] Step 32: Build a retrieval interaction layer to receive policy query keywords or statements input by users, and after text preprocessing, connect to the retrieval interface of the knowledge graph;

[0083] Step 33: Based on the MTQA model, reason about the structural features of the quadruplets, construct targeted retrieval rules, and map the user's query intent to the retrieval dimensions of the quadruplets in the graph;

[0084] Step 34: Call the Neo4j query interface and execute a customized Cypher statement using the Py2neo library to retrieve MTQA inference quadruple data that match the query from the knowledge graph, and extract the policy attribute values, inter-policy relationships, and associated multimedia identifiers.

[0085] Step 35: Parse and reconstruct the retrieved MTQA inference quadruple data. The policy attribute values ​​are organized in the form of key-value pairs, and the policy relationships are displayed in the form of visual graph fragments. At the same time, retrieve the corresponding policy interpretation images, implementation case videos, and other materials according to the multimedia identifiers.

[0086] Step 36: Integrate the parsed quadruple knowledge, relational visualizations, and multimodal materials, and return them to the user through the front-end interface. Among them, the core quadruple information of MTQA reasoning is displayed as the core module to ensure that the user can clearly obtain the key reasoning knowledge of the policy.

[0087] Step 4: Construct an intelligent question answering model. The intelligent question answering model includes the Temporal Knowledge Graph Question Answering Inference Model (MTQA) and the Zidong Changkong Low-Altitude Large Model. The MTAQ inference model includes a question embedding module, a time-aware constraint subgraph extraction module, a multimodal temporal knowledge graph embedding module (MMTE), and an information aggregation and answer prediction module. The two work together to achieve accurate question answering in the low-altitude economic field.

[0088] Figure 2 This is a structural diagram of the MTQA inference model. The following is a combination of... Figure 2 The technical principles and processing methods are described in detail.

[0089] Step 41: The MTQA inference model, through four stages—question embedding, subgraph extraction, knowledge graph embedding, and answer prediction—infers highly relevant vertical domain knowledge from the low-level economic knowledge base. Using vertical domain knowledge effectively avoids the generalization problem of generalized large models, ensuring that the answers conform to the low-level domain and have better matching.

[0090] Step 411: The question embedding module preprocesses the input question text and then inputs it into the pre-trained language model to output the question embedding vector Q. T Specifically, this includes:

[0091] The system preprocesses the text of user-submitted questions by inserting a [CLS] tag at the beginning of the text, identifying all subject entities and replacing them with [MASK], such as "Sichuan Province" and "Low-altitude economy related policies".

[0092] The pre-processed question text is input into the pre-trained DistilBERT model. The semantic matrix QR is generated, and the embedding corresponding to the [CLS] tag is selected as the global representation q of the question.

[0093] The subsequent role of the semantic matrix QR is to provide basic semantic support for the construction of question embeddings. The semantic matrix QR contains the semantic embeddings of all tokens in the question text, which can provide textual semantic basis for entity recognition and time information extraction in the subsequent subgraph extraction stage. It helps to accurately locate the topic entities and time range constraints in the question, and ensures that the time-aware constraint-based subgraph extraction module can efficiently filter out knowledge subgraphs that are highly relevant to the question.

[0094] To supplement entity and time embedding, entity information is masked by entity embedding from a multimodal temporal knowledge graph. If there are two topic entities, then Qe = q[CLS] + e1 + e2. The time range of entity association facts is extracted from the knowledge graph, and the sorted start time t1 and end time t2 are taken, and finally integrated into Q. T =q[CLS]+e1+e2+t1+t2 composite problem embedding vector Q T This is used for subsequent subgraph modeling and feature fusion.

[0095] Step 412: The time-aware constraint-based subgraph extraction module runs, including three stages: subgraph extraction, subgraph pruning, and temporal graph neural network modeling;

[0096] Subgraph extraction phase: For the input problem (Qt), extract the set of all topic entities Ent1, Ent2, ..., Ent i ; for each entity i Retrieve all related facts from the temporal knowledge graph, extract relevant entity and time information, and construct an m-hop neighbor subgraph GEnt. i Merge duplicate entities in all neighboring subgraphs and construct a fused subgraph GS through bidirectional expansion. Bidirectional expansion includes: tracing back the original entity and extending related entities.

[0097] The neighbor subgraph is defined as shown in the formula GEnti(m)=(e,r)|e∈Nm(Enti),r∈R(Enti(m))).

[0098] Subgraph pruning stage: Select the time constraint acquisition method through a gating mechanism (direct extraction when the graph is complete, supplementation through reasoning when incomplete, weight p∈{0,1}); merge all timestamps of associated entities and sort them, take the minimum value t1 as the start time and the maximum value t2 as the end time to obtain the time range [t1,t2]; prune the edges and nodes in the fused subgraph that do not satisfy the time constraint, and finally obtain the answer subgraph GA;

[0099] The temporal graph neural network modeling stage involves using TGNN to mine subgraph structure and temporal information, calculating the correlation scores between adjacent nodes, updating node representations through an attention mechanism, and finally obtaining the subgraph information Q of the problem through average pooling.G .

[0100] The mathematical expression for calculating the correlation score between adjacent nodes is as follows:

[0101]

[0102] in, This represents the entity of the previous layer. For relational embedding, Embed the timestamp.

[0103] Step 413: The knowledge graph embedding stage is implemented through the Multimodal Temporal Knowledge Graph Embedding Module (MMTE), which includes four stages: multimodal embedding extraction and alignment, adaptive fusion and expert mechanism, temporal-modal interaction modeling, and multimodal quadruple scoring. For example... Figure 3 The diagram shown is a schematic of the specific structure of the MMTE module.

[0104] Multimodal embedding extraction and alignment stage: Extract text embedding, visual embedding, structural embedding and temporal embedding of entities respectively, and align the four types of embedding to a unified complex representation space through modality-specific MLP mapping function;

[0105] Adaptive fusion and expert mechanism stage: Text and visual embeddings are fused through a gating function to obtain the text and visual embedding representation ef. A multi-expert learning mechanism is introduced, which extracts expert features through a feature extraction function and combines them with gating weights. Calculate the weighted sum to obtain the expert-driven entity embedding. By fusing multimodal embeddings through an expert-based fusion mechanism, optimal feature integration is ensured.

[0106] The mathematical expression for the gate function is as follows: ,in , For learnable weights, satisfying + = The mathematical expression for ef: The mathematical expression for the feature extraction function: The mathematical expression for the gating weights: Mathematical table expression for expert-driven entity embedding: Where K represents the total number of expert networks designed in the multi-expert learning mechanism.

[0107] Time-modal interaction modeling phase: Complex embeddings of entities, relations, and timestamps are generated based on the TComplEx algorithm, and entity embedding is driven by experts. Multimodal features are mapped to the complex domain, resulting in enhanced multimodal embeddings that include entities, relations, and time. Among these, entities... ,relation ,time Embedding takes into account the temporal changes of multimodal data to enhance relational reasoning.

[0108] Multimodal quadruple scoring stage: Scores are calculated using the quadruple scoring formula, and the embedding quality is optimized using a loss function. The core function of this scoring is to quantify the semantic rationality of quadruples in the multimodal temporal knowledge graph, distinguish between positive and negative quadruples, and provide a basis for subsequent loss function calculation.

[0109] The four-tuple scoring formula is as follows: ModalAlign is the text-image similarity alignment term, and its loss function is expressed mathematically as follows: , For modal alignment loss, For time-modal smoothing loss.

[0110] Step 414: Information aggregation and answer prediction module, input question embedding vector Q F Subgraph Information Q G Stacked operations generate an enhanced problem representation Q. F Then, the entity prediction vector is output through two layers of MLP. and timestamp prediction vector The probability distribution is then obtained through the Softmax function. The core function of this probability distribution is to quantify the confidence level of candidate entities and timestamps as true answers, transforming the model's output prediction vector into a normalized score that conforms to probabilistic statistics, intuitively reflecting the matching priority of different candidate answers. During model training, this probability distribution directly affects the calculation of the cross-entropy loss function.

[0111] Constructing a knowledge fusion layer composed of Transformer encoders ( ), input problem embedding vector Q T Subgraph Information Q G Stacking operations are performed using a stacking operation function to generate an enhanced problem representation Q. F .

[0112] The mathematical expression for the stacking operation function is: and .

[0113] Employing a two-layer MLP architecture, through and Output the entity prediction vector and the timestamp prediction vector;

[0114] Define the scoring function and Calculate candidate scores, obtain probability distributions using Softmax, and optimize model prediction accuracy using cross-entropy loss.

[0115] Step 42: Construct the Zidong Changkong low-altitude large-scale model question-and-answer module, set up on the backend of the MTQA inference model, and design four question-and-answer processes including knowledge input, structured design of prompt words, parameter configuration and answer generation. By passing through these four processes, the vertical domain knowledge output by the MTQA inference model is transformed into natural language answers, avoiding the generalization problem of general large-scale models and improving the matching degree and rigor of answers in the low-altitude economic field.

[0116] Knowledge input stage: The MTQA reasoning model is invoked to extract domain knowledge highly relevant to the user's question from the low-altitude economic knowledge base, including quadruple reasoning results, entity attributes, and relationships;

[0117] Structured prompt design: The prompt content integrates the user's original question with MTQA reasoning knowledge and consists of two parts: first, question guidance: clarifying the direction of the query to avoid misunderstanding of intent; second, domain knowledge anchors: marking key reasoning knowledge to constrain the professional scope of the answer.

[0118] Parameter configuration: Set the model temperature coefficient parameter to 0.1 to reduce output randomness, prioritize high-probability semantic sequences, and ensure the rigor and consistency of the answers;

[0119] Answer generation: Based on the prompts and parameter settings, Zidong Changkong's low-altitude big model generates natural language answers that conform to the norms of the low-altitude economic field and returns them to the user, achieving synergistic effects of "domain knowledge + big model capabilities".

[0120] Step 43: Train the MTQA inference model using a publicly available question-answering dataset. Optimize the model using the cross-entropy loss function and iterate until the entity prediction accuracy and timestamp prediction recall of the inference model on the validation set stabilize. Stop training at this point.

[0121] For dataset preparation, training data was obtained by expanding upon two types of publicly available authoritative datasets: Wikidata-Time, QA, and EventKG-QA. The datasets were then divided into training, validation, and test sets in a 7:1.5:1.5 ratio.

[0122] Training optimization, with the cross-entropy loss function as the optimization objective, iteratively trains until the entity prediction accuracy and timestamp prediction recall of the model on the validation set tend to stabilize.

[0123] like Figure 4The diagram shows the intelligent question-answering flowchart of this invention. First, the user asks, "What low-altitude economy-related policies has Sichuan Province promulgated since 2025?" The MTQA inference model identifies entities based on the user's question and infers highly relevant vertical domain knowledge based on the low-altitude economy knowledge base, performing knowledge reasoning to obtain a knowledge quadruple. This knowledge quadruple is then integrated with the original question and input into the Zidong Changkong Big Data Model, which in turn integrates with the user's original question. Finally, the Zidong Changkong Big Data Model outputs the intelligent answer.

[0124] To implement the above-mentioned intelligent question-answering method, this invention also designs an intelligent question-answering system for implementing the above method, such as... Figure 5 As shown. This intelligent question-answering system includes: a multimodal data acquisition and preprocessing module, a knowledge graph construction module, a knowledge retrieval module, a multimodal temporal knowledge graph module, an intelligent question-answering module, and a reasoning module. Among them,

[0125] The multimodal data acquisition and preprocessing module is used for text data acquisition and preprocessing. First, it defines the data pattern, crawls the policy data related to each item in the policy list and performs data cleaning, and then constructs policy quadruple data.

[0126] The knowledge graph construction module is used to create nodes and relationships through the local Neo4j service and store knowledge point quadruple data in the Neo4j graph database.

[0127] The knowledge retrieval module is used to query policy questions entered by users. The intelligent question-answering system returns attribute values, relationship displays, and video displays corresponding to the user's input.

[0128] The multimodal temporal knowledge graph module is used to align text, visual, structural, and timestamp embeddings into a unified complex representation space. It introduces a multi-expert learning mechanism to extract expert features and maps multimodal features to the complex domain, aggregates them to obtain multimodal enhanced embeddings, and then calculates scores based on the quadruplet scoring formula.

[0129] The intelligent question-answering module includes a reasoning module, which takes the user's question and uses the reasoning module to infer highly relevant vertical domain knowledge from the low-altitude economic knowledge base. This knowledge is then integrated with the user's original question and input into the Zidong Changkong low-altitude big model. Finally, the big model outputs an intelligent answer.

[0130] The processing flow of the aforementioned intelligent question-answering system revolves around two main modules: "multimodal temporal knowledge graph" and "intelligent question answering." First, the user inputs a question, which is simultaneously transmitted to both modules. In the multimodal temporal knowledge graph module, knowledge is first embedded to transform it into a graph-recognizable form. Then, based on the question, subgraph retrieval extracts related knowledge subgraphs. Next, subgraph processing filters effective information, ultimately yielding an answer subgraph which is transmitted to the prediction generation stage of the intelligent question answering module. In the intelligent question answering module, entity recognition is performed on the user's question to extract key entities. Then, question embedding is completed based on these entities. Subsequently, prediction generation is performed based on the answer subgraph to obtain preliminary answer information. Afterward, the system enters the question-answering model stage—sequentially performing structured processing of information through prompt word engineering, generating answers, and finally completing enhanced responses, outputting the final user reply.

[0131] It should be noted that the specific embodiments described above are exemplary, and those skilled in the art can devise various solutions inspired by the disclosure of this invention. These solutions all fall within the scope of this invention and its protection. Those skilled in the art should understand that this specification and its accompanying drawings are illustrative and not intended to limit the scope of the claims. The scope of protection of this invention is defined by the claims and their equivalents.

Claims

1. A low-altitude economic intelligent question-answering method based on multimodal temporal knowledge graph, characterized in that, The question-answering method constructs a multimodal temporal knowledge graph reasoning question-answering model (MTQA) based on time-aware constraints. The MTQA reasoning model includes a multimodal temporal knowledge graph embedding module (MMTE), which integrates multimodal information such as text, images, and videos and incorporates a temporal dimension to construct a knowledge graph that clearly represents the semantic and temporal relationships of entities. The subgraph extraction of the MTQA reasoning model uses topic entities as anchor points to fuse subgraphs. After time-aware constraint pruning, a temporal graph neural network is used to transmit temporal associations, balancing scale with the preservation of key information. Specifically, this includes: Step 1: Text, image, and video dataset collection and preprocessing. First, define the data schema, crawl the policy data related to each item in the policy list and clean it, then construct the policy quadruple data, which includes entity-attribute-attribute value-timestamp and entity-relationship-entity-timestamp. Step 2: Based on the quadruple data, construct a multimodal temporal knowledge graph; Step 3: Knowledge retrieval. Based on the constructed multimodal temporal knowledge graph and MTQA inference model, the system queries the policies input by the user and returns the corresponding attribute values ​​and partial relationship displays, as well as related images and videos. Step 31: Based on the constructed knowledge graph, accurately retrieve the quadruple knowledge data generated by the MTQA model, transform fragmented reasoning knowledge into structured and understandable policy information, and at the same time associate it with supporting multimodal materials to provide users with policy knowledge services that are both accurate and complete. Step 32: Build a retrieval interaction layer to receive policy query keywords or statements input by users, and after text preprocessing, connect to the retrieval interface of the knowledge graph; Step 33: Based on the MTQA model, reason about the structural features of the quadruplets, construct targeted retrieval rules, and map the user's query intent to the retrieval dimensions of the quadruplets in the graph; Step 34: Call the Neo4j query interface and execute a customized Cypher statement using the Py2neo library to retrieve MTQA inference quadruple data that match the query from the knowledge graph, and extract the policy attribute values, inter-policy relationships, and associated multimedia identifiers. Step 35: Parse and reconstruct the retrieved MTQA inference quadruple data. The policy attribute values ​​are organized in key-value pairs, and the policy relationships are displayed in a visual graph fragment. At the same time, retrieve the corresponding policy interpretation images and implementation case video materials according to the multimedia identifiers. Step 36: Integrate the parsed quadruple knowledge, relational visualizations, and multimodal materials, and return them to the user through the front-end interface. Among them, the core quadruple information of MTQA reasoning is displayed as the core module to ensure that the user can clearly obtain the key reasoning knowledge of the policy. Step 4: Construct an intelligent question-answering model, which includes the MTQA model and the Zidong Changkong low-altitude large model. The MTAQ model includes a question embedding module, a time-aware constraint subgraph extraction module, a multimodal temporal knowledge graph embedding module (MMTE), and an information aggregation and answer prediction module. The two work together to achieve accurate question answering in the low-altitude economic field. Step 41: The MTQA model goes through four stages: question embedding, subgraph extraction, knowledge graph embedding, and answer prediction, to infer highly relevant vertical domain knowledge from the low-altitude economic knowledge base; Step 411: The question embedding module preprocesses the input question text and then inputs it into the pre-trained language model to output the question embedding vector Q. T ; Step 412: The time-aware constraint-based subgraph extraction module runs, which includes three stages: subgraph extraction, subgraph pruning, and temporal graph neural network modeling. First, entity and time information of the input question are extracted and neighbor subgraphs are constructed. Then, the subgraph is pruned by selecting the time constraint acquisition method to obtain the answer subgraph. Finally, the subgraph structure and time information are mined by the temporal graph neural network TGNN to output the answer subgraph information of the question. Step 413: The knowledge graph embedding stage is implemented through the MMTE module, including four stages: multimodal embedding extraction and alignment, adaptive fusion and expert mechanism, time-modal interaction modeling, and multimodal quadruple scoring. In the first stage, the textual, visual, structural, and temporal embeddings of entities are extracted and aligned to a unified complex representation space. In the second stage, a multi-expert learning mechanism is introduced to extract expert features to obtain expert-driven entity embeddings. In the third stage, complex embeddings of entities, relations, and timestamps are generated based on the TComplEx algorithm. The multimodal features are mapped to the complex domain through the expert-driven entity embedding to obtain multimodal enhanced embeddings. In the fourth stage, the score is calculated based on the quadruple scoring formula. Step 414: Information aggregation and answer prediction module, input question embedding vector and subgraph information, generate enhanced question representation based on stacking operation, then output entity prediction vector and timestamp prediction vector through two layers of MLP, and then obtain probability distribution through Softmax function; Step 42: Construct the Zidong Changkong Low-Altitude Large Model Question Answering Module, set up in the backend of the MTQA inference model, and design four question answering processes including knowledge input, structured design of prompt words, parameter configuration and answer generation. Through these four question answering processes, the vertical domain knowledge output by the MTQA inference model is transformed into natural language answers. Step 43: Train the MTQA inference model using a publicly available question-answering dataset. Optimize the model using the cross-entropy loss function and iterate until the entity prediction accuracy and timestamp prediction recall of the inference model on the validation set stabilize. Stop training at this point.

2. The low-altitude economic intelligent question-and-answer method as described in claim 1, characterized in that, The three-stage processing of the subgraph extraction module specifically includes: Subgraph extraction stage: For the input problem, extract the entire set of topic entities, retrieve all related facts for each entity in the temporal knowledge graph, extract relevant entity and time information, construct an m-hop neighbor subgraph, merge duplicate entities in all neighbor subgraphs, and construct a fusion subgraph through bidirectional expansion; Subgraph pruning stage: Select the time constraint acquisition method through gating mechanism, merge all timestamps of related entities and sort them, take the minimum value as the start time and the maximum value as the end time to obtain the time range; prune the edges and nodes in the fused subgraph that do not meet the time constraint, and finally obtain the answer subgraph; The temporal graph neural network modeling stage involves using TGNN to mine subgraph structure and temporal information, calculating the correlation scores of adjacent nodes, updating node representations through an attention mechanism, and finally obtaining subgraph information of the problem through average pooling.

3. The low-altitude economic intelligent question-and-answer method as described in claim 2, characterized in that, The four-stage processing procedure of the MMTE module includes: Multimodal embedding extraction and alignment stage: Extract text embedding, visual embedding, structural embedding and temporal embedding of entities respectively, and align the four types of embedding to a unified complex representation space through modality-specific MLP mapping function; Adaptive fusion and expert mechanism stage: Text and visual embeddings are fused through a gating function, a multi-expert learning mechanism is introduced, expert features are extracted through a feature extraction function, and a weighted sum is calculated by combining gating weights to obtain expert-driven entity embeddings; The time-modal interaction modeling stage: Based on the TComplEx algorithm, complex embeddings of entities, relations, and timestamps are generated. Multimodal features are mapped to the complex domain through expert-driven entity embedding, and multimodal enhanced embeddings are obtained, including entities, relations, and time. Multimodal quadruplet scoring stage: Scores are calculated using the quadruplet scoring formula, and the embedding quality is optimized using a loss function.

4. The low-altitude economic intelligent question-and-answer method as described in claim 3, characterized in that, Step 42, the four question-and-answer processes of the Zidong Changkong Low-Altitude Large Model Q&A module, specifically include: Knowledge input stage: The MTQA model is invoked to extract domain knowledge highly relevant to the user's question from the low-altitude economic knowledge base, including quadruple reasoning results, entity attributes, and relationships; Structured design of prompt words: The original user question is integrated with the reasoning knowledge of the MTQA model into prompt content. The prompt content consists of two parts: first, question guidance: clarifying the direction of the query to avoid misunderstanding of intent; second, domain knowledge anchors: marking key reasoning knowledge to constrain the professional scope of the answer. Parameter configuration: Set the model temperature coefficient parameter to 0.1 to reduce output randomness and prioritize high-probability semantic sequences; Answer generation: Based on the prompts and parameter configurations, the Zidong Changkong Low-Altitude Big Data Model generates natural language answers that conform to the standards of the low-altitude economic field and returns them to the user.

5. A low-altitude economic intelligent question-answering system based on multimodal temporal knowledge graph, characterized in that, The intelligent question-answering system is used to implement the method of claim 1, and the system includes at least: a multimodal data acquisition and preprocessing module, a knowledge graph construction module, a knowledge retrieval module, a multimodal temporal knowledge graph module, an intelligent question-answering module, and a reasoning module, wherein, The multimodal data acquisition and preprocessing module is used to acquire and preprocess text, image, and video data. First, it defines the data pattern, crawls the policy data related to each item in the policy list and cleans the data, and then constructs the policy quadruple data. The knowledge graph construction module is used to create nodes and relationships through the local Neo4j service and store knowledge point quadruple data in the Neo4j graph database; The knowledge retrieval module is used to query policy questions entered by users. The intelligent question-answering system returns attribute values, relationship displays, and video displays corresponding to the user's input. The multimodal temporal knowledge graph module is used to align text, visual, structural and timestamp embeddings to a unified complex representation space. It introduces a multi-expert learning mechanism to extract expert features and maps multimodal features to the complex domain. The resulting multimodal enhanced embeddings are then aggregated and scores are calculated based on the quadruplet scoring formula. The intelligent question-answering module includes a reasoning module, which takes the user's question and infers highly relevant vertical domain knowledge from the low-altitude economic knowledge base. This knowledge is then integrated with the user's original question and input into the Zidong Changkong low-altitude big model. Finally, the big model outputs an intelligent answer. The intelligent question-answering system's processing flow revolves around two main modules: a multimodal temporal knowledge graph and intelligent question answering. First, the user inputs a question, which is simultaneously transmitted to both modules. In the multimodal temporal knowledge graph module, knowledge is first embedded to transform it into a graph-recognizable form. Then, based on the question, subgraph retrieval extracts related knowledge subgraphs. Next, subgraph processing filters effective information, ultimately yielding an answer subgraph which is then transmitted to the prediction and generation stage of the intelligent question-answering module. In the intelligent question-answering module, entity recognition is performed on the user's question to extract key entities. Then, question embedding is completed based on these entities. Subsequently, prediction and generation are performed based on the answer subgraph to obtain preliminary answer information. Afterward, the question-answering model enters the stage where information is structured through prompt word engineering, answer generation is performed, and finally, an enhanced answer is completed, outputting the final user response.

Images