Adaptive graph database multi-modal power knowledge question-answering method and system

Abstract

The invention relates to the technical field of intelligent data processing and artificial intelligence in the power industry, and provides an adaptive graph database multi-modal power knowledge question-answering method and system.The method comprises the steps that for a data query subtask in a semantic nested tree, intelligent error correction and expansion are conducted on natural language query, and after an expanded query statement is obtained, the data query subtask in the semantic nested tree is obtained; extracting a data result from the power industry database; for the knowledge query sub-tasks in the semantic nested tree, matching data results with nodes in the graph knowledge base, and obtaining knowledge results through a large language model based on feature representation of the matched nodes and the data results; based on the knowledge result and the data result, a natural language answer is obtained through a generation function of a large language model; and cross-channel cooperative processing and comprehensive answering are realized.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent data processing and artificial intelligence technology in the electric power industry, and in particular to an adaptive graph database multimodal electric power knowledge question and answer method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In the process of digital transformation of the power industry, massive amounts of equipment operation data, fault information, historical records, industry regulations, system standards and other data and text information provide strong support for applications such as intelligent equipment management, fault diagnosis, and predictive analysis. However, the data and text knowledge structure of the power system is complex, involving multiple levels of equipment information, operating parameters, fault types, regulations, system standards, and maintenance history, and the data and knowledge types are different, including multimodal data such as text, images, and time series. The heterogeneity and diversity of these data and knowledge make it a major challenge for the power industry to efficiently and accurately query and analyze data knowledge and extract key insights. The intelligent question-and-answer system that emerged as a result has gradually become an important tool for equipment management, operation monitoring, and decision support. However, the existing technology still has significant deficiencies in multimodal data fusion, natural language parsing and query enhancement, and dynamic updating of knowledge bases. At the same time, the traditional intelligent question-and-answer system is also inadequate in terms of problem intent recognition and processing and path optimization. These problems seriously restrict the widespread application and efficient operation of the intelligent question-and-answer system: (1) Current technologies show obvious limitations in multimodal data fusion. Traditional intelligent question-answering systems usually process single-modal data, such as only processing text data or time series data. They have weak fusion and joint analysis capabilities for multimodal data such as images, audio, and video. The feature distribution differences between multimodal data are large, and existing methods are difficult to achieve unified feature representation and deep fusion, resulting in the system being unable to comprehensively utilize the complementary information of multimodal data to provide more accurate answers, making it difficult for the system to provide comprehensive and accurate answers when processing queries that require multimodal collaboration. In addition, existing technologies lack the ability to perceive the semantic context of user input, especially for the recognition of specific terms and implicit intent in the power industry. For example, for a user query such as "abnormalities of high-load transformers in the past week", the system often cannot accurately infer the specific parameters or time range that the user is concerned about, resulting in insufficient accuracy in the response.

[0004] (2) In terms of natural language parsing and query enhancement, the existing technology also has many shortcomings. First, the system's ability to parse natural language queries entered by users is weak, especially when dealing with professional terms and complex expressions in the power industry, which can easily lead to semantic deviations. For example, when a user queries "temperature control status of high-voltage transformers", the system may misunderstand the query intent and fail to provide an accurate answer. Second, the current system lacks intelligent error correction and optimization capabilities. The system cannot effectively identify and correct common term confusion, logical errors or fuzzy expressions in user queries. For example, when a user enters "voltage operation log of switchgear", the system may cause query failure due to logical errors, and cannot actively prompt or optimize the query content. In addition, the existing system has serious defects in supporting multi-round interactions and dynamically optimized query paths. It can usually only perform single-round queries and responses, and cannot dynamically adjust the query strategy based on the subsequent information provided by the user, resulting in low response accuracy in complex query scenarios. For example, when a user enters a fuzzy query, the system lacks the ability to guide the user to clarify the intent, which ultimately affects the accuracy of the query results and user experience. At the same time, traditional query systems have limited capabilities in processing complex queries across multiple tables and multiple conditions that are common in the power industry. For scenarios where data is distributed in multiple heterogeneous tables, the system response speed and query accuracy cannot meet actual needs.

[0005] (3) In terms of dynamic updating of knowledge bases, traditional knowledge bases in the power industry mostly adopt a static structure and lack the ability to update in real time and maintain automatically. Usually, such knowledge bases require manual data updates and maintenance on a regular basis, which results in high costs, long update cycles and low efficiency, and it is difficult to reflect the latest industry changes in a timely manner. Especially when faced with dynamic information such as regulatory updates and equipment status changes, static knowledge bases cannot meet the requirements of real-time queries, which can easily lead to the question-and-answer system providing answers based on outdated data, affecting its practicality and credibility. In addition, traditional knowledge bases are mostly based on single-modal data, such as text information, and cannot fully utilize the complementary information of multi-modal data such as images, audio, and video. Such systems find it difficult to achieve accurate and comprehensive responses when faced with multi-modal complex queries.

[0006] (4) More importantly, the existing system is difficult to accurately distinguish the types of user queries, and the problem intention diversion mechanism is missing. In the power industry, user queries usually involve structured data (such as equipment operating status, fault records) and unstructured knowledge (such as operating procedures, regulations and standards). The existing technology lacks the ability to accurately classify and divert query intentions. Traditional systems use a single mode to process these two types of queries. They lack accurate judgment of the type of questions and find it difficult to adopt the optimal processing path for different query types, resulting in low query efficiency and difficulty in calling the most suitable processing module according to different question types, thus affecting the response speed and result quality of complex queries. This lack of problem diversion capability directly leads to the problems of low query efficiency and inaccurate response results. For example, when a user simultaneously queries "transformer operating data and fault handling methods in the past week", the system finds it difficult to identify that it involves both data query and knowledge query, and cannot dynamically allocate them to the corresponding module for processing, which ultimately affects the integrity and response speed of the query.

[0007] In summary, the shortcomings of existing technologies in multimodal data fusion, natural language parsing and query enhancement, dynamic knowledge base update, and question intention diversion have seriously limited the application value of intelligent question-answering systems in the power industry. Therefore, a new intelligent query system that integrates multimodal deep fusion, intelligent error correction, dynamic knowledge base update, and question diversion is urgently needed. Through accurate intent recognition and multi-channel collaborative processing, the query efficiency, response accuracy, and scenario adaptability of the system can be significantly improved, providing key technical support for the intelligent upgrade of the power industry. Summary of the invention

[0008] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides an adaptive graph database multimodal power knowledge question and answer method and system, which dynamically determines whether the natural language query input by the user belongs to a data query or a knowledge query by analyzing the natural language query input by the user, and utilizes multimodal semantic parsing technology and proprietary knowledge graphs of the power industry to deeply understand the context and potential intentions of the query, thereby realizing cross-channel collaborative processing and comprehensive answers.

[0009] In order to achieve the above object, the present invention adopts the following technical solution: A first aspect of the present invention provides an adaptive graph database multimodal power knowledge question answering method, which comprises: Obtain natural language queries for the power industry and generate semantic nested trees using a large language model; For the data query subtask in the semantic nested tree, the natural language query is intelligently corrected and expanded, and after the expanded query statement is obtained, the data results are extracted from the power industry database; For the knowledge query subtask in the semantic nested tree, after matching the data results with the nodes in the graph knowledge base, the knowledge results are obtained through the large language model based on the feature representation of the matching nodes and data results; the natural language answer is obtained based on the knowledge results and data results through the generation function of the large language model; Among them, the construction steps of the graph knowledge base include: obtaining new data, calculating the similarity between the new data and the corresponding nodes in the graph knowledge base, if the similarity is less than the threshold, updating the feature representation of the corresponding node; obtaining new nodes, calculating the association strength between the new nodes and all nodes in the graph knowledge base, if the association strength is greater than the set value, establishing an edge between the two nodes and giving the weight of the edge.

[0010] Furthermore, the steps of intelligent error correction and expansion include: For natural language queries in the power industry and their related images, feature extraction is performed using a large language model and a deep learning model respectively, and then the features of different modalities are unified into the same representation in a two-way fusion manner using the knowledge graph and the visual question-answering graph neural network to obtain fused features. Based on the fused features, intelligent error correction is performed through a hypergraph neural network and multiple rounds of interaction. After optimizing the natural language query, an extended query statement is obtained through hypergraph reasoning.

[0011] Furthermore, the initialization step of the graph knowledge base includes: taking different modal data as different node types, extracting feature representations of each node, and defining the relationship between different nodes, giving the strength of the relationship, and obtaining the graph knowledge base.

[0012] Further, the node types include device nodes, regulation nodes, operating procedure nodes, image nodes, video nodes, audio nodes, and segmented area nodes; Alternatively, the relationship includes a connection relationship, a device dependency relationship, a compliance relationship, an operation dependency relationship, and a subordinate relationship.

[0013] Furthermore, in the graph database, if the similarity between two nodes is less than a similarity threshold, the two nodes are merged into one node; Alternatively, in a graph database, if there is more than one edge between two nodes, the weights of all edges are merged.

[0014] Furthermore, the data query subtask includes querying equipment operation data, fault records and image data; Alternatively, the knowledge query subtask includes query of laws and regulations, operating procedures, pictures, videos and audios.

[0015] A second aspect of the present invention provides an adaptive graph database multimodal power knowledge question answering system, which includes: A graph knowledge base construction module is configured to: obtain new data, calculate the similarity between the new data and the corresponding nodes in the graph knowledge base, and if the similarity is less than a threshold, update the feature representation of the corresponding node; obtain new nodes, calculate the association strength between the new nodes and all nodes in the graph knowledge base, and if the association strength is greater than a set value, establish an edge between the two nodes and give the weight of the edge; A data acquisition module is configured to: acquire natural language queries for the power industry and generate a semantic nested tree through a large language model; A data query module is configured to: for the data query subtask in the semantic nested tree, intelligently correct and expand the natural language query, and after obtaining the expanded query statement, extract the data result from the power industry database; The knowledge query module is configured as follows: for the knowledge query subtask in the semantic nested tree, after matching the data results with the nodes in the graph knowledge base, the knowledge results are obtained through a large language model based on the feature representation of the matching nodes and data results; based on the knowledge results and the data results, a natural language answer is obtained through the generation function of the large language model.

[0016] Furthermore, it also includes an initialization module for a graph knowledge base, which is configured to: take different modal data as different node types, extract the feature representation of each node, and define the relationship between different nodes, give the strength of the relationship, and obtain a graph knowledge base; the node types include device nodes, regulatory nodes, operating procedure nodes, image nodes, video nodes, audio nodes, and segmented area nodes; the relationships include connection relationships, device dependency relationships, compliance relationships, operation dependency relationships, and subordinate relationships.

[0017] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the adaptive graph database multimodal power knowledge question and answer method as described above.

[0018] The fourth aspect of the present invention provides a computer device, including a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, wherein when the processor executes the program, the steps in the adaptive graph database multimodal power knowledge question and answer method as described above are implemented.

[0019] Compared with the prior art, the present invention has the following beneficial effects: The present invention innovatively proposes to combine the semantic parsing technology of the power industry-specific knowledge base and a large language model (such as GPT) to identify problem intent and make diversion decisions. By analyzing the natural language query input by the user, it dynamically determines whether it is a data query (such as equipment operation data, fault records, etc.) or a knowledge query (such as laws and regulations, procedures, etc.). By using multimodal semantic parsing technology and the power industry's proprietary knowledge graph, it deeply understands the context and potential intent of the query, and dynamically decomposes sub-problems according to the complexity of the problem, realizing cross-channel collaborative processing and comprehensive answers.

[0020] The present invention innovatively proposes semantic incremental technology, which incrementally updates the newly added labels and segmentation results after each input processing, avoids redundant reconstruction of the entire knowledge base, identifies changes in important information, and automatically corrects or expands the content of the graph knowledge base, performs local optimization and node expansion of the existing knowledge graph, identifies the relationship between the new information and the existing content, and optimizes the structure and association paths of the knowledge base.

[0021] The present invention innovatively proposes a graph knowledge base including equipment nodes, regulatory nodes, operating procedure nodes, image nodes, video nodes, audio nodes and segmented area nodes, etc., which supports the joint retrieval of multimodal data and text in the power industry, supports the retrieval and question-answering of multimodal data such as images, text, audio and video, and can perform semantic understanding and answer related questions of power equipment images, construction site videos, equipment operation sounds, etc., to realize the intelligence of the entire process from data input to information acquisition; in addition, the multimodal large model can perform unified feature representation for different types of data, and perform deep feature fusion and alignment of multimodal data.

[0022] This invention innovatively proposes a hypergraph big model driven intelligent error correction and query enhancement method, combining hypergraph neural network and big language model, and through self-learning error correction cycle, performs real-time semantic analysis, intelligent error correction and query enhancement on the user's natural language query. By using multi-level relationship modeling in the hypergraph structure, it can better understand the user's implicit intention, and automatically identify semantic deviations, terminology confusion or logical errors in user query input, such as device name confusion, parameter setting errors, etc., and prompt possible correct options, while providing specific optimization suggestions through the graph knowledge base. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0024] Figure 1 It is a flow chart of the adaptive graph database multi-modal power knowledge question and answer method according to the first embodiment of the present invention; Figure 2It is a structural diagram of a computer device according to a fourth embodiment of the present invention. DETAILED DESCRIPTION

[0025] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0026] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0027] Embodiment 1 This embodiment provides an adaptive graph database multimodal power knowledge question and answer method.

[0028] The adaptive graph database multimodal power knowledge question-answering method provided in this embodiment introduces a question intention diversion function, accurately classifies user questions, and dynamically distributes them to a data question-answering system or a knowledge question-answering system, improving query efficiency and response quality through multi-channel collaborative processing; through adaptive context perception and multimodal deep fusion technology, it can efficiently parse multimodal data and make full use of its complementary information; combining the intelligent error correction and query optimization technology of hypergraph neural network and large language model, it improves the accuracy of query parsing and multi-round interaction capabilities; through the dynamic update mechanism of the graph knowledge base with semantic incremental updates, it realizes the rapid capture and automatic update of real-time data and regulatory changes. The present invention significantly improves the adaptability, real-time and intelligence level of the intelligent question-answering system, and provides key technical support for the intelligent upgrade of the power industry.

[0029] The adaptive graph database multi-modal power knowledge question answering method provided in this embodiment is as follows: Figure 1 As shown, the specific steps are as follows: Step 1: A graph knowledge base maintenance method with semantic incremental update is proposed to automatically detect and integrate new information in the power industry, including multimodal data such as equipment operation data, on-site video, on-site audio, regulatory changes, operating procedures, and on-site images. Through semantic incremental technology, the newly added labels and segmentation results after each input processing are incrementally updated to avoid redundant reconstruction of the entire knowledge base, identify changes in important information, and automatically correct or expand the content of the graph knowledge base, perform local optimization and node expansion of the existing knowledge graph, identify the relationship between the new information and the existing content, and optimize the structure and association paths of the knowledge base. In particular, for recurring power engineering terms and specifications, their priorities and retrieval weights can be dynamically adjusted, so that the knowledge base can dynamically reflect the latest industry knowledge and data. The method supports the joint retrieval of multimodal data and text in the power industry, and supports the retrieval and question-answering of multimodal data such as images, text, audio, and video. It can perform semantic understanding and answer related questions for power equipment images, on-site videos, equipment operation sounds, etc., and realize the intelligence of the entire process from data input to information acquisition. Among them, the multimodal large model can unify the feature representation of different types of data, and deeply fuse and align the features of multimodal data. It supports device recognition in images, device voiceprint analysis in audio, and interpretation of construction processes in videos, and combines text question-and-answer modules to achieve cross-modal data association and intelligent answers. For example, when a user inquires about the operating status of a device, it can provide users with multi-level comprehensive answers by combining device recognition in images, voiceprint analysis in sounds, and operating logs in text. This ability to associate and understand cross-modal data allows it to not only answer direct queries, but also provide more comprehensive and intelligent answers to complex problems by analyzing the correlation of multimodal data.

[0030] Step 101, identification and classification of multimodal data sources. First, it is necessary to clarify the various data sources and types in the power industry to ensure comprehensive coverage of key data. The main data types include equipment operation data, on-site video, on-site audio, regulatory changes, operating procedures and on-site images. Equipment operation data usually contains real-time monitoring information, such as current, voltage, power and temperature, through which the operating status and health of the equipment can be evaluated; regulatory change data covers the latest policies, regulations and safety standards issued by the government and the industry, and is an important basis for guiding equipment operation and management; and operating procedures include standardized operating procedures and emergency treatment plans for various equipment to ensure that operations in daily and emergency situations meet specifications; finally, on-site images and video data are taken by drones or collected by on-site monitoring, which can provide visual information of the equipment and the surrounding environment to assist fault diagnosis and on-site inspection; in addition, the sound of equipment operation is collected by a sound collector, which can provide auditory information of the equipment and the surrounding environment to assist fault diagnosis and on-site inspection.

[0031] In terms of data collection, after obtaining data from different data sources, each data type needs to be systematically organized for subsequent processing and analysis: First, format and store text data such as regulations and operating procedures, and convert the content into a structured format (such as JSON or XML). This ensures that the text information is easy to process and integrate in the subsequent knowledge base construction. In addition, since the equipment operation data is usually tabular data, it can be directly stored in a structured table format to facilitate subsequent data query and analysis; In addition, in order to ensure that on-site images, videos, and audio can be accurately associated with other modal data, detailed metadata tags need to be added to the image data, such as shooting time, device identification, and geographic location; In addition, in the processing of video data, after extracting key frames, visual semantic segmentation technology is used on the key frames to further extract key semantic information in the image; In addition, in the processing of image data, visual semantic segmentation technology is used to further extract key semantic information in the image.

[0032] Here, let the image data set or key frame set be I, where I = {I1, I2,…,In}, representing a key frame set extracted from multiple on-site image sets or video data. For each image Ii, the segmentation region Si is generated by the semantic segmentation function f, that is: , where Si represents the segmented area, which contains the semantic information of each marked area, such as equipment, background, etc. This segmentation technology can help automatically identify the core objects in the image and convert them into a standardized data format for subsequent feature extraction and analysis.

[0033] In order to integrate data of different modalities into a unified knowledge graph, the data needs to be standardized and characterized. Let the multimodal data set be D, which includes equipment operation data Dop, regulatory data Dreg, operating procedures Dproc, image data Dimg, audio data Daud, and video data Dvid, that is, D={Dop, Dreg, Dproc, Dimg, Daud, Dvid}.

[0034] Step 102: Deep fusion of multimodal features. For data of different modes in a multimodal data set, a suitable feature extraction model is used to generate its feature vector representation: Text data: Use the GPT model to process text data (regulatory data or operating procedures) and map the text content into a feature vector xtext. The formula is: xtext=GPT(T), where T is the text data.

[0035] Image data: Use the Visual Transformer (ViT) model to process each segmented region in the image data to extract the deep feature vector ximage of the image. The deep feature vectors xs=ViT(Si) of all segmented regions Si corresponding to an image are concatenated to obtain the deep feature vector ximage of the image.

[0036] Audio data: The Transformer for Audio (TFA) model is used to process audio data and extract the audio feature vector xaudio. The formula is: xaudio=TFA(A), where A is the audio data.

[0037] Video data: For each segmented region of a key frame in the video data, use ViT to extract features. The deep feature vectors xs=ViT(Si) of all segmented regions Si corresponding to a key frame are concatenated to obtain the deep feature vector of the key frame; the deep feature vectors of all key frames of a video are concatenated to obtain the deep feature vector xvideo of the video.

[0038] Step 103: All feature vectors obtained in step 102 are preliminarily aligned through a multimodal feature alignment model to ensure that each modal feature has a similar representation scale in the same feature space.

[0039] In this embodiment, the training of the multimodal feature alignment model adopts a spatiotemporal contrastive learning framework.

[0040] Assume that there is a training set containing N data instances. For each data instance, extract the feature vectors of its modalities such as text, image, audio, and video, such as the text feature xtexti, image feature ximagei, audio feature xaudioi, and video feature xvideoi of instance i. A positive sample pair is a multimodal feature pair of the same instance, such as (xtexti, ximagei) and (xaudioi, xvideoi); while a negative sample pair is composed of modal features of different instances, such as (xtexti, ximagej) and (xaudioi, xvideok), where i≠j≠k.

[0041] To achieve the alignment of multimodal features, the similarity is calculated for the positive sample pairs and the negative sample pairs respectively. The similarity function s can select cosine similarity or Euclidean distance. For the positive sample pairs (xa, xb) and the negative sample pairs (xa, xc), the similarity is calculated as s(xa, xb) and s(xa, xc) respectively, where s(xa, xb) aims to maximize the similarity of the positive sample pairs, while s(xa, xc) is used to minimize the similarity of the negative sample pairs.

[0042] To this end, the contrast loss function Lcontrastive used in training the multimodal feature alignment model is defined. Its goal is to maximize the similarity of positive samples and minimize the similarity of negative samples by optimization. The specific loss function can be expressed as:

[0043] Among them, the first term is the loss of positive sample pairs, which ensures that the positive sample pairs are highly aligned in the feature space by making s(xa, xb) as close to 1 as possible; the second term is the loss of negative sample pairs, which controls the similarity of negative sample pairs by minimizing s(xa, xc) and setting the threshold δ to prevent the features between negative samples from being too close.

[0044] In order to enhance the feature alignment effect of the contrastive learning framework, temporal and spatial information is introduced in the similarity calculation to ensure that the different modal features of the same instance are consistent in time and space. Assuming that the modal features of the instance contain temporal information t and spatial information l, the calculation of similarity is expanded to:

[0045] Among them, α and β are the attenuation coefficients of time and space, which are used to control the influence of time and space distance on similarity. This formula ensures the consistency of multimodal features of the same instance in time and space, making the alignment of multimodal features more reasonable and tight.

[0046] During the training process of the multimodal feature alignment model, by minimizing the contrast loss function Lcontrastive, the similarity of positive sample pairs is continuously optimized and the similarity of negative sample pairs is reduced, and the alignment of multimodal features in the same feature space is gradually achieved.

[0047] Through the multimodal feature alignment model, a unified feature space is generated, so that data of different modalities can be highly consistent in the feature space, thus providing more accurate multimodal retrieval results when facing fuzzy queries. Moreover, through the aligned multimodal feature representation, a comprehensive understanding of multimodal data can be achieved in intelligent question answering and data retrieval, ensuring the accuracy and consistency of query responses.

[0048] Step 103: Initialization construction of the graph knowledge base.

[0049] First, construct a node set of the knowledge graph to represent different types of power industry knowledge entities. Define the graph structure G = (V, E), where V represents the node set, E represents the edge set between nodes, and the edge represents the relationship between nodes. For different types of multimodal data (equipment operation data, regulations, operating procedures, images, videos, and audio, etc.), create different types of nodes: Device node Vdev: represents the power device and its status information. A device node represents a power device, including all device operation data of the power device.

[0050] Regulation node Vreg: represents the policy and regulation information of the power industry. A regulation node represents a regulation, including the text and feature vector representation of the regulation.

[0051] Operation procedure node Vproc: represents the operation process and procedure steps of the equipment. An operation procedure node represents the operation procedure of a type of equipment, including the text and feature vector representation of the operation procedure.

[0052] Image node Vimg: represents image information related to the device. An image node represents an image of a certain device at a certain monitoring angle, including images at all times under the monitoring angle and their feature vector representations.

[0053] Video node Vvid: represents the video information related to the device. A video node represents the video of a certain monitoring angle of a certain device, including the video at all times under the monitoring angle and its feature vector representation.

[0054] Audio node Vaud: represents the audio information related to the device. An audio node represents the audio of a device, including the audio of the device at all times and its feature vector representation.

[0055] Segmentation region node Vs: represents the segmentation region associated with the image or video. A segmentation region node represents a segmentation region, including the segmentation region and its feature vector representation. For example, the foreground of an image of a certain monitoring angle of a certain device can be used as an independent segmentation region node, and the segmentation region node contains the foreground region of the image of the device at all times under the monitoring angle.

[0056] Therefore, the node set can be expressed as: V = Vdev ∪ Vreg ∪ Vproc ∪ Vimg ∪ Vvid ∪ Vaud ∪ Vs.

[0057] In the process of constructing the knowledge graph of this embodiment, for the segmented area Sj of image data or video data, each labeled area ri can be mapped to a graph node vj, and the node of the segmented area is connected to the image node, and the connection relationship is a subordinate relationship.

[0058] In this embodiment, the mapping relationship between the image data or video data and the segmentation region node is expressed as vj=g(Sj,ri), and the feature vector of the segmentation region node vj is ; Wherein, g is the region mapping function, h is the feature extraction function, Sj represents the jth segmented region image obtained, and ri represents the i-th region in the image or video frame.

[0059] In this embodiment, a feature vector is assigned to each node so that the feature information can be used during retrieval and analysis. Assuming that the feature vector of node v∈V is Xv, the feature representation can be defined as:

[0060] in, Represents the feature vector representation extracted from the device operation data.

[0061] Next, we need to define the relationship between nodes, that is, the edges in the knowledge graph. The edge set E is used to describe the association relationship between different node types, for example: The relationship between device nodes: such as connection relationship, device dependency relationship, etc., used to represent the physical or logical association between devices; The relationship between the device node and the regulation node: For example, the compliance relationship indicates the regulatory terms that a device must comply with when operating; The relationship between the equipment node and the operation procedure node: such as the operation dependency, which indicates the operation steps that a certain equipment needs to follow at different operation stages; The relationship between device nodes and image nodes: such as subordinate relationships, which represent image information of a specific device at different viewing angles, and are used for fault diagnosis and device status monitoring; The relationship between device nodes and video nodes: such as subordinate relationships, which represent videos of specific devices from different perspectives, used for fault diagnosis and device status monitoring; The relationship between device nodes and audio nodes: such as subordinate relationships, which represent the audio of a specific device in different environments, and are used for fault diagnosis and device status monitoring; The relationship between the video node or image node and the segmentation region node: such as a subordinate relationship, indicating the region segmented by the video node or image node.

[0062] Assuming that the weight of edge e∈E is we, which represents the strength or importance of the relationship, the edge set is expressed as: ; Among them, (vi, vj) represents the edge between two nodes, and we represents its weight.

[0063] Next, to ensure that the knowledge base structure is free of redundancy, the graph structure needs to be optimized, especially the deduplication of nodes and edges.

[0064] First, for duplicate nodes of the same entity, the following conditions are used to determine whether they are redundant: ; where d(vi, vj) represents the similarity between two nodes (calculated by the cosine similarity of the feature vectors). is the similarity threshold; when d(vi, vj) is less than the threshold , merge nodes vi and vj into one node to reduce duplicate information.

[0065] Secondly, for edge redundancy detection, the edge weight update rule is defined to prevent the appearance of duplicate edges: if there is already an edge (vi, vj) between two identical nodes, the newly added edge weight w'e is merged with the existing edge weight we: .

[0066] This ensures that the graph structure has a reasonable weight distribution when representing multimodal data associations.

[0067] Finally, after node construction, feature assignment, edge definition and structure optimization, a complete initial graph structure G = (V, E) is generated, where V represents all nodes mapped by multimodal data and E represents the relationship between nodes. Through this initial construction step, the knowledge base has a complete representation of power industry information and can maintain data consistency during incremental updates.

[0068] Step 104: semantic incremental update detection.

[0069] First, we obtain the newly added data (newly added data refers to the data contained in a node, such as a piece of status information of a device, a sentence in an operating procedure, the latest image or video of a monitoring angle of a device, etc.), and generate its semantic feature representation through feature extraction technology. Let the newly added multimodal data set be Dnew = {d1, d2, ..., dn}, where di represents the i-th newly added data item; for each newly added data item di, its feature vector is obtained through the feature extraction function f. , where f represents a feature extraction function, and different feature extraction methods are used for different modal data, including the method in step 102.

[0070] The semantic similarity calculation is performed between the feature representation xdi of the newly added data and the feature vector xvj of the corresponding node in the knowledge base to detect the changes between the newly added data and the existing data. Let the similarity calculation function be s(·,·), which is defined as: ; Among them, the value range of s(xdi, xvj) is [0, 1]. The closer the value is to 1, the higher the similarity between the new data and the existing knowledge base data.

[0071] By setting a threshold θ1, we can determine whether the new data is significantly different from the existing data. When s(xdi, xvj)<θ1, it is considered that there is a large difference between di and vj, so di is marked as a change point that needs to be updated; otherwise, the new data is not added to the graph structure. That is, if s(xdi, xvj)<θ1, di is a change point, where the value of θ1 can be adjusted according to the specific application scenario to ensure the accuracy and recall rate of change point detection.

[0072] Once a change point is detected, it is further classified into types such as device state change, procedure update, image change, video or audio change, etc. Let the classification function of the change point be c(di), then each change point is marked as a specific update type.

[0073] For example: .

[0074] This classification helps in targeted updating of relevant nodes and relationships in the graph knowledge base.

[0075] Furthermore, for each change point di, corresponding incremental update information is generated. Let the incremental update information be U(di), then the generation process of the incremental update information can be expressed as: ; where vj is the node that needs to be updated (i.e., the node corresponding to the newly added data), xdi is the feature vector of the change point, and "association edge" represents the edge between the change point and the related node vj in the knowledge base. For example, if a change in the device status is detected, the association edge between di and the device node is updated to the latest status to reflect the current operation status.

[0076] Finally, the generated update information U(di) is added to the knowledge base, the feature vector of the corresponding node is updated, and the associated edges of the corresponding node are readjusted. Through local optimization, the position and semantic relationship of the new information in the knowledge base are ensured to be reasonable. For example, for the device node vj whose state has changed, the feature vector of the updated node vj is: ; Among them, α∈[0, 1] is the smoothing coefficient, which is used to control the degree of fusion of new information and existing information to prevent the excessive impact of new information on the node state.

[0077] Step 105: Optimize and update the incremental graph knowledge base.

[0078] Get the newly added nodes, which refer to the newly added device nodes, image nodes, video nodes, etc.

[0079] For the newly added nodes identified in change detection, they are not added independently, but integrated based on their association with existing nodes.

[0080] First, the semantic similarity between the newly added node and the existing node is calculated to determine whether a connection needs to be established: for the newly added node vnew and the existing node vexisting, their association strength is calculated through the similarity function s(vnew, vexisting). If it exceeds the set threshold θ2, it is considered that the two are strongly associated and an edge e needs to be established between them: e=(vnew, vexisting) If, s(vnew, vexisting)>θ2.

[0081] When a new node is connected to an existing node, in order to ensure the accuracy of the association, the weight of the edge is adjusted to reflect the actual strength of the association between the nodes. The initial weight of the edge is set to winit, and it is adjusted according to the similarity of the nodes to ensure the dynamic nature of the relationship in the knowledge base. The final weight of the edge we is calculated as: ; Where β is the adjustment coefficient, which is used to balance the influence of initial weight and similarity. This step ensures that the relationship strength between the newly added nodes and the nodes in the existing knowledge base reasonably reflects the actual situation and avoids overly weakening or strengthening certain associations.

[0082] After the integration of the newly added nodes and edges is completed, the affected node neighborhoods in the graph structure (graph knowledge base) are optimized to remove redundant or weakly associated nodes. Specifically, the neighborhood set N(v) of each node v is updated, and the nodes with a correlation below the threshold are removed to ensure the simplicity of the graph structure. Let the correlation between node v and the neighboring node u be g(v, u), and the optimized neighborhood set is expressed as: ; where γ is the minimum association threshold for neighborhood optimization. This optimization step reduces unnecessary connections and improves the efficiency of graph representation.

[0083] After adding new nodes and edges and adjusting the neighborhood structure, a consistency check is performed to ensure that the updated knowledge base has no structural conflicts. The check includes node uniqueness, edge rationality, and graph connectivity. For redundant nodes or edges in the graph structure, they are marked and removed after being detected during the check process. If two nodes are highly similar in semantics (i.e., the similarity is close to 1), they are merged into one node to reduce information redundancy. The consistency check function C(Gnew) is used to verify whether the graph structure meets the consistency requirements: .

[0084] After incremental updates, edge weight adjustments, neighborhood optimization, and consistency checks, an optimized knowledge graph (graph knowledge base) Gnew = (Vnew, Enew) is generated to ensure that it reflects the latest industry knowledge and maintains a simple and reasonable structure. This optimized graph provides a dynamic and accurate knowledge foundation for subsequent queries and intelligent question-answering, while avoiding unnecessary redundant information and ensuring efficient retrieval.

[0085] Step 2: Propose a method for problem intent diversion and multi-channel query processing, which combines the semantic parsing technology of the power industry-specific knowledge base and a large language model (such as GPT) for problem intent identification and diversion decision-making.

[0086] By analyzing the natural language query input by the user, it dynamically determines whether it is a data query (such as equipment operation data, fault records, images, etc.) or a knowledge query (such as regulations, procedures, etc.), and assigns the question to the data question-answering system or the knowledge question-answering system. Using multimodal semantic parsing technology and proprietary knowledge graphs in the power industry, it deeply understands the context and potential intent of the query, and dynamically decomposes sub-questions according to the complexity of the question, achieving cross-channel collaborative processing and comprehensive answers.

[0087] This embodiment focuses on the efficiency and intelligence of multi-channel query processing, and aims to solve the diverse needs in complex power industry query scenarios through the collaborative work of the data question answering system and the knowledge question answering system. It is mainly composed of a multi-channel processing module and a collaborative processing module.

[0088] The multi-channel processing module consists of two core subsystems: the data question-answering system focuses on processing structured data queries such as equipment operation data and fault records; while the knowledge question-answering system focuses on the retrieval and answering of unstructured knowledge such as laws and regulations. Through division of labor and collaboration, these two subsystems provide flexible query processing capabilities in the system architecture.

[0089] The collaborative processing module focuses on the decomposition of complex queries and multi-channel system collaboration through dynamic task decomposition and collaborative processing. The core is to ensure efficient processing and accurate response to multi-intent queries through problem complexity assessment, sub-problem decomposition and collaborative information sharing between channels.

[0090] First, the complexity of the query input by the user is evaluated, and the semantic parsing model and intent classifier are used to determine whether the query contains multi-level intent. For simple single-intent queries, such as "obtain the operating status of the transformer", they will be directly assigned to the data question-answering system for processing. For complex multi-intent queries, such as "the operating status of the transformer in the past week and related operating procedures", two sub-intents will be identified: "extract equipment operating status data" and "find operating procedure knowledge", thus deciding to adopt a dynamic decomposition strategy.

[0091] For queries with multi-level intent, a semantic nested tree is generated through a large language model (such as GPT-4), the logical structure of the query is parsed, and a tree decomposition representation is generated:

[0092] Among them, Q1, Q2, etc. are the subtasks of the query, and T represents its hierarchical logical structure.

[0093] Next, each subtask is dynamically assigned to the corresponding processing channel according to the decomposed subtask type. Taking "the operating status and related operating procedures of the transformer in the past week" as an example, the query of the operating status is assigned to the data question-answering system, which extracts the equipment operation log from the power industry database and generates a time series trend chart. The query of "related operating procedures" is routed to the knowledge question-answering system, which returns a detailed description of the procedures through knowledge graph retrieval and semantic reasoning.

[0094] Step 3: Data question answering system.

[0095] An adaptive context-aware and multi-modal deep fusion method for intelligent power data query is proposed, combining adaptive context-aware and multi-modal deep fusion natural language conversion technology with the power industry's unique knowledge base. The proprietary knowledge base of the power industry (power knowledge base) is integrated, and through semantic embedding and multi-modal data (such as text, images, and time series data) interaction, it can not only identify the power professional terms implicit in the natural language context of user input, but also predict and intelligently complete the user's current query intention through a large model. For example, for the query "abnormalities of high-load transformers in the past week", it can intelligently infer the specific parameters that users are concerned about, automatically compare these descriptions with the background knowledge base, and generate complete natural language query statements.

[0096] Specifically, the natural language queries input by users and images related to the queries are obtained, and natural language understanding and reasoning are achieved through multimodal knowledge graphs and graph neural networks (GNNs), especially for contextual awareness in the specific context of the power industry.

[0097] Step 301: Receive a natural language query (text data) input by a user; in parallel, data in other modalities and images related to the query (such as photos of power equipment) are also received.

[0098] The query-related image refers to the input image associated with the user's query statement.

[0099] Step 302: parse the natural language query through a pre-trained language model (such as a large language model such as GPT), generate a contextual representation of the question, encode it into a vector representation, and obtain query statement features; and the image data will be extracted through a deep learning model (such as a VIT (vision transformer) model) to obtain image features.

[0100] Step 303: Generate scene graph and concept graph.

[0101] The input multimodal information (text and image) is converted into a unified semantic representation through a graph neural network (GNN), including: based on query statement features and knowledge graphs, the power knowledge base is used to extract relevant concepts from the query statement and construct a concept graph; based on image features, a pre-trained scene graph generator is used to extract visual information from the image and construct a scene graph.

[0102] Step 304: parse the natural language query input by the user through a combination of vision and language.

[0103] The VQA-GNN (Visual Question Answering Graph Neural Network) model uses a bidirectional fusion approach to unify the information of these different modalities into the same representation, and performs reasoning through the message passing mechanism of the graph neural network to obtain fused features.

[0104] Among them, the input of VQA-GNN is divided into three main sources: images, natural language questions and knowledge base; the output includes feature vectors that can be used for intelligent query completion or fuzzy completion.

[0105] Among them, the processing steps of the VQA-GNN (Visual Question Answering Graph Neural Network) model include: (1) Construction of multimodal semantic graph.

[0106] The construction formula of the multimodal graph is: Where E represents a set of edges; each scene graph node vs and concept graph node vc are connected by a supernode z.

[0107] (2) Two-way information fusion.

[0108] VQA-GNN uses two modality-specific graph neural networks (GNNs) to perform message passing on the scene graph and concept graph respectively.

[0109] Node feature update formula: The feature of each node v is updated by aggregating the features of neighboring nodes: ; Where, N(v): the set of neighbor nodes of node v; AGG: aggregation function (such as summation or attention mechanism); W: learnable weight matrix; σ: activation function (such as ReLU).

[0110] (3) Cross-modal messaging.

[0111] The information of scene graph and concept graph is passed through super nodes: ; ;in, and : Attention weight, used to measure the importance between super nodes and neighbor nodes. Super nodes connect scene graphs and concept graphs. Their features represent the global semantic fusion results of visual modality and language modality, and are an important summary of the entire multimodal semantic graph.

[0112] Supernode feature update formula: ; : Information aggregated from the scene graph; : information aggregated from the concept graph; ||: feature concatenation operation; fz: linear transformation or nonlinear activation function.

[0113] (4) Extract the global representation of the scene graph and concept graph through pooling operation, concatenate the feature vectors of QA-context Node and QA-concept Node, and output the fused feature , used for subsequent analysis or as a query of the associated vector to provide key field completion for fuzzy search.

[0114] Step 305: Based on the fusion features, intelligent error correction is performed through hypergraph neural network and multiple rounds of interaction. After optimizing the natural language query, an extended query statement is obtained through hypergraph reasoning, and the extended query statement is converted into a structured query statement through an autoregressive model to obtain the query result. For the query result, dynamic permission adaptation, sensitive data protection, and compliance and security checks are performed to achieve dynamic response.

[0115] A hypergraph big model-driven intelligent error correction and query enhancement method is proposed. This method combines the hypergraph neural network and performs real-time semantic analysis, intelligent error correction and query enhancement on users' natural language queries through a self-learning error correction cycle.

[0116] Specifically, by using multi-level relationship modeling in the hypergraph structure, it can better understand the user's implicit intentions, and automatically identify semantic deviations, terminology confusion or logical errors in user query input, such as device name confusion, parameter setting errors, etc., and prompt possible correct options, while providing specific optimization suggestions through the graph knowledge base. It also has multi-round interaction capabilities, and can dynamically generate candidate queries based on ambiguous or conflicting information input by users, and prioritize and recommend them in an interactive manner. This error correction and enhancement mechanism significantly improves the context adaptability of queries and the accuracy of conversion from natural language to data language in the process of interaction with users.

[0117] Specifically, intelligent error correction and query optimization mainly rely on Hypergraph Neural Networks (HGNN) to automatically identify and correct semantic errors or logical deviations in user queries.

[0118] Based on the fusion features (supernodes and related node features updated by GNN), missing field information, semantic errors, logical deviations, fuzziness or logical ambiguity (such as time range) in the query are identified, and candidate query statements are generated by aligning with the concept graph node features. For example, "find abnormal records of high-voltage equipment" can be supplemented as "abnormal records of high-voltage equipment in the past year."

[0119] In query enhancement and multi-round interaction optimization, the focus is on generating multiple candidate queries by detecting ambiguous or conflicting information in user input, and continuously optimizing the query content through multiple rounds of interaction, ultimately ensuring that the query results meet user needs.

[0120] In each round of interaction, these candidate query statements are presented to the user, allowing the user to select the query path that best matches his or her intent.

[0121] In each round of interaction, after receiving the candidate query selected by the user, the candidate query selected by the user is weightedly fused with the natural language query of the previous round to obtain the natural language query of this round, and the fusion features and candidate query are regenerated until the candidate query no longer changes.

[0122] In the deep reasoning and enhancement of hypergraph structures, the hypergraph structure is used to model the potential multi-layer relationships in complex query contexts to achieve refined association discovery and data insight enhancement. Different from error correction or simple query optimization, this step aims to provide users with in-depth data patterns and trends beyond the surface results of the query, thereby helping users gain a richer analytical perspective in the context of complex data.

[0123] Specifically, based on the supernodes and related node features updated by GNN, the extended query statement is obtained through hypergraph reasoning.

[0124] Multimodal query expansion, input image: monitoring screen, showing damaged transmission lines. Query statement: "Are there any recent maintenance records for transmission lines similar to the one in the image?"; System processing: The scene graph extracts the damage features in the image. The concept graph expands the "similar transmission lines" condition in the query. Return query results and suggestions: "The recent maintenance record shows that transmission line ID 12345 completed maintenance in October 2024.

[0125] In natural language to SQL query, it is necessary to convert the user's natural language query into a structured SQL query based on understanding the user's intention to obtain the data query result. This process uses an autoregressive model SQLformer, which can gradually generate SQL (structured query statement) statements based on natural language input. In order to accurately complete this conversion process, it not only relies on the semantic embedding in the knowledge graph, but also combines the structured information of the power equipment.

[0126] First, when receiving a user's natural language query, such as "abnormal conditions of high-load transformers in the past week", the natural language processing model will decompose the key components of the query and identify the core objects in the query (such as "transformer"), time range (such as "past week"), and query intent (such as "abnormal conditions"). In order to convert these unstructured natural language elements into structured SQL queries, the entity embedding in the knowledge graph is used to locate the specific entities and attributes of these elements in the power system. For example, "transformer" is mapped to the corresponding equipment table in the database, and "abnormal conditions" corresponds to the abnormal field in the fault record table.

[0127] In the process of generating SQL queries, the autoregressive model will gradually predict each component of the query, from SELECT, FROM to WHERE conditions. The SQLformer model is used to represent the SQL query as an abstract syntax tree (AST) and use the autoregressive model to generate each node of the SQL query. In this process, the corresponding SQL structure is generated according to the database table structure, field information and query intent.

[0128] In addition, when generating SQL queries, the relationship between tables and fields in the knowledge graph is referenced. For example, it is known in which tables the relevant information of "transformer" is stored, and the terms in the natural language query are matched with the fields in the database through semantic embedding. In this way, complex SQL queries can be automatically generated, and the query statements are guaranteed to be consistent with the database structure specific to the power industry. Through this autoregressive SQL generation model, the user's natural language query can be efficiently and accurately converted into SQL queries, ensuring that users can quickly obtain the required power data.

[0129] Step 4: Knowledge question and answer system.

[0130] Step 401: If there is only knowledge query, the query and response of the intelligent question answering system are generated. The specific steps include: First, the user's query sentence Q is represented as a feature vector xQ in order to match it with the multimodal features in the graph knowledge base in the feature space. Assuming that the query is in text form, it is converted into a feature vector through an embedding model (e.g., a GPT model): , this feature vector xQ is located in the same alignment space as the multimodal features of the graph knowledge base so as to calculate the similarity with the nodes in the graph knowledge base.

[0131] Next, semantic matching is performed in the graph knowledge base, and the most relevant node or node combination is found by calculating the similarity between the query feature vector xQ and the feature xV of each node in the graph structure. The similarity s(xQ, xV) can be calculated using cosine similarity: The most relevant nodes Vrelevant to the query are retrieved from the graph knowledge base according to the similarity value, so as to generate a targeted response to the query.

[0132] In order to provide more accurate suggestions, the content and associated information of the selected nodes are dynamically analyzed. For example, if the query involves the device status, the device operation data node and the related image node are extracted from the knowledge base, the current status is evaluated and combined with the corresponding historical data and images to generate processing suggestions. Assuming that the device status is S(t) and the image detection result is E(t), the processing suggestion R can be generated by the following logic: ; Where f is a rule or model function that automatically generates corresponding response suggestions based on the current state of the device and the image detection results.

[0133] Finally, the multimodal information is integrated to generate a detailed response for the user. If the user's query requires a multimodal explanation (such as a text description, an image example, or an audio prompt), the data of the relevant modalities is obtained from the graph knowledge base and integrated into a unified answer. Assuming that the final response contains a text answer Tresp and an image example Iresp, the generated multimodal response is: .

[0134] The response can comprehensively cover different aspects of the user's query and provide the user with specific processing suggestions and reference information.

[0135] Step 402: If there is both data query and knowledge query.

[0136] During task execution, the collaborative processing mechanism realizes cross-channel data sharing through internal communication modules. When the intermediate results of a module are critical to the processing of another module, these results can be dynamically transferred. For example, when querying "equipment operation data and fault handling solutions", the data question and answer system first extracts the equipment operation status , and input it into the knowledge question answering system as context information to help the latter more accurately match relevant regulatory provisions and treatment plans .

[0137] In the process of inter-channel collaboration, the results are shared through a unified communication protocol. After generating the device operation data results, the data question answering system will represent the semantic vector of the data query results. Passed to the knowledge question answering system for contextual assistance.

[0138] The knowledge question answering system uses the following formula to match the data query results with the knowledge nodes in the graph knowledge base obtained in step 1:

[0139] Among them, N is the node set in the graph knowledge base, vRdata is the vector representation of the running data result, vR represents the semantic vector of the data result, and cos(vRdata,vn) is the semantic similarity between the running data and the knowledge node.

[0140] By maximizing the cosine similarity, the knowledge node nbest that is most relevant to the data result is located, for example, the processing procedure corresponding to the abnormal operating temperature of the transformer.

[0141] Next, the knowledge question answering system generates a specific regulatory description Rknowledge based on the matched node nbest and context information. This process is completed through a large language model, and the formula is as follows:

[0142] Among them, LMknowledge is a large language model dedicated to knowledge generation, and its input includes the most relevant node nbest (matching node) and the semantic vector vRdata of the data result.

[0143] Comprehensive answer generation is the last key step, which aims to integrate and optimize partial answers from different channels to generate a coherent, complete, and user-friendly final answer. Using a large language model (such as GPT-4) as the core generation engine, the accuracy of the answer and the coherence of the information are ensured by performing contextual analysis and logic optimization on the data results returned by each module.

[0144] After the data question answering system and the knowledge question answering system return results respectively, assuming that the equipment operation data result returned by the data question answering system is Rdata, and the regulations and procedures answers returned by the knowledge question answering system are Rknowledge, the large language model generates the final comprehensive answer based on these two results:

[0145] LM represents the generative function of the large language model, which can combine the input content of the two channels to generate a complete natural language answer. This method fully utilizes the context understanding and language generation capabilities of the large language model to ensure the coherence and accuracy of the answer content.

[0146] For example, for the query "Transformer operating temperature and abnormal reasons in the past week", the system combines the time series curve of the equipment temperature with the regulatory interpretation, and generates the following answer through the language model: "The transformer operating temperature has been on an upward trend in the past week, especially reaching an abnormal peak (85°C) on March 15. According to the operating procedures, the cooling system should be checked in time and cooling measures should be taken." After the answer is generated, the answer content will be optimized based on the user's personalized preferences and interactive feedback. If the user indicates that more detailed operation steps or a more concise answer are needed, the system will dynamically adjust the answer structure. For example, if the user makes further demands on the "solution to abnormal transformer temperature", the system will add specific operation steps, such as "(1) Check the coolant level, (2) Clean the heat sink, (3) Adjust the load operating parameters."

[0147] In addition, multimodal results are visualized. For equipment operation data, intuitive graphics such as trend graphs and pie charts are automatically generated; for regulations and procedures, they are presented in a step-by-step list. This integrated display of multimodal results enables users to quickly understand the core information. For example, when returning "Causes and solutions for temperature anomalies", a temperature change curve is directly displayed, and targeted operating procedures are attached, forming a comprehensive report that emphasizes both text and graphics.

[0148] Through these processes, the comprehensive answer generation module achieves full-process optimization from multi-channel results to user-satisfactory answers, ensuring that multimodal comprehensive answers that are both accurate and easy to understand can be generated in complex scenarios, significantly improving user experience and system practicality.

[0149] Embodiment 2 This embodiment provides an adaptive graph database multimodal power knowledge question answering system, which specifically includes: The initialization module of the graph knowledge base is configured to: take different modal data as different node types, extract the feature representation of each node, and define the relationship between different nodes, give the strength of the relationship, and obtain the graph knowledge base; the node types include device nodes, regulation nodes, operation procedure nodes, image nodes, video nodes, audio nodes and segmented area nodes; the relationships include connection relationships, device dependency relationships, compliance relationships, operation dependency relationships and subordinate relationships; A graph knowledge base construction module is configured to: obtain new data, calculate the similarity between the new data and the corresponding nodes in the graph knowledge base, and if the similarity is less than a threshold, update the feature representation of the corresponding node; obtain new nodes, calculate the association strength between the new nodes and all nodes in the graph knowledge base, and if the association strength is greater than a set value, establish an edge between the two nodes and give the weight of the edge; A data acquisition module is configured to: acquire natural language queries for the power industry and generate a semantic nested tree through a large language model; A data query module is configured to: for the data query subtask in the semantic nested tree, intelligently correct and expand the natural language query, and after obtaining the expanded query statement, extract the data result from the power industry database; The knowledge query module is configured as follows: for the knowledge query subtask in the semantic nested tree, after matching the data results with the nodes in the graph knowledge base, the knowledge results are obtained through a large language model based on the feature representation of the matching nodes and data results; based on the knowledge results and the data results, a natural language answer is obtained through the generation function of the large language model.

[0150] This embodiment proposes an adaptive context-aware and multi-modal deep fusion method for intelligent power query, which combines the adaptive context-aware and multi-modal deep fusion natural language conversion technology of the power industry-specific knowledge base. The system integrates the proprietary graph knowledge base of the power industry, and uses information such as equipment type, operating rules, common faults, regulations and standards as background knowledge through semantic embedding and multi-modal data (such as text, images, and time series data). The system can not only identify the power professional terms implicit in the natural language context of the user's input, but also predict and intelligently complete the user's current query intention through a large model. For example, for the query "abnormalities of high-load transformers in the past week", the system can intelligently infer the specific parameters that the user is concerned about, automatically compare these descriptions with the background knowledge base, and generate a complete natural language query statement.

[0151] This embodiment proposes an intelligent error correction and query enhancement method driven by a hypergraph large model. The method combines a hypergraph neural network and a large language model, and performs real-time semantic analysis, intelligent error correction and query enhancement on the user's natural language query through a self-learning error correction cycle. The system uses multi-level relationship modeling in the hypergraph structure to better understand the user's implicit intentions, and automatically identifies semantic deviations, term confusion or logical errors in user query input, such as device name confusion, parameter setting errors, etc., and prompts possible correct options, while providing specific optimization suggestions through the graph knowledge base. The system also has multi-round interaction capabilities, and can dynamically generate candidate queries based on the fuzzy or conflicting information input by the user, and prioritize and recommend them in an interactive manner. This error correction and enhancement mechanism significantly improves the context adaptability of the query and the conversion accuracy of natural language to data language in the process of interacting with the user. And convert natural language into precise SQL or other data query languages, so that it can adapt to complex query scenarios across multiple tables and multiple conditions, and improve query accuracy and response speed.

[0152] This embodiment proposes a graph knowledge base maintenance method for semantic incremental update, which automatically detects and integrates new information in the power industry, including multimodal data such as equipment operation data, regulatory changes, operating procedures, and on-site images. The system uses semantic incremental technology to incrementally update the newly added labels and segmentation results after each input processing to avoid redundant reconstruction of the entire knowledge base. The system identifies changes in important information and automatically corrects or expands the content of the graph knowledge base, performs local optimization and node expansion of the existing knowledge graph, identifies the relationship between the new information and the existing content, and optimizes the structure and association path of the knowledge base. In particular, for recurring power engineering terms and specifications, the system can dynamically adjust their priority and retrieval weight, so that the knowledge base can dynamically reflect the latest industry knowledge and data. The method supports the joint retrieval of multimodal data and text in the power industry, and supports the retrieval and question-answering of multimodal data such as images, text, audio, and video. The system can perform semantic understanding and answer related questions for power equipment images, construction site videos, equipment operation sounds, etc., and realize the full process intelligence from data input to information acquisition. Among them, the multimodal large model can unify the feature representation of different types of data, and deeply fuse and align the features of multimodal data. The system supports device recognition in images, device voiceprint analysis in audio, and interpretation of construction processes in videos, and combines text question-and-answer modules to achieve cross-modal data association and intelligent answers. For example, when a user inquires about the operating status of a device, the system can combine device recognition in images, voiceprint analysis in sounds, and operating logs in text to provide users with multi-level comprehensive answers. This ability to associate and understand cross-modal data allows the system to not only answer direct queries, but also provide more comprehensive and intelligent answers to complex problems by analyzing the correlation of multimodal data.

[0153] This embodiment proposes a method for question intent diversion and multi-channel query processing, which combines the semantic parsing technology of the power industry-specific knowledge base and a large language model (such as GPT) to identify question intent and make diversion decisions. The system analyzes the natural language query input by the user, dynamically determines whether it belongs to a data query (such as equipment operation data, fault records, etc.) or a knowledge query (such as regulations, procedures, etc.), and assigns the question to a data question-and-answer system or a knowledge question-and-answer system. The system uses multimodal semantic parsing technology and a proprietary knowledge graph for the power industry to deeply understand the context and potential intent of the query, and dynamically decomposes sub-questions according to the complexity of the question to achieve cross-channel collaborative processing and comprehensive answers.

[0154] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 1, and the specific implementation process is the same, which will not be repeated here.

[0155] Embodiment 3 This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the adaptive graph database multimodal power knowledge question and answer method as described in the first embodiment above are implemented.

[0156] Embodiment 4 This embodiment provides a computer device, such as Figure 2 As shown, it includes a display device, an input device, a computer-readable storage medium (volatile memory and non-volatile storage medium), a processor, a communication interface (i.e., a network interface), and a computer program stored on the computer-readable storage medium and executable on the processor, wherein the processor, the communication interface, and the computer-readable storage medium can be connected via a bus or other means. The communication interface is used to receive and send data, and when the processor executes the program, the steps in the adaptive graph database multimodal power knowledge question and answer method described in the first embodiment are implemented.

[0157] Among them, any reference to memory, storage, database or other media provided in the present application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0158] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0159] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0161] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An adaptive graph database multimodal power knowledge question answering method, characterized in that: include: Obtain natural language queries for the power industry and generate a semantic nested tree using a large language model; For the data query subtask in the semantic nested tree, the natural language query is intelligently corrected and expanded, and after the extended query statement is obtained, the data results are extracted from the power industry database; For the knowledge query subtask in the semantic nested tree, after matching the data results with the nodes in the graph knowledge base, the knowledge results are obtained through the large language model based on the feature representation of the matching nodes and data results; Based on the knowledge results and data results, the natural language answer is obtained through the generative function of the large language model; Among them, the construction steps of the graph knowledge base include: obtaining new data, calculating the similarity between the new data and the corresponding nodes in the graph knowledge base, if the similarity is less than the threshold, updating the feature representation of the corresponding node; obtaining new nodes, calculating the association strength between the new nodes and all nodes in the graph knowledge base, if the association strength is greater than the set value, establishing an edge between the two nodes and giving the weight of the edge.

2. The adaptive graph database multi-modal power knowledge question answering method according to claim 1, characterized in that: The steps of intelligent error correction and expansion include: For natural language queries in the power industry and their related images, feature extraction is performed using a large language model and a deep learning model respectively, and then the features of different modalities are unified into the same representation in a two-way fusion manner using the knowledge graph and the visual question-answering graph neural network to obtain fused features. Based on the fused features, intelligent error correction is performed through a hypergraph neural network and multiple rounds of interaction. After optimizing the natural language query, an extended query statement is obtained through hypergraph reasoning.

3. The adaptive graph database multi-modal power knowledge question answering method according to claim 1, characterized in that: The initialization step of the graph knowledge base includes: taking different modal data as different node types, extracting the feature representation of each node, and defining the relationship between different nodes, giving the strength of the relationship, and obtaining the graph knowledge base.

4. The adaptive graph database multi-modal power knowledge question answering method according to claim 3, characterized in that: The node types include device nodes, regulation nodes, operating procedure nodes, image nodes, video nodes, audio nodes and segmented area nodes; Alternatively, the relationship includes a connection relationship, a device dependency relationship, a compliance relationship, an operation dependency relationship, and a subordinate relationship.

5. The adaptive graph database multi-modal power knowledge question answering method according to claim 1, characterized in that: In a graph database, if the similarity between two nodes is less than the similarity threshold, the two nodes are merged into one node; Alternatively, in a graph database, if there is more than one edge between two nodes, the weights of all edges are merged.

6. The adaptive graph database multi-modal power knowledge question answering method according to claim 1, characterized in that: The data query subtask includes querying equipment operation data, fault records and image data; Alternatively, the knowledge query subtask includes query of laws and regulations, operating procedures, pictures, videos and audios.

7. Adaptive graph database multimodal power knowledge question answering system, characterized by: include: A graph knowledge base construction module is configured to: obtain new data, calculate the similarity between the new data and the corresponding nodes in the graph knowledge base, and if the similarity is less than a threshold, update the feature representation of the corresponding node; obtain new nodes, calculate the association strength between the new nodes and all nodes in the graph knowledge base, and if the association strength is greater than a set value, establish an edge between the two nodes and give the weight of the edge; A data acquisition module is configured to: acquire natural language queries for the power industry and generate a semantic nested tree through a large language model; A data query module is configured to: for the data query subtask in the semantic nested tree, intelligently correct and expand the natural language query, and after obtaining the expanded query statement, extract the data result from the power industry database; The knowledge query module is configured to: for the knowledge query subtask in the semantic nested tree, after matching the data result with the node in the graph knowledge base, obtain the knowledge result through the large language model based on the feature representation of the matching node and the data result; Based on the knowledge results and data results, natural language answers are obtained through the generation function of the large language model.

8. The adaptive graph database multimodal power knowledge question answering system according to claim 1, characterized in that: It also includes an initialization module for a graph knowledge base, which is configured to: take different modal data as different node types, extract the feature representation of each node, define the relationship between different nodes, give the strength of the relationship, and obtain a graph knowledge base; the node types include device nodes, regulatory nodes, operating procedure nodes, image nodes, video nodes, audio nodes, and segmented area nodes; the relationships include connection relationships, device dependency relationships, compliance relationships, operation dependency relationships, and subordinate relationships.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the adaptive graph database multimodal power knowledge question and answer method as described in any one of claims 1 to 6 are implemented.

10. A computer device comprising a computer-readable storage medium, a processor, and a computer program stored in the computer-readable storage medium and executable on the processor, characterized in that: When the processor executes the program, the steps in the adaptive graph database multimodal power knowledge question and answer method as described in any one of claims 1-6 are implemented.

Images