Electric intelligent question and answer multi-mode retrieval method and system

Abstract

The invention belongs to the field of intelligent operation and maintenance of an electric power system, and provides an electric power intelligent question and answer multi-modal retrieval method and system. Converting the multi-modal data of the power industry into structured representation of nodes and edges of the knowledge graph; performing optimization and de-duplication processing on the edges between the data nodes to obtain optimized edges between the nodes, and constructing an initial knowledge graph; incremental change information is determined based on the newly added multi-modal data, and after neighborhood optimization and consistency check are adjusted according to the incremental change information and the edge weight, an optimized knowledge graph is generated; extracting multi-modal features, and aligning the multi-modal features in the same feature space to obtain aligned multi-modal features; and matching the query features with the knowledge graph, retrieving to obtain a node most related to the query, dynamically analyzing the content and associated information of the selected node, and generating corresponding multi-modal response data. And the complex query requirement of the user is met.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent operation and maintenance of power systems, and in particular to a method and system for intelligent question-answering multimodal retrieval of power systems. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, the power industry has gradually introduced multimodal data (such as text, images, audio, and video) for comprehensive management and analysis. For example, equipment operation data, inspection pictures, regulatory texts, and environmental audio all contain rich equipment status and operation information. With the rapid growth of data volume and the continuous changes in power equipment and regulations, how to timely and accurately convert multimodal data into effective knowledge information has become a key requirement for the intelligent development of the industry.

[0004] The application of intelligent question-answering systems in the power industry faces the challenge of multimodal data fusion. There is heterogeneity between multimodal data. For example, the semantic information of text, the visual features of images, and the sound features of audio each have their own unique representation forms, and it is very difficult to directly integrate this information. Existing intelligent question-answering systems can often only process a single type of data, such as text-based questions and answers or simple knowledge retrieval, and cannot comprehensively consider information from multiple data modalities. The deep fusion of multimodal information, especially the combination of text and non-text modalities (such as images and videos), can significantly improve the quality of answers to complex queries. For example, when it is necessary to judge the operating status of equipment, combining text records of equipment operating data, multimodal information of on-site images and ambient audio can provide more accurate judgments, but it is difficult for existing systems to achieve this function.

[0005] In addition, traditional knowledge bases in the power industry mostly adopt a static structure and lack the ability to update in real time and maintain automatically. Usually, such knowledge bases require manual data updates and maintenance on a regular basis, which is costly, inefficient, and difficult to reflect the latest industry changes in a timely manner. Especially when faced with dynamic information such as regulatory updates and equipment status changes, static knowledge bases cannot meet the requirements of real-time queries. In addition, traditional knowledge bases are mostly based on single-modal data, such as text information, and cannot fully utilize the complementary information of multi-modal data such as images, audio, and video. Such systems are difficult to achieve accurate and comprehensive responses when faced with multi-modal complex queries. Summary of the invention

[0006] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention provides a multimodal retrieval method and system for electric power intelligent question and answer, which proposes an electric power intelligent question and answer system based on a graph knowledge base with semantic incremental update and multimodal spatiotemporal comparative learning. Through the alignment and deep fusion of multimodal features, the system can process multimodal data, automatically update the knowledge base, and provide accurate and real-time responses.

[0007] In order to achieve the above object, the present invention adopts the following technical solution: A first aspect of the present invention provides a multi-modal retrieval method for electric power intelligent question and answer, comprising the following steps: Obtain multimodal data for the power industry; Convert the multimodal data of the power industry into a structured representation of knowledge graph nodes and edges; After optimizing and removing duplicates from the edges between data nodes, the optimized edges between nodes are obtained, and the initial knowledge graph is constructed; Determine incremental change information based on newly added multimodal data, and generate an optimized knowledge graph after adjusting neighborhood optimization and consistency check based on incremental change information and edge weights; Extract multimodal features, align the multimodal features in the same feature space, and obtain aligned multimodal features; According to the aligned multimodal features, the query is represented as a query feature, the query feature is matched with the optimized knowledge graph, the nodes most relevant to the query are retrieved, the content and associated information of the selected nodes are dynamically analyzed, and the corresponding multimodal response data is generated.

[0008] Furthermore, the data nodes and the edges between the nodes are optimized and deduplicated, including: For duplicate nodes of the same entity, if , the node and merged into one node, where Represents the similarity between two nodes, calculated by the cosine similarity of the feature vectors. is the similarity threshold; For edge redundancy detection, define the edge weight update rule. If there is already an edge between two identical nodes , then the weight of the newly added edge With existing edge weights to merge.

[0009] Further, the determining of the incremental change information based on the newly added multimodal data includes: Extract semantic feature representation of each modality’s newly added data; The semantic features of the newly added data are calculated with the semantic feature vectors of the corresponding nodes in the knowledge base. If the similarity result is less than the set threshold, the newly added data is marked as a change point that needs to be updated. Determine the change type of the change point, and generate corresponding incremental update information for each change point; The generated incremental update information is added to the knowledge base, the feature vector of the corresponding node is updated, and the associated edges of the node are readjusted.

[0010] Furthermore, according to the incremental change information, the edge weights are adjusted for neighborhood optimization and consistency checking, including: Determine the semantic similarity between the newly added node and the existing node. If the similarity exceeds the set threshold, establish an edge between the newly added node and the existing node. When a new node is connected to an existing node, the weight of the edge is adjusted according to the semantic similarity between the new node and the existing node to obtain the edge with adjusted weight; After the integration of new nodes and edges is completed, the affected node neighborhoods in the knowledge graph are optimized to remove redundant or weakly associated nodes; After adjusting the neighborhood structure, check whether the nodes or edges in the knowledge graph are redundant. If so, mark and remove the redundant nodes or edges.

[0011] Furthermore, the multimodal features are aligned in the same feature space to obtain aligned multimodal features, including: For data of different modes, extract the feature vector of the corresponding mode; The generated multimodal feature vectors are used to construct positive samples and negative samples, and the similarities of the positive samples and negative samples are calculated respectively. When calculating the similarity, the event information and spatial information of the multimodal features are introduced; By maximizing the similarity of positive samples and minimizing the similarity of negative samples, multimodal features are aligned in the same feature space.

[0012] Furthermore, the solution also includes automatically adjusting the structure and feature alignment strategy of the knowledge graph based on the query and multimodal response, including: Analyze query patterns to identify query types and information needs; When the access frequency of a node increases, the associated edges of the node are dynamically increased, and the weight of the edges is increased; If the number of data queries for a certain modality increases, increase the feature weight of the modality.

[0013] Furthermore, multimodal data includes equipment operation data, regulatory changes, operating procedures, and on-site images.

[0014] A second aspect of the present invention provides an electric power intelligent question-answering multimodal retrieval system, comprising: A data acquisition module, which is used to acquire multimodal data of the power industry; The knowledge graph construction module is used to convert the multimodal data of the power industry into a structured representation of knowledge graph nodes and edges; after optimizing and removing duplicates on the data nodes and the edges between the nodes, the optimized nodes and the edges between the nodes are obtained, and the initial knowledge graph is constructed; The knowledge graph optimization module is used to determine the incremental change information based on the newly added multimodal data, and generate the optimized knowledge graph after adjusting the neighborhood optimization and consistency check according to the incremental change information and edge weights; A multimodal data alignment module is used to extract multimodal features and align the multimodal features in the same feature space to obtain aligned multimodal features; The query module is used to represent the query as a query feature based on the aligned multimodal features, match the query feature with the optimized knowledge graph, retrieve the nodes most relevant to the query, dynamically analyze the content and associated information of the selected nodes, and generate corresponding multimodal response data.

[0015] A third aspect of the present invention provides a computer-readable storage medium.

[0016] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the electric power intelligent question-answering multimodal retrieval method as described above.

[0017] A fourth aspect of the present invention provides a computer device.

[0018] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the above-mentioned multimodal retrieval method for intelligent question and answer of electric power are implemented.

[0019] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention innovatively proposes an intelligent question-answering retrieval technology that integrates incremental updates and multimodal fusion, and introduces an updated graph knowledge base maintenance method, so that the knowledge base can automatically integrate new multimodal data in the power industry, overcoming the limitations of traditional knowledge bases. While ensuring the consistency of multimodal data and the real-time nature of the knowledge base, it can provide the power industry with dynamic and accurate intelligent question-answering services, meet the industry's urgent needs for complex data processing and real-time knowledge acquisition, and achieve dynamic data management and comprehensive improvement of multimodal question-answering.

[0020] 2. The present invention introduces semantic incremental update technology, which enables the knowledge base to automatically integrate new data in the power industry, including multimodal content such as equipment operation information, regulatory updates, operating procedures, and on-site images. Traditional systems often require manual regular updates of the knowledge base, but the incremental update method of this patent significantly reduces the workload of manual maintenance, ensures that the knowledge base reflects the latest industry knowledge in real time, enables users to query the latest information at any time, and significantly improves the applicability and effectiveness of the system.

[0021] 3. The spatiotemporal contrast learning framework of the present invention further improves the accuracy of multimodal feature alignment. Data of different modalities often have different feature spaces, so alignment processing is required. Through the spatiotemporal contrast learning framework, the system accurately aligns the different modal features of the same instance in the same space, so that the system can also provide high-precision retrieval results when processing fuzzy queries. For example, a fuzzy query proposed by a user can provide the user with a comprehensive answer with semantic relevance based on the alignment results of multimodal features, ensuring that the system can accurately understand and respond to user needs.

[0022] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0024] Figure 1 It is a flow chart of an intelligent question-answering retrieval method based on knowledge base updating and multimodal fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0026] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0027] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0028] In the process of addressing the above challenges, the alignment and fusion of multimodal features have become technical difficulties. Data of different modes are usually distributed in different feature spaces, and there is a lack of direct alignment methods, which makes multimodal feature fusion a complex project. In addition, the knowledge of the power industry is updated quickly, and the traditional static knowledge base structure cannot meet the needs of timely updates. Therefore, an incremental update mechanism is needed that can dynamically capture new data and effectively integrate it into the existing knowledge base so that the system can reflect the latest industry knowledge and equipment status changes at any time.

[0029] In addition, the introduction of spatiotemporal contrastive learning provides a new approach to solving the deep fusion of multimodal data and the completion of missing information. The spatiotemporal contrastive learning method can achieve effective alignment between multimodal features, and by utilizing the temporal and spatial information of the same instance, multimodal features can maintain consistency in the same space. This method has great advantages in multimodal question-answering systems, especially in complex query scenarios that require reasoning or comprehensive analysis.

[0030] In summary, the present invention proposes an electric power intelligent question-answering system based on a semantic incremental updated graph knowledge base and multimodal spatiotemporal comparative learning. Through the alignment and deep fusion of multimodal features, the system can process multimodal data, automatically update the knowledge base, and provide accurate and real-time responses. It can help the electric power industry overcome the limitations of traditional knowledge bases, realize dynamic data management and comprehensive improvement of multimodal question-answering, and meet the needs of the rapidly developing electric power industry for intelligent and multimodal data processing.

[0031] Embodiment 1 This embodiment provides an intelligent question-answering retrieval method based on knowledge base updating and multimodal fusion, comprising the following steps: Step 1: Obtain multimodal data of the power industry; In this embodiment, the multimodal data includes equipment operation data, regulatory changes, operating procedures, and on-site images.

[0032] Among them, equipment operation data usually contains real-time monitoring information, such as current, voltage, power and temperature, etc., through which the operating status and health of the equipment can be evaluated.

[0033] Regulatory change data covers the latest policies, regulations and safety standards issued by the government and industry, and is an important basis for guiding equipment operation and management.

[0034] The operating procedures include standardized operating procedures and emergency handling plans for various equipment to ensure that operations in daily and emergency situations comply with regulations.

[0035] On-site image data is captured by drones or collected by on-site monitoring, which can provide visual information of the equipment and surrounding environment to assist fault diagnosis and on-site inspection.

[0036] Step 2: Preprocess the acquired multimodal data of the power industry; Formatting and storing text data such as regulations and operating procedures, and converting the content into a structured format (such as JSON or XML) ensures that the text information is easy to process and integrate in the subsequent knowledge base construction.

[0037] For equipment operation data, since it is usually tabular data, it can be directly stored in a structured table format to facilitate subsequent data query and analysis.

[0038] In addition, in order to ensure that on-site images can be accurately associated with other modal data, detailed metadata tags need to be added to the image data, such as shooting time, device identification, and geographic location.

[0039] In the processing of image data, the system uses visual semantic segmentation technology to further extract key semantic information in the image. Suppose the image data set is I ,in I ={I1,I2,…,In}, represents a set of images collected from multiple sites. For each image I i , the system uses semantic segmentation function f Generate segmented image S i ,Right now: , in, The segmented image contains the semantic information of each marked area, such as equipment, background, environment, etc. This segmentation technology can help the system automatically identify the core objects in the image and convert them into a standardized data format for subsequent feature extraction and analysis.

[0040] Assume that the multimodal data set is , which contains equipment operation data , Regulatory Data , Operating Procedures and image data ,Right now: , In the process of building the knowledge graph, the segmentation area of ​​the image data , each marked area Can be further mapped as graph nodes , and generate node feature vector Therefore, the image Ii The representation in the graph structure is: , , in, g () is the area mapping function, h () is the feature extraction function.

[0041] Ultimately, the system will standardize all modal data to ensure that the formats and structures of various types of data are consistent, facilitating subsequent integration operations, thereby achieving efficient fusion of multimodal power data and providing basic support for the dynamic update of the graph knowledge base.

[0042] Step 3: Build a knowledge graph based on the acquired multimodal data; For each type of data, it is converted into a structured representation of knowledge graph nodes and edges. Equipment operation data, regulatory data, operating procedure data, and image data are mapped to graph structures respectively. Nodes in and edge , where nodes represent entities such as equipment, procedures, and environment, and edges represent the relationships between these entities.

[0043] The specific steps include: Step 301: Convert different multimodal data into data nodes; Create different multimodal data nodes and obtain the node set by finding their union; Construct a node set of the knowledge graph to represent different types of power industry knowledge entities. Define the graph structure ,in Represents a collection of nodes. Represents the edge set between nodes. Create different node types for different types of multimodal data (equipment operation data, regulations, operating procedures, images, etc.): Device Node V dev : Indicates power equipment and its status information.

[0044] Regulation Node V reg : Represents policy and regulatory information for the power industry.

[0045] Operation Procedure Node V proc : Indicates the operating procedures and procedural steps of the equipment.

[0046] Image Node V img : Represents device-related image information (objects or scenes extracted through image segmentation).

[0047] Therefore, the node set can be represented as: , Step 302: assign a feature vector to each node, expressed as: , in, , , , The feature vectors respectively represent the equipment, regulations, procedures and image nodes; these feature vectors can be obtained through multimodal feature extraction methods, such as using visual semantic segmentation to process image data, text vectorization to process regulation and procedure text data, etc.

[0048] Step 303: Define the relationship between nodes, i.e. the edge in the knowledge graph .

[0049] Edge Set Used to describe the relationship between different node types. For example: The relationship between devices: such as connection relationship, dependency relationship, etc., is used to represent the physical or logical association between devices.

[0050] The relationship between equipment and regulations: For example, the compliance relationship indicates the regulatory provisions that the equipment must comply with during operation.

[0051] The relationship between equipment and operating procedures: such as operation dependency, which indicates the operating steps that a certain equipment needs to follow at different operation stages.

[0052] Relationship between equipment and images: image information of specific equipment in different environments, used for fault diagnosis and equipment status monitoring.

[0053] Assume Edge The weight of , indicating the strength or importance of the relationship, then the edge is represented as: , in, represents an edge between two nodes, Indicates the weight corresponding to the edge.

[0054] Step 304: Optimize and remove duplicates on the edges between the data nodes to obtain optimized edges between the nodes; To ensure that the knowledge base structure is free of redundancy, the graph structure needs to be optimized, especially the deduplication of nodes and edges. First, for duplicate nodes of the same entity, the following conditions are used to determine whether they are redundant: if , the node and Merge into one node to reduce duplicate information.

[0055] in, Represents the similarity between two nodes, which can be calculated by the cosine similarity of the feature vectors. is the similarity threshold.

[0056] For edge redundancy detection, edge weight update rules are defined to prevent the occurrence of duplicate edges.

[0057] If there is already an edge between two identical nodes , then the weight of the newly added edge With existing edge weights To perform the merge: , This ensures that the graph structure has a reasonable weight distribution when representing multimodal data associations.

[0058] Step 305: Generate a complete initial graph structure based on node construction, feature assignment, edge definition and structure optimization; through this initial construction step, the knowledge base has a complete representation of power industry information and can maintain data consistency during the incremental update process.

[0059] Step 4: Determine incremental change information based on the newly added multimodal data; Step 401, extracting semantic feature representation of each modality's newly added data; For each type of newly added data (such as newly added equipment status information, regulatory texts, operating procedures, and on-site images), its semantic feature representation is first generated through feature extraction technology; let the newly added multimodal data set be Dnew ={ d 1, d 2, ..., dn}, where d i Indicates i For each data item d i , through the feature extraction function f Get its eigenvector ,Right now: , in, f Represents a feature extraction function, which uses different feature extraction methods for different modal data, such as using a text embedding model to process text data and using visual semantic segmentation to process image data.

[0060] Step 402: Calculate the semantic similarity between the semantic features of the newly added data and the semantic feature vectors of the corresponding nodes in the knowledge base of the knowledge graph constructed in step 3, and determine whether the corresponding nodes need to be marked as change points; The features of the newly added data are represented The feature vector of the corresponding node in the knowledge base Perform semantic similarity calculation to detect changes between new data and existing data. The similarity calculation function for: , in, The value range of is [0,1]. The closer the value is to 1, the higher the similarity between the new data and the existing knowledge base data.

[0061] By setting a threshold θ, we can determine whether the new data is significantly different from the existing data.

[0062] when <θ, it is considered d and vj There is a big difference, so d Marked as the change point that needs to be updated. The value of θ can be adjusted according to the specific application scenario to ensure the accuracy and recall rate of change point detection.

[0063] Step 403: Determine the change type of the change point, and for each change point d i , generate corresponding incremental update information; Once a change point is detected, it is further classified into types such as equipment status change, procedure update, or field environment change. Suppose the classification function of the change point is , then each change point is marked as a specific update type:

[0064] This classification helps in targeted updating of relevant nodes and relationships in the graph knowledge base.

[0065] For each change point d i , generate the corresponding incremental update information. Let the update information be , then the generation process of update information can be expressed as: , in, vj For the nodes that need to be updated, is the feature vector of the change point, and the "association edge" represents the edge between the change point and other related nodes in the knowledge base. For example, if a change in the device state is detected, d i The edges associated with the device nodes are updated to the latest status to reflect the current operating status.

[0066] Step 404: Generated update information Add it to the knowledge base, update the feature vector of the corresponding node, and readjust the associated edges of the node.

[0067] Through local optimization, we ensure that the location and semantic relationship of the newly added information in the knowledge base are reasonable. For example, for a device node whose state has changed, the feature vector of the updated node is: , Among them, α∈[0,1] is the smoothing coefficient, which is used to control the degree of fusion of new information and existing information to prevent the excessive influence of new information on the node state.

[0068] Through automated knowledge base updates and semantic incremental maintenance methods, the system reduces the labor cost of knowledge base maintenance. In traditional methods, the power knowledge base often requires professionals to regularly update and review data, while the automatic update and semantic maintenance methods adopted by the patent significantly reduce labor consumption and improve system efficiency, allowing the system to be deployed on a large scale at a lower cost and maintain its efficient operation.

[0069] Step 5: Based on the incremental change information, edge weight adjustment, neighborhood optimization, and consistency check, the system generates an optimized knowledge graph; The steps include: Step 501: Determine the semantic similarity between the newly added node and the existing node to determine whether a connection needs to be established; For the newly added node information identified in change detection, it is not directly added independently, but integrated based on the association with the existing nodes.

[0070] For new nodes v new and existing nodes v existing , through the similarity function Calculate their correlation strength, if it exceeds the set threshold , it is considered that there is a strong correlation between the two, and an edge needs to be established between them. e : , Step 502: After the new node is connected to the existing node, in order to ensure the accuracy of the association relationship, the weight of the edge is adjusted to reflect the actual association strength between the nodes.

[0071] The initial weight of the edge is set to winit, and it is adjusted according to the similarity of the nodes to ensure the dynamic nature of the relationship in the knowledge base. The final weight of the edge is w e The calculation formula is: , in, is the adjustment coefficient, which is used to balance the influence of the initial weight and similarity. This step ensures that the strength of the relationship between the new information and the nodes in the existing knowledge base reasonably reflects the actual situation and avoids overly weakening or strengthening certain associations.

[0072] Step 503: After the integration of the newly added nodes and edges is completed, the affected node neighborhoods in the knowledge graph are optimized to remove redundant or weakly associated nodes.

[0073] Specifically, for each node v Neighborhood set of N ( v ) to update and remove nodes whose correlation is lower than the threshold to ensure the simplicity of the graph structure. v With neighboring nodes u The correlation degree is g( v , u ), the optimized representation of the neighborhood set is: , Among them, γ is the minimum correlation threshold of neighborhood optimization. This optimization step reduces unnecessary connections and improves the efficiency of graph representation.

[0074] Step 504: After adding new nodes and edges and adjusting the neighborhood structure, a consistency check is performed to ensure that the updated knowledge base has no structural conflicts.

[0075] The checks include node uniqueness, edge rationality, and graph connectivity.

[0076] For redundant nodes or edges in the graph, the system marks and removes them after detecting them during the inspection process. If two nodes are highly similar in semantics (i.e., the similarity is close to 1), they are merged into one node to reduce information redundancy. The consistency check function C(Gnew) is used to verify whether the graph structure meets the consistency requirements: , Step 505: After incremental update, edge weight adjustment, neighborhood optimization and consistency check, the system generates an optimized knowledge graph Gnew=(Vnew,Enew), ensuring that it reflects the latest industry knowledge and maintains a simple and reasonable structure. This optimized graph provides a dynamic and accurate knowledge basis for subsequent queries and intelligent question-answering, while avoiding unnecessary redundant information, ensuring improved system performance and efficient retrieval.

[0077] Through automated knowledge base updates and semantic incremental maintenance methods, the system reduces the labor costs of knowledge base maintenance. In traditional methods, the power knowledge base often requires professionals to regularly update and review data, while the automatic update and semantic maintenance methods adopted by the present invention significantly reduce manpower consumption and improve system efficiency, allowing the system to be deployed on a large scale at a lower cost and maintain its efficient operation. These advantages combined enable the patented intelligent question-answering retrieval system to provide dynamic, accurate and comprehensive services in the power industry to meet the ever-changing needs of the industry.

[0078] Step 6: Fuse the extracted multimodal features; The steps include: Step 601: For data of different modalities, a suitable feature extraction model is used to generate a feature vector representation thereof: In this embodiment, text data: The GPT model is used to process the text data and map the text content into feature vectors. , the formula is: , Where T is text data.

[0079] Image data: Use the Visual Transformer (ViT) model to process image data and extract deep feature vectors of the image , the formula is: , Where I represents the input image.

[0080] Audio data: Use the Transformer for Audio (TFA) model to process audio data and extract audio feature vectors , the formula is: , Among them, A is audio data.

[0081] Video data: For video data, extract key frames and use ViT or other suitable video models for feature extraction to represent the video as a feature vector : , in, A collection of keyframes representing video data.

[0082] Step 602: construct positive samples and negative samples from the generated multimodal feature vectors, and calculate the similarities of the positive samples and the negative samples respectively; For each data instance, extract the feature vector of its text, image, audio, and video modalities, such as instancei Text features , Image features , audio features and video features . A positive sample pair is a multimodal feature pair of the same instance, such as ( , )and( , ); while the negative sample pairs are composed of modal features of different instances, such as ( , )and( , ), where i≠j≠k.

[0083] In order to achieve the alignment of multimodal features, the similarity is calculated for positive sample pairs and negative sample pairs respectively.

[0084] The similarity function s can choose cosine similarity or Euclidean distance. And negative sample pairs , the similarity is calculated as and ,in Aims to maximize the similarity of positive sample pairs, while It is used to minimize the similarity of negative sample pairs.

[0085] To this end, the contrast loss function Lcontrastive is defined, whose goal is to maximize the similarity of positive samples and minimize the similarity of negative samples by optimization. The specific loss function can be expressed as: , Among them, the first term is the loss of positive sample pairs, which is obtained by As close to 1 as possible to ensure that the positive sample pairs are highly aligned in the feature space. The second term is the loss of negative sample pairs, which is minimized And set the threshold δ to control the similarity of negative sample pairs to prevent the features between negative samples from being too close.

[0086] Step 603: Introduce spatiotemporal information into similarity calculation to ensure that different modal features of the same instance remain consistent in time and space.

[0087] Assume that the modal features of the instance contain temporal information t and spatial information l , then the calculation of similarity is expanded to: , Among them, α and β are the attenuation coefficients of time and space, which are used to control the influence of time and space distance on similarity. This formula ensures the consistency of multimodal features of the same instance in time and space, making the alignment of multimodal features more reasonable and tight.

[0088] During the training process, the system continuously optimizes the similarity of positive sample pairs and reduces the similarity of negative sample pairs by minimizing the contrast loss function Lcontrastive, gradually realizing the alignment of multimodal features in the same feature space. In this way, the system generates a unified feature space so that data of different modalities can be highly consistent in the feature space, thereby providing more accurate multimodal retrieval results when facing fuzzy queries. Ultimately, through the aligned multimodal feature representation, the system can achieve a comprehensive understanding of multimodal data in intelligent question and answer and data retrieval, ensuring the accuracy and consistency of query responses.

[0089] Step 7: Generate query and response of the intelligent question answering system.

[0090] The specific steps include: Step 701: Represent the user's query Q as a query feature vector x Q , in order to match the multimodal features in the knowledge base in the feature space.

[0091] Assuming the query is in text form, the system converts it into a feature vector through an embedding model (such as the GPT model): , This feature vector x Q Located in the same alignment space as the multimodal features of the knowledge base, so as to calculate similarity with nodes in the graph knowledge base.

[0092] Step 702: semantically match the query feature vector with the graph knowledge base to retrieve the nodes most relevant to the query; The system performs semantic matching in the graph knowledge base and calculates the query feature vector x Q and the characteristics x of each node in the graph V similarity to find the most relevant node or node combination.

[0093] Similarity s( x Q , x V ) can be calculated using cosine similarity: , The system retrieves the node V that is most relevant to the query from the knowledge base based on the similarity value. relevant , in order to generate targeted responses to queries.

[0094] Step 703: Dynamically analyze the content and associated information of the selected node and generate corresponding response data; To provide more accurate suggestions, the system dynamically analyzes the content and associated information of the selected nodes. For example, if the query involves the status of the device, the system extracts the device operation data node and the related abnormal information node from the knowledge base, evaluates the current status and generates processing suggestions in combination with the corresponding historical data and abnormal situations.

[0095] Assuming the device status is S(t) and the anomaly detection result is E(t), the processing suggestion R can be generated by the following logic: , in, f It is a rule or model function that automatically generates corresponding response suggestions based on the current status of the device and abnormal conditions.

[0096] Finally, the system integrates multimodal information to generate a detailed response for the user. If the user's query requires a multimodal explanation (such as a text description, an image example, or an audio prompt), the system obtains data from the relevant modalities from the graph knowledge base and integrates it into a unified answer. Assuming that the final response contains a text answer Tresp and an image example Iresp, the generated multimodal response is: , The response can comprehensively cover different aspects of the user's query and provide the user with specific processing suggestions and reference information.

[0097] The intelligent question-and-answer interface and automatic response generation function greatly improve the user experience. Users can enter queries through a unified interface, and the system quickly performs semantic matching based on aligned multimodal features and dynamically updated knowledge bases, generating detailed multimodal response results such as device status analysis and processing suggestions. This question-and-answer mode allows users to query without being limited to a single modality and directly obtain multimodal integrated information, greatly enhancing the interactive experience of the system and the richness of the response.

[0098] Step 8: Automatically adjust the structure and feature alignment strategy of the graph knowledge base based on new user queries and responses; The specific steps include: Step 801: Analyze the user query pattern to identify common query types and key information needs.

[0099] For example, the system can analyze the frequency and content of queries to identify specific topics that users focus on (such as equipment operating status, abnormal diagnosis, or regulatory updates). Based on this pattern analysis, the system can dynamically generate query templates to identify users' high-frequency needs and strengthen the priority of relevant knowledge nodes.

[0100] After each response is generated, the system evaluates the effectiveness of the response to ensure that it meets user needs and provides accurate information. The effectiveness evaluation can calculate the matching degree between the user query Q and the response R by the following formula: , in, is the similarity between query and response features. When the match is low, the system marks the response as needing improvement and triggers the graph structure optimization process.

[0101] Step 802: Based on the focus of the user's query and the effectiveness feedback of the response, the system adjusts the structure of the knowledge base by increasing the connection of important nodes or reducing redundant connections to ensure that the nodes that the user pays most attention to receive a higher access priority.

[0102] Assume that a node v If the access frequency of a node increases, the system can dynamically increase the associated edges of the node. e , and adjust its weight w e , so that it is displayed first in the search. The weight update formula is as follows: , Among them, γ is the adjustment coefficient, f 访问频率 Represents the access frequency function of the node. This step optimizes the graph structure to ensure that important information is more prominent in user queries, thereby improving the retrieval effect.

[0103] Step 803: The system will dynamically adjust the alignment strategy of the multimodal features to adapt to changes in user needs.

[0104] If a modality data shows high importance in multiple queries (for example, image data is more critical in equipment fault diagnosis), the system will increase the feature weight of the modality. Suppose the alignment weight is adjusted to α m , for important modes m The feature weights are updated as: , Among them, λ is the adjustment parameter, f 模态重要性 is the importance function of the modality in a specific query. This dynamic adjustment of the alignment strategy ensures more accurate extraction of multimodal information in specific tasks.

[0105] The above technical solution is based on the characteristics of adaptive adjustment, and can automatically optimize the structure and feature alignment strategy of the graph knowledge base based on the frequency of user queries and response effect feedback. This adaptive mechanism ensures that the system can adapt to the rapid changes in the industry at any time and remain sensitive to user needs. With changes in regulations, technological advances, and equipment upgrades in the power industry, the system can continuously adjust its internal structure and information processing methods, so that the system's question-answering and retrieval effects are always at the forefront of the industry, thereby providing high-quality user services.

[0106] Embodiment 2 This embodiment provides an electric power intelligent question-answering multimodal retrieval system, including: A data acquisition module, which is used to acquire multimodal data of the power industry; The knowledge graph construction module is used to convert the multimodal data of the power industry into a structured representation of knowledge graph nodes and edges; after optimizing and removing duplicates on the data nodes and the edges between the nodes, the optimized nodes and the edges between the nodes are obtained, and the initial knowledge graph is constructed; The knowledge graph optimization module is used to determine the incremental change information based on the newly added multimodal data, and generate the optimized knowledge graph after adjusting the neighborhood optimization and consistency check according to the incremental change information and edge weights; A multimodal data alignment module is used to extract multimodal features and align the multimodal features in the same feature space to obtain aligned multimodal features; The query module is used to represent the query as a query feature based on the aligned multimodal features, match the query feature with the optimized knowledge graph, retrieve the nodes most relevant to the query, dynamically analyze the content and associated information of the selected nodes, and generate corresponding multimodal response data.

[0107] Embodiment 3 This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the electric power intelligent question and answer multimodal retrieval method as described above are implemented.

[0108] Embodiment 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the multimodal retrieval method for intelligent electric power question and answer are implemented as described above.

[0109] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0110] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0111] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0113] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal retrieval method for intelligent power question answering, characterized in that: The steps include: Obtain multimodal data for the power industry; Convert the multimodal data of the power industry into a structured representation of knowledge graph nodes and edges; After optimizing and removing duplicates from the edges between data nodes, the optimized edges between nodes are obtained, and the initial knowledge graph is constructed; Determine incremental change information based on newly added multimodal data, and generate an optimized knowledge graph after adjusting neighborhood optimization and consistency check based on incremental change information and edge weights; Extract multimodal features, align the multimodal features in the same feature space, and obtain aligned multimodal features; According to the aligned multimodal features, the query is represented as a query feature, the query feature is matched with the optimized knowledge graph, the nodes most relevant to the query are retrieved, the content and associated information of the selected nodes are dynamically analyzed, and the corresponding multimodal response data is generated.

2. The electric power intelligent question-answering multimodal retrieval method according to claim 1, characterized in that: Optimize and remove duplicate data nodes and edges between nodes, including: For duplicate nodes of the same entity, if , the node and merged into one node, where Represents the similarity between two nodes, calculated by the cosine similarity of the feature vectors. is the similarity threshold; For edge redundancy detection, define the edge weight update rule. If there is already an edge between two identical nodes , then the weight of the newly added edge With existing edge weights to merge.

3. The electric power intelligent question-answering multimodal retrieval method according to claim 1, characterized in that: The determining of incremental change information based on the newly added multimodal data includes: Extract semantic feature representation of each modality’s newly added data; The semantic features of the newly added data are calculated with the semantic feature vectors of the corresponding nodes in the knowledge base. If the similarity result is less than the set threshold, the newly added data is marked as a change point that needs to be updated. Determine the change type of the change point, and generate corresponding incremental update information for each change point; The generated incremental update information is added to the knowledge base, the feature vector of the corresponding node is updated, and the associated edges of the node are readjusted.

4. The electric power intelligent question-answering multimodal retrieval method according to claim 1, characterized in that: According to the incremental change information, edge weights are adjusted for neighborhood optimization and consistency checks, including: Determine the semantic similarity between the newly added node and the existing node. If the similarity exceeds the set threshold, establish an edge between the newly added node and the existing node. When a new node is connected to an existing node, the weight of the edge is adjusted according to the semantic similarity between the new node and the existing node to obtain the edge with adjusted weight; After the integration of new nodes and edges is completed, the affected node neighborhoods in the knowledge graph are optimized to remove redundant or weakly associated nodes; After adjusting the neighborhood structure, check whether the nodes or edges in the knowledge graph are redundant. If so, mark and remove the redundant nodes or edges.

5. The electric power intelligent question-answering multimodal retrieval method according to claim 1, characterized in that: Align the multimodal features in the same feature space to obtain aligned multimodal features, including: For data of different modes, extract the feature vector of the corresponding mode; The generated multimodal feature vectors are used to construct positive samples and negative samples, and the similarities of the positive samples and negative samples are calculated respectively. When calculating the similarity, the event information and spatial information of the multimodal features are introduced; By maximizing the similarity of positive samples and minimizing the similarity of negative samples, multimodal features are aligned in the same feature space.

6. The electric power intelligent question-answering multimodal retrieval method according to claim 1, characterized in that: The solution also includes automatically adjusting the structure and feature alignment strategy of the knowledge graph based on the query and multimodal response, including: Analyze query patterns to identify query types and information needs; When the access frequency of a node increases, the associated edges of the node are dynamically increased, and the weight of the edges is increased; If the number of data queries for a certain modality increases, increase the feature weight of the modality.

7. The electric power intelligent question-answering multimodal retrieval method according to claim 1, characterized in that: Multimodal data includes equipment operating data, regulatory changes, operating procedures, and on-site images.

8. The power intelligent question-answering multimodal retrieval system is characterized by: include: A data acquisition module, which is used to acquire multimodal data of the power industry; The knowledge graph construction module is used to convert the multimodal data of the power industry into a structured representation of knowledge graph nodes and edges; after optimizing and removing duplicates on the data nodes and the edges between the nodes, the optimized nodes and the edges between the nodes are obtained, and the initial knowledge graph is constructed; The knowledge graph optimization module is used to determine the incremental change information based on the newly added multimodal data, and generate the optimized knowledge graph after adjusting the neighborhood optimization and consistency check according to the incremental change information and edge weights; A multimodal data alignment module is used to extract multimodal features and align the multimodal features in the same feature space to obtain aligned multimodal features; The query module is used to represent the query as a query feature based on the aligned multimodal features, match the query feature with the optimized knowledge graph, retrieve the nodes most relevant to the query, dynamically analyze the content and associated information of the selected nodes, and generate corresponding multimodal response data.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the electric power intelligent question-answering multimodal retrieval method as described in any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the electric power intelligent question and answer multimodal retrieval method according to any one of claims 1 to 7 are implemented.

Images