A multi-modal fusion power equipment defect knowledge graph construction method and system

Abstract

The application discloses a kind of multi-modal fusion's power equipment defect knowledge graph construction method and system, it is related to digital model application technical field, the method of the present application includes multi-modal data fusion, knowledge graph construction and graph reasoning early warning, first pass through multidimensional feature vector, calculate cross-modal correlation strength index, filter out relevant unique code, extract effective dimension, construct power multidimensional cross-modal fusion feature vector, secondly, extract entity and attribute from fusion feature vector, construct initial knowledge graph;Based on motor equipment similarity law, supplement initial graph, form strengthened knowledge graph, finally according to current equipment inspection data matching to staff prompt relevant information, defect early warning, the present application improves feature representation ability by multi-modal data deep fusion, relies on similarity law to supplement and optimize the integrity of knowledge graph, finally through graph reasoning early warning, improve the effect of power equipment defect management.

Description

Technical Field

[0001] This invention relates to the field of digital model application technology, specifically to a method and system for constructing a multimodal fusion knowledge graph of power equipment defects. Background Technology

[0002] With the expansion of power grid scale, diversification of equipment types, and increasing complexity of operating environments, the incidence of power equipment defects has significantly increased. If defects are not detected and warned in a timely manner, they may lead to equipment failures, power outages, or even major safety risks such as fires and explosions. Therefore, a multimodal fusion method and system for constructing a knowledge graph of power equipment defects is needed.

[0003] Existing technologies, such as the invention patent application CN116612481B (publication number CN116612481B), provide a method and system for power equipment defect identification based on knowledge graphs and multi-source images. This relates to the field of power equipment image recognition technology. The method includes acquiring multiple inspection images of power equipment; performing text attribute recognition and defect recognition on the inspection images to obtain power equipment text data and power equipment classification results with defect tags; defining a first defect image tag and a second attribute text tag; associating the first defect image tag with the second attribute text tag; using the second attribute text tag and the first defect image tag, constructing a multimodal knowledge graph containing power equipment triples and image and text tag knowledge for each entity; representing the multi-source heterogeneous data of images and text; and outputting visualized results of power equipment defects. This disclosure improves the efficiency of power equipment inspection.

[0004] The above solution has the following technical problems: 1. The solution lacks a graph optimization step: it only constructs a triplet of power equipment, defect type, nameplate, and control area through defect labels and attribute labels, without designing any graph enhancement mechanism. This solution does not consider similar defects of similar equipment, such as common defects of the same type of transformer, nor does it cover the evolution of defects in the time dimension, such as the change of equipment from minor defects to serious faults, resulting in low information density and narrow coverage of the graph.

[0005] 2. The entity association logic of this solution is simple: it only establishes basic relationships between equipment and defects, equipment and nameplates, and control areas, without including key relationships such as environmental factors such as high temperature causing insulation aging and human-computer interaction that affects equipment lifespan. The knowledge representation capability of the graph is insufficient and cannot support complex defect analysis.

[0006] 3. The final output of this solution is only a visualization of power equipment defects. It lacks reasoning logic based on knowledge graphs and does not incorporate early warning functions. Therefore, it cannot transform the constructed graphs into practical value for guiding inspections and providing early warnings. It only completes the first half of graph construction and fails to form a closed loop for practical application, resulting in low practicality. 4. This solution only focuses on inspection images and text extracted from images, completely lacking sensor data. However, early warning of defects in power equipment relies heavily on sensor data. The lack of this modality results in a one-sided data dimension, which cannot reflect the real-time operating conditions of the equipment. At the same time, this solution only constructs the map by tag-level association between the first defect image label and the second attribute text label, without performing feature-level deep fusion. The fused features lack validity verification and are prone to introducing redundant information, affecting the accuracy of subsequent maps. Summary of the Invention

[0007] To address the aforementioned technical shortcomings, the present invention aims to provide a method and system for constructing a multimodal fusion-based knowledge graph of power equipment defects.

[0008] To solve the above technical problems, the present invention adopts the following technical solution: The present invention provides a method for constructing a multimodal fusion power equipment defect knowledge graph, including the following steps: Step 1, multimodal data fusion: collect multimodal data of power equipment, analyze and process the multimodal data of power equipment through a power multidimensional cross-modal fusion calculation model, and output a power multidimensional cross-modal fusion feature vector.

[0009] Step 2: Knowledge Graph Construction: Based on the multi-dimensional cross-modal fusion feature vector of power, an initial knowledge graph is constructed through a knowledge graph generation calculation model. Then, the initial knowledge graph is strengthened through a multi-dimensional equipment similarity law calculation model specific to power equipment structure, parameters and operating conditions to construct an enhanced knowledge graph.

[0010] Step 3, Knowledge Graph Reasoning and Early Warning: Collect current equipment inspection data, and use the reasoning and matching calculation model to perform power equipment similar feature matching analysis between the current equipment inspection data and the enhanced knowledge graph, and output early warning information related to power equipment defects and handling.

[0011] Preferably, an enhanced knowledge graph is constructed, and the specific construction process is as follows: select target-related unique codes for analysis, and record each non-target-related unique code as an other related unique code.

[0012] Run the similarity analysis subroutine to obtain the structural similarity, parameter similarity, and operating condition similarity of each other related unique code. Obtain the pre-set structural similarity weight factor, parameter similarity weight factor, and operating condition similarity weight factor in the power equipment field from the database. Multiply the structural similarity, parameter similarity, and operating condition similarity of each other related unique code by the corresponding weight factor, and add them together to obtain the similarity rate of each other related unique code. In this way, the power equipment-specific similarity rate between each related unique code is obtained.

[0013] Based on the similarity rate between each relevant unique code, and combined with the unique coding rules of power equipment code and timestamp code, the initial knowledge graph is supplemented with the same equipment and the same sequence, resulting in an enhanced knowledge graph.

[0014] On the other hand, the present invention provides a multimodal fusion power equipment defect knowledge graph construction system, including the following modules: a multimodal data fusion module, used to collect multimodal data of power equipment, analyze and process the multimodal data of power equipment through a power multidimensional cross-modal fusion calculation model, and output a power multidimensional cross-modal fusion feature vector.

[0015] The knowledge graph construction module is used to build an initial knowledge graph based on the multi-dimensional cross-modal fusion feature vector of power, and then to strengthen the initial knowledge graph by using a multi-dimensional equipment similarity law calculation model specific to power equipment structure, parameters and operating conditions, thus constructing an enhanced knowledge graph.

[0016] The graph reasoning and early warning module is used to collect current equipment inspection data, and through the reasoning and matching calculation model, it performs matching analysis of the current equipment inspection data with the enhanced knowledge graph for similar characteristics of power equipment, and outputs early warning information related to power equipment defects and handling.

[0017] The beneficial effects of this invention are as follows: 1. First, by using multi-dimensional feature vectors, the cross-modal correlation strength index is calculated, relevant unique codes are screened out, effective dimensions are extracted, and a multi-dimensional cross-modal fusion feature vector for power is constructed. Second, entities and attributes are extracted from the fusion feature vector to construct an initial knowledge graph. Based on the similarity law of motor equipment, the initial graph is supplemented to form an enhanced knowledge graph. Finally, based on the current equipment inspection data, relevant information is prompted to the staff, and defect warnings are given. This method improves the feature representation capability through deep fusion of multi-modal data, supplements and optimizes the integrity of the knowledge graph by relying on the similarity law, and finally improves the defect management effect of power equipment through graph reasoning and warning.

[0018] 2. This invention captures equipment appearance defects through graphic data, records static attributes such as equipment nameplates and maintenance records through text data, and fills the gap in the comparison case where the real-time operating status of the equipment could not be perceived through sensor data. The combination of three modal data not only covers the static identity information and explicit appearance features of the equipment, but also captures the precursors of hidden defects in the operation of the equipment. This provides a more complete data source support for subsequent defect analysis and early warning, which is closer to the actual operating scenario of the equipment, and improves the ability to identify explicit appearance defects and hidden faults.

[0019] 3. This invention analyzes the cross-modal correlation strength index to screen out effective codes that are highly correlated with equipment defects, thus eliminating irrelevant or low-correlation noise data from the source. Secondly, it introduces principal component analysis to process multi-dimensional feature vectors, compressing redundant information and retaining the most critical features for defect identification. The two-layer fusion mechanism improves the quality of the final generated data, reduces redundancy, lays a precise feature foundation for subsequent knowledge graph construction, and alleviates the problem of noise data interfering with the accuracy of the model.

[0020] 4. The knowledge graph of this invention not only records defect results but also presents the complete logical associations related to defects. Furthermore, through similarity law analysis of electrical equipment, defect knowledge of similar equipment under the same operating conditions is added to the power target equipment graph. Simultaneously, by utilizing the unique coding design of power equipment codes and timestamps, the state changes of the same equipment at different times are associated, enabling the tracking of defect temporal evolution. This design upgrades the knowledge graph from a static database of comparative cases to a dynamic knowledge network, covering richer knowledge dimensions such as experience reuse for similar equipment and defect temporal tracing. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the implementation steps of the method of the present invention.

[0023] Figure 2 This is a schematic diagram of the system structure connection of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] according to Figure 1 As shown, the present invention provides a method for constructing a multimodal fusion power equipment defect knowledge graph, including the following steps: Step 1, multimodal data fusion: collect multimodal data of power equipment, analyze and process the multimodal data of power equipment through a power multidimensional cross-modal fusion calculation model, and output a power multidimensional cross-modal fusion feature vector.

[0026] In one specific embodiment, the acquisition process of the multimodal data of the power equipment is as follows: the power equipment data includes various uniquely encoded feature vectors of various graphic dimensions, various text dimensions, and various sensor dimensions.

[0027] Images are captured from each device at each timestamp using cameras, and each timestamp is recorded as a unique code. Images with these unique codes are then obtained. Image recognition technology is used to acquire various graphic feature vectors for each unique code. These graphic feature vectors include, but are not limited to, those from... Textual dimension data comes from textual records throughout the device's entire lifecycle, while feature vectors focus on the structured representation of semantic information, such as: identity attribute features of basic device information, defect type of defect description features, and maintenance duration of maintenance record features.

[0028] By collecting data from the sensors of each device, uniquely encoded feature vectors for various sensing dimensions are obtained. These feature vectors include, but are not limited to, temperature, vibration amplitude, and stress.

[0029] In one specific embodiment, the power multidimensional cross-modal fusion computing model analyzes and processes the following: power equipment data includes uniquely coded graphical feature vectors of power equipment appearance defects, textual feature vectors of power equipment nameplates and maintenance records, and sensor feature vectors of power equipment temperature, vibration, and stress. The power equipment data is then input into the power multidimensional cross-modal fusion computing model.

[0030] First, feature aggregation processing is performed on the power equipment data to obtain image feature vectors, text feature vectors, and sensor feature vectors with unique codes.

[0031] The input is executed by a feature vector similarity calculation subroutine based on the cosine similarity calculation formula. The output is the image-text similarity, image-sensor similarity, and text-sensor similarity of each unique code. The cross-modal association strength index of each unique code is calculated by weighting. If the cross-modal association strength index of a certain unique code is greater than the preset standard cross-modal association strength index in the field of power equipment, the unique code is recorded as the relevant unique code, and thus the relevant unique codes are obtained.

[0032] It should be noted that the image-text similarity of each unique identifier is obtained by substituting the angle between the image feature vector and the text feature vector of each unique identifier into the cosine similarity calculation formula. The image-sensor similarity is obtained by substituting the image feature vector and the sensor feature vector into the Manhattan distance similarity calculation formula. The semantic matching degree of the sensor features of the text feature vector and the sensor feature vector of each unique identifier is calculated based on the power BERT model to obtain the text-sensor similarity of each unique identifier.

[0033] The feature vector similarity calculation subroutine is set using the cosine similarity calculation formula.

[0034] It should be noted that the image-text similarity weight factor, image-sensing similarity weight factor, and text-sensing similarity weight factor for each unique code were obtained by staff through experiments, and the specific values ​​were set by staff.

[0035] The standard cross-modal association strength index is a threshold set by the staff. The setting always revolves around the core objective of distinguishing between effective and ineffective associations between multimodal data. The specific value is set by the staff.

[0036] In one specific embodiment, the power multidimensional cross-modal fusion calculation model further includes a feature fusion processing subroutine, the specific processing process of which is as follows: based on the normalized quality score of image clarity, text keyword coverage, and sensor signal-to-noise ratio, image weight factors, text weight factors, and sensor weight factors are set for each unique code. The feature vectors of each type of graphic dimension of each relevant unique code are multiplied by the corresponding image weight factors to obtain the weighted feature vectors of each type of graphic dimension of each relevant unique code. The feature vectors are then multiplied by the corresponding weight factors to obtain the weighted feature vectors of each type of text dimension and each type of sensor dimension of each relevant unique code. Finally, the weighted feature vectors of each type of dimension of each relevant unique code are summarized to obtain the weighted feature vectors of each type of dimension of each relevant unique code.

[0037] It should be noted that the sharpness calculation is based on edge detection: the number of effective edge pixels and the total number of pixels are collected through image recognition technology, and the sharpness is obtained by dividing the number of effective edge pixels by the total number of pixels. The text keyword coverage is calculated as follows: the structured text is directly exported from the power equipment management system, the text data of the power target equipment is filtered by the unique identification code, the keyword library is automatically matched, and the keyword fuzzy matching is performed using the pandas library of Python to obtain the number of occurrences of the core keywords and the total number of core keywords. The text keyword coverage is obtained by dividing the number of occurrences of the core keywords by the total number of core keywords. The signal-to-noise ratio of the sensor signal is calculated by using signal processing tools to calculate the power of the effective signal and the power of the noise signal. The signal-to-noise ratio is obtained by dividing the power of the effective signal by the power of the noise signal.

[0038] Python's pandas library is a core library for data processing and analysis.

[0039] The image clarity, text keyword coverage, and sensor signal-to-noise ratio of each relevant unique identifier are collected. After normalization, the image quality score, text quality score, and sensor quality score of each relevant unique identifier are obtained. The image quality score, text quality score, and sensor quality score of each relevant unique identifier are added together to obtain the corresponding total quality score. The image quality score, text quality score, and sensor quality score of each relevant unique identifier are divided by the corresponding score to obtain the image weight factor, text weight factor, and sensor weight factor of each unique code.

[0040] Run the principal component analysis algorithm to obtain the feature values ​​of each dimension. Arrange the feature values ​​of each dimension in descending order to obtain the feature value sequence of each dimension. Select a preset number of each dimension from the feature value sequence of each dimension and record them as each effective dimension. Select the weighted feature vector of each effective dimension from the weighted feature vector of each dimension with each relevant unique code, and summarize them to obtain the fusion feature vector of each dimension with each relevant unique code.

[0041] The cosine similarity of each type of fused feature vector with the original feature vector is verified. If the similarity of the fused image, text, and sensor is greater than the preset standard fusion similarity in the power equipment field, it is determined to be a valid unique code. The various types of fused feature vectors of each valid unique code are obtained. The various types of fused feature vectors of each valid unique code are summarized to obtain the power multidimensional cross-modal fused feature vector.

[0042] It should be noted that the verification process for the various fusion feature vectors of each relevant unique code is as follows: the various fusion feature vectors of each relevant unique code are compared with the corresponding various graphic dimension feature vectors, various text dimension feature vectors, and various sensor dimension feature vectors using cosine similarity calculation to obtain the fusion image similarity, fusion text similarity, and fusion sensor similarity of each relevant unique code. If the fusion image similarity, fusion text similarity, and fusion sensor similarity of a certain relevant unique code are all greater than the standard fusion similarity, the relevant unique code is recorded as a valid unique code, and thus each valid unique code is obtained.

[0043] It should be noted that the standard fusion similarity is a threshold set by the staff. The setting always revolves around the core goal of accurately distinguishing between effective and invalid associations between multimodal data. The specific value is set by the staff.

[0044] This leads to the construction of a multi-dimensional cross-modal fusion model for power systems: inputting various uniquely encoded feature vectors for different graphic dimensions, text dimensions, and sensor dimensions, and outputting a multi-dimensional cross-modal fusion feature vector for power systems.

[0045] Step 2: Knowledge Graph Construction: Based on the multi-dimensional cross-modal fusion feature vector of power, an initial knowledge graph is constructed through a knowledge graph generation calculation model. Then, the initial knowledge graph is strengthened through a multi-dimensional equipment similarity law calculation model specific to power equipment structure, parameters and operating conditions to construct an enhanced knowledge graph.

[0046] In one specific embodiment, the knowledge graph generation computing model constructs an initial knowledge graph. The specific construction process is as follows: extracting each relevant uniquely encoded power equipment entity, equipment component entity, state defect entity, environmental factor entity, human-computer interaction entity, and corresponding power equipment defect handling association information from the power multi-dimensional cross-modal fusion feature vector. All of these entities are dedicated to the operation of power equipment.

[0047] Run the power equipment feature association type relationship determination algorithm, input the power equipment entity, equipment component entity, status defect entity, environmental factor entity, human-computer interaction entity and the corresponding power equipment defect handling association information of each relevant unique code, and obtain the power equipment exclusive relationship attributes of each relevant unique code, including inclusion, existence of defects, environmental impact and handling association.

[0048] By summarizing the relevant uniquely coded power equipment entities, equipment component entities, state defect entities, environmental factor entities, human-computer interaction entities, and various relational attributes, an initial knowledge graph is obtained.

[0049] It should be noted that the power multidimensional cross-modal fusion feature vector is divided into a set of effective feature vectors in the graphic dimension, a set of effective feature vectors in the sensing dimension, and a set of effective feature vectors in the text dimension. From the set of effective feature vectors in the graphic dimension of the fusion features, component contour features and defect area features are extracted to obtain the equipment component entity and status defect entity corresponding to each effective unique identifier. From the set of effective feature vectors in the sensing dimension of the fusion features, environmental parameter features are extracted to obtain the environmental factor entity corresponding to each effective unique identifier. From the set of effective feature vectors in the text dimension of the fusion features, operation subject features and operation record features are extracted to obtain the human-computer interaction entity corresponding to each effective unique identifier. Each effective unique identifier is directly used as a type of power equipment entity to obtain various entities.

[0050] Relationship attributes include, but are not limited to, inclusion, existence of defects, similarity, environmental impact, and handling association. Based on the inclusion relationship between component coordinates and equipment outlines in graphical features, the inclusion relationship between a certain type of power equipment entity and a certain type of equipment component entity is obtained; based on the cross-modal association strength index, the existence of defects relationship between a certain type of power equipment entity and a certain type of state defect entity is obtained; based on the cosine similarity of the fused feature vectors, the similarity association relationship between different types of power equipment entities is obtained; based on the correlation coefficient between environmental features and equipment state features, the environmental impact relationship between a certain type of environmental factor entity and a certain type of power equipment entity is obtained; based on text matching degree and historical handling success rate, the handling association relationship between a certain type of human-computer interaction entity and a certain type of state defect entity is obtained; thus, the relationship attributes between various entities are obtained.

[0051] The technique of setting up a knowledge graph using feature vectors of power equipment in various dimensions is an existing technology that can be found on the Internet, so it will not be elaborated further.

[0052] Knowledge graph generation computational model: Input each relevant uniquely coded power equipment entity, equipment component entity, status defect entity, environmental factor entity, human-computer interaction entity and corresponding power equipment defect handling association information, and output an initial knowledge graph by running a power equipment feature association type relationship determination algorithm.

[0053] In one specific embodiment, the construction process of the enhanced knowledge graph is as follows: select target-related unique codes for analysis, and record each non-target-related unique code as an other related unique code.

[0054] Run the similarity analysis subroutine to obtain the structural similarity, parameter similarity, and operating condition similarity of each other related unique code. Obtain the pre-set structural similarity weight factor, parameter similarity weight factor, and operating condition similarity weight factor in the power equipment field from the database. Multiply the structural similarity, parameter similarity, and operating condition similarity of each other related unique code by the corresponding weight factor, and add them together to obtain the similarity rate of each other related unique code. In this way, the power equipment-specific similarity rate between each related unique code is obtained.

[0055] It should be noted that the structural similarity weight factor, parameter similarity weight factor, and operating condition similarity weight factor were obtained by the staff through experiments, and the specific values ​​were set by the staff.

[0056] Based on the similarity rate between each relevant unique code, and combined with the unique coding rules of power equipment code and timestamp code, the initial knowledge graph is supplemented with the same equipment and the same sequence, resulting in an enhanced knowledge graph.

[0057] In one specific embodiment, the similarity analysis subroutine performs the following calculation: setting the structural score of each component to obtain the structural score of each component of each relevant unique code; summing the structural scores of each component that is the same as the target relevant unique code in each other relevant unique code to obtain the total structural score of each other relevant unique code; and dividing the total structural score of each other relevant unique code by the standard total structural score to obtain the structural similarity of each other relevant unique code.

[0058] It should be noted that the structural scores of each component are set by the staff according to their importance, and the specific values ​​are set by the staff.

[0059] The feature values ​​of various motor-specific parameter feature data of various power equipment are obtained from the various fusion feature vectors of various relevant unique codes. The feature values ​​of various motor parameter feature data of the target relevant unique code are calculated with the feature values ​​of various motor parameter feature data of other relevant unique codes to obtain the Euclidean distance of other relevant unique codes. The standard Euclidean distance is set to 1 and then subtracted from the Euclidean distance of other relevant unique codes to obtain the parameter similarity of other relevant unique codes.

[0060] It should be noted that the standard Euclidean distance is set to the unit "1".

[0061] The fusion feature vectors of various power equipment operation-specific sensors are obtained from the fusion feature vectors of various related unique codes. The Pearson correlation coefficient is calculated between the fusion feature vectors of various sensors with the target related unique codes and the fusion feature vectors of various other related unique codes to obtain the operating condition similarity.

[0062] The device similarity law calculation model obtains the structural similarity of each other related unique code by running a similarity analysis subroutine.

[0063] In one specific embodiment, the initial knowledge graph is supplemented with same-device and same-time sequences using a unique encoding rule that combines power equipment codes and timestamp codes. The specific supplementation process is as follows: The unique code is a combination of the power equipment code and the timestamp code. In this way, the relevant unique codes with the same power equipment code are recorded as the relevant unique codes of the same equipment, and the relevant unique codes with the same timestamp code are recorded as the relevant unique codes of the same timestamp.

[0064] Obtain the target-related unique code and the corresponding timestamp of each related unique code, and then obtain the similarity rate of the target-related unique code and the corresponding timestamp of each related unique code. Record each related unique code with a similarity rate greater than the preset equipment similarity rate in the power equipment field as the effective unique code of each equipment. Summarize the state defect entity, human-computer interaction entity and various relation attributes of each effective unique code of the target-related unique code and record them as the knowledge graph supplementary group of each effective unique code of the target-related unique code. In this way, the knowledge graph supplementary group of each time-series effective unique code of the target-related unique code is obtained. Finally, the knowledge graph supplementary group of each target-related unique code is obtained.

[0065] The initial knowledge graph supplementation process is as follows: each knowledge graph supplementation group with a unique code related to the target is filled into the initial knowledge graph group with a unique code related to the target, and each unique code is filled in sequentially to obtain the enhanced knowledge graph.

[0066] Step 3, Knowledge Graph Reasoning and Early Warning: Collect current equipment inspection data, and use the reasoning and matching calculation model to perform power equipment similar feature matching analysis between the current equipment inspection data and the enhanced knowledge graph, and output early warning information related to power equipment defects and handling.

[0067] In one specific embodiment, the acquisition of current equipment inspection data is carried out as follows: the current equipment inspection data includes the image feature vector, text feature vector, and sensor feature vector of the current power target equipment.

[0068] The current process for collecting equipment inspection data is the same as the process for collecting multimodal data of power equipment, so it will not be described in detail here.

[0069] In one specific embodiment, the analysis of matching the current equipment inspection data with the enhanced knowledge graph for similar features of power equipment is performed as follows: based on the power equipment code corresponding to the power target equipment, obtain each similar unique code of the power target equipment and the equipment type, and obtain the image feature vector, text feature vector and sensor feature vector of each similar unique code of the power target equipment and the equipment type from the enhanced knowledge graph.

[0070] The current equipment inspection data is input into the inference matching calculation model along with the image feature vectors, text feature vectors, and sensor feature vectors of the same type of power target equipment that are similar to each other.

[0071] Reasoning and matching calculation model: Run the cosine similarity calculation algorithm to obtain the historical similarity of each similar unique code of the power target equipment and the equipment type. Record each similar unique code of the power target equipment and the equipment type whose historical similarity is greater than the preset historical similarity as each valid unique code. In this way, the valid unique codes of the current power target equipment are obtained.

[0072] In one specific embodiment, the output of power equipment defects and related early warning information includes the following process: obtaining the status defect entity group, human-machine interaction entity, and corresponding power equipment defect handling association information group of each valid unique code of the current power target equipment from the enhanced knowledge graph, and prompting the staff with the status defect entity, human-machine interaction entity, and corresponding power equipment defect handling association information of each valid unique code.

[0073] according to Figure 2 As shown, the present invention provides a multimodal fusion power equipment defect knowledge graph construction system, including the following modules: multimodal data fusion module, knowledge graph construction module, graph reasoning and early warning module, and database.

[0074] The knowledge graph construction module is connected to the multimodal data fusion module and the graph reasoning early warning module, respectively. The multimodal data fusion module, the knowledge graph construction module, and the graph reasoning early warning module are all connected to the database.

[0075] The multimodal data fusion module is used to collect multimodal data from power equipment, analyze and process the multimodal data of power equipment through a multidimensional cross-modal fusion calculation model, and output a multidimensional cross-modal fusion feature vector of power equipment.

[0076] The knowledge graph construction module is used to build an initial knowledge graph based on the multi-dimensional cross-modal fusion feature vector of power, and then to strengthen the initial knowledge graph by using a multi-dimensional equipment similarity law calculation model specific to power equipment structure, parameters and operating conditions, thus constructing an enhanced knowledge graph.

[0077] The graph reasoning and early warning module is used to collect current equipment inspection data, and through the reasoning and matching calculation model, it performs matching analysis of the current equipment inspection data with the enhanced knowledge graph for similar characteristics of power equipment, and outputs early warning information related to power equipment defects and handling.

[0078] The database is used to store structural similarity weight factors, parameter similarity weight factors, and operating condition similarity weight factors.

[0079] The Manhattan distance similarity calculation formula, the power BERT model, the cosine similarity calculation, the Euclidean distance calculation, and the Pearson correlation coefficient calculation described in this invention are all existing technologies and can be found on the Internet, so they will not be elaborated further.

[0080] The examples described in this invention are not limited to the specific embodiments listed above. The examples are merely illustrative to facilitate understanding of the invention and do not constitute a limitation on the scope of protection of this invention. Any modifications, equivalent substitutions, etc., made within the spirit and principles of this invention should be included within the scope of protection.

[0081] The above description is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined in this specification, they should all fall within the protection scope of the present invention.

Claims

1. A method for constructing a multimodal fusion knowledge graph of power equipment defects, characterized in that, Includes the following steps: Step 1: Multimodal data fusion: Collect multimodal data of power equipment, analyze and process the multimodal data of power equipment through a multidimensional cross-modal fusion calculation model of power, and output multidimensional cross-modal fusion feature vector of power. Step 2: Knowledge Graph Construction: Based on the multi-dimensional cross-modal fusion feature vector of power, an initial knowledge graph is constructed through a knowledge graph generation calculation model. Then, the initial knowledge graph is strengthened through a multi-dimensional equipment similarity law calculation model specific to power equipment structure, parameters and operating conditions to construct an enhanced knowledge graph. Step 3, Knowledge Graph Reasoning and Early Warning: Collect current equipment inspection data, and use the reasoning and matching calculation model to perform power equipment similar feature matching analysis between the current equipment inspection data and the enhanced knowledge graph, and output early warning information related to power equipment defects and handling.

2. The method for constructing a multimodal fusion-based knowledge graph of power equipment defects according to claim 1, characterized in that, The analysis and processing process of the aforementioned multidimensional cross-modal fusion calculation model for power is as follows: The power equipment data includes graphic feature vectors of appearance defects of each uniquely coded power equipment, text feature vectors of nameplates and maintenance records of power equipment, and sensor feature vectors of temperature, vibration and stress of power equipment. The power equipment data is input into the power multidimensional cross-modal fusion computing model. First, feature aggregation processing is performed on the power equipment data to obtain image feature vectors, text feature vectors, and sensor feature vectors with unique codes. The input is a feature vector similarity calculation subroutine set based on the cosine similarity calculation formula. The output is the image-text similarity, image-sensor similarity, and text-sensor similarity of each unique code. The cross-modal association strength index of each unique code is calculated by weighting. If the cross-modal association strength index of a certain unique code is greater than the preset standard cross-modal association strength index in the field of power equipment, the unique code is recorded as the relevant unique code. In this way, each relevant unique code is obtained. The feature vector similarity calculation subroutine is set using the cosine similarity calculation formula.

3. The method for constructing a multimodal fusion-based knowledge graph of power equipment defects according to claim 2, characterized in that, The power multidimensional cross-modal fusion calculation model also includes a feature fusion processing subroutine, the specific processing procedure of which is as follows: Based on the normalized quality score of image sharpness, text keyword coverage, and sensor signal-to-noise ratio, image weight factors, text weight factors, and sensor weight factors are set for each unique code. The feature vectors of each relevant unique code in various graphic dimensions are multiplied by the corresponding image weight factors to obtain the weighted feature vectors of each relevant unique code in various graphic dimensions. The feature vectors are then multiplied by the corresponding weight factors to obtain the weighted feature vectors of each relevant unique code in various text dimensions and various sensor dimensions. Finally, the weighted feature vectors of each relevant unique code in various dimensions are obtained. Run the principal component analysis algorithm to obtain the feature values ​​of various dimensions. Arrange the feature values ​​of various dimensions in descending order to obtain the feature value sequence of various dimensions. Select the first preset number of each dimension in the feature value sequence of each dimension and record them as each effective dimension. Select the weighted feature vector of each effective dimension from the weighted feature vector of each dimension with each relevant unique code, and summarize to obtain the fusion feature vector of each dimension with each relevant unique code. The cosine similarity of each type of fusion feature vector of each relevant unique code is verified with the original feature vector. If the similarity of the fusion image, text, and sensor is greater than the preset standard fusion similarity in the field of power equipment, it is determined to be a valid unique code. The various types of fusion feature vectors of each valid unique code are obtained. The various types of fusion feature vectors of each valid unique code are summarized to obtain the power multidimensional cross-modal fusion feature vector. This leads to the construction of a multi-dimensional cross-modal fusion model for power systems: inputting various uniquely encoded feature vectors for different graphic dimensions, text dimensions, and sensor dimensions, and outputting a multi-dimensional cross-modal fusion feature vector for power systems.

4. The method for constructing a multimodal fusion-based knowledge graph of power equipment defects according to claim 1, characterized in that, The knowledge graph generation computational model constructs the initial knowledge graph, and the specific construction process is as follows: Extract the uniquely encoded power equipment entities, equipment component entities, state defect entities, environmental factor entities, human-computer interaction entities, and corresponding power equipment defect handling association information from the multi-dimensional cross-modal fusion feature vector of power. All of these entities are exclusive to the operation of power equipment. Run the power equipment feature association type relationship determination algorithm, input the power equipment entity, equipment component entity, status defect entity, environmental factor entity, human-computer interaction entity and corresponding power equipment defect handling association information of each relevant unique code, and obtain the power equipment exclusive relationship attributes of each relevant unique code, including inclusion, existence of defects, environmental impact and handling association. The initial knowledge graph is obtained by summarizing the relevant uniquely coded power equipment entities, equipment component entities, status defect entities, environmental factor entities, human-computer interaction entities, and various relational attributes. Knowledge graph generation computational model: Input each relevant uniquely coded power equipment entity, equipment component entity, status defect entity, environmental factor entity, human-computer interaction entity and corresponding power equipment defect handling association information, and output an initial knowledge graph by running a power equipment feature association type relationship determination algorithm.

5. The method for constructing a multimodal fusion-based knowledge graph of power equipment defects according to claim 1, characterized in that, The specific construction process for building the enhanced knowledge graph is as follows: Select the target-related unique code for analysis, and record each non-target-related unique code as the other related unique code; Run the similarity analysis subroutine to obtain the structural similarity, parameter similarity, and operating condition similarity of each other related unique code. Obtain the pre-set structural similarity weight factor, parameter similarity weight factor, and operating condition similarity weight factor in the power equipment field from the database. Multiply the structural similarity, parameter similarity, and operating condition similarity of each other related unique code by the corresponding weight factor, and add them together to obtain the similarity rate of each other related unique code. In this way, the power equipment-specific similarity rate between each related unique code is obtained. Based on the similarity rate between each relevant unique code, and combined with the unique coding rules of power equipment code and timestamp code, the initial knowledge graph is supplemented with the same equipment and the same sequence, resulting in an enhanced knowledge graph.

6. The method for constructing a multimodal fusion-based knowledge graph of power equipment defects according to claim 5, characterized in that, The similarity analysis subroutine performs the following calculations: Set the structural score for each component to obtain the structural score for each component of each relevant unique code. Sum the structural scores of each component that is the same as the target relevant unique code in each other relevant unique code to obtain the total structural score of each other relevant unique code. Divide the total structural score of each other relevant unique code by the total standard structural score to obtain the structural similarity of each other relevant unique code. The feature values ​​of various motor-specific parameter feature data of various power equipment are obtained from the various fusion feature vectors of various relevant unique codes. The feature values ​​of various motor parameter feature data of the target relevant unique code are calculated with the feature values ​​of various motor parameter feature data of other relevant unique codes to obtain the Euclidean distance of other relevant unique codes. The standard Euclidean distance is set to 1 and then subtracted from the Euclidean distance of other relevant unique codes to obtain the parameter similarity of other relevant unique codes. The fusion feature vectors of various power equipment operation-specific sensors are obtained from the fusion feature vectors of various related unique codes. The Pearson correlation coefficient is calculated between the fusion feature vectors of various sensors with the target related unique codes and the fusion feature vectors of various other related unique codes to obtain the operating condition similarity.

7. The method for constructing a multimodal fusion-based knowledge graph of power equipment defects according to claim 5, characterized in that, The initial knowledge graph is supplemented with information on the same device and the same sequence of events by combining the unique encoding rules of power equipment codes and timestamp codes. The specific supplementation process is as follows: The unique code is a combination of the power equipment code and the timestamp code. In this way, the relevant unique codes with the same power equipment code are recorded as the relevant unique codes of the same equipment, and the relevant unique codes with the same timestamp code are recorded as the relevant unique codes of the same timestamp. Obtain the target-related unique code and the time stamp of each related unique code, and then obtain the similarity rate of the target-related unique code and the time stamp of each related unique code. Record each related unique code with a similarity rate greater than the preset equipment similarity rate in the power equipment field as the effective unique code of each equipment. Summarize the state defect entity, human-computer interaction entity and various relation attributes of each effective unique code of the target-related unique code and record them as the knowledge graph supplementary group of each effective unique code of the target-related unique code. In this way, the knowledge graph supplementary group of each time-series effective unique code of the target-related unique code is obtained. Summarize to obtain the knowledge graph supplementary group of each knowledge graph supplementary code of the target-related unique code. The initial knowledge graph supplementation process is as follows: each knowledge graph supplementation group with a unique code related to the target is filled into the initial knowledge graph group with a unique code related to the target, and each unique code is filled in sequentially to obtain the enhanced knowledge graph.

8. The method for constructing a multimodal fusion knowledge graph of power equipment defects according to claim 7, characterized in that, The specific analysis process for matching the current equipment inspection data with the enhanced knowledge graph based on similar characteristics of power equipment is as follows: Current equipment inspection data includes image feature vectors, text feature vectors, and sensor feature vectors of the current power target equipment; Based on the power equipment code corresponding to the power target equipment, obtain each similar unique code of the power target equipment and equipment type, and obtain the image feature vector, text feature vector and sensor feature vector of each similar unique code of the power target equipment and equipment type from the enhanced knowledge graph; Input the image feature vectors, text feature vectors, and sensor feature vectors of the current equipment inspection data and the similar unique codes of the power target equipment of the same type into the inference matching calculation model; Reasoning and matching calculation model: Run the cosine similarity calculation algorithm to obtain the historical similarity of each similar unique code of the power target equipment and the equipment type. Record each similar unique code of the power target equipment and the equipment type whose historical similarity is greater than the preset historical similarity as each valid unique code. In this way, the valid unique codes of the current power target equipment are obtained.

9. The method for constructing a multimodal fusion knowledge graph of power equipment defects according to claim 8, characterized in that, The early warning information related to defects and handling of the output power equipment is provided in the following specific early warning process: The system retrieves the status defect entity groups, human-machine interaction entities, and corresponding power equipment defect handling association information groups of each valid unique code of the current power target equipment from the enhanced knowledge graph, and prompts the staff with the status defect entities, human-machine interaction entities, and corresponding power equipment defect handling association information of each valid unique code.

10. A knowledge graph construction system applying the multimodal fusion method for constructing a knowledge graph of power equipment defects according to any one of claims 1-9, characterized in that, Includes the following modules: The multimodal data fusion module is used to collect multimodal data from power equipment, analyze and process the multimodal data of power equipment through a multidimensional cross-modal fusion calculation model, and output a multidimensional cross-modal fusion feature vector of power. The knowledge graph construction module is used to build an initial knowledge graph based on the multi-dimensional cross-modal fusion feature vector of power, and then to strengthen the initial knowledge graph through a multi-dimensional equipment similarity law calculation model of power equipment's specific structure, parameters and operating conditions, thus building an enhanced knowledge graph. The graph reasoning and early warning module is used to collect current equipment inspection data, and through the reasoning and matching calculation model, it performs matching analysis of the current equipment inspection data with the enhanced knowledge graph for similar characteristics of power equipment, and outputs early warning information related to power equipment defects and handling.

Images