A Complex Equipment Fault Diagnosis Method and System Based on a Multimodal Knowledge Graph

Abstract

The present invention relates to the technical field of equipment fault diagnosis, and in particular to a complex equipment fault diagnosis method and system based on a multimodal knowledge graph. The method comprises: extracting multimodal features from acquired equipment operating data; constructing a knowledge graph based on the extracted multimodal features; enhancing the multimodal features of the knowledge graph through cross-modal comparative learning; defining small sample tasks and initializing the features of the enhanced knowledge graph. By combining the causal relationships inherent in the knowledge graph, the system can dynamically perceive and predict the health status of the equipment, providing early warning of potential fault risks.

Description

Technical Field

[0001] The present invention relates to the technical field of equipment fault diagnosis, and in particular to a complex equipment fault diagnosis method and system based on a multimodal knowledge graph. Background Art

[0002] To ensure the safe and stable operation of power systems, early detection and accurate warning of potential faults such as partial discharge in transformer windings, abnormal generator vibration, worn circuit breaker contacts, or loose bolts on transmission line conductors have become a hot topic and a key challenge. Traditional scheduled inspections and post-event maintenance models, with their fixed cycles and delayed response times, have made it difficult to balance high reliability with low O&M costs. This has led to the emergence of predictive maintenance, which shifts from passive maintenance to proactive prevention, significantly improving equipment availability and maintenance efficiency.

[0003] Existing predictive fault diagnosis methods can be broadly categorized into three main types: First, threshold- and rule-based diagnosis, which achieves real-time alarms by manually setting upper and lower limits for indicators such as vibration, temperature, and current. While simple and efficient, these methods struggle to adapt to changing operating conditions and new faults. Second, physical model-based diagnosis, which identifies anomalies by comparing residuals or parameter estimates using mathematical or simulation models, offers mechanistic explanations but is susceptible to model bias and parameter inaccuracies. Third, data-driven diagnosis, which utilizes machine learning or deep learning to train on large-scale historical data, can mine complex features but relies on a large number of labeled samples, lacking generalization capabilities for new faults and small sample sizes. Furthermore, multimodal data, such as vibration, images, and text, is often fragmented, failing to fully exploit potential causal chains and cross-modal associations. In practical applications, these methods struggle to dynamically characterize the causal relationships between multiple variables in equipment and are prone to overfitting or failure when samples are extremely scarce or when new faults emerge. Furthermore, single-modal analysis methods struggle to meet the requirements for comprehensive perception of the "environment-operation-fault" landscape of power equipment.

[0004] In addition, the current fault diagnosis methods for complex equipment mainly rely on single data source modeling, feature engineering or deep learning technology based on large-scale sample training, which has the following significant defects: (1) Insufficient data heterogeneity processing, making it difficult to integrate multimodal information such as sensor data, text records, and operation and maintenance knowledge, resulting in one-sided diagnosis results and limited accuracy; (2) Weak knowledge expression and reasoning capabilities, making it difficult for existing methods to effectively model the internal structure, functional relationships, and fault evolution mechanism of equipment, lacking interpretability and reasoning depth; (3) Insufficient ability to identify small samples and early faults, especially in practical applications, a small number of noisy samples often leads to a significant decline in model performance, affecting system stability and reliability. Therefore, it is urgent to develop a small sample complex equipment fault diagnosis system based on multimodal knowledge graphs, fully integrate multi-source heterogeneous data, build an intelligent diagnosis system that combines equipment mechanism knowledge with data-driven, and achieve high-precision, explainable predictive fault diagnosis for complex equipment. Summary of the Invention

[0005] To address the problems of insufficient multi-source heterogeneous data fusion, weak fault mechanism knowledge modeling and reasoning capabilities, and difficulty identifying small-sample faults in the predictive fault diagnosis process for complex equipment, the present invention provides a complex equipment fault diagnosis method and system based on a multimodal knowledge graph. Leveraging multi-source heterogeneous data such as vibration signals, maintenance documentation, and images, this system maps multimodal features into entity-relationship-attribute triples through a unified semantic ontology to construct an equipment health knowledge graph. Furthermore, graph neural network encoding and a small-sample learning strategy are introduced to generate semantically enhanced fault prototypes, guided by the structured semantics of the knowledge graph, enabling efficient identification and location of new fault types.

[0006] In a first aspect, the present invention provides a complex equipment fault diagnosis method based on a multimodal knowledge graph, which adopts the following technical solutions:

[0007] A complex equipment fault diagnosis method based on multimodal knowledge graph, including:

[0008] Obtain equipment operation data;

[0009] Perform multimodal feature extraction on the acquired equipment operation data;

[0010] Construct a knowledge graph based on the extracted multimodal features;

[0011] Multimodal feature enhancement of knowledge graphs through cross-modal contrastive learning;

[0012] Define small sample tasks and initialize features of the enhanced knowledge graph;

[0013] Perform subgraph extraction and Mamba enhancement on the knowledge graph after feature initialization;

[0014] Utilize graph neural networks to perform fault classification based on knowledge graphs.

[0015] Furthermore, the multimodal feature extraction is performed on the acquired equipment operation data, including the original vibration signal data collected by the high-precision sensor, and the vibration signal data collected by the sensor is set as X v , using 1D-CNN to extract local features; using the BERT pre-trained language model to perform entity recognition on operation and maintenance logs and operation records, complete the identification of fault entities, and output the corresponding semantic embedding vector; and using the singular value decomposition SVD method to reduce the dimension of the image matrix, and input the reduced dimension image into the standard CNN network to obtain the image feature representation h i .

[0016] Furthermore, the knowledge graph is constructed based on the extracted multimodal features, including constructing a feature-fault template dictionary for vibration signal features, and using template matching methods and spectrum analysis methods to identify and classify the extracted signal features; for text data features, natural language generation (NLG) is further introduced for the semantic embedding vectors and entity recognition results extracted from operation and maintenance logs and operation information through the BERT model, and sentence-level information is summarized and summarized to extract key semantic content related to diagnosis; for image data features, high-dimensional image features extracted through CNN need to be semantically annotated in combination with computer vision.

[0017] Furthermore, the multimodal feature enhancement of the knowledge graph through cross-modal contrastive learning includes inter-modal contrastive learning, performing positive and negative comparisons on sample pairs of different modalities, and completing cross-modal feature alignment. Among them, in view of the different dimensions of the three modal features, they are projected and mapped into a contrastive learning space of unified dimension. In the inter-modal contrastive learning, the distinguishability is further strengthened within each individual modality, and the attribute similarity of each sample in the same modality is quantified, and positive and negative pairs are generated based on this; context mask filling is performed for text data features; Gaussian blur is added to image features, and the image is convolved with a two-dimensional Gaussian kernel to obtain a blurred enhanced image.

[0018] Furthermore, the multimodal feature enhancement of the knowledge graph through cross-modal contrastive learning also includes constructing positive and negative sample pairs within the modality based on the sample-enhanced views, optimizing the feature distribution structure within the modality, defining a positive sample pair as a pairing between two views of the same sample, and a negative sample pair as a pairing between the sample view and other sample views. To optimize the aggregation and separation effects of intra-modal features, intra-modal contrastive learning uses an improved InfoNCE loss function to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. For modality m, the loss form is expressed as:

[0019]

[0020] in, Representative j The representation vector of each sample after the enhancement operation passes through the projection network, Representative j The representation vector of a sample after the projection network without enhancement operation, τ is the temperature hyperparameter that controls the smoothness of the contrast distribution.

[0021] Furthermore, the small sample task is defined and the features of the enhanced knowledge graph are initialized, including constructing a support set S as a small number of samples of each type of fault in the current task for the model to learn the current task; the query set Q is used to evaluate the learning effect of the model on the task; the knowledge graph is used to perform structured screening of fault categories, wherein the graph structure measurement indicators, eigenvector centrality and path length in graph theory are used to evaluate the fault nodes in the graph, and for each fault category, a third-order neighbor sampling strategy is executed from the knowledge graph, that is, starting from the current node, traversing the graph outward to the third hop, selecting entity nodes closely related to the category, and forming a subgraph structure containing semantic neighbors.

[0022] Furthermore, the subgraph extraction and Mamba enhancement of the knowledge graph after feature initialization include the introduction of structural indicators to dynamically control the number of sampling hops S hop , for each sample corresponding to the knowledge graph entity e i , dynamically adjust the neighborhood sampling hop count S according to the topological importance of the node hop , extract its local neighborhood subgraph G i :

[0023]

[0024] Among them, Betweenness represents the betweenness centrality of the node, Degree represents the degree of the node, parameters α and β are used to weigh the influence of betweenness centrality and degree of the node, and dist represents the number of path hops. V i Represents a collection of nodes, E i Represents a set of edges.

[0025] Furthermore, the subgraph extraction and Mamba enhancement of the knowledge graph after feature initialization also include the introduction of the selective state space model Mamba, the construction of a multi-level, adaptive feature interaction architecture, and the implementation of input-dependent dynamic parameter adjustment. The core formula of MambaBlock is:

[0026]

[0027] in, represents the hidden state vector at time step t, represents the gating function, represents the matrix exponential function, A is the learnable state transition matrix, B is the input projection matrix, I is the identity matrix, Represents the input node features, Represents the high-order neighbor node features of node v.

[0028] Furthermore, the use of graph neural networks to perform fault classification based on knowledge graphs includes compressing the support sample vectors of the same category into a prototype vector in a small sample scenario, and comparing the similarity between the query sample and the prototypes of each category through a distance metric to determine the category to which it belongs. q The distance between the sample and the prototype of each category is mapped into a probability distribution to quantify the probability that the sample belongs to each fault category:

[0029]

[0030] Among them, k is the fault category, h q represents the vector representation of the query test sample, and d represents the cosine similarity measure.

[0031] The second aspect is a complex equipment fault diagnosis system based on a multimodal knowledge graph, including:

[0032] The data acquisition module is configured to acquire equipment operation data;

[0033] The feature module is configured to perform multimodal feature extraction on the acquired equipment operation data;

[0034] The graph module is configured to construct a knowledge graph based on the extracted multimodal features;

[0035] The enhancement module is configured to perform multimodal feature enhancement on the knowledge graph through cross-modal contrastive learning;

[0036] The small sample module is configured to define small sample tasks and perform feature initialization on the enhanced knowledge graph;

[0037] The classification module is configured to perform subgraph extraction and Mamba enhancement on the knowledge graph after feature initialization; and use graph neural network to perform fault classification based on the knowledge graph.

[0038] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, for a complex equipment fault diagnosis method based on a multimodal knowledge graph.

[0039] In a fourth aspect, the present invention provides a terminal device comprising a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded and executed by the processor to form a complex equipment fault diagnosis method based on a multimodal knowledge graph.

[0040] In summary, the present invention has the following beneficial technical effects:

[0041] Compared with the existing technology, the present invention provides a method and system for predictive fault diagnosis of complex equipment based on a multimodal knowledge graph, which has the following beneficial effects: by constructing a multimodal knowledge graph and integrating vibration signals, equipment logs and image features, a unified semantic representation of fault information is achieved, which significantly improves the comprehensiveness and accuracy of the extraction of fault features of complex equipment; by utilizing small sample task construction and prototype network metric learning guided by the knowledge graph, the system can quickly adapt to new fault categories under conditions of very few labeled samples, effectively overcoming the problems of data scarcity and category imbalance; at the same time, combined with the causal relationship contained in the knowledge graph, the system can realize dynamic perception and trend prediction of the health status of the equipment, and provide early warning of potential fault risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a schematic diagram of a complex equipment fault diagnosis method based on a multimodal knowledge graph according to Example 1 of the present invention.

[0043] Figure 2 1 is a comparison chart of ACC and F1 of various models in Example 1 of the present invention.

[0044] Figure 3 It is a radar chart of different models under six major indicators of Example 1 of the present invention.

[0045] Figure 4 This is a schematic diagram showing the early warning of the comparison results of the model robustness of Example 1 of the present invention.

[0046] Figure 5 2 is a schematic diagram of the comparison results of the advance amount distribution of Example 1 of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described in detail below with reference to the accompanying drawings.

[0048] Example 1

[0049] Reference Figure 1In this embodiment, a complex equipment fault diagnosis method based on a multimodal knowledge graph includes:

[0050] Obtain equipment operation data;

[0051] Perform multimodal feature extraction on the acquired equipment operation data;

[0052] Construct a knowledge graph based on the extracted multimodal features;

[0053] Multimodal feature enhancement of knowledge graphs through cross-modal contrastive learning;

[0054] Define small sample tasks and initialize features of the enhanced knowledge graph;

[0055] Perform subgraph extraction and Mamba enhancement on the knowledge graph after feature initialization;

[0056] Utilize graph neural networks to perform fault classification based on knowledge graphs.

[0057] Specifically:

[0058] S1. Multimodal feature extraction module,

[0059] During the operation of complex equipment, various types of operating data are often generated, including structural vibration, thermal imaging images, environmental parameters, operation records, and operation and maintenance logs. These data have different modal characteristics, each reflecting the operating status of the equipment in a certain dimension. Single-modal information often has limitations and cannot fully characterize the health status of the equipment under actual working conditions. The core goal of the multimodal feature extraction module is to make full use of data from multiple perception channels, extract the significant features of each data modality through a deep learning model, and achieve unified feature expression, providing a solid foundation for subsequent knowledge graph construction and reasoning tasks. It specifically includes the following three parts:

[0060] 1) Extraction of vibration signal features: During the operation of equipment, vibration signal is one of the most commonly used and most direct physical quantities that reflects the state of the machine. The original vibration signal data collected by the high-precision sensor is set as X v , use 1D-CNN to extract local features:

[0061]

[0062] in, k is the convolution kernel size, d is the hidden layer dimension, W v for Weight parameters. Transformer global dependency modeling is used to capture long-range periodicity:

[0063]

[0064] 2) Text feature extraction: Operation and maintenance logs and operation records are text information manually recorded by front-line maintenance personnel during equipment maintenance. They contain a large number of semantic descriptions related to the equipment's operating status, such as fault phenomena, maintenance measures, and affected components. This type of text data is usually unstructured, highly redundant, and has non-standard semantic expressions. Let the collected operation and maintenance log data be X t , the BERT pre-trained language model is used to perform entity recognition on operation and maintenance logs and operation records, complete the identification of fault entities (such as equipment components, fault phenomena, etc.), and output the corresponding semantic embedding vector.

[0065]

[0066] Among them, CLS is a special marker used to mark the entire sequence.

[0067] Infrared thermal imaging image feature extraction, infrared thermal imaging as a non-contact monitoring method, can obtain real-time equipment surface temperature distribution images, revealing fault precursors such as temperature rise and hot spot concentration. Assume that the image data collected by the infrared thermal imaging sensor is X i Since thermal imaging images are essentially grayscale images or heat distribution maps, directly inputting them into convolutional neural networks is easily affected by noise and redundant information. Therefore, the singular value decomposition (SVD) method is first used to reduce the dimensionality of the image matrix:

[0068]

[0069] in, Indicates that the image is approximately represented by the matrix after SVD dimension reduction, U is the left singular vector matrix, Σ is the singular value matrix, and V is the right singular vector matrix. The reduced dimension image is input into the standard CNN network to obtain the image feature representation h i :

[0070]

[0071] in, Indicates that the image is approximately represented by the matrix after SVD dimension reduction.

[0072] S2. Knowledge graph building module,

[0073] The deep features of multi-source heterogeneous data such as vibration signals, texts, and images obtained by the first module are semantically understood and summarized, mapped into entities and relationships, and a knowledge graph that can express equipment structure, operating status, and fault factors is constructed.

[0074] 1) Vibration signal features are modeled jointly using 1D-CNN and Transformer to generate high-dimensional temporal feature representations. To construct the data structure centered around "entity-relationship-entity" triples within the knowledge graph, these feature vectors need to be further semantically interpreted.

[0075] Specifically, a feature-fault template dictionary is constructed, and the extracted signal features are identified and classified using template matching and spectrum analysis methods. For example, spectrum features are used to determine whether there are abnormal amplitudes, harmonic peaks, or changes in energy concentration at specific frequencies. These features are matched with typical fault modes to generate standardized description texts, such as: "The main shaft is unbalanced", "Damage to the bearing inner ring is detected", "Possible structural looseness". These fault descriptions will serve as the primary semantic entity "fault mode" node in the knowledge graph, and will be further associated with the corresponding "equipment components" and "vibration characteristics" attributes. The formula of the template matching method is as follows:

[0076]

[0077] in, h v Represents the vibration signal characteristics, p i Represents the template feature, l i represents the device description label, and || || represents the L2 norm.

[0078] 2) Targeting text data characteristics, the semantic embedding vectors and entity recognition results extracted from maintenance logs and operational information using the BERT model are further integrated with natural language generation (NLG) technology to summarize sentence-level information and extract key semantic content relevant to diagnosis. For example, after identifying raw text fragments such as "the pump body vibrates violently," "lubrication is abnormal," and "the bearing has been replaced," NLG models (such as T5 and GPT) are used to reconstruct and summarize them, resulting in structured semantic sentences such as "the pump body vibrates violently, suspected bearing wear due to insufficient lubrication," and "regular maintenance records indicate a recurring failure trend for this component." These contents can be further abstracted into multiple entity nodes in the knowledge graph (such as "pump body," "bearing wear," and "inadequate lubrication") and connected through relationship types such as "cause and effect."

[0079] In addition, attribute fields such as "fault type", "severity", and "fault time" can be extracted to form entities to enhance the expressive power of the graph.

[0080] 3) Regarding image data features, the high-dimensional image features extracted by CNN need to be semantically annotated in combination with computer vision technology (such as segmentation results, defect location, etc.). For example, the regional distribution of temperature hotspots in infrared images can generate the following description: "Local overheating areas appear on the surface of the motor, and the center temperature exceeds 80°C." This type of description is converted into text through rule templates or language generation mechanisms, and then introduced into the atlas as an "image diagnosis result" entity, improving the searchability and reasonability of image-related information, and describing visual information such as temperature anomalies and structural damage. The formula for identifying abnormal temperature hotspot distribution is as follows:

[0081]

[0082] in, h i represents the image features extracted by CNN, T represents the temperature matrix, Z represents the composite representation of temperature and semantics, and S represents the probability that the calculated area is an abnormal hotspot. Descriptive information is generated using the rule template in combination with the coordinates x and y.

[0083] 4) Based on the multimodal textual features, define the basic elements of the knowledge graph, including entities, attributes, and relations, and clarify various equipment components, failure modes, characteristic indicators, and their interrelationships.

[0084] S3. Common modality contrastive learning feature enhancement module,

[0085] The purpose of the co-modal contrastive learning feature enhancement module is to fully explore the associations and differences between vibration signal features, text data features, and image information features through cross-modal and intra-modal contrast learning, map the complementary features of each modality into the same representation space, and thereby inject richer and more discriminative vector representations into knowledge graph nodes.

[0086] 1) Intermodal contrastive learning: Perform positive and negative comparisons on pairs of samples from different modalities to achieve cross-modal feature alignment. First, since the three modal features have different dimensions, they need to be projected into a contrastive learning space with a unified dimension d:

[0087]

[0088] Among them, m represents the modal index, and its values are v, t, and i, which represent vibration, text, and image modes respectively. Represents the pre-coded feature vector of sample k on modality m, w represents the linear transformation matrix, and b represents the bias vector. To facilitate the calculation of cosine similarity, the projection vector of each modality is L2 normalized:

[0089]

[0090] in, Represents the unnormalized vector representation obtained by mapping sample k on modality m, where m represents the modality index.

[0091] Construct positive and negative sample pairs. A training batch contains N nodes, each node corresponds to three modes {v k ,t k ,i k}. The positive sample pair includes the combination of cross-modal under the same node

[0092]

[0093] Negative pairs include pairs between different nodes in the same modality, or pairs between different nodes across modalities, e.g. In the negative sample set, the pairs that are most similar to the positive samples (with the largest dot product) but have different labels are selected to increase the training difficulty. For any pair of modalities (m,n)∈{(v,t),(v,u),(t,u)}, the InfoNCE loss is defined as:

[0094]

[0095] Among them, sim represents cosine similarity, N represents the number of samples, Represents the temperature parameter, which controls the sharpness of the similarity distribution.

[0096] Sum up the losses of the three pairs of modalities to get the overall cross-modal loss :

[0097]

[0098] 2) Network pruning: Under the premise of ensuring that the contrastive learning performance does not drop significantly, the number of model parameters is reduced and the reasoning speed is accelerated, making subsequent large-scale knowledge graph applications more efficient. Under the premise of not significantly losing the cross-modal contrastive learning effect, the parameter size of the projection head is compressed as much as possible and the reasoning efficiency is improved. First, for each weight w Calculate the importance score s(w) and use a gradient sensitivity-based metric to obtain the importance score s(w):

[0099]

[0100] in, is the cross-modal InfoNCE loss. After obtaining all s(w), the threshold is determined globally according to the target pruning ratio p%. τ , mark the p% weights with the lowest scores as redundant. Specifically, construct a binary mask:

[0101]

[0102] And update the weight parameters to sparse form .

[0103] In order to avoid a sharp drop in performance due to a one-time large-scale pruning, it is recommended to adopt a progressive pruning strategy: the total pruning ratio p max Amortize it into T stages and calculate the instantaneous pruning rate of each step using linear or cosine scheduling.

[0104]

[0105] After pruning is completed, the pruning effect is quantified by calculating the model sparsity.

[0106]

[0107] 3) Intermodal contrast learning further strengthens the distinction within each individual modality, such as different fault categories within a vibration signal, different fault modes in text descriptions, and different fault locations in an image. Quantify the attribute similarity of each sample within the same modality and use this as a basis to generate positive and negative pairs. j and k The cosine similarity under modality m is defined as attribute similarity:

[0108]

[0109] Three data enhancement operations are used for the three modes, and the vibration signal feature h v Add signal shifting,

[0110]

[0111] Where Δt represents the time offset of the translation, U(-δ,δ) It means that the offset Δt is randomly sampled from a uniform distribution in the interval [-δ,δ]. δ is the maximum allowed offset, which controls the disturbance intensity.

[0112] The text data features are filled with context masks, that is, some features are randomly masked in the input text data features and filled with language models. Gaussian blur is added to the image features to make the image h i Convolution with a two-dimensional Gaussian kernel yields a blurred enhanced image:

[0113]

[0114] in, Represents the Gaussian kernel function. The value of the image coordinate point (x, y) determines the weight distribution of the blur. σ controls the standard deviation of the Gaussian kernel and the blur intensity.

[0115] Based on the sample's augmented views, intra-modal positive and negative sample pairs are constructed to optimize the intra-modal feature distribution structure. A positive pair is defined as a pairing between two views of the same sample, and a negative pair is a pairing between that sample's view and another sample's view. To optimize the aggregation and separation of intra-modal features, intra-modal contrastive learning uses a modified InfoNCE loss function to maximize the similarity of positive pairs and minimize the similarity of negative pairs. For modality m, the loss is as follows:

[0116]

[0117] in, Representative j The representation vector of each sample after the enhancement operation passes through the projection network, Representative j The representation vector of the sample without augmentation is obtained after the projection network. τ is a temperature hyperparameter that controls the smoothness of the contrast distribution. Finally, the intra-modal losses of the three modes are weighted and combined into the overall loss:

[0118]

[0119] in, Represents the trainable weight coefficient.

[0120] Jointly optimize cross-modal and intra-modal contrast losses to achieve multimodal collaboration and category balance:

[0121]

[0122] in, Represents the trainable weight coefficient.

[0123] Fine-tuning and knowledge graph fusion injects enhanced multimodal features into knowledge graph nodes. Specifically, a fault instance node is set up in the knowledge graph and connected to the structural nodes (such as equipment components and fault descriptions) newly added to the graph under the new sample conditions. The weighted fusion of multimodal features is used as the feature of the fault instance node.

[0124] S4. Small sample task construction module,

[0125] In actual production, common fault types may have abundant data, but rare and sudden faults often have only a very small number of samples. Small sample task preparation can be achieved by extracting multiple fault categories from the knowledge graph, either evenly or based on structural importance. This forces the model to learn to distinguish each fault category using limited samples, thereby improving the diagnosis ability of rare faults.

[0126] 1) Define a small-shot task T. Small-shot learning is based on the concept of "meta-learning." It does not directly learn a classification model, but rather learns how to learn the classification task. Each training session is considered a task, which contains a support set S (Support Set) and a query set Q (Query Set). The support set S is a small number of samples for each fault type in the current task, which the model uses to "learn" the current task. The query set Q is used to evaluate the model's learning performance on the task. Assume that the current task is an N-way K-shot learning task, that is, there are N fault categories, each category has K samples in the support set and M samples in the query set. The task format is as follows:

[0127]

[0128] Among them, N represents the number of task categories, K represents the number of support samples for each category, and M represents the number of test query samples for each category. x Indicates characteristics, y Indicates the fault type.

[0129] 2) Use the constructed multimodal equipment knowledge graph to perform structured screening of fault categories to enhance the representativeness and generalization of the training task. Specifically, graph structure metrics in graph theory (such as degree centrality, eigenvector centrality, and shortest path length) are used to evaluate the fault nodes in the graph: high centrality nodes indicate associations with multiple entities and represent typical and common faults; low centrality or edge nodes may represent rare or long-tail faults. Select 𝑁 fault categories distributed in different structural positions from them C , which not only covers the backbone knowledge in the core area of the graph, but also retains the long-tail information in the marginal area, forming a structurally balanced small sample task configuration.

[0130]

[0131] 3) To improve the semantic integrity and context expression ability of each type of sample, for each fault category C i , executes a third-order neighbor sampling strategy from the knowledge graph, that is, starting from the current node, traverses the graph outward to the third hop, selects entity nodes closely related to the category (such as affected components, symptoms, historical maintenance records, etc.), and forms a subgraph structure containing semantic neighbors. The semantic text of these neighbor nodes is encoded by the pre-trained language model SBERT (Sentence-BERT) to generate the basic feature vector of each sample. The encoded feature vector is concatenated with the feature vector of the fault instance node to generate the feature vector of each sample. xi,j , complete the small sample task preparation:

[0132]

[0133] S5. Subgraph Extraction and Mamba-Enhanced Graph Neural Network Encoding,

[0134] By utilizing the constructed knowledge graph, the structural information around the fault entity corresponding to each sample is extracted and high-dimensional features are extracted through the graph neural network (GNN) to obtain a sample representation enhanced by "structure + semantics".

[0135] 1) Dynamic neighborhood subgraph extraction, based on adaptive structural perception of node importance. In order to effectively capture the structural context information around the fault entity, a local subgraph is extracted from the knowledge graph for each target entity. G i =( V i , E i ),in, V i is a collection of nodes, E i is the edge set. Considering the differences in the structural roles of different fault nodes in the graph, such as core hub nodes and edge isolated nodes, the neighborhood depths they need to perceive are different, so a structural indicator is introduced to dynamically control the number of sampling hops S. hop . For each sample corresponding to the knowledge graph entity e i , dynamically adjust the neighborhood sampling hop count S according to the topological importance of the node hop , extract its local neighborhood subgraph G i :

[0136]

[0137] Among them, Betweenness represents the betweenness centrality of the node, Degree represents the degree of the node, and parameters α and β are used to weigh the influence of betweenness centrality and degree of the node. Dist represents the number of path hops. V i Represents a collection of nodes, E i Represents a set of edges.

[0138] 2) Graph Neural Network Encoding, Mamba-enhanced Gated State Space Propagation. To achieve effective propagation of semantic information and feature fusion in subgraphs, a selective state space model (Mamba) is introduced based on the traditional gated recurrent unit (GRU) and residual connection to build a multi-level, adaptive feature interaction architecture and implement input-dependent dynamic parameter adjustment.v In the l The state update formula for the +1 layer is:

[0139]

[0140] in, MPNN stands for Message Passing Neural Network, which accelerates training and improves stability. represents the set of neighbor nodes of node v, Represents the hidden state at the current moment, MambaBlock represents the Mamba architecture model, Represents the gating function to realize the current input With the previous hidden state Fine-grained fusion. Represents the set of high-order neighbor nodes of node v. The Mamba model, as a key link in the graph neural network encoding process, solves the core bottleneck of traditional graph neural networks in long-range dependencies. The core formula of MambaBlock is:

[0141]

[0142] in, represents the hidden state vector at time step t, represents the gating function, represents the matrix exponential function, A is the learnable state transition matrix, B is the input projection matrix, I is the identity matrix, Represents the input node features, Represents the high-order neighbor node features of node v.

[0143] 3) Graph-level representation generation, readout based on structured aggregation. After multiple rounds of propagation through L layers of graph neural networks, the final embedding of each node is obtained. The entire subgraph representation needs to be aggregated into a single vector for input to downstream tasks. To this end, the structured graph readout function DiffPool is used to aggregate nodes into supernodes or the entire graph representation in a differentiable manner. The aggregated vectors of all nodes in the Lth layer form the sample representation:

[0144]

[0145] Among them, DiffPool is a readout function with structural adaptability, which can maintain the topological characteristics and semantic composition of the subgraph during the aggregation process.

[0146] S6. Prototype Network Classification,

[0147] In small-sample scenarios, few-shot classification is achieved by calculating the "category prototype" of the support set and measuring the distance between the query test sample and the prototype. The core idea of the prototypical network is to compress the support sample vectors of the same category into a prototype vector and use a distance metric to compare the similarity between the query sample and each category prototype to determine its category. Compared to traditional classifiers, the prototypical network does not rely on large-scale parameter optimization and has natural adaptability to small samples and strong generalization capabilities.

[0148] 1) Prototype vector calculation,

[0149] In a given small sample task T, there are support set S and query set Q. The support set S contains N categories and K support samples for each category. Each support sample is encoded by the graph neural network to obtain a structured subgraph encoding. k The subgraph encoding is averaged to obtain fault C k The prototype vector of:

[0150]

[0151] Where N represents the number of samples with the same fault type.

[0152] 2) Distance measurement and probability estimation,

[0153] For any test sample in the query set x q , first obtain its feature vector representation through graph neural network encoding h q Then, the cosine similarity is calculated between it and each category prototype c k Distance between:

[0154]

[0155] Among them, C represents the prototype vector, h T A test vector representing the query.

[0156] 3) Loss function,

[0157] In order to train the prototype network so that it can accurately divide the boundaries between different categories, the cross entropy loss function is introduced as a supervisory signal to minimize the difference between the predicted distribution and the true label distribution:

[0158]

[0159] Among them, Q represents the query set of task T, Represents the model input x q Predicted as the correct label y q The conditional probability of .

[0160] If there are multiple test samples, batch optimization can be used to continuously sample a batch of tasks of size B from the sample set to form a mini-batch. The overall loss is:

[0161]

[0162] 4) The probability of the classification to which the test sample belongs,

[0163] For actual diagnosis tasks, it is necessary not only to give a clear prediction category, but more importantly, to provide a probability estimate for each fault category to assist experts in evaluating the confidence level of the diagnosis results and form a risk assessment mechanism. Specifically, the query test sample x q The distance between the sample and the prototype of each category is mapped into a probability distribution to quantify the probability that the sample belongs to each fault category:

[0164]

[0165] Among them, k is the fault category, h q represents the vector representation of the query test sample, and d represents the cosine similarity measure.

[0166] S7. Online fault detection and early warning,

[0167] The offline-trained multimodal knowledge graph and small-sample prototype network model are deployed in the production environment to achieve real-time diagnosis and graded warning of new streaming data, and provide decision-making basis to operation and maintenance personnel through the explainability module.

[0168] 1) Data access and synchronization preprocessing,

[0169] The system uses multi-source sensors deployed at key locations on the equipment to synchronously collect multimodal raw data streams at time t. These streams primarily include: mechanical dynamic signals captured by vibration sensors (such as accelerometers); textual data such as operation logs and alarm messages generated by the industrial control system; and structural or thermal images captured by visible light or infrared cameras. All data is processed by preprocessing and feature extraction modules at the access point before being uniformly mapped to a target entity node in the knowledge graph space, achieving semantic alignment with the offline model.

[0170] 2) Threshold determination and multi-level warning,

[0171] Using the trained small sample prototype network model, the system performs forward reasoning on each new input sample and calculates whether it belongs to each known fault category. C kA fault identification probability threshold δ is set. If the probability of a certain fault classification is greater than the threshold δ, the probability of the fault being present is determined to be high. Based on the "severity" attribute label associated with each fault node in the knowledge graph, such as "warning level", "serious level", "emergency level", etc., the alarm level is determined based on the probability output by the model and the attribute information in the knowledge graph.

[0172] 3) Interpretability of results,

[0173] Calculate the vector h of the fault sample q and the prototype vector c of each category k distance, find the prototype category with the highest similarity k *:

[0174]

[0175] Then find the entity node e to which the sample belongs in the knowledge graph q , and the fault node e corresponding to the prototype k ∗ , search in the knowledge graph from e q to e k ∗ Several paths of {Π i}, each path is defined as:

[0176] π=(e q →r j →v j →⋯→e k ∗ ), where r j Indicates the path j The relationship type of the edge, v j is the intermediate node passed through in the path. In order to measure the credibility of each path, the path confidence function is introduced to calculate a confidence score for each path:

[0177]

[0178] in, Represents the entity node e corresponding to the current fault sample q Embedding vector in the knowledge graph, r i The embedding vector representing the relation type of the i-th edge in the path π represents the semantic relationship between entities. Represents the target fault node e corresponding to the prototype category k ∗ The embedding vector of , ||·|| represents the L2 norm. The top M most likely inference links are selected in descending order of Confidence (Π) for interpretation.

[0179] Experimental verification:

[0180] To verify the effectiveness of the proposed method in predictive fault diagnosis of power equipment, this paper designed multiple sets of comparative experiments under typical power system operating scenarios. These experiments covered key equipment such as transformers (partial discharge monitoring units), generators (bearing and stator vibration monitoring systems), high-voltage circuit breakers (contact wear detection), and transmission lines (bolt loosening image recognition and infrared temperature measurement). These experiments incorporated multimodal sensing and recording information, including vibration, infrared images, and inspection text. A total of approximately 13,000 complete diagnostic cycles were sampled. To ensure practical application, the experimental setup introduced a variety of real-world operating conditions, including grid load fluctuations, long-term equipment fatigue, false alarms caused by environmental interference, and sudden loss of key data channels. The model's adaptability to complex scenarios and fault identification accuracy were comprehensively examined.

[0181] The comparison methods include the current mainstream fault identification models, such as the attention mechanism-dominated Transformer, MMAN based on the shared attention mechanism, the multi-resolution feature fusion model MF-CNN, the KGGRN based on the graph neural network, and the small sample adaptive optimization model Proto-MAML. All models are trained under the same training / validation / testing partition (6:2:2), unified loss function and optimization strategy to ensure the comparability and consistency of the experimental results. In order to comprehensively evaluate the performance of the fault diagnosis method, six evaluation indicators are selected: ACC, F1-Score, warning lead time, missed alarm rate, false alarm rate and inference delay. The experimental results are as follows Figure 2 , Figure 3 and as shown in Table 1.

[0182] Table 1 Data comparison of different methods under six indicators

[0183] Model Name ACC F1 Early warning lead time False negative rate False alarm rate Inference latency Transformer 80% 79% 17min 19% 14% 65s MMAN 87% 85% 28min 16% 13% 72s MF-CNN 85% 82% 23min 21% 15% 60s KGGRN 88% 87% 30min 14% 11% 85s Proto-AML 87% 86% 27min 15% 13% 78s Method of the present invention 93% 91% 42min 8% 7% 42s

[0184] from Figure 2 , Figure 3 As can be seen from Table 1, traditional Transformer, MMAN, MF-CNN, KGGRN, and Proto-AML methods demonstrate some recognition capabilities in complex equipment fault diagnosis, but each has its limitations. While the Transformer method possesses strong global modeling capabilities and can model long-term dependencies to a certain extent, its native structure is less sensitive to local features and lacks structural path perception, resulting in a high false positive rate. While MMAN and KGGRN perform well in terms of accuracy and F1 score, there is still room for improvement in terms of missed detection rate and false positive rate. While MF-CNN and Proto-AML perform well in terms of inference latency, their performance is poor in terms of accuracy and F1 score.

[0185] In contrast, the method of the present invention effectively integrates multi-source heterogeneous data and enhances the expressive power of fault features through the construction of a multimodal knowledge graph and graph neural network encoding. At the same time, the small sample learning strategy introduced enables the system to make accurate judgments even when there is less sample data, thereby improving the adaptability and generalization ability of the model. In the experiment, this method outperformed the comparative methods in six dimensions: accuracy, F1 value, warning lead time, missed detection rate, false alarm rate, and reasoning delay, verifying its practicality and efficiency in predictive maintenance tasks. In particular, the accuracy and F1 value reached 93% and 91%, the warning lead time reached 42 minutes, the missed detection rate and false alarm rate were reduced to 8% and 7% respectively, and the reasoning delay was only 42 seconds, which fully surpassed other comparative methods.

[0186] In order to verify the robustness of the model, the system set different degrees of dropout rate for the original data in the experiment. The results are as follows: Figure 4 As shown in the figure, comparing the robustness of the models under different dropout ratios, the proposed method (red solid line) demonstrates significant superiority. Its accuracy (ACC) remains the highest at all dropout ratios, with only a slight decrease from 93% to 88%, demonstrating strong robustness. In contrast, the accuracy of other models decreases as the dropout ratio increases, with the Transformer's accuracy dropping from 80% to 55%, demonstrating its sensitivity to dropout changes.

[0187] In order to verify the model's early warning lead time distribution, the system tested different models in the experiment, and the results are as follows: Figure 5 As shown. According to the comparison results of the warning lead time distribution of different models in the figure, the median warning lead time of the method of the present invention is the highest, about 40 minutes, indicating that this method performs best in terms of warning lead time and can predict faults earlier. The median warning lead time of other models is less than 30 minutes, and the distribution range is large, indicating that the prediction effect is poor and the stability is poor. Therefore, the method of the present invention not only has an advantage in warning lead time, but also provides a more stable and reliable fault warning capability.

[0188] Example 2

[0189] This embodiment provides a complex equipment fault diagnosis system based on a multimodal knowledge graph, including:

[0190] The data acquisition module is configured to acquire equipment operation data;

[0191] The feature module is configured to perform multimodal feature extraction on the acquired equipment operation data;

[0192] The graph module is configured to construct a knowledge graph based on the extracted multimodal features;

[0193] The enhancement module is configured to perform multimodal feature enhancement on the knowledge graph through cross-modal contrastive learning;

[0194] The small sample module is configured to define small sample tasks and perform feature initialization on the enhanced knowledge graph;

[0195] The classification module is configured to perform subgraph extraction and Mamba enhancement on the knowledge graph after feature initialization; and use graph neural network to perform fault classification based on the knowledge graph.

[0196] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, a complex equipment fault diagnosis method based on a multimodal knowledge graph.

[0197] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded and executed by the processor to perform a complex equipment fault diagnosis method based on a multimodal knowledge graph.

[0198] The above are all preferred embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A complex equipment fault diagnosis method based on multimodal knowledge graph, characterized by: include: Obtain equipment operation data; Perform multimodal feature extraction on the acquired equipment operation data; Construct a knowledge graph based on the extracted multimodal features; Multimodal feature enhancement of knowledge graphs through cross-modal contrastive learning; Define small sample tasks and initialize features of the enhanced knowledge graph; Perform subgraph extraction and Mamba enhancement on the knowledge graph after feature initialization; Use graph neural networks to classify faults based on knowledge graphs; The multimodal feature extraction is performed on the acquired equipment operation data, including the original vibration signal data collected by the high-precision sensor, and the vibration signal data collected by the sensor is set as X v , using 1D-CNN to extract local features; using the BERT pre-trained language model to perform entity recognition on operation and maintenance logs and operation records, complete the identification of fault entities, and output the corresponding semantic embedding vector; and using the singular value decomposition SVD method to reduce the dimension of the image matrix, and input the reduced dimension image into the standard CNN network to obtain the image feature representation h i ; The knowledge graph is constructed based on the extracted multimodal features. This includes constructing a feature-fault template dictionary for vibration signal features, and using template matching and spectrum analysis to identify and classify the extracted signal features. For text data features, the semantic embedding vectors and entity recognition results extracted from operation and maintenance logs and operation information using the BERT model are further introduced into natural language generation (NLG) to summarize sentence-level information and extract key semantic content related to diagnosis. For image data features, the high-dimensional image features extracted by CNN need to be semantically annotated in combination with computer vision. The multimodal feature enhancement of the knowledge graph through cross-modal contrastive learning includes inter-modal contrastive learning, performing positive and negative comparisons on sample pairs of different modalities, and completing cross-modal feature alignment. In particular, the three modal features are projected onto a contrastive learning space of unified dimension, with the different dimensions of the features being projected. In inter-modal contrastive learning, the distinguishability is further enhanced within each individual modality, and the attribute similarity of each sample within the same modality is quantified, based on which positive and negative pairs are generated. Context mask filling is performed on text data features. Gaussian blur is added to image features, and the image is convolved with a two-dimensional Gaussian kernel to obtain a blurred enhanced image. The multimodal feature enhancement of the knowledge graph through cross-modal contrastive learning also includes constructing positive and negative sample pairs within the modality based on the sample-enhanced views, optimizing the feature distribution structure within the modality, defining a positive sample pair as a pairing between two views of the same sample, and a negative sample pair as a pairing between the sample view and other sample views. To optimize the aggregation and separation of features within the modality, the intra-modal contrastive learning uses an improved InfoNCE loss function to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. For modality m, the loss form is expressed as: ， in, Representative j The representation vector of each sample after the enhancement operation passes through the projection network, Representative j The representation vector of the sample after the projection network without the enhancement operation, τ is the temperature hyperparameter that controls the smoothness of the contrast distribution; The method defines a small sample task and performs feature initialization on the enhanced knowledge graph, including constructing a support set S as a small number of samples of each type of fault in the current task for the model to learn the current task; the query set Q is used to evaluate the learning effect of the model on the task; the knowledge graph is used to perform structured screening of fault categories, wherein the graph structure metrics, eigenvector centrality, and path length in graph theory are used to evaluate the fault nodes in the graph. For each fault category, a third-order neighbor sampling strategy is executed from the knowledge graph, that is, starting from the current node, traversing the graph outward to the third hop, selecting entity nodes closely related to the category, and forming a subgraph structure containing semantic neighbors; The subgraph extraction and Mamba enhancement of the knowledge graph after feature initialization include the introduction of structural indicators to dynamically control the number of sampling hops S hop , for each sample corresponding to the knowledge graph entity e i , dynamically adjust the neighborhood sampling hop count S according to the topological importance of the node hop , extract its local neighborhood subgraph G i : ， Among them, Betweenness represents the betweenness centrality of the node, Degree represents the degree of the node, parameters α and β are used to weigh the influence of betweenness centrality and degree of the node, and dist represents the number of path hops. V i Represents a collection of nodes, E i Represents a set of edges.

2. A complex equipment fault diagnosis method based on multimodal knowledge graph according to claim 1, characterized in that: The subgraph extraction and Mamba enhancement of the knowledge graph after feature initialization also include the introduction of the selective state space model Mamba, the construction of a multi-level, adaptive feature interaction architecture, and the implementation of input-dependent dynamic parameter adjustment. The core formula of MambaBlock is: ， in, represents the hidden state vector at time step t, represents the gating function, represents the matrix exponential function, A is the learnable state transition matrix, B is the input projection matrix, I is the identity matrix, Represents the input node features, Represents the high-order neighbor node features of node v.

3. A complex equipment fault diagnosis method based on multimodal knowledge graph according to claim 2, characterized in that: The method of using graph neural network to classify faults based on knowledge graph includes compressing the support sample vectors of the same category into a prototype vector in a small sample scenario, and comparing the similarity between the query sample and the prototypes of each category by distance measurement to determine the category to which it belongs. q The distance between the sample and the prototype of each category is mapped into a probability distribution to quantify the probability that the sample belongs to each fault category: ， Among them, k is the fault category, h q represents the vector representation of the query test sample, and d represents the cosine similarity measure.

4. A complex equipment fault diagnosis system based on a multimodal knowledge graph, executing the complex equipment fault diagnosis method based on a multimodal knowledge graph as claimed in claim 1, characterized in that: include: The data acquisition module is configured to acquire equipment operation data; The feature module is configured to perform multimodal feature extraction on the acquired equipment operation data; The graph module is configured to construct a knowledge graph based on the extracted multimodal features; The enhancement module is configured to perform multimodal feature enhancement on the knowledge graph through cross-modal contrastive learning; The small sample module is configured to define small sample tasks and perform feature initialization on the enhanced knowledge graph; The classification module is configured to perform subgraph extraction and Mamba enhancement on the knowledge graph after feature initialization; and use graph neural network to perform fault classification based on the knowledge graph.

Images