A dual-prototype driven intelligent fault diagnosis method based on multi-modal knowledge

Abstract

The application provides a dual-prototype driven intelligent fault diagnosis method based on multi-modal knowledge, relates to the technical field of network model diagnosis fault, and constructs a multi-modal fault knowledge graph through historical operation data of bearings, so as to take the multi-modal fault knowledge graph as auxiliary fault knowledge prototypes in a knowledge and data dual-driven meta-learning network. The meta-learning network comprises a knowledge feature extractor and a data feature extractor, the meta-learning network can utilize the knowledge prototypes outside the data to assist the original data prototypes of the meta-learning in identifying new class faults, so as to realize the generalization ability from known faults to unknown faults. Therefore, the application can effectively improve the efficiency and accuracy of intelligent fault diagnosis of bearings.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of network model fault diagnosis, and particularly relates to a dual-prototype-driven intelligent fault diagnosis method based on multi-modal knowledge. BACKGROUND

[0002] Bearing is an indispensable component in rotating machinery, widely used in various industrial equipment, such as wind turbine generators, automobiles, aircrafts and production lines, etc. Due to its continuous work in high speed and heavy load environment, bearing is prone to failure, which leads to equipment performance degradation or even shutdown. According to industry statistics, about 40% of the failures of rotating machinery are caused by bearing failure. Therefore, timely detection of bearing failure is of great significance to ensure normal operation of equipment and production safety.

[0003] Bearing fault detection technology can help operators and maintenance personnel to discover potential faults in advance, so as to take appropriate maintenance measures to avoid high repair costs and production losses caused by sudden failures. Traditional bearing fault detection methods mainly rely on manual experience, identifying abnormal sound and vibration through hearing and touch. However, this method is subjective, low in accuracy, and cannot adapt to the increasingly complex industrial environment. With the development of signal processing technology and machine learning algorithms, especially deep learning algorithms, deep learning uses convolutional neural network (CNN) and recurrent neural network (RNN) to detect bearing faults. However, deep learning models usually need a large amount of data for training, and the lack of samples will cause underfitting problem in the training process. In addition, most current deep learning researches have the problem of large data resource demand, and using small sample learning to solve fault detection in the case of data scarcity is obviously more practical.

[0004] In order to promote the deep combination of small sample learning model and fault detection, some researches try to use transfer learning, data augmentation and metric learning methods to improve the generalization ability and robustness of the model. Transfer learning improves the performance of new models by using the prior knowledge of existing models, but the performance is highly dependent on the similarity of fault sample distribution. In order to expand the sample, data augmentation method is often used, which generates new samples to expand the training data, but the model training is difficult, and the gradient vanishing problem is difficult to solve.

[0005] Small sample learning model, especially meta-learning, can make the model learn how to learn. This method, which conforms to the way humans learn, has been widely used in small sample research in recent years. The addition of meta-learning can enable deep learning models to generalize the model's recognition ability of the original base class to new classes, allowing the model to generalize the classification of new classes using a small number of samples. However, the current development of meta-learning is hindered by the limitation of data-driven. The requirement of the current artificial intelligence 2.0 era for deep learning models is to have more information to be added, thus giving rise to knowledge-driven deep learning models. In the latest deep learning research, dual driving of knowledge and data has been proven to be superior to single data-driven, allowing the model to learn more information about new classes, enabling the meta-learning model to learn the scarce new class information using the base class information in the case of data scarcity, thus achieving the purpose of small sample learning.

[0006] Current knowledge-driven deep learning models can be roughly classified into three categories. First, deep learning based on symbolic reasoning, which realizes the collaborative optimization of "perception + reasoning" by integrating data-driven learning and logical rule reasoning. Symbolic systems can perform transparent reasoning, but they are difficult to handle unstructured data. The implementation form of this method is to convert the perception results of neural networks (such as identifying objects in images) into symbolic inputs to reasoning engines for logical inference (such as judging object relationships). The current differences between symbols and vectors, dynamic knowledge updating efficiency, and embedding of complex logic are bottlenecks. Second, the combination of expert systems and deep learning, which focuses on using expert knowledge to assist and constrain the training of neural networks. In this collaborative mechanism, expert systems provide interpretable domain knowledge (such as fault diagnosis rules) to make up for the "black box" defects of neural networks, while neural networks mine complex patterns (such as signal features and semantic associations) from massive data to break through the bottleneck of expert systems relying on artificial rules and being difficult to handle unstructured data. This method also has obvious defects such as strict logical constraints that may weaken the flexibility of neural networks (such as "minor flaws" being misjudged as faults by rules in industrial quality inspection). Finally, neural networks based on knowledge graph assistance, which constructs the relationship between fault categories through knowledge graph, starting from the attributes of fault categories, and constructs the intra-class and inter-class relationships of faults. This method can construct a knowledge graph with new classes based on the base class to assist the meta-learning network in recognizing new classes in the case of new class data scarcity.

[0007] In existing academic research, the prototype network has better learning ability in the case of a small amount of samples, because the prototype network can determine the data prototype of the base class in the feature space, and even a small amount of samples can be used for metric learning. The existence of the prototype can also play a role in the learning of the new class by measuring the base class. The bottleneck of the current prototype network and most neural networks is data, and the method to break through this research gap is to use knowledge-driven auxiliary data-driven to further improve the learning ability of the prototype network for new classes. In the past, the conventional use is only the use of symbolic reasoning, expert knowledge, or knowledge graph, which has many shortcomings in the adaptability of the prototype network, and is not sufficient to fully utilize knowledge information to assist, thereby causing the defects of low detection efficiency and low detection accuracy for bearing fault. SUMMARY

[0008] In order to solve the above problems existing in the prior art, the present application provides a double-prototype-driven intelligent fault diagnosis method based on multi-modal knowledge. The technical problem to be solved by the present application is solved by the following technical scheme:

[0009] A double-prototype-driven intelligent fault diagnosis method based on multi-modal knowledge comprises:

[0010] S100, a data set is composed of historical running data of a bearing, and a multi-modal fault knowledge graph is constructed using the data set; wherein the label category of the data set is , The plurality of fault categories and the one normal category are included in the plurality of label categories;

[0011] S200, a knowledge prototype is extracted from the multi-modal fault knowledge graph using a knowledge feature extractor;

[0012] S300, the data set is divided into a support set and a query set, and a data prototype is extracted from the support set using a data feature extractor;

[0013] S400, the knowledge prototype and the data prototype are fused to obtain a fused prototype, and a metric learning loss is calculated using the fused prototype, the knowledge prototype and the data prototype;

[0014] S500, the knowledge feature extractor and the data feature extractor are trained by back propagation until the metric learning loss converges, to obtain trained knowledge feature extractor and data feature extractor;

[0015] S600, current running data of the bearing is received and input into the trained knowledge feature extractor and data feature extractor respectively to obtain a current fused prototype; the fault category of the bearing is determined using the current fused prototype.

[0016] Beneficial effects:

[0017] The application provides a dual-prototype driven intelligent fault diagnosis method based on multi-modal knowledge, constructs a multi-modal fault knowledge graph through historical operation data of a bearing, and takes the multi-modal fault knowledge graph as an auxiliary fault knowledge prototype in a knowledge-data dual-driven meta-learning network. The meta-learning network comprises a knowledge feature extractor and a data feature extractor. The meta-learning network can utilize the knowledge prototype in addition to data to assist the original data prototype of the meta-learning in identifying new fault types, thereby realizing the generalization ability from known faults to unknown faults. Therefore, the application can effectively improve the efficiency and accuracy of intelligent fault diagnosis of the bearing.

[0018] The application will be further described in detail below in combination with the drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flowchart of a dual-prototype driven intelligent fault diagnosis method based on multi-modal knowledge provided by the application;

[0020] Figure 2 is a full-process schematic diagram of a dual-prototype driven intelligent fault diagnosis method based on multi-modal knowledge provided by the application;

[0021] Figure 3 is a schematic diagram of a multi-modal fault knowledge graph provided by the application;

[0022] Figure 4 is a structural schematic diagram of a data feature extractor provided by the application. DETAILED DESCRIPTION

[0023] The application will be further described in detail below in combination with specific embodiments, but the embodiments of the application are not limited thereto.

[0024] The application aims to break through the limitations of traditional single data driven fault diagnosis models, realizes the deep integration of knowledge driven and data driven in industrial scenarios through the construction of a multi-modal fault knowledge graph. The addition of knowledge driven not only makes the industrial small sample fault diagnosis have a more accurate recognition degree for fault diagnosis under the condition of data scarcity, but also helps the model to generalize from the features of old faults to the diagnosis of new fault features through knowledge assistance, further improves the existing data driven deep learning model, and thereby improves the efficiency of intelligent fault diagnosis.

[0025] In combination with Figure 1 and Figure 2 , the application provides a dual-prototype driven intelligent fault diagnosis method based on multi-modal knowledge, which comprises the following steps:

[0026] S100, a data set is formed by using historical operation data of the bearing, and a multi-modal fault knowledge graph is constructed by using the data set; wherein a label category of the data set includes a plurality of fault categories and one normal category; , The plurality of fault categories and the one normal category are included in the one label category;

[0027] In a specific embodiment of the present application, S100 includes:

[0028] S110, historical operation data of the bearing is collected and a data set is formed, the data set including a plurality of data recording fault information; the fault information including fault depth, load type, fault type, fault location and rotating speed;

[0029] Taking the CWRU data set as an example, in the CWRU data set, signal data (stored in.mat file format), data sampling process description (including equipment, bearing model, sampling method, etc.), data attribute description (load, fault depth, rotating speed, fault location, etc. corresponding to each data file) are included.

[0030] S120, each data is assigned a label, and each label is taken as a node and the fault information is taken as attribute data of the node;

[0031] S130, the same load, the same rotating speed, the same fault location, the same fault depth and the same fault type are taken as connection relationship between nodes, so as to construct an undirected graph;

[0032] S140, the fault information in each data is described by using text to obtain fault text, and the fault text is corresponded to the attribute data, so as to obtain a multi-modal fault knowledge graph.

[0033] The conventional single-modal knowledge graph mainly includes nodes, attributes and relationships. By reorganizing the data attributes, attribute data relationships can be obtained, as shown in Table 1.

[0034] Table 1

[0035]

[0036] Each piece of data in the table 1 is assigned a label, and the label format is a non-negative integer (for example, when the CWRU data set is used for training, the number of labels corresponds to the number of files). The label is taken as a node, and other main data (fault depth, load type, fault type, fault location, speed) in the same row is taken as an attribute to construct a single node of the knowledge graph. Normal is N, and fault types O6, O3, O12, I, and B represent outer ring faults in the 6 o'clock direction, outer ring faults in the 3 o'clock direction, outer ring faults in the 12 o'clock direction, inner ring faults, and ball faults, respectively. The same load, the same speed, the same fault location, the same fault depth, and the same fault type are taken as relations of the knowledge graph to construct an undirected graph. In addition, the information in the data file is described by using a text description, for example, a text format of “the X bearing rotates at X rpm under the load X, and XX faults occur at the X end, the fault depth is X, and the fault signal file X is generated at the sampling frequency X”, to construct a multi-modal fault knowledge graph of attribute data and fault text, and a local structure of the graph is as shown in Figure 3

[0037] S200, extracting a knowledge prototype from the multi-modal fault knowledge graph by using a knowledge feature extractor;

[0038] In a specific embodiment of the present application, S200 includes:

[0039] S210, encoding attribute data in the multi-modal fault knowledge graph to obtain attribute encoded data;

[0040] Specifically, S210 includes:

[0041] S211, determining first attribute data for fault type and fault location, and second attribute data of the remaining numerical types from the multi-modal fault knowledge graph;

[0042] S212, encoding the first attribute data by using a one-hot encoding method to obtain first attribute encoded data, and encoding the second attribute data by using a normalization method to obtain second attribute encoded data.

[0043] ​Before entering the knowledge feature extractor, the multi-modal fault knowledge graph needs to encode the attribute data and extract the semantics of the fault text. The specific encoding method of the attribute data is that the fault type and fault position use one-hot encoding, and other numerical data uses the normalization method. Taking the CWRU dataset as an example, normal operation is N, and fault types O6, O3, O12, I and B represent 6 o'clock direction outer ring fault, 3 o'clock direction outer ring fault, 12 o'clock direction outer ring fault, inner ring fault and ball fault, respectively, which can be represented by one-hot encoding [1, 0, 0, 0, 0, 0], [0, 1, 0, 0, 0, 0], [0, 0, 1, 0, 0, 0], [0, 0, 0, 1, 0, 0], [0, 0, 0, 0, 1, 0], [0, 0, 0, 0, 0, 1] respectively. The fault position is divided into drive end (drive end) and fan end (fan end) and no, so it is encoded as [1, 0, 0], [0, 1, 0], [0, 0, 1] respectively. Other numerical attribute data is normalized according to the following formula:

[0044] ;

[0045] wherein, denotes the normalized attribute data, is the original attribute data, is the minimum value of the attribute data, is the maximum value.

[0046] S220, for the fault text in the multi-modal fault knowledge graph, the semantic feature extraction method is used to extract features to obtain a fault description knowledge prototype;

[0047] The formula used by the semantic feature extraction method to extract features is:

[0048] ;

[0049] ;

[0050] ;

[0051] ;

[0052] ;

[0053] ;

[0054] wherein, denotes the fault description knowledge prototype, denotes the extracted text language vector , Used to measure the difference between the co-occurrence probability and independent probability of two elements (words / pixels). Indicates the semantic units, Indicates the semantic units, represents the Hadamard product, 、 and They all represent intermediate parameters used for presentation and have no specific meaning; represents the concatenation of vectors, Indicates the use of pre-trained Bert model extraction The global semantics obtained by the features, Indicates the Semantic unit and The co-occurrence frequency of semantic units, Indicates the The frequency of occurrence of semantic units, Indicates the The frequency of occurrence of semantic units, and Both represent intermediate parameters used for presentation, have no specific meaning, and are superscripted represents the word vector dimension, represents the sigmoid function, represents the entropy function, Indicates word frequency, represents the inverse text frequency, represents a learnable weight matrix parameter, Indicates the The vector representation of the output of the layer Transformer encoder, It is The weight coefficients of the layer Transformer encoder, Indicates the total number of semantic units.

[0055] S230, generating the node feature matrix and adjacency matrix from the attribute data;

[0056] S240, constructing a knowledge feature extractor using the node feature matrix, the adjacency matrix, and the data characteristics of the fault description knowledge prototype;

[0057] The attribute data except the fault text is used to generate the node feature matrix and adjacency matrix of the graph. The dimension is [ , ], adjacency matrix The dimension is [ , ] , is the number of nodes, is the node feature dimension.

[0058] S250: Send the attribute code to the knowledge feature extractor for feature extraction to obtain a fault attribute knowledge prototype, and use it as the knowledge prototype.

[0059] Fault knowledge feature extraction combines the extraction results of semantic features to construct a knowledge feature extractor, and finally extracts the fault attribute knowledge prototype. The extraction process of the knowledge feature extractor is expressed as:

[0060] ;

[0061] Where, Represents the knowledge prototype of fault attributes, and also represents the knowledge prototype of the final output. Represents a learnable parameter, ranging from [0,1]; Represents a feature extractor for a graph convolutional neural network.

[0062] S300, dividing the data set into a support set and a query set, and extracting data prototypes from the support set using a data feature extractor;

[0063] The structure of the data feature extractor of the present invention refers to Figure 4 As shown in the figure, it mainly includes a one-dimensional convolution layer, a time series convolution module, a multi-level convolution layer, a time series convolution module and a global average pooling layer; the time series convolution module consists of a causal convolution layer, a ReLU activation function, a regularization layer and a downsampling layer connected in sequence; the multi-level convolution layer consists of a one-dimensional convolution layer, multiple pooling layers and a feature splicing layer.

[0064] In a specific embodiment of the present invention, S300 includes:

[0065] S310, the data is concentrated into the front Fault categories And add 1 normal category of running data as the base class;

[0066] Among them, the requirements The normal data labels are included in the base class. The purpose of this is to make the base class data the main body of model training, and to generalize the model well to new classes during the testing phase.

[0067] S320, after the data is centralized The operating data of each fault category is taken as a new category;

[0068] S330, dividing the base class into a training set and a test set according to a predetermined ratio;

[0069] The predetermined ratio can be 4:1, and can also be adjusted according to actual conditions.

[0070] S340, dividing the training set into a first support set and a first query set of meta-learning and dividing the test set into a second support set and a second query set of meta-learning;

[0071] S350, dividing the bearing operation data of the new class into the second query set;

[0072] In the first support set and the second support set, each kind of sample is 5, and in the first query set and the second query set, each kind of sample is 15. The data of the test set is divided in the same way, but the number of tasks (episodes) of the test set needs to be much larger than that of the training set.

[0073] S360, inputting the first support set into a data feature extractor to obtain a data prototype and inputting the first query set into the data feature extractor to obtain a first query set feature.

[0074] S400, fusing the knowledge prototype and the data prototype to obtain a fused prototype, and calculating a metric learning loss by using the fused prototype, the knowledge prototype and the data prototype;

[0075] In a specific embodiment of the present application, S400 includes:

[0076] S410, fusing the knowledge prototype and the data prototype by using a fusion formula to obtain a fused prototype; the fusion formula is represented as:

[0077] ;

[0078] In the formula, represents the fused prototype, represents the data prototype, represents a learnable interaction term coefficient, represents the knowledge prototype, represents a gating matrix, represents splicing and

[0079] S420, calculating a metric learning loss by using the fused prototype, the first query set feature, the data prototype and the knowledge prototype.

[0080] In a specific embodiment of the present application, S420 includes:

[0081] S421, calculating a cross-entropy loss between the fused prototype and the first query set feature, and taking it as a first loss, represented as: ​

[0082] ;

[0083] wherein, represents cross-entropy loss, represents the Euclidean distance between and represents the first query set feature, represents the label corresponding to the data sample of the first query set.

[0084] S422, calculate the orthogonal regularization loss of the data prototype and the knowledge prototype as the second loss, denoted as:

[0085]

[0086] wherein, the superscript T represents transposition, represents the square of the Frobenius norm of the knowledge prototype and the data prototype, which aims to force the feature space of the two types of prototypes to be decoupled.

[0087] The purpose of the second loss is to force the feature space of the two types of prototypes to be decoupled, separate shared features and private features.

[0088] S423, calculate the metric learning loss by using the first loss and the second loss, denoted as:

[0089] ;

[0090] wherein, represents the metric learning loss, represents the first loss, represents the second loss, represents a data set-specific hyperparameter, and the value range is [0, 1].

[0091] S500, train the knowledge feature extractor and the data feature extractor by back propagation until the metric learning loss converges, to obtain the trained knowledge feature extractor and the data feature extractor;

[0092] Wherein, the knowledge feature extractor and the data feature extractor are trained by back propagation, so that when optimizing the parameters, the optimizer uses the Adam optimizer with default parameters. The convergence of the metric learning loss means that the metric learning loss is minimized or no longer changes.

[0093] In a specific embodiment of the present application, after S500, the multi-modal knowledge-based dual prototype driven intelligent fault diagnosis method further comprises:

[0094] ​a. Inputting the second support set into the trained knowledge feature extractor and data feature extractor to obtain a knowledge prototype and a data prototype respectively; then fusing the knowledge prototype and the data prototype to obtain a fused prototype of the second support set;

[0095] b. Input the second query set into the data feature extractor to obtain the second query set features; calculate the Euclidean distance between the fusion prototype of the second support set and the second query set features;

[0096] c. Calculate the classification evaluation index using the Euclidean distance in b.

[0097] d. Evaluate the trained knowledge feature extractor and data feature extractor using the classification evaluation index.

[0098] In a meta-learning task (episode) during the testing process, samples from the second support set are extracted using the pre-trained feature extractor RTCN to obtain data prototypes. During the training process, knowledge prototypes are fused to obtain fused prototypes. Data from the second query set (including base classes and new classes) is then input into the RTCN for feature extraction to obtain data prototypes. The distance between the fused prototype and the data prototype is calculated, and classification and statistical accuracy are evaluated. The evaluation metric calculation formula is as follows:

[0099] ;

[0100] in, Four statistical indicators are used, namely accuracy, precision, recall, and F1 score; represents the fusion prototype corresponding to the second support set, Represents the data prototype corresponding to the second query set, Indicates the number of samples in the second query set.

[0101] S600, receiving the current operating data of the bearing, and inputting it into the trained knowledge feature extractor and data feature extractor respectively to obtain the current fusion prototype; and determining the fault category of the bearing using the current fusion prototype.

[0102] When predicting a bearing's fault category based on its current operating data, the present invention simultaneously acquires existing fault operating data. This fault operating data is used as the support set, and the current operating data as the query set. Similar to the testing process, the closest fault category in the support set is obtained as the bearing's current fault category, thereby predicting the bearing's fault category.

[0103] It is to be noted that the terms "first", "second", and the like in the description do not necessarily connote an absolute order among the technical features and do not imply or connote a quantity of technical features. Thus, features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0104] The above is a further detailed description of the present application in combination with specific preferred embodiments, and cannot be deemed to limit the specific implementation of the present application to these descriptions. For ordinary skilled persons in the art to which the present application belongs, a number of simple deductions or replacements can be made without departing from the concept of the present application, and all of them shall be deemed to fall within the protection scope of the present application.

Claims

1. A dual-prototype driven intelligent fault diagnosis method based on multi-modal knowledge, characterized in that, Comprise: S100, a data set is composed by using historical operation data of the bearing, and a multi-modal fault knowledge graph is constructed by using the data set; wherein a label category of the data set comprises a plurality of fault categories and one normal category; , ​ S200, extracting a knowledge prototype from the multi-modal fault knowledge graph by using a knowledge feature extractor; S300, dividing the data set into a support set and a query set, and extracting a data prototype from the support set by using a data feature extractor; S400, fusing the knowledge prototype and the data prototype to obtain a fused prototype, and calculating a metric learning loss by using the fused prototype, the knowledge prototype and the data prototype; S500, training the knowledge feature extractor and the data feature extractor by back propagation until the metric learning loss converges, to obtain trained knowledge feature extractor and data feature extractor; S600, receiving current running data of the bearing, and inputting the current running data into the trained knowledge feature extractor and the data feature extractor respectively to obtain a current fused prototype; determining the fault category of the bearing by using the current fused prototype; S100 comprises: S110, collecting historical running data of the bearing and forming a data set, the data set comprising a plurality of data recording fault information; the fault information comprising fault depth, load type, fault type, fault position and rotating speed; S120, assigning a label to each data, and taking each label as a node and taking the fault information as attribute data of the node; S130, taking the same load, the same rotating speed, the same fault position, the same fault depth and the same fault type as the connection relationship between the nodes, thereby constructing an undirected graph; S140, obtaining fault text by describing the fault information in each data by using text, and corresponding the fault text with the attribute data, thereby obtaining a multi-modal fault knowledge graph; S300 comprises: S310, the data is concentrated into the front Fault categories And add 1 normal category of running data as the base class; S320, after the data set operational data of the individual failure categories as new classes; S330, dividing the base class into a training set and a test set according to a predetermined proportion; S340, dividing the training set into a first support set and a first query set of meta-learning and dividing the test set into a second support set and a second query set of meta-learning; S350, dividing the bearing running data of the new class into the second query set; S360, inputting the first support set into the data feature extractor to obtain a data prototype, and inputting the first query set into the data feature extractor to obtain a first query set feature; S400 comprises: S410, fusing the knowledge prototype and the data prototype by using a fusion formula to obtain a fused prototype; the fusion formula is represented as: ； wherein, represents a fusion prototype, represents a data prototype, represents a learnable interaction term coefficient, represents a knowledge prototype, represents a gating matrix, represents concatenating and represents a Hadamard product;​ S420, calculating a metric learning loss by using the fused prototype, the first query set feature, the data prototype and the knowledge prototype.

2. The dual-prototype driven intelligent fault diagnosis method based on multi-modal knowledge according to claim 1, characterized in that, S200 comprises: S210, encoding the attribute data in the multi-modal fault knowledge graph to obtain attribute encoded data; S220, extracting features from the fault text in the multi-modal fault knowledge graph by using a semantic feature extraction method to obtain a fault description knowledge prototype; S230, generating a node feature matrix and an adjacency matrix from the attribute data; S240, constructing a knowledge feature extractor by using the node feature matrix, the adjacency matrix and the data characteristics of the fault description knowledge prototype; S250, input the attribute code into the knowledge feature extractor for feature extraction to obtain a fault attribute knowledge prototype.

3. The dual-prototype driven intelligent fault diagnosis method based on multi-modal knowledge according to claim 2, characterized in that, S210 includes: S211, determine first attribute data of fault type and fault location and second attribute data of remaining numerical values from the multi-modal fault knowledge graph; S212, encode the first attribute data by using one-hot encoding to obtain first attribute encoding data, and encode the second attribute data by using normalization to obtain second attribute encoding data.

4. The dual-prototype-driven intelligent fault diagnosis method based on multi-modal knowledge according to claim 2, characterized in that, The formula used by the semantic feature extraction method in S220 for feature extraction is: ； ； ； ； ； ； In the formula, represents a fault description knowledge prototype, represents an extracted text language vector , is used to measure the difference between the co-occurrence probability and the independent probability of two elements, represents the first semantic unit, represents the first semantic unit, represents Hadamard product, , and all represent intermediate variables for conversion, and have no specific meaning; represents the concatenation of vectors, represents the global semantics obtained by using a pre-trained Bert model to extract features, represents the co-occurrence frequency of the first semantic unit and the first semantic unit, represents the frequency of the first semantic unit, represents the frequency of the first semantic unit, and all represent intermediate variables for conversion, and have no specific meaning, and the upper subscript represents the dimension of the word vector, represents the sigmoid function, represents the entropy function, represents the word frequency, represents the inverse text frequency, represents a learnable weight matrix parameter, represents the vector representation output by the first layer Transformer encoder, is the weight coefficient of the first layer Transformer encoder, represents the total number of semantic units.

5. The dual-prototype-driven intelligent fault diagnosis method based on multi-modal knowledge according to claim 1, characterized in that, S420 includes: S421, calculate the cross-entropy loss between the fusion prototype and the first query set feature, and take it as the first loss; S422, calculate the orthogonal regularization loss between the data prototype and the knowledge prototype, and take it as the second loss; S423, calculate the metric learning loss by using the first loss and the second loss.

6. The dual-prototype driven intelligent fault diagnosis method based on multi-modal knowledge according to claim 5, characterized in that, The metric learning loss is represented by the formula: ； In the formula, denotes a metric learning loss, denotes a first loss, denotes a second loss, denotes a data set-specific hyperparameter, and takes a value in the range [0, 1]. ； In the formula, denotes the cross-entropy loss, denotes the Euclidean distance between and denotes the first query set feature, denotes the label corresponding to the data sample of the first query set;​ where the superscripts T denote the transpose, denote the square of the Frobenius norm of the knowledge and data prototypes.

7. The dual-prototype driven intelligent fault diagnosis method based on multi-modal knowledge according to claim 1, characterized in that, After S500, the multi-modal knowledge based double prototype driven intelligent fault diagnosis method further includes: a, input the second support set into the trained knowledge feature extractor and data feature extractor to obtain the knowledge prototype and data prototype respectively; then fuse the knowledge prototype and data prototype to obtain the fusion prototype of the second support set; b, input the second query set into the data feature extractor to obtain the second query set feature; calculate the Euclidean distance between the fusion prototype of the second support set and the second query set feature; c, calculate the classification evaluation index by using the Euclidean distance in b; d, evaluate the trained knowledge feature extractor and data feature extractor by using the classification evaluation index.

Images