Double-prototype drive intelligent fault diagnosis method based on multi-modal knowledge

Abstract

The invention provides a double-prototype drive intelligent fault diagnosis method based on multi-modal knowledge, and relates to the technical field of network model fault diagnosis. And taking the multi-mode fault knowledge graph as knowledge and data to learn an auxiliary fault knowledge prototype in a network. The meta-learning network comprises a knowledge feature extractor and a data feature extractor, and the meta-learning network can utilize knowledge prototypes except for data to assist meta-learning of the identification capability of original data prototypes on new faults, so that the generalization capability from known faults to unknown faults is realized. Therefore, the bearing intelligent fault diagnosis efficiency and accuracy can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of network model fault diagnosis, and particularly relates to a dual-prototype-driven intelligent fault diagnosis method based on multimodal knowledge. Background Technique

[0002] Bearings are indispensable components in rotating machinery and are widely used in various industrial equipment such as wind turbines, automobiles, airplanes, and production lines. Due to their continuous operation in high-speed and heavy-load environments, bearings are prone to failures, which can lead to a decline in equipment performance and even shutdown. According to industrial statistics, approximately 40% of the failures in rotating machinery are caused by bearing failures. Therefore, the timely detection of bearing failures is of great significance for ensuring the normal operation of equipment and production safety.

[0003] Bearing fault detection technology can help operators and maintenance personnel detect potential failures in advance, so as to take corresponding maintenance measures and avoid the high maintenance costs and production losses caused by sudden failures. Traditional bearing fault detection methods mainly rely on manual experience to identify abnormal sounds and vibrations through hearing and touch. However, this method is highly subjective, has low accuracy, and cannot adapt to the increasingly complex industrial environment. With the development of signal processing technology and machine learning algorithms, especially deep learning algorithms, deep learning uses convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to detect bearing faults. However, deep learning models usually require a large amount of data for training, and the lack of samples will lead to underfitting problems during training. In addition, most current deep learning research has the problem of large data resource requirements. It is obviously more practical to use few-shot learning to solve fault detection in the case of scarce data.

[0004] To promote the deep integration of few-shot learning models and fault detection, some studies have tried to use methods such as transfer learning, data augmentation, and metric learning to improve the generalization ability and robustness of the models. Transfer learning improves the performance of new models by leveraging the prior knowledge of existing models, but the performance highly depends on the similarity of the fault sample distribution. To expand the samples, data augmentation methods are often used. This method expands the training data by generating new samples, but the model training is difficult and the problem of gradient disappearance is difficult to solve.

[0005] Few-shot learning models, especially meta-learning, enable the model to learn how to learn. This method, which conforms to the human learning approach, has been widely used in recent few-shot research. The addition of meta-learning allows the deep learning model to generalize its class discrimination ability to new classes based on the original base classes, enabling generalization classification of new classes with a small number of samples. However, the current resistance to the development of meta-learning is the limitation of data-driven. In the current era of AI 2.0, the requirement for deep learning models is that more information needs to be incorporated, thus giving rise to knowledge-driven deep learning models. In the latest deep learning research, the dual drive of knowledge and data has been proven to be superior to single data-driven, enabling the model to learn more information about new classes, allowing the meta-learning model to make good use of base class information to learn scarce new class information even in the case of scarce data, thereby achieving the goal of few-shot learning.

[0006] Currently, knowledge-driven deep learning models can be roughly classified into three categories. First, there is deep learning based on symbolic reasoning, which realizes the collaborative optimization of "perception + reasoning" by integrating data-driven learning and logical rule reasoning. The symbolic system can perform transparent reasoning but is difficult to handle unstructured data. The implementation form of this method is to convert the perceptual results of the neural network (such as identifying objects in an image) into symbolic inputs to the inference engine for logical inference (such as judging object relationships). Currently, there are differences in symbolic and vector representations, dynamic knowledge update efficiency, and bottlenecks in embedding complex logics. Second, there is the combination of expert systems and deep learning. This method mainly uses expert knowledge to assist and constrain the training of neural networks. In this collaborative mechanism, the expert system provides interpretable domain knowledge (such as fault diagnosis rules) to make up for the "black box" defect of the neural network, while the neural network mines complex patterns (such as signal features, semantic associations) from massive data, breaking through the bottleneck that expert systems rely on artificial rules and are difficult to handle unstructured data. This method also has obvious defects. For example, strict logical constraints may weaken the flexibility of the neural network (such as in industrial quality inspection, "minor defects" are misjudged as faults by the rules). Finally, there are neural networks assisted by knowledge graphs. Knowledge graphs construct the relationships between fault categories, starting from the attributes of the fault types themselves, and build intra-class and inter-class relationships of faults. This method can construct a knowledge graph with new classes based on the base classes to assist the meta-learning network in completing the recognition of new classes in the case of scarce new class data.

[0007] In existing academic research, the prototype network has better learning ability in the case of a small number of samples. This is because the prototype network can determine data prototypes for base classes in the feature space, and metric learning can be carried out even with a small number of samples. The existence of prototypes can also enable the learning of new classes by measuring base classes. However, the development bottlenecks of current prototype networks and most neural networks lie in data. The way to break through this research gap is to use knowledge-driven to assist data-driven, so as to further improve the learning ability of the prototype network for new classes. In the past, the conventional usage methods, whether it is symbolic reasoning, expert knowledge, or knowledge graphs, have many deficiencies in the adaptability of the prototype network and are not sufficient to make full use of knowledge information for assistance, resulting in low detection efficiency and insufficient detection accuracy for bearing faults. Summary of the Invention

[0008] To solve the above problems existing in the prior art, the present invention provides a dual-prototype-driven intelligent fault diagnosis method based on multi-modal knowledge. The technical problems to be solved by the present invention are realized through the following technical solutions: A dual-prototype-driven intelligent fault diagnosis method based on multi-modal knowledge includes: S100, using the historical operation data of the bearing to form a data set, and using the data set to construct a multi-modal fault knowledge graph; wherein, the label categories of the data set are , Among the label categories, there are multiple fault categories and 1 normal category; S200, using a knowledge feature extractor to extract knowledge prototypes from the multi-modal fault knowledge graph; S300, dividing the data set into a support set and a query set, and using a data feature extractor to extract data prototypes from the support set; S400, fusing the knowledge prototypes and the data prototypes to obtain fused prototypes, and calculating metric learning losses using the fused prototypes, knowledge prototypes, and data prototypes; S500, training the knowledge feature extractor and the data feature extractor through backpropagation until the metric learning loss converges, to obtain the trained knowledge feature extractor and data feature extractor; S600, receiving the current operation data of the bearing, and inputting it into the trained knowledge feature extractor and data feature extractor respectively to obtain the current fused prototype; using the current fused prototype to determine the fault category of the bearing.

[0009] Advantageous Effects: The present invention provides a dual-prototype-driven intelligent fault diagnosis method based on multimodal knowledge. A multimodal fault knowledge graph is constructed using the historical operation data of bearings, and this multimodal fault knowledge graph is used as an auxiliary fault knowledge prototype in a knowledge- and data-driven meta-learning network. The meta-learning network includes a knowledge feature extractor and a data feature extractor. The meta-learning network can utilize the knowledge prototype outside the data to assist the original data prototype in the meta-learning to identify new types of faults, thereby realizing the generalization ability from known faults to unknown faults. Therefore, the present invention can effectively improve the efficiency and accuracy of intelligent fault diagnosis of bearings.

[0010] The following will further elaborate on the present invention in conjunction with the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a schematic flowchart of a dual-prototype-driven intelligent fault diagnosis method based on multimodal knowledge provided by the present invention; Figure 2 is a schematic diagram of the whole process of a dual-prototype-driven intelligent fault diagnosis method based on multimodal knowledge provided by the present invention; Figure 3 is a schematic diagram of the multimodal fault knowledge graph provided by the present invention; Figure 4 is a schematic diagram of the structure of the data feature extractor provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0012] The following further describes the present invention in detail with specific embodiments, but the embodiments of the present invention are not limited thereto.

[0013] The present invention aims to break through the limitations of traditional single-data-driven fault diagnosis models and achieve the deep integration of knowledge-driven and data-driven in industrial scenarios by constructing a multimodal fault knowledge graph. The addition of knowledge-driven not only enables industrial small-sample fault diagnosis to have a more accurate recognition ability for fault diagnosis under data-scarce conditions, but also, with the assistance of knowledge, helps the model generalize from the features of old faults to the diagnosis of new fault features, further improving on the existing data-driven deep learning model, thereby improving the efficiency of intelligent fault diagnosis.

[0014] Combined with Figure 1 and Figure 2 , the present invention provides a dual-prototype-driven intelligent fault diagnosis method based on multimodal knowledge, including: S100, using the historical operation data of bearings to form a data set, and using this data set to construct a multimodal fault knowledge graph; wherein, the label categories of the data set are , label categories include multiple fault categories and 1 normal category; In a specific embodiment of the present invention, S100 includes: S110, collecting historical operation data of the bearing and forming a data set, where the data set includes multiple pieces of data recording fault information; the fault information includes fault depth, load type, fault type, fault location, and rotational speed; Taking the CWRU data set as an example for illustration. In the CWRU data set, it is divided into signal data (stored in the.mat file format), description of the data sampling process (including equipment, bearing model, sampling method, etc.), and description of data attributes (load, fault depth, rotational speed, fault location, etc. corresponding to each data file).

[0015] S120, assigning a label to each piece of data, and taking each label as a node and the fault information as the attribute data of the node; S130, taking the same load, the same rotational speed, the same fault location, the same fault depth, and the same fault type as the connection relationship between nodes, thereby constructing an undirected graph; S140, using text to describe the fault information in each piece of data to obtain a fault text, and corresponding the fault text with the attribute data, thereby obtaining a multi-modal fault knowledge graph.

[0016] Conventional single-modal knowledge graphs are mainly divided into nodes, attributes, and relationships. By reorganizing the data attributes, the attribute data relationship can be obtained, as shown in Table 1.

[0017] Table 1

[0018] Assign a label to each piece of data in Table 1, and the label format is a non-negative integer (for example, when training using the CWRU data set, the number of labels corresponds to the number of files). Take the label as a node, and the other main data in the same row (fault depth, load type, fault type, fault location, rotational speed) as the attributes to construct a single node of the knowledge graph. Normal is N, and the fault types O6, O3, O12, I, B represent the outer ring fault at the 6 o'clock direction, the outer ring fault at the 3 o'clock direction, the outer ring fault at the 12 o'clock direction, the inner ring fault, and the ball fault respectively. Take "the same load", "the same rotational speed", "the same fault location", "the same fault depth", and "the same fault type" as the relationships of the knowledge graph to construct an undirected graph. In addition, use text description to describe the information in the data file, for example, in the text format of "Bearing X has a rotational speed of X rpm under load X, and an XX fault occurred at the X end, the fault depth is X, and now a fault signal file X is generated at a sampling frequency of X", to construct a multi-modal fault knowledge graph of attribute data and fault text, and the local structure of the graph is as Figure 3 shown.

[0019] S200, extract the knowledge prototype from the multi-modal fault knowledge graph using a knowledge feature extractor; In a specific embodiment of the present invention, S200 includes: S210, encode the attribute data in the multi-modal fault knowledge graph to obtain attribute encoded data; Specifically, S210 includes: S211, determine the first attribute data of the fault type and fault location from the multi-modal fault knowledge graph, and the remaining numerical second attribute data; S212, encode the first attribute data using one-hot encoding to obtain first attribute encoded data, and encode the second attribute data using normalization to obtain second attribute encoded data.

[0020] Before the multi-modal fault knowledge graph enters the knowledge feature extractor, it is necessary to encode the attribute data and perform semantic extraction on the fault text. The specific encoding method for the attribute data is that the fault type and fault location use one-hot encoding, and other numerical data use the normalization method. Taking the CWRU dataset as an example, normal operation is N, and the fault types O6, O3, O12, I, B represent the outer race fault at the 6 o'clock position, the outer race fault at the 3 o'clock position, the outer race fault at the 12 o'clock position, the inner race fault, and the ball fault, respectively, which can be represented by the one-hot encodings [1,0,0,0,0,0], [0,1, 0,0,0,0], [ 0,0, 1,0,0,0], [ 0,0,0,1,0,0], [0,0,0,0, 1,0], [0,0,0,0,0,1] respectively. The fault location is divided into the drive end, the fan end, and none, so they are encoded as [1,0,0], [0,1,0], [0,0,1] respectively. Other numerical attribute data are normalized according to the following formula: ; where, represents the normalized attribute data, is the original attribute data, is the minimum value of this type of attribute data, is the maximum value.

[0021] S220, for the fault text in the multi-modal fault knowledge graph, extract features using a semantic feature extraction method to obtain a fault description knowledge prototype; The formula used for feature extraction by the semantic feature extraction method is expressed as: ; ; ; ; ; ; In the formula, represents the fault description knowledge prototype, represents the extraction of the text language vector , is used to measure the difference between the co-occurrence probability and the independent probability of two elements (words / pixels), represents the th semantic unit, represents the th semantic unit, represents the Hadamard product, , and both represent intermediate parameters for forwarding, without specific meanings; represents the concatenation of vectors, represents the global semantics obtained by using the pre-trained Bert model to extract features, represents the th semantic unit and the th semantic unit co-occurrence frequency, represents the th semantic unit occurrence frequency, represents the th semantic unit occurrence frequency, and both represent intermediate parameters for forwarding, without specific meanings, and the superscript represents the word vector dimension, represents the sigmoid function, represents the entropy function, represents the word frequency, represents the inverse document frequency, represents a learnable weight matrix parameter, represents the vector representation output by the th layer of the Transformer encoder, is the weight coefficient of the th layer of the Transformer encoder, represents the total number of semantic units.

[0022] S230, generating the node feature matrix and the adjacency matrix from the attribute data; S240, constructing a knowledge feature extractor by using the data characteristics of the node feature matrix, the adjacency matrix, and the fault description knowledge prototype; Generate the node feature matrix and adjacency matrix of the atlas from the attribute data except the fault text. Among them, the node feature matrix has a dimension of , , and the adjacency matrix has a dimension of , , is the number of nodes, is the node feature dimension.

[0023] S250. Send the attribute encoding into the knowledge feature extractor for feature extraction to obtain the fault attribute knowledge prototype, and use it as the knowledge prototype.

[0024] Construct a knowledge feature extractor by combining the extraction results of the fault knowledge features and semantic features, and finally extract the fault attribute knowledge prototype. The extraction process of the knowledge feature extractor is expressed as: ; In the formula, represents the fault attribute knowledge prototype and also represents the finally output knowledge prototype, represents the learnable parameter, and the range is [0,1]; represents the feature extractor of the graph convolutional neural network.

[0025] S300. Divide the data set into a support set and a query set, and extract the data prototype from the support set by using the data feature extractor; The structure of the data feature extractor of the present invention refers to Figure 4 as shown, and mainly includes a one-dimensional convolutional layer, a time series convolutional module, a multi-level convolutional layer, a time series convolutional module, and a global average pooling layer; the time series convolutional module is composed of a causal convolutional layer, a ReLU activation function, a regularization layer, and a downsampling layer connected in sequence; the multi-level convolutional layer is composed of a one-dimensional convolutional layer, multiple pooling layers, and a feature splicing layer.

[0026] In a specific embodiment of the present invention, S300 includes: S310. Use the first fault categories in the data set and add the running data of 1 normal category as the base class; Among them, it is required that and the normal data label is included in the base class. The purpose of doing this is to make the base class data the main body of model training, and the model can be well generalized to new classes in the test stage.

[0027] S320. Use the running data of the last fault categories in the data set as new classes; S330, divide the base class into a training set and a test set according to a predetermined ratio; Among them, the predetermined ratio can be 4:1. Of course, it can also be adjusted according to the actual situation.

[0028] S340, divide the training set into a first support set and a first query set for meta-learning and divide the test set into a second support set and a second query set for meta-learning; S350, divide the bearing operation data of the new class into the second query set; Among them, there are 5 samples of each type in the first support set and the second support set, and 15 samples of each type in the first query set and the second query set. Divide the data of the test set in the same way, but the number of tasks (episodes) in the test set needs to be much larger than that in the training set.

[0029] S360, input the first support set into the data feature extractor to obtain a data prototype and input the first query set into the data feature extractor to obtain first query set features.

[0030] S400, fuse the knowledge prototype and the data prototype to obtain a fused prototype, and calculate the metric learning loss using the fused prototype, the knowledge prototype, and the data prototype; In a specific embodiment of the present invention, S400 includes: S410, fuse the knowledge prototype and the data prototype using a fusion formula to obtain a fused prototype; the fusion formula is expressed as: ; In the formula, represents the fused prototype, represents the data prototype, represents a learnable interaction term coefficient, represents the knowledge prototype, represents the gating matrix, represents concatenating and ; S420, calculate the metric learning loss using the fused prototype, the first query set features, the data prototype, and the knowledge prototype.

[0031] In a specific embodiment of the present invention, S420 includes: S421, calculate the cross-entropy loss between the fused prototype and the first query set features and use it as the first loss, which is expressed as: ; In the formula, represents the cross-entropy loss, Indicates the calculation of and the Euclidean distance between represents the first query set feature, and represents the label corresponding to the data sample of the first query set.

[0032] S422. Calculate the orthogonal regularization loss between the data prototype and the knowledge prototype, and use it as the second loss, expressed as:

[0033] In the formula, the superscript T represents the transpose, represents the square of the Frobenius norm of the knowledge prototype and the data prototype, aiming to force the decoupling of the feature spaces of the two types of prototypes.

[0034] The purpose of the second loss is to force the decoupling of the feature spaces of the two types of prototypes and separate the shared features and private features.

[0035] S423. Use the first loss and the second loss to calculate the metric learning loss, expressed as: ; In the formula, represents the metric learning loss, represents the first loss, represents the second loss, represents a hyperparameter that varies depending on the dataset, and its value range is [0, 1].

[0036] S500. Train the knowledge feature extractor and the data feature extractor through backpropagation until the metric learning loss converges, and obtain the trained knowledge feature extractor and data feature extractor; Among them, when training the knowledge feature extractor and the data feature extractor through backpropagation to optimize the parameters, the optimizer uses the Adam optimizer with default parameters. The convergence of the metric learning loss means that the metric learning loss is minimized or no longer changes.

[0037] In a specific embodiment of the present invention, after S500, the dual-prototype-driven intelligent fault diagnosis method based on multi-modal knowledge further includes: a. Input the second support set into the trained knowledge feature extractor and data feature extractor to obtain the knowledge prototype and the data prototype respectively; then fuse the knowledge prototype and the data prototype to obtain the fusion prototype of the second support set; b. Input the second query set into the data feature extractor to obtain the second query set feature; calculate the Euclidean distance between the fusion prototype of the second support set and the second query set feature; c. Calculate the classification evaluation index using the Euclidean distance in b; d. Evaluate the trained knowledge feature extractor and data feature extractor using the classification evaluation index.

[0038] In a meta - learning task (episode) during the testing process, the samples in the second support set are feature - extracted by the pre - trained feature extractor RTCN to obtain data prototypes, and the knowledge prototypes are fused in the training process steps to obtain fused prototypes. Then, the data in the second query set (including base classes and new classes) is input into RTCN for feature extraction to obtain data prototypes, and the distance between the fused prototypes and the data prototypes is calculated, and then the classification and the evaluation of the statistical accuracy rate are carried out. The formula for the evaluation index is as follows: ; where, Four types of statistical indicators are adopted, namely accuracy rate, precision rate, recall rate, and F1 - score; represents the fused prototype corresponding to the second support set, represents the data prototype corresponding to the second query set, represents the number of samples in the second query set.

[0039] S600. Receive the current operating data of the bearing and input it into the trained knowledge feature extractor and data feature extractor respectively to obtain the current fused prototype; use the current fused prototype to determine the fault category of the bearing.

[0040] When predicting the fault category of the bearing according to the current operating data of the bearing in the present invention, the existing fault operating data will be obtained synchronously. Take this fault operating data as the support set and the current operating data as the query set. The same as the testing process, the fault category with the closest distance in the support set can be obtained as the fault category of the bearing at the current time, realizing the prediction of the bearing fault category.

[0041] It should be noted that the terms "first" and "second" in the present invention are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, the meaning of "a plurality" is two or more unless otherwise specifically defined.

[0042] The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention belongs, without departing from the concept of the present invention, several simple deductions or substitutions can still be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. A dual-prototype-driven intelligent fault diagnosis method based on multimodal knowledge, characterized in that, Including: S100, compose a data set using the historical operation data of the bearing, and construct a multi-modal fault knowledge graph using the data set; among them, the label categories of the data set are , Among the label categories, there are multiple fault categories and 1 normal category; S200, extracting knowledge prototypes from the multi-modal fault knowledge graph by using a knowledge feature extractor; S300, dividing the data set into a support set and a query set, and extracting data prototypes from the support set by using a data feature extractor; S400, fusing the knowledge prototypes and the data prototypes to obtain fused prototypes, and calculating a metric learning loss by using the fused prototypes, knowledge prototypes, and data prototypes; S500, training the knowledge feature extractor and the data feature extractor through backpropagation until the metric learning loss converges, to obtain the trained knowledge feature extractor and data feature extractor; S600, receiving the current operation data of the bearing, and respectively inputting it into the trained knowledge feature extractor and data feature extractor to obtain the current fused prototype; determining the fault category of the bearing by using the current fused prototype.

2. The dual-prototype-driven intelligent fault diagnosis method based on multimodal knowledge according to claim 1, wherein S100 includes: S110, collecting the historical operation data of the bearing and forming a data set, where the data set includes multiple pieces of data recording fault information; the fault information includes fault depth, load type, fault type, fault location, and rotational speed; S120, assigning a label to each piece of data, and taking each label as a node and the fault information as the attribute data of the node; S130, taking the same load, the same rotational speed, the same fault location, the same fault depth, and the same fault type as the connection relationship between nodes, so as to construct an undirected graph; S140, using text to describe the fault information in each piece of data to obtain fault texts, and corresponding the fault texts to the attribute data, so as to obtain a multi-modal fault knowledge graph.

3. The dual-prototype-driven intelligent fault diagnosis method based on multimodal knowledge according to claim 2, wherein S200 includes: S210, encoding the attribute data in the multi-modal fault knowledge graph to obtain attribute encoded data; S220, extracting features from the fault texts in the multi-modal fault knowledge graph by using a semantic feature extraction method to obtain fault description knowledge prototypes; S230, generating a node feature matrix and an adjacency matrix from the attribute data; S240, constructing a knowledge feature extractor by using the data characteristics of the node feature matrix, the adjacency matrix, and the fault description knowledge prototypes; S250, inputting the attribute encoding into the knowledge feature extractor for feature extraction to obtain fault attribute knowledge prototypes.

4. The dual-prototype-driven intelligent fault diagnosis method based on multimodal knowledge according to claim 3, wherein S210 includes: S211, determining, from the multi-modal fault knowledge graph, first attribute data that are fault types and fault locations, and second attribute data that are the remaining numerical types; S212, encoding the first attribute data by using a one-hot encoding method to obtain first attribute encoded data, and encoding the second attribute data by using a normalization method to obtain second attribute encoded data.

5. The dual-prototype-driven intelligent fault diagnosis method based on multimodal knowledge according to claim 3, wherein, The formula used for feature extraction by the semantic feature extraction method in S220 is: ； ； ； ； ； ； Wherein, represents the fault description knowledge prototype, represents the extraction of the text language vector , which is used to measure the difference between the co-occurrence probability and the independent probability of two elements, represents the th semantic unit, represents the th semantic unit, represents the Hadamard product, , and both represent intermediate parameters for forwarding, without specific meanings; represents the concatenation of vectors, represents the global semantics obtained by using the pre-trained Bert model to extract the features, represents the co-occurrence frequency of the th semantic unit and the th semantic unit, represents the frequency of occurrence of the th semantic unit, represents the frequency of occurrence of the th semantic unit, and both represent intermediate parameters for forwarding, without specific meanings. The superscript represents the dimension of the word vector, represents the sigmoid function, represents the entropy function, represents the word frequency, represents the inverse document frequency, represents a learnable weight matrix parameter, represents the vector representation output by the th layer of the Transformer encoder, is the weight coefficient of the th layer of the Transformer encoder, represents the total number of semantic units.

6. The dual-prototype-driven intelligent fault diagnosis method based on multimodal knowledge according to claim 1, wherein S300 includes: S310, use the operation data of the first fault categories in the dataset and add the operation data of 1 normal category as the base class; S320, post the data in the dataset The operation data of the following failure categories as new categories; S330, dividing the base class into a training set and a test set according to a predetermined ratio; S340, dividing the training set into a first support set and a first query set for meta-learning and dividing the test set into a second support set and a second query set for meta-learning; S350, divide the bearing operation data of the new class into the second query set; S360, input the first support set into the data feature extractor to obtain a data prototype and input the first query set into the data feature extractor to obtain first query set features.

7. The dual-prototype-driven intelligent fault diagnosis method based on multimodal knowledge according to claim 6, wherein S400 includes: S410, fuse the knowledge prototype and the data prototype using a fusion formula to obtain a fused prototype; the fusion formula is expressed as: ； In the formula, represents the fusion prototype, represents the data prototype, represents the learnable interaction term coefficient, represents the knowledge prototype, represents the gating matrix, represents concatenating and ; represents the Hadamard product; S420, calculate a metric learning loss using the fused prototype, the first query set features, the data prototype, and the knowledge prototype.

8. The dual-prototype-driven intelligent fault diagnosis method based on multimodal knowledge according to claim 7, wherein, S420 includes: S421, calculate the cross-entropy loss between the fused prototype and the first query set features and use it as the first loss; S422, calculate the orthogonal regularization loss between the data prototype and the knowledge prototype and use it as the second loss; S423, calculate the metric learning loss using the first loss and the second loss.

9. The dual-prototype-driven intelligent fault diagnosis method based on multimodal knowledge according to claim 8, wherein The metric learning loss is expressed by the formula: ； In the formula, represents the metric learning loss, represents the first loss, represents the second loss, represents a dataset-dependent hyperparameter with a value range of [0, 1]; ； In the formula, represents the cross-entropy loss, represents the calculation of and the Euclidean distance between them, represents the first query set feature, represents the label corresponding to the data sample of the first query set; In the formula, the superscript T represents transpose, represents the square of the Frobenius norm of the knowledge prototype and the data prototype.

10. The dual-prototype-driven intelligent fault diagnosis method based on multimodal knowledge according to claim 6, wherein After S500, the dual-prototype driven intelligent fault diagnosis method based on multi-modal knowledge further includes: a, input the second support set into the trained knowledge feature extractor and data feature extractor to respectively obtain a knowledge prototype and a data prototype; then fuse the knowledge prototype and the data prototype to obtain a fused prototype of the second support set; b, input the second query set into the data feature extractor to obtain second query set features; calculate the Euclidean distance between the fused prototype of the second support set and the second query set features; c, calculate a classification evaluation metric using the Euclidean distance in b; d, evaluate the trained knowledge feature extractor and data feature extractor using the classification evaluation metric.

Images